Commands is a team devoted to dynamic optimization, both for deterministic and stochastic systems. This includes the following approaches: trajectory optimization, deterministic and stochastic optimal control, stochastic programming, dynamic programming and Hamilton-Jacobi-Bellman equation.

Our aim is to derive new and powerful algorithms for solving numerically these problems, with applications in several industrial fields. While the numerical aspects are the core of our approach it happens that the study of convergence of these algorithms and the verification of their well-posedness and accuracy raises interesting and difficult theoretical questions, such as, for trajectory optimization: qualification conditions and second-order optimality condition, well-posedness of the shooting algorithm, estimates for discretization errors; for the Hamilton-Jacobi-Bellman approach: accuracy estimates, strong uniqueness principles when state constraints are present, for stochastic programming problems: sensitivity analysis.

For many years the team members have been deeply involved in various industrial applications. The Commands team itself has dealt since its foundation in 2007 with two main types of applications:

Space vehicle trajectories, in collaboration with CNES, the French space agency,

Production, management, storage and trading of energy resources (in collaboration with EDF, GDF and TOTAL).

Energy management for hybrid vehicles (in collaboration with Renault).

We give more details in the Application domain section.

This year is the last one of the "Opale" project (launchers trajectory optimization) with CNES that has been the occasion of several fruitful collaborations over the last years.

In Fall 2010 we have seen the defence of the PhD thesis of Francisco Silva, and the one of the Habilitation thesis of Hasnaa Zidani (let us mention that Oana Serea who was in "delegation" until September passed her habilitation thesis in December). We have also seen the beginning of the PhD theses of Xavier Dupuis, Laurent Pfeiffer and Zhiping Rao.

Several projects started this year: ALMA (analysis of leukaemia models, Digiteo DIM), BOCOP (optimal control Toolbox, INRIA), BiNoPe-HJ (parallel calculus toolbox for high dimensional HJB equations, with the SME HPC-Project), and others will start next year: SADCO (Sensitivity Analysis for Deterministic Controller Design, a Marie Curie European network), and "Energy optimization and control" (within the CIRIC project of research center in Chile).

The roots of deterministic optimal control are the “classical” theory of the calculus of variations, illustrated by the work of Newton, Bernoulli, Euler, and Lagrange (whose famous multipliers were introduced in ), with improvements due to the “Chicago school”, Bliss during the first part of the 20th century, and by the notion of relaxed problem and generalized solution (Young ).

*Trajectory optimization*really started with the
spectacular achievement done by Pontryagin's group
during the fifties, by stating,
for general optimal control problems, nonlocal optimality
conditions generalizing those of Weierstrass. This motivated
the application to many industrial problems (see the
classical books by Bryson and Ho
, Leitmann
, Lee and Markus
, Ioffe and Tihomirov
). Since then, various
theoretical achievements have been obtained by extending the
results to nonsmooth problems, see Aubin
, Clarke
, Ekeland
.

*Dynamic programming*was introduced and systematically
studied by R. Bellman during the fifties. The HJB equation,
whose solution is the value function of the (parameterized)
optimal control problem, is a variant of the classical
Hamilton-Jacobi equation of mechanics for the case of
dynamics parameterized by a control variable. It may be
viewed as a differential form of the dynamic programming
principle. This nonlinear first-order PDE appears to be
well-posed in the framework of
*viscosity solutions*introduced by Crandall and Lions
,
,
. These tools also allow to
perform the numerical analysis of discretization schemes. The
theoretical contributions in this direction did not cease
growing, see the books by Barles
and Bardi and Capuzzo-Dolcetta
.

The so-called
*direct methods*consist in an optimization of the
trajectory, after having discretized time, by a nonlinear
programming solver that possibly takes into account the
dynamic structure. So the two main problems are the choice of
the discretization and the nonlinear programming algorithm. A
third problem is the possibility of refinement of the
discretization once after solving on a coarser grid.

In the
*full discretization approach*, general Runge-Kutta
schemes with different values of control for each inner step
are used. This allows to obtain and control high orders of
precision, see Hager
, Bonnans
. In an interior-point algorithm
context, controls can be eliminated and the resulting system
of equation is easily solved due to its band structure.
Discretization errors due to constraints are discussed in
Dontchev et al.
. See also Malanowski et al.
.

In the
*indirect*approach, the control is eliminated thanks to
Pontryagin's maximum principle. One has then to solve the
two-points boundary value problem (with differential
variables state and costate) by a single or multiple shooting
method. The questions are here the choice of a discretization
scheme for the integration of the boundary value problem, of
a (possibly globalized) Newton type algorithm for solving the
resulting finite dimensional problem in
IR^{n}(
nis the number of state variables), and a methodology
for finding an initial point.

For state constrained problems the formulation of the shooting function may be quite elaborated , . As initiated in , we focus more specifically on the handling of discontinuities, with ongoing work on the geometric integration aspects (Hamiltonian conservation).

This approach consists in calculating the value function associated with the optimal control problem, and then synthesizing the feedback control and the optimal trajectory using Pontryagin's principle. The method has the great particular advantage of reaching directly the global optimum, which can be very interesting, when the problem is not convex.

*Characterization of the value function*From the dynamic
programming principle, we derive a characterization of the
value function as being a solution (in viscosity sense) of an
Hamilton-Jacobi-Bellman equation, wich is a nonlinear PDE of
dimension equal to the number n of state variables. Since the
pioneer works of Crandall and Lions
,
,
, many theoretical contributions
were carried out, allowing an understanding of the properties
of the value function as well as of the set of admissible
trajectories. However, there remains an important effort to
provide for the development of effective and adapted
numerical tools, mainly because of numerical complexity
(complexity is exponential with respect to n).

*Numerical approximation for continuous value
function*Several numerical schemes have been already
studied to treat the case when the solution of the HJB
equation (the value function) is continuous. Let us quote for
example the Semi-Lagrangian methods
,
studied by the team of M. Falcone
(La Sapienza, Rome), the high order schemes WENO, ENO,
Discrete galerkin introduced by S. Osher, C.-W. Shu, E.
Harten
,
,
,
, and also the schemes on
nonregular grids by R. Abgrall
,
. All these schemes rely on
finite differences or/and interpolation techniques which lead
to numerical diffusions. Hence, the numerical solution is
unsatisfying for long time approximations even in the
continuous case.

One of the (nonmonotone) schemes for solving the HJB equation is based on the Ultrabee algorithm proposed, in the case of advection equation with constant velocity, by Roe and recently revisited by Després-Lagoutière , . The numerical results on several academic problems show the relevance of the antidiffusive schemes. However, the theoretical study of the convergence is a difficult question and is only partially done.

*Optimal stochastic control problems*occur when the
dynamical system is uncertain. A decision typically has to be
taken at each time, while realizations of future events are
unknown (but some information is given on their distribution
of probabilities). In particular, problems of economic nature
deal with large uncertainties (on prices, production and
demand). Specific examples are the portfolio selection
problems in a market with risky and non-risky assets,
super-replication with uncertain volatility, management of
power resources (dams, gas). Air traffic control is another
example of such problems.

Sometimes this value function is smooth (e.g. in the case of Merton's portfolio problem, Oksendal ) and the associated HJB equation can be solved explicitly. Still, the value function is not smooth enough to satisfy the HJB equation in the classical sense. As for the deterministic case, the notion of viscosity solution provides a convenient framework for dealing with the lack of smoothness, see Pham , that happens also to be well adapted to the study of discretization errors for numerical discretization schemes , .

*Numerical approximation*for optimal stochastic control
problems. The numerical discretization of second order HJB
equations was the subject of several contributions. The book
of Kushner-Dupuis
gives a complete synthesis on the
chain Markov schemes (i.e Finite Differences,
semi-Lagrangian, Finite Elements, ...). Here a main
difficulty of these equations comes from the fact that the
second order operator (i.e. the diffusion term) is not
uniformly elliptic and can be degenerated. Moreover, the
diffusion term (covariance matrix) may change direction at
any space point and at any time (this matrix is associated
the dynamics volatility).

For solving stochastic control problems, we studied the so-called Generalized Finite Differences (GFD), that allow to choose at any node, the stencil approximating the diffusion matrix up to a certain threshold . Determining the stencil and the associated coefficients boils down to a quadratic program to be solved at each point of the grid, and for each control. This is definitely expensive, with the exception of special structures where the coefficients can be computed at low cost. For two dimensional systems, we designed a (very) fast algorithm for computing the coefficients of the GFD scheme, based on the Stern-Brocot tree .

We have mainly contributed to the following fields

Trajectory optimization for space launcher problems (with CNES).

Trading of Liquefied Natural Gas (with TOTAL).

Energy management for hybrid vehicles (with Renault).

The
Shoot2.0package
implements an indirect method for optimal control problems,
and is a complete rewrite of the previous
Shootsoftware.
New features include a parallel (OpenMP) grid shooting for
an easier intialization, as well as the generic handling of
mixed state-control constraints. The software also retains
the automatic switching detection and embedded continuation
of the previous version. Additionnal features under
development include the numerical minimization of the
Hamiltonian and the handling of pure state constraints. The
package has been used for the practical solving of
trajectory optimization problems for space launchers, and
is available at
http://

Developed in the framework of the PhD Thesis of J. Laurent-Varin, supported by CNES and ONERA. Implementation of an interior-point algorithm for multiarc trajectory optimization, with built-in refinement. Applied to several academic, launcher and reentry problems.

Developped since 2004 in C++ for solving the stochastic HJB equations in dimension 2. The code is based on the Generalized Finite Differences, and includes a decomposition of the covariance matrices in elementary diffusions pointing towards grid points. The implementation is very fast and was mainly tested on academic examples.

Developped in C++ for solving HJB equations in dimension 4. This code is based on the Ultra-Bee scheme and an efficient storage technique with sparse matrices.

Developped in C++ for solving HJB equations. This code does not depend on the dimension of the problem.

This is a project of toolbox in parallel calculus for solving high dimensional HJB equations. The project gathers mathematicians (F. Bonnans, O. Bokanowski, N. Forcadel, H. Zidani, J. Zhao), and researchers in computer science (P. Fiorini, T. Porcher) from he SME HPC-Project. This project has also the support of Inria (DTI), and takes now the form of an I-Lab.

This is an “ADT” (action of software development) project of toolbox in optimal control involving P. Martinon (chairman of the board, F. Bonnans, and V. Grélard. Its kernel will be based on open source software and in particular the IPOPT and ADOL-C facilities from COIN-OR, as well as the Maxima computer algebra system. The project began in October 2010. A white paper has been issued in December 2010.

We consider the optimal control problem of a class of integral equations with initial and final state constraints, as well as running state constraints. We prove Pontryagin's principle, and study the continuity of the optimal control and of the measure associated with first order state constraints. We also establish the Lipschitz continuity of these two functions of time for problems with only first order state constraints. The results were published as an INRIA report 7257 (2010) and in a journal .

In optimal control, there is a well-known link between the Hamilton-Jacobi-Bellman (HJB) equation and Pontryagin's Minimum Principle (PMP). Namely, the costate (or adjoint state) in PMP corresponds to the gradient of the value function in HJB. We investigate from the numerical point of view the possibility of coupling these two approaches to solve control problems. First a rough approximation of the value function is computed by the HJB method, and then used to obtain an initial guess for the PMP method. The advantage of our approach over other initialization techniques (such as continuation or direct methods) is to provide an initial guess close to the global minimum. Numerical tests have been conducted over simple problems involving multiple minima, discontinuous control, singular arcs and state constraints ( ).

Thanks to our recent achievments on the HJB (efficient numerical methods) and PMP approaches (numerical minimization of the Hamiltonian, grid shooting), we were able to apply this coupling method to a realistic space launcher problem (Ariane 5), in the framework of a research contract with the Cnes (french space agency).

This work aims to investigate a control problem governed by differential equations with Radon measure as data and with final state constraints. By using a known reparametrization method by Dal Maso and Rampazzo, we obtain that the value function can be characterized by means of an auxiliary control problem of absolutely continuous trajectories, involving time-measurable Hamiltonian. We study the characterization of the value function of this auxiliary problem and discuss its numerical approximations. A preprint corresponding to this paper is available .

In this work we consider a stochastic optimal control problem with convex control constraints. Using the variational approach, we are able to obtain first and second order expansions for the state and cost function, around a local minimum. This fact allows us to prove general first order necessary condition and, under a geometrical assumption over the constraint set, second order necessary conditions are also established. The result is published as report .

We consider a linear quadratic stochastic optimal control problem with non-negativity control constraints. The latter are penalized with the classical logarithmic barrier. Using a duality argument and the stochastic minimum principle, we provide an error estimate for the solution of the penalized problem which is the natural extension of the well known estimate in the deterministic framework. The result is published as report .

Within a collaboration with F. Camilli, we consider approximation schemes for monotone systems of fully nonlinear second order partial differential equations. We first prove a general convergence result for monotone, consistent and regular schemes. This result is a generalization to the well known framework of Barles-Souganidis, in the case of scalar nonlinear equation. Our second main result provides the convergence rate of approximation schemes for weakly coupled systems of Hamilton-Jacobi-Bellman equations. Examples including finite difference schemes and Semi-Lagrangian schemes are discussed.

We consider the problem of optimal management of energy contracts, with bounds on the local (time step) amounts and global (whole period) amounts to be traded, integer constraint on the decision variables and uncertainty on prices only. After building a finite state Markov chain by using vectorial quantization tree method, we rely on the stochastic dual dynamic programming (SDDP) method to solve the continuous relaxation of this stochastic optimization problem. An heuristic for computing sub optimal solutions to the integer optimization problem, based on the Bellman values of the continuous relaxation, is provided. Combining the previous techniques, we are able to deal with high-dimension state variables problems. Numerical tests applied to realistic energy markets problems have been performed. The results have been published in .

We consider a model of medium-term commodity contracts management. Randomness takes place only in the prices on which the commodities are exchanged whilst state variable is multi-dimensional. In , we proposed an algorithm to deal with such problem, based on quantization of random process and a dual dynamic programming type approach. We obtained accurate estimates of the optimal value and a suboptimal strategy from this algorithm. In this paper, we analyse the sensitivity with respect to parameters driving the price model. We discuss the estimate of marginal price based on the Danskin's theorem. Finally, some numerical results applied to realistic energy market problems have been performed. Comparisons between results obtained in and other classical methods are provided and evidence the accuracy of the estimate of marginal prices.The objective is to check how the optimal value changes when random process parameters, exactly the forward price in our model changes. This point is crucial for risk management and hedging strategy. Here, we give a partial response by applying Danskin theorem, which demands highly accuracy on optimal decision policy. We compare our algorithm with another traditional algorithm on the simple example of a swing option.

INRIA - RENAULT,
*Energy management for hybrid vehicles*, PhD.
fellowship of G. Granato, Dec 2009 - Nov 2012. Involved
researchers: F. Bonnans,
**H. Zidani**.

INRIA - TOTAL,
*Trading of Liquefied Natural Gas*, PhD fellowship
(CIFRE) of Y. Cen, Dec 2008 - Dec 2011. Involved
researchers:
**F. Bonnans**.

INRIA - CNES, R&T
*Trajectory optimization for climbing problems*,
2009-2010. Involved researchers: O. Bokanowski, P.
Martinon,
**H. Zidani**.

INRIA - HPC PROJECT,
*Bibliothèque numérique de calcul parallèle: Equations
HJB*, Dec 2009 - Dec 2011. Involved researchers: O.
Bokanowski, F. Bonnans, N. Forcadel,
**H. Zidani**.

ENSTA-DGA,
*Optimisation de trajectoires pour des systèmes
dynamiques de grande taille*, Jan 2010 - Dec 2010.
Involved researchers: O. Bokanowski,
**H. Zidani**.

F. Bonnans: Coorganizer of the
*Séminaire Parisien d'Optimisation*:

F. Bonnans: Coorganizer of the
*Conference of the FIME laboratory*, HEC,
Jouy-en-Josas, June 28-29, 2010.

S. Aronna:

(i)
*Second order sufficient conditions for a linear
Optimal Control Problem*. MODE 2010 (Conference on
optimization and decision making of Société française
de Mathématiques Appliquées et Industrielles). Limoges,
France. March 24th. 2010.

(ii)
*Second order conditions for bang-singular
extremals*. Séminaires des doctorants COMMANDS
(INRIA) et Optimisation et Commands (UMA-ENSTA). Paris,
France. June 18th. 2010.

(iii)
*Second order conditions for bang-singular
extremals*. Séminaires des doctorants Université
Paris VI. Paris, France. June 21st. 2010.

F. Bonnans:

(i)
*Interior-point algorithms for optimal control
problems*. Convex analysis, optimization and
applications. Les Houches, Jan. 5-8, 2010.

(ii)
*Interior-point algorithms for optimal control
problems*. Advanced methods and perspectives in
nonlinear optimization and control. Toulouse, STAE
Foundation, Feb. 3-5, 2010.

(iii)
*Optimal control of state constrained integral
equations*. JBHU 2010: Analyse Variationnelle,
Optimisation et Applications. Bayonne, October 25-27,
2010.

Z. Cen:

(i)
*LNG contract managements: optimal decision and
sensitivity analysis*. SMAI-MODE, Limoges, March
24-26, 2010.

(ii)
*Stochastic dual dynamic programming technique in
finance : pricing and sensitivity analysis – from swing
to LNG portfolio*. Workshop on numerical methods in
finance, Bordeaux, June 1-2, 2010.
http://

P. Martinon:

(i)
*Contrôle optimal et Principe de Pontryagin: tirer
meilleur parti de la puissance de calcul
disponible*. Journées MODE 2010, Limoges, France,
24-26 Mars, 2010.

F. Silva:

(i)
*Second order necessary optimality conditions for
stochastic optimal control problems*. SMAI-MODE,
Limoges, March 24-26, 2010.

F. Silva:
*Asymptotic expansions for interior penalty solutions
of some control constrained optimal control
problems*Summer school Optimal Control of Partial
Differential Equations at Cortona, Italy. July, 12-17,
2010.

F. Bonnans: coorganizer and lecturer of the CIMPA course on dynamic optimization, Tandil (Argentina), 30/8/10-10/9/10.

H. Zidani: lecturer of the CIMPA course on dynamic optimization, Tandil (Argentina), 30/8/10-10/9/10.

F. Bonnans:

*Second order conditions for bang-singular
extremals*. International conference on continuous
optimization (ICCOPT), Santiago, Chile. July 26-29,
2010.

Z. Cen:

(i)
*Stochastic dual dynamic programming and its
application in energy contracts portfolio
optimization*. International conference on
continuous optimization (ICCOPT), Santiago, Chile. July
26-29, 2010.

(ii)
*Cutting plane technique and its application in
energy contracts portfolio optimization*. 12th
International Conference on Stochastic Programming.
August 14-20, 2010, Halifax, Nova Scotia.

G. Granato:

*Stochastic Optimization of Hybrid Systems with
Activation Delay and Decision Lag*. International
conference on continuous optimization (ICCOPT),
Santiago, Chile. July 26-29, 2010.

F. Silva:

(i)
*Second order necessary optimality conditions for
stochastic optimal control problems*. International
conference on continuous optimization (ICCOPT),
Santiago, Chile. July 26-29, 2010.

H. Zidani:

*State-constrained optimal control problems lacking
controllabiltiy assumptions*. 9th Brazilian
Conference on Dynamics, Control and Their Applications
(DINCON), Sao Paulo, Brezil (Keynote speaker). June
07-11, 2010.

Andrei Dmitruk (Moscow State University), 2 weeks.

Pablo Lotito (U. Tandil, Argentina), 2 weeks.

Mohamed Mnif (ENIT, Tunis), 3 weeks.

Lars Grüne (University of Bayreuth), 1 week.

Antonio Siconolfi (University La Sapienza, Rome 1), 2 weeks.

F. Bonnans:

(i) Co-editor of the Journal “Series on Mathematics and its Applications” of the Annals of The Academy of Romanian Scientists (AOSR), since 2009.

(ii) Corresponding Editor, “ESAIM:COCV” (Control, Optimisation and Calculus of Variations),

(iii) Associate Editor of “Applied Mathematics and Optimization”, and “Optimization, Methods and Software”.

F. Bonnans is chair of the board of the SMAI-MODE group.

E. Trélat is Associate Editor of “ESAIM:COCV” (Control, Optimisation and Calculus of Variations) and of "International Journal of Mathematics and Statistics".

F. Bonnans: Professeur Chargé de
Cours, Ecole Polytechnique. M2 (second year of Master
studies): two courses on
*Stochastic programming*and
*Numerical methodes for partial differential equations
in finance*, 54 h (both courses in common Master
studies with the University of Paris VI). M1: comanager
of the Applied Mathematics program, supervisor in
internships in applied mathematics.

P. Martinon: Teaching Assistant, (i)
Ensta ParisTech:
*Quadratic Optimization*(15h) and
*Introduction à Matlab*(20h). (ii) ENSAE:
*Numerical Analysis*(16h).

F. Silva: Ensta ParisTech: Teaching Assistant in the course Quadratic Optimization, 16 hours.

Z. Rao: Teaching Assistant at Ensta ParisTech. (i) Introduction to automatic (15h) (ii) Markov chains (15h).

H. Zidani: Professeur Chargée de Cours
à l'Ensta ParisTech. (i) 1st year
*Continuous optimisation*(22h), (ii) 3rd year and
Master MMMEF Paris 1, course on Numerical methods for
finance (25h), (iii) 3rd year and Master ”Modélisation et
Simulation de Versailles St-Quentin et de l'INSTN“,
course on
*Numerical methods for front propagation*(22h), (iv)
M2 MIME Univ. Orsay:
*Optimal control*(22h).