Many authors prefer to have a coarse discretization for the control variables (typically constant or piecewise-linear on each time step) and a higher order discretization for the state equation. The idea is both to have an accurate discretization of dynamics (since otherwise the numerical solution may be meaningless) and to obtain a small-scale resulting nonlinear programming problem. See e.g. Kraft . A typical situation is when a few dozen of time-steps are enough and there are no more than five controls, so that the resulting NLP has at most a few hundreds of unknowns and can be solved using full matrices software. On the other hand, the error order (assuming the problem to be unconstrained) is governed by the (poor) control discretization. Note that the integration scheme does not need to be specified (provided it allows to compute functions and gradients with enough precision) and hence general Ordinary Differential Equations integrators may be used.

On the other hand, a full discretization (i.e. in a context of Runge-Kutta methods, with different values of control for each inner substep of the scheme) allows to obtain higher orders that can be effectively computed, see Hager , Bonnans , being related to the theory of partitioned Runge-Kutta schemes, Hairer et al. . In an interior-point algorithm context, controls can be eliminated and the resulting system of equation is easily solved due to its band structure. Discretization errors due to constraints are discussed in Dontchev et al. . See also Malanowski et al. .

For large horizon problems integrating from the initial time to the final time may be impossible (finding a feasible point can be very hard !). Analogously to the indirect method of multiple shooting algorithm, a possibility is to add (to the control variables), as optimization parameters, the state variables for a given set of times, subject of course to “sticking” constraint. Note that once more the integration scheme does not need to be specified. Integration of the ODE can be performed in parallel. See Bock .

Recent proposals were made of methods based on a reformulation of the problem based on (possibly flat) output variables. By the definition, control and state variables are combinations of derivatives of these output variables. When the latter are represented on a basis of smooth functions such as polynomials, their derivatives are given linear combinations of the coefficients, and so the need for integration is avoided. One must of course take care of the possibly complicated expression of constraints that can make numerical difficulties. The numerical analysis of these methods seems largely open. See on this subject Petit, Milam and Murray .

The collocation approach for solving an ODE consists in a polynomial interpolation of the dynamic variable, the dynamic equation being enforced only at limited number of points (equal to the degree of the polynomial). Collocation can also be performed on each time step of a one-step method; it can be checked than collocation methods are a particular case of Runge-Kutta methods.

It is known that the polynomial interpolation with equidistant points is unstable for more than about 20 points, and that the Tchebycheff points should be preferred, see e.g. Section 5.2.6. Nevertheless, several papers suggested the use of pseudospectral methods Ross and Fahroo in which a single (over time) high-order polynomial approximation is used for the control and state. Therefore pseudospectral methods should not be used in the case of nonsmooth (e.g. discontinuous) control.

In view of model and data uncertainties there is a need for robust solutions. Robust optimization has been a subject of increasing importance in recent years see Ben-Tal and Nemirovski . For dynamic problems taking the worst-case of the perturbation at each time-step may be too conservative. Specific remedies have been proposed in specific contexts, see Ben-Tal et al. , Diehl and Björnberg .

A relatively simple method taking into account robustness, applicable to optimal control problems, was proposed in Diehl, Bock and Kostina .

The dominant ones (for optimal control problems as well as for other fields) have been successively the augmented Lagrangian approach (1969, due to Hestenes and Powell , see also Bertsekas ) successive quadratic programming (SQP: late seventies, due to , , and interior-point algorithms since 1984, Karmarkar . See the general textbooks on nonlinear programming , , .

When ordered by time the optimality system has a “band structure”. One can take easily advantage of this with interior-point algorithms whereas it is not so easy for SQP methods; see Berend et al. . There exist some very reliable SQP softwares SNOPT, some of them dedicated to optimal control problems, Betts , as well as robust interior-point software, see Morales et al. , Wächter and Biegler , and for application to optimal control Jockenhövel et al. .

We have developed a general SQP algorithm , for sparse nonlinear programming problems, and the associated software for optimal control problems; it has been applied to atmospheric reentry problems, in collaboration with CNES .

More recently, in collaboration with CNES and ONERA, we have developed a sparse interior-point algorithm with an embedded refinement procedure. The resulting TOPAZE code has been applied to various space trajectory problems , , . The method takes advantage of the analysis of discretization errors, is well-understood for unconstrained problems .

The choice of the discretization scheme for the numerical integration of the boundary value problem can have a great impact on the convergence of the method. First, the integration itself can be tricky. If the state equation is stiff (the linearized system has fast modes) then the state-costate has both fast and unstable modes. Also, discontinuities of the control or its derivative, due to commutations or changes in the set of active constraints, lead to the use of sophisticated variable step integrators and/or switching detection mechanisms, see Hairer et al. , . Another point is the computation of gradients for the Newton method, for which basic finite differences can give inaccurate results with variable step integrators (see Bock ). This difficulty can be treated in several ways, such as the so-called “internal differentiation” or the use of variational equations, see Gergaud and Martinon .

Most optimal control problems include control and state constraints. In that case the formulation of the TPBVP must take into account entry and exit times of boundary arcs for these constraints, and (for state constraints of order at least two) times of touch points (isolated contact points). In addition for state constrained problems, the so-called “alternative formulation” (that allows to eliminate the “algebraic variables, i.e. control and state, from the algebraic constraints) has to be used, see Hartl, Sethi and Vickson .

Another interesting point is the presence of singular arcs, appearing for instance when the control enters in the system dynamics and cost function in a linear way, which is common in practical applications. As for state contraints, the formulation of the boundary value problem must take into account these singular arcs, over which the expression of the optimal control typically involves higher derivatives of the Hamiltonian, see Goh and Robbins .

As mentioned before, finding a suitable initial point can be extremely difficult for indirect methods, due to the small convergence radius of the Newton type method used to solve the boundary value problem. Homotopy methods are an effective way to address this issue, starting from the solution of an easier problem to obtain a solution of the target problem (see Allgower and Georg ). It is sometimes possible to combine the homotopy approach with the Newton method used for the shooting, see Deuflhard .

With a given value of the initial costate are associated (through in integration of the reduced state-costate system) a control and a state, and therefore a cost function. The latter can therefore be minimized by ad-hoc minimization algorithms, see Dixon and Bartholomew-Biggs . The advantage of this point of view is the possibility to use the various descent methods in order to avoid convergence to a local maximum or saddle-point. The extension of this approach to constrained problems (especially in the case of state constraints) is an open and difficult question.

We have recently clarified under which properties shooting algorithms are well-posed in the presence of state constraints. The (difficult) analysis was carried out in , . A related homotopy algorithm, restricted to the case of a single first-order state constraint, has been proposed in .

We also conducted a study of optimal trajectories with singular arcs for space launcher problems. The results obtained for the generalized three-dimensional Goddard problem (see ) have been successfully adapted for the numerical solution of a realistic launcher model (Ariane 5 class).

Furthermore, we continue to investigate the effects of the numerical integration of the boundary value problem and the accurate computation of Jacobians on the convergence of the shooting method. As initiated in , we focus more specifically on the handling of discontinuities, with ongoing work on the geometric integration aspects (Hamiltonian conservation).

In , we have extended that kind of result to the case of state constraints: we described a complete classification, in terms of the geometry (Lie configuration) of the system, of local minimal time syntheses, in dimension two and three. This theoretical study was motivated by the problem of atmospheric re-entry posed by the CNES, and in , we showed how to apply this theory to this concrete problem, thus obtaining the precise structure of the optimal trajectory.

From the dynamic programming principle, we derive a characterization of the value function as being a solution (in viscosity sense) of an Hamilton-Jacobi-Bellman equation, wich is a nonlinear PDE of dimension equal to the number n of state variables. Since the pioneer works of Crandall and Lions , , , many theoretical contributions were carried out, allowing an understanding of the properties of the value function as well as of the set of admissible trajectories. However, there remains an important effort to provide for the development of effective and adapted numerical tools, mainly because of numerical complexity (complexity is exponential with respect to n).

Several numerical schemes have been already studied to treat the case when the solution of the HJB equation (the value function) is continuous. Let us quote for example the Semi-Lagrangian methods , studied by the team of M. Falcone (La Sapienza, Rome), the high order schemes WENO, ENO, Discrete galerkin introduced by S. Osher, C.-W. Shu, E. Harten , , , , and also the schemes on nonregular grids by R. Abgrall , . All these schemes rely on finite differences or/and interpolation techniques which lead to numerical diffusions. Hence, the numerical solution is unsatisfying for long time approximations even in the continuous case.

In a realistic optimal control problem, there are often constraints on the state (reaching a target, restricting the state of the system in an acceptable domain, ...). When some controlability assumptions are not satisfied, the value function associated to such a problem is discontinuous and the region of discontinuity is of great importance since it separates the zone of admissible trajectories and the nonadmissible zone.

In this case, it is not reasonable to use the usual numerical schemes (based on finite differences) for solving the HJB equation. Indeed, these schemes provide poor approximation quality because of the numerical diffusion.

Discrete approximations of the Hamilton-Jacobi equation for an optimal control problem of a differential-algebraic system were studied in .

Numerical methods for the HJB equation in a bilevel optimization scheme where the upper-level variables are design parameters were used in . The algorithm has been applied to the parametric optimization of hybrid car engines.

Within the framework of the thesis of N. Megdich, we have studied new antidiffusive schemes for HJB equations with discontinuous data , . One of these schemes is based on the Ultrabee algorithm proposed, in the case of advection equation with constant velocity, by Roe and recently revisited by Després-Lagoutière , . The numerical results on several academic problems show the relevance of the antidiffusive schemes. However, the theoretical study of the convergence is a difficult question and is only partially done .

In the standard case of minimization of an expectation (possibly of a utility function) a dynamic programming principleholds. Essentially, this says that the decision is a function of the present state (we can ignore the past) and that a certain reverse-time induction over the associated values holds. Unfortunately a straighforward resolution of the dynamic programming principle based on a discretization of the state space is out of reach (again this is the curse of dimensionality). For convex problems one can build lower convex approximations of the value function: this is the Stochastic dual dynamic programming(SDDP) approach, Pereira and Pinto . Another possibility is a parametric approximation of the value function; however determining the basis functions is not easy and identifying (or, we could say in this context, learning) the best parameters is a nonconvex problem, see however Bertsekas and J. Tsitsiklis , Munos .

A popular approach is to sample the uncertainties in a structured way of a tree(branching occurs typically at each time). Computational limits allow only a small number of branching, far less than the amount needed for an accurate solution Shapiro and Nemirovski . Such a poor accuracy may nevertheless (in the absence of a more powerful approach) be a good way for obtaining a reasonable policy. Very often the resulting programs are linear, possibly with integer variables (on-off switches of plants, investment decisions), allowing to use (possibly dedicated) mixed integer linear programming codes. The tree structure (coupling variables) can be exploited by the numerical algorithms, see Dantzig and Wolfe , Kall and Wallace .

By Monte Carlo we mean here sampling a given number of independent trajectories (of uncertainties). In the special case of optimal stopping (e.g., American options) it happens that the state space and the uncertainty space coincide. Then one can compute the transition probabilities of a Markov chain whose law approaches the original one, and then the problem reduces to the one of a Markov chain, see . Let us mention also the quantization approach, see .

In the general case a useful possibility is to compute a tree by agregating the original sample, as done in .

Maximizing the expectation of gains can lead to a solution with a too high probability of important losses (bankruptcy). In view of this it is wise to make a compromise between expected gains and risk of high losses. A simple and efficient way to achieve that may be to maximize the expectation of a utility function; this, however, needs an ad-hoc tuning. An alternative is the mean-variance compromise, presented in the case of portfolio optimization in Markowitz . A useful generalization of the variance, uncluding dissymetric functions such as semideviations, is the theory of deviation measures, Rockafellar et al. .

Another possibility is to put a constraint on the level of gain to be obtained with a high probability value say at least 99%. The corresponding concept of value-at-risk leads to difficult nonconvex optimization problems, although convex relaxations may be derived, see Shapiro and Nemirovski .

Yet the most important contribution of the recent years is the axiomatized theory of risk measures Artzner et al. , satisfying the properties of monotonicity and possibly convexity.

In a dynamic setting, risk measures (over the total gains) are not coherent (they do not obey a dynamic programming principle). The theory of coherent risk measuresis an answer in which risk measures over successive time steps are inductively applied; see Ruszczyński and Shapiro . Their drawback is to have no clear economic interpretation at the moment. Also, associated numerical methods still have to be developed.

The study of relations between chance constraints (constraints on the probability of some event) and robust optimization is the subject of intense research. The idea is, roughly speaking, to solve a robust optimization (some classes of which are tractable in the sense of algorithmic complexity). See the recent work by Ben-Tal and Teboulle .

Sometimes this value function is smooth (e.g. in the case of Merton's portfolio problem, Oksendal ) and the associated HJB equation can be solved explicitly. Still, the value function is not smooth enough to satisfy the HJB equation in the classical sense. As for the deterministic case, the notion of viscosity solution provides a convenient framework for dealing with the lack of smoothness, see Pham , that happens also to be well adapted to the study of discretization errors for numerical discretization schemes , .

The numerical discretization of second order HJB equations was the subject of several contributions. The book of Kushner-Dupuis gives a complete synthesis on the chain Markov schemes (i.e Finite Differences, semi-Lagrangian, Finite Elements, ...). Here a main difficulty of these equations comes from the fact that the second order operator (i.e. the diffusion term) is not uniformly elliptic and can be degenerated. Moreover, the diffusion term (covariance matrix) may change direction at any space point and at any time (this matrix is associated the dynamics volatility).

In the framework of the thesis of R. Apparigliato (that will finish at the end of 2007) we have studied the robust optimization approach to stochastic programming problems, in the case of hydroelectric production, for one valley. The main difficulty lies with both the dynamic character of the system and the large number of constraints (capacity of each dam). We have also studied the simplified electricity production models for respecting the “margin” constraint. In the framework of the thesis of G. Emiel and in collaboration with CEPEL, we have studied large-scale bundle algorithms for solving (through a dual “price decomposition” method) stochastic problems for the Brazilian case.

For solving stochastic control problems, we studied the so-called Generalized Finite Differences (GFD), that allow to choose at any node, the stencil approximating the diffusion matrix up to a certain threshold . Determining the stencil and the associated coefficients boils down to a quadratic program to be solved at each point of the grid, and for each control. This is definitely expensive, with the exception of special structures where the coefficients can be computed at low cost. For two dimensional systems, we designed a (very) fast algorithm for computing the coefficients of the GFD scheme, based on the Stern-Brocot tree . The GFD scheme was used as a basis for the approximation of an HJB equation coming from a super-replication problem. The problem was motivated by a study conducted in collaboration with Société Générale, see .

Within the framework of the thesis of Stefania Maroso, we also contributed to the study of the error estimate of the approximation of Isaac equation associated to a differential game with one player , and also for the approximation of HJB equation associated with the impulse problem .