Team Ecuador studies Algorithmic Differentiation (AD) of computer programs, blending :

**AD theory:** We study software engineering techniques, to
analyze and transform programs mechanically. Algorithmic Differentiation (AD)
transforms a program `P` that computes a function `P'`
that computes analytical derivatives of *adjoint mode* of AD,
a sophisticated transformation that yields gradients for optimization at a remarkably low cost.

**AD application to Scientific Computing:**
We adapt the strategies of Scientific Computing
to take full advantage of AD.
We validate our work on real-size applications.

We aim to produce AD code that can compete with hand-written sensitivity and adjoint programs used in the industry. We implement our algorithms into the tool Tapenade, one of the most popular AD tools at present.

Our research directions :

Efficient adjoint AD of frequent dialects e.g. Fixed-Point loops.

Development of the adjoint AD model towards Dynamic Memory Management.

Evolution of the adjoint AD model to keep in pace with with modern programming languages constructs.

Optimal shape design and optimal control for steady and unsteady simulations. Higher-order derivatives for uncertainty quantification.

Adjoint-driven mesh adaptation.

(AD, aka Automatic Differentiation) Transformation of a program, that returns a new program that computes derivatives of the initial program, i.e. some combination of the partial derivatives of the program's outputs with respect to its inputs.

Mathematical manipulation of the Partial Differential Equations that define a problem, obtaining new differential equations that define the gradient of the original problem's solution.

General trade-off technique, used in adjoint AD, that trades duplicate execution of a part of the program to save some memory space that was used to save intermediate results.

Algorithmic Differentiation (AD) differentiates
*programs*. The input of AD is
a source program

Any execution of

where each

which can be mechanically written as a sequence of instructions

The above computation of

**Sensitivities**, defined for a given direction

This expression is easily computed from right to left, interleaved with the original
program instructions. This is the *tangent mode* of AD.

**Adjoints**, defined after transposition (

This expression is most efficiently computed from right to left,
because matrix*adjoint mode* of AD, most effective for
optimization, data assimilation ,
adjoint problems , or inverse problems.

Adjoint AD builds a very efficient program Section 3.3,
which computes the gradient in a time independent from the number of parameters *tangent mode*
would require running the tangent differentiated program

However, the *inverse* of their computation order. If the
original program *overwrites* a part of

Another research issue is to make the AD model cope with the constant evolution of modern language constructs. From the old days of Fortran77, novelties include pointers and dynamic allocation, modularity, structured data types, objects, vectorial notation and parallel programming. We keep developing our models and tools to handle these new constructs.

Tree representation of a computer program, that keeps only the semantically significant information and abstracts away syntactic sugar such as indentation, parentheses, or separators.

Representation of a procedure body as a directed graph, whose nodes, known as basic blocks, each contain a sequence of instructions and whose arrows represent all possible control jumps that can occur at run-time.

Model that describes program static analysis
as a special sort of execution, in which all branches of control switches are taken
concurrently, and where computed values are replaced by abstract values
from a given *semantic domain*. Each particular analysis gives birth to
a specific semantic domain.

Program analysis that studies how a given property of variables evolves with execution of the program. Data Flow analysis is static, therefore studying all possible run-time behaviors and making conservative approximations. A typical data-flow analysis is to detect, at any location in the source program, whether a variable is initialized or not.

The most obvious example of a program transformation tool is certainly a compiler. Other examples are program translators, that go from one language or formalism to another, or optimizers, that transform a program to make it run better. AD is just one such transformation. These tools share the technological basis that lets them implement the sophisticated analyses required. In particular there are common mathematical models to specify these analyses and analyze their properties.

An important principle is *abstraction*: the core of a compiler
should not bother about syntactic details of the compiled program.
The optimization and code generation phases must be independent
from the particular input programming language. This is generally achieved
using language-specific *front-ends*, language-independent *middle-ends*,
and target-specific *back-ends*.
In the middle-end, analysis can concentrate on the semantics
of a reduced set of constructs. This analysis operates
on an abstract representation of programs made of one
*call graph*, whose nodes are themselves *flow graphs* whose
nodes (*basic blocks*) contain abstract *syntax trees* for the individual
atomic instructions.
To each level are attached symbol tables, nested to capture scoping.

Static program analysis can be defined on this internal representation,
which is largely language independent. The simplest analyses on trees can be
specified with inference rules , , .
But many *data-flow analyses* are more complex, and better defined on graphs than on trees.
Since both call graphs and flow graphs may be cyclic, these global analyses will be solved iteratively.
*Abstract Interpretation* is a theoretical framework to
study complexity and termination of these analyses.

Data flow analyses must be carefully designed to avoid or control
combinatorial explosion. At the call graph level, they can run bottom-up or top-down,
and they yield more accurate results when they take into account the different
call sites of each procedure, which is called *context sensitivity*.
At the flow graph level, they can run forwards or backwards, and
yield more accurate results when they take into account only the possible
execution flows resulting from possible control, which is called *flow sensitivity*.

Even then, data flow analyses are limited, because they are static and thus have very
little knowledge of actual run-time values. Far before reaching the very theoretical limit of
*undecidability*, one reaches practical limitations to how much information one can infer
from programs that use arrays , or pointers.
Therefore, conservative *over-approximations* must be made, leading to
derivative code less efficient than ideal.

In Scientific Computing, the mathematical model often consists of Partial Differential Equations, that are discretized and then solved by a computer program. Linearization of these equations, or alternatively linearization of the computer program, predict the behavior of the model when small perturbations are applied. This is useful when the perturbations are effectively small, as in acoustics, or when one wants the sensitivity of the system with respect to one parameter, as in optimization.

Consider a system of Partial Differential Equations
that define some characteristics of a system with respect to some
parameters. Consider one particular scalar characteristic.
Its sensitivity (or gradient) with respect to the parameters
can be defined by means of *adjoint* equations, deduced from the
original equations through linearization and transposition.
The solution of the adjoint equations is known as the adjoint state.

Scientific Computing provides reliable simulations
of complex systems. For example it is possible to *simulate*
the steady or unsteady 3D air flow around a plane that captures the physical phenomena
of shocks and turbulence. Next comes *optimization*,
one degree higher in complexity because it repeatedly simulates and
applies gradient-based optimization steps until an optimum is reached.
The next sophistication is *robustness*, that detects undesirable solutions which,
although maybe optimal, are very sensitive to uncertainty on design parameters or
on manufacturing tolerances. This makes second derivatives come into play.
Similarly *Uncertainty Quantification* can use second derivatives to evaluate how uncertainty on
the simulation inputs imply uncertainty on its outputs.

To obtain this gradient and possibly higher derivatives,
we advocate adjoint AD (*cf* )
of the program that discretizes and solves the direct system.
This gives the exact gradient of the discrete function
computed by the program, which is quicker and more sound than differentiating
the original mathematical equations .
Theoretical results guarantee convergence
of these derivatives when the direct program converges.
This approach is highly mechanizable. However, it requires
careful study and special developments of the AD model ,
to master possibly heavy memory usage.
Among these additional developments, we promote in particular
specialized AD models for Fixed-Point iterations , ,
efficient adjoints for linear algebra operators such as solvers, or exploitation
of parallel properties of the adjoint code.

Algorithmic Differentiation of programs gives sensitivities or gradients, useful for instance for :

optimum shape design under constraints, multidisciplinary optimization, and more generally any algorithm based on local linearization,

inverse problems, such as parameter estimation and in particular 4Dvar data assimilation in climate sciences (meteorology, oceanography),

first-order linearization of complex systems, or higher-order simulations, yielding reduced models for simulation of complex systems around a given state,

adaption of parameters for classification tools such as Machine Learning systems,
in which Adjoint Differentiation is also known as *backpropagation*.

mesh adaptation and mesh optimization with gradients or adjoints,

equation solving with the Newton method,

sensitivity analysis, propagation of truncation errors.

A CFD program computes the flow around a shape, starting from a number of inputs that define the shape and other parameters. On this flow one can define optimization criteria e.g. the lift of an aircraft. To optimize a criterion by a gradient descent, one needs the gradient of the criterion with respect to all inputs, and possibly additional gradients when there are constraints. Adjoint AD is the most efficient way to compute these gradients.

Inverse problems aim at estimating the value of hidden parameters from other measurable values, that depend on the hidden parameters through a system of equations. For example, the hidden parameter might be the shape of the ocean floor, and the measurable values of the altitude and velocities of the surface. Figure shows an example of an inverse problem using the glaciology code ALIF (a pure C version of ISSM ) and its AD-adjoint produced by Tapenade.

One particular case of inverse problems is *data assimilation*
in weather forecasting or in oceanography.
The quality of the initial state of the simulation conditions the quality of the
prediction. But this initial state is not well known. Only some
measurements at arbitrary places and times are available.
A good initial state is found by solving a least squares problem
between the measurements and a guessed initial state which itself must verify the
equations of meteorology. This boils down to solving an adjoint problem,
which can be done though AD .
The special case of *4Dvar* data assimilation is particularly challenging.
The 4^{th} dimension in “4D” is time, as available measurements are distributed
over a given assimilation period. Therefore the least squares mechanism must be
applied to a simulation over time that follows the time evolution model.
This process gives a much better estimation of the initial state, because
both position and time of measurements are taken into account.
On the other hand, the adjoint problem involved is more complex,
because it must run (backwards) over many time steps.
This demanding application of AD justifies our efforts in
reducing the runtime and memory costs of AD adjoint codes.

Simulating a complex system often requires solving a system of Partial Differential Equations.
This can be too expensive, in particular for real-time simulations.
When one wants to simulate the reaction of this complex system to small perturbations around a fixed
set of parameters, there is an efficient approximation: just suppose that the system
is linear in a small neighborhood of the current set of parameters. The reaction of the system
is thus approximated by a simple product of the variation of the parameters with the
Jacobian matrix of the system. This Jacobian matrix can be obtained by AD.
This is especially cheap when the Jacobian matrix is sparse.
The simulation can be improved further by introducing higher-order derivatives, such as Taylor
expansions, which can also be computed through AD.
The result is often called a *reduced model*.

Some approximation errors can be expressed by an adjoint state. Mesh adaptation can benefit from this. The classical optimization step can give an optimization direction not only for the control parameters, but also for the approximation parameters, and in particular the mesh geometry. The ultimate goal is to obtain optimal control parameters up to a precision prescribed in advance.

Keywords: Computational Fluid Dynamics - Turbulence

Functional Description: Aironum is an experimental software that solves the unsteady compressible Navier-Stokes equations with k-epsilon, LES-VMS and hybrid turbulence modelling on parallel platforms, using MPI. The mesh model is unstructured tetrahedrization, with possible mesh motion.

Participant: Alain Dervieux

Contact: Alain Dervieux

Keywords: Static analysis - Optimization - Compilation - Gradients

Scientific Description: Tapenade implements the results of our research about models and static analyses for AD. Tapenade can be downloaded and installed on most architectures. Alternatively, it can be used as a web server. Higher-order derivatives can be obtained through repeated application.

Tapenade performs sophisticated data-flow analysis, flow-sensitive and context-sensitive, on the complete source program to produce an efficient differentiated code. Analyses include Type-Checking, Read-Write analysis, and Pointer analysis. AD-specific analyses include the so-called Activity analysis, Adjoint Liveness analysis, and TBR analysis.

Functional Description: Tapenade is an Algorithmic Differentiation tool that transforms an original program into a new program that computes derivatives of the original program. Algorithmic Differentiation produces analytical derivatives, that are exact up to machine precision. Adjoint-mode AD can compute gradients at a cost which is independent from the number of input variables. Tapenade accepts source programs written in Fortran77, Fortran90, or C. It provides differentiation in the following modes: tangent, vector tangent, adjoint, and vector adjoint.

News Of The Year: - Continued development of multi-language capacity: AD of codes mixing Fortran and C - Continued front-end for C++ (using Clang-LLVM) - Preliminary work, including refactoring, in view of future Open-Source distribution

Participants: Laurent Hascoët and Valérie Pascual

Contact: Laurent Hascoët

Our goal is to extend Tapenade for C++. We further developed our external parser for C++, built
on top of Clang-LLVM https://

In the present development stage, this back-and-forth chain works on several small C++ codes. Still, many issues remain on the large example provided by the ABS team. We are working to solve those progressively. The lack of serious development documentation on the Clang internals obviously doesn't help.

The next development stage will be to adapt the analysis and differentiation components of Tapenade to the new Object constructs of the IR. This development has not started yet. Upstream this development, we need to devise an extended AD model correspondingly. This research part is in progress.

We extend Tapenade to differentiate codes that mix different languages, for both tangent and adjoint modes of AD. Our motivating application is Calculix, a 3-D Structural Finite Element code that mixes Fortran and C. This year we improved the memory representation of Tapenade's IR to handle the C-Fortran memory correspondence (commons, structs...) defined by the Fortran standard.

C files (aka “translation units”) and Fortran modules are two instances of the more general notion of “package” for which we have developed a unified representation in Tapenade, that also handles C++ namespaces.

We support industrial users with their first experiments of Algorithmic Differentiation of large in-house codes. This concerned two industrial codes this year.

One application is with ONERA on their ElsA CFD platform (Fortran 90). This is the continuation of a collaboration started in 2018. Both tangent and adjoint models of the kernel of ElsA were built successfully with Tapenade. This year's work was mostly about improving efficiency. It is worth noticing that this application was performed inside ONERA by ONERA engineers (Bruno Maugars, Sébastien Bourasseau, Cédric Content) with no need for installation of ElsA inside Inria. We take this as a sign of maturity of Tapenade. Our contribution is driven by development meetings, in which we point out some strategies and tool options to improve efficiency of the adjoint code. As a result from these discussions, we developed improved strategies and AD model, that will be useful to other tools. These improvements deal mostly with the adjoint of vectorized code. We prepared together an aticle that describes the architecture of a modular and AD-friendly ElsA, together with the corresponding extensions of the AD model of Tapenade. This article has been submitted to “Computers and Fluids”.

The other application ultimately targets AD of the “Jaguar” code, developed jointly by ONERA, CERFACS, and IMFT in Toulouse. This is a collaboration with Jose I. Cardesa and Christophe Airiau, both at IMFT. After a relatively easy tangent differentiation, most of the effort was devoted to obtaining a running and efficient adjoint code. Not too surprisingly, the main source of trouble was the Message-Passing parallel aspect on the application code. This underlined a lack of debugging support for adjoint-differentiated code. Given the run-time of the simulation that we consider (from hours to a few days on a 4110 processors platform), efficiency is crucial. We used the optimal binomial checkpointing scheme at the time-stepping level. However, performance of the adjoint code can probably be improved further with a better checkpointing scheme on the call tree. This calls in particular for AD-specific performance profiling tools, that we are planning to develop. We prepared together an article that describes this succesful experiment, which is now submitted to “Journal of Computational Science”.

Two collaborations are in preparation for next year, one with Jan Hueckelheim at Argonne National Lab. about SIMD parallel codes, and one with Stefano Carli at KU Leuven about adjoint AD of the plasma code SOLPS-ITER.

Reducing approximation errors as much as possible is a particular kind of optimal control problem. We formulate it exactly this way when we look for the optimal metric of the mesh, which minimizes a user-specified functional (goal-oriented mesh adaptation). In that case, the usual methods of optimal control apply, using adjoint states that can be produced by Algorithmic Differentiation.

This year, we published the final revised versions of two conference papers , , we published in a journal the final version of the adjoint-based mesh adaptation for Navier-Stokes flows ), and we published in “Numerical Methods in Fluids” a work on nonlinear correctors extending . Let us also mention the final publication of the book “Uncertainty Management for Robust Industrial Design in Aeronautics”, edited by C. Hirsch et al. in the Springer series Notes on Numerical Fluid Mechanics and Multidisciplinary Design (2019) in which we have contributed chapters 20, 21, 45, and 48.

The monography on mesh adaptation currently being written by Alauzet, Loseille, Koobus and Dervieux now involves all its chapters (14 chapters) and is being finalized.

Modeling turbulence is an essential aspect of CFD. The purpose of our work in hybrid RANS/LES (Reynolds Averaged Navier-Stokes / Large Eddy Simulation) is to develop new approaches for industrial applications of LES-based analyses. In the applications targetted (aeronautics, hydraulics), the Reynolds number can be as high as several tens of millions, far too high for pure LES models. However, certain regions in the flow can be predicted better with LES than with usual statistical RANS models. These are mainly vortical separated regions as assumed in one of the most popular hybrid models, the hybrid Detached Eddy Simulation (DES) model. Here, “hybrid” means that a blending is applied between LES and RANS. An important difference between a real life flow and a wind tunnel or basin is that the turbulence of the flow upstream of each body is not well known.

The development of hybrid models, in particular DES in the litterature, has raised the question of the domain of validity of these models. According to theory, these models should not be applied to flow involving laminar boundary layers (BL). But industrial flows are complex flows and often present regions of laminar BL, regions of fully developed turbulent BL and regions of non-equilibrium vortical BL. It is then mandatory for industrial use that the new hybrid models give a reasonable prediction for all these types of flow. We concentrated on evaluating the behavior of hybrid models for laminar BL and for vortical wakes. While less predictive than pure LES on laminar BL, some hybrid models still give reasonable predictions for rather low Reynolds numbers.

We have developed a new model relying on the hybridation of a DES model
based on a k-

High order approximations for compressible flows on unstructured meshes
are facing many constraints that increase their complexity
i.e. their computational cost.
This is clear for the largest class of approximation, the class of

The progress in highly accurate schemes for compressible flows on unstructured meshes (together with advances in massive parallelization of these schemes) allows us to solve problems previously out of reach. The three teams of Montpellier university (coordinator), Inria-Sophia and Keldysh Institute of Moscow have written a proposal for cooperation on the subject of the extension of these methods to simulate the noise emission of rotating machines (helicopters, future aerial vehicles, unmanned aerial vehicles, wind turbines...). The proposal has been selected by ANR and RSF (Russian Science Foundation) for support for a program duration of 4 years.

Laurent Hascoët is on the organizing commitee of the EuroAD Workshops
on Algorithmic Differentiation (http://

Laurent Hascoët was on the organizing and program committees of the workshop “Program Transformations for Machine Learning” at NeurIPS2019, Vancouver Canada, December 14th.

Laurent Hascoët was invited to give a talk on AD for the “GdR Calcul”, at “Institut de Physique du Globe”, Paris, January 24th.

Alain Dervieux is Scientific Director for the LEMMA company.