Section: New Software and Platforms

AeroSol

Participants : Simon Delmas [Université de Bordeaux] , Sébastien de Brye [Université de Bordeaux] , Benjamin Lux [Cagire] , Nikolaos Pattakos [Cardamom] , Vincent Perrier [Cagire, correspondent] , Mario Ricchiuto [Cardamom] .

Developed since 2011 by V. Perrier in partnership with the Cardamom Inria team, the AeroSol library is a high order finite element library written in C++. The code design has been carried for being able to perform efficient computations, with continuous and discontinuous finite element methods on hybrid and possibly curvilinear meshes. The work of the Cardamom team is focused on continuous finite element methods, while we focus on discontinuous Galerkin methods. However, everything is done for sharing the largest possible part of code. The distribution of the unknowns is made with the software PaMPA, first developed within the Inria teams Bacchus and Castor, and currently maintained in the Tadaam team.

The generic features of the library are Adaptive wall treatment for a second moment closure in the industrial context

• High order. It can be theoretically any order of accuracy, but the finite element basis, and quadrature formula are implemented for having up to a fifth order of accuracy.

• Hybrid and curvilinear meshes. AeroSol can deal with up to fifth order conformal meshes composed of lines, triangles, quadrangles, tetrahedra, hexahedra, prism, and pyramids.

• Continuous and discontinuous discretization. AeroSol deals with both continuous and discontinuous finite element methods.

We would like to emphasize three assets of this library:

• Its development environment For allowing a good collaborative work and a functional library, a strong emphasis has been put on the use of modern collaborative tools for developing our software. This includes the active use of a repository, the use of CMake for the compilation, the constant development of unitary and functional tests for all the parts of the library (using CTest), and the use of the continuous integration tool Jenkins for testing the different configurations of AeroSol and its dependencies. Efficiency is regularly tested with direct interfacing with the PAPI library or with tools like scalasca.

• Its genericity A lot of classes are common to all the discretization, for example classes concerning I/O, finite element functions, quadrature, geometry, time integration, linear solver, models and interface with PaMPA. Adding simple features (e.g. models, numerical flux, finite element basis or quadrature formula) can be easily done by writing the class, and declaring its use in only one class of the code.

• Its efficiency This modularity is achieved by means of template abstraction for keeping good performances. Dedicated efficient implementation, based on the data locality of the discontinuous Galerkin method has been developed. As far as parallelism is concerned, we use point-to-point communications, the HDF5 library for parallel I/O. The behavior of the AeroSol library at medium scale (1000 to 2000 cores) was studied in [27].

The AeroSol project fits with the first axis of the Bordeaux Sud Ouest development strategy, which is to build a coherent software suite scalable and efficient on new architectures, as the AeroSol library relies on several tools developed in other Inria teams, especially for the management of the parallel aspects. At the end of 2015, AeroSol had the following features:

• Boundary conditions Periodic boundary conditions, time-dependent inlet and outlet boundary conditions. Adiabatic wall and isothermal wall. Steger-Warming based boundary condition. Synthetic Eddy Method for generating turbulence.

• C++/Fortran interface Tests for binding fortran with C++.

• Development environment An upgraded use of CMake for compilation (gcc, icc and xlc), CTest for automatic tests and memory checking, lcov and gcov for code coverage reports. A CDash server for collecting the unitary tests and the memory checking. An under development interface for functional tests. Optional linking with HDF5, PAPI, with dense small matrices libraries (BLAS, Eigen). An updated shared project Plafrim and joint project Aerosol/Scotch/PaMPA project on the continuous integration platform. An on-going integration of SPack for handling dependencies. A fixed ESSL interface.

• Finite elements up to fourth degree for Lagrange finite elements and hierarchical orthogonal finite element basis (with Dubiner transform on simplices) on lines, triangles, quadrangles, tetrahedra, prisms, hexaedra and pyramids. Finite element basis that are interpolation basis on Gauss-Legendre points for lines, quadrangles, and hexaedra, and triangle (only 1st and 2nd order).

• Geometry Elementary geometrical functions for first order lines, triangles, quadrangles, prisms, tetrahedra, hexaedra and pyramids. Handling of high order meshes.

• In/Out Link with the XML library for handling with parameter files. Parallel reader for GMSH, with an embedded geometrical pre-partitioner. Writer on the VTK-ASCII legacy format (cell and point centered). Parallel output in vtu and pvtu (Paraview) for cell-centered visualization, and XDMF/HDF5 format for both cell and point centered visualization. Ability of saving the high order solution and restarting from it. Computation of volumic and probe statistics. Ability of saving averaged layer data in quad and hexa meshes. Ability of defining user defined output visualization variables.

• Instrumentation Aerosol can give some traces on memory consumption/problems with an interfacing with the PAPI library. Tests have also been performed with VTUNE and TAU. Tests with Maqao and Scalasca (VIHPS workshop).

• Linear Solvers Link with the external linear solver UMFPack, PETSc and MUMPS. Internal solver for diagonal and block-diagonal matrices.

• Memory handling discontinuous and continuous, sequential and parallel discretizations based on PaMPA for generic meshes, including hybrid meshes.

• Models Perfect gas Euler system, real gas Euler system (template based abstraction for a generic equation of state), scalar advection, Waves equation in first order formulation, generic interface for defining space-time models from space models. Diffusive models: isotropic and anisotropic diffusion, compressible Navier-Stokes. Scalar advection-diffusion model. Linearized Euler equations, and Sutherland model for non isothermal diffusive flows. Shallow-water model.

• Multigrid Development of $p$-multigrid methods. This includes also the possibility of beginning a computation with an order and to decrease or increase the order of approximation when restarting. For the $p$ multigrid methods, $V$ and $W$ cycle have been developed, and restriction and prolongation opertors have also been developed. In progress implementation of $h$-multigrid, with the development of tests of the aggregation methods of PaMPA, and the definition of finite element basis on arbitrary cells.

• Numerical fluxes Centered fluxes, exact Godunov' flux for linear hyperbolic systems, and Lax-Friedrich flux. Riemann solvers for Low Mach flows. Numerical flux accurate for steady and unsteady computations.

• Numerical schemes Continuous Galerkin method for the Laplace problem (up to fifth order) with non consistent time iteration or with direct matrix inversion. Explicit and implicit discontinuous Galerkin methods for hyperbolic systems, diffusive and advection-diffusion problems. In progress optimization by stocking the geometry for advection problems. SUPG and Residual distribution schemes. Optimization of DG schemes for advection-diffusion problems: stocking of the geometry and use of BLAS for all the linear phases of the scheme.

• Parallel computing Mesh redistribution, computation of Overlap with PaMPA. Collective asynchronous communications (PaMPA based). Asynchronous point to point communications. Tests on the cluster Avakas from MCIA, and on Mésocentre de Marseille, and PlaFRIM. Tier-1 Turing (BlueGene). Weighted load balancing for hybrid meshes.

• Postprocessing High order projections over line postprocessing, possibility of stocking averaged data, such as the average flow and the Reynolds stresses.

• Quadrature formula up to 11th order for Lines, Quadrangles, Hexaedra, Pyramids, Prisms, up to 14th order for tetrahedron, up to 21st order for triangles. Gauss-Lobatto type quadrature formula for lines, triangles, quadrangles and hexaedra.

• Time iteration explicit Runge-Kutta up to fourth order, explicit Strong Stability Preserving schemes up to third order. Optimized CFL time schemes: SSP(2,3) and SSP(3,4). CFL time stepping. Implicit integration with BDF schemes from 2nd to 6th order Newton method for stationary problems. Implicit unstationary time iterator non consistent in time for stationary problems. Implementation of in house GMRES and conjugate gradient based on Jacobian free iterations.

• Validation Poiseuille, Taylor-Green vortex. Laplace equation on a ring and Poiseuille flow on a ring. Volumic forcing based on wall dissipation.Turbulent channel flow.

In 2016, the following features have been added:

• Geometric multigrid methods: aggregation of the mesh based on PaMPA, definition of finite element basis on arbitrary shape cells. Definition of geometry, quadratures and numerical schemes on aggregated finite elements.

• Sutherland law in the Navier-Stokes equations.

• Mass matrix free implementation of discontinuous Galerkin methods.

• Improvement of installation documentation. Spack based installation.

• Implementation of Boussinesq type models and Shallow water discretizations with well balancing, positivity preserving, wet-dry handling, limiters based on entropy viscosity,

• Implementation of Barotropic Euler equations

• Implementation of Taylor-based basis on simplices.