3.3.1 Use of Runtime systems

Participants: Olivier Beaumont, Aurélien Esnard, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Laércio Lima Pilla, Philippe Swartvagher.

In previous works, our main goal was to study the methodology needed to efficiently exploit the new generation of high-performance computers with all the constraints that it induces (number of cores, heterogeneity, co-scheduling effects, etc.). To achieve this goal, we successfully proposed a methodology based on the use of modern task-based runtime systems to ensure both portability and performance portability (the ability to achieve high performance by only tuning few parameters of the application). This work was done in the context of several projects (ANR Solhar, ANR SOLHARIS, Projet Région HPC Scalable Ecosystem, etc.). The work done mainly targeted single multicore nodes equipped with several accelerator devices and the extension of these techniques to the multi-node case will be the focus of our future works, especially with the arrival of Philippe Swartvagher in the team. Indeed, it has been observed that in the context of distributed nodes, the placement strategies of runtime systems are insufficient and generate too much communication. In this context, it is therefore crucial to develop efficient placement strategies 41, 34. The extension of these mixed (static/dynamic) strategies in the case of tensors is largely open.

3.3.2 Design of compression techniques

Participants: Abdou Guermouche, Yulia Gusak, Mathieu Faverge, Pierre Ramet, Philippe Swartvagher.

The memory consumption of the applications has been and will remain an important challenge for solving larger problems that will lead to exascale computations. In the recent years we have demonstrated the interest of data compression techniques in linear solvers, both to save space and computations. Increasingly complex compression schemes require programming models to evolve to properly express the parallelism of these formats and to accommodate the increasing irregularity of applications. In TOPAL, we propose to continue the study of data compression techniques (low-rank, mixed precision, ...) in the context of solvers, but also in the context of training and multi-linear algebra. This part will be a very pertinent field for the study of applications over runtime systems, because of the strong irregularities that make the load balancing more complicated. At the same time, it is an original and promising approach for energy reduction. Representing convolutional / fully-connected weights in tensor formats is an effective way to reduce the parameters/FLOP in neural networks. However, post-quantization (reduction of parameters precision, for example, from float32 to int8) of networks with factorized weights yields a significant drop in accuracy. Due to memory/power consumption limitations of real devices, the quantization step is necessary, when pre-trained models are deployed. Therefore, our goal is to find algorithms that build tensorized neural networks, where weight factors are directly contain elements in low-precision format. Efficient implementation of operations on tensors represented in low-bit format will be required, as well as development of regularization techniques to tackle instability issues when training deep learning models with low-bit weights.

3.3.3 Energy minimization

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Laércio Lima Pilla.

Running computations with resource frugality is an important challenge, both for the upcoming exascale shift and for generally reducing the carbon impact of scientific computing. In addition to the usual objective of making computations run faster, we thus intend to design and evaluate our techniques and algorithms with the purpose of limiting their carbon footprint. In particular, given the lasting trend that the time and energy costs of computing are becoming ever lower than the costs of accessing and communicating data, we want to explore the tradeoffs of trading more computation for less data movements. This can be achieved in several ways: compression techniques as described above, replication of some computations, or use of lower precision. We are planning to work on this issue from two points of views: more frugal numerical algorithms, and energy-aware scheduling techniques. As for the embedded architectures in the phone, but also in the latest generation of laptops (Apple M1 Pro and Max chips), we are starting to see the emergence of Big-Little type technologies in the design of HPC oriented chips. In general, thermal design power (TDP) constraints push architects to increase the diversity and number of energy efficient circuits, even if they cannot all be powered simultaneously. If this hardware solution is very debatable from the point of view of carbon impact, it raises difficult and original questions about the optimization of computing performance under energy constraints. This kind of approach opens new perspectives, both from the point of view of scheduling algorithms but also in the design of computational kernels in linear algebra. We are also seeing the emergence of new processors (ARM or RISC-V technologies, Rhea from the SiPearl company within the EPI consortium, which should seriously compete with the supremacy of x86 architectures (Intel and AMD) with Nvidia accelerator cards in the search for a compromise between pure performance and energy sobriety.

In the field of training, a complementary opportunity is available. Indeed, contrary to classical HPC, the renewal of computational resources is often linked to the need to run larger models (and data with a better resolution to a lesser extent), rather than by the acceleration of computations. In this context, the possibility offered by tools such as Rotor 7.1.4 to limit memory requirements contributes to limiting the carbon footprint. Our goal is to extend the scope of these techniques, including to other fields of application than training. Our collaboration with Qarnot Computing is consistent with this objective. The co-design environment of the TextaRossa and Eupex projects 10 are also great avenues to explore these questions.

3.4.1 Task-based Linear Algebra and Tensor Computations

Participants: Olivier Beaumont, Aurélien Esnard, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Pierre Ramet, Philippe Swartvagher.

We plan to continue our activity on task-based linear algebra to find solutions for expressing high level algorithms in an elegant way while ensuring high performance. First, we want to consider the expressivity of the algorithms for large scale distributed architectures while considering the specific problems of scheduling, data and task mapping, and data granularity. This work will be done in tight collaboration with the Storm and Tadaam teams and is a key objective of the ANR SOLHARIS project. Moreover, the foundations of this topic fall back to the HiePACS project. Thus, we plan to collaborate and exchange with the CONCACE team on topics which are of interest to both teams (mainly expressivity and scalability). Second, as mentioned above, we plan to study data compression techniques in linear algebra 48, 53, 57, which brings new algorithmic schemes that are outside of the scope of the classical programming model used until now. As mid and long term objectives, we would like to find new ways to express these linear algebra algorithms to efficiently exploit large heterogeneous architectures. A second research topic focuses on the extension of the techniques developed in the framework of linear algebra, in particular with the Chameleon library, to multi-linear algebra and tensors. The idea is to build on the expertise we have in the field of compression and in the use of runtimes to use heterogeneous resources in particular.

Another challenge would be to redesign the graph partitioning & matrix ordering algorithms in a task-based runtime, in order to facilitate the integration of this basic building block in modern tasked-based solvers. This work has already been initiated in the StarPart 7.1.2 project.

3.4.2 Multi-Linear Algebra and Tensor Decompositions

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Mathieu Faverge, Abdou Guermouche, Julia Gusak, Pierre Ramet.

Tensor decompositions are a natural extension of SVD-type decompositions in linear algebra. Unlike linear algebra, there are several types of decompositions, which play an important role in the analysis of large data and in the compression of networks, in particular to increase the efficiency of inference. The arrival of Julia Gusak in the project allows us to reinforce this competence. In addition to the basic kernels to be integrated in Chameleon proposed in the Topic 3.4.1, we will propose distributed tensor decomposition algorithms compression algorithms, focusing mainly on the case of small but large tensors, which is the most common in the context of neural networks.

3.4.3 Energy Minimization in Linear Solvers

Participants: Mathieu Faverge, Abdou Guermouche.

We plan to investigate how to reduce the energy consumption of linear algebra libraries (either sparse or dense). To do so we will rely on an algorithmic approach rather than a system approach. The idea, in a first step, is to consider several implementations of a same kernel and select the implementation while taking into account energy consumption 32, 31, 33. For instance a low-rank implementation of a given operation will be slower than a regular high-performance implementation but it will tend to require less energy. In the longer term, we plan also to investigate how to design energy efficient implementations of basic kernels. They will then be used within higher level algorithms in order to find a better trade-off between energy consumption and high performance. In the context of developing linear algebra solvers using compression techniques, a research axis we would like to develop is the energy consumption study of these solvers: is it possible to provide computation kernels with different energy consumption levels that can be easily exchanged to lower the final energy consumption of the application while keeping the same numerical accuracy. Low-rank compression techniques, as well as mixed-precision solution are envisioned toward this objective.

3.4.4 Task-based Approaches for Deep Learning

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Mathieu Faverge, Julia Gusak, Abdou Guermouche, Laércio Lima Pilla, Pierre Ramet, Philippe Swartvagher.

In popular Deep Learning frameworks like TensorFlow or PyTorch, the parallelization of the training process is performed with a large granularity, mostly relying on Data Parallelism. Specialized frameworks have been proposed to explore finer parallel schemes, like PipeDream for model parallelism 59. These implementations are however very static and require explicit and error-prone data management policies. We believe that our expertise in using task-based runtime systems can be used to provide much simpler approaches for a finer grain control on the execution of the corresponding task graphs and communications patterns, for both training and inference phases. We plan to design a prototype implementation that would allow to easily use clever scheduling and optimization techniques to improve the performance of inference. In the longer term, we expect that this approach will provide better scalability and flexibility, and unlock new opportunities for optimization, for a wide range of deep learning applications.

3.4.5 Tensor Compression for Inference

Participants: Olivier Beaumont, Julia Gusak.

We envision a more exploratory research activity around the use of tensor compression for inference. Initially, the objective is to use tensor compression techniques and quantization to allow inference to be performed with little memory or low latency. These techniques can also be further extended in the context of online training performed after installation on the device itself, which requires in particular memory-saving approaches. Finally, an even more ambitious goal would be to combine these approaches with techniques for designing neural networks that are inherently efficient in terms of memory needs, such as extensions of RevNets 49, 52, 61.

3.4.6 Carbon Saving and Energy-Efficient Training

Participants: Olivier Beaumont, Lionel Eyraud Dubois, Julia Gusak, Laércio Lima Pilla.

The training phase of Deep Neural Networks is notoriously very resource-hungry, especially regarding its energy consumption. In the last years, we have proposed several algorithmic solutions (re-materialization 6, offloading 38, their combination 36, pipelining 39) to reduce the resource consumption of this training phase, with a focus on reducing the training time. We plan to broaden the scope of these studies, by also taking into account the energy usage. A heterogeneous context and a flexible runtime system, as planned in Topic 3.4.4, may also be an opportunity to reduce energy consumption by allocating some tasks, typically the non-critical ones, to the most efficient resources for them, or by selecting a different implementation with better energy efficiency. This can be seen as a generalization of mixed-precision techniques, which are also very popular in this context to help achieving a better frugality. However, care must be taken to not degrade the convergence of the training phase. Moreover, the carbon footprint comes essentially from the manufacturing 60, 51 of the computing resources (GPUs) and the main goal is to facilitate their non-renewal, as enabled by memory saving techniques.

5.2.1 Carbon Impact of Cloud Platforms

To limit the environmental impact of Qarnot focuses on re-using the heat produced by computations in heat circuits or boilers. As part of the Pulse Inria challenge, we are working with Qarnot on algorithms for placing computations on their infrastructure, so as to maximize the use of reusable heat sources, depending on computation demand and task characteristics. The aim is to enable users of the Qarnot platform to specify their objective function on the (carbon footprint, time, cost) axes, and to be able to meet it.

5.2.2 Democratization of Large Models Training

In the context of training, at one end of the spectrum we see the provision of computing resources, such as the Jean Zay supercomputer, whose efficient use requires large-scale parallel training algorithms and frameworks to optimize resource utilization and accelerate time to discovery. At the other end of the spectrum, we see the importance of enabling researchers from different communities to use the resources at their disposal (often just a few GPUs) to develop original models without being constrained by hardware limitations. In particular, recent transformer-based models are very heavy-weight, and techniques must be employed to run them on GPUs that are only a few years old, without compromising data quality, computational accuracy, or model size. In particular, the Topal team has been working for several years on memory-saving strategies to enable the training of large models on limited-capacity resources (re-materialization and offloading), and on software 7 such as Rotor and Rockmate, which are recognized and visible in the AI applications community and enable researchers with access to limited capacity resources to train large models.

7.1.1 Chameleon

• Keywords:
  Runtime system, Task-based algorithm, Dense linear algebra, HPC, Task scheduling
• Scientific Description:

  Chameleon is part of the MORSE (Matrices Over Runtime Systems @ Exascale) project. The overall objective is to develop robust linear algebra libraries relying on innovative runtime systems that can fully benefit from the potential of those future large-scale complex machines.

  We expect advances in three directions based first on strong and closed interactions between the runtime and numerical linear algebra communities. This initial activity will then naturally expand to more focused but still joint research in both fields.

  1. Fine interaction between linear algebra and runtime systems. On parallel machines, HPC applications need to take care of data movement and consistency, which can be either explicitly managed at the level of the application itself or delegated to a runtime system. We adopt the latter approach in order to better keep up with hardware trends whose complexity is growing exponentially. One major task in this project is to define a proper interface between HPC applications and runtime systems in order to maximize productivity and expressivity. As mentioned in the next section, a widely used approach consists in abstracting the application as a DAG that the runtime system is in charge of scheduling. Scheduling such a DAG over a set of heterogeneous processing units introduces a lot of new challenges, such as predicting accurately the execution time of each type of task over each kind of unit, minimizing data transfers between memory banks, performing data prefetching, etc. Expected advances: In a nutshell, a new runtime system API will be designed to allow applications to provide scheduling hints to the runtime system and to get real-time feedback about the consequences of scheduling decisions.

  2. Runtime systems. A runtime environment is an intermediate layer between the system and the application. It provides low-level functionality not provided by the system (such as scheduling or management of the heterogeneity) and high-level features (such as performance portability). In the framework of this proposal, we will work on the scalability of runtime environment. To achieve scalability it is required to avoid all centralization. Here, the main problem is the scheduling of the tasks. In many task-based runtime environments the scheduler is centralized and becomes a bottleneck as soon as too many cores are involved. It is therefore required to distribute the scheduling decision or to compute a data distribution that impose the mapping of task using, for instance the so-called “owner-compute” rule. Expected advances: We will design runtime systems that enable an efficient and scalable use of thousands of distributed multicore nodes enhanced with accelerators.

  3. Linear algebra. Because of its central position in HPC and of the well understood structure of its algorithms, dense linear algebra has often pioneered new challenges that HPC had to face. Again, dense linear algebra has been in the vanguard of the new era of petascale computing with the design of new algorithms that can efficiently run on a multicore node with GPU accelerators. These algorithms are called “communication-avoiding” since they have been redesigned to limit the amount of communication between processing units (and between the different levels of memory hierarchy). They are expressed through Direct Acyclic Graphs (DAG) of fine-grained tasks that are dynamically scheduled. Expected advances: First, we plan to investigate the impact of these principles in the case of sparse applications (whose algorithms are slightly more complicated but often rely on dense kernels). Furthermore, both in the dense and sparse cases, the scalability on thousands of nodes is still limited, new numerical approaches need to be found. We will specifically design sparse hybrid direct/iterative methods that represent a promising approach.

  Overall end point. The overall goal of the MORSE associate team is to enable advanced numerical algorithms to be executed on a scalable unified runtime system for exploiting the full potential of future exascale machines.

• Functional Description:
  Chameleon is a dense linear algebra software relying on sequential task-based algorithms where sub-tasks of the overall algorithms are submitted to a Runtime system. A Runtime system such as StarPU is able to manage automatically data transfers between not shared memory area (CPUs-GPUs, distributed nodes). This kind of implementation paradigm allows to design high performing linear algebra algorithms on very different type of architecture: laptop, many-core nodes, CPUs-GPUs, multiple nodes. For example, Chameleon is able to perform a Cholesky factorization (double-precision) at 80 TFlop/s on a dense matrix of order 400 000 (i.e. 4 min 30 s).
• Release Contributions:

  Chameleon includes the following features:

  - BLAS 3, LAPACK one-sided and LAPACK norms tile algorithms - Support QUARK and StarPU runtime systems and PaRSEC since 2018 - Exploitation of homogeneous and heterogeneous platforms through the use of BLAS/LAPACK CPU kernels and cuBLAS/MAGMA CUDA kernels - Exploitation of clusters of interconnected nodes with distributed memory (using OpenMPI)

• URL:
  https://­gitlab.­inria.­fr/­solverstack/­chameleon
• Contact:
  Mathieu Faverge
• Participants:
  Samuel Thibault, Emmanuel Agullo, Florent Pruvost, Mathieu Faverge
• Partners:
  Innovative Computing Laboratory (ICL), King Abdullha University of Science and Technology, University of Colorado Denver

7.1.2 StarPart

• Functional Description:
  StarPart is a flexible and extensible framework that integrates state-of-the-art methods for graph partitioning and sparse matrix ordering. More precisely, StarPart is a framework that offers a uniform API to manipulate graph, hypergraph and mesh structures. It is designed to be easily extensible by adding new methods and to plug all these methods into a comprehensive framework. It is initially designed to provide graph partitioning and sparse matrix ordering methods, that come from sate-of-the-art software such as Metis, Scotch, Patoh, Zoltan, etc. Besides, it provides some facilities for IO, diagnostic, benchmark, visualization (VTK, SVG, ...). StarPart is the core of the MetaPart project. It is built upon the LibGraph library.
• URL:
  https://­gitlab.­inria.­fr/­metapart/­starpart
• Contact:
  Aurélien Esnard
• Participant:
  Aurélien Esnard

7.1.3 PaStiX

• Name:
  Parallel Sparse matriX package
• Keywords:
  Direct solvers, Parallel numerical solvers, Linear Systems Solver
• Scientific Description:
  PaStiX is based on an efficient static scheduling and memory manager, in order to solve 3D problems with more than 50 million of unknowns. The mapping and scheduling algorithm handles a combination of 1D and 2D block distributions. A dynamic scheduling can also be applied to take care of NUMA architectures while taking into account very precisely the computational costs of the BLAS 3 primitives, the communication costs and the cost of local aggregations.
• Functional Description:

  PaStiX is a scientific library that provides a high performance parallel solver for very large sparse linear systems based on block direct and block ILU(k) methods. It can handle low-rank compression techniques to reduce the computation and the memory complexity. Numerical algorithms are implemented in single or double precision (real or complex) for LLt, LDLt and LU factorization with static pivoting (for non symmetric matrices having a symmetric pattern). The PaStiX library uses the graph partitioning and sparse matrix block ordering packages Scotch or Metis.

  The PaStiX solver is suitable for any heterogeneous parallel/distributed architecture when its performance is predictable, such as clusters of multicore nodes with GPU accelerators or KNL processors. In particular, we provide a high-performance version with a low memory overhead for multicore node architectures, which fully exploits the advantage of shared memory by using a hybrid MPI-thread implementation.

  The solver also provides some low-rank compression methods to reduce the memory footprint and/or the time-to-solution.

• URL:
  https://­gitlab.­inria.­fr/­solverstack/­pastix
• Publications:
  inria-00346017, inria-00346018, hal-01485507, hal-01824275, hal-03361299, hal-04527103
• Contact:
  Pierre Ramet
• Participants:
  Alycia Lisito, Grégoire Pichon, Mathieu Faverge, Pierre Ramet

7.1.4 rotor

• Name:
  Re-materializing Optimally with pyTORch
• Keywords:
  Deep learning, Optimization, Python, GPU, Automatic differentiation
• Scientific Description:

  This software implements in PyTorch a new activation checkpointing method which allows to significantly decrease memory usage when training Deep Neural Networks with the back-propagation algorithm. Similarly to checkpointing techniques coming from the literature on Automatic Differentiation, it consists in dynamically selecting the forward activations that are saved during the training phase, and then automatically recomputing missing activations from those previously recorded. We propose an original computation model that combines two types of activation savings: either only storing the layer inputs, or recording the complete history of operations that produced the outputs (this uses more memory, but requires fewer recomputations in the backward phase), and we provide in https://hal.inria.fr/hal-02352969 an algorithm to compute the optimal computation sequence for this model.

  Our PyTorch implementation processes the entire chain, dealing with any sequential DNN whose internal layers may be arbitrarily complex and automatically executing it according to the optimal checkpointing strategy computed given a memory limit. In https://hal.inria.fr/hal-02352969, through extensive experiments, we show that our implementation consistently outperforms existing checkpoint-ing approaches for a large class of networks, image sizes and batch sizes.

• Functional Description:
  Allows to train very large convolutional networks on limited memory by optimally selecting which activations should be kept and which should be recomputed. This code is meant to replace the checkpoint.py utility available in pytorch, by providing more efficient rematerialization strategies. The algorithm is easier to tune: the only required parameter is the available memory, instead of the number of segments.
• URL:
  https://­gitlab.­inria.­fr/­hiepacs/­rotor
• Publication:
  hal-02352969
• Contact:
  Lionel Eyraud Dubois
• Participants:
  Olivier Beaumont, Alena Shilova, Alexis Joly, Lionel Eyraud Dubois, Julien Herrmann

7.1.5 StarPU

• Name:
  The StarPU Runtime System
• Keywords:
  Runtime system, High performance computing
• Scientific Description:

  Traditional processors have reached architectural limits which heterogeneous multicore designs and hardware specialization (eg. coprocessors, accelerators, ...) intend to address. However, exploiting such machines introduces numerous challenging issues at all levels, ranging from programming models and compilers to the design of scalable hardware solutions. The design of efficient runtime systems for these architectures is a critical issue. StarPU typically makes it much easier for high performance libraries or compiler environments to exploit heterogeneous multicore machines possibly equipped with GPGPUs or Cell processors: rather than handling low-level issues, programmers may concentrate on algorithmic concerns.Portability is obtained by the means of a unified abstraction of the machine. StarPU offers a unified offloadable task abstraction named "codelet". Rather than rewriting the entire code, programmers can encapsulate existing functions within codelets. In case a codelet may run on heterogeneous architectures, it is possible to specify one function for each architectures (eg. one function for CUDA and one function for CPUs). StarPU takes care to schedule and execute those codelets as efficiently as possible over the entire machine. In order to relieve programmers from the burden of explicit data transfers, a high-level data management library enforces memory coherency over the machine: before a codelet starts (eg. on an accelerator), all its data are transparently made available on the compute resource.Given its expressive interface and portable scheduling policies, StarPU obtains portable performances by efficiently (and easily) using all computing resources at the same time. StarPU also takes advantage of the heterogeneous nature of a machine, for instance by using scheduling strategies based on auto-tuned performance models.

  StarPU is a task programming library for hybrid architectures.

  The application provides algorithms and constraints: - CPU/GPU implementations of tasks, - A graph of tasks, using StarPU's rich C API.

  StarPU handles run-time concerns: - Task dependencies, - Optimized heterogeneous scheduling, - Optimized data transfers and replication between main memory and discrete memories, - Optimized cluster communications.

  Rather than handling low-level scheduling and optimizing issues, programmers can concentrate on algorithmic concerns!

• Functional Description:
  StarPU is a runtime system that offers support for heterogeneous multicore machines. While many efforts are devoted to design efficient computation kernels for those architectures (e.g. to implement BLAS kernels on GPUs), StarPU not only takes care of offloading such kernels (and implementing data coherency across the machine), but it also makes sure the kernels are executed as efficiently as possible.
• Release Contributions:
  StarPU is a runtime system that offers support for heterogeneous multicore machines. While many efforts are devoted to design efficient computation kernels for those architectures (e.g. to implement BLAS kernels on GPUs), StarPU not only takes care of offloading such kernels (and implementing data coherency across the machine), but it also makes sure the kernels are executed as efficiently as possible.
• URL:
  https://­starpu.­gitlabpages.­inria.­fr/
• Publications:
  tel-04213186, inria-00326917, inria-00378705, inria-00384363, inria-00411581, inria-00421333, inria-00467677, inria-00523937, inria-00547614, inria-00547616, inria-00547847, inria-00550877, inria-00590670, inria-00606195, inria-00606200, inria-00619654, hal-00643257, hal-00648480, hal-00654193, hal-00661320, hal-00697020, hal-00714858, hal-00725477, hal-00772742, hal-00773114, hal-00773571, hal-00773610, hal-00776610, tel-00777154, hal-00803304, hal-00807033, hal-00824514, hal-00851122, hal-00853423, hal-00858350, hal-00911856, hal-00920915, hal-00925017, hal-00926144, tel-00948309, hal-00966862, hal-00978364, hal-00978602, hal-00987094, hal-00992208, hal-01005765, hal-01011633, hal-01081974, hal-01101045, hal-01101054, hal-01120507, hal-01147997, tel-01162975, hal-01180272, hal-01181135, hal-01182746, hal-01223573, tel-01230876, hal-01283949, hal-01284004, hal-01284136, hal-01284235, hal-01316982, hal-01332774, hal-01353962, hal-01355385, hal-01361992, hal-01372022, hal-01386174, hal-01387482, hal-01409965, hal-01410103, hal-01473475, hal-01474556, tel-01483666, hal-01502749, hal-01507613, hal-01517153, tel-01538516, hal-01616632, hal-01618526, hal-01718280, tel-01816341, hal-01842038, tel-01959127, hal-02120736, hal-02275363, hal-02296118, hal-02403109, hal-02421327, hal-02872765, hal-02914793, hal-02933803, hal-02943753, hal-02970529, hal-02985721, hal-03144290, hal-03273509, hal-03290998, hal-03298021, hal-03318644, hal-03348787, hal-03552243, hal-03609275, hal-03623220, hal-03773486, hal-03773985, hal-03789625, hal-03936659, tel-03989856, hal-04005071, hal-04088833, hal-04115280, hal-04146714, hal-04236246, tel-04260094, tel-04316145, hal-04548787, hal-04646530, hal-04668550, hal-04690154
• Contact:
  Nathalie Furmento
• Participants:
  Cedric Augonnet, Olivier Aumage, Nathalie Furmento, Samuel Thibault, Simon Archipoff, Bérenger Bramas, Alfredo Buttari, Jérôme Clet-Ortega, Terry Cojean, Nicolas Collin, Camille Coti, Ludovic Courtes, Alexandre Denis, Lionel Eyraud Dubois, Maxime Gonthier, Amina Guermouche, Kun He, Sylvain Henry, Andra Hugo, Antoine Jego, LoÏc Jouans, Mehdi Juhoor, Yanis Khorsi, Xavier Lacoste, Romain Lion, Benoit Lize, Gwenole Lucas, Mariem Makni, Thomas Morin, Raymond Namyst, Cyril Roelandt, Corentin Salingue, Lucas Schnorr, Marc Sergent, Luka Stanisic, Ludovic Stordeur, Philippe Swartvagher, François Tessier, Leo Villeveygoux, Philippe Virouleau, Pierre Wacrenier

7.1.6 VITE

• Name:
  Visual Trace Explorer
• Keywords:
  Visualization, Execution trace
• Functional Description:
  ViTE is a trace explorer. It is a tool made to visualize execution traces of large parallel programs. It supports Pajé, a trace format created by Inria Grenoble, and OTF and OTF2 formats, developed by the University of Dresden and allows the programmer a simpler way to analyse, debug and/or profile large parallel applications.
• URL:
  https://­solverstack.­gitlabpages.­inria.­fr/­vite/
• Publications:
  hal-00707236, hal-04725983
• Contact:
  Mathieu Faverge
• Participants:
  Mathieu Faverge, Philippe Swartvagher

7.1.7 pmtool

• Keywords:
  Scheduling, Task scheduling, StarPU, Heterogeneity, GPGPU, Performance analysis
• Functional Description:
  Analyse post-mortem the behavior of StarPU applications. Provide lower bounds on makespan. Study the performance of different schedulers in a simple context. Provide implementations of many scheduling algorithms from the literature
• URL:
  https://­gitlab.­inria.­fr/­eyrauddu/­pmtool
• Publications:
  hal-01386174, hal-01878606
• Contact:
  Lionel Eyraud Dubois
• Participant:
  Lionel Eyraud Dubois

7.1.8 rockmate

• Name:
  rockmate
• Keywords:
  Deep learning, Optimization, Python, Pytorch, GPU, Automatic differentiation
• Scientific Description:

  We propose Rockmate to control the memory requirements when training PyTorch DNN models. Rockmate is an automatic tool that starts from the model code and generates an equivalent model, using a predefined amount of memory for activations, at the cost of a few re-computations. Rockmate automatically detects the structure of computational and data dependencies and rewrites the initial model as a sequence of complex blocks. We show that such a structure is widespread and can be found in many models in the literature (Transformer based models, ResNet, RegNets,...). This structure allows us to solve the problem in a fast and efficient way, using an adaptation of Checkmate (too slow on the whole model but general) at the level of individual blocks and an adaptation of Rotor (fast but limited to sequential models) at the level of the sequence itself. We show through experiments on many models that Rockmate is as fast as Rotor and as efficient as Checkmate, and that it allows in many cases to obtain a significantly lower memory consumption for activations (by a factor of 2 to 5) for a rather negligible overhead (of the order of 10% to 20%). Rockmate is open source and available at https://github.com/topal-team/rockmate.

  Complete paper: https://openreview.net/pdf?id=wLAMOoL0KD

• Functional Description:

  Given a PyTorch model, a sample input, and a GPU memory budget, Rockmate builds a new torch.nn.Module, which performs forward and backward pass while keeping the memory of activations under the given budget.

  The new model produces the same outputs and gradients as the original one. Training the model with a lower memory than PyTorch Autodiff is achieved by re-computing some of the activations instead of storing them for gradient calculation. Based on the budget, Rockmate determines automatically which activations should be recomputed.

• URL:
  https://­github.­com/­topal-team/­rockmate
• Contact:
  Lionel Eyraud Dubois

10.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

ELF Associate Team on on Efficient deep Learning Frameworks.

Partners

• TOPAL
• California Institute of Technology (Caltech)

Nowadays, Deep Learning (DL) and Artificial Intelligence (AI) technologies are incorporated in more and more areas to solve various problems of video, audio, natural language processing, content generation, etc. Frameworks based on neural networks, which are core modules of deep learning models, have been already successfully used for action recognition, weather forecasting, robotic surgery and other inspiring applications [24, 44, 48]. The drawbacks of modern neural networks are that they usually require a significant amount of data and a lot of GPU devices to be trained, which makes them expensive in terms of energy and money costs, and harmful in terms of air emissions [27]. The general question we are going to address during the work of the associate team is: given your application and your computation platform, how to perform the model training efficiently in terms of time/energy?

10.2.1 Visits of international scientists

10.3.1 H2020 projects

10.4.1 ANR

10.4.2 Inria Challenge

11.1.1 Scientific events: organisation

The Workshop on Advancing Neural Network Training WANT Workshop at ICML'2024 focused on improving efficiency, scalability, and resource optimization in neural network training. Organized by experts from INRIA, NVIDIA, Oak Ridge National Laboratory, and others, it features 42 accepted papers (including 7 oral presentations) reviewed by 77 reviewers and 7 area chairs. The workshop highlights topics like efficient data management, scaling across devices, hardware and cluster optimization, and advanced techniques such as quantization, pruning, and low-rank adaptation, with applications in computer vision, NLP, reinforcement learning, and scientific domains. Paper contributors represented a wide range of expertise, with 130 affiliations from academia and 57 from industry, showcasing a strong collaborative effort. The hybrid-format event included offline and online activities like panel discussions, poster presentations, and networking sessions via Gather Town and Discord. Key speakers included experts from Hugging Face, NVIDIA, Carnegie Mellon University, and TogetherAI, who shared insights on large-scale training, hardware design, and optimization techniques. With a focus on fostering impactful research, the workshop is an essential platform for advancing computationally efficient AI solutions.

11.1.2 Scientific events: selection

11.1.3 Journal

11.1.4 Invited talks

• Lionel Eyraud Dubois gave a lecture at the New Trends in Computing Summer School, entitled “Scheduling on heterogeneous machines”.
• Philippe Swartvagher gave an invited talk at the ComPAS 2024 conference and at the FOSDEMHPC, Bigdata and Data Science devroom, entitled “Making reproducible and publishable experiments”.
• Philippe Swartvagher gave a talk at the Journées non-thématiques du GDR RSD, entitled “On the Interactions between HPC Task-based Runtime Systems and Communication Libraries”.
• Julia Gusak gave a talk at GAP'24 (Grenoble) entitled “Neural ODEs, neural operators, and their efficiency“.
• Julia Gusak gave a talk at LoRAINNE'24 (Nancy) entitled “Tensor Methods in Deep Learning and their Efficiency“.
• Julia Gusak gave a talk at AI4Industry'24 entitled “Efficient Training of Neural Networks“.

11.1.5 Leadership within the scientific community

• Olivier Beaumont is a member of the IEEE CS Babbage Award selection commitee
• Olivier Beaumont is the Inria representative on the EuroHPC Mirror Group, which helps define the position and strategy of the French Ministry of Research at the EuroHPC Governing Board.

11.1.6 Scientific expertise

• Olivier Beaumont acts as Cross Reader for the European Innovation Council and the Pathfinder Open program.
• Olivier Beaumont acted as external evaluator for several EuroHPC calls: Inno4Scale, Energy, FFPlus
• Pierre Ramet is Scientific Advisor at the CEA-DAM CESTA.
• Pierre Ramet participated in the HCERES evaluation committee of the IRFU (Institut de recherche sur les lois fondamentales de l'Univers) at CEA Saclay.

11.1.7 Research administration

• Pierre Ramet is the head of the CNRS Satanas department.
• Pierre Ramet is member of Scientific comittee of the LaBRI.
• Philippe Swartvagher is the communication referent for the NumPEx/Exa-SofT project.
• Philippe Swartvagher is the point of contact in Bordeaux for Grid5000/SLICES-FR infrastructure.
• Philippe Swartvagher is the representative of the TOPAL project at the Bordeaux CUMI.
• Philippe Swartvagher is elected member of the school council of the Enseirb-MatMeca engineering school.
• Abdou Guermouche is the scientific lead of the numerical library work package of the ExaSoft project (PEPR NumPEx).
• Abdou Guermouche is member of the Scientific Committee of LaBRI.
• Julia Gusak is a PI of the ELF associate team between Topal and Caltech.

11.2.1 Supervision

• Defended PhD: Clément Richefort ; Development of an algebraic multigrid solver for the indefinite Helmholtz equation ; defended November 2024; advisor Pierre Ramet .
• Defended PhD: Xunyi Zhao ; Optimizing Memory Usage when Training Deep Neural Networks; defended December 2024; advisors Olivier Beaumont , Yulia Gusak , Lionel Eyraud Dubois .
• PhD in progress: Abel Calluaud; Combined compiler and runtime approach for a direct hierarchical solver; started Nov. 2022; advisors Pierre Ramet , Mathieu Faverge .
• PhD in progress: Jean-François David; Dynamic Scheduling for Inference in Deep Neural Networks; advisors Olivier Beaumont , Lionel Eyraud Dubois .
• PhD in progress: Alycia Lisito; Design and implementation of a portable linear algebra benchmark on runtime systems for performance evaluation of heterogeneous Exascale architectures ; started Nov. 2023; advisors Pierre Ramet , Mathieu Faverge , Matthieu Kuhn (Eviden).
• PhD in progress: Dimitri Walther; ; started Nov. 2024; advisors Pierre Ramet , Mathieu Faverge , M. Lecouvez (CEA Cesta).
• Mathieu Faverge and Philippe Swartvagher supervised the internship of Camille Ordronneau  26.
• PhD in progress: Hayfa Tayeb ; Optimization of high-performance applications on heterogeneous computing nodes; started Nov. 2021; A. Guermouche , B. Bramas , M. Faverge. Defense March 25th, 2025.
• PhD in progress: Albert D'Aviau de Piolant ; started October 2023; Energy aware scheduling for exascale architectures. Advisors: Abdou Guermouche and Amina Guermouche.
• PhD in progress: Thomas Morin ; started October 2023; Scheduling recursive task graphs. Advisors: Abdou Guermouche, Samuel Thibault, Pierre-André Wacrenier.
• PhD in progress : Alice Lasserre ; Started Oct. 2022; Optimization of a task-based simulation code on a distributed supercomputer; Advisors: Jean-Marie Couteyen-Carpaye, Raymond Namyst and Abdou Guermouche.
• PhD in progress: Adrien Aguilla–Multner , Started October 2024; Efficient Training of Neural Networks. Advisors: Julia Gusak, Olivier Beaumont.
• Internship in progress: Enrique Galves , started in September 2024; Task-based systems for efficient deep learning. Advisors: Julia Gusak, Olivier Beaumont.

11.3.1 Productions (articles, videos, podcasts, serious games, ...)

Olivier Beaumont participated with Remi Bouzel (Qarnot Computing) to the podcast "Déassemblons le numérique" (in French) to present the joint Inria/Qarnot Challenge "Pulse" (podcast)

11.3.2 Participation in Live events

• As part of the "Circuit Scientifique Bordelais", Philippe Swartvagher presented to high school pupils from the Lycée Bastide at Villeneuve-sur-Lot what is research in computer science and how to become a researcher.
• Olivier Beaumont participated in several internal events (closed doors, unité ou café) to present the activities of the Inria Bordeaux center teams at the interface between HPC and AI.
• As part of Maths en Jeans, Olivier Beaumont worked with groups of students from Andernos high school on combinatorial problems linked to data distribution for Linear Algebra problems.
• On several occasions, we have welcomed 3rd and 2nd grade students into the team, with the participation of Topal's PhD students, for periods of 2 hours to half a day.

International journals

• 6 articleO.Olivier Beaumont, L.Lionel Eyraud-Dubois, J.Julien Herrmann, A.Alexis Joly and A.Alena Shilova. Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory.ACM Transactions on Mathematical Software2024. In press. HALback to textback to textback to text

International peer-reviewed conferences

• 7 inproceedingsO.Olivier Beaumont, R.Rémi Bouzel, L.Lionel Eyraud-Dubois, E.Esragul Korkmaz, L.Laércio Lima Pilla and A.Alexandre van Kempen. A 1.25(1+ε)-Approximation Algorithm for Scheduling with Rejection Costs Proportional to Processing Times.International European Conference on Parallel and Distributed Computing (Euro-Par)14801Lecture Notes in Computer ScienceMadrid, SpainSpringer Nature SwitzerlandAugust 2024, 225-238HALDOIback to textback to text
• 8 inproceedingsO.Olivier Beaumont, J.-F.Jean-François David, L.Lionel Eyraud-Dubois and S.Samuel Thibault. Exploiting Processor Heterogeneity to Improve Throughput and Reduce Latency for Deep Neural Network Inference.SBAC-PAD 2024 - IEEE 36th International Symposium on Computer Architecture and High Performance ComputingHilo, Hawaii, United StatesNovember 2024HALback to text
• 9 inproceedingsO.Olivier Beaumont, J.-F.Jean-François David, L.Lionel Eyraud-Dubois and S.Samuel Thibault. StarONNX: a Dynamic Scheduler for Low Latency and High Throughput Inference on Heterogeneous Resources.HeteroPar 2024 - 22ND INTERNATIONAL WORKSHOP Algorithms, Models and Tools for Parallel Computing on Heterogeneous PlatformsHeteroPar'24 ProceedingsMadrid, Spain2024HALback to text
• 10 inproceedingsL.Lionel Eyraud-Dubois, G.Guillaume Iooss, J.Julien Langou and F.Fabrice Rastello. Tightening I/O Lower Bounds through the Hourglass Dependency Pattern.SPAA 2024 - 36th ACM Symposium on Parallelism in Algorithms and ArchitecturesNantes, FranceApril 2024, 1-34HALback to text
• 11 inproceedingsN.Nathalie Furmento, A.Abdou Guermouche, G.Gwenolé Lucas, T.Thomas Morin, S.Samuel Thibault and P.-A.Pierre-André Wacrenier. Optimizing Parallel System Efficiency: Dynamic Task Graph Adaptation with Recursive Tasks.WAMTA 2024 - Workshop on Asynchronous Many-Task Systems and Applications 2024Knoxville, United Stateshttps://wamta24.icl.utk.edu/February 2024HALback to text
• 12 inproceedingsA.Alice Lasserre, J. M.Jean Marie Couteyen Carpaye, A.Abdou Guermouche and R.Raymond Namyst. Multi-Criteria Mesh Partitioning for an Explicit Temporal Adaptive Task-Distributed Finite-Volume Solver.The 25th IEEE International Workshop on Parallel and Distributed Scientific and Engineering Computing (PDSEC 2024)San Francisco, United StatesMay 2024, 10HALback to text
• 13 inproceedingsA.Alycia Lisito. Efficient HPL on top of runtime systems.compas 2024 - Conférence francophone d'informatique en Parallélisme, Architecture et SystèmeNantes, FranceJuly 2024HALback to text
• 14 inproceedingsA.Alycia Lisito, M.Mathieu Faverge, G.Grégoire Pichon and P.Pierre Ramet. Enhancing sparse direct solver scalability through runtime system automatic data partition.WAMTA 2024 - Workshop on Asynchronous Many-Task Systems and Applications 202414626Lecture Notes in Computer ScienceKnoxville, United StatesSpringer Nature SwitzerlandMarch 2024, 105-110HALDOIback to text
• 15 inproceedingsH.Hayfa Tayeb, B.Bérenger Bramas, M.Mathieu Faverge and A.Abdou Guermouche. Dynamic Tasks Scheduling with Multiple Priorities on Heterogeneous Computing Systems.2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)38th IEEE International Parallel & Distributed Processing SymposiumSan franscisco, CA, United States2024, 31-40HALDOIback to text

Conferences without proceedings

• 16 inproceedingsO.Olivier Beaumont, J.-F.Jean-François David, L.Lionel Eyraud-Dubois and S.Samuel Thibault. StarONNX : Un ordonanceur dynamique pour une inférence rapide et à haut débit sur des ressources hétérogènes.Compas 2024 - Conférence francophone d'informatique en Parallélisme, Architecture et SystèmeNantes, FranceJuly 2024HALback to text
• 17 inproceedingsR.Rob Falgout, M.Matthieu Lecouvez, P.Pierre Ramet and C.Clément Richefort. Slides 18th CMCIM 2024, Toward an algebraic multigrid method for the indefinite Helmholtz equation.18th Copper Mountain Conference On Iterative MethodsFrisco (Co), United StatesApril 2024HALback to text
• 18 inproceedingsB.Brieuc Nicolas, M.Mathieu Faverge, P.Pierre Ramet, A.Alycia Lisito and M.Mohamed Kherraz. Comparative Study of Mixed-Precision and Low-Rank Compression Techniques in Sparse Direct Solvers.2024 SIAM Conference on Applied Linear AlgebraParis, FranceMay 2024HALback to text

Reports & preprints

• 19 miscO.Olivier Beaumont, R.Rémi Bouzel, L.Lionel Eyraud-Dubois, E.Esragul Korkmaz, L.Laércio Lima Pilla and A.Alexandre van Kempen. Approximation Algorithms for Scheduling with/without Deadline Constraints where Rejection Costs are Proportional to Processing Times.October 2024HALback to text
• 20 miscR. D.Robert D Falgout, M.Matthieu Lecouvez, P.Pierre Ramet and C.Clément Richefort. Toward an algebraic multigrid method for the indefinite Helmholtz equation.June 2024HAL
• 21 miscJ.Julia Gusak, X.Xunyi Zhao, T.Théotime Le Hellard, Z.Zhe Li, L.Lionel Eyraud-Dubois and O.Olivier Beaumont. HiRemate: Hierarchical Approach for Efficient Re-materialization of Large Neural Networks.July 2024HALback to text
• 22 miscX.Xunyi Zhao, L.Lionel Eyraud-Dubois, T.Théotime Le Hellard, J.Julia Gusak and O.Olivier Beaumont. OFFMATE: full fine-tuning of LLMs on a single GPU by re-materialization and offloading.July 2024HALback to text
• 23 miscA.Albert d'Aviau de Piolant, H.Hayfa Tayeb, B.Bérenger Bramas, M.Mathieu Faverge, A.Abdou Guermouche and A.Amina Guermouche. Improving energy efficiency of HPC applications using unbalanced GPU power capping.October 2024HAL

Other scientific publications

• 24 inproceedingsA.Alice Lasserre, J. M.Jean Marie Couteyen Carpaye, A.Abdou Guermouche and R.Raymond Namyst. Multi-Criteria Mesh Partitioning for an Explicit Temporal Adaptive Task-Distributed Finite-Volume Solver.Doctoral students' day of the Mathematics and Computer Science doctoral school of the University of Bordeaux (EDMI)Bordeaux, FranceApril 2024HAL
• 25 inproceedingsA.Alycia Lisito, M.Mathieu Faverge, D.David Goudin, M.Matthieu Kuhn and P.Pierre Ramet. Scalable and Portable LU Factorization with Partial Pivoting on top of Runtime Systems.journée calculs et données 2024Bordeaux, FranceNovember 2024HALback to text
• 26 thesisC.Camille Ordronneau. Développement et maintenance du logiciel ViTE.ENSEIRB-MATMECATalenceOctober 2024, 21HALback to text

Software

• 27 softwareO.Olivier Beaumont, L.Lionel Eyraud-Dubois, E.Esragul Korkmaz and L.Laércio Lima Pilla. Experimental codes and results for the paper "A 5/4(1+eps)-Approximation Algorithm for Scheduling with Rejection Costs Proportional to Processing Times".March 2024Inria & Labri, Univ. Bordeaux lic: CeCILL Free Software License Agreement v2.0.HALSoftware Heritageback to text