Audio-to-fixed easier than float-to-fixed

In audio processing, we know that the inputs and outputs are fixed-point data, and we also have a lot of domain knowledge about audio physics. This gives serious hope that Faust audio can be compiled directly to fixed-point. This is a requirement for FPGAs, but it will also reduce the latency and power consumption on software targets if we can use their integer units. It will also enable the compilation of Faust to ultra-low-power microcontrollers without floating-point hardware.

“Domain-specific” is the key word here making us confident that a problem that is generally intractable (float-to-fixed conversion) can be addressed with little or no modification to a Faust program. The challenge here is to keep this additional work so high-level and sound-related that it is not a burden for a musician or a sound engineer. A central objective is that Faust programmers should not need to become fixed-point experts. They should actually not be anymore aware of the underlying arithmetic than they currently are with floating-point. Being high-level is a key reason for the success of Faust .

Automated error analysis for hardware computing just right

The main issue is to understand how arithmetic errors propagate, are amplified, are accumulated, etc. in a computation and in a circuit. This is called error analysis. Then a general technique 35 is to add enough bits to the right of internal fixed-point formats so that errors accumulate in these bits and the overall error accumulation does not hinder the final quality of the result. Error analysis is also managed by a worst-case combination, but here there is nothing implicit or hidden. This is therefore a comparatively well understood problem, and there is no reason to believe it cannot be fully automated in a compiler that is already able to derive the format information, building on the experience accumulated when designing complex FloPoCo operators 34, 33, 24, 72, 76.

Digital filters as arithmetic objects

Digital filters are essential components of everyday electronics like radios, mobile phones, etc., but also in audio systems of course. Their design is a core topic in digital signal processing and control theory, one that has received significant research interest for the better part of the last half century. A lot of effort has gone into constructing flexible filter design methods. For designing software-based digital filters with floating-point coefficients, there are many powerful approaches that are relatively easy to use by the filter designer (all the more as they rely on over-dimensioned floating-point operators). When designing hardware, things are not that simple for several reasons:

• algorithms developed for software-implemented filters cannot be transferred directly to hardware: what is a constraint in software (e.g., “use a 32-bit fixed-point format”) becomes a degree of freedom in hardware design (“What is the smallest fixed-point format that can be used?”);
• another degree of freedom comes from different available realization techniques to implement the arithmetic itself, for instance the construction of multipliers by constants.

A consequence is that popular tools, such as the popular fdatool (filter design and analysis tool) from Matlab's Signal Processing toolbox, offer a complex interface, requiring a tedious hand-tuning process, and expect some domain expertise. Such tools input a frequency response, and decompose the filter implementation problem in three steps: 1/ the filter design (FD) step consists in finding a filter with ideal (high precision) coefficients that adheres to the frequency response; 2/ the quantization (Q) step converts the obtained coefficients to hardware-friendly fixed-point values ; 3/ the implementation (I) step generates a valid hardware description (e.g., a VHDL or Verilog description) using the quantized coefficients.

The objective of this research axis is to offer an optimal solution to the global FD + Q + I problem. Optimal techniques exist for each of the FD, Q and I steps in isolation. The combination of the FD & Q steps have been studied since the 1960's  45, and can even be regarded as solved for certain practical instances of fixed-point Finite Impulse Response (FIR) design 46. A large body of work also exists for the I step, with recent optimal approaches 18, 47, 48. However, these approaches are only optimal for a given set of coefficients, and therefore strongly depend on the FD and Q steps.

Arithmetic-oriented scheduling and tiling for low-latency audio

Finally, we also want to formally insert arithmetic considerations in the global problem of distributing a very heavy computation between space (we have up to several thousands multipliers in an FPGA, and many more if we multiply by a constant) and time (we have thousands of FPGA cycles within one audio cycle). These are well-researched compilation issues, called the scheduling and tiling problems. There is local expertise in Lyon (in particular in the CASH team and its spin-off XtremLogic21) who have worked on these problems for FPGAs. However, scheduling and tiling techniques so far consider each operation as having a standard, constant cost (e.g., multiplication costs $c_m$ and has latency $t_m$, addition costs $c_a$ in space and $t_a$ in time). This is a very crude simplification if one attempts to optimize each operator, think for multiplications by constant for instance. The availability of many audio-related case studies in Emeraude will allow us (hopefully in collaboration with CASH) to develop arithmetic-aware scheduling and tiling techniques that will eventually prove useful well beyond the world of digital audio.

Towards provably optimal solutions

Recent arrivals of Christine Solnon and Anastasia Volkova into the team bring more focus into the optimization. Given a mathematical object to implement in hardware (for instance the 2D norm $\sqrt{x^2+y^2}$), the standard practice so far has been to 1/ define a family of hardware algorithms parameterized by architectural parameters (typically the number of bits for the intermediate results), 2/ express the constraints for a given solution to be acceptable (typically that the hardware should provide a faithful or correct rounding of the exact result), and 3/ finally define a heuristic that provides the parameters for an acceptable solution with good performance (performance being, for instance, area or delay of the hardware). The shift is to replace the heuristic in the third step with a mathematical model that can be solved with standard solvers (ILP, SAT or CP) to provide hardware operators with provably optimal performance. This approach was pioneered by Martin Kumm about ten years ago, and used in Emeraude recently for squarers 22, elementary function evaluators 31, and digital filters 10. In the coming years we will apply this approach more broadly to operators for digital signal processing (in collaboration with axes 1 and 3) and artificial intelligence accelerators (in collaboration with other members of the PEPR IA). We will also use optimization for core classical arithmetic problems such as function approximation and evaluation, compressor tree synthesis, or word-length optimization, again in collaboration with Axis 3.

The Faust language and its compiler

A large part of Emeraude's research results is visible thanks to the Faust ecosystem development. Faust has gained an international recognition, especially since it is used for teaching at Stanford University (in the context of courses on signal processing, physical interaction design, etc.) and developing new audio plugins 59. The efforts needed to keep Faust as the most efficient language for real-time audio processing involve research in: language design, compiler design, and development of DSP libraries.

One of the reason of Faust 's success is that it is both a language and an environment for audio signal processing. The Faust compiler typically generates high-level codes (in languages such as C, C++, etc.), following every compiler's goal: providing better code than manually written code. For that, it has to stick to the most recent compiler technologies and processors evolutions 53. For instance, a back-end for WebAssembly was recently added to the Faust compiler 52. An important deployment step was the embedding of the Faust compiler in a web browser 51 which makes it easily accessible on all computers.

Faust language design research in Emeraude

The current design of Faust , inspired by lambda-calculus, combinatory logic and John Backus’ functional programming, has to be extended to face new challenges, in particular multi-dimensional and multi-rate signals and linear algebra.

Faust allows for the description of synchronous mono-rate scalar DSP computations. This is sufficient to implement most time-domain algorithms such as filters, oscillators, waveguides, etc. However, this makes the implementation of frequency-domain algorithms (e.g., FFT, convolution, etc.) very inefficient, not to say impossible. One of our goals is to extend the language to enable multi-rate as well as vector computations. While we already have a working prototype for this, some challenges have yet to be overcome.

Along the lines of the previous point, Faust currently doesn't provide any support for efficient matrix operations and more generally linear algebra. This prevents the implementation of some classes of DSP algorithms such as Finite-Difference Time-Domain (FDTD) method for physical modeling. The skills of former Socrate members on seminal Alpha language 36 and polyhedral optimization are very useful here.

Support for the main target programming languages in Faust is essential. Recently added languages (WebAssembly, Rust, and CMajor) have opened many new opportunities. The FPGA target, studied in §3.1, introduces new challenges such as the ability to use fixed-point arithmetic or the use of HLS for targeting hardware platforms (e.g., VHDL, Verilog, etc.). Other “exotic” architectures such as GPUs or MPSoCs should be studied for compute-bound algorithms.

Musicians have to deal with a large variety of operating systems, software environments and hardware architectures. Faust is designed to favor an easy deployment of dsp programs on all these targets by making a clear separation between computation itself, as described by the program code, and how this computation should be related to the external world. This relation (with audio drivers, GUIs, sensors, etc.) is described in specific architecture files40. Architecture files concern both hardware (i.e., audio interfaces/sound cards) as well as software control interfaces (e.g., GUI, OSC,22 MIDI), new luthieries (e.g., SmartFaust, Gramophone), Web platforms (Web audio Plugin), etc. One of the goal of the work of Emeraude on Faust is to ease the programming of these audio systems.

Faust ecosystem and DSP libraries

Faust users are very attached to its ecosystem, including native applications, online and “embedded” audio applications, Just In Time (JIT) compiler, etc. Recent developments include a JIT Faust compiler on the Web, a JIT compiler in the Max/MSP environment, tools to find the best compilation parameters and ease compilation for multiple CPUs. This is constantly evolving to answer to users' demand.

The Faust DSP libraries currently implement hundreds of functions/objects ranging from simple oscillators and filters to advanced filter architectures, physical models, and complete ready-to-use audio plugins. These libraries are at the heart of Faust 's success and international position. Julius Smith23 (Stanford professor) is one of the most respected figures in the field of audio DSP and one of the main contributors to the Faust libraries. One of the ambitions of the Emeraude team is to maintain and extend this tool to make it as exhaustive and as universal as possible. Along these lines, new developments made to the language presented above (e.g., multi-rate, linear algebra, etc.) should be ported to the libraries. Finally, dedicated libraries targeting specific hardware platforms (e.g., microcontrollers, FPGAs) should be made available too.

Machine learning for digital signal processing

Machine learning and deep learning in particular, are playing an increasingly important role in the field of audio DSP. Researchers are revisiting the algorithmic techniques of signal synthesis and processing in the light of machine learning, for instance for speech processing 41. Recent breakthroughs such as the use of machine learning use in the context of Differentiable Digital Signal Processing (DDSP) 39 demonstrate its power. The extension of Faust applications to artificial intelligence began with the PhD work of Benjamin Quiédeville, expanding the scope of Faust in the field of AI. The objective is the introduction of an autodifferentiation primitive for Faust programs, aimed at machine learning applications. The development of new backends is planned, particularly for MLIR, MOJO, and GPUs, as well as support for emerging architectures, especially in the context of game engines and VR/AR applications.

Embedded systems for audio processing

As Emeraude's name suggests it, the implementation of audio Digital Signal Processing on embedded hardware is at the heart of the project. We naturally rely on the Faust language for these implementations. The skills of Emeraude members in compilation and embedded systems are used to add new embedded target for audio processing, in particular FPGAs, as explained previously. This action is a mix of research and engineering work, it should be very useful for the dissemination of audio processing programming.

Haptics is a huge topic, especially in the field of New Interfaces for Musical Expression (NIME), which has been studied for many years 29, 74. It has always been tightly coupled to physical modeling because this sound synthesis technique provides natural connections to the physical world. A big part of the challenge is technological because haptics requires ultra low-latency and high sampling resolution in order to be accurate. This is at the heart of Emeraude 's goals.

Virtual and Augmented Reality (VR/AR) is not limited to immersive 3D graphics, sound also has an important role to play in that emerging field. Lots of work has been done around using VR environments as a creative tool for audio 50, 27, 25. While many VR-based musical instruments have been created in the past 68, little work has been done around implementing interfaces specifically targeting VR/AR audio environments, especially in the context of 3D sound. This is something that we plan to explore as part of Emeraude.

Finally, beside ergonomic and HCI aspects, the design of musical interfaces is impacted by various kinds of technical limitations that we plan to address as part of Emeraude. First, just like for real-time audio processing, latency plays a crucial role in this context. Similarly, the “time resolution” (e.g., the sampling rate of the interfaces) can have a huge impact, especially when targeting specific kinds of instruments such as drums. Finally, the “spatial resolution” (e.g., the number of sensor points per squared centimeters on a tabletop interface) also impacts its quality. In this context, we would like to develop an embedded, high-resolution, high-sampling-rate, multi-touch multi-dimensional (X and Y + pressure) interface/instrument leveraging the development carried out in the previous axes. This work would be followed by a user study to measure the impact of this type of advanced system on perception.

6.1.1 FloPoCo

• Name:
  Floating-Point Cores, but not only
• Keyword:
  Synthesizable VHDL generator
• Functional Description:
  The purpose of the open-source FloPoCo project is to explore the many ways in which the flexibility of the FPGA target can be exploited in the arithmetic realm.
• URL:
  http://­flopoco.­org
• Contact:
  Florent De Dinechin
• Participants:
  Florent De Dinechin, Luc Forget
• Partners:
  ENS Lyon, Insa de Lyon, Inria, Fulda University of Applied Science

6.1.2 Syfala

• Name:
  Low-Latency Synthesizer on FPGA
• Keywords:
  FPGA, Compilers, High-level synthesis, Audio signal processing
• Functional Description:

  The goal of Syfala is to design an FPGA-based platform for multichannel ultra-low-latency audio Digital Signal Processing programmable at a high-level with Faust and C++ and usable for various applications ranging from sound synthesis and processing to active sound control and artificial sound field/room acoustics.

  A series of tools are currently being developed around SyFaLa. SyFaLa is freely accessible on GitHub: https://github.com/inria-emeraude/syfala.

• URL:
  https://­faust.­grame.­fr/­syfala/
• Contact:
  Tanguy Risset

6.1.3 FAUST

• Name:
  Functional Audio Stream)
• Keywords:
  Audio, Functional programming
• Functional Description:

  The core component of Faust is its compiler. It allows to "translate" any Faust digital signal processing (DSP) specification to a wide range of non-domain specific languages such as C++, C, LLVM bit code, WebAssembly, Rust, etc. In this regard, Faust can be seen as an alternative to C++ but is much simpler and intuitive to learn.

  Thanks to a wrapping system called "architectures," codes generated by Faust can be easily compiled into a wide variety of objects ranging from audio plug-ins to standalone applications or smartphone and web apps, etc.

• URL:
  https://­faust.­grame.­fr/
• Contact:
  Yann Orlarey
• Partners:
  GRAME, Insa de Lyon, Inria

7.1.1 C++ Optimizations In the Context of High-Level Synthesis

Participants: Tanguy Risset, Romain Michon, Pierre Cochard.

When compiling Faust programs to FPGA, Syfala relies on the High Level Synthesis (HLS) tool provided by Xilinx, which takes a C++ program as an input. Hence, Faust generates C++ code from a Faust program and Syfala feeds it to HLS. The topology of the C++ code provided to HLS has a huge impact on the performances of the generated Intellectual Property (IP). In 2023, we conducted a study aiming at understanding the kind of optimizations that can be carried out on C++ code in the context of the high-level synthesis of real-time audio DSP programs. Thanks to this work, we managed to significantly optimize the applications generated by Syfala allowing for much more complex audio DSP algorithms to be run on the FPGA. While these findings haven't been integrated to the Faust Syfala backend, they can be used with the new Syfala C++ support. Indeed, we recently added a new mode in Syfala allowing for C++ code to be used as a substitute for Faust . This, combined with an exhaustive public documentation of the aforementionned optimizations will help increasing the attractivity of Syfala.

Our work around the new Syfala C++ support led to a publication at a conference in 2024 4.

7.1.2 Syfala Ethernet Audio Support

Participants: Pierre Cochard, Tanguy Risset, Romain Michon.

As we worked on applications for Syfala requiring the handling of a large number of audio channels in parallel for spatial audio, we realized that we needed a way to send and receive audio streams in parallel between a laptop computer and our FPGA board. For this, we opted for an open standard named PipeWire which allows for the transmission of digital audio streams in real-time over an ethernet connection. PipeWire was implemented in the Linux layer of Syfala and is now perfectly integrated to the toolchain. It will allow to significantly expand the scope of the various spatial audio systems that we've been working on in the context of PLASMA (see §7.2)

This work led to a publication at the IEEE Symposium on Internet of Sound in 2024 5.

7.2.1 Distributed Spatial Audio System

Participants: Romain Michon, Tanguy Risset.

We developed a distributed systems for spatial audio DSP based on a network of microcontrollers (see Fig. 7). We chose this type of platform because they are cost-effective, very lightweight, and OS-free. We used PJRC's Teensys 4.126 as they host a powerful Cortex M7 clocked at 600 MHz as well as built-in ethernet support. PJRC also provides an “audio shield” which is just a breakout board for a stereo audio codec (SGTL-5000) compatible with the Teensy 4.1.

A preliminary task was to send audio streams over the Ethernet from a laptop to the Teensy. For that, we decided to use the JackTrip protocol which is open source and used a lot in the audio/music tech community 26. Implementing a JackTip client on the Teensy was fairly straightforward. We then implemented our own protocol which can be easily accessed through an audio plugin directly runnable in a Digital Audio Workstation (DAW).

Audio DSP is carried out directly on the Teensys which are programmed with the Faust programming language thanks to the faust2teensy 57 tool developed by the Emeraude team. A laptop is used to transmit audio streams to the Teensys which are controlled using the Open Sound Control (OSC) standard. OSC messages are multicast/broadcast to save bandwidth. The same audio streams are sent to all the Teensys in the network (all audio processing is carried out on the Teensys, not on the laptop computer).

This work led to a journal paper in 2024 3 and is now the main focus of Thomas Rushton's PhD.

Show image description

Photo of our Wave Field Synthesis system based on a network of microcontrollers installed in GRAME studio

7.2.2 FPGA-Based Spatial Audio

Participants: Romain Michon, Tanguy Risset, Pierre Cochard.

The implementation of audio spatialization and active acoustic control techniques requires a large number of audio channels and significant computational resources, making these techniques costly and hardly accessible. The use of Field-Programmable Gate Arrays (FPGAs) offers a solution by enabling DSP programming within a centralized system with enough computational power and inputs/outputs to handle all audio streams. However, physically interfacing an embedded system with multiple audio channels is a complex process that poses a significant challenge to system accessibility. In 2024, we introduced two open-source circuit boards (see Fig. 8 and Fig. 9) facilitating the interfacing between an FPGA and large multichannel systems, both aiming different kinds of approaches. The first board is designed to be cost-effective and easily reproducible. The second involves more complex manufacturing techniques, but it allows us to fully exploit the performances of the FPGA. Each board provides 32 audio channels and has stacking capabilities, allowing for the interfacing of up to 512 channels on a single FPGA. Their integration into the Syfala compilation flow facilitates the implementation of algorithms without hardware concerns. These interfaces aim to enhance accessibility and affordability when implementing audio spatialization and active acoustic control systems.

This work led to a publication at the SMC conference in 2024 8.

Show image description

Affordable sister board.

Show image description

Sister board with 32 inputs and outputs designed by Maxime Popoff during his PhD 8.

7.3.1 The Faust programming language

Faust is a synchronous functional programming language inspired by lambda calculus, combinatory logic, and John Backus' FP. Its semantics is centered around the concept of audio circuits. Programming in Faust primarily involves constructing new audio circuits by assembling primitive ones using an algebra of five composition operations. During the period, however, Faust has evolved beyond simple circuit assembly. The language has been extended with new primitive operations, such as Ondemand, Widget Modulation and Automatic Differentiation, which provide users with new higher-order functions for manipulating audio circuits.

7.3.2 The Faust compiler

To support Faust 's evolving functionality and optimize performance across a wider range of platforms, recent advancements in the Faust compiler—driven in part by the Syfala project—have introduced innovative compilation techniques that now target not only CPUs but also FPGAs, enabling efficient code generation and dynamic processing capabilities on diverse hardware architectures.

7.3.3 The Faust ecosystem

Beyond the Faust compiler, there exists a comprehensive ecosystem of complementary tools and resources. This includes extensive libraries for audio processing, synthesis, and effects, as well as development tools like the Faust Web IDE, which enables real-time code writing and testing. The ecosystem also features Faust architecture files, which facilitate seamless integration into diverse audio environments such as plugins, standalone applications, and web platforms. Deployment is further simplified with the faust2target suite, which provides tailored scripts for a wide range of use cases.

7.4.1 Activation functions in low precision with high accuracy

As machine learning hardware uses ever smaller number formats, the article 15 surveys simple and effective techniques for the implementation of activation functions in low precision (fewer than 16 bits) with high accuracy. The implementation combines a fixed-point centric approach, efficient function-specific range reduction techniques, and state-of-the-art polynomial approximation. The resulting trade-offs are studied on both FPGA and ASIC. Functions considered in this article include tanh, sigmoid, ReLU variants such as GELU, ELU, SiLU, and exp for stable softmax, but the methodology can apply to more functions. These techniques are implemented in an opensource hardware generator that produces readable synthesizable VHDL.

Show image description

ReLU variants in FPGAs

7.4.2 Design-space exploration for Reconfigurable Single Constant Multiplication

The scalar product of two vectors is the arithmetic core of many applications, in particular in machine learning and in digital signal processing. In both cases, one of the vectors can be considered constant for a long period of time. It is then interesting to reformulate the problem as a shift-and-add (SA) graph in such a way that the costly multiplications are replaced by additions and bit shifts. Our goal within the PhD thesis of Bastien Barbe is to propose a new reconfigurable computational unit that implements SA graphs for Multiple Constant Multiplication (MCM). The research hypothesis is that the number of configuration bits for such a Reconfigurable MCM unit (RMCM) is less than the bitwidth of the constant coefficients stored in memory and the overall cost is smaller than that of the generic multiplier. This is possible only because for the machine learning applications, and probably for DSP, we do not need to cover the whole range of representable values for target coefficients but only for a small subset of those. Indeed, recent results in quantization-aware training (QAT) for hardware neural networks demonstrates that weights could be adapted to fit a subset of numerical values that a realizable by the underlying hardware.

Show image description

RSCM unit

After a preliminary study of the design of an RMCM unit we concluded that the combinatorial explosion of the problem makes it difficult to find optimal parameters for the architecture (e.g. the depth of the SA graph, the number of adders, the routing between nodes and levels, etc) for a given probability distribution of target coefficients. Hence, we fell back to solving the problem for a single constant and its duplication, in parallel, up to as many constants as needed. In particular, we target 8-bit integer coefficients, which are realizable with an RSCM architecture in Fig. 10. Then, given a set of constants that should be realizable by each RSCM unit, or a distribution that should be more-or-less satisfied, we reduce as much as possible the number of configuration bits (in Fig. 10 it is the values of shifts, signs and multiplexers). This solution is based on design-space exploration using a constraint programming model and a custom branch-and-bound algorithm for optimization. This is work in progress, planned to be submitted in the first half of 2025.

7.4.3 Constant modular multipliers

Modular multipliers play a critical role in various cryptographic algorithms, including RSA and Diffie–Hellman, which require large coefficients (e.g., 2048 bits for RSA) to ensure security. This work, presented at RAIM in Perpignan, aims to reduce the resource usage of the Gribok’s et al. state-of-the-art modular multiplier implementation on FPGAs. By exploiting the constant nature of these coefficients, we are able to replace the DSPs with Multiple Constant Multipliers (MCM), thus creating a DSP-less architecture. To further reduce resource consumption, we made the design sequential, achieving up to 79% (277k to 58k) savings in LUTs at the cost of an increased but reasonable register usage +92% (1,5k to 20k) and latency +77% (557 Mhz to 130 Mhz). This approach enables resource-constrained FPGAs to handle large modular multiplications. The design is integrated into the FloPoCo arithmetic core generator and is freely available for use.

Participants: Florent de Dinechin, Stephane Letz, Romain Michon, Yann Orlarey, Tanguy Risset, Anastasia Volkova, Christine Solnon.

9.1.1 Associate Teams in the framework of an Inria International Lab or in the framework of an Inria International Program

9.2.1 ANR FAST

Embedded systems for audio and multimedia are increasingly used in the arts and culture (e.g., interactive systems, musical instruments, virtual and augmented reality, artistic creation tools, etc.). They are typically based on a CPU (Central Processing Unit) which limits their computational power and induces some latency. FPGAs (Field Programmable Gate Arrays) can be seen as a solution to these problems. However, these types of chips are extremely complex to program, making them largely inaccessible to musicians, digital artists and makers communities.

The goal of the FAST ANR project is to enable high-level programming of FPGA-based platforms for multichannel ultra-low-latency audio processing using the Faust programming language (a standard in the field of computer music). We plan to use this system for various applications ranging from sound synthesis and processing to active sound control and artificial sound field/room acoustics.

FAST officially started in March 2021 and will end in January 2025. It gathers the strength of GRAME-CNCM, CITI Lab (INRIA/INSA Lyon), and LMFA (École Centrale Lyon).

9.2.2 PEPR HoliGrail

A key challenge to reach the maximal efficiency is to match algorithms with the underlying hardware. As the end of Moore’s law steadily approaches, hardware becomes increasingly heterogeneous, and the interplay between algorithms and hardware even more important. From this point of view, artificial intelligence algorithms are not efficient when run on commodity hardware such as microprocessors and GPUs. The last decade has seen a boom of accelerators for common inference tasks (first in line was Google’s TPU, followed by many others), now embedded in many consumer products. These accelerators offer hardware that better matches the algorithms, but are still far from achieving the “inference per joule” performance of the Human brain. Training deep neural networks (DNNs) is even less efficient, requiring orders of magnitude more computation operations than inference.

We believe that there is a significant room for improvements in both “inference per joule" and "training per joule”. The vision of this action is to create a synergy with the research on the foundations of AI frugality (as proposed in SHARP action) to propose cutting-edge methods that significantly improve the energy efficiency of both inference and training of a model. We will propose (i) more compact and efficient number representations that still maintain a quality of inference or training close to the reference, (ii) hardware-aware training algorithms that enhance certain types of sparsity (e.g., more structured), coding compactness (aggressive quantization, maximum entropy) and tensor transformations. Most state-of-the-art solutions are agnostic of the hardware they run on. By taking advantage of this interplay between the hardware and the algorithms, we can achieve breakthroughs beyond current solutions, in particular by developing (iii) efficient hardware mechanisms, especially optimized to take advantage of sparsity, extreme quantization and ad-hoc number representations, together with (iv) compiler optimizations, to demonstrate the effectiveness of the proposed methods. Our approaches are holistic in the sense that they will jointly optimize the whole computing stack of AI, i.e., at the algorithm, arithmetic, compiler and hardware levels.

PEPR HOLIGRAIL kicked off in March 2024 and will end in 2029 (extension has been anticipated). The project combines following partners: Inria/IRISA Taran (Univ. Rennes, Inria, CNRS), List /LIAE (Université Paris Saclay, CEA), Inria Corse (Université Grenoble Alpes), TIMA SLS (CNRS, Université Grenoble Alpes), CITI lab (INSA Lyon / Inria), List / LVML (Université Paris Saclay, CEA). The scientific leaders are Olivier Bichler (CEA List) and Olivier Sentieys (Inria Taran).

9.3.1 FIL inter-lab project FORTE

With this 2-year inter-lab project funded by FIL (Fédération d'Informatique de Lyon) we will be working towards frugality of DNN training by acting on two levels: (1) the fundamental level of optimization algorithms used for training DNNs and (2) application level of number representation/ precision choice for implementation in embedded systems, in particular Filed Programmable Gate Arrays (FPGAs). With this collaboration between Anastasia Volkova (Inria Emeraude), an expert in computer arithmetic, and Elisa Riccietti (Inria Okham), expert in continuous optimization, the goal is to achieve the frugality by working at the interface of these two domains to explore the design of new mixed numerical precision algorithms that are offer increased performance in a resource-restricted environment while minimising computational cost.

10.1.1 Scientific events: organisation

10.1.2 Scientific events: selection

10.1.3 Invited talks

• Christine Solnon has given an invited talk at the 23rd workshop on Constraint Modelling and Reformulation (ModRef) in September 2024, and she has given invited seminars at GSCOP and at ENS Lyon
• Florent de Dinechin was invited to give a keynote talk at the International Conference on Field-Programmable Logic and Applications (FPL) in September 2024.
• Romain Michon was a keynote speaker at the “International Open Seminar on Synchronous Reactive Programming” (SYNCHRON) in November 2024.
• Romain Michon has given invited talks at the University of Michigan (USA) and Stanford University (USA) on October 2024.

10.1.4 Leadership within the scientific community

• Romain Michon has been nominated President of the Sound and Music Computing (SMC) Network that steers the planning of the SMC conference, one of the most prestigious scientific event in the sound and music technology community.
• Since September 2024, Christine Solnon is the Deputy Scientific Director (DSA) of the Inria Lyon center.
• Christine Solnon is member of the advisory and scientific boards (comité de direction et comité scientifique) of the GDR RADIA
• Florent de Dinechin is a member of the scientific committee of the DEEPGREEN project (https://deepgreen.ai).

10.2.1 Teaching

• Tanguy Risset is professor at the Telecommunications Department of Insa Lyon.
• Florent de Dinechin is a professor at the Computer Science Department of Insa Lyon and head of the 4th year. He also teaches computer architecture at ENS-Lyon.
• Christine Solnon is a professor at the Computer Science Department of Insa Lyon.
• Romain Michon is a part-time associate professor at the Telecommunications Department of Insa Lyon.
• Romain Michon teaches 2 courses as part of the RIM/RAN Masters Program at the université of Saint-Étienne.
• Romain Michon teaches 2 one week workshops at Aalborg University in Copenhagen every year.
• Stephane Letz teaches 1 course as part of the RIM/RAN Masters Program at the université of Saint-Étienne.

10.2.2 Juries

• Christine Solnon was in the PhD jury of:
  • Maryia Hryhoryeva (ENTPE, Lyon, 2024), as president
  • Assaad Oussama Zeghina (U. Strasbourg, 2024), as president
  • Shuolin Li (U. Aix Marseille, 2024), as president
  • Alexandre Borthomieu (U. Grenoble Alpes, 2024), as reviewer
  • Jovial Cheukam Ngouonou (U. Laval, Canada, 2024)
  • Maxime Popoff (U. Lyon, 2024)
• Christine Solnon was in the HdR jury of
  • Laure Brisoux-Devendeville (U. de Picardie Jules Verne, 2024)
• Tanguy Risset was in the Phd Jury of:
  • Marie Badaroux (U. Grenoble Alpes, 2024 ) as a reviewer.
  • Hugo Reymond (U. Rennes, 2024) as a reviewer.
  • Arthur Vianès (U. Grenoble Alpes, 2024 ) as jury president.
• Tanguy Risset was in the HdR Jury of:
  • Karol Desnos (U. Rennes, 2024), as a reviewer.
  • Romain Michon (U. Lyon, 2024), as HDR referent.
• Florent de Dinechin was in the Phd Jury of:
  • Cedric Gernigon (U. Rennes, 2024), as a reviewer.
  • Corentin Ferry (U. Rennes, ).
• Romain Michon was in the PhD Jury of:
  • Shihong Ren (U. Saint-Étienne, 2024), as a reviewer.
  • Loïc Alexandre (École Centrale de Lyon, 2024), as a reviewer.
  • Bérangère Daviaud (Polytech Angers, 2024), as a reviewer.
• Anastasia Volkova was in the PhD Jury of:
  • Debasmita Lohar (Max Plank Institute, 2024), as a reviewer
  • Hugo Le Blevec (IMT Atlantique de Brest, 2024)

10.3.1 Participation in Live events

Christine Solnon has given a seminar on environmental and societal impacts of AI at ENSSIB, and she has participated at a round table about AI after a public screening of the film "The thinking game".

International journals

• 2 articleX.Xiao Peng, F.François Schwarzentruber, O.Olivier Simonin and C.Christine Solnon. Spanning-tree based coverage for a tethered robot.IEEE Robotics and Automation LettersDecember 2024, 1-8In press. HAL
• 3 articleT. A.Thomas Albert Rushton, R.Romain Michon, S.Stefania Serafin, T.Tanguy Risset and S.Stéphane Letz. Networked microcontrollers for accessible, distributed spatial audio.Frontiers in Virtual Reality5November 2024HALDOIback to text

International peer-reviewed conferences

• 4 inproceedingsP.Pierre Cochard, M.Maxime Popoff, R.Romain Michon and T.Tanguy Risset. Programming FPGA Platforms for Real-Time Audio Signal Processing In C++.Proceedings of the 2024 Sound and Music Computing Conference (SMC-24)Proceedings of the 2024 Sound and Music Computing Conference (SMC-24)Porto, PortugalJuly 2024HALback to text
• 5 inproceedingsP.Pierre Cochard, J.Jurek Weber, R.Romain Michon, T.Tanguy Risset and S.Stéphane Letz. Ethernet Real-time Audio Transmission to FPGA.2024 IEEE 5th International Symposium on the Internet of Sounds (IS2)5th IEEE International Symposium on the Internet of SoundsErlangen, GermanyIEEE2024, 1-7HALDOIback to text
• 6 inproceedingsS.Stéphane Letz, R.Romain Michon and Y.Yann Orlarey. What’s New in the Faust Ecosystem in 2024?IFC 2024 - 4th International Faust ConferenceTurin (Italie), ItalyNovember 2024, pp.27-33HAL
• 7 inproceedingsY.Yann Orlarey, S.Stéphane Letz and R.Romain Michon. Widget Modulation in FAUST.Proceedings of the 4th International Faust ConferenceInternational Faust ConferenceTurin, ItalyNovember 2024HALback to text
• 8 inproceedingsM.Maxime Popoff, R.Romain Michon and T.Tanguy Risset. Enabling Affordable and Scalable Audio Spatialization With Multichannel Audio Expansion Boards for FPGA.Proceedings of the 2024 Sound and Music Computing Conference (SMC-24)Proceedings of the 2024 Sound and Music Computing Conference (SMC-24)Porto, PortugalJuly 2024HALback to textback to textback to text
• 9 inproceedingsT. A.Thomas Albert Rushton. Differentiable DSP in Faust.IFC 2024 - 4th International Faust ConferenceTurin, ItalyDecember 2024HALback to text
• 10 inproceedingsA.Anastasia Volkova, R.Rémi Garcia, F.Florent de Dinechin and M.Martin Kumm. Hardware-optimal digital FIR filters: one ILP to rule them all and in faithfulness bind them.Proceedings of the Asilomar conference2023 Asilomar Conference on Signals, Systems, and ComputersAsilomar, United StatesMarch 2024HALback to text

National peer-reviewed Conferences

• 11 inproceedingsA.Agathe Herrou, F.Florent de Dinechin, S.Stéphane Letz, Y.Yann Orlarey and A.Anastasia Volkova. Towards Fixed-Point Formats Determination for Faust Programs.Journées d'Informatique Musicale 2024Marseille, FranceMay 2024HAL

Scientific books

• 12 bookF.Florent de Dinechin and M.Martin Kumm. Application-Specific Arithmetic.Springer International Publishing2024HALDOIback to text

Edition (books, proceedings, special issue of a journal)

• 13 proceedingsProceedings of the 4th International Faust Conference.Proceedings of the 4th International Faust ConferenceDecember 2024HALback to text

Reports & preprints

• 14 reportP.Pierre Cochard, J.Jurek Weber, R.Romain Michon, T.Tanguy Risset and S.Stephane Letz. Open Source Ethernet Real-time Audio Transmission to FPGA.RR-9542INRIAFebruary 2024, 16HAL
• 15 miscT.Tom Hubrecht, O.Orégane Desrentes and F.Florent de Dinechin. Activations in Low Precision with High Accuracy.November 2024HALback to text

Scientific popularization

• 16 articleC.Christine Solnon. Nos écrans : manipulation, addiction et remède.La Recherche5792024HAL
• 17 article C.Christine Solnon. Peut-on remplacer le raisonnement humain par des algorithmes ? La Recherche 580 January 2025 HAL