Section: New Results

Sparse Representations, Inverse Problems, and Dimension Reduction

Sparsity, low-rank, dimension-reduction, inverse problem, sparse recovery, scalability, compressive sensing

The team has had a substantial activity ranging from theoretical results to algorithmic design and software contributions in the fields of sparse representations, inverse problems, and dimension reduction.

Algorithmic and Theoretical results on Computational Representation Learning

Participants : Rémi Gribonval, Nicolas Bellot, Cássio Fraga Dantas.

Main collaborations: Luc Le Magoarou (IRT b$<>$com, Rennes), Nicolas Tremblay (GIPSA-Lab, Grenoble), R. R. Lopes and M. N. Da Costa (DSPCom, Univ. Campinas, Brazil)

An important practical problem in sparse modeling is to choose the adequate dictionary to model a class of signals or images of interest. While diverse heuristic techniques have been proposed in the literature to learn a dictionary from a collection of training samples, classical dictionary learning is limited to small-scale problems. Inspired by usual fast transforms, we proposed a general dictionary structure (called FA$\mu$ST for Flexible Approximate Multilayer Sparse Transforms) that allows cheaper manipulation, and an algorithm to learn such dictionaries together with their fast implementation.

The principle and its application to image denoising appeared at ICASSP 2015 [81] and an application to speedup linear inverse problems was published at EUSIPCO 2015 [80]. A Matlab library has been released (see FA$\mu$ST in Section 6.2) to reproduce the experiments from the comprehensive journal paper published in 2016 [83], which additionally includes theoretical results on the improved sample complexity of learning such dictionaries. Pioneering identifiability results have been obtained in the Ph.D. thesis of Luc Le Magoarou on this topic [84].

We further explored the application of this technique to obtain fast approximations of Graph Fourier Transforms. A conference paper on this latter topic appeared in ICASSP 2016 [82], and a journal paper has been published this year [17] where we empirically show that $𝒪(nlogn)$ approximate implementations of Graph Fourier Transforms are possible for certain families of graphs. This opens the way to substantial accelerations for Fourier Transforms on large graphs. The approximation error of such Fast Graph Fourier Transforms has been studied in a conference paper [31].

A C++ version of the FA$\mu$ST software library has been developed (see Section 6) to release the resulting algorithms and interface them with both Matlab and Python (work in progress).

As a complement to the FA$\mu$ST structure for matrix approximation, we proposed a learning algorithm that constrains the dictionary to be a sum of Kronecker products of smaller sub-dictionaries. A special case of the proposed structure is the widespread separable dictionary. This approach, named SuKro, was evaluated experimentally on an image denoising application [39].

We combined accelerated matrix-vector multiplications offered by FAuST matrix approximations with dynamic screening [53], that safely eliminates inactive variables to speedup iterative sparse recovery algorithms. First, we showed how to obtain safe screening rules for the exact problem while manipulating an approximate dictionary. We then adapted an existing screening rule to this new framework and define a general procedure to leverage the advantages of both strategies. Significant complexity reductions were obtained in comparison to screening rules alone [35].

Theoretical results on generalized matrix inverses, and the sparse pseudo-inverse

Participant : Rémi Gribonval.

Main collaboration: Ivan Dokmanic (University of Illinois at Urbana Champaign, USA)

We studied linear generalized inverses that minimize matrix norms. Such generalized inverses are famously represented by the Moore-Penrose pseudoinverse (MPP) which happens to minimize the Frobenius norm. Freeing up the degrees of freedom associated with Frobenius optimality enables us to promote other interesting properties. In a first part of this work [37], we looked at the basic properties of norm-minimizing generalized inverses, especially in terms of uniqueness and relation to the MPP. We first showed that the MPP minimizes many norms beyond those unitarily invariant, thus further bolstering its role as a robust choice in many situations. We then concentrated on some norms which are generally not minimized by the MPP, but whose minimization is relevant for linear inverse problems and sparse representations. In particular, we looked at mixed norms and the induced ${\ell }^p\to {\ell }^q$ norms.

An interesting representative is the sparse pseudoinverse which we studied in much more detail in a second part of this work [38], motivated by the idea to replace the Moore-Penrose pseudoinverse by a sparser generalized inverse which is in some sense well-behaved. Sparsity implies that it is faster to apply the resulting matrix; well-behavedness would imply that we do not lose much in stability with respect to the least-squares performance of the MPP. We first addressed questions of uniqueness and non-zero count of (putative) sparse pseudoinverses. We showed that a sparse pseudoinverse is generically unique, and that it indeed reaches optimal sparsity for almost all matrices. We then turned to proving a stability result: finite-size concentration bounds for the Frobenius norm of p-minimal inverses for $1\le p\le 2$. Our proof is based on tools from convex analysis and random matrix theory, in particular the recently developed convex Gaussian min-max theorem. Along the way we proved several results about sparse representations and convex programming that were known folklore, but of which we could find no proof.

Algorithmic exploration of large-scale Compressive Learning via Sketching

Participants : Rémi Gribonval, Nicolas Keriven, Antoine Chatalic, Antoine Deleforge.

Main collaborations: Patrick Perez (Technicolor R&I France, Rennes), Anthony Bourrier (formerly Technicolor R&I France, Rennes; then GIPSA-Lab, Grenoble), Antoine Liutkus (ZENITH Inria project-team, Montpellier), Nicolas Tremblay (GIPSA-Lab, Grenoble), Phil Schniter & Evan Byrne (Ohio State University, USA)

Sketching for Large-Scale Mixture Estimation. When fitting a probability model to voluminous data, memory and computational time can become prohibitive. We proposed during the Ph.D. thesis of Anthony Bourrier [54], [57], [55], [56] a framework aimed at fitting a mixture of isotropic Gaussians to data vectors by computing a low-dimensional sketch of the data. The sketch represents empirical moments of the underlying probability distribution. Deriving a reconstruction algorithm by analogy with compressive sensing, we experimentally showed that it is possible to precisely estimate the mixture parameters provided that the sketch is large enough. The proposed algorithm provided good reconstruction and scaled to higher dimensions than previous probability mixture estimation algorithms, while consuming less memory in the case of voluminous datasets. It also provided a potentially privacy-preserving data analysis tool, since the sketch does not explicitly disclose information about individual datum.

During the Ph.D. thesis of Nicolas Keriven [12], we consolidated our extensions to non-isotropic Gaussians, with a new algorithm called CL-OMP [73] and conducted large-scale experiments demonstrating its potential for speaker verification. A conference paper appeared at ICASSP 2016 [72] and the journal version has been accepted this year [44], accompanied by a toolbox for reproducible research (see SketchMLBox, Section 6.3). Nicolas Keriven was awarded the SPARS 2017 Best Student Paper Award for this work [46].

Sketching for Compressive Clustering and beyond. Last year we started a new endeavor to extend the approach beyond the case of Gaussian Mixture Estimation.

First, we showed empirically that sketching can be adapted to compress a training collection while still allowing large-scale clustering. The approach, called “Compressive K-means”, uses CL-OMP at the learning stage and is described in a paper published at ICASSP 2017 [23]. In the high-dimensional setting, it is also possible to substantially speedup both the sketching stage and the learning stage by replacing Gaussian random matrices with fast random matrices in the sketching procedure. This has been demonstrated by Antoine Chatalic during his internship and submitted for publication to a conference. An alternative algorithm for cluster recovery from a sketch was proposed this year, based on simplified hybrid generalized approximate message passing (SHyGAMP). Numerical experiments suggest that this approach is more efficient than CL-OMP (in both computational and sample complexity) and more efficient than k-means++ in certain regimes [25].

Then, we leveraged the parallel between the mathematical expression of sketched clustering and super-resolution to explore the potential of sketching and CL-OMP for the stable recovery of signals made of few spikes (in the gridless setting) from few random weighted Fourier measurements [48]. We also demonstrated that sketching can be used in blind source localization and separation, by learning mixtures of alpha-stable distributions [45], see details in Section 7.4.2.

Theoretical results on Low-dimensional Representations, Inverse problems, and Dimension Reduction

Participants : Rémi Gribonval, Yann Traonmilin.

Main collaboration: Mike Davies (University of Edinburgh, UK), Gilles Puy (Technicolor R&I France, Rennes),

Inverse problems and compressive sensing in Hilbert spaces.

Many inverse problems in signal processing deal with the robust estimation of unknown data from underdetermined linear observations. Low dimensional models, when combined with appropriate regularizers, have been shown to be efficient at performing this task. Sparse models with the ${\ell }^1$-norm or low-rank models with the nuclear norm are examples of such successful combinations. Stable recovery guarantees in these settings have been established using a common tool adapted to each case: the notion of restricted isometry property (RIP). Last year we published a comprehensive paper [97] establishing generic RIP-based guarantees for the stable recovery of cones (positively homogeneous model sets) with arbitrary regularizers. We also described a generic technique to construct linear maps from a Hilbert space to ${ℝ}^m$ that satisfy the RIP [20]. These results have been surveyed in a book chapter completed this year[49].

Information preservation guarantees with low-dimensional sketches. We established a theoretical framework for sketched learning, encompassing statistical learning guarantees as well as dimension reduction guarantees. The framework provides theoretical grounds supporting the experimental success of our algorithmic approaches to compressive K-means, compressive Gaussian Mixture Modeling, as well as compressive Principal Component Analysis (PCA). A comprehensive preprint has been completed and submitted to a journal [42]. Future work will include expliciting the impact of the proposed framework on a wider set of concrete learning problems.

Algorithmic Exploration of Sparse Representations in Virtual Reality and Neurofeedback

Participant : Rémi Gribonval.

Ferran Argelaget & Anatole Lecuyer (HYBRID Inria project-team, Rennes), Saman Noorzadeh, Pierre Maurel & Christian Barillot (VISAGES Inria project-team, Rennes)

In collaboration with the VISAGES team we validated a technique to estimate brain neuronal activity by combining EEG and fMRI modalities in a joint framework exploiting sparsity [34].

Our work in collaboration with the HYBRID team on sparse dictionary learning for spatial and rotation invariant gesture recognition has been published this year [24]. Our work on multi-modal

An Alternative Framework for Sparse Representations: Sparse “Analysis” Models

Participants : Rémi Gribonval, Nancy Bertin, Clément Gaultier.

Main collaborations: Srdan Kitic (Technicolor R & I France, Rennes), Laurent Albera and Siouar Bensaid (LTSI, Univ. Rennes)

In the past decade there has been a great interest in a synthesis-based model for signals, based on sparse and redundant representations. Such a model assumes that the signal of interest can be composed as a linear combination of few columns from a given matrix (the dictionary). An alternative analysis-based model can be envisioned, where an analysis operator multiplies the signal, leading to a cosparse outcome.

Building on our pioneering work on the cosparse model [71], [90][8], successful applications of this approach to sound source localization, brain imaging and audio restoration have been developed in the team during the last years [74], [76], [75], [51]. Along this line, two main achievements were obtained this year. First, and following the publication in 2016 of a journal paper embedding in a unified fashion our results in source localization [5], we wrote a book chapter (currently in press) gathering our contributions in physics-driven cosparse regularization, including new results and algorithms demonstrating the versatility, robustness and computational efficiency of our methods in realistic, large scale scenarios in acoustics and EEG signal processing [47]. Second, we continued extending the cosparse framework on audio restoration problems: improvements on our released real-time declipping algorithm (A-SPADE - see Section 6), new results on the denoising task [41], [28], and the submission of a journal paper encompassing several denoising and declipping methods in a common framework [40].