Section: New Results

Fast and Memory Optimal Low-Rank Matrix Approximation

In this paper, we revisit the problem of constructing a near-optimal rank $k$ approximation of a matrix $M\in {[0,1]}^{m\times n}$ under the streaming data model where the columns of $M$ are revealed sequentially. We present SLA (Streaming Low-rank Approximation), an algorithm that is asymptotically accurate, when $k{s}_{k+1}\left(M\right)=o\left(\sqrt{mn}\right)$ where $s_{k+1}\left(M\right)$ is the $(k+1)$-th largest singular value of $M$. This means that its average mean-square error converges to 0 as $m$ and $n$ grow large (i.e., $\parallel {\widehat{M}}^{\left(k\right)}-M^{\left(k\right)}{\parallel }_F^2=o\left(mn\right)$ with high probability, where ${\widehat{M}}^{\left(k\right)}$ and $M^{\left(k\right)}$ denote the output of SLA and the optimal rank $k$ approximation of $M$, respectively). Our algorithm makes one pass on the data if the columns of $M$ are revealed in a random order, and two passes if the columns of $M$ arrive in an arbitrary order. To reduce its memory footprint and complexity, SLA uses random sparsification, and samples each entry of $M$ with a small probability $\delta$. In turn, SLA is memory optimal as its required memory space scales as $k(m+n)$, the dimension of its output. Furthermore, SLA is computationally efficient as it runs in $O\left(\delta kmn\right)$ time (a constant number of operations is made for each observed entry of $M$), which can be as small as $O(klog{\left(m\right)}^{4}n)$ for an appropriate choice of $\delta$ and if $n\ge m$.