Storage of a large, but low-level matrix

Question

I know that R has packages for efficient storage of sparse matrices. Is there a way to efficiently store a low rank matrix? For example:

A <- matrix(rnorm(1e6), nrow=1e5, ncol=1e1) B <- A %*% t(A) 

Now B too large to hold in memory, but it is low in rank. Is there a way to build and save B efficient way, so that some basic reading methods ( rowSums , colSums , etc.) are colSums on the fly to trade for the processor or memory?

Answer 1

Your question is already the answer: for the efficient storage of such a matrix with a low rank, you create a data structure containing both factors. If matrix vector multiplication is required, this can be done from right to left using matrix-vector products of factors.

One application of this strategy and data structure can be found in implementations of quasi-Newton methods with limited memory Broyden or BFGS.

Answer 2

Here is another approach, although I miss the experience to see how effective this is for large matrices:

If the rank is low, this means that the matrix contains many irrelevant rows, which are linear combinations of others. If the matrix is ​​a linear system of equations, you can develop an algorithm that sequentially deletes these rows.

To check if a row is irrelevant, check if the matrix rank is not without this row. To calculate the rank of the matrix, see this and which answer.