Geek Answers Handbook

Matrix operations on large matrices on limited memory

I need to perform some operations with matrices on my computer. These matrices are large 1,000,000x1,000,000 or more, some operations require TB memory. Obviously, they cannot be directly loaded into memory and calculated. What approaches can I use to solve these matrices on my computer? Assuming that the matrices cannot be further reduced using matrix optimizations and are already stored in compact form. I am thinking of using some kind of memory mapping scheme, but I need some ideas.

+4
source share
1 answer

Two suggestions:

  • Use the mmap2 system call to map files containing both input and output data. This allows you to display files up to 2 ^ 44 bytes and process them as if they were already in memory. That is, you simply use the standard pointer syntax for accessing data, and the OS takes care to read or write it from / to disk without worrying about it. Not only that, but mmap is many times significantly faster than manual file input / output - See this SO post.

  • Read “ What Every Programmer Should Know About Memory, ” by Ulrich Drapper. One of the problems he faces is the very optimizing operations with matrices.

+2
source

More articles:


All Articles