Scipy Large Sparse Matrix

Question

I am trying to use large allowed matrices of size 10 ^ 5 × 10 ^ 5, but it seems to work against scipy:

n = 10 ** 5 x = scipy.sparse.rand(n, n, .001) 

gets

 ValueError: Trying to generate a random sparse matrix such as the product of dimensions is greater than 2147483647 - this is not supported on this machine 

Does anyone know why there is a limit, and can I avoid it? (fyi, I use macbook air with 4GB memory and distribution enthusiastically)

Answer 1

This is a limitation resulting from the implementation of scipy.sparse.rand() . You can generate your own random matrix generation to get around this limitation:

 n = 10 ** 5 density = 1e-3 ij = numpy.random.randint(n, size=(2, n * n * density)) data = numpy.random.rand(n * n * density) matrix = scipy.sparse.coo.coo_matrix((data, ij), (n, n))