Parallel Processing with Numpy.loadtxt ()

Question

I have a file> 100 MB in size that needs to be read with numpy.loadtxt()

The reading part is the main bottleneck in my code. A file with 72 MB requires 17.3s

Is it possible to somehow read a file in parallel using loadtxt()

If possible, without breaking the file.

+1

python numpy parallel-processing

Tengis Mar 24 '14 at 17:20

source share

1 answer

horta · Accepted Answer · 2014-03-24T17:52:55+0000

It seems that numpy.loadtxt () is your problem.

According to these sites, you better not use the numpy download function.

pandas.read_csv and read_table should be useful from the pandas module.