Geek Answers Handbook

How does numpy handle mmap on npz files?

I have a case where I would like to open a numpy compressed file using mmap mode, but it seems that I can not find any documentation on how it will work under covers. For example, will it unpack the archive in memory and then mmap? Will it be unpacked on the fly?

No documentation is available for this configuration.

+4
source share
1 answer

The short answer, based on code review, is that archiving and compression using np.savezor gzipis incompatible with file access in mmap_mode. This is not just a question of how this is done, but is it even possible to do so.

np.load

elif isinstance(file, gzip.GzipFile):
    fid = seek_gzip_factory(file)
...
    if magic.startswith(_ZIP_PREFIX):
        # zip-file (assume .npz)
        # Transfer file ownership to NpzFile
        tmp = own_fid 
        own_fid = False
        return NpzFile(fid, own_fid=tmp)
...
    if mmap_mode:
        return format.open_memmap(file, mode=mmap_mode)

np.lib.npyio.NpzFile. npz ZIP- .npy. () () (, obj[key]). There no provision in its code for opening those individual files in mmap_mode`.

, , np.savez, mmap. ZIP gzip, np.load.

, np.save, gzipped? , format.open_memmap file, fid ( gzip).

open_memmap np.lib.npyio.format. , file , fid. np.memmap. gzip.

+6

More articles:


All Articles