Loading python zip with modules from memory

Question

So let's say I have a zip file with modules / classes inside. Then I read this file - I read the binary ("rb") to save it in memory. How can I take this zip file into memory and load a module from it. Do I need to write an import hook for this? You can't just run exec on binary zip data from memory, right?

I know that it is just easy to load a module from a simple zip file on disk, as this is done automatically with python2.7. I, however; want to know if this is possible through memory.

Update: Many people mention importing zip from disk. The problem is that I want to import zip from NOT disk memory. I will obviously read it from disk and byte byte. I want to take all these bytes from memory that make up a zip file and use it as a regular import.

Answer 1

EDIT: Fixed ZipImporter to work in everything (I think)

Test data:

mkdir mypkg vim mypkg/__init__.py vim mypkg/test_submodule.py 

__init__.py Contents:

 def test(): print("Test") 

test_submodule.py Contents:

 def test_submodule_func(): print("This is a function") 

Create a test zip code (on Mac):

 zip -r mypkg.zip mypkg rm -r mypkg # don't want to accidentally load the directory 

Special zip import in inmem_zip_importer.py :

 import os import imp import zipfile class ZipImporter(object): def __init__(self, zip_file): self.z = zip_file self.zfile = zipfile.ZipFile(self.z) self._paths = [x.filename for x in self.zfile.filelist] def _mod_to_paths(self, fullname): # get the python module name py_filename = fullname.replace(".", os.sep) + ".py" # get the filename if it is a package/subpackage py_package = fullname.replace(".", os.sep, fullname.count(".") - 1) + "/__init__.py" if py_filename in self._paths: return py_filename elif py_package in self._paths: return py_package else: return None def find_module(self, fullname, path): if self._mod_to_paths(fullname) is not None: return self return None def load_module(self, fullname): filename = self._mod_to_paths(fullname) if not filename in self._paths: raise ImportError(fullname) new_module = imp.new_module(fullname) exec self.zfile.open(filename, 'r').read() in new_module.__dict__ new_module.__file__ = filename new_module.__loader__ = self if filename.endswith("__init__.py"): new_module.__path__ = [] new_module.__package__ = fullname else: new_module.__package__ = fullname.rpartition('.')[0] return new_module 

Using:

 In [1]: from inmem_zip_importer import ZipImporter In [2]: sys.meta_path.append(ZipImporter(open("mypkg.zip", "rb"))) In [3]: from mypkg import test In [4]: test() Test function In [5]: from mypkg.test_submodule import test_submodule_func In [6]: test_submodule_func() This is a function 

---

(from efel) one more thing ...:

To read directly from memory, you will need to do this:

 f = open("mypkg.zip", "rb") # read binary data we are now in memory data = f.read() f.close() #important! close the file! we are now in memory # at this point we can essentially delete the actual on disk zip file # convert in memory bytes to file like object zipbytes = io.BytesIO(data) zipfile.ZipFile(zipbytes) 

Answer 2

read test.txt from zip (do not unpack or write to disk):

python 3 (if you are using py2.x, you must change the py3 zip API to py2)

 import zipfile file_rst = [] with zipfile.ZipFile('/test/xxx.zip') as my_zip: with my_zip.open('test.txt') as my_file: for line in my_file: file_rst.append(line.strip())