What Python modules are available for saving and loading data?

Question

StackOverflow has a lot of scattered posts regarding Python modules used to save and load data.

I myself am familiar with json and pickle , and I heard about pytables . There are probably more. In addition, each module seems to correspond to a specific purpose and has its own limitations (for example, loading a large list or a dictionary with pickling takes a lot of time, if you work at all). Therefore, it would be nice to have a proper overview of the possibilities.

Could you help provide a complete list of modules used to save and load data describing for each module:

• What is the overall goal of the module,
• its limits
• why do you choose this module over others?

Answer 1

marshal :

• Pros:

  • Can read and write Python values ​​in binary format. Therefore, it is much faster than pickle (which is based on characters).

• Against:

  • Not all types of Python objects are supported. Some unsupported types, such as subclasses of built-in functions, will display correctly and parallelize correctly.
  • Not intended to protect against erroneous or malicious data.
  • Python compilers reserve the right to reverse the format of the marshal, if necessary.

---

shelve

• Arguments:

  • Values ​​on a shelf can be essentially arbitrary Python objects.

• Against:

  • It does not support simultaneous read / write access to private objects.

---

ZODB (suggested by @Duncan)

• Pro:

  • transparent perseverance
  • full transaction support
  • pluggable storage
  • scalable architecture

• Against

  • not included in the standard library.
  • it is not possible (easy) to reload data if the original python object model used for saving is not available (consider versioning issues and data portability).

Answer 2

There is an overview of the standard lib data storage modules.