Download a huge Python pickle dictionary

Question

I generated the pickle.dump () file with a size of about 5 GB. It takes about noon to download this file, and about 50 GB of RAM. My question is whether it is possible to read this file by accessing a separate record by record (one at a time), rather than loading it all into memory, or if you have any other suggestions on accessing data in such a file.

Many thanks.

Answer 1

There is absolutely no doubt that this should be done using a database, and not for pickle databases designed specifically for this kind of problem.

, , sqllite . , , SQL, , , . , , SQLAlchemy, " Mapper ", , .

import os
import sqlite3

# an enormous dictionary too big to be stored in pickle
my_huge_dictionary = {"A": 1, "B": 2, "C": 3, "D": 4}

# create a database in the file my.db
conn = sqlite3.connect('my.db')
c = conn.cursor()

# Create table with two columns: k and v (for key and value). Here your key
# is assumed to be a string of length 10 or less, and your value is assumed
# to be an integer. I'm sure this is NOT the structure of your dictionary;
# you'll have to read into SQL data types
c.execute("""
create table dictionary (
k char[10] NOT NULL,
v integer NOT NULL,
PRIMARY KEY (k))
""")

# dump your enormous dictionary into a database. This will take a while for
# your large dictionary, but you should do it only once, and then in the future
# make changes to your database rather than to a pickled file.
for k, v in my_huge_dictionary.items():
    c.execute("insert into dictionary VALUES ('%s', %d)" % (k, v))

# retrieve a value from the database
my_key = "A"
c.execute("select v from dictionary where k == '%s'" % my_key)
my_value = c.next()[0]
print my_value

!

Answer 2

- , - ZODB, , - , , ,

ZODB Zope - Python, Plone .

, Zope - , , SQL.

http://www.zodb.org/