Geek Answers Handbook

Python json memory bloat

import json
import time
from itertools import count

def keygen(size):
    for i in count(1):
        s = str(i)
        yield '0' * (size - len(s)) + str(s)

def jsontest(num):
    keys = keygen(20)
    kvjson = json.dumps(dict((keys.next(), '0' * 200) for i in range(num)))
    kvpairs = json.loads(kvjson)
    del kvpairs # Not required. Just to check if it makes any difference                            
    print 'load completed'

jsontest(500000)

while 1:
    time.sleep(1)

Linux top indicates that the python process contains ~ 450 MB of RAM after the jsontest function completes. If the call to " json.loads " is omitted, this problem is not observed. A gc.collect after executing this function frees up memory .

It seems that the memory is not stored in any caches or memory allocation of python's internal memory, since an explicit call to gc.collect frees up memory.

Is this because the garbage collection threshold (700, 10, 10) was never reached?

I made the code after jsontest to simulate a threshold. But it did not help.

+5
1

import gc
gc.set_debug(gc.DEBUG_STATS)

, . , jsontest, .

print gc.get_count()

. - 0; ( ) - , 0 ( 1) 1 ( 2). jsontest, , (548, 6, 0) - ( , Python). , , .

, . , ( , ).

+2

More articles:


All Articles