How to implement intensive python with memory script for testing

Question

I applied the group rule for a specific user, and I would like to check if the memory of programs running from the specified user is limited, as expected. I tried with the following script:

import string import random if __name__ == '__main__': d = {} i = 0; for i in range(0, 100000000): val = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(200)) # generate ramdom string of size 200 d[i] = val if i % 10000 == 0: print i 

When I tracked the process using the ps command, it turned out that% MEM was increased to 4.8 and never changed when the cgroups service was turned on and off:

 $ ps aux | grep mem_intensive.py USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jason 11531 88.5 4.8 3312972 3191236 pts/0 R+ 22:16 0:07 python mem_intensive.py 

In this case, the total memory is 62 GB, while 4.8% is about 3 GB. I set a limit of 4 GB without any other processes running on this user.

So can anyone give me some idea about this problematic python script? Thanks in advance.

Answer 1

If you want to see if cgroup is working, just set the limit to 100 MB and try running the script. The point is not to see if the large limit works better or worse than the small one - you just want to make sure that the limit is met. A small limit is enough for this.

To make sure the dict grows as expected, you can print its size using the answers to this question: Using memory in a Python dictionary?

Answer 2

range creates a list in memory, which then executes the loop, xrange creates a generator that is an object that passes the loop as a sequence, but does not create this sequence in memory. There are few differences between range and xrange for short ranges, but significant differences for large ranges, citing Python docs: https://docs.python.org/2/library/functions.html#xrange

In Python 3, the functionality provided by xrange becomes standard for the built-in range. As a result of this and the benefits of xrange built-in memory in Python 2, I saw that Python compatibility levels 2 through 3 display the Python 2 range function to call xrange instead under the hood.

Answer 3

I played a little with your script and it continues to grow, albeit slowly. Bottleneck uses random.choice . If you want to quickly fill up the memory, randomness generation will work against you. Thus, using fixed strings really runs out of memory pretty quickly. If you use the following, although you want to see how it grows, you will probably throw away time.sleep() after your print :

 if __name__ == '__main__': d = {} i = 0; for i in range(0, 100000000): d[i] = 'A'*1024 if i % 10000 == 0: print(i) 

memory filling faster:

just single line:

 ['A'*1024 for _ in xrange(0, 1024*1024*1024)]