Python disables reference counting for some objects

Question

This question is derived from here .

I have three large lists containing python objects ( l1 , l2 and l3 ). These lists are created when the program starts, and they occupy a total of 16 GB of RAM. The program will be used exclusively for Linux.

I do not need to modify these lists or the objects in these lists in any way or form after they are created. They must remain in memory until the program exits.

I use os.fork () and the multiprocessing module in my program to create many subprocesses (up to 20 at present). Each of these subprocesses should be able to read three lists ( l1 , l2 and l3 ).

My program is working fine and pretty fast. However, I have problems with memory consumption. I was hoping that each subprocess could use three lists without copying them into memory due to the copy-on-write approach in Linux. However, this is not so, since a link to an object in any of these lists will lead to an increase in related links and, therefore, will lead to a copy of the entire memory page.

So my question is:

Is it possible to disable the counting of links to l1 , l2 and l3 and all objects in these lists? Basically, for the whole object (including metadata, such as ref count), it is read-only, so it will never be modified under any circumstances (this, I believe, will allow me to use copy-to-write).

Currently, I'm afraid that I need to switch to another programming language in order to complete this task due to a “function” (reference counting) that I do not need at the moment, but which is still being imposed on me and causes unnecessary problems.

Answer 1

You cannot, reference counting is fundamental to CPython (the reference implementation and the one you use). Using methods for objects causes a change in the number of links, subscribing to an object or accessing an attribute causes objects to add and remove from the stack, which uses reference counting, etc. You cannot get around this.

And if the contents of the lists do not change, use tuple() . This will not change the fact that they will be recounted, though.

Other Python implementations (Jython (using the Java virtual machine), IronPython (.NET runtime), or PyPy (Python implemented in Python but experimenting with JIT and other compiler methods) may use different control memory methods and may or may not solve your memory problem.