Is there a reason to use malloc over PyMem_Malloc?

Question

I am reading documentation for memory management in Python C extensions , and as far as I can tell, it doesn't really seem to be reasonable to use malloc , not PyMem_Malloc . Suppose I want to allocate an array that should not be exposed to the Python source code, and will be stored in an object that will be garbage collected. Is there a reason to use malloc ?

Answer 1

EDIT : mixed corrections PyMem_Malloc and PyObject_Malloc ; these are two different challenges.

Without the PYMALLOC_DEBUG macro PYMALLOC_DEBUG PyMem_Malloc is an alias of libc malloc() , which has one special case: a call to PyMem_Malloc to select null bytes returns a non-NULL pointer, whereas malloc (zero_bytes) may return NULL or raise a system error ( link to source code ):

/ * malloc. Note that nbytes == 0 tries to return a non-NULL * pointer from all other currently existing pointers. This may not be possible. * /

In addition, there is an advisory note in the pymem.h header file :

Never mix calls with PyMem_ with calls to the malloc / realloc / calloc / platform for free. For example, on Windows, different DLLs can have different heaps, and if you use PyMem_Malloc you will get memory from the heap used by the Python DLL; this can be a disaster if you are free (), which is right in your own extension. Instead, use PyMem_Free to ensure that Python can return memory to the correct heap. Like another example, in PYMALLOC_DEBUG mode, Python wraps all calls to all PyMem_ and PyObject_ memory functions in special debug shells that add additional information for debugging dynamic memory blocks. The system routines do not have a clue what to do with this material, and Python wrappers do not have a clue what to do with the raw blocks received directly then the system will be executed.

Then there are some Python settings inside PyMem_Malloc PyObject_Malloc , a function used not only for C-extensions, but for all dynamic distributions when starting a Python program, for example 100*234 , str(100) or 10 + 4j :

 >>> id(10 + 4j) 139721697591440 >>> id(10 + 4j) 139721697591504 >>> id(10 + 4j) 139721697591440 

Previous instances of complex() are small objects allocated in a dedicated pool.

Distributing small objects (<256 bytes) using PyMem_Malloc PyObject_Malloc quite efficient because it is made from a pool of 8 bytes with alignment, there is one pool for each block size. There are also Pages and Arenas blocks for large distributions.

This source comment explains how the PyObject_Malloc call is PyObject_Malloc :

 /* * The basic blocks are ordered by decreasing execution frequency, * which minimizes the number of jumps in the most common cases, * improves branching prediction and instruction scheduling (small * block allocations typically result in a couple of instructions). * Unless the optimizer reorders everything, being too smart... */ 

Pools, pages, and arenas are optimizations designed to reduce fragmentation of the external memory of long Python programs.

Check out the source code for full detailed documentation on Python's internal internals.

Answer 2

This is normal for extensions to allocate memory using malloc or other system allocators. This is normal and inevitable for many types of modules — most modules that wrap other libraries that themselves don’t know anything about Python will invoke their own distributions when they happen in this library. (Some libraries allow you to control distribution to prevent this, and most do not.)

There is a serious drawback to using PyMem_Malloc: you need to hold the GIL while using it. Native libraries often want to issue a GIL when doing computationally intensive use of the processor or make any calls that might block, such as I / O. The need to block the GIL before distribution can be somewhere between a very inconvenient and a performance issue.

Using Python shells to allocate memory allows you to use Python memory debugging code. However, with tools like Valgrind, I doubt the real value of this.

You will need to use these functions if it requires an API; for example, if the API passed a pointer that should be highlighted by these functions, so it can be freed with them. If you prohibit an explicit reason for use, I adhere to a normal distribution.

Answer 3

From my experience writing MATLAB.mex functions, I believe that portability is the most important determining factor in whether you use malloc or not. Let's say you have a header file that loads useful functions only using internal data types c (there is no need to interact with the Python object, so there is no problem using malloc), and you suddenly realize that you want to transfer this header file to another a code base that has nothing to do with Python (maybe this is a project written exclusively in C), using malloc will obviously be a much more portable solution.

But for your code, which is purely an extension of Python, my initial reaction was to expect the c function of the native language to execute faster. I have no evidence to support this :)