STL container memory issue

Question

I implement my own graph library on linux (Fedora 10 and CentOS 5) with gcc 4.3.2 and using STL containers, then I found some memory problems. When I create my schedule, I use a lot of memory for viewing in the top or another tool for using memory. I am sure that I free this memory (I looked through the code again and again and I used valgrind to check for memory leak), but the memory remains in use (I can view it at the top or cat / proc / meminfo) and when I create graph again, this does not increase memory usage, obviously reuses allocated memory.

After several days of debugging, I created a very simple code with the same problem.

#include <iostream> #include <list> // Object that occupies 128KB. // Data is not important. class MyObject { public: int * a; int * b; int * c; int * d; MyObject( ) { a = new int[ 8192 ]; b = new int[ 8192 ]; c = new int[ 8192 ]; d = new int[ 8192 ]; } MyObject( const MyObject & m ) { a = new int[ 8192 ]; b = new int[ 8192 ]; c = new int[ 8192 ]; d = new int[ 8192 ]; } ~MyObject( ) { delete [] a; delete [] b; delete [] c; delete [] d; } void operator=( const MyObject &m ) { //Do nothing. } }; typedef std::list< MyObject > list_t; #define MB_TO_ALLOC 1000 // Size in MB that the program must alloc. #define SLEEP_TIME 5 // Time in seconds that the program must wait until go to another step. // It used to give sufficient time for tools update the memory usage int main( ) { std::cout << "Alloc..." << std::endl; list_t * list = new list_t( ); // Number of objects for alloc MB_TO_ALLOC amount of memory int nObjects = MB_TO_ALLOC * 1024 / 128; for( int i = 0; i < nObjects; ++i ) list->push_back( MyObject( ) ); std::cout << SLEEP_TIME << "s to Dealloc..." << std::endl; // Wait some time for a tool (like top) to update the memory usage sleep( SLEEP_TIME ); std::cout << "Dealloc..." << std::endl; delete list; std::cout << SLEEP_TIME << "s to Alloc..." << std::endl; // Wait some time for a tool (like top) to update the memory usage sleep( SLEEP_TIME ); //Repeats the procedure for evaluating the reuse of memory std::cout << "Alloc..." << std::endl; list = new list_t( ); for( int i = 0; i < nObjects; ++i ) list->push_back( MyObject( ) ); std::cout << SLEEP_TIME << "s to Dealloc..." << std::endl; sleep( SLEEP_TIME ); delete list; } 

I tried using a simple array or my own list class, but in these cases the memory is usually freed.

Does anyone know what is going on? How to prevent "backup" of this memory?

Thanks!

- Bruno Caponi

Answer 1

gcc STL has its own level of memory management, which captures large chunks of memory and does not return them; there is an env variable that you can configure to use unprocessed new calls

 GLIBCPP_FORCE_NEW=1 

I guess that frees him. This env var is usually used when using valgrind, so valgrind does not think that something has leaked.

Answer 2

In addition to the STL container, it may be libc itself (new / delete - malloc / free). User space libraries can save memory for later reuse. Allocation / deallocation is an expensive operation (in terms of clock cycles), so many implementations try to avoid this.

Answer 3

The memory usage for the container class is handled by the allocator container (which is passed as an argument to the std::list constructor, and the default is std::allocator ). The default allocator implementation may not immediately return memory to the system to prevent excessive heap fragmentation.

If you need more direct control over this, you will probably have to implement a custom allocator.

Answer 4

In Linux user memory, a process from the kernel is allocated using the BRK script, which extends the data pointer down to make the process available. This is the only way for the kernel to transfer normal memory to the process. It is also possible to obtain memory using mmap, which allows the process to specify the starting address (except for the data pointer), but no allocators do this, since it significantly increases the amount of work both the allocator and the kernel allocator do. For this reason, the memory provided to the user process can be easily restored by the kernel until the process terminates. If your particular application creates many large allocs / deallocs, then using mmap for these allocs may be the solution to reduce the size of the image in memory.

Answer 5

I cannot say exactly what is happening, but the problem (if it is a problem) is reproducible.

I thought it was a memory pool problem, wandering that the STL was doing this in the particular allocator used. However, the drilldown list <>, I found only "new_allocator", which returns nothing more, returning the result of the global new operator:

 pointer allocate(size_type __n, const void* = 0) { return static_cast<_Tp*>(::operator new(__n * sizeof(_Tp))); } 

The behavior, as I understand it, then goes to glibc or stdlibC ++. In my quick look, I could not figure out how to get around this behavior without implementing a custom allocator, and if the custom allocator will certainly behave differently.

I tried another test without STL, and then I could see how resources grow. I would suggest creating your own distributor, which allocates an arbitrary number of elements in the array with the placement of a new one and takes care of the distribution / deallocation of these arrays. Logic tells me that resource use should behave the same as the "non STL" test.

Please try and tell what will happen. I won’t do it myself, because I don’t have time to get rid of it now, despite my curiosity;)

Note: The Big Three rule is not affected here. As far as I understand, there is no memory leak, and the contents of the object does not matter. Bruno could make a copy of the data, homing, etc., but just created an empty constructor to illustrate his point.

Answer 6

I ran into the same problem and after a long debugging I wrote an example program that illustrates that this is a problem with the kernel (or maybe g ++).

 #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <ctime> static const size_t CHUNKS_COUNT = 1024 * 1024; void* chunks[CHUNKS_COUNT]; int main(int argc, const char *argv[]) { bool additionalAlloc = false; if (argc > 1) additionalAlloc = true; fprintf(stdout, "%lu: starting allocating chunks, additionalAlloc=%d\n", time(NULL), additionalAlloc); for (size_t n = 0; n < 2; ++n) { void* additionalChunk; // 1GB for (size_t c = 0; c < CHUNKS_COUNT; ++c) { static const size_t BASE_CHUNK_SIZE = 1024; chunks[c] = malloc(BASE_CHUNK_SIZE); } if (additionalAlloc) { // 33 is arbitrary, but for instance for 123 given example // is not working - magic :-) additionalChunk = malloc(33); } fprintf(stdout, "%lu: finished allocating chunks, n=%lu\n", time(NULL), n); sleep(60); for (size_t c = 0; c < CHUNKS_COUNT; ++c) { free(chunks[c]); } if (additionalAlloc) free(additionalChunk); fprintf(stdout, "%lu: finished freeing chunks, n=%lu\n", time(NULL), n); sleep(60); } sleep(60); fprintf(stdout, "%lu: finishing program\n", time(NULL)); return 0; } 

When it starts without a parameter (optional Alloc is false), memory is freed into the system after a free call. But when it starts with the parameter (optional Alloc is true), memory is freed only after the program terminates. I run it on Debian Squeeze with 4.4.5-1 g ++ on the kernel 2.6.18-6-xen-amd64. I don’t know how this works on other systems, but seeing that 123 bytes for an additional fragment leads to different program behavior, there is a high probability that it will not work, but believe me, it worked for these settings :-)

PS. Can someone explain why the 33 and 123 values ​​for the extra fragment lead to different behavior?