C ++ application memory resiliency

Question

I want to check the memory stability of a C ++ application that I wrote and compiled for Linux. This is a network application that responds to remote client connections at a rate of 10-20 connections per second. In the long run, memory increased to 50 MB, while the application made calls to delete ...

Research shows that Linux does not immediately free memory. So here are my questions:

How to get Linux to free the memory that I really freed? At least I want to do this once to check memory stability. Otherwise, is there any reliable memory indicator that can report the memory that my application is actually holding?

Answer 1

What you see is most likely not a memory leak. Currently, operating systems and malloc / new heaps make very complex memory accounting. This is, in general, a very good thing. Most likely, any attempt on your part to force the OS to free memory will only damage both the performance of your application and the overall system performance.

To illustrate:

• The Heap reserves several areas of virtual memory for use. None of this holds (supported by physical memory) until malloc'd.

• You allocate memory. The pile grows accordingly. You see this in the task manager.

• You allocate more memory to the heap. He is growing more.

• You free the memory allocated in step 2. However, the heap cannot be compressed, since the memory in # 3 is still allocated, and Heaps cannot compress the memory (this will invalidate your pointers).

• You malloc / new stuff. This can be tied after the memory allocated in step # 3, because it cannot fit into the area open on the left with free number # 2, or because the heap manager is inefficient for processing the heap for the remaining block using # 2. (depends on heap and block size implementations for allocated memory / free 'd)

So, the memory at step number 2 is now dead to the world? Not necessary. Firstly, it will probably be reused again once it becomes effective. In cases where it is not reused, the operating system itself can use the functions of the CPU virtual memory (TLB) to "reassign" unused memory directly from within your application and assign it to another application - to fly. The heap knows about this and usually manages things to help improve the OS's ability to reassign pages.

These are valuable memory management techniques that have an unrivaled side effect for rendering fine-grained memory leak detection through Process Explorer is basically useless. If you want to detect small memory leaks on the heap, then you will need tools to detect leaks at runtime. Since you mentioned that you can also use Windows, I want to note that Microsoft CRT has built-in leak checking tools. Instructions for use found here:

http://msdn.microsoft.com/en-us/library/974tc9t1(v=vs.100).aspx

There are also open source versions for malloc available for use with the GCC / Clang tools, although I have no direct experience with them. I think Linux Valgrind is the preferred and reliable leak detection method anyway. (and in my experience it’s easier to use than MSVCRT Debug).

Answer 2

I would suggest using valgrind with the memcheck tool or any other tool for profiling memory leaks

on the Valgrind page:

Memcheck

detects memory management problems and is primarily focused on C and C ++. When the program starts under Memcheck's control, all reads and writes of the memory are checked and calls malloc / new / free / delete are intercepted. As a result, Memcheck may if your program:

• Access to memory should not (areas not yet allocated, freed areas, areas located beyond the end of heap blocks, inaccessible areas of the stack).
• Using uninitialized values ​​in a dangerous way.
• Memory leak.
• Poor heap blocks (double release, mismatch frees).
• Skips memcpy () overlapping source and destination blocks and related functions.

Memcheck reports these errors as soon as they appear, giving the source the line number on which it occurred, as well as the stack trace of the functions called to reach this line. Memcheck tracks byte addressing and bit level value initialization. As a result, it can detect the use of single uninitialized bits and does not report false errors in bitstream operations. Memcheck programs run about 10-30x slower than usual. Similar to Cachegrind

an array

Massif is a heap profiler. It performs detailed heap profiling by taking regular snapshots of the software heap. He creates a graph showing heap usage over time, including information on which parts of the program are responsible for most memory allocations. the graph is complemented by a text or HTML file, which includes more information to determine where most of the memory is allocated. Massif runs programs about 20 times slower than usual.

Using valgrind is as simple as a working application with the right keys and gives it as input to valgrind:

valgrind --tool=memcheck ./myapplication -f foo -b bar 

Answer 3

I highly doubt that anything other than wrapping malloc and free [or new and delete ] with another function can actually get you something other than very rude ratings.

One problem is that freed memory can only be released if there is a long contiguous piece of memory. It usually happens that there are “small bits” of memory that are used throughout the heap, and you cannot find a large chunk that can be freed.

It is very unlikely that you can fix this in any simple way.

And by the way, your application will probably need these 50 MB later, when you will again have more load, so it’s just wasted to free it.

(If the memory you are not using is needed for something else, it will be replaced, and pages that have not been touched for a long time are the first candidates, so if the system runs on low memory for some other tasks, it’s all it will also reuse RAM in your machine for this space, so it doesn’t sit there for nothing - you just can’t use "ps" or some of them to find out how many times your program uses it!)

As suggested in the comment: you can also write your own memory allocator using mmap() to create a “chunk” for extracting parts from. If you have a section of code that does many memory allocations, then ALL of them will surely be freed later to allocate everything from a separate chunk of memory, and when all this is freed, you can put mmap 'd back in the list " free mmap ", and when the list is large enough, free some of the mmap distributions [this is an attempt to avoid calling mmap LOTS times and then munmap again after a few milliseconds]. However, if you ALWAYS allowed one of these memory allocations to “exit” your fenced area, your application will likely crash (or, even worse, not crash, but use memory belonging to some other part of the application, and you get a very strange result somewhere, for example, one user gets access to network content intended for another user!)

Answer 4

Use valgrind to look for memory leaks: valgrind./your_application

It will indicate where you allocated the memory and did not free it.

I do not think this is a Linux problem, but in your application. If you track memory usage with "top", you won’t get very accurate ways to use it. Try using an array (valgrind tool): valgrind --tool = massif./your_application to learn about using real memory.

As a more general rule, to avoid leaks in C ++: use smart pointers instead of regular pointers. Also, in many situations, you can use RAII (http://en.wikipedia.org/wiki/Resource_Acquisition_Is_Initialization) instead of allocating memory "new."

Answer 5

For the OS, it is not recommended to release memory when calling free or delete . This memory is returned to the heap manager in the runtime library.

If you want to free memory, you can use brk . But this opens up a very large capacity of worms for memory management. If you directly call brk , you better not call malloc . For C ++, you can override new to use brk directly.

Not an easy task.

Answer 6

In the latter, dlmalloc () has a concept called mspace (others call it scope). You can call malloc () and free () in mspace. Or you can delete mspace to free all memory allocated from mspace at the same time. Removing mspace will free up memory from the process.

If you create mspace with a connection, allocate all the memory to connect from this mspace and delete mspace when the connection closes, you will not have any process.

If you have a pointer in one mspace pointing to memory in another mspace, and you delete the second mspace, then, as the lawyers say, the results are undefined.