Upstart init is a memory leak, how do you debug it?

Question

I have a memory leak during the initialization process of Upstart (pid 1), what parameters do I have when debugging?

EDIT: Offer me some real tools for this by manually printing printfs or calculating the memory allocation manually, this will not reduce it. Also resetting the initialization kernel and gouging around this is not really an option.

UPD1: valgrind does not work. Replacing / sbin / init on the kernel command line with the correct magic valgrind + init does not seem to be such an option, because it tries to access / proc for self for smaps, but they are not available until init starts.

UPD2: dmalloc also does not work (does not compile on ARM).

Answer 1

The solution for the poor should be to simply log every call to malloc and free , then comb through the logs and look for the pattern.

ld provides an awesome feature that can help here.

--wrap=symbol

Use the wrapper function for the character. Any undefined symbol references will be resolved by __wrap_symbol. Any undefined reference to "__real_symbol" will be allowed to use the character.

This can be used to provide a wrapper for a system function. The wrapper function should be called "__wrap_symbol". If he wishes to call the system function, it should call "__real_symbol".

Here is a trivial example:

 void * __wrap_malloc (size_t c) { printf ("malloc called with %zu\n", c); return __real_malloc (c); } 

If you link other code to this file using --wrap malloc, then all calls to "malloc" will call the __wrap_malloc function. Calling __real_malloc in __wrap_malloc will call the real function malloc.

You might also want to provide the __real_malloc function so that links without the --wrap option succeed. If you do this, you should not put the definition of __real_malloc in the same file as __wrap_malloc; if you do this, assembler can allow the call before the linker; there is the possibility to wrap it in "malloc".

---

Update

Just to see how useful this is.

• Add a custom file to the Upstart assembly.

Like this:

 void*__wrap_malloc( size_t c ) { void *malloced = __real_malloc(c); /* log malloced with its associated backtrace*/ /* something like: <malloced>: <bt-symbol-1>, <bt-symbol-2>, .. */ return malloced } void __wrap_free( void* addr ) { /* log addr with its associated backtrace*/ /* something like: <addr>: <bt-symbol-1>, <bt-symbol-2>, .. */ __real_free(addr); } 

• Compile upstarts with debugging symbols ( -g ) so you can get nice backtracks. You can still optimize ( -O2/-O3 ) code if you want.

• Link Upstart with optional LD_FLAGS --wrap=malloc , --wrap=free .
  Now, anywhere Upstart calls the malloc character will be magically resolved to your new __wrap_malloc character. Beautifully, this is all transparent to the compiled code, as it happens during a connection.
  It is like a shimming or tool without any mess.

• Run the recompiled Upstart as usual until you are sure that a leak has occurred.

• Browse the logs for malloced and addr s mismatch.

A few notes:

• The --wrap=symbol function does not work with function names, which are actually macros. So watch out for #define malloc nih_malloc . This is what libnih you need to use instead of --wrap=nih_malloc and __wrap_nih_malloc .
• Use gcc's built-in backtracing features .
• All these changes only affect the recompiled Upstart executable.
• Instead, you can flush logs to the sqlite database to make it easier to find inconsistent mallocs and frees.
• you can make it so that you enter the format of the SQL insert format, and then simply insert them into the post-mortem database for further analysis.

Answer 2

You can configure the memory allocation by connecting malloc / free calls and counting the number of bytes allocated by you and freeing each time.

Answer 3

You can also use init without change, but create a wrapper that sets the MALLOC_CHECK environment variable to 1 or higher . This will allow you to see some diagnostics for memory allocation.

The change is to slightly modify the initialization code to set this environment variable before it starts using malloc.

You can also suggest AmineK to add debug code to the init source code.

Answer 4

You can try linking your upstart version to Google TCMalloc . It comes with a built-in bunch of checks .

The heap controller can be enabled in two ways:

• set the HEAPCHECK environment HEAPCHECK to one of {normal | strict | draconian}.
• set HEAPCHECK to local and check the code manually using HeapProfileLeakChecker objects.

I do not know how to set the environment variable for init, however.

Answer 5

How about starting pmap in the process and learning which memory segments are growing. This may give you some idea of ​​what memory is. A little scripting can make this process almost automatic **.

** In a past life, I actually wrote a script that will take n pmap snapshots of a set of running processes, separated by a few seconds. The result of this was served in a perl script that identified the segments that resized them. I used it to detect multiple memory leaks in some commercial code. [I would share the scenarios, but they are covered by the IP (copyright) of the previous employer.]

• John