How to debug a memory issue in Rust?

Question

I hope this question is not too open. I ran into a memory problem with Rust, where I got "out of memory" from a call to the next object of the Iterator object . I'm not sure how to debug it. The printouts only led me to where the failure occurred. I am not very familiar with other tools such as ltrace , so although I could create a trace (231MiB, pff), I didn really know what to do with it. Is the footprint so useful? I wish I had better capture gdb / lldb? Or Walgrind?

Answer 1

In general, I would try to do the following approach:

• Boiler room reduction:. Try narrowing your OOM problem so you don't have too much extra code. In other words: the faster your program crashes, the better. Sometimes it’s also possible to snatch a specific piece of code and put it in additional binary code, just for investigation.

• Reducing the size of the problem: Lower the problem with OOM to a simple “too much memory”, so you can say that some part takes up something, but that it does not lead to OOM, If it is too difficult to say that you see the problem or not , you can lower the memory limit. On Linux, this can be done using ulimit :

   ulimit -Sv 500000 # that 500MB ./path/to/exe --foo 

• Gathering information: If the problem is small enough, you are ready to collect information with a lower noise level. There are several ways you can try. Remember to compile your program with debugging symbols. It may also be an advantage to turn off optimization, since this usually leads to loss of information. Both can be archived NOT using the --release flag at compile time.

  • Heap profiling: one way too gperftools is used:

     LD_PRELOAD="/usr/lib/libtcmalloc.so" HEAPPROFILE=/tmp/profile ./path/to/exe --foo pprof --gv ./path/to/exe /tmp/profile/profile.0100.heap 

    This shows a graph that symbolizes what parts of your program are, how much memory. See official docs for more details.

  • rr: Sometimes it’s very difficult to understand what is really happening, especially after creating a profile. Assuming you did a good job in step 2, you can use rr :

     rr record ./path/to/exe --foo rr replay 

    This spawns GDB with super powers. The difference with a regular debugging session is that you can not only continue , but also reverse-continue . Basically, your program runs from a recording where you can jump back and forth as you wish. This wiki page contains some additional examples. It's one thing to note that rr only works with GDB.

  • Good old debugging: sometimes you get tracks and records that are still too large. In this case, you can (in combination with the ulimit trick) just use GDB and wait for the program to crash:

     gdb --args ./path/to/exe --foo 

    Now you should get a normal debugging session where you can check what the current state of the program is. GDB can also be run using cores. A common problem with this approach is that you cannot go back in time and you cannot continue execution. This way you only see the current state, including all stack frames and variables. You can also use LLDB here if you want.

• (Potential) fix + repeat: After you have the glue, what could go wrong, you can try changing your code. Then try again. If it still does not work, go back to step 3 and try again.

Answer 2

In general, for debugging you can either use a logging approach (either by inserting the logs themselves, or using the ltrace , ptrace , ... tool to create logs for you) or you can use the debugger.

Note that ltrace , ptrace or debugger approaches require you to reproduce the problem; I prefer manual logs because I work in an industry where error reports are generally too inaccurate to allow immediate reproduction (and therefore we use logs to create a regenerator script).

Rust supports both approaches, and the standard toolkit used for C or C ++ programs works well for it.

My personal approach is to have some protocols to quickly narrow down where the problem arose, and if logging is not enough to run the debugger for finer checking. In this case, I would advise you to immediately work out the debugger.

A panic , which means that breaking the call into a panicky hook, you will see both the call stack and the memory status at the moment when the situation goes awry.

Run the program using the debugger, set a breakpoint on the panic hook, run the program, make a profit.

Answer 3

Valgrind and other tools work fine.

Here is a program that leaks from memory by putting a 1MiB String in Vec and never freeing it:

 use std::{thread, time::Duration}; fn main() { let mut held_forever = Vec::new(); loop { held_forever.push("x".repeat(1024 * 1024)); println!("Allocated another"); thread::sleep(Duration::from_secs(3)); } } 

Using the developer tool of MacOS Instruments, you can easily see the memory growth over time, as well as the exact trace of the stack that allocated the memory:

And here is an example of a memory leak by creating an infinite reference loop:

 use std::{cell::RefCell, rc::Rc}; struct Leaked { data: String, me: RefCell<Option<Rc<Leaked>>>, } fn main() { let data = "x".repeat(5 * 1024 * 1024); let leaked = Rc::new(Leaked { data, me: RefCell::new(None), }); let me = leaked.clone(); *leaked.me.borrow_mut() = Some(me); } 

See also:

• Why doesn't Valgrind detect a memory leak in Rust using nightly 1.29.0?
• Memory leak handling in cyclic graphs using RefCell and Rc
• Minimum Rc Dependence Cycle