How to identify places accumulating memory usage in a Perl script?

Question

In my Perl script, it works with high memory speed of busy memory. I tried to make suspicious variables clear immediately when they are no longer needed, but the problem cannot be fixed. Is there any method for monitoring memory changes before and after block execution?

Answer 1

I recently had to fix a problem in one of my programs. Although I do not pretend to be an expert in this matter in any way, I am going to share my findings in the hope that it will benefit someone.

1. High, but stable, memory usage

First, you need to make sure that you have not just a case with high, but stable memory usage. If memory usage is stable, even if your process does not fit in available memory, the discussion below will not help. Here are some notes to read in the Perl documentation here and here , in this SO question , in this discussion of PerlMonks . There is an interesting analysis here if you are familiar with the internal components of Perl. A lot of in-depth information can be found in the presentation of Tim Banz . You should be aware that Perl cannot return memory to the system even if you are undef material . Finally, this is an opinion from the Perl developer that you should not worry too much about memory usage.

2. The ever-increasing use of memory

In the event that memory usage is steadily increasing, this may ultimately lead to a lack of memory situation. My problem turned out to be an example of circular references. According to this answer on StackOverflow, circular links are a common source of memory leaks in Perl. The main reason is that Perl uses a link counting mechanism and cannot free up circular link memory until the program exits . (Note: I could not find the latest version in the Perl documentation on the latest statement.)

You can use Scalar :: Util :: weaken to “weaken” the round chain of links (see also http://perlmaven.com/eliminate-circular-reference-memory-leak-using-weaken ).

3. Further reading

• Tim Banss presentation (slides here ); also in this post.
• http://www.perlmonks.org/?node_id=472366
• Perl Usage and Leak Detection Profiling
• and of course the link given by @mpapec: http://perlmaven.com/how-much-memory-does-the-perl-application-use

4. Tools

• on Unix, you can do system("ps -p $$ -o vsz,rsz,sz,size") Caution: as explained in Tim Banz's presentation, you will need to track VSIZE instead of RSS
• How to find the amount of physical memory occupied by a hash in Perl?
• https://metacpan.org/pod/Devel::Size
• and Tim Bans’s more recent approach, which adds the ability to estimate the interpreter’s total memory: https://metacpan.org/pod/Devel::SizeMe
• in test cases you can use https://metacpan.org/pod/Test::LeakTrace and https://metacpan.org/pod/Test::Memory::Cycle ; example here
• https://metacpan.org/pod/Devel::InterpreterSize