Geek Answers Handbook

What tool should I use to determine memory allocation in Perl?

I climbed into a large file using File :: Slurp, but given the size of the file, I see that it should have it in memory twice, or maybe it was pumped up, turning into a 16-bit unicode. How can I best diagnose such a problem in Perl?

The file I pulled out is 800 MB in size, and my perl process, which parses this data, has approximately 1.6 GB allocated at runtime.

I understand that I can be wrong in my reason, but I'm not sure what the most effective way to prove / disprove my theory.

Update:

I banished cunning character encoding from the list of suspects. It seems like I'm copying a variable at some point, I just can't figure out where.

Update 2:

Now I have done some more research and found that they are actually just getting data from File :: Slurp that is causing the problem. I looked through the documentation and found that I can make it return scalar_ref, i.e.

my $data = read_file($file, binmode => ':raw', scalar_ref => 1);

Then I do not get inflation of my memory. This makes sense and is most logical when you need to make data in my situation.

Information about which variables exist, etc. usually helps, although thanks.

+6
memory-management memory-leaks perl
source share
2 answers

Maybe Devel::DumpSizes and / or Devel::Size can help? I think the first would be more useful in your case.

Devel :: DumpSizes - resets the name and size in bytes (in ascending order) of the variables available at the selection point in the script.

Devel :: Size - Perl extension for finding memory usage of Perl variables

+4
source share

Here are some common resources on memory issues in Perl:

As for your own suggestion, the easiest way to refute would be to write a simple Perl program that:

  • Creates a large (100M) plain text file, perhaps just by outputting one line in a loop to a file or for binary files that execute the dd via the system() call

  • Read the file using the standard Perl open()/@a=<>;

  • Measurement of memory consumption.

Then repeat # 2- # 3 for your 800M file.

This will tell you if the problem is a :: Slurp file, some kind of weird logic in your program, or some specific content in the file (for example, non-ascii, although I would be surprised if that were the reason)

+4
source share

More articles:


All Articles