Geek Answers Handbook

Need advice on memory usage in perl

I am not new to perl, but I have not been able to solve this problem.

Here is my problem:

I am using Activestate perl 5.12 32-bit. I want a 32 x 8MB array of double numbers to be stored in memory at the same time. unfortunately, in this case I get an error "from memory". for example, the code below will run out of memory.

my $aref1 = [(.25) x (8*1024*1024)]; my $aref2 = [(.25) x (8*1024*1024)]; my $aref3 = [(.25) x (8*1024*1024)]; my $aref4 = [(.25) x (8*1024*1024)]; my $aref5 = [(.25) x (8*1024*1024)]; my $aref6 = [(.25) x (8*1024*1024)]; my $aref7 = [(.25) x (8*1024*1024)]; my $aref8 = [(.25) x (8*1024*1024)]; my $aref9 = [(.25) x (8*1024*1024)]; my $aref10 = [(.25) x (8*1024*1024)];

Is there a way to deal with this effectively?

Note: In any case, access to arrays is required at any time as quickly as possible (since the user cannot wait long)

The options I tried:

  • DBM :: Deep - takes more time
  • PDL - "out of memory" error for 32 x 8MB double array

Waiting for your valuable suggestions!

+4
source share
3 answers

32 arrays * 8 Mi-doubles / array * 8 bytes / double = 2 GiB.

32-bit processes on Windows have only 2GiB of usable address space. Your source data occupies the entire available address space, leaving nothing for the overhead of the data structure, perl , your program, and other variables. The only way it will fit into memory is to switch to 64-bit Perl. Otherwise, you will have to deal with the inevitable slower solution.

+11
source

This will create a perl value for each element, which will be quite heavy. You might want to look at what stores values ​​as doubles, such as Tie :: CArray or Tie :: Array :: PackedC .

+3
source

This is what I get on my Windows XP SP3 system using ActiveState Perl 5.14.2. Task Manager showed Commit charge: 778M/3958M when I ran the following script:

 #!/usr/bin/env perl use strict; use warnings; use Devel::Size qw(total_size); my $unit = 1024 * 1024; my $topj = (8 * $unit) - 1; my @data; for my $i (0 .. 31) { print "$i: "; my @row; $#row = $topj; for my $j (0 .. $topj) { $row[$j] = 0.25; } push @data, \@row; printf "%.0f\n", total_size(\@data)/$unit; }

Output:

  C: \ temp> yy
 0: 224
 1: 448
 2: 672
 3: 896
 4: 1120
 5: 1344
 6: 1568
 7: 1792
 Out of memory!
 eight:

On the other hand, the following C program improves:

 #include <stdlib.h> #include <stdio.h> #define ROWSIZE 8*1024*1024 int main(void) { int n = 1; while (calloc(ROWSIZE, sizeof(double))) { printf("%d: success!\n", n); n += 1; } return 0; }

Output:

  1: success!
 2: success!
 3: success!
 ...
 26: success!
 27: success!

for the same mark of 1.7 GB.

+2
source

More articles:


All Articles