Prove that using a range operator in a loop does not use extra memory

Question

The current documentation for the range operator .. states that it does not burn memory for counting cycles :

... The range operator is useful for writing foreach (1..10) loops foreach (1..10) and for slice operations on arrays. In the current implementation, a temporary array is not created when the range operator is used as an expression in foreach loops, but older versions of Perl can burn a lot of memory when you write something like this:

  1. for (1 .. 1_000_000) { 2. # code 3. } 

Due to the aforementioned early implementations of for (MIN .. MAX) , I still come to experts who are wary of using counting loops because they think this is equivalent:

 my @temp_array = (MIN .. MAX); # Needlessly using up memory for (@temp_array) { 

Compared to more logical and memory efficient:

 for ($_ = MIN; $_ <= MAX; $_++) { # Logical counting from MIN to MAX 

Questions:

• Is there a way in which one could prove that the count cycle is not wasting memory?

• Does anyone know which versions of Perl had memory issues and when were they fixed?

I can prove to myself that loop counts do not lose memory using the bottom single-line layer, which would certainly crash my system if it actually created a temporary array. Nevertheless, it would be nice if there was more convincing information on this issue, so that we could tell this story of old wives to relax.

 $ perl -e 'for (1 .. 1_000_000_000_000_000) { print "$_\n"; last if $_ == 5 }' 1 2 3 4 5 

---

Decision

Each of the three answers below explains this problem:

• ikegami answer carefully breaks down the various types of cycles and demonstrates how counting cycles differ on the front panel and the back end. & Star;
• friedo answer shows how to use top to monitor memory usage.
• Borodin's answer refers to the second part of my query when this no longer became a problem:
  • perlop v5.4_68 (released June 23, 1998) warns that a temporary array is being used.
  • perlop v5.4_69 (released June 29, 1998) indicates that the temporary array is no longer in use.
  • perldelta v5.4_71 (released July 9, 1998) indicates that the optimization of counting cycles.

At some point, I could do some specific version checks, but given that this is apparently a 16-year-old problem, I am sure that the warning in perlop can be postponed.

Answer 1

First of all, here is a list of the various types of for loops and optimizations that can be applied. All of them are present in each version of Perl from 5.6 to 5.20 (present) inclusive, and I believe that it is comprehensive.

• for (EXPR; EXPR; EXPR)
  → "C-style for loop", extended while loop.
• for (EXPRX..EXPRY)
  → Range and nothing else is optimized in the counting cycle.
• for (@ARRAY)
  → The array and nothing else get directly, but do not flatten.
• for (reverse LIST)
  → None of the above optimizations apply, but the list moves in the reverse order, and not vice versa.
• for (LIST)
  → In a general foreach loop, the LIST expression is evaluated before the start of the loop.

When CONSTX..CONSTY smoothed (i.e., anywhere other than for (CONSTX..CONSTY) ), it is smoothed at compile time, rather than at run time.

---

Black box

Base memory usage:

 $ perl -e'system(ps, ho, rss, 0+$$);' 1540 # 1.5 MiB 

The general case is smoothed out.

 $ perl -e'$y=2_000_000; for ((),1..$y) { system(ps, ho, rss, 0+$$); last }' 80208 # 78 MiB 

Or worse. (It is flattened into an array at compile time in addition to the normal use of the stack.)

 $ perl -e'for ((),1..2_000_000) { system(ps, ho, rss, 0+$$); last }' 143224 # 140 MiB 

for (CONST..CONST) not smoothing.

 $ perl -e'for (1..2_000_000) { system(ps, ho, rss, 0+$$); last }' 1540 # 1.5 MiB 

Indeed, for (EXPR..EXPR) is not smoothed at all.

 $ perl -e'$y=2_000_000; for (1..$y) { system(ps, ho, rss, 0+$$); last }' 1540 # 1.5 MiB 

Even without tools, you can find out the difference in compilation time.

 $ time perl -c -e'1 for 1..2_000_000' -e syntax OK real 0m0.010s user 0m0.004s sys 0m0.000s $ time perl -c -e'1 for (),1..2_000_000' -e syntax OK real 0m1.197s user 0m0.952s sys 0m0.232s 

---

White box

A non-optimized case uses the range operator in the context of l ist. Full list in mind.

 $ perl -MO=Concise,-exec -e'$y=1_000_000; 1 for (),1..$y;' ... 8 <|> range(other->9)[t3] lK/1 <-- Range operator 9 <#> gvsv[*y] s a <1> flop lKM goto b i <$> const[IV 1] s j <1> flip[t4] lK/LINENUM b <#> gv[*_] s c <{> enteriter(next->d last->g redo->d) lK/8 <-- No S ... 

This is what the range flattened out at compile time is as follows:

 $ perl -MO=Concise,-exec -e'1 for (),1..1_000_000;' ... 4 <$> const[AV ] s <-- Constant array 5 <1> rv2av lKPM/1 6 <#> gv[*_] s 7 <{> enteriter(next->8 last->b redo->8) lK/8 <-- No S ... 

You can see that for (CONST..CONST) creates enteriter with the "S" flag. At enteriter this means it is a counting cycle.

 $ perl -MO=Concise,-exec -e'1 for 1..1_000_000;' ... 4 <$> const[IV 1] s 5 <$> const[IV 1000000] s 6 <#> gv[*_] s 7 <{> enteriter(next->8 last->b redo->8) lKS/8 <-- S ... 

The same goes for for (EXPR..EXPR) in general.

 $ perl -MO=Concise,-exec -e'$y=1_000_000; 1 for 1..$y;' ... 8 <$> const[IV 1] s 9 <#> gvsv[*y] s a <#> gv[*_] s b <{> enteriter(next->c last->f redo->c) lKS/8 <-- S ... 

Even for (@a) not flattened!

 $ perl -MO=Concise,-exec -e'1 for @a;' ... 4 <#> gv[*a] s 5 <1> rv2av[t2] sKRM/1 6 <#> gv[*_] s 7 <{> enteriter(next->8 last->b redo->8) lKS/8 <-- S ... 

Double check

 $ perl -MO=Concise,-exec -e'1 for (),@a;' ... 4 <#> gv[*a] s 5 <1> rv2av[t2] lKM/1 6 <#> gv[*_] s 7 <{> enteriter(next->8 last->b redo->8) lK/8 <-- No S ... 

A search for the code for the "S" flag will confirm all this.

• pp_enteriter ( PL_op->op_flags & OPf_STACKED checks for "S")
• pp_iter .

Answer 2

Is there a way that could prove that the counting cycle does not lose memory?

One way would be to control the process with top . For example:

 perl -E 'for( 1 .. 1_000_000_000_000 ) { say $_ }' > /tmp/blah & top -pid $! 

This shows that memory usage remains fairly constant (and trivial) while the program is running, despite repeating a list of one trillion items.

Does anyone know which versions of Perl had memory issues and when were they fixed?

I could not find the exact version when this was changed using quick grep perldeltas. I note that perlop mentions it:

 The range operator is useful for writing "foreach (1..10)" loops and for doing slice operations on arrays. In the current implementation, no temporary array is created when the range operator is used as the expression in "foreach" loops, but older versions of Perl might burn a lot of memory when you write something like this: for (1 .. 1_000_000) { # code } 

But this does not say when this optimization was introduced. In any case, it was a long time ago.

Answer 3

Last time perlop talks about it

Remember that a temporary array is created in the current implementation, so you will write a lot of memory if you write something like this:

is in version 5.4_68 .

On 5.4_69 (released June 29, 1998) it changes to very close to the current version

In the current implementation, a temporary array is not created when the range operator is used as an expression in C loops, but older versions of Perl can burn a lot of memory when you write something like this:

But I cannot find the change mentioned in perldelta anywhere!

Anyway, we're talking about a sixteen-year-old lawsuit,