Maximum TCL Array Size

I am working on an engineering application, and the interface is written in TCL TK.

Everything went fine until I needed a (extremely) large array. 370.000.000 elements, each element from 2 to 10 characters long (linear growth).

My question is: ¿where is the size limit for TCL arrays? I read and investigated, and just found, this is “2 GB” of string data, but I don’t know if it is reliable because it does not explain the reason.

I did an experiment:

set lista [list ] catch { for {set i 0} {$i < 370000000} {incr i} { lappend lista $i } } puts $i 

returns $ i = 50,000,000 more or less by 32 bits of Windows 7

+5
source share
1 answer

This is a little hard to explain. The 2 GB limit comes from a low-level memory allocator, which has a size limit since it uses a signed 32-bit integer to describe how much memory is allocated. It was good on 32-bit systems, but it was an open error (which can be assigned to me) that it is still relevant for 64-bit systems; in fact, the correct type in API C is actually ssize_t (yes, still signed, negative values ​​are used for signaling), but fixing it completely destroys many APIs, so a significant version change is required to change it.

But the maximum size of the list is something else. This is fundamentally connected with a combination of several things. Firstly, there is a maximum size of the memory structure that can be allocated (2 GB limitation), which means that you probably cannot reliably get more than 256 M items in a list on a 64-bit system. Then the total number of selected elements, although this is not a problem in practice, especially if you actually put the elements in the list several times (as the links are shared). Finally, there is the size of the string representation of the list: if you generate so much, you are still mistaken, but it will be a real limiting factor in your example if you create it (since it will reach the 2 GB limit faster).

The actual point at which you end up with a memory limit may be lower, depending on when your system begins to refuse memory allocation requests. All this concerns the OS, which, as a rule, bases its decision on what else is happening in the system, so it is incredibly difficult to give any general rule. My (64-bit, OSX) system took a long time, but you managed to run the sample code:

 $ tclsh8.6 % eval { set lista [list ] catch { for {set i 0} {$i < 370000000} {incr i} { lappend lista $i } } puts $i } 370000000 % llength $lista 370000000 % unset lista % exit 

llength was the only really fast operation (since it could pull the length from the list metadata). unset took age. exit was pretty fast, but took a few seconds.

+9
source

All Articles