Initialize a huge uint8_t array statically with reasonable compilation time

Question

I would like to statically initialize the array (megabytes) of uint8_t.

In the beginning I tried this:

constexpr uint8_t arr[HUGE_SIZE] = { 0, 255, ... };

Unfortunately, the compilation time above is very long (optimization takes about 30 seconds, optimization takes about an hour).

I found out that compilation time can be reduced to insignificant (both during optimization and in case), if we use the initialization of the c-style string:

constexpr uint8_t arr[HUGE_SIZE + 1] = "\x00\xFF\x...";

Is this a good approach in C ++? Should I use any string literal to make the types of both sides above the destination equal?

Answer 1

If the array is really large, consider using a utility to directly create an object file from the array. For example, using the GNU assembler, you can do something like this:

  .section .rodata # or .data, as needed .globl arr arr: .incbin "arr.bin" # assuming arr.bin is a file that contains the data .size arr,.-arr 

Then compile this file using the GNU assembler and attach it to your program. To use this data elsewhere in your program, simply declare it as extern "C" :

 extern "C" const uint8_t arr[]; 

Answer 2

It has been found that the compilation time of large arrays improves slightly if the array is broken into smaller pieces. However, the string approach is still significantly faster. With such a scheme, a true array could union array of arrays.

The publication below is an example of how to test the OP problem without explicitly encoding files with million bytes of bytes. Since this is not a big answer, it’s a research resource that marks this community wiki.

 #include <iostream> using namespace std; #include <cstdint> #define METHOD 5 #if METHOD == 1 // 1 byte blocks 28 secs #define ZZ16 65, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, #define ZZ256 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 #define ZZ4K ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 #define ARR constexpr uint8_t arr[] #define COUT cout << arr << endl #elif METHOD == 2 // 16 byte blocks 16 secs #define ZZ16 {66, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255}, #define ZZ256 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 #define ZZ4K ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 #define ARR constexpr uint8_t arr[][16] #define COUT cout << arr[0] << endl #elif METHOD == 3 // 256 byte blocks 16 secs #define ZZ16 67, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, #define ZZ256 {ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16}, #define ZZ4K ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 #define ARR constexpr uint8_t arr[][256] #define COUT cout << arr[0] << endl #elif METHOD == 4 // 4K byte blocks 13 secs #define ZZ16 68, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, #define ZZ256 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 #define ZZ4K {ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256}, #define ARR constexpr uint8_t arr[][4096] #define COUT cout << arr[0] << endl #elif METHOD == 5 // String 4 sec #define ZZ16 "\x45\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF" #define ZZ256 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 ZZ16 #define ZZ4K ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 #define ARR constexpr uint8_t arr[] #define COUT cout << arr << endl #endif #define ZZ64K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K ZZ4K #define ZZ1M ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K #define ZZ16M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M ZZ1M // 3 million bytes ARR = { ZZ1M ZZ1M ZZ1M }; int main() { cout << "!!!Hello World!!!" << endl; COUT; cout << sizeof(arr) << endl; return 0; } 

Answer 3

Are you going to recompile the file in which the array is defined very often? If not, you can put the array definition in a separate .cpp file with the declaration forward in the .h file. Thus, you will encounter overhead only when changing the array.

Answer 4

Move the array definition to a separate C file and compile it as such. C ++ can reference external global data from C object modules.

If gcc takes too long to compile it, use tcc .