How to make make / ninja limit parallelism based on memory pressure?

If I use make -j2 , it builds fine, but underuses the CPU:

j2

If I use make -j4 , it builds quickly, but for some specific files with a large number of templates, it consumes a lot of memory, slowing down the entire system and the build process due to replacing to the hard drive:

j4

How can I automatically limit the number of parallel tasks based on memory, for example:

enter image description here

so he builds the project at maximum speed, but slows down in some places to avoid getting on the memory wall?

Ideas:

  • Use -l and artificially adjust the average load if the memory is busy (average load grows naturally when the system is already in trouble).
  • Creates memory allocation system calls (for example, sbrk (2) or mmap (2)) or page errors that continue to hang the process until memory is restored by completed jobs instead of replacing other processes. Sorry, Deadlock-prone ...
+7
parallel-processing memory build makefile
source share

No one has answered this question yet.

See related questions:

772
How to find out the memory usage of my application in Android?
672
How to measure the actual memory usage of an application or process?
505
How to determine CPU and memory consumption from inside the process?
449
How can I run multiple npm scripts in parallel?
4
Is there a parallel development system that is smart enough to intelligently respond to low memory / replacement conditions?
2
Troubleshoot Java Resident Size Unlimited Size (RSS)
2
Can multiple threads on the same primary system improve performance?
one
Alternative to niceload command in ubuntu?
one
munmap () when processes share a file descriptor table but not virtual memory
0
How to prevent my parallel code using all available system memory?

All Articles