Memory usage limitations when counting threads using PLINQ

Question

I use PLINQ to split the data into an operation. Each thread consumes a very large amount of memory, which is a variable based on the algorithm settings; there is no easy way to calculate in advance how much memory will be required based on the algorithm options. When memory requirements are low, the optimal number of threads is 7 out of a total of 8; when the memory requirements are high, 2 threads improve significantly compared to 1, but after three threads are used, the system starts hitting the paging file, and the processor load drops to 0, and the performance becomes hundreds of times worse.

What I would like to do is force PLINQ to increase the number of threads step by step, controlling the memory load, and as soon as the system runs out of available physical memory, set the undo marker on the last thread so that it can roll back its memory, allowing other threads to perform optimally.

Any ideas on how to get this behavior from PLINQ, or do I need a full roll?

Answer 1

PLINQ uses TreadPool Themes (default TaskScheduler and TaskFactory ), so you can use ThreadPool.SetMaxThreads(#N, #Q) to control the number of threads based on the current memory consumption of your application.

But it seems to me that the memory consumption of this value - this application, perhaps should be redesigned and transferred to another module (for example, to a database - Lite RDBMS or NoSQL - or to a caching system) - just a suggestion.