Geek Answers Handbook

Performance issue with a very large number of lists created in a loop

This is my first post here ... Please think I'm a hobby programmer.
Problem:
My goal is to improve the performance of my code. The program uses only 65% ​​of the processor and 500 MB of memory. There is another 800 MB of free physical memory available for the program, and about 30% of the processor is idling. I don’t know where there can be a bottleneck for further use of resources and increase of code performance.
Background:
I wrote a program that is a massive testing of a financial algorithm. This algorithm has a number of parameters, and I'm trying to find the best combination of parameters. To do this, I run it for all possible combination of parameters.
This algorithm has an input data series. It iterates through a series of data and gives a result.
Therefore, for this, I introduced the algorithm code into the Parallel.Foreach loop, which runs once for each combination of parameters. The critical code for optimization is a loop.
Since the code is very long, I post the framework:

...
class candle : IComparable<candle>  //Data element to represent a candle and all chart data
    {
        public double open;
        public double high;
        ...40 more
    }
class param : IComparable<param>
{
    public int par1;
    public int par2;
    public int par3;
    ... a few more    
}
//Running program
{
List<candle> sourceCandleList = new List<candle>();
List<param> paramList = new List<param>(1000000);

// Code to populate sourceCandleList and paramList in here
}
// EDIT: Start parallel processing
Parallel.ForEach(paramList,
    p =>
    {
        List<candle> CandleList = new List<candle>(sourceCandleList.Count);
        foreach (var cndl in sourceCandleList)
           {
            candle c = new candle();
            c.open = cndl.open;
            c.high = cndl.high;
            ...
            //Run calculations and populate fields in CandleList
            }
         //Evaluate results
     }

140 000 . sourceCandleList 2.000 . , 2.000 , . , GC 200 . 1 Parallel.ForEach 80 . .
, GC , CandleList , . , , . , , , TryEnter, ?

EDIT
I did not clearly say that the program first generates 2 lists: sourceCandleList and paramList. Then they no longer change as soon as they are completed. After these 2 lists are completely filled, the program starts checking the algorithm: sourceCandleList is the input, and one record from paramList is the set of applied parameters. The problem is that the program defines a new CandleList each time. so i need something like:
ConcurrentQueue<List<candle>> ListOfCanldes = new ConcurrentQueue<List<candle>>();

However, I cannot understand the correct syntax.

Questions:
Why is the processor memory not fully used? Am I reaching maximum read / write speed of memory ??
How can I avoid GC program slowdowns?
How can I find out how much time I am losing due to the GC?
How to increase productivity?
+4
2

, . , , , . .
50% , . 100%.

static void Main(string[] args)
    {
        // Parameters to test
        List<candle> paramList = new List<candle>();

        // Original list
        List<candle> sourceList = new List<candle>();

        // Create a copy for each thread
        int maxThreads = 16;
        ConcurrentBag<int> pool = new ConcurrentBag<int>();
        List<candle>[] processList = new List<candle>[maxThreads];
        for (int i = 0; i <= maxThreads - 1; i++)
        {
            pool.Add(i);
            processList[i] = sourceList.ConvertAll(p => p);
        }
        Parallel.For(0, paramList.Count,
        new ParallelOptions { MaxDegreeOfParallelism = maxThreads },
        p =>
        {
            int slot = 0;
            int item;
            for (int i = 0; i <= 2; i++)
            {
                if (pool.TryTake(out item))
                {
                    slot = item;
                    break;
                }
                else
                {
                    i = 0;
                }
            }
            lock (processList[slot])
            {
                // Do processing here
            }
            pool.Add(slot);
        });
    }
0

? / ??

  • . Parallel.ForEach , . , , - ConcurrentQueue ConcurrentStack.

- , :

class Program
    {
        static ConcurrentQueue<candle> sourceCandleList = new ConcurrentQueue<candle>();
        static ConcurrentBag<param> paramList = new ConcurrentBag<param>();
        static void Main(string[] args)
        {
            var threads = new List<Thread>();
            var numberOfThreads = 10;
            for (int i = 0; i < numberOfThreads; i++)
            {
                threads.Add(new Thread(Run));
            }
            threads.ForEach(i => i.Start());
        }
        static void Run()
        {
            candle item;
            while (sourceCandleList.TryDequeue(out item))
            {
                //do you processing here
            }
        }
    }

GC ?: GC.

, - GC? VS 2015 - , , Ant Profiler. GC.

? . . , .

+2

More articles:


All Articles