Geek Answers Handbook

Performance problem: CPU intensive works better with more concurrency in Erlang

TL, Dr.
I improve the performance of my erlang program when I perform my intensive tasks with a higher level concurrency processor (e.g. 10K versus 4 at the same time). Why?

I am writing a wireframe reduction scheme using erlang and I am doing performance tests.

My card function is characterized by a high processor intensity (mostly pure calculation). It also needs access to some static data, so I have several constant (protracted, that is, life through the application life cycle) work processes on my machine, each of which has some of this data in memory and is waiting for requests for a map. The output of the card is sent to the dispatching process (which sends the card requests to the workers), where the reduction is performed (very easy).

In any case, I noticed that I get a higher throughput when I immediately create a new process for each card request that employees receive, rather than letting the workflow itself synchronously execute the card request (one by one) like this leaving an empty request map in the process queue, because I immediately start the map requests).

Code snippet:

%% When I remove the comment, I get significant performance boost (90% -> 96%)
%% spawn_link(fun()->
                %% One invocation uses around 250ms of CPU time
                do_map(Map, AssignedSet, Emit, Data),
                Manager ! {finished, JobId, self(), AssignedSet, normal},
%%       end),

Compared to when I perform the same calculation in a closed loop, I get 96% throughput (efficiency) using the "immediate spawning" method (for example, 10,000 cards reduce jobs that are executed in full parallel). When I use the "employee does one after another" method, I get only about 90%.

, Erlang, , , , , 10K-, , 100 ..! , 4 , , , concurrency 4 , , 5.

, ( 100% ). . 100 , - 96% " " 90%, " ". , 200, 500, 1000, 10K.

, , - 25 , . 25 , ( , , ).

, . - , - ?

, ( , , ). , !

+5
2

1 3 , :

  _______   _______   _______
 |   m   | |   m   | |   m   |
 |       | |       | |       |
_|       |_|       |_|       |_
a         a         a         r

a - ( , ..) m - , r . , :

  _________________._
 |   m              r
 |  ___________________._
 | |   m                r
 | |  _____________________._
_|_|_|   m                  r
a a a

, (a), (m) , (r).

(.. ) , . , , , .

concurrency , . ( ), .

+3

-, , . :

  • ([rq: x] ) - : Erlang BIF , . . , . ( , , : " " , , , : 50 000 , : 5 * 10 000 = 50 000 .) ( )

  • 4 1 : , - . , 100%, , () / , (-) / "" . , "" , . -, , , . (/)

  • concurrency , (/ OTP-/etc.). : . ( )

+1

More articles:


All Articles