Geek Answers Handbook

Doing millions of Java tasks in a short amount of time?

I need to complete about 8 million tasks in Java. Each task runs between 1 and 50 ms. I tried to complete these tasks alternately with a loop and in parallel with ThreadPoolExecutor. Obviously, starting them sequentially in a loop is much slower, and then starting through ThreadPoolExecutorhas a significant increase in performance. However, I am still not satisfied with the performance using ThreadPoolExecutor.

I can run all 8,000,000 tasks using 15 threads on 4 cores. I perform 50,000 tasks when sending tasks to ThreadPoolExecutor. I allow a timeout of 1000 ms for ExecutorCompletionServiceto return the result. After setting up my application with these parameters, I get about 25 minutes to process all 8,000,000 tasks.

Is there a way to run my application faster than 25 minutes, perhaps Hadoop (distributed system using multiple machines)? I am open to using some other technology or framework.

+4
source share
2 answers

This Akka tutorial shows how Akka can help with parallel tasks.

Akka, , (, , , ...) .

: , Akka

+1

, .

, . -, ( , - , , ).

'amount of cores 'x 'utilization percentage' - . , :

int cores = Runtime.getRuntime().availableProcessors();


int poolcount = Math.round( cores * utilizationFactor);

, .

, , , , .

, , , . - . .

Hadoop, , , . .

Cassandra, , , /. , , .

, , , , / . Akka, , .

, - LMAX Disruptor (http://lmax-exchange.imtqy.com/disruptor/). . ( ) , .

0

More articles:


All Articles