Geek Answers Handbook

Creating too many threads in java

I use threads in my Java application to get data in parallel (using network calls). I have a method (which is not part of the stream class) that creates a pool of threads with a given size (10-15 max.) And uses them for network calls, and I call this method several times from the loop.

When I run this application on a slow machine (3 GB RAM, Pentium-IV), everything works fine, but when I run it on the iMac (32 GB RAM, i7 processor), it creates too many threads, 2500 sometimes throws memory error.

I suspect that the JVM does not return completed threads back to the pool, they are finished soon, so new threads are created.

And even on iMac, if I continue Thread.sleep (1000); in the for loop I mentioned above, everything works fine. Create about 900 threads.

The following are sample code from this application:

public ArrayList<String> getValuesForKeyFromMaps(String key, ArrayList<Meta> locations) throws InterruptedException, ExecutionException{ int threadNum = locations.size(); // 10-15 at max ExecutorService executor = Executors.newFixedThreadPool(threadNum); List<FutureTask<ArrayList<String>>> taskList = new ArrayList<FutureTask<ArrayList<String>>>(); for(final Meta location : locations){ FutureTask<ArrayList<String>> futureTask_1 = new FutureTask<ArrayList<String>>(new Callable<ArrayList<String>>() { public ArrayList<String> call() throws Exception { // service call return getValues(key, location); } }); taskList.add(futureTask_1); executor.execute(futureTask_1); } ArrayList<String> values = new ArrayList<String>(); // Wait until all results are available and combine them at the same time for (int j = 0; j < threadNum; j++) { FutureTask<ArrayList<String>> futureTask = taskList.get(j); values.addAll(futureTask.get()); } executor.shutdown(); return values; }

If I call the above method using below for a loop on the iMac, it throws a memory error, as it creates about 2500 threads. But it works on a slow machine.

  for(String key : keySet){ getValuesForKeyFromMaps(key, metaMap.get(key)); }

And, with the code below, about 900 threads work fine on the iMac.

  for(String key : keySet){ getValuesForKeyFromMaps(key, metaMap.get(key)); Thread.sleep(200); //sleeping for 200ms }

If I increased the latency in the above for the cycle to 1000 ms, its creation is only 30-50 threads, and the application works fine.

How to control the maximum threads allowed in my application. I am going to create / use 10-15 threads with maximum value at a given time, but java creates too much.

+6
source share
2 answers

This is not java creating too many threads, you!

Do not create an executor every time you call a function. If you have 100 collections of 100 elements each, you will create 10,000 threads - this is very resource-intensive ... And it makes no sense.

 ExecutorService executor = Executors.newFixedThreadPool(threadNum);

You most likely have 8 cores - just create one artist with 8 threads and use it everywhere. Your code will run faster and your application will consume less, much less resources.

Check out this singleton reviewer reviewer question. You can use this solution in your application.

+4
source

Using ExecutorService executor = Executors.newFixedThreadPool(threadNum); , you create a new thread pool for each call to getValuesForKeyFromMaps . Therefore, when your keySet contains 100 entries, you will receive 100 pools of 10-15 threads. Store one thread pool as an instance or class variable and use it when necessary.

+3
source

More articles:


All Articles