Java-FixedThreadPool with known pool size but unknown workers

Question

So, I think I understand how fixed-thread pools work (using Executor.fixedThreadPool built into Java), but from what I see, the number of tasks you want to do is usually given, and you know how much until you run the program. for example

int numWorkers = Integer.parseInt(args[0]); int threadPoolSize = Integer.parseInt(args[1]); ExecutorService tpes = Executors.newFixedThreadPool(threadPoolSize); WorkerThread[] workers = new WorkerThread[numWorkers]; for (int i = 0; i < numWorkers; i++) { workers[i] = new WorkerThread(i); tpes.execute(workers[i]); } 

Where every workerThread does something really simple, this part is arbitrary. I want to know that if you have a fixed pool size (say, 8 max.), But you don’t know how many workers you need to complete the task before execution.

A specific example: if I have a pool size of 8, and I read it from standard input. When I read, I break the input into blocks of a given size. Each of these blocks is assigned a stream (along with some other information) so that they can compress it. This way, I don't know how many threads I need to create, since I need to continue until I get to the end of the input. I also have to somehow make sure that the data remains in the same order. If thread 2 ends before thread 1 and just sends it to work, my data will be inoperative!

Would a thread pool be the wrong approach in this situation? It seems to be great (since I can't use more than 8 threads at a time).

Basically, I want to do something like this:

 ExecutorService tpes = Executors.newFixedThreadPool(threadPoolSize); BufferedInputStream inBytes = new BufferedInputStream(System.in); byte[] buff = new byte[BLOCK_SIZE]; byte[] dict = new byte[DICT_SIZE]; WorkerThread worker; int bytesRead = 0; while((bytesRead = inBytes.read(buff)) != -1) { System.arraycopy(buff, BLOCK_SIZE-DICT_SIZE, dict, 0, DICT_SIZE); worker = new WorkerThread(buff, dict) tpes.execute(worker); } 

This is not working code, I know, but I'm just trying to illustrate what I want.

I forgot a bit, but I'll see how the buff and dict values ​​change, and I don't know how long the input is. I don’t think that I can’t actually make this thought, because a good worker already exists after the first call! I can’t just say work = new WorkerThread a lot of time, since it no longer points to an existing thread (true, a thread that may be dead) and, obviously, in this embodiment, if it works, I would not work in parallel. But I want to say that I want to continue to create threads until I get the maximum pool size, wait for the thread to finish, and then continue to create threads until I get to the end of the input.

I also need to keep things in order, which is the part that is really annoying.

Answer 1

Your solution is completely fine (the only thing parallelism might not be needed if your WorkerThread very small).

With a thread pool, the number of tasks transferred does not matter. There may be fewer or more threads in the pool, the thread pool will take care of this.

However, this is important: you rely on some order of results for your WorkerThread s, but when using parallelism this order is not guaranteed! It doesn't matter if you use a thread pool or how many work threads you have, etc., it will always be possible that your results will be completed in random order!

To maintain the correct order, give each WorkerThread number of the current element in its constructor and give them the result in the correct order after completion:

 int noOfWorkItem = 0; while((bytesRead = inBytes.read(buff)) != -1) { System.arraycopy(buff, BLOCK_SIZE-DICT_SIZE, dict, 0, DICT_SIZE); worker = new WorkerThread(buff, dict, noOfWorkItem++) tpes.execute(worker); } 

Answer 2

As @ignis points out, parallel execution may not be the best answer for your situation. However, to answer a more general question, there are several other Executor implementations to consider with the exception of FixedThreadPool, some of which may have the characteristics you want.

As for things in the order, as a rule, you send tasks to the performer, and for each view you get the future (this is an object that promises gives you the result later when the task is completed), so you can track the futures in the order in which you submitted tasks, and then, when all tasks are completed, call get () for each Future to get the results.