Java memory usage / thread pool performance issue

Question

These things obviously require careful testing and the availability of code to carefully analyze and provide good suggestions. However, this is not always possible, and I hope that it will be possible to provide me with useful tips based on the information presented below.

I have a server application that uses a listener thread to listen for incoming data. Incoming data is interpreted in the messages of specific applications, and these messages then lead to events.

Until this moment, I have no control over how everything is done.

Since this is an outdated application, these events previously dealt with the same listener thread (mostly a single-threaded application). Events are sent to the black box, and the output appears, which should be written to disk.

To increase throughput, I wanted to use threadpool to monitor events. The idea was that the listener thread could simply create new tasks every time an event is created, and the threads would take care of calling the black box. Finally, I have a background thread that writes to disk.

Only with the previous installation and the background author does everything work fine, and the throughput is 1.6 times more than before.

When I add a thread pool, but performance degrades. From the very beginning everything seems smooth, but after a while everything happens very slowly, and finally I get OutOfMemoryExceptions. The strange thing is that when I print the number of active threads every time a task is added to the pool (along with information about how many tasks are queued, etc.), it seems that the thread pool has no problems with to keep up with the producer (stream of listeners).

Using top -H to test CPU usage, it is fairly evenly distributed from the start, but in the end, worker threads are barely active, and only the listener thread is active. However, it does not seem to represent any more tasks ...

Can anyone suggest the cause of these symptoms? Do you find it more likely that there is something wrong with the legacy code (which I don’t control) that when adding multiple threads? The problem with the lack of memory should be due to the fact that some queue is somewhere too large, but since the thread stream almost never contains tasks in the queue, it cannot be.

Any ideas are welcome. Especially ideas on how to more effectively diagnose a similar situation. How can I get a better profile on what my threads do, etc.

Thanks.

Answer 1

Thanks for answers. I read in VisualVM in Java and used this as a tool. The results and conclusions are described in detail below. We hope that the photos will work long enough.

First, I ran the program and created some heap heaps, thinking that I could just analyze the dumps and see what all the memory takes. This probably worked, except that the dump file was so large and my workstation had limited use when trying to access it. After waiting two hours in one operation, I realized that I could not do this.

So, my next option was something that I, stupidly, did not think. I could just reduce the number of messages sent to the application, and the trend of increasing memory usage should still be there. In addition, the dump file will be smaller and faster for analysis.

It turns out that when sending messages at a lower speed there was no memory problem! The memory usage schedule is shown below.

slow sending http://img197.imageshack.us/img197/1628/slowsend.png

Peaks are the results of cumulative memory allocations, and subsequent gutters after the garbage collector starts. Although the amount of memory used is certainly alarming and there are probably problems, there is no long-term trend for memory leaks.

I began to gradually increase the speed of sending messages per second to see where the application hits the wall. The image below shows a completely different scenario than the previous one ...

fast transfer http://img200.imageshack.us/img200/151/fastsend.png

Since this happens when the speed of sent messages increases, I assume that my release of the listener stream leads to the fact that it is able to receive many messages very quickly, and this causes more and more deductions. The garbage collector does not start and memory usage hits the wall.

There is, of course, more to this problem, but given what I learned today, I have a pretty good idea of ​​where to go from here. Of course, any additional suggestions and comments are welcome.

These questions should probably be reclassified as working using memory, not file paths ... Threadpool was not a problem at all.

Answer 2

Slowing and then running out of memory means a memory leak.

So, I would start by using some Java memory analyzer tools to determine if there is a leak and what is happening. Sometimes you are lucky and the leaked object is well known, and it becomes pretty clear who is hanging on things that they should not.

Answer 3

I agree with @djna. The java concurrency pool package works. It does not create threads if they do not need them. You see that the number of threads is in line with expectations. This means that something in your legacy code is probably not ready for multithreading. For example, a piece of code is out of sync. As a result, some item is not deleted from the collection. Or some additional items are stored in the collection. Thus, memory usage is increasing.

By the way, I didn’t understand exactly which part of the application now uses threadpool. Do you have one thread that processes events, and now you have several threads that do this? Perhaps you have changed the mechanism for exchanging data between threads? Queues added? This may be another area of ​​your investigation.

Good luck

Answer 4

As djna mentioned, this is probably some type of memory leak. I assume that you will reference the request somewhere:

• In the dispatcher thread, which is in the request queue
• In threads that handle requests
• In a black box that processes requests
• In the write stream that is being written to disk.

Since you said that everything works before adding the thread pool to the mix, I assume that the threads in the pool will store the request link somewhere. The idea was that without threadpool you are not reusing threads so that information disappears.

As recommended by djna, you can use the Java memory analyzer to find out where the data is stacked.