Linux memory usage history

Question

I had a problem when my server started to crash some normal processes and checks, because the server memory was full and taken.

I looked at the registration history and found that these were some Java processes.

I used the "top" command to see which processes are taking up most of the memory right now (after fixing the problem), and it was a Java process. Thus, in fact, I can say which processes occupy most of the memory right now.

What I want to know is there a way to see which processes take up most of the memory at the time the failures occurred? Perhaps Linux monitors or logs memory usage at specific times? I really have no idea, but it would be great if I could see such details.

Answer 1

Are you saying that the OOM kernel killer is gone? What does the magazine say in dmesg? Note that you can restrict the JVM to use a fixed heap size, which means that it will fail unsuccessfully when it is full, instead of letting the kernel kill something else. But the general answer to your question is no: there is no way to reliably start something during an OOM failure, because the system does not have enough memory! In the best case scenario, you can use a separate process to poll the process table and log sizes for catching memory leaks, etc.

Answer 2

@ Andy answered your question. However, I would like to add that for future reference use a monitoring tool. Something like . This will give you what happened during the failure, as you obviously cannot constantly monitor all your servers. Hope this helps.

Answer 3

Linux's memory history is not used by default, but you can access it with a simple command line tool such as sar .

As for your memory problem: If it were an OOM-killer that messed up the machine a bit, you have one great option to make sure it doesn't happen again (of course, after reducing the heap size of the JVM).

By default, linux kernel allocates more memory than it actually does. In some cases, this can cause the OOM-killer to kill most memory-related processes if there is no memory for kernel tasks. This behavior is controlled by the vm.overcommit sysctl parameter.

So you can try setting it to vm.overcommit = 2 is sysctl.conf and then run sysctl -p .

This will prohibit excessive work and make killing an OOM killer very unpleasant. You can also consider adding a small amount of swap space (if you don’t already have one) and setting vm.swappiness to some really low value (for example, 5 , for example, the default value is 60 ), so in an ordinary workflow your application will not will be inserted into swap, but if you really will be a bit in memory, it will start using it temporarily, and you can even see it with df

WARNING , this can cause processes to receive a "Cannot allocate memory" error if your server is overloaded with memory. In this case:

• Try to limit memory usage by applications
• Move some of them to another machine.