RabbitMQ Generic server rabbit_disk_monitor terminating / eheap_alloc: Unable to allocate 229520 bytes of memory (like "old_heap")

RabbitMQ crashed. RabbitMQ worked correctly for many days (10-15 days). I don’t understand why it crashed.


I am using RabbitMQ 3.4.0 on Erlang 17.0


erlang created a dump file for failure. What shows

eheap_alloc: Cannot allocate 229520 bytes of memory (of type "old_heap"). 

Also note that the download message for a rabbitmq subscription is very low. (max: 1-2 messages per second). And RabbitMQ messages are handled the way it happens, so RabbitMQ is almost empty all the time. There is also enough disk space and memory.

 More system info: Limiting to approx 8092 file handles (7280 sockets) Memory limit set to 6553MB of 16383MB total. Disk free limit set to 50MB. 

RabbitMQ logs are listed below.

 =ERROR REPORT==== 18-Jul-2015::04:29:31 === ** Generic server rabbit_disk_monitor terminating ** Last message in was update ** When Server state == {state,"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/ rabbit@localhost-mnesia ", 50000000,28358258688,100,10000, #Ref<0.0.106.70488>,false} ** Reason for termination == ** {eacces,[{erlang,open_port, [{spawn,"C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/ rabbit@localhost-mnesia \""}, [stream,in,eof,hide]], []}, {os,cmd,1,[{file,"os.erl"},{line,204}]}, {rabbit_disk_monitor,get_disk_free,2,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,599}]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]} =INFO REPORT==== 18-Jul-2015::04:29:31 === Disabling disk free space monitoring on unsupported platform: {{'EXIT',{eacces,[{erlang,open_port, [{spawn,"C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/ rabbit@localhost-mnesia \""}, [stream,in,eof,hide]], []}, {os,cmd,1,[{file,"os.erl"},{line,204}]}, {rabbit_disk_monitor,get_disk_free,2,[]}, {rabbit_disk_monitor,init,1,[]}, {gen_server,init_it,6,[{file,"gen_server.erl"},{line,306}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]}}, 17179336704} =INFO REPORT==== 18-Jul-2015::04:29:31 === Disabling disk free space monitoring on unsupported platform: {{'EXIT',{eacces,[{erlang,open_port, [{spawn,"C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/ rabbit@localhost-mnesia \""}, [stream,in,eof,hide]], []}, {os,cmd,1,[{file,"os.erl"},{line,204}]}, {rabbit_disk_monitor,get_disk_free,2,[]}, {rabbit_disk_monitor,init,1,[]}, {gen_server,init_it,6,[{file,"gen_server.erl"},{line,306}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]}}, 17179336704} 

 =CRASH REPORT==== 18-Jul-2015::04:29:31 === crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.167.0> registered_name: rabbit_disk_monitor exception exit: {eacces, [{erlang,open_port, [{spawn, "C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/ rabbit@localhost-mnesia \""}, [stream,in,eof,hide]], []}, {os,cmd,1,[{file,"os.erl"},{line,204}]}, {rabbit_disk_monitor,get_disk_free,2,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,599}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]} in function gen_server:terminate/6 (gen_server.erl, line 746) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.140.0>] messages: [] links: [<0.166.0>] dictionary: [] trap_exit: false status: running heap_size: 4185 stack_size: 27 reductions: 481081978 neighbours: =SUPERVISOR REPORT==== 18-Jul-2015::04:29:31 === Supervisor: {local,rabbit_disk_monitor_sup} Context: child_terminated Reason: {eacces, [{erlang,open_port, [{spawn, "C:\\Windows\\system32\\cmd.exe /c dir /-C /W \"c:/Users/jasmin.joshi/AppData/Roaming/RabbitMQ/db/ rabbit@localhost-mnesia \""}, [stream,in,eof,hide]], []}, {os,cmd,1,[{file,"os.erl"},{line,204}]}, {rabbit_disk_monitor,get_disk_free,2,[]}, {rabbit_disk_monitor,internal_update,1,[]}, {rabbit_disk_monitor,handle_info,2,[]}, {gen_server,handle_msg,5, [{file,"gen_server.erl"},{line,599}]}, {proc_lib,init_p_do_apply,3, [{file,"proc_lib.erl"},{line,239}]}]} Offender: [{pid,<0.167.0>}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,{transient,1}}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 18-Jul-2015::04:29:31 === crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.24989.51> registered_name: [] exception exit: unsupported_platform in function gen_server:init_it/6 (gen_server.erl, line 322) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.140.0>] messages: [] links: [<0.166.0>] dictionary: [] trap_exit: false status: running heap_size: 1598 stack_size: 27 reductions: 650 neighbours: =SUPERVISOR REPORT==== 18-Jul-2015::04:29:31 === Supervisor: {local,rabbit_disk_monitor_sup} Context: start_error Reason: unsupported_platform Offender: [{pid,<0.167.0>}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,{transient,1}}, {shutdown,4294967295}, {child_type,worker}] =CRASH REPORT==== 18-Jul-2015::04:29:31 === crasher: initial call: rabbit_disk_monitor:init/1 pid: <0.24991.51> registered_name: [] exception exit: unsupported_platform in function gen_server:init_it/6 (gen_server.erl, line 322) ancestors: [rabbit_disk_monitor_sup,rabbit_sup,<0.140.0>] messages: [] links: [<0.166.0>] dictionary: [] trap_exit: false status: running heap_size: 1598 stack_size: 27 reductions: 650 neighbours: =SUPERVISOR REPORT==== 18-Jul-2015::04:29:31 === Supervisor: {local,rabbit_disk_monitor_sup} Context: start_error Reason: unsupported_platform Offender: [{pid,{restarting,<0.167.0>}}, {name,rabbit_disk_monitor}, {mfargs,{rabbit_disk_monitor,start_link,[50000000]}}, {restart_type,{transient,1}}, {shutdown,4294967295}, {child_type,worker}] 
+5
source share
2 answers

From the error message, rabbitmq could not open more files due to system restrictions. You can set the maximum open file numbers to the upper value to avoid the problem.

https://serverfault.com/questions/249477/windows-server-2008-r2-max-open-files-limit

+2
source

There are two unrelated errors here: one is the virtual machine's failure to allocate memory. Another is stopping disk space. Disk Disk Monitor is optional, and on some less common platforms or with certain security restrictions, it is known to fail. This does not lead to a decrease in VM and, of course, has nothing to do with failures in the allocation of heaps.

Heap allocation error usually comes down to the two most common cases:

  • Known bug fixed in Erlang 17.x (don’t remember which patch release, so use 17.5)
  • You are running 32-bit Erlang / OTP on a 64-bit OS.

Chen Yu comment on the EACCESS system call error is correct.

+1
source

All Articles