Geek Answers Handbook

Apache httpd using mem until hang

It looks like we might have a (growing) memory issue in our apache httpd somewhere.

Quick pic: http://screencast.com/t/NGAzarD87O

Please note that it has been running on the source (physical) server for some time. Now it works on a new virtual machine (with a large amount of memory and a processor), but it slowly leaves for mem / swap until the system freezes.

If I restart httpd, mem will come back (if we catch it).

httpd.x86_64 2.2.3-76.el5_9 installed PHP 5.1.6 (cli) (built: Jun 22 2012 06:20:25) MySQL Server version: 5.0.95

I do not think that this can be any of the scripts that are run / etc, since they have been working on a physical machine for many years. We tried to match all the configs (http, php, etc.) on the new computer, but we cannot understand why httpd continues to grow.

 $ ps -ylC httpd --sort:rss S UID PID PPID C PRI NI RSS SZ WCHAN TTY TIME CMD S 0 13814 1 0 78 0 29208 68382 - ? 00:00:00 httpd S 48 20854 13814 0 76 0 34876 70930 semtim ? 00:00:00 httpd S 48 20853 13814 0 75 0 36592 71387 semtim ? 00:00:00 httpd S 48 13822 13814 0 75 0 36780 71430 semtim ? 00:00:00 httpd S 48 20696 13814 0 75 0 37092 71520 semtim ? 00:00:00 httpd S 48 13821 13814 0 75 0 37184 71529 semtim ? 00:00:01 httpd S 48 13820 13814 0 75 0 37220 71527 - ? 00:00:01 httpd S 48 13824 13814 0 75 0 37236 71513 semtim ? 00:00:01 httpd S 48 13818 13814 0 75 0 37636 71547 semtim ? 00:00:01 httpd S 48 13819 13814 0 75 0 37636 71617 semtim ? 00:00:01 httpd S 48 13823 13814 0 75 0 37888 71689 semtim ? 00:00:01 httpd S 48 13825 13814 0 75 0 37900 71676 semtim ? 00:00:01 httpd

UPDATE: when writing this question (maybe 10-15 minutes) I will repeat above, and RSS all sit at ~ 51072 instead of above at ~ 37000

Python program run here

  Private + Shared = RAM used Program .... SNIPPED OUT .... 208.0 MiB + 25.5 MiB = 233.5 MiB httpd (12) --------------------------------- 477.1 MiB

PHP mem settings:

 max_execution_time = 30 max_input_time = 60 memory_limit = 152M

Modules:

 $ apachectl -M httpd: Could not reliably determine the server fully qualified domain name, using 192.168.1.2 for ServerName Loaded Modules: core_module (static) mpm_prefork_module (static) http_module (static) so_module (static) auth_basic_module (shared) auth_digest_module (shared) authn_file_module (shared) authn_alias_module (shared) authn_anon_module (shared) authn_dbm_module (shared) authn_default_module (shared) authz_host_module (shared) authz_user_module (shared) authz_owner_module (shared) authz_groupfile_module (shared) authz_dbm_module (shared) authz_default_module (shared) ldap_module (shared) authnz_ldap_module (shared) include_module (shared) log_config_module (shared) logio_module (shared) env_module (shared) ext_filter_module (shared) mime_magic_module (shared) expires_module (shared) deflate_module (shared) headers_module (shared) usertrack_module (shared) setenvif_module (shared) mime_module (shared) dav_module (shared) status_module (shared) autoindex_module (shared) info_module (shared) dav_fs_module (shared) vhost_alias_module (shared) negotiation_module (shared) dir_module (shared) actions_module (shared) speling_module (shared) userdir_module (shared) alias_module (shared) rewrite_module (shared) proxy_module (shared) proxy_balancer_module (shared) proxy_ftp_module (shared) proxy_http_module (shared) proxy_connect_module (shared) cache_module (shared) suexec_module (shared) disk_cache_module (shared) file_cache_module (shared) mem_cache_module (shared) cgi_module (shared) version_module (shared) perl_module (shared) php5_module (shared) proxy_ajp_module (shared) python_module (shared) ssl_module (shared) Syntax OK

Httpd.conf settings:

 IfModule prefork.c> ignore format on these tags StartServers 8 MinSpareServers 5 MaxSpareServers 20 ServerLimit 256 MaxClients 256 MaxRequestsPerChild 4000 # worker MPM StartServers 2 MaxClients 150 MinSpareThreads 25 MaxSpareThreads 75 ThreadsPerChild 25 MaxRequestsPerChild 0

Using the program at https://code.google.com/p/check-httpd-limits/wiki/Documentation I received:

  Check Apache Httpd MPM Config Limits (Version 2.4)
 by Jean-Sebastien Morisset - http://surniaulula.com/

 Httpd binary

  - CONFIG: /etc/httpd/conf/httpd.conf
  - EXE: / usr / sbin / httpd
  - MPM: prefork
  - ROOT: / etc / httpd
  - VERSION: 2.2

 Httpd processes

  - PID 10860 (httpd): 106.93 MB / 3.95 MB shared
  - PID 13814 (httpd): 28.52 MB / 6.36 MB shared [excluded from averages]
  - PID 13818 (httpd): 180.28 MB / 4.29 MB shared
  - PID 13819 (httpd): 182.67 MB / 4.04 MB shared
  - PID 13820 (httpd): 182.45 MB / 4.08 MB shared
  - PID 13821 (httpd): 185.53 MB / 4.04 MB shared
  - PID 13822 (httpd): 176.12 MB / 4.36 MB shared
  - PID 13823 (httpd): 180.05 MB / 4.04 MB shared
  - PID 13824 (httpd): 182.21 MB / 4.05 MB shared
  - PID 13825 (httpd): 179.36 MB / 4.04 MB shared
  - PID 20696 (httpd): 180.10 MB / 4.04 MB shared
  - PID 20853 (httpd): 180.39 MB / 4.03 MB shared
  - PID 20854 (httpd): 180.79 MB / 4.04 MB shared
  - PID 21003 (httpd): 159.77 MB / 4.05 MB shared

  - HttpdRealAvg: 166.09 MB [excludes shared]
  - HttpdSharedAvg: 4.05 MB
  - HttpdRealTot: 2225.76 MB [excludes shared]
  - HttpdRunning: 14

 Httpd Config

  - StartServers: 8
  - ServerLimit: 256
  - MinSpareServers: 5
  - MaxSpareServers: 20
  - MaxRequestsPerChild: 4000
  - MaxClients: 256

 Server memory

  - Cached: 671.46 MB
  - MemFree: 547.88 MB
  - MemTotal: 3819.89 MB
  - SwapFree: 5951.89 MB
  - SwapTotal: 5951.99 MB

 Calculations Summary

  - OtherProcsMem: 370.74 MB (MemTotal - Cached - MemFree - HttpdRealTot - HttpdSharedAvg)
  - FreeMemNoHttpd: 3449.15 MB (MemFree + Cached + HttpdRealTot + HttpdSharedAvg)
  - MaxLimitHttpdMem: 42523.09.09 MB (HttpdRealAvg * MaxClients + HttpdSharedAvg)
  - AllProcsTotalMem: 42893.83 MB (OtherProcsMem + MaxLimitHttpdMem)

 Maximum Values ​​for MemTotal (3819.89 MB)

   
         StartServers 8 # (no change) Default is 5
         ServerLimit 21 # (256 -> 21) MaxClients
         MinSpareServers 5 # (no change) Default is 5
         MaxSpareServers 20 # (no change) Default is 10
         MaxRequestsPerChild 4000 # (no change) Default is 10000
         MaxClients 21 # (256 -> 21) (MemFree + Cached + HttpdRealTot + HttpdSharedAvg) / HttpdRealAvg
   

 Result

 ERROR: AllProcsTotalMem (42893.83 MB) exceeds MemTotal (3819.89 MB) and free swap (5951.89 MB) by 33122.05 MB.
+7
source share
2 answers

There can be any number of reasons why Apache can take up too much memory. Although there is no universal “answer”, there is a troubleshooting list:

  • Reinstall Apache and all modules.
  • Disconnect all modules that you do not need. Besides the potential use of additional resources, this can also be a security risk.
    • mod_autoindex will allow anyone to view all files in any directory in your public_html directory if there is no index. * in this directory.
    • mod_alias will not only show that you are using Apache, but it will also be a good indicator that you have not removed additional modules and will make you a more likely target.
    • Remove all proxy modules if you are not actually using them.
    • Remove all mod_cgi , mod_php etc. that you don't need. The more file types you allow Apache, the more opportunities for an attacker to be malicious.
  • Check server configuration. Don't be afraid to research what specific directives do, and don't be afraid to back up your apache.conf and change a few things. The worst case is that Apache does not start or Apache crashes.
+1
source

Sometimes allocating too much memory creates a memory problem. If there is no particular reason for allocating 152M for PHP, this is high. Take it down to 32M or 64M. If more is needed for a particular script, enable it only for that script.

Then check the database distribution. Often, memory is the first thing that people set aside to do something faster when there are other elements that need to be executed (for example, setting up a query).

In fact, too much allocation creates back-up information for I / O. Sounds backwards, I know. I had a similar problem and this was related to resource allocation. Drop your resources to their defaults and see if that changes memory usage at all. Tune in from there to find the best settings.

+1
source

More articles:


All Articles