PHP memory usage tracking

Question

I am trying to track the memory usage of a script that processes URLs. The basic idea is to check for a reasonable buffer before adding another URL to the cURL multiprocessor. I use the concept of "roll cURL", which processes the URL data when the multiprocessor is running. This means that I can keep N connections active by adding a new URL from the pool each time an existing URL is processed and deleted.

I used memory_get_usage() with some positive results. Adding the real_usage flag helped (the difference between the "system" memory and the "emalloc" memory is not entirely clear, but the system shows large numbers). memory_get_usage() grows when URLs are added and then down as the set of URLs is exhausted. However, I just exceeded the 32M limit when my last memory check was ~ 18M.

I examine memory usage every time cURL multi signals return a request. Since multiple requests may return at the same time, it is likely that multiple URLs will return data at the same time and actually translate memory usage into 14M. However, if memory_get_usage() is accurate, I guess what happens.

[ Update . Should have run more tests before asking, I think the php memory limit was increased (but left the “safe” amount the same in the script), and the memory usage was reported to jump from below below my limit ranging from 25M to 32M. Then, as expected, slowly declined as URLs that were not added. But I will leave the question: is this the right way to do this?]

Can I trust memory_get_usage() this way? Are there any better alternative methods for using memory (I saw some scripts analyze shell command output)?

Answer 1

real_usage works as follows:

Zend's memory manager does not use the system malloc for every block it needs. Instead, it allocates a large block of system memory (with a step of 256 KB, can be changed by setting the environment variable ZEND_MM_SEG_SIZE ) and manages it internally. So there are two types of memory usage:

• How much memory the engine took from the OS ("real use")
• How much of this memory was actually used by the application ("internal use")

Any of these can be returned by memory_get_usage() . Which one is more useful to you depends on what you study. If you are looking for code optimization in certain parts, the “inside” may be more useful to you. If you track memory usage around the world, "real" will be more useful. memory_limit limits the "real" number, therefore, as soon as all the blocks that are allowed by the limit are taken from the system and the memory manager cannot allocate the requested block, the distribution is not performed. Note that the “internal” use in this case may be less than the limit, but the distribution may still fail due to fragmentation.

In addition, if you use the external memory tracking tool, you can set this environment variable USE_ZEND_ALLOC=0 , which will disable the above mechanism and force the engine to always use malloc() . This will have much worse performance, but allows the use of malloc tracking tools.

See also the article about this memory manager , there are also code examples.

Answer 2

I also assume that memory_get_usage() is safe, but I think you can compare both methods and decide for yourself, here is a function that parses system calls:

 function Memory_Usage($decimals = 2) { $result = 0; if (function_exists('memory_get_usage')) { $result = memory_get_usage() / 1024; } else { if (function_exists('exec')) { $output = array(); if (substr(strtoupper(PHP_OS), 0, 3) == 'WIN') { exec('tasklist /FI "PID eq ' . getmypid() . '" /FO LIST', $output); $result = preg_replace('/[\D]/', '', $output[5]); } else { exec('ps -eo%mem,rss,pid | grep ' . getmypid(), $output); $output = explode(' ', $output[0]); $result = $output[1]; } } } return number_format(intval($result) / 1024, $decimals, '.', ''); } 

Answer 3

Well, I never had a memory problem with my PHP scripts, so I don’t think I could help find the cause of the problem, but I can recommend that you get a PHP accelerator, you will notice that there is a serious increase in performance and memory usage with a decline. Here is a list of accelerators and an article comparing some of them (3 times better than any of them)

Wikipedia List

Benchmark

Tests are designed for 2 years, but you get an idea of ​​increasing productivity.

If you need to, you can also increase the memory limit in PHP if you still have problems even with an accelerator. Open php.ini and find:

 memory_limit = 32M; 

and increase it a little.