I have a program that receives data from a socket, performs some quality control and distinguishes other settings for it, and then writes it to a named pipe. I ran into it and fixed all the memory leaks that originally existed. Then I created the “demo” environment on a system in which I had 32 instances of this program, each of which supplied unique data and each of them displayed its own channel on it. We tested it, and everything was in order. Then I tried stress testing, increasing the data transfer rate to an absurd speed, and at first everything was fine, but my programs continued to consume more and more memory until I had no resources left.
I turned to valgrind and did the same setup, with the exception of every program running inside valgrind, by checking for leak = complete. A few strange things happened. Firstly, the memory was leaking, but only until the moment when each program consumed 0.9% of my memory (previously the largest memory was full 6% of my memory). When valgrind ran the cost of the processor for programs, and now I was on a 100% processor with a huge average load, so maybe a lack of available processor caused all the programs to run slowly enough for the leak to appear for too long. When I tried to stop these programs, valgrind did not detect direct memory leaks, it showed some potential memory leaks, but I checked them, and I do not think that any of them represents real memory leaks; and, in addition, a possible memory leak appeared only in a few kilobytes, while the program consumed more than 100 MB. The reachable (non-leaking) memory reported by valgrind was also in the KB range, so valgrind seems to believe that my programs consume some of the memory that Top says they use.
I conducted several other tests and got odd results. One program, even three times faster than my original memory leak, was detected, it never seems to consume more than 0.9% of memory, two programs leak up to 1.9 and 1.3% of memory, respectively, but no more etc., as if the amount of memory leak and its leak rate somehow depend on how many instances of my program are running at a time; which makes no sense, each instance should be 100% independent of the others.
I also found that if I run 32 instances, and only one instance running in valgrind has a value of valgrinded (this word, if I say it!) Memory leak, but slower than those that work outside of valgrind. The valgrind instance will still say that I have no direct leaks and reports much less memory consumption than Top shows.
I'm pretty dumb about what might cause this result, and why valgrind refuses to be aware of a memory leak. I thought it could be an external library, but I really do not use any external libraries; just basic C ++ functions / objects. I also thought that it could be data written to the output channel in order to quickly cause the buffer to grow indefinitely, but 1) there should be an upper limit that such a buffer can have, and 2) as soon as the memory was leaked, if I delete the data speed entering to zero, the memory remains consumed, and then slowly returns to a reasonable amount.
Can someone give me a hint about where I should look from here? I am completely obsessed with why memory behaves this way.
Thanks.