Dynamic memory failure recovery

Question

I am working on an integrated processor (400 MHz Intel PXA255 XScale), and it seemed to me that I saw one case when there is not enough memory to satisfy the “new” operation. The program did not crash, so I assumed that the other threads freed their memory, and that was just a transitional thing. This is pretty critical code, so the exit is not an option, and some error should be returned to the remote user.

Will a small fix be enough to solve the problem, or is there a better way? Before replacing each new code with the following code, I thought I'd ask.

char someArr[]; do{ someArr = new char[10]; Sleep(100); // no justification for choosing 100 ms } while ( someArr == NULL ); 

Does Sleep Help? Should I set a number of attempts? Is it possible to use static initialization everywhere?

FINAL UPDATE: Thank you very much for the helpful answers, but it turned out that the code check failed due to a memory failure. I will take into account all these answers and replace as many malloc and innovations as possible (especially in the error handling code).

Answer 1

There are several different ways to attack this - note that the tool instructions will be slightly different depending on which version of Windows CE / Windows Mobile you are using.

Some questions to answer:

1. Is your application a memory leak leading to this low memory condition?

2. Does your application simply use too much memory at certain stages, what leads to this low memory condition?

1 and 2 can be explored using the Windows CE AppVerifier tool, which can provide detailed memory logging tools for your product. Other heap wrapping tools may also provide similar information (and may be more productive), depending on the design of your product.

http://msdn.microsoft.com/en-us/library/aa446904.aspx

3. Do you allocate and free memory very often in this process?

Windows CE, prior to OS version 6.0 (not to be confused with Windows Mobile 6.x), has a 32 MB / process virtual memory limit, which usually causes a lot of fragmentation problems. In this case, even if you do not have enough physical memory, you may run out of virtual memory. The use of specialized distribution blocks is usually a mitigation for this problem.

4. Do you allocate very large blocks of memory? (> 2 MB)

Associated with 3, you could just run out of virtual process memory space. There are tricks, somewhat depending on the version of the OS, for allocating memory in the general space of the VM, outside the process space. If you run out of virtual machine but not physical RAM, this may help.

5. Do you use a large number of DLLs?

Also applies to 3, depending on the OS version, DLLs can also quickly reduce the total available virtual machine.

Further jumping from points:

CE Memory Tools Overview

http://blogs.msdn.com/ce_base/archive/2006/01/11/511883.aspx

The mi tool of the target control window

http://msdn.microsoft.com/en-us/library/aa450013.aspx

Answer 2

You are trying to solve a global problem with local reasoning. The global problem is that the entire device has a limited amount of RAM (and possibly backup storage) for the operating system and all applications. To make sure that this amount of RAM is not exceeded, you have several options:

• Each process runs in a fixed amount of RAM, which must be determined for each process at startup; the programmer makes reasoning to make sure that everything fits. So, yes, you can select everything statically . This is just a lot of work, and when you change the system configuration, you need to review the selection .

• Processes are aware of their use and memory needs and constantly tell each other how much memory they need. They interact, so they do not have enough memory . This suggests that at least some processes in the system can adjust their own memory requirements (for example, by changing the size of the internal cache). Alonso and Appel wrote a paper on this approach .

• Every process knows that memory can run out and can go into a state in which it consumes minimal memory . Often this strategy is implemented through exclusion from memory. An exception is handled at or near main (), and an out-of-memory event significantly restarts the program from scratch. This mode of switching to another resource may work if the memory grows in response to user requests; if the requirements for program memory are growing no matter what the user does, it can lead to shredding.

Your offer does not match any of the scenarios. Instead, you hope that some other process solves the problem , and eventually you need memory. You may be lucky. You can not.

If you want your system to work reliably, you would be better off reviewing the design of each process running on the system due to the need to exchange limited memory. It may be more work than you expected, but if you understand the problem, you can do it. Good luck

Answer 3

Other answers have a lot of good things, but I thought it’s worth adding that if all the threads fall into a similar loop, then the program will be blocked.

The “correct” answer to this situation probably has strict limits for different parts of the program to ensure that they do not exceed memory. This will probably require rewriting the main sections in all parts of the program.

The next best solution would be to have a callback when a failed allocation attempt can tell the rest of the program that more memory is needed. Perhaps other parts of the program may release some buffers more aggressively than usual, or release memory used to cache search results, or something else. This will require new code for other parts of the program. However, this can be done gradually, instead of requiring rewriting throughout the program.

Another solution would be to protect the program with large (temporary) memory requests using a mutex. It looks like you are sure that the memory will be released soon if you can just try again later. I suggest you use the mutex for operations that can consume a lot of memory, this will allow the thread to wake up immediately when another thread frees up the required memory. Otherwise, your thread will sleep for a tenth of a second, even if memory is freed immediately.

You can also try sleep (0), which simply transfers control to any other thread that is ready to run. This will allow your thread to regain control immediately if all other threads fall asleep, instead of waiting for its 100-millisecond offer. But if at least one thread still wants to work, you still have to wait until it gives up control. This is usually 10 milliseconds on Linux machines, the last time I checked. I do not know about other platforms. Your thread may also have a lower priority in the scheduler if he voluntarily went to bed.

Answer 4

Based on your question, I assume that your heap is split between multiple threads.

If this is not the case, then the code above will not work, because nothing will be freed from the heap during the loop.

If the heap is split, then it will probably work higher. However, if you have a common heap, then calling the “new” one will probably result in a spin lock (a similar loop to the one you have, but using CAS instructions), or it will block based on some kernel resources.

In both cases, the loop you have will reduce the throughput of your system. This is due to the fact that you will either encounter additional context switches or you will need more time to respond to the "memory is now available" event.

I would prefer to override the "new" and "delete" statements. With a new failure, you can lock (or lock the lock of some counter variable), waiting for another thread to free memory, and then delete it can either signal a blocked "new" thread or increase the counter variable using CAS.

This should give you better bandwidth and be more efficient.

Answer 5

A few points:

• Nested programs often allocate all memory at startup or use only static memory to avoid such situations.
• If there is anything else on the device that frees up memory on a regular basis, your solution is unlikely to be effective.
• I have 64 MB of RAM in Viper, I don’t think they have less than 32 MB, how much memory does your application use?

Answer 6

Secondly, the most reasonable is the use of static memory allocation, so you have an idea of ​​what is happening. Dynamic memory allocation is a bad habit of programming on the desktop, which is not suitable for machines with limited resources (unless you spend a lot of time and effort on creating a well-managed and controlled memory management system).

Also, check which OS features on your device (provided that it has one, high-performance ARM devices like this one, usually for working with the OS) to process memory.

Answer 7

You are using C ++. This way you can use some C ++ utilities to make your life easier. For example, why not use new_handler?

 void my_new_handler() { // make room for memory, then return, or throw bad_alloc if // nothing can be freed. } int main() { std::set_new_handler(&my_new_handler); // every allocation done will ask my_new_handler if there is // no memory for use anymore. This answer tells you what the // standard allocator function does: // https://stackoverflow.com/questions/377178 } 

In new_handler, you can signal all applications to let them know that some applications require memory, and then wait a bit to give other applications time to complete a memory request. It is important that you do something and do not tacitly hope for available memory. The new statement will call your handler again if there is still not enough memory, so you do not need to worry about whether all applications have already freed free memory. You can also overload the new operator if you need to know the size of memory needed by new_handler. See my other answer on how to do this. Thus, you have one central place to solve memory problems, and not in many places related to it.

Answer 8

As others have pointed out, ideally you would have avoided this problem with the help of design architecture and software, but Im, assuming that at the moment this is really not an option.

As mentioned in another publication, it would be nice to wrap the logic in some utility functions so that you do not write code out of memory to the entire place.

To get to the real issue, you are trying to use a shared resource, memory, but you cannot, because this shared resource is being used by another thread in the system. Ideally, you need to wait until one of the other threads in the system releases the resource you need, and then acquires that resource. If you had a way to intercept all the allocation and free calls, you could set something up so that the allocation of the thread would be blocked until the memory was available, and the release signaled the allocation of the stream if there was memory. But I guess this is just too much work.

Given the limitations of the inability to completely redesign the system or rewrite the memory allocator, I think that your solution is most practical if you (and others in your team) understand the limitations and problems that it will cause on the track.

Now, to improve your specific approach, you can measure your workload to see how often memory is allocated and freed. This would give you the opportunity to calculate what the repetition interval should be.

Secondly, you want to try to increase the timeout for each iteration in order to reduce the load on this thread in the system.

Finally, you should definitely have some time with an error / panic case if the thread cannot make progress after a certain number of iterations. This will allow you to at least see a potential real-time lock case that you might encounter if all threads expect another thread in the system to free up memory. You could just pick a few iterations based on what is empirically shown at work, or you could get smarter about it and keep track of how many threads are stuck waiting for memory, and if it ends up with all the threads panic.

Note. This is obviously not an ideal solution, and other posters claim that to solve the problem, a more global view of the application as a whole is needed, but above it is a practical method that should work in short -term.

Answer 9

Of course, this will depend on whether you have a reasonable expectation of memory available in sleep 100 (milliseconds?)? Of course, you should limit the number of attempts.

Something doesn't smell right to me. Hmmm ...

Embedded systems, as a rule, should be extremely deterministic - perhaps you should consider the entire system and lead the potential for this in order to fall forward; and then just fails, it actually happens in practice.