Hard and soft AWS ECS task memory limits

Question

I am confused by the fact that both hard and soft memory limits are specified for defining ECS ​​tasks.

IIRC. The size limit is the amount of memory that the scheduler reserves for the instance to complete the task, and the hard limit is how much memory the container can use before it is killed.

My problem is that if the ECS scheduler distributes tasks to instances based on the soft limit, you may have a situation where a task that uses memory above the soft limit but below the hard limit can cause the instance to exceed its maximum memory ( provided that all other tasks use memory slightly lower or equal to their soft limit).

Is it correct?

thanks

Answer 1

If you expect to run a computational workload that is mainly related to memory, not the processor, then you should use only a hard limit, not a soft limit. From the docs:

You must specify a nonzero integer for one or both memory elements or memoryReservation in the container definitions. If you specify both, the memory should be larger than memoryReservation. If you specify memoryReservation, this value will be subtracted from the available memory resources for the container instance in which the container is placed; otherwise, the memory value is used.

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html

By specifying only a hard memory limit for your tasks, you avoid running out of memory, since ECS stops putting tasks in an instance, and Docker kills all containers that try to break the hard limit.

The soft memory limit function is designed for applications with limited processor use, where you want to reserve a small minimum of memory (soft limit), but allow random outliers to a hard limit. With this type of large CPU workload, you don’t really care about the special value of using memory for containers, since containers will run out of CPUs long before they run out of instance memory, so you can place tasks based on CPU redundancy and soft memory limits . In this setting, the hard limit is simply fault tolerant in case something gets out of hand or a memory leak occurs.

So, in the end, you should evaluate your workload using stress tests and see if it tends to run out of CPU or memory out of stock first. If you are attached to a processor, you can use a soft memory limit with an additional hard limit as a fault tolerant. If you are limited by memory, you will only need to use a hard limit without a soft limit.