AWS Redis + uWSGI for NGINX - High Load

Question

I am running a python application (flask + redis-py) with uwsgi + nginx and using aws elasticache (redis 2.8.24).

trying to improve application response time, I noticed that under high load (500 requests per second / for 30 seconds using loader.io) I lose requests (for this test I use only one server without load balancing, 1 instance of uwsgi, 4 process intended for testing).

I went deeper and found out that under this download some requests to ElastiCache are slow. eg:

• normal load: cache_set time 0.000654935836792
• heavy load: cache_set time 0.0122258663177 this does not happen for all requests, it just happens by accident.

My AWS ElastiCache is based on 2 nodes on cache.m4.xlarge (AWS default configuration settings). See current clients connected in the last 3 hours:

I think this does not make sense since there are currently 14 servers (8 of them with high traffic XX RPS use this cluster), I expect to see a much higher client speed.

UWSGI configuration (version 2.0.5.1)

processes = 4 enable-threads = true threads = 20 vacuum = true die-on-term = true harakiri = 10 max-requests = 5000 thread-stacksize = 2048 thunder-lock = true max-fd = 150000 # currently disabled for testing #cheaper-algo = spare2 #cheaper = 2 #cheaper-initial = 2 #workers = 4 #cheaper-step = 1 

Nginx is just a web proxy for uWSGI using a unix socket.

This is how I open a connection with redis:

 rdb = [ redis.StrictRedis(host='server-endpoint', port=6379, db=0), redis.StrictRedis(host='server-endpoint', port=6379, db=1) ] 

This is how I set the value, for example:

 def cache_set(key, subkey, val, db, cache_timeout=DEFAULT_TIMEOUT): t = time.time() merged_key = key + ':' + subkey res = rdb[db].set(merged_key, val, cache_timeout) print 'cache_set time ' + str(time.time() - t) return res cache_set('prefix', 'key_name', 'my glorious value', 0, 20) 

This is how I get the value:

 def cache_get(key, subkey, db, _eval=False): t = time.time() merged_key = key + ':' + subkey val = rdb[db].get(merged_key) if _eval: if val: val = eval(val) else: # None val = 0 print 'cache_get time ' + str(time.time() - t) return val cache_get('prefix', 'key_name', 0) 

Version:

• uWSGI: 2.0.5.1
• Flask: 0.11.1
• redis-py: 2.10.5
• Redis: 2.8.24

So conclude:

• Why AWS clients are considered low if 14 servers are connected, each of which has 4 processes, and each of them opens a connection to 8 different databases in a redis cluster
• What makes query response times rise?
• Would thank for any recommendations regarding ElastiCache and / or uWSGI performance under heavy load

Answer 1

Short answer

So, if I understood correctly, in my case the problem was not in Elasticache requests, but in the use of uWSGI memory.

Long answer

I installed uwsgitop with this setting:

 ### Stats ### --- ### disabled by default ### To see stats run: uwsgitop /tmp/uwsgi_stats.socket ### uwsgitop must be install (pip install uwsgitop) stats = /tmp/uwsgi_stats.socket 

this will show uwsgi statistics for uwsgitop.

Then I used loader.io to emphasize the application with 350-500 requests per second.

What I found in my previous configuration was that uWSGI workers continued to grow in used memory until the memory was suffocated and then the processor crashed. new workers who needed to reappear also needed a processor that caused some congestion on the servers, which caused nginx timings and closure of these connections.

So, I did some research and configuration modification until I was able to get the installation below, which currently manages ~ 650rps in each instance with a response time of ~ 13 ms, which is great for me.

* My application used (still uses some) disks produced by dat files, some of which were hard to load - I reduced the dependence on the disk to the minimum *

For those who can see this in the future - if you need quick answers - asynchronize everything you can. for example use celery + rabbitmq for any database queries if possible

UWSGI configuration:

 listen = 128 processes = 8 threads = 2 max-requests = 10000 reload-on-as = 4095 reload-mercy = 5 #reload-on-rss = 1024 limit-as = 8192 cpu-affinity = 3 thread-stacksize = 1024 max-fd = 250000 buffer-size = 30000 thunder-lock = true vacuum = true enable-threads = true no-orphans = true die-on-term = true 

Relevant parts of NGINX:

 user nginx; worker_processes 4; worker_rlimit_nofile 20000; thread_pool my_threads threads=16; pid /run/nginx.pid; events { accept_mutex off; # determines how much clients will be served per worker # max clients = worker_connections * worker_processes # max clients is also limited by the number of socket connections available on the system (~64k) worker_connections 19000; # optmized to serve many clients with each thread, essential for linux -- for testing environment use epoll; # accept as many connections as possible, may flood worker connections if set too low -- for testing environment multi_accept on; } http { ... aio threads; sendfile on; sendfile_max_chunk 512k; tcp_nopush on; tcp_nodelay on; keepalive_timeout 5 5; keepalive_requests 0; types_hash_max_size 2048; send_timeout 15; ... } 

Hope this helps!