ELB, RDS mysql, EC2, NGINX, where to look for concurrency performance issues

Question

Setup: EC2 servers autoscale behind ELB, connecting to mysql RDS database, all static files that are served from the cloud interface.

I am running nginx as a web server on EC2 servers, keepalive is set to 20, workflows 4 ,. Codeigniter is the backend and use of codeigniter sessions.

I had a lot of tests to try to test performance, siege, apache test, blitz.io.

I am testing two specific pages, the first performance is very good, it uses codeigniter sessions, so it gets into the database for reading and updating the ci_sessions database. The second page is a problem that I am having problems with, it launches a request with several joins, which are filled in after about 0.4 seconds by one user. This query is optimized and I am using InnoDB tables. In apache test with c10 and n1000, 100% of requests are returned within 634 ms.

When I start concurrent users> 200, I start to run into problems. Adding more EC2 servers does not help, the CPU is about 50%. Monitoring the RDS database also shows that CPU and memory consumption is less than 70%, and average DB connections are 35.

Performance improved by moving to a large RDS instance and large EC2 instances, which makes me wonder if there will be input / output.

If I load a server outside of ELB during load tests and click on this page, it returns in less than a second, but if I start another server in ELB, it will remain up to 4 or 5 seconds. This suggests that I do not overload RDS.

I tried to slowly build up ELBs with 5-minute bursts, and that didn't seem to help.

I am wondering where to look for the next problem, be it some kind of I / O problem or something else, because the RDS and EC2 servers do not seem to be pushing their capabilities. Any suggestions or ideas to look at further will be greatly appreciated.

Answer 1

Good. Well, this is a very broad subject, as you know. But I will try and help.

• ELBs are usually not very good at scaling packages. After talking with Amazon engineers about this, I realized that in reality they would not scale ELBs on queues, because this is not possible. You must have a consistent load over time to increase ELB. Because of this, I switched to haproxy. In addition to the ELB, which is not scalable under packet load, it also uses CNAME to look for DNS, which will also affect your performance. Therefore, if you plan to download the package frequently or require a DNS lookup, it might be best to exit ELB.

• RDS is a black box. Well, that’s not entirely true, but I generally avoid RDS if I don’t know that the backend is a simple setup that scales easily. Having said that, RDS really helps with scaling, but I would omit the backend and ensure that your query starts quickly. Run it on a regular MySQL instance and see if it is subsecond. In my experience, when you say that the request is “optimized”, it doesn’t really mean that there is no other way to make it more “optimized” if you catch my drift.