I built a set of 3 APIs using Asp.net Web Api 2 , self-hosted using the OWIN in the Azure Cloud service, the role of the worker . The worker role is on the Internet with a custom domain.
Each API has one controller, performs some common operations with a dictionary, table calls and Azure Redis calls . 1 on two, just make one call to Redis and go back around 10 ms . The average call when viewing the entire API code is 150 ms . The answer is a JSON object about 10k in size.
Everything works fine, but I have a problem.
I have approximately 25 connection peaks per second and no more than 2 million requests per day, and I can hardly get the processor below 40% with 3 Azure D2_V2 starts (2 cores, 8 GB of RAM) . I have problems because I spend almost 1.5 thousand dollars a month on Api, which serves only 15-25 calls per second. If I delete or reduce the instance of the instance, the processor reaches 55-60%, Redis and Azure table calls slow down, and the API request takes 3-5 seconds to return.
I tried my best, I thought it could be some bots or a DDos attack, so I installed the Nuget WebApi Throttle package and set a maximum of 1 IP request per second. Nothing changed.
I looked at the whole code configuration to cut the non-optimized parts, but 1 call of 2 just calls redis and returns, and the rest are very clean and simple C #, returning in 150 ms, with two calls to azure table + 1 azure queue set.
The async API controllers are all asynchronous.
I turned on profiling, the central processor is in the main azure process and the Redis Get method, nothing more stupid here, no bottlenecks. I turned on the diagnostics, no errors.
I installed Application Insights , and here I see something strange that cannot say whether it is normal or not.
I see this IP: 13.88.23.0 executes thousands of API requests with request values commonly used in regular requests. Many of them fail. This IP is Azure, why is it calling Api? Some of these requests got stuck for several minutes, I see that from the Application Properties panel there is always the same IP address.
Then I see the remaining logs, dependencies, etc., nothing significant.
Also , what can I do to understand the problem? I don’t think it’s normal to consume so much processor resources for an API with 2 million calls per day, or not?
Is there an additional profiling method that I could use?
Based on your experience, how many API calls should I expect to serve 3 dual-core 8 gigabytes of RAM under normal conditions? (if something is wrong in my configuration)
thanks
UPDATE
I separated the API in two cloud services, 2 in one and 1 in the other. I still see calls from another Microsoft-owned IP address in Insights applications. I believe this is normal, perhaps the Insights application cannot determine the real IP address of the client, since this is the role of the worker and show the internal one.
But the problem is that you need to use so much energy for so many challenges. Any thoughts on this?