We use the Spring cloud in our project. We have several microservices, each of which has its own .yml file.
Below properties only on zuul server
hystrix.command.default.execution.isolation.thread.timeoutInMilliseconds: 60000 ribbon: ConnectTimeout: 3000 ReadTimeout: 60000
Test 1:
Account Service:
This service is what I call to check the timeout, and I call the request through zuul, i.e. using port 8006.
@RequestMapping(value = "/accountholders/{cardHolderId}/accounts", produces = "application/json; charset=utf-8", method = RequestMethod.GET) @ResponseBody public AllAccountsVO getAccounts(@PathVariable("cardHolderId") final String cardHolderId, @RequestHeader("userContextId") final String userContextId, @RequestParam final MultiValueMap<String, String> allRequestParams, final HttpServletRequest request) { return iAccountService.getCardHolderAccountsInfo(cardHolderId, userContextId, request, allRequestParams, ApplicationConstants.ACCOUNTHOLDER); }
The above service internally calls below using Spring RestTemplate. I started testing by adding a sleep time of 5000 ms, as shown below, to the Association Service and made a request for Account Service (call getAccounts).
Support service:
@RequestMapping(value = "/internal/userassociationstatus", produces = "application/json; charset=utf-8", consumes = "application/json", method = RequestMethod.GET) @ResponseBody public UserAssociationStatusVO getUserAssociationStatus(@RequestParam final Map<String, String> allRequestParams) { try { Thread.sleep(5000); } catch (InterruptedException e) {
Below is the error I received in the association service
org.apache.catalina.connector.ClientAbortException: java.io.IOException: An established connection was aborted by the software in your host machine at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393) ~[tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426) ~[tomcat-embed-core-8.0.30.jar:8.0.30] at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342) ~[tomcat-embed-core-8.0.30.jar:8.0.30]
The following is the error I received in the Account Service
org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://USERASSOCIATIONS-V1/user/v1/internal/userassociationstatus?cardholderid=123&usercontextid=222&role=ACCOUNT": com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out; nested exception is java.io.IOException: com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:607) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:557) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE] at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:475) ~[spring-web-4.2.4.RELEASE.jar:4.2.4.RELEASE]
If I save the sleep time as 4500, it gives me the answer, but if is> = 4800, it throws the above exception. I think this is not due to Ribbon timeouts, but something else. Any specific reason for the above exception after a certain point.
Test 2
Then I tried to save the 75000 ms sleep time in the Account Service directly and delete the support timeout.
@RequestMapping(value = "/accountholders/{cardHolderId}/accounts", produces = "application/json; charset=utf-8", method = RequestMethod.GET) @ResponseBody public AllAccountsVO getAccounts(@PathVariable("cardHolderId") final String cardHolderId, @RequestHeader("userContextId") final String userContextId, @RequestParam final MultiValueMap<String, String> allRequestParams, final HttpServletRequest request) { try { Thread.sleep(75000); } catch (InterruptedException ex) {
In this case, I got an "exception": "com.netflix.zuul.exception.ZuulException",
And in my APIGateway (Zuul application) log I see an error below.
com.netflix.zuul.exception.ZuulException: Forwarding error at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.forward(RibbonRoutingFilter.java:134) ~[spring-cloud-netflix-core-1.1.0.M5.jar:1.1.0.M5] at org.springframework.cloud.netflix.zuul.filters.route.RibbonRoutingFilter.run(RibbonRoutingFilter.java:76) ~[spring-cloud-netflix-core-1.1.0.M5.jar:1.1.0.M5] at com.netflix.zuul.ZuulFilter.runFilter(ZuulFilter.java:112) ~[zuul-core-1.1.0.jar:1.1.0] at com.netflix.zuul.FilterProcessor.processZuulFilter(FilterProcessor.java:197) ~[zuul-core-1.1.0.jar:1.1.0] Caused by: com.netflix.hystrix.exception.HystrixRuntimeException: useraccounts-v1RibbonCommand timed-out and no fallback available. at com.netflix.hystrix.AbstractCommand$16.call(AbstractCommand.java:806) ~[hystrix-core-1.4.23.jar:1.4.23] at com.netflix.hystrix.AbstractCommand$16.call(AbstractCommand.java:790) ~[hystrix-core-1.4.23.jar:1.4.23] at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$1.onError(OperatorOnErrorResumeNextViaFunction.java:99) ~[rxjava-1.0.14.jar:1.0.14] at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:70) ~[rxjava-1.0.14.jar:1.0.14]
I think this has nothing to do with Ribbon ConnectTimeout or ReadTimeout. This error occurs due to the "execute.isolation.thread.timeoutInMilliseconds: 60000" property. I also reduced this property to 10,000 ms to test the behavior and got the same exception if the sleep time is longer (for example: 12000).
I want to understand Ribbon ConnectTimeout and Read-timeout vs Hystrix timeout and how to test tape timeouts in my application. Also, if I need different timeouts for different microservices, do I save these properties in the corresponding .yml files? Any thoughts?
I am trying to create a document that will be used by my team so that the developer can know how these timeout options work in the Spring cloud.
(This is a long description, but for clarity, I had to write in detail)