Varnish 503 error (cache overload?)

Question

Recently, my company received media coverage, and therefore a lot of traffic comes to our site. We are an online tutoring at http://rayku.com that allows students to get on-demand help with teachers on the HTML5 interactive whiteboard. Tutors are notified via google talk, and both users connect to the board as soon as the tenant clicks the link in the automatic message sent via google talk.

The problem that we are connected with is the varnish when connected to the board. Unfortunately, at random moments, the mentor sometimes cannot reach the board and receives the following error message:

http://grab.by/i65A

error connecting to server: 503 Service unavailable

Error 503 Service unavailable

Service is unavailable

Guru of Meditation:

XID: 1564976246

---

Varnish Caching Server

After clearing the cookies, this problem is resolved (but not the cache). Unfortunately, this problem is difficult to replicate, and I suspect that it is related to Varnish cache overload and does not accept the correct parameters.

Could you help me debug this problem? Many teachers reported this problem, and because of this, many of them are deleted :).

Thank you very much! Donnie

Answer 1

VARNISH 3.0.7: There are various reasons for this. I am currently browsing this on one of my servers and discover that I have several issues. In my case, the "first read error" is due to a failure with slow pages (when receiving images). The way I found this was to use the varnishlog command.

https://www.varnish-cache.org/docs/3.0/tutorial/troubleshooting.html

varnishlog -d -c -m TxStatus:503 24 SessionOpen c 127.0.0.1 39370 :6081 24 ReqStart c 127.0.0.1 39370 657793361 24 RxRequest c GET 24 RxURL c /inventory/part/MYPAGE.HTML 24 RxProtocol c HTTP/1.1 24 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36 24 RxHeader c Referer: https://MYWEBSITE.com/inventory/new?limit=100 24 RxHeader c Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/msword, application/xaml+xml, application/vnd.ms-xpsdocument, application/x-ms-xbap, application/x-ms-application, */* 24 RxHeader c Accept-Encoding: gzip, deflate 24 RxHeader c Accept-Language: en-US 24 RxHeader c Pragma: no-cache 24 RxHeader c Host: MYWEBSITE.com 24 RxHeader c Cookie: PHPSESSID=crp0881ji0qgfdqgtde10ovh72; laravel_session=eyJpdiI6ImIrRVVGRFBWdHErdk85cU9oQThqemc9PSIsInZhbHVlIjoid0g2Zk56elVybUdlVkVQb0dCdzlVVVBhMWVmVlwvZnRPOFlEOVwvQjRWOW5ITUVyNUFCMGZyRUI5aDlGSVBoWWpsR0Z3NGxZK2NjQ2Z6Q01Lam5IWVdcL3c9PSIsIm1hYyI6ImZlZ .... .... .... 

It seems to me that on MYSITE.com it was not possible to get the URI above. In this case, this is expected, as there is a problem with the page.

Varnish has some meanings to help mitigate this and other problems. The following to pay attention to this:

  .connect_timeout = 1s; # Wait a maximum of 1s for backend connection (Apache, Nginx, etc...) .first_byte_timeout = 120s; # Wait a maximum of 120s for the first byte to come from your backend .between_bytes_timeout = 2s; # Wait a maximum of 2s between each bytes sent 

If you see a “first byte error” in the log, you can solve it with setting first_byte_timeout (in many cases this works). Not in my case, but in the case above. What I'm going to experiment with is the timeout of the response from the connect_timeout backend. If you receive unhealthy messages, you will need to edit the .probe parameters in the varnish configuration file. Another simple problem is that the ports are not configured correctly or the profile file on / httpcheck or something is actually inaccessible (make sure that you can contact it using curl).

PS> I answer this topic because I noticed that the answer is not enough, I hope that my investigation helps someone at least a little.

More details here: https://varnish-cache.org/tips/varnishlog/fetcherror.html

Answer 2

This seems to have been resolved thanks to the great guys at Rackspace! Turns out this is a cache issue, and by directing Varnish to only cache image, js and css files, this solved my problem.