Geek Answers Handbook

Haproxy and intermittent problems 503

I use Haproxy 1.5.14 in VirtualBox VM (Boot2docker), where the resources it loads periodically show 503 without any real rhyme or reason, this is especially true when starting the cluster.

The cluster looks as follows: 1 interface with ports 80 and 443 by 2, serving static resources and web memory material, respectively.

Haproxy

  • FE (front end for static resources)
  • BE (back end, for network connections)

For example, a static asset serviced by the front end may be

https://local.dev.myproject.com/assets/images/back.png

Despite the fact that the front end server is up and nothing has changed, clicking update and looking at the chrome debugger, I will see many statuses 503 or OK 200 304, but this is not deterministic. It can go from 503 to OK to return to 503, for any asset. When connecting directly to the web server, the assets are returned in order, so there seems to be something with haproxy.

The best I can understand is that the health check is not working properly, and the FE / BE server is temporarily removed from the internal haproxy list, but it does not make sense checking every half second, and I see the haproxy calls being sent, returning to FE / BE ok terminal output window, every half second as expected.

If I look at the haproxy statistics report, I see that the servers periodically come and go, flicker, despite the fact that in the terminal window, haproxy still causes health checks without spaces, and the servers return them as expected.

The attached current haproxy configuration I am using, any help is appreciated.

#--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # http://haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # #log 127.0.0.1 local2 # log /lnl_zoom_shared/log local0 # log /lnl_zoom_shared/log local1 notice chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # SSL #ca-base /etc/ssl #crt-base /etc/ssl ca-base /myproject_shared/SECURITY/local.dev.myproject.com/ crt-base /myproject_shared/SECURITY/local.dev.myproject.com/ tune.ssl.default-dh-param 1024 # turn on stats unix socket #stats socket /var/lib/haproxy/stats # Exposes the stat socket so we can manage the proxy through node.js stats socket /tmp/haproxy.sock level admin #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option http-server-close option http-pretend-keepalive option dontlognull option redispatch option contstats option forwardfor except 127.0.0.0/8 retries 3 backlog 10000 timeout client 25s timeout connect 10s timeout server 25s #long timeoutfor websocket connections timeout tunnel 3600s timeout http-keep-alive 1s timeout http-request 15s timeout queue 30s timeout tarpit 60s default-server inter 3s rise 2 fall 3 #timeout check 10s maxconn 256 #--------------------------------------------------------------------- # Haproxy internal stats on the servers below: password protected #--------------------------------------------------------------------- stats enable stats auth admin:myadminpassword stats uri /haproxy stats refresh 5s #--------------------------------------------------------------------- # #--------------------------------------------------------------------- frontend public # HTTP bind *:80 # Redirect all HTTP traffic to HTTPS redirect scheme https if !{ ssl_fc } # HTTPS # Example with CA certificate bundle # bind :443 ssl crt cert.pem ca-file bundle.crt # Example without CA certification bunch bind *:443 ssl crt /myproject_shared/SECURITY/local.dev.myproject.com/local.dev.myproject.com.pem acl url_static_BE path_beg -i /BE /primus use_backend BE if url_static_BE # FRONT END (aka FE) STATIC ASSETS SERVER # if path is a static asset, assume the front end server to handle it acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js .html .ico use_backend FE if url_static # GIT HOOKS for UPDATE on the git repo changes acl url_githook path_beg -i /gitupdate use_backend HACNTL if url_githook #BACK END (aka BE) default_backend BE #--------------------------------------------------------------------- # controller for haproxy #--------------------------------------------------------------------- backend HACNTL # Tell the backend that this is a secure connection, # even though it getting plain HTTP. option forwardfor http-request add-header X-Forwarded-Proto https if { ssl_fc } server SELF 127.0.0.1:3300 #--------------------------------------------------------------------- # static backend for serving up images, stylesheets and such #--------------------------------------------------------------------- backend FE # Tell the backend that this is a secure connection, # even though it getting plain HTTP. option forwardfor http-request add-header X-Forwarded-Proto https if { ssl_fc } option httpchk GET /haproxy/getstatus option httpchk HEAD / balance roundrobin #server FE1 11.22.33.44:8000 maxconn 256 server FE_172.17.0.2 172.17.0.2:8000 maxconn 256 check inter 500ms #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend BE # Tell the backend that this is a secure connection, # even though it getting plain HTTP. option forwardfor http-request add-header X-Forwarded-Proto https if { ssl_fc } #http-request set-header X-Custom-Header %[url] #http-request set-header Connection upgrade #http-request set-header Upgrade websocket option httpchk GET /haproxy/getstatus cookie SRVNAME insert nocache balance roundrobin server BE_172.17.0.3 172.17.0.3:8888 maxconn 256 cookie BE_172.17.0.3 check inter 500ms
+5
source share
1 answer

although this is not an absolute fix, allowing each server to start in turn, it fixed the problem at the moment. basically adding hibernation between the docker launch command

0
source

More articles:


All Articles