manual haproxy backend failover

If you want to perform a failover on another haproxy backend server this is the way you should do it:

Gather information

Via hatop

Note: Please mind that the names of frontends / backends / servers are only examples. Mind this when you want to use the shown CLI commands. The path to the haproxy socket may also vary.

Example: We have two MySQL servers with primary-primary replication configured as backends in haproxy.

Your frontend / backend looks like this in hatop:

mysql1 is preferred to use as mysql2 is configured as backup. So all connections will be sent to mysql1. We want to do some maintenance work on mysql1 so we need to make a failover to mysql2.

Via HTTP statistics page

There is direct access on our load balancers to the stats page.

On every other host you can use a SSH tunnel to access the site:

Copyssh -L 18082:127.0.0.1:18082 $fqn

Access the page: http://127.0.0.1:18082/haproxy?stats Show archive.org snapshot

Via socat

Get all frontends

Copyecho "show stat" | sudo socat unix-connect:/var/run/haproxy/haproxy.stat stdio | cut -d ',' -f 1-2 | grep FRONTEND

Get all backends

Copyecho "show backend" | sudo socat unix-connect:/var/run/haproxy/haproxy.stat stdio

Get information about a specific backend

Copyecho "show servers state $backend" | sudo socat unix-connect:/var/run/haproxy/haproxy.stat stdio

Example

Copyecho "show servers state nginx-upstream" | sudo socat unix-connect:/var/run/haproxy/haproxy.stat stdio
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
35 nginx-upstream 1 nginx-server-1 10.10.0.1 2 0 50 50 598114 15 3 5 6 0 0 0 - 12345 -
35 nginx-upstream 2 nginx-server-2 10.10.0.2 2 0 50 50 598114 15 3 5 6 0 0 0 - 122345 -
35 nginx-upstream 3 nginx-backup-1 10.10.0.1 2 0 50 50 598114 1 0 2 0 0 0 0 - 12345 -
35 nginx-upstream 4 nginx-backup-2 10.10.0.2 2 0 50 50 47163 1 0 0 0 0 0 0 - 12345 -

This output is not easy to parse. Make sure to check out the documentation Show archive.org snapshot , especially for the commands show servers state Show archive.org snapshot and show stat Show archive.org snapshot .

You're advised to use the HTTP statistics page or hatop to prevent mistakes.

Put server into maintenance

1. Set mysql1 to maintenance. Do this by selecting the backend in hatop and pressing F10. Via socat it works this way:

   Copy# disable server <backend>/<server>
   echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio
   

   Now no new connections will be opened to mysql1 but to mysql2

2. We still need to close existing connections to mysql1. Do this in the hatop CLI or via socat:

   Copy# shutdown sessions server <backend>/<server>
   echo "shutdown sessions server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio
   

3. When you want to fail back, re-enable mysql1 and kill the existing connections to mysql2

   Copyecho "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio
   echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio
   

Recovery a previously down server

When the primary server went down you need to manually recover it (or it will take days to recover, you risk a splitbrain too).

1. disable the primary server

   Copy# disable server <backend>/<server>
   echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio
   

2. enable the primary server

   Copyecho "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio
   

3. shutdown sessions on the replica

   Copyecho "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy/haproxy.stat stdio