If you want to perform a failover on another haproxy backend server this is the way you should do it:

Gather information

Via hatop

Note: Please mind that the names of frontends / backends / servers are only examples. Mind this when you want to use the shown CLI commands. The path to the haproxy socket may also vary.

Example: We have two MySQL servers with Master-Master replication configured as backends in haproxy.

Your frontend / backend looks like this in hatop:

Copy
>>> mysql-front
FRONTEND                            0 OPEN                                           0             0               0               0                0                0               75                0

>>> mysql-back
mysql1                             50 UP               L7OK                          1             0               0               0                0                0                0                0
mysql2                             50 UP               L7OK                          0             1               0               0                0                0                0                0
BACKEND                            50 UP                                             1             1               0               0                0                0                8                0

mysql1 is preferred to use as mysql2 is configured as backup. So all connections will be sent to mysql1. We want to do some maintenance work on mysql1 so we need to make a failover to mysql2.

Via HTTP statistics page

There is direct access on our load balancers to the stats page.

On every other host you can use a SSH tunnel to access the site:

Copy
ssh -L 18082:127.0.0.1:18082 $fqn

Access the page: http://127.0.0.1:18082/haproxy?stats

Via socat

Get all frontends

Copy
echo "show stat" | sudo socat unix-connect:/var/run/haproxy.stat stdio | cut -d ',' -f 1-2 | grep FRONTEND

Get all backends

Copy
echo "show backend" | sudo socat unix-connect:/var/run/haproxy.stat stdio

Get information about a specific backend

Copy
echo "show servers state $backend" | sudo socat unix-connect:/var/run/haproxy.stat stdio

Example

Copy
echo "show servers state nginx-upstream-upstream_reydix_p-20990" | sudo socat unix-connect:/var/run/haproxy.stat stdio
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
35 nginx-upstream-upstream_reydix_p-20990 1 reydix_p-server-1 10.22.0.149 2 0 50 50 598114 15 3 5 6 0 0 0 - 20990 -
35 nginx-upstream-upstream_reydix_p-20990 2 reydix_p-server-2 10.22.0.150 2 0 50 50 598114 15 3 5 6 0 0 0 - 20990 -
35 nginx-upstream-upstream_reydix_p-20990 3 reydix_p-backup-1 10.22.0.149 2 0 50 50 598114 1 0 2 0 0 0 0 - 20990 -
35 nginx-upstream-upstream_reydix_p-20990 4 reydix_p-backup-2 10.22.0.150 2 0 50 50 47163 1 0 0 0 0 0 0 - 20990 -

This output is not easy to parse. Make sure to check out the documentation, especially for the commands show servers state and show stat.

You're advised to use the HTTP statistics page or hatop to prevent mistakes.

Put server into maintenance

1. Set mysql1 to maintenance. Do this by selecting the backend in hatop and pressing F10. Via socat it works this way:

   Copy
   # disable server <backend>/<server>
   echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
   

   Now no new connections will be opened to mysql1 but to mysql2

2. We still need to close existing connections to mysql1. Do this in the hatop CLI or via socat:

   Copy
   # shutdown sessions server <backend>/<server>
   echo "shutdown sessions server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
   

3. When you want to fail back, re-enable mysql1 and kill the existing connections to mysql2

   Copy
   echo "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
   echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy.stat stdio
   

Recovery a previously down server

When the master server went down you need to manually recover it (or it will take days to recover, you risk a splitbrain too).

1. disable the master server

   Copy
   # disable server <backend>/<server>
   echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
   

2. enable the master server

   Copy
   echo "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
   

3. shutdown sessions on the slave

   Copy
   echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy.stat stdio