52 cards
Posted about 5 years ago. Visible to the public. Repeats.

manual haproxy backend failover

If you want to perform a failover on another haproxy backend server this is the way you should do it:

Gather information

Via hatop

Note: Please mind that the names of frontends / backends / servers are only examples. Mind this when you want to use the shown CLI commands. The path to the haproxy socket may also vary.

Example: We have two MySQL servers with Master-Master replication configured as backends in haproxy.

Your frontend / backend looks like this in hatop:

Copy
>>> mysql-front FRONTEND 0 OPEN 0 0 0 0 0 0 75 0 >>> mysql-back mysql1 50 UP L7OK 1 0 0 0 0 0 0 0 mysql2 50 UP L7OK 0 1 0 0 0 0 0 0 BACKEND 50 UP 1 1 0 0 0 0 8 0

mysql1 is preferred to use as mysql2 is configured as backup. So all connections will be sent to mysql1. We want to do some maintenance work on mysql1 so we need to make a failover to mysql2.

Via HTTP statistics page

There is direct access on our load balancers to the stats page.

On every other host you can use a SSH tunnel to access the site:

Copy
ssh -L 18082:127.0.0.1:18082 $fqn

Access the page: http://127.0.0.1:18082/haproxy?stats

Via socat

Get all frontends

Copy
echo "show stat" | sudo socat unix-connect:/var/run/haproxy.stat stdio | cut -d ',' -f 1-2 | grep FRONTEND

Get all backends

Copy
echo "show backend" | sudo socat unix-connect:/var/run/haproxy.stat stdio

Get information about a specific backend

Copy
echo "show servers state $backend" | sudo socat unix-connect:/var/run/haproxy.stat stdio

Example

Copy
echo "show servers state nginx-upstream-upstream_reydix_p-20990" | sudo socat unix-connect:/var/run/haproxy.stat stdio # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord 35 nginx-upstream-upstream_reydix_p-20990 1 reydix_p-server-1 10.22.0.149 2 0 50 50 598114 15 3 5 6 0 0 0 - 20990 - 35 nginx-upstream-upstream_reydix_p-20990 2 reydix_p-server-2 10.22.0.150 2 0 50 50 598114 15 3 5 6 0 0 0 - 20990 - 35 nginx-upstream-upstream_reydix_p-20990 3 reydix_p-backup-1 10.22.0.149 2 0 50 50 598114 1 0 2 0 0 0 0 - 20990 - 35 nginx-upstream-upstream_reydix_p-20990 4 reydix_p-backup-2 10.22.0.150 2 0 50 50 47163 1 0 0 0 0 0 0 - 20990 -

This output is not easy to parse. Make sure to check out the documentation, especially for the commands show servers state and show stat.

You're advised to use the HTTP statistics page or hatop to prevent mistakes.

Put server into maintenance

  1. Set mysql1 to maintenance. Do this by selecting the backend in hatop and pressing F10. Via socat it works this way:

    Copy
    # disable server <backend>/<server> echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio

    Now no new connections will be opened to mysql1 but to mysql2

  2. We still need to close existing connections to mysql1. Do this in the hatop CLI or via socat:

    Copy
    # shutdown sessions server <backend>/<server> echo "shutdown sessions server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
  3. When you want to fail back, re-enable mysql1 and kill the existing connections to mysql2

    Copy
    echo "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy.stat stdio

Recovery a previously down server

When the master server went down you need to manually recover it (or it will take days to recover, you risk a splitbrain too).

  1. disable the master server

    Copy
    # disable server <backend>/<server> echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
  2. enable the master server

    Copy
    echo "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
  3. shutdown sessions on the slave

    Copy
    echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy.stat stdio

Owner of this card:

Avatar
Claus-Theodor Riegg
Last edit:
2 months ago
by Andreas Vöst
Keywords:
ops
This website uses short-lived cookies to improve usability.
Accept or learn more