38 cards
View
Posted about 3 years ago. Visible to the public. Repeats.

manual haproxy backend failover

If you want to perform a failover on another haproxy backend server this is the way you should do it:

note: please mind that the names of frontends / backends / servers are only examples. Mind this when you want to use the shown CLI commands. The path to the haproxy socket may also vary.

Example: We have two MySQL servers with Master-Master replication configured as backends in haproxy.

Your frontend / backend looks like this in hatop:

Copy
>>> mysql-front FRONTEND 0 OPEN 0 0 0 0 0 0 75 0 >>> mysql-back mysql1 50 UP L7OK 1 0 0 0 0 0 0 0 mysql2 50 UP L7OK 0 1 0 0 0 0 0 0 BACKEND 50 UP 1 1 0 0 0 0 8 0

mysql1 is preferred to use as mysql2 is configured as backup. So all connections will be sent to mysql1. We want to do some maintenance work on mysql1 so we need to make a failover to mysql2.

  1. Set mysql1 to maintenance. Do this by selecting the backend in hatop and pressing F10. Via socat it works this way:

    Copy
    # disable server <backend>/<server> echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio

    Now no new connections will be opened to mysql1 but to mysql2

  2. We still need to close existing connections to mysql1. Do this in the hatop CLI or via socat:

    Copy
    # shutdown sessions server <backend>/<server> echo "shutdown sessions server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
  3. When you want to fail back, re-enable mysql1 and kill the existing connections to mysql2

    Copy
    echo "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy.stat stdio

Recovery a previously down server

When the master server went down you need to manually recover it (or it will take days to recover, you risk a splitbrain too).

  1. disable the master server

    Copy
    # disable server <backend>/<server> echo "disable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
  2. enable the master server

    Copy
    echo "enable server mysql-back/mysql1" | socat unix-connect:/var/run/haproxy.stat stdio
  3. shutdown sessions on the slave

    Copy
    echo "shutdown sessions server mysql-back/mysql2" | socat unix-connect:/var/run/haproxy.stat stdio

Owner of this card:

Avatar
Claus-Theodor Riegg
Last edit:
7 months ago
by Stefan Langenmaier
Keywords:
ops
This website uses cookies to improve usability and analyze traffic.
Accept or learn more