500 Backup

Posted Over 6 years ago. Visible to the public.

Backups are stored in a physically separated environment from their origin to protect against physical dangers such as a fire. Backup data is generally encrypted to ensure confidentiality and integrity.

Access to backups is restricted and available to authorized personell upon legitimate requests.

Backups are created in an append-only fashion. Additionally, the underlying storage system is taking snapshots that are only available read-only, to ensure backup integrity.


Regular Intervals

Backups in the form of database dumps of our relational databases happen every day (usually between 12 AM and 1 AM CET/CEST). The backups will be created on the database replica server to prevent load impacts on the primary instance.
The dumps will be stored on our dedicated backup storage and kept there for 7 days (previously 30 days). Database dumps are encrypted with individual cryptographic keys for customers.


Additionally and upon request we can provide Point-In-Time-Recoverability for dedicated databases. For our shared PostgreSQL cluster PITR is available by default. Restore Points are available up to 7 days in the past.


We do not backup Redis data but let it write its data to disk in certain intervals determined by amount of data changed and a time interval. Writing data to disk is not a backup. It's not copied anywhere. It's not incremental. It's not rotated. Disk persistence in Redis is needed for replication.

Application an Filesystem data

In our datacenter at noris, we use borgbackup Show archive.org snapshot . With that, the borg client is in charge of sending the data to the backup server. This way we can fully encrypt all backups on a per-customer basis.

All in all, the following is backed up:

  • Your application deployment, with releases (/var/www)
  • Your deploy users home (/home)
    • We save all your cronjobs in your home directory (in ~/.crontab-dump)
  • The shared data in the GlusterFS (/gluster/shared)
  • all files in /etc

We do this, in order to restore functionality completely without outside (e.g. a developer of the customer) interaction to any point within the last 7 days.

Backup restore test and monitoring

We frequently check that our backup is working as expected and we are able to restore backups:

  • every week database dumps of every type (PostgreSQL, MySQL and every version) are randomly selected and are restored to a local database by an operations engineer
  • every week we verify that new files are included in the filesystem backups
  • additionally we monitor if the sync for the filesystem backup runs every day
  • we get notified if backup jobs fail
  • there is a continuous monitoring if PITR backup is receiving new data
  • every week we also restore a PITR backup
Thomas Eisenbarth
Last edit
22 days ago
Florian Heinle
Source code in this card is licensed under the MIT License.
Posted by Thomas Eisenbarth to opscomplete (2017-03-31 11:33)