Build and install exim4-daemon-custom on Ubuntu 16.04

Unfortunately I couldn't find a complete and working documentation on how to do this. But this steps succeeded (at least once).

  1. Install build dependencies

    $ sudo apt-get install pbuilder devscripts dpatch grep-dctrl debhelper
  2. Add deb-src sources to apt sources list (if you don't have it yet)

    deb-src xenial-security main universe multiverse restricted
  3. Start the process:

    $ mkdir exim
    $ apt-get source exim4
    $ cd exim4-4.86.2
    $ fakeroot debian/r...

HowTo: Downgrade a Port on FreeBSD


collectd5 in version 5.9.0 is broken and you upgraded it everywhere. But because it's FreeBSD it does not restart running services. After every old version is removed from /var/cache/pkg you restarted collectd and it does not start anymore, you really want to downgrade to 5.8.1 again. But even pkg now has 5.9.0 and you need your own make config setup anyway.


Enter portdowngrade. Install it via pkg install portdowngrade.

You might want go to /root and run the next command should the fol...

Debug Puppet "Error: Failed to apply catalog: undefined method [...] for nil:NilClass"

If you get e.g. this error message when you try to run puppet agent:

Error: Failed to apply catalog: undefined method `strip' for nil:NilClass

you could have a very bad time debugging it. Because there will be no messages in the logs which could help you.

You should always remember puppet agents --trace option, because that's probably the only way how you can track this error. For example if you have an unparsable mail aliases file you will get this:

# puppet agent --test --trace --noop
Info: Using configured environment ...

GitLab: Rails Console Tasks

Sometimes you might need to do some task in GitLab which would be tedious if you'd have to do it via the Browser.

There is also an API which you could maybe use!

You can connect to the GitLab internal Rails with executing gitlab-rails console. It might be slow at times, especially if you use a lot of Tab for complex structures, so be aware of this.

The following code listings are expected to be entered inside the gitlab-rails console.

Remove GitLab Labels from already closed Issues

p = Pr...

HowTo: let puppet agent run from different branch

This is not encouraged because this change is easily forgotten. We recommend using this only if you know what you are doing.

Should you be in need to run puppet from a branch (called environment in puppet land) other than production for a long time and can't or don't want to disable puppet you can do the following:

Add a new section to /etc/puppetlabs/puppet/puppet.conf on the server that should apply the environment. It usually looks like this:

< some stuff >

Add the agent section so it looks like this:


start a systemd service blocked by rate limiting

If a systemd service unit is blocked by rate limiting (see in documentation) you can reset the failcounter via:

systemctl reset-failed $UNIT

Afterwards you can try to start the service again:

systemctl start $UNIT

Correct sequence of ssl cert and intermediate certificates

In case you're wondering, when concat-ing server certificate and intermediate certificates, the server certificate comes first.

RFC 4346:

This is a sequence (chain) of X.509v3 certificates. The sender's
certificate must come first in the list. Each following
certificate must directly certify the one preceding it.

convert PostgreSQL custom dump format to textdump

If you have a PostgreSQL dump in the custom format you can can view the text format dump (plain SQL statements).

PostgreSQL 12 and newer

Write it as a textdump to a file:

pg_restore $dumpname -f name.textdump

Applying parameters

You can apply the same parameters for the output of the textdump as for restoring the dump (for e.g. --clean to drop database objects before recreating them):

pg_restore --clean $dumpname -f $name.textdump

PostgreSQL 11 and older

You can output the contents to STDOUT and redirect i...

System Administration Screencasts

Like Railscasts or Ruby Tapas, but for Linux.

Elasticsearch unassigned replica shards on single node instances

If you have a single node elasticsearch instance and indices with replicas enabled your cluster state will be yellow. If you have replica shards they should be moved to a different node for high availability purposes. With a single node this can't be accomplished. So you either build a ES cluster or you disable the replicas.

Building a cluster is beyond the scope of this card, but we can have a look how we can disable the replicas.

get the unassigned shards

curl -s -XGET 'localhost:9200/_cat/shards?pretty' | grep UNASSIGNED


Manage Gluster Mount on boot

There are two ways within systemd to manage the fact that you need the gluster service active before mounting the mountpoint. Thus in the past the mount triggered before the gluster service was running and didn't mount again.


You can use the x-systemd.autmount feature which mounts the mountpoint as soon as it's accessed

localhost:/shared /gluster/shared glusterfs defaults,_netdev,noauto,x-systemd.automount 0 0

Unit file

This unit file takes care of the mountpoint and makes sure gluster is running before:


fix clamav freshclam failing database updates

If freshclam updates are failing even though the update servers are available and you find error messages like the following in the log you may have outdated or wrong data in freshclams data files:

Thu Mar 29 12:49:52 2018 -> ClamAV update process started at Thu Mar 29 12:49:52 2018
Thu Mar 29 12:49:52 2018 -> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Thu Mar 29 12:49:52 2018 -> WARNING: getpatch: Can't download daily-24428.cdiff from
Thu Mar 29 12:49:52 2018 -> WARNING: getpat...

Capistrano 3: assets:precompile only on one server


This should not be necessary in any case and is only for special cases, for e.g. if the assets directory is linked to a shared storage (for e.g. glusterfs). Please mind that it isn't wise to move your assets directory to a shared storage. The data in this directory should always be reproducible on all appservers when executing a assets precompile (so you don't need to sync this data). Other data should not be stored in the assets directory.
User uploads or files generated by requests should be stored in public/system.


Test websocket connections

You can use wscat:

sudo apt-get install node-ws
wscat -c ws://
connected (press CTRL+C to quit)
> foo

< foo
> bar

< bar

Graphite: reset a users password

If a users password for is unknown one can set it like this:

sudo python /opt/graphite/webapp/graphite/ changepassword ${USERNAME}

dumping and restoring PostgreSQL databases

This card is just about creating simple PostgreSQL dumps. This is no instruction for a backup strategy nor a guide for SQL dump performance optimization.

Read before starting

  • I will assume that all commands will be executed as local postgres user on a database server master. Please mind that you should stop the replication on a slave PostgreSQL server before creating dumps
  • Dumps can get huge, be careful so that the system running a production PostgreSQL won't run out of disk space
  • dumps should always be gzipped to reduce the siz...

controlling multiple PostgreSQL installations on Debian/Ubuntu

This applies only to distribusions based on Debian

If you have multiple Postgres Versions installed on a server and want to start/stop/restore/.. them separately you need to use pg_ctlcluster

Usage: /usr/bin/pg_ctlcluster <version> <cluster> <action>

For e.g.

/usr/bin/pg_ctlcluster 9.5 main start

With systemd you can start/stop the services via:

systemctl <action> postgresql@<version>-<cluster>.service

For e.g.

systemctl start postgresql@10-main.service



The Version of P...

How to remove cloud init from ubuntu

If you're trying to start a cloud-init based Ubuntu VM with KVM you will suffer long boot times and confusing output on the terminal. If you want to get rid of it you need to remove cloud-init.

  1. wait until the VM boots
  2. login
  3. echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
  4. sudo apt-get purge cloud-init
  5. sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
  6. reboot

Disable only

Touch the file /etc/cloud/cloud-init.disabled or use the kernel parameter cloud-init=disabled to disabl...

stopping / restarting libvirt on Ubuntu 16.04 with systemd

Because systemd is the greatest thing ever it's not enough to stop the libvirt service. You need to disable the libvirt socket too.

sudo systemctl stop libvirt-bin
sudo systemctl stop libvirt-bin.socket
sudo systemctl start libvirt-bin

HowTo: verify SSL private key matches SSL certificate

When receiving a new SSL-Certificate for an existing SSL-key it should be checked that they match cryptographically.
Maybe the customer accidentally created a new key and certificate and sent us just the certificate.

It's also possible that the certificate chain is in the wrong order. Make sure that the server certificate is the first. This is also necessary for nginx.

It is recommended to pipe the public keys of both files through an hashing algorithm, to easier identify differences. Instead of comparing the modulus the same syntax appli...

Lenovo Laptop BIOS Update with Linux

You will need to look here and find your own device. Look for the "bootable CD" download, which will give you an ISO file.

Then you need a Perl script from the internet (, I attached it too, because, you know. Internet.) to extract the El Torito boot image from the Lenovo-supplied ISO.

Pro-Tip: Don't clic...

How to unban host with Fail2ban

To see all jails:

fail2ban status

Our $JAIL is usually ssh.

To see which IPs are banned currently:

fail2ban-client status ssh

If you want to unban a host banned by fail2ban on Ubuntu 14.04 or later use this method:

fail2ban-client set $JAIL unbanip $IP
#example with a jail named ssh
fail2ban-client set ssh unbanip

Please don't try to undo the ban by deleting the iptables rule. Fail2ban will recreate the rule if it is missing but fail2ban has still stored the ban.

On 12.04 hosts fail2ban ...