convert PostgreSQL custom dump format to textdump

If you have a PostgreSQL dump in the custom format you can can view the text format dump (plain SQL statements).

PostgreSQL 12 and newer

Write it as a textdump to a file:

pg_restore $dumpname -f name.textdump

Applying parameters

You can apply the same parameters for the output of the textdump as for restoring the dump (for e.g. --clean to drop database objects before recreating them):

pg_restore --clean $dumpname -f $name.textdump

PostgreSQL 11 and older

You can output the contents to STDOUT and redirect i...

Elasticsearch unassigned replica shards on single node instances

If you have a single node elasticsearch instance and indices with replicas enabled your cluster state will be yellow. If you have replica shards they should be moved to a different node for high availability purposes. With a single node this can't be accomplished. So you either build a ES cluster or you disable the replicas.

Building a cluster is beyond the scope of this card, but we can have a look how we can disable the replicas.

get the unassigned shards

curl -s -XGET 'localhost:9200/_cat/shards?pretty' | grep UNASSIGNED

I...

Manage Gluster Mount on boot

There are two ways within systemd to manage the fact that you need the gluster service active before mounting the mountpoint. Thus in the past the mount triggered before the gluster service was running and didn't mount again.

/etc/fstab

You can use the x-systemd.autmount feature which mounts the mountpoint as soon as it's accessed

localhost:/shared /gluster/shared glusterfs defaults,_netdev,noauto,x-systemd.automount 0 0

Unit file

This unit file takes care of the mountpoint and makes sure gluster is running before:

[U...

fix clamav freshclam failing database updates

If freshclam updates are failing even though the update servers are available and you find error messages like the following in the log you may have outdated or wrong data in freshclams data files:

Thu Mar 29 12:49:52 2018 -> ClamAV update process started at Thu Mar 29 12:49:52 2018
Thu Mar 29 12:49:52 2018 -> main.cvd is up to date (version: 58, sigs: 4566249, f-level: 60, builder: sigmgr)
Thu Mar 29 12:49:52 2018 -> WARNING: getpatch: Can't download daily-24428.cdiff from db.local.clamav.net
Thu Mar 29 12:49:52 2018 -> WARNING: getpat...

Capistrano 3: assets:precompile only on one server

Disclaimer

This should not be necessary in any case and is only for special cases, for e.g. if the assets directory is linked to a shared storage (for e.g. glusterfs). Please mind that it isn't wise to move your assets directory to a shared storage. The data in this directory should always be reproducible on all appservers when executing a assets precompile (so you don't need to sync this data). Other data should not be stored in the assets directory.
User uploads or files generated by requests should be stored in public/system.

...

Test websocket connections

You can use wscat:

sudo apt-get install node-ws
#
wscat -c ws://echo.websocket.org
connected (press CTRL+C to quit)
> foo

< foo
> bar

< bar
> 

Graphite: reset a users password

If a users password for https://monitor.makandra.de/ is unknown one can set it like this:

sudo python /opt/graphite/webapp/graphite/manage.py changepassword ${USERNAME}

dumping and restoring PostgreSQL databases

This card is just about creating simple PostgreSQL dumps. This is no instruction for a backup strategy nor a guide for SQL dump performance optimization.

Read before starting

  • I will assume that all commands will be executed as local postgres user on a database server master. Please mind that you should stop the replication on a slave PostgreSQL server before creating dumps
  • Dumps can get huge, be careful so that the system running a production PostgreSQL won't run out of disk space
  • dumps should always be gzipped to reduce the siz...

How to remove cloud init from ubuntu

If you're trying to start a cloud-init based Ubuntu VM with KVM you will suffer long boot times and confusing output on the terminal. If you want to get rid of it you need to remove cloud-init.

  1. wait until the VM boots
  2. login
  3. echo 'datasource_list: [ None ]' | sudo -s tee /etc/cloud/cloud.cfg.d/90_dpkg.cfg
  4. sudo apt-get purge cloud-init
  5. sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
  6. reboot

Disable only

Touch the file /etc/cloud/cloud-init.disabled or use the kernel parameter cloud-init=disabled to disabl...

stopping / restarting libvirt on Ubuntu 16.04 with systemd

Because systemd is the greatest thing ever it's not enough to stop the libvirt service. You need to disable the libvirt socket too.

sudo systemctl stop libvirt-bin
sudo systemctl stop libvirt-bin.socket
sudo systemctl start libvirt-bin

How to unban host with Fail2ban

To see all jails:

fail2ban status

Our $JAIL is usually ssh.

To see which IPs are banned currently:

fail2ban-client status ssh

If you want to unban a host banned by fail2ban on Ubuntu 14.04 or later use this method:

fail2ban-client set $JAIL unbanip $IP
#example with a jail named ssh
fail2ban-client set ssh unbanip 10.0.0.20

Please don't try to undo the ban by deleting the iptables rule. Fail2ban will recreate the rule if it is missing but fail2ban has still stored the ban.

On 12.04 hosts fail2ban ...

create htpasswd entry and print to stdout

Create htpasswd entry and print to stdout:

$ sudo apt install apache2-utils # Optional

$ htpasswd -n $USERNAME
New password: 
Re-type new password: 
$USERNAME:<Password Hash>

puppet variable variable name

example

We want to make the following firewall rule to be applyable to different network interfaces (for e.g. different environments) with just one variable:

firewall { "010-reject-port":
  ensure      => present,
  dport       => [ 80 ],
  destination => $::ipaddress_eth0,
  proto       => 'tcp',
  action      => 'drop',
  iniface     => 'eth0',
}

We can create a $firewall_interface variable and apply it to iniface but how can we ensure that the correct ipaddress factof the corresponding interface is used for `destina...

get debug output for puppetmaster running with passenger

If you need the debug output of the puppetmaster running with passenger you have to uncomment this setting in the config.ru:

#ARGV << "--debug"

If you don't know where your config.ru is, check the Apache DocumentRoot. It's in /path/to/DocumentRoot/../config.ru. For example if your DocumentRoot is /etc/puppet/rack/public/ the config.ru is in /etc/puppet/rack/config.ru.

Exim: investigating frozen messages in the mailqueue

Investigate why mails are frozen

The exim documentation says:

Freezing occurs when a bounce message encounters a permanent failure because the sender address of the original message that caused the bounce is invalid, so the bounce cannot be delivered. This is probably the most common case, but there are also other conditions that cause freezing, and frozen messages are not always bounce messages.

By default, frozen bounce messages will b...

Change / Update SSL certificate for Amazon Elastic Load Balancer with AWS Command Line Interface

  1. Install and configure the AWS Command Line Interface

  2. Show existing certificates to test if the AWS Cli is working:

    $ aws iam list-server-certificates
    {
      "ServerCertificateMetadataList": [
          {
              "Path": "/", 
              "Arn": "arn:aws:iam::5xxxxxxxxxxx:server-certificate/www.example.com-201307-201407", 
              "ServerCertificateId": "AXXXXXXXXXXXXXXXXXXXX", 
              "ServerCertificateName": "www.example.com-201210-201310", 
     ...
    

Fix "A client error (MalformedCertificate) occurred: Invalid Private Key." at AWS SSL Certificate upload

I'm creating certificate requests with this command:

openssl req -new -out www.example.com.csr -keyout www.example.com.key -newkey rsa:2048 -nodes

When I try to upload the certificate to AWS IAM I get this error:

$ aws iam upload-server-certificate --server-certificate-name www.example.com-2013010-2014010 --certificate-body www.example.com.crt --private-key www.example.com.key --certificate-chain www.example.com.ca-bundle 
A client error (MalformedCertificate) occurred: Invalid Public Key Certificate.

That's because o...