controlling multiple PostgreSQL installations on Debian/Ubuntu

Hint

This applies only to distributions based on Debian.

If you have multiple Postgres Versions installed on a server and want to start/stop/restore/.. them separately you need to use pg_ctlcluster

Usage: /usr/bin/pg_ctlcluster <version> <cluster> <action>

For e.g.

/usr/bin/pg_ctlcluster 9.5 main start

With systemd you can start/stop the services via:

systemctl <action> postgresql@<version>-<cluster>.service

For e.g.

systemctl start postgresql@10-main.service

Details

<version>

The Vers...

HowTo apply Test Driven Development to Container Images

Apply Test Driven Development(TDD) to the process of building container images by defining test before writing code and automate the testing process. Iterate through the TDD cycle while developing and running the tests later in continuous integration to ensure robust and reliable container images.

Installation

We create a Gemfile for installing all required gems.

# Gemfile
gem 'docker-api'
gem 'serverspec'
gem 'rspec'

Then we install them

bundle install

Preparation

Create the directory where the Tests a...

HowTo Generate Nagios Config with puppet fast

Creating Nagios Config with puppet

Let's have a look at the classic way of managing Nagios configuration with exported Puppet resources.
Here is a good article about the topic written by Eric Holzbach.

First we export the checks on the nodes we want to monitor:

@@nagios_service { “check_unicorns-${::hostname}”:
  host_name           => $::hostname,
  service_description => ‘Unicorn Status’,
  check_command       => ‘check_unicorn-${::hostname}“,
...

How to Protect container images with production tag from ECR lifecycle rules

If you want to automatically delete old container images from your Elastic Container Registry, the solution is a quite simple ECR Lifecycle Rule that deletes images e.g. 7 days after they have been pushed to the registry.

If you however want to always keep the image tagged production, possibly because that is a floating tag always associated with the image currently deployed to production, the situation suddenly is not so simple any more. ECR does not provide a keep action in their lifecycle rules, only "expire". The logic here is t...

Find dead code in puppet with puppet-ghostbuster

There is a gem puppet-ghostbuster which can help you find dead code in puppet projects.

To use it, (as of now) it's best to get the latest version from git, as the last release is not yet on rubygems.org.

# Gemfile
gem 'puppet-ghostbuster', git: 'https://github.com/voxpupuli/puppet-ghostbuster.git', ref: 'XXXXX'

After that you can use puppet-lint to find dead code in your project:

# You probably need to set some ENV Variables, see https://github.com/voxpupuli/puppet-ghostbu...

Change bwlimit for Proxmox storages

It's not possible to change the bwlimit of Proxmox storages via the Web-UI (at least in Proxmox 7.x).

To change the bwlimit you need to use the pvesm command and specify the limit for the desired category.

For example to increase the bwlimit for migrations:

pvesm set $NAME_OF_STORAGE --bwlimit migration=$INTEGER_LIMIT
# example
pvesm set ssd --bwlimit migration=500000

These are the different categories:

  • default
  • clone
  • migration
  • move
  • restore

Repair broken etcd node

If one etcd node is no longer a member of the remaining etcd cluster or fails to connect you need to remove it from the cluster and then add it again:

  1. Stop etcd on the broken node : sudo stop etcd
  2. delete the data on the broken node sudo rm -r /var/lib/etcd/data/*
  3. delete the wal data on the broken node: sudo rm -r /var/lib/etcd/wal/*
  4. Follow the instructions for etcd runtime-configuration, remove the broken node from the cluster, then re-add it again and update t...

Barman recovery fails with missing history file

When restoring a barman PITR backup you may encounter this error:

Copying required WAL segments.
EXCEPTION: {'ret': 2, 'err': '/bin/sh: 1: cannot open /var/lib/barman/foopostgres/wals/00000007.history: No such file\n', 'out': ''}

The reason is that the barman backups xlog.db file contains a history file which is no longer present in the wals directory of your backup. The most likely reason is that someone deleted this file in the past. If you do not need this file for restoring your current backup (maybe because it's very old a...

Terraform/Terragrunt: Hide Refreshing State..., Reading..., Read complete after... output

Terraform (and terragrunt) gives a lot of output when running plan or apply, outputting a lot of progress information when refreshing the state before creating the plan. They have reasons for this but it doesn't look like there is going to be any silent flags any time soon. Especially when running larger environments or running multiple modules with terragrunt, this will clog the output on your terminal or in your CI jobs, possibly making the signal hard to find among the noise.

H...

ACM certificate not showing up in CloudFront

Preface

Before you continue, ensure that you've created your certificate in the region us-east-1 (N. Virginia). Otherwise the certificate is not available for CloudFront.

The issue

At some point in time you may be confronted with the following issue:

  • you've requested an SSL certificate via ACM
  • the validation was successful
  • you try to add the freshly issued ACM certificate to a CloudFront configuration via AWS console
  • the certificate is not selectable from the dropdown in the distribution configuration

Fixing the is...

Pay attention to trailing slashes when using rsync

When you synchronize directories with rsync you have to pay attention to use (or not use) trailing /.

Example:

# without trailing slash
$ mkdir -p a/foo/bar/baz
$ mkdir b
$ rsync -a a b
$ find b
b
b/a
b/a/foo
b/a/foo/bar
b/a/foo/bar/baz

# with trailing slash
$ mkdir -p a/foo/bar/baz
$ mkdir b
$ rsync -a a/ b/
$ find b
b
b/foo
b/foo/bar
b/foo/bar/baz

Merge two PostgreSQL dumps

Attention

This is an edge-case. You probably don't want to mix different database dumps. It also requires that the mixed tables do not share relations to other database objects.

Scenario

  • You've got two database dumps:
    • dump_a.pgdump
    • dump_b.pgdump
  • Both dumps include these tables:
    • foobar
    • barfoo
    • foobaz
  • You want to create a dump with:
    • dump_a.foobar
    • dump_b.barfoo
    • dump_a.foobaz

Invalid and incomplete restore

These commands might be obvious but won't actually work:

pg_re...

Find unmaintained packages with apt-forktracer

If you use third party APT sources you might end up with unmaintained packages after removing the external source or performing a dist-upgrade. The reason for this is how external sources overwrite official package versions.

apt-forktracer helps you to identify such packages:

APT will not warn you when newer versions of official packages (point releases, security updates) will appear in the stable release. This means you may miss some important change.

Example output

This is the output of `apt...

Why Apple Silicon MacBooks Can Only Use One External Display Through A Dock

It's quite confusing how many external displays are usable with a MacBook that uses an M1 or M2 Chip. Documentation on the Internet is sparse and not even the Geniuses at the Genius Bar are 100% sure. We found out some things and can explain the backgrounds.

tl;dr: when using a dock for your MacBook, using more than one external display is only possible with workarounds.

Docking stations

Whenever you're considering using more than 1 external display, it makes sense to consider using a Docking Station for your Laptop of some sort. Af...

Intel Microcode Versions and Updates

Intel CPUs receive updates, including security relevant upgrades, through 2 channels:

  • Firmware/UEFI BIOS updates can also update the microcode in CPUs. This is the preferred and persistent way
  • the intel-microcode package can patch the microcode in the CPU at boot time, given the kernel is cooperating. This patch is ephemeral and will be lost after a processor hard-reset or power-off.

Is it important to install microcode updates?

Yes. From the README.Debian.gz in the intel-microcode package:

While most of the microcode up...

Security considerations in MIME-Type configuration

When you are using the default MIME-Type configuration and your application allows uploading files, it can be a security issue.

Example:

A user uploads a file with HTML/JavaScript content using no file extension.

In the Apache default configuration if you access the file it will have no Content-Type. Some browsers will guess/autodetect it as HTML and now you are vulnerable to XSS.

To prevent this, you can set a default Content-Type (e.g. plain/text or application/octet-stream).

manual haproxy backend failover

If you want to perform a failover on another haproxy backend server this is the way you should do it:

Gather information

Via hatop

Note: Please mind that the names of frontends / backends / servers are only examples. Mind this when you want to use the shown CLI commands. The path to the haproxy socket may also vary.

Example: We have two MySQL servers with Master-Master replication configured as backends in haproxy.

Your frontend / backend looks like this in hatop:

| NAME | W | STATUS | CHECK | ACT | BCK | QCUR...

HowTo: Easier selecting and copying of text in tmux

When using tmux, selecting and copying multiple lines of text can be a hassle, especially when using splits (highlighting lines will cross pane borders, copying contents from the other pane too) and when the user wishes to copy (thus, select) lines that have already scrolled out of the viewport in the current pane.

One idea would be to enable mouse mode in tmux, which makes the selection tmux aware. However, this will no longer populate the selection clipboard of the graphical environment, limiting copying and pasting to the same tmux ins...

Testing AWS SNS Subscription Filter Policies

A convenient way to test SNS Subscription Filter Policies is using an email address as the subscription endpoint. However, for this to work when the filter is applied to the MessageBody the protocol must be set to email-json. Using the email protocol will change the message format and filtering will not work as expected.

SSH add/remove port forwarding to active session

It is possible to manipulate the forwarded ports of an established interactive SSH session.
This is done by opening the command line with the escape character ~C. This will open a ssh> prompt. If it doesn't, make sure it's the first character in a line, i.e. press enter first.

You have following commands at your disposal

-L[bind_address:]port:host:hostport    Request local forward
-R[bind_address:]port:host:hostport    Request remote forward
-D[bind_address:]port                  Request dynamic forward
-KL[bind_address:]port    ...

HowTo: List packages in an apt repository

When you're using a third party repository on your Ubuntu or Debian system and need to find out which packages are available in this repo and which versions, you can consult the local apt cache.

In /var/lib/apt/lists you'll find a lot of *_Packages files that list the contents of apt repositories. Most repositories will have one file for the i386 architecture and one for the amd64 architecture. Make sure to pick the correct one.

To list the contents of the repository, view the respective files or, for a short summary:

/...