Barman recovery fails with missing history file

When restoring a barman PITR backup you may encounter this error:

Copying required WAL segments.
EXCEPTION: {'ret': 2, 'err': '/bin/sh: 1: cannot open /var/lib/barman/foopostgres/wals/00000007.history: No such file\n', 'out': ''}

The reason is that the barman backups xlog.db file contains a history file which is no longer present in the wals directory of your backup. The most likely reason is that someone deleted this file in the past. If you do not need this file for restoring your current backup (maybe because it's very old a...

Terraform/Terragrunt: Hide Refreshing State..., Reading..., Read complete after... output

Terraform (and terragrunt) gives a lot of output when running plan or apply, outputting a lot of progress information when refreshing the state before creating the plan. They have reasons for this but it doesn't look like there is going to be any silent flags any time soon. Especially when running larger environments or running multiple modules with terragrunt, this will clog the output on your terminal or in your CI jobs, possibly making the signal hard to find among the noise.

H...

Don't use puppet `exec` type without `cwd` and `user` parameter

  1. Don't use exec without user parameter

    If you use exec without user parameter, the command will get executed as root. You mostly don't want this.

  2. There is a difference in the env variables of the exec if you run puppet manually or if the daemon runs.

  3. Never ever use exec without cwd parameter

    If you use exec without cwd parameter, the command get executed in the cwd of your puppet run. This can cause problems if you run the puppet agent manually.

    Example:

    # exec resource:
    e...
    

ACM certificate not showing up in CloudFront

Preface

Before you continue, ensure that you've created your certificate in the region us-east-1 (N. Virginia). Otherwise the certificate is not available for CloudFront.

The issue

At some point in time you may be confronted with the following issue:

  • you've requested an SSL certificate via ACM
  • the validation was successful
  • you try to add the freshly issued ACM certificate to a CloudFront configuration via AWS console
  • the certificate is not selectable from the dropdown in the distribution configuration

Fixing the is...

Pay attention to trailing slashes when using rsync

When you synchronize directories with rsync you have to pay attention to use (or not use) trailing /.

Example:

# without trailing slash
$ mkdir -p a/foo/bar/baz
$ mkdir b
$ rsync -a a b
$ find b
b
b/a
b/a/foo
b/a/foo/bar
b/a/foo/bar/baz

# with trailing slash
$ mkdir -p a/foo/bar/baz
$ mkdir b
$ rsync -a a/ b/
$ find b
b
b/foo
b/foo/bar
b/foo/bar/baz

Merge two PostgreSQL dumps

Attention

This is an edge-case. You probably don't want to mix different database dumps. It also requires that the mixed tables do not share relations to other database objects.

Scenario

  • You've got two database dumps:
    • dump_a.pgdump
    • dump_b.pgdump
  • Both dumps include these tables:
    • foobar
    • barfoo
    • foobaz
  • You want to create a dump with:
    • dump_a.foobar
    • dump_b.barfoo
    • dump_a.foobaz

Invalid and incomplete restore

These commands might be obvious but won't actually work:

pg_re...

Find unmaintained packages with apt-forktracer

If you use third party APT sources you might end up with unmaintained packages after removing the external source or performing a dist-upgrade. The reason for this is how external sources overwrite official package versions.

apt-forktracer helps you to identify such packages:

APT will not warn you when newer versions of official packages (point releases, security updates) will appear in the stable release. This means you may miss some important change.

Example output

This is the output of `apt...

Why Apple Silicon MacBooks Can Only Use One External Display Through A Dock

It's quite confusing how many external displays are usable with a MacBook that uses an M1 or M2 Chip. Documentation on the Internet is sparse and not even the Geniuses at the Genius Bar are 100% sure. We found out some things and can explain the backgrounds.

tl;dr: when using a dock for your MacBook, using more than one external display is only possible with workarounds.

Docking stations

Whenever you're considering using more than 1 external display, it makes sense to consider using a Docking Station for your Laptop of some sort. Af...

Intel Microcode Versions and Updates

Intel CPUs receive updates, including security relevant upgrades, through 2 channels:

  • Firmware/UEFI BIOS updates can also update the microcode in CPUs. This is the preferred and persistent way
  • the intel-microcode package can patch the microcode in the CPU at boot time, given the kernel is cooperating. This patch is ephemeral and will be lost after a processor hard-reset or power-off.

Is it important to install microcode updates?

Yes. From the README.Debian.gz in the intel-microcode package:

While most of the microcode up...

Security considerations in MIME-Type configuration

When you are using the default MIME-Type configuration and your application allows uploading files, it can be a security issue.

Example:

A user uploads a file with HTML/JavaScript content using no file extension.

In the Apache default configuration if you access the file it will have no Content-Type. Some browsers will guess/autodetect it as HTML and now you are vulnerable to XSS.

To prevent this, you can set a default Content-Type (e.g. plain/text or application/octet-stream).

manual haproxy backend failover

If you want to perform a failover on another haproxy backend server this is the way you should do it:

Gather information

Via hatop

Note: Please mind that the names of frontends / backends / servers are only examples. Mind this when you want to use the shown CLI commands. The path to the haproxy socket may also vary.

Example: We have two MySQL servers with Master-Master replication configured as backends in haproxy.

Your frontend / backend looks like this in hatop:

>>> mysql-front
FRONTEND                            0 O...

HowTo: Easier selecting and copying of text in tmux

When using tmux, selecting and copying multiple lines of text can be a hassle, especially when using splits (highlighting lines will cross pane borders, copying contents from the other pane too) and when the user wishes to copy (thus, select) lines that have already scrolled out of the viewport in the current pane.

One idea would be to enable mouse mode in tmux, which makes the selection tmux aware. However, this will no longer populate the selection clipboard of the graphical environment, limiting copying and pasting to the same tmux ins...

Testing AWS SNS Subscription Filter Policies

A convenient way to test SNS Subscription Filter Policies is using an email address as the subscription endpoint. However, for this to work when the filter is applied to the MessageBody the protocol must be set to email-json. Using the email protocol will change the message format and filtering will not work as expected.

SSH add/remove port forwarding to active session

It is possible to manipulate the forwarded ports of an established interactive SSH session.
This is done by opening the command line with the escape character ~C. This will open a ssh> prompt. If it doesn't, make sure it's the first character in a line, i.e. press enter first.

You have following commands at your disposal

-L[bind_address:]port:host:hostport    Request local forward
-R[bind_address:]port:host:hostport    Request remote forward
-D[bind_address:]port                  Request dynamic forward
-KL[bind_address:]port    ...

HowTo: List packages in an apt repository

When you're using a third party repository on your Ubuntu or Debian system and need to find out which packages are available in this repo and which versions, you can consult the local apt cache.

In /var/lib/apt/lists you'll find a lot of *_Packages files that list the contents of apt repositories. Most repositories will have one file for the i386 architecture and one for the amd64 architecture. Make sure to pick the correct one.

To list the contents of the repository, view the respective files or, for a short summary:

/...

Don't use flock with GlusterFS

We recently encountered a problem with GlusterFS (7.x) when an application used the flock syscall on a GlusterFS path. If somehow two flock syscalls are made at the same time, the lock will never be released. And all future flock syscalls will wait forever.

This problem doesn't happen only on our infrastructure. There are a lot of bugreports, e.g.:

HowTo: Curl applications that are usually behind reverse proxies with TLS termination without the application redirecting to https schema

A lot of web applications require being called over https, which is a good thing. It's possible to configure this requirement at the web- or proxy server level, where nginx or apache will just redirect every request on http to https. Some applications additionally detect if the URL they've been called with contains the http or the https scheme and issue their own redirect response (usually 301 or 302) to https. This card is for the later kind.

The Problem

When you want to test your application directly on the VM it...

Parameter naming and checking in shell script

It is a good idea to use named variables for storing parameters of a script or function. We can use parameter expansion to either set a default or check mandatory arguments

Mandatory parameter

hello() {
    NAME=${1:?provide name as first parameter}
    echo "Hello $NAME!"
}

$ hello  # $?=1
bash: 1: provide name as first parameter

$ hello Foo # $?=1
Hello Foo!

Parameter with default

hello() {
  NAME=${1:-Marvin}
  echo "Hello $NAME!"
}

$ hello  # $?=0
Hello Marvin!

$ hello Foo # $?=1
Hello Foo!

netfilter's Connection Tracking system (nf_conntrack)

What is netfilter's Connection Tracking system?

The connection tracking system often referenced as nf_conntrack is part of the Netfilter framework. It allows the Linux kernel to keep track of all logical network connections and sessions. In combination with iptables this feature is used to achieve a stateful firewall.

Why to care about nf_conntrack?

All connections are stored in the connection tracking table. The size of the tracking table is based o...

restart puppetmaster after changing custom functions

The puppet master caches custom functions. If you edit an existing function (e.g. while you’re developing it), you’ll need to restart the puppet master before the new version can be used.

That also means you can't test functions in a different puppet environment. And also if you have changed functions in a different puppet environment (where you e.g. test some new module version) and this environment runs at first after a puppetmaster restart, you have this changed functions also in your production environment.

Changes in `modules/$FOOBA...

Get information about current running passenger processes

With passenger-status --show=requests you can get a huge JSON showing current information about the running passenger processes.

This can be useful if you want to find out what a passenger process is doing at the moment (for e.g. if one worker seems to be stuck):

  * PID: 4273    Sessions: 1       Processed: 47      Uptime: 50m 53s
    CPU: 43%     Memory  : 3644M   Last used: 49m 24s ago
    Shutting down...

This passenger process is using too much memory and seems it's Last used timestamp is old. The worker is processing a ...

Use Terraform grouping mode like Golang's Ellipsis expression

The terraform documentation states the ... syntax as (grouping mode*. See: Grouping-Results).

But this seems not the be the whole truth. Instead the ... syntax behaves like Go's Ellipsis expression which is used to pass a list as multiple parameters to a Variadic Function.

You can use this behavior for example if you want to merge a list of maps into one map:

locals {
  list_of_ma...

Playbook: fetch container logs from journald

docker

journalctl _CMDLINE=dockerd

podman

journalctl SYSLOG_IDENTIFIER=podman

identifying potential filter fields

journalctl -o verbose
journalctl -o json | jq

Get a more colorful output for man pages

You can add this function to your .bashrc (or the configuration file of the shell your using instead):

man() {
    LESS_TERMCAP_mb=$'\e'"[1;31m" \
    LESS_TERMCAP_md=$'\e'"[1;31m" \
    LESS_TERMCAP_me=$'\e'"[0m" \
    LESS_TERMCAP_se=$'\e'"[0m" \
    LESS_TERMCAP_so=$'\e'"[1;44;33m" \
    LESS_TERMCAP_ue=$'\e'"[0m" \
    LESS_TERMCAP_us=$'\e'"[1;32m" \
    command man "$@"
}