HowTo: List packages in an apt repository

When you're using a third party repository on your Ubuntu or Debian system and need to find out which packages are available in this repo and which versions, you can consult the local apt cache.

In /var/lib/apt/lists you'll find a lot of *_Packages files that list the contents of apt repositories. Most repositories will have one file for the i386 architecture and one for the amd64 architecture. Make sure to pick the correct one.

To list the contents of the repository, view the respective files or, for a short summary:

/...

Don't use flock with GlusterFS

We recently encountered a problem with GlusterFS (7.x) when an application used the flock syscall on a GlusterFS path. If somehow two flock syscalls are made at the same time, the lock will never be released. And all future flock syscalls will wait forever.

This problem doesn't happen only on our infrastructure. There are a lot of bugreports, e.g.:

HowTo: Curl applications that are usually behind reverse proxies with TLS termination without the application redirecting to https schema

A lot of web applications require being called over https, which is a good thing. It's possible to configure this requirement at the web- or proxy server level, where nginx or apache will just redirect every request on http to https. Some applications additionally detect if the URL they've been called with contains the http or the https scheme and issue their own redirect response (usually 301 or 302) to https. This card is for the later kind.

The Problem

When you want to test your application directly on the VM it...

Parameter naming and checking in shell script

It is a good idea to use named variables for storing parameters of a script or function. We can use parameter expansion to either set a default or check mandatory arguments

Mandatory parameter

hello() {
    NAME=${1:?provide name as first parameter}
    echo "Hello $NAME!"
}

$ hello  # $?=1
bash: 1: provide name as first parameter

$ hello Foo # $?=1
Hello Foo!

Parameter with default

hello() {
  NAME=${1:-Marvin}
  echo "Hello $NAME!"
}

$ hello  # $?=0
Hello Marvin!

$ hello Foo # $?=1
Hello Foo!

netfilter's Connection Tracking system (nf_conntrack)

What is netfilter's Connection Tracking system?

The connection tracking system often referenced as nf_conntrack is part of the Netfilter framework. It allows the Linux kernel to keep track of all logical network connections and sessions. In combination with iptables this feature is used to achieve a stateful firewall.

Why to care about nf_conntrack?

All connections are stored in the connection tracking table. The size of the tracking table is based o...

restart puppetmaster after changing custom functions

The puppet master caches custom functions. If you edit an existing function (e.g. while you’re developing it), you’ll need to restart the puppet master before the new version can be used.

That also means you can't test functions in a different puppet environment. And also if you have changed functions in a different puppet environment (where you e.g. test some new module version) and this environment runs at first after a puppetmaster restart, you have this changed functions also in your production environment.

Changes in `modules/$FOOBA...

Get information about current running passenger processes

With passenger-status --show=requests you can get a huge JSON showing current information about the running passenger processes.

This can be useful if you want to find out what a passenger process is doing at the moment (for e.g. if one worker seems to be stuck):

  * PID: 4273    Sessions: 1       Processed: 47      Uptime: 50m 53s
    CPU: 43%     Memory  : 3644M   Last used: 49m 24s ago
    Shutting down...

This passenger process is using too much memory and seems it's Last used timestamp is old. The worker is processing a ...

Use Terraform grouping mode like Golang's Ellipsis expression

The terraform documentation states the ... syntax as (grouping mode*. See: Grouping-Results).

But this seems not the be the whole truth. Instead the ... syntax behaves like Go's Ellipsis expression which is used to pass a list as multiple parameters to a Variadic Function.

You can use this behavior for example if you want to merge a list of maps into one map:

locals {
  list_of_ma...

Playbook: fetch container logs from journald

docker

journalctl _CMDLINE=dockerd

podman

journalctl SYSLOG_IDENTIFIER=podman

identifying potential filter fields

journalctl -o verbose
journalctl -o json | jq

Get a more colorful output for man pages

You can add this function to your .bashrc (or the configuration file of the shell your using instead):

man() {
    LESS_TERMCAP_mb=$'\e'"[1;31m" \
    LESS_TERMCAP_md=$'\e'"[1;31m" \
    LESS_TERMCAP_me=$'\e'"[0m" \
    LESS_TERMCAP_se=$'\e'"[0m" \
    LESS_TERMCAP_so=$'\e'"[1;44;33m" \
    LESS_TERMCAP_ue=$'\e'"[0m" \
    LESS_TERMCAP_us=$'\e'"[1;32m" \
    command man "$@"
}

Nginx Proxy buffer tuning

When a nginx reverse proxy complains about upstreams sending too big headers, tweaking the buffers responsibly can help to prevent this issue.

Example log message:

upstream sent too big header while reading response header from upstream, client: 192.0.2.100, server: localhost, request: "GET /index.html HTTP/1.1", upstream: "http://198.51.100.123:80/index.html", host: "192.0.2.10:80"

The cause

This behaviour was caused by an application that transforms parts of the query from the URL into a response header. If the query in the ...

Fix Imagemagick CVE-2022-44268 in Ubuntu packages

A severe bug was found in ImageMagick by Bryan Gonzalez from Ocelot Team.
It allows to embed the content of an arbitrary remote file when ImageMagick parses PNG files.
We found lots of older versions of ImageMagick to be vulnerable.

So far there is no information on updated Packages for Ubuntu (https://ubuntu.com/security/CVE-2022-44268).

Due to that we patched our systems as follows:

Ubuntu 22.04:

Get package source on a Ubuntu 22.04 syst...

Check JVM settings of running java process

If you want to check the configuration of a running JVM process you can use jcmd.

  1. List the running processes:

    $ jcmd -l
    1 /app.jar
    140 jdk.jcmd/sun.tools.jcmd.JCmd -l
    

    The first column shows the PID of the process.

  2. Print system configuration with the PID:

    $ jcmd 1 VM.system_properties
    1:
    #Thu Jan 26 10:34:20 UTC 2023
    java.specification.version=17
    sun.jnu.encoding=UTF-8
    java.class.path=/app.jar
    java.vm.vendor=Eclipse Adoptium
    sun.arch.data.model=64
    catalina.use...
    

Parsing multiline container logs with fluent-bit

If you have for e.g. a Java application which outputs multiline stack traces inside a container running in kubernetes you might wonder how you can concat alle these lines to a single log message in fluent-bit. If fluent-bit is receiving the log output directly you can just set the multiline.parser to java. But when you're reading the logs from /var/log/containers this is not possible, because every line of the log message is logged in the CRI log format. This means that every line of the log becomes a JSON object containing addit...

Delete specific Redis-DBs

To delete a specific redis-DB you need to use the FLUSHDB-command in combination with the SELECT-command. For more information have a look at the documentation for FLUSHDB and SELECT.

Attention

By default when connecting to a redis-instance you always connect with db0.

When connecting to the db you can list your keyspaces/databases with:

# Show info about all databases
127.0.0.1:6379> INFO keyspace
# Keyspace
db0:keys=2674,expires=2663,avg_ttl=99821...

Desktop notification for failed systemd user services

You can execute systemctl --user --failed to check for failed systemd user units. But let's face it: It's inconvenient and you'll probably miss failures. Better use desktop notifications.

~/.config/systemd/user/service.d/user-failure-notification.conf

Add a OnFailure handler to all user units.

[Unit]
OnFailure=user-failure-notification@%n

~/.config/systemd/user/user-failure-notification@.service

Add a template service unit which sends the notifications. The instance variable %i is replaced by the calling unit.

Measure HTTP Connection times (TTFB) with curl

To measure the time of your HTTP request with curl, you can use the -w (--write-out) option:

curl --silent -o /dev/null -w "%{time_connect};%{time_starttransfer};%{time_total};%{time_appconnect};%{time_pretransfer}\n" https://example.org
0.121080;0.685826;0.685881;0.571310;0.571396

With -w you can make curl display informations for your HTTP request. As seen in the example above you can e.g. output this in a CSV format for further analysis.

Informations to the specific informations can be found in the [man page](https://curl...

Root-account tries to access mariadb-database regularly

We had a strange behaviour on one of our mariadb-servers:
Everyday at around midnight we saw that the root-account on one of our servers is trying to access the database.

# journalctl -u mariadb
Dec 16 00:00:03 cool-server mariadbd[788]: 2022-12-16  0:00:03 34996 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
Dec 17 00:00:02 cool-server mariadbd[788]: 2022-12-17  0:00:02 89012 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
Dec 18 00:00:02 cool-server mariadbd[788]: 2022-12-18  0:00...

FAQ for When PostgreSQL Indexes Are Corrupted After Locale Changes

When changing glibc versions, it's possible to end up with corrupt indexes in PostgreSQL.

Are My Indexes Affected?

If your database has been created on an operating system with glibc version < 2.28 and later upgraded to an operating system with glibc >= 2.28, you're most likely affected. To find out which indexes in our database are affected, run the following slightly modified [query from the PostgreSQL documentation](https://...

Mind your locales with glibc upgrades when using PostgreSQL

When changing the glibc version, it's possible that the upgrade also includes changes to how locales work.

This is especially relevant when using PostgreSQL databases (of any version), since those depend on the locales functionality provided by glibc as well. It's recommended to re-build all indexes, to be sure no index that depends on collation features is left in an inconsistent state.

Am I affected?

There has been a big change in glibc regarding loca...

How to prevent duplicate exported resources across a Puppet Infrastructure

There are cases where we have multiple exported resources that are identical. There are nodes that provide the same services for failover reasons. It is likely to have duplicate resources, when exporting nagios_check from these nodes. From the exporting node's point of view everything is fine. But when it comes to realizing the resources on the monitoring server, the puppetrun will fail due to the duplicate addresses. So how to mitigate this issue?

Nodename in the title

To disentangle the nagios_check resources it is easy to just s...

When to use a function over a defined in Puppet

In Puppet, there are some differences between defined and function types when it comes to code encapsulation. In most cases, a defined type is best, but there are some situations where a function is more appropriate.

Use a defined type when:

  • You need to retrieve the resource or its parameter via puppetdb_query().
  • The containing code creates real resources.
  • You need a specific wrapper for the code.

Use a function when:

  • You need to return data from the code.
  • You don't need to specify a resource name.
  • You need a general ...

Create gitlab container expiry config in every repo in a group

The link contains a simple shell script to push a cleanup policy for gitlab repositories in a group.

Please mind that this will overwrite the configuration for all the repositories with the given settings.

MySQL client: Double check the used character set with utf8mb4

The Oracle mysql client has an odd behavior if your server uses latin1 as default character-set-server.

Command

mysql --version
mysql  Ver 8.0.31-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))

mysql --default-character-set utf8mb4 -e "SHOW VARIABLES LIKE '%char%';"

Expectation

utf8mb4 will be used as character set in this session.

Reality

The mysql client falls back to latin1:

+--------------------------+----------------------------------------------+
| Variable_name            | Value           ...