Measure HTTP Connection times (TTFB) with curl

To measure the time of your HTTP request with curl, you can use the -w (--write-out) option:

curl --silent -o /dev/null -w "%{time_connect};%{time_starttransfer};%{time_total};%{time_appconnect};%{time_pretransfer}\n" https://example.org
0.121080;0.685826;0.685881;0.571310;0.571396

With -w you can make curl display informations for your HTTP request. As seen in the example above you can e.g. output this in a CSV format for further analysis.

Informations to the specific informations can be found in the [man page](https://curl...

Root-account tries to access mariadb-database regularly

We had a strange behaviour on one of our mariadb-servers:
Everyday at around midnight we saw that the root-account on one of our servers is trying to access the database.

# journalctl -u mariadb
Dec 16 00:00:03 cool-server mariadbd[788]: 2022-12-16  0:00:03 34996 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
Dec 17 00:00:02 cool-server mariadbd[788]: 2022-12-17  0:00:02 89012 [Warning] Access denied for user 'root'@'localhost' (using password: NO)
Dec 18 00:00:02 cool-server mariadbd[788]: 2022-12-18  0:00...

FAQ for When PostgreSQL Indexes Are Corrupted After Locale Changes

When changing glibc versions, it's possible to end up with corrupt indexes in PostgreSQL.

Are My Indexes Affected?

If your database has been created on an operating system with glibc version < 2.28 and later upgraded to an operating system with glibc >= 2.28, you're most likely affected. To find out which indexes in our database are affected, run the following slightly modified [query from the PostgreSQL documentation](https://...

Mind your locales with glibc upgrades when using PostgreSQL

When changing the glibc version, it's possible that the upgrade also includes changes to how locales work.

This is especially relevant when using PostgreSQL databases (of any version), since those depend on the locales functionality provided by glibc as well. It's recommended to re-build all indexes, to be sure no index that depends on collation features is left in an inconsistent state.

Am I affected?

There has been a big change in glibc regarding loca...

How to prevent duplicate exported resources across a Puppet Infrastructure

There are cases where we have multiple exported resources that are identical. There are nodes that provide the same services for failover reasons. It is likely to have duplicate resources, when exporting nagios_check from these nodes. From the exporting node's point of view everything is fine. But when it comes to realizing the resources on the monitoring server, the puppetrun will fail due to the duplicate addresses. So how to mitigate this issue?

Nodename in the title

To disentangle the nagios_check resources it is easy to just s...

When to use a function over a defined in Puppet

In Puppet, there are some differences between defined and function types when it comes to code encapsulation. In most cases, a defined type is best, but there are some situations where a function is more appropriate.

Use a defined type when:

  • You need to retrieve the resource or its parameter via puppetdb_query().
  • The containing code creates real resources.
  • You need a specific wrapper for the code.

Use a function when:

  • You need to return data from the code.
  • You don't need to specify a resource name.
  • You need a general ...

MySQL client: Double check the used character set with utf8mb4

The Oracle mysql client has an odd behavior if your server uses latin1 as default character-set-server.

Command

mysql --version
mysql  Ver 8.0.31-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))

mysql --default-character-set utf8mb4 -e "SHOW VARIABLES LIKE '%char%';"

Expectation

utf8mb4 will be used as character set in this session.

Reality

The mysql client falls back to latin1:

+--------------------------+----------------------------------------------+
| Variable_name            | Value           ...

Why doesn't my prometheus relabel_config work?

When configuring prometheus scrape_configs, you may use relabel_configs to filter your metrics or change some fields. Your config may look like this:

scrape_configs
        - job_name: kubernetes-service-endpoints
          sample_limit: 10000
          kubernetes_sd_configs:
          - role: endpoints
          relabel_configs:
          - action: kee...

load balance unix sockets to UDP destinations

Nginx is capable of forwarding a unix socket to UDP backend servers. This is quite handy for load balance syslog traffic.

Example nginx configuration

load_module /usr/share/nginx/modules/ngx_stream_module.so;

stream {
    upstream syslog_servers {
        server 192.0.2.10:514;
        server 192.0.2.11:514;
        server 192.0.2.12:514;
    }
    server {
        listen unix:/run/nginx/log.sock udp;
        proxy_pass syslog_server;
    }
}

Testing the connection

echo "Hello Syslog!" | socat - /run/nginx/log.sock...

Use systemd-run as an alternative for screen

You might use screen or tmux to run a temporary command on a server which continues to run after the SSH session is closed.

Consider systemd-run as alternative. It will turn every command in a systemd service unit:

# Run `openssl speed` as unit run-benchmark.service
$ sudo systemd-run --unit=run-benchmark openssl speed

# Query the current status
$ systemctl status run-benchmark.service
● run-benchmark.service - /usr/bin/openssl speed
   Loaded: loaded (/run/systemd/transient/run-benchmark.serv...

Networking restart on FreeBSD

If you try to restart you may encouter the problem that your networking connection gets shutdown but not start again. Here is the right way to restart networking on FreeBSD:

service netif restart && service routing restart

Disable AWS Free Tier Usage Alerts

Ever felt annoyed by AWS Free Tier limit alert emails?

Just disable them:

Billing preferences -> Cost Management Preferences -> Receive Free Tier Usage Alerts

Terragrunt/terraform: fork/exec argument list too long

When terragrunt is relaying information to input variables it's happening via environment variables. Depending on the size of the content of the variable it might exceed your OS limits. This is independent of your shell.

A possible workaround is to use a generated file to load the input instead of the env variable, e.g.

# WORKAROUND
# the variable my_huge_input cannot be loaded as part of the inputs
generate "dependencies" {
  path      = "dependencies.auto.tfvars"
  if_exists = "overwrite_terragrunt"
  contents = <<EOF
my_input    ...

HowTo: Clone and refresh all repos in a GitLab Group

If the project you're working on has, say, 39 repositories and counting in GitLab and you need all the repos checked out for some reason, here's how to do it.

Checking out all repos

  1. Create a personal access token for GitLab that has the API permissions. In your terminal, store this key in an env variable.
  2. For each group you want to check out:
    1. Create a new directory where you want all the checkouts to live.
    2. In GitLab, navigate to the Group's overview page so you can see the Group ID.
    3. In the directory you created...

HowTo: Get postgres shell in kubernetes

If your postgres database is only accessible from inside a kubernetes cluster, e.g. if it's configured in AWS RDS and not available to the public (as it should be!), here's how to open a psql shell inside Kubernetes and connect to the database. Make sure to replace the variables appropriately.

$ kubectl run postgresql-client \
  --image=postgres      \
  --namespace=$NAMESPACE \
  --stdin=true --tty=true \
  --rm=true                \
  --env="PGPASSWORD=$PASSWORD_FOR_POSTGRES \
  --command -- \
  psql --host=$HOSTNAME_FOR_POSTG...

HowTo: Get kubernetes secrets in plaintext

Here's a one-liner to view base64 encoded secrets in kubernetes. Make sure you have jq installed.

$ kubectl get -n $NAMESPACE secret/$SECRET_NAME -o json| jq '.data | map_values(@base64d)'
{
  "database": "secret1",
  "endpoint": "secret2",
  "username": "secret3",
  "password": "secret4"
}

Bolt: Run commands from a file

There's a simple way in bolt to run commands from a file without caring about BASH escaping:

# /home/user/foo.sh
echo "$(hostname -f): $(uptime)"
echo "${USER}"
echo "${SERVERLIST}" | bolt command run @foo.sh --run-as root --targets -

Use script run to run a ruby script:

#!/usr/bin/env ruby
# /home/user/bar.rb

puts 'Hello, world!'...

Delete unresponsive rabbitmq queue

In our monitoring, RabbitMQ queues like aliveness-test may show up as unresponsive, with a ping timeout after 10 seconds. The logfile will generally read like this:

operation queue.delete caused a channel exception not_found: failed to perform operation on queue 'example' in vhost '/' due to timeout

For the aliveness-test queue, you can can use this command to delete it:

rabbitmqctl eval 'rabbit_amqqueue:internal_delete({resource,<<"/">>,queue,<<"aliveness-test">>}).'

This queue is only used for monitoring if RabbitMQ...

Replacing exported resources with puppetdb queries

Instead of using Puppet exported resources you can use the puppetdb_query feature.

This can result in more complex code but has several benefits:

  • you can use more complex puppetdb queries to get the resources you want than with the limited filtering options of exported resources
  • because you receive a data object of the resources you can only use a part of the information contained
  • you c...

Keepalived VRRP FAQ

How can I configure virtual IP's?

There are two parameter to set up virtual ips in Keepalived:

virtual_ipaddress

Addresses defined here are included into the VRRP Packages and are therefore limited in number, especially with IPv6.

Address families cannot be mixed here.

If this contains IPv6 addresses, Keepalived will use VRRP over IPv6.

The inclusion of the addresses into the VRRP packages is for troubleshooting reasons. See RFC5798 Section 5.2.9 and [RFC3768 Secti...