Terraform: Deploying code for lambda functions

If you're deploying code for your lambda function via terraform, this code is usually zipped and uploaded to Amazon S3 by terraform. The ZIP file's hash is then stored to terraform's state. However we have observed that zipping files can create ZIP archives with different hashes on different machines. This means that if you're collaborating with colleages e.g. via git, each run of terraform will possibly see a different hash of the code's ZIP archive and try to replace the lambda function.

Workaround

This workaround is for single file l...

Terraform plugin cache

If you're using many terraform root modules in a repository, downloading common providers like the AWS one can quickly add up to a lot of storage space wasted. Each root module will download a copy of the same provider file. The AWS provider is over 200 MB in size.

To avoid creating multiple copies of the same file, you can use the Terraform Plugin Cache.

Configuring the plugin cache:

It's possible to add the following setting to $HOME/.terraformrc:

plugin_...

Resize an Areca Raid by swapping disks

If you want to expand your Areca Raid by swapping out the disks for larger ones you will need to do the following:

  1. Swap out all disks one by one and wait until the syncronisation is finished in between.

  2. Reboot your server and enter the Raid Controller Configuration by pressing TAB at the right time.

  3. In the menu go the following path:

    3.1. Raid Set Function
    3.2. Rescue Raid Set
    3.3. Type RESETCAPACITY HDD RAID1 where HDD RAID1 is the name of the Raid Set
    3.3. It should now say: **Raid Set Capacity Recomputed...

loginctl: Failed to look up user $USER: No such process

When you get this error message from e.g. loginctl show-user [...] after you deleted a user from your system:

# $USER is the username you deleted
loginctl: Failed to look up user $USER: No such process

then you can either switch to a Linux distribution without systemd or restart systemd-logind to fix this problem.

systemctl restart systemd-logind

HowTo: Rebalance Elasticsearch Shards

If you end up with an Elasticsearch cluster which has a very different disk usage on it's nodes you can use these steps to rebalance the shards.

Before we begin it's important to understand how Elasticsearch defines balance:

The balance of the cluster depends only on the number of shards on each node and the indices to which those shards belong. It considers neither the sizes of these shards nor the available disk space on ...

Installing throttled to fix Lenovo laptops with nvidia graphics running Ubuntu losing performance after a while

We have observed Lenovo laptops with nvidia graphics losing performance after they have been in use for a few minutes. While we have not been able to pinpoint exactly what triggered the performance loss, we have found it mostly correlated with high load or during video calls (i.e. GPU usage), possibly related to temperature sensors picking up rising temps.

What happens is that the CPU clock falls to e.g. 400 MHz and will stay there, rendering the system unusable. This problem can be mitigated by using [throttled](https://github.com/erpalma/...

Prevent systemd from dropping IP-Addresses configured from other processes

Systemd's networkd will drop IP-Addresses configured by other processes like Keepalived's VRRP, when it is restarted. This may happens on updates.

To prevent this following settings are required in /run/systemd/network/10-netplan-eth0.network

v242 or earlier (even if you do not use DHCP client in networkd)

[DHCP]
CriticalConnection=yes

v243 or newer

[Network]
KeepConfiguration=yes

If there is netplan in place these files will be generated.

Add critical: yes to /etc/netplan/50-cloud-init.yaml

eth1:
...

Listing sizes in AWS S3 Buckets

Getting the whole bucket size

aws s3 ls s3://$BUCKETNAME/ --recursive --human-readable --summarize | tail -n2

Tail is used because otherwise all files will be printed on screen (but you may want that for some reason).

Getting the size of a specific directory/file

You just need to add the path to the bucket name:

aws s3 ls s3://$BUCKETNAME/some/dir --recursive --human-readable --summarize | tail -n2

Getting the size of all subdirs

If you imagine there is something like ncdu or `du -ha -d 1 /foo/bar...

Fix timedrifts after VM live migration

After performing a live migration of a VM you may encounter issues with the clock of the migrated VM. The time drifts fairly fast in the future or the past and ntpd doesn't seem to be able sync the time.

A bit of testing showed that the problem disappeared when we replaced openntpd with chrony. So it seems to be a openntpd issue probably.

To fix the issue on Ubuntu:

  1. sudo apt-get purge openntpd

  2. sudo rm /var/lib/openntpd/db/ntpd.drift

  3. reinstall openntpd and deploy the config again. In the makandra setup this is don...

HowTo: Clone a MariaDB database with mariabackup, mbstream and netcat

If you have a very large datadir in MariaDB and you want to transfer the data to another host (e.g. for replication) you may want to avoid storing it locally and copying it between the hosts.

You can stream the backup directly via netcat.

Transferring the stream data

  1. On the destination host
$ mkdir mariabackup
$ cd mariabackup
$ nc -l 9999 | cat - | mbstream -x
  1. On the source host
$ mariabackup --backup --stream=xbstream | nc destination-host 9999

Restoring the backup

After the streaming of the ...

XCA: Easy SSL certificate management

If you're about to handle X509 certificates and don't want to remember/google a handful of openssl commands you can use the graphical tool XCA.

Setup

  1. Install XCA sudo apt install xca
  2. Create a new database with a strong master password

Features

  • Create CSR
  • Verify SSL private key matches SSL certificate
    • enable Key name column
    • import certificate and private key
  • Import, export and convert PKCS#12, PKCS#7, PEM chain, DER and more
  • Check issuer chain
    • import certificate, intermediat...

send a testmail via exim

Sending a testmail from a server with a configured exim is easy as pie.

From your usershell:

  1. tell exim that we want to send a mail to an recipient and tell us about the details of the sending procedure

    $ exim -v receipient@foo.bar
    
  2. Tell exim about the sender, the subject and input some text.

    From: user@your.domain.example
    Subject: Foobar
    Text Text Text
    
    I like Pie!
    
  3. Press Ctrl+d

  4. You should see the details about the SMTP connection. You can return to your shell with...

HowTo: enable termination protection for all EC2 instances not in autoscaling groups via aws cli

aws ec2 describe-instances --output text --query 'Reservations[].Instances[?!not_null(Tags[?Key == `aws:autoscaling:groupName`].Value)] | [].[InstanceId]' | xargs -L1 -t aws ec2 modify-instance-attribute --disable-api-termination --instance-id

HowTo: enable DeletionProtection for all RDS instances via aws cli and jq

aws rds describe-db-instances | jq '.[][] | select(.DeletionProtection == false) | .DBInstanceIdentifier' | xargs -L1 -t aws rds modify-db-instance --deletion-protection --db-instance-identifier

Build and install exim4-daemon-custom on Ubuntu 16.04

Unfortunately I couldn't find a complete and working documentation on how to do this. But this steps succeeded (at least once).

  1. Install build dependencies

    $ sudo apt-get install pbuilder devscripts dpatch grep-dctrl debhelper
    
  2. Add deb-src sources to apt sources list (if you don't have it yet)

    deb-src http://security.ubuntu.com/ubuntu xenial-security main universe multiverse restricted
    
  3. Start the process:

    $ mkdir exim
    $ apt-get source exim4
    $ cd exim4-4.86.2
    $ fakeroot debian/r...
    

HowTo: Downgrade a Port on FreeBSD

Example

collectd5 in version 5.9.0 is broken and you upgraded it everywhere. But because it's FreeBSD it does not restart running services. After every old version is removed from /var/cache/pkg you restarted collectd and it does not start anymore, you really want to downgrade to 5.8.1 again. But even pkg now has 5.9.0 and you need your own make config setup anyway.

portdowngrade

Enter portdowngrade. Install it via pkg install portdowngrade.

You might want go to /root and run the next command should the fol...

Debug Puppet "Error: Failed to apply catalog: undefined method [...] for nil:NilClass"

If you get e.g. this error message when you try to run puppet agent:

Error: Failed to apply catalog: undefined method `strip' for nil:NilClass

you could have a very bad time debugging it. Because there will be no messages in the logs which could help you.

You should always remember puppet agents --trace option, because that's probably the only way how you can track this error. For example if you have an unparsable mail aliases file you will get this:

# puppet agent --test --trace --noop
Info: Using configured environment ...

GitLab: Rails Console Tasks

Sometimes you might need to do some task in GitLab which would be tedious if you'd have to do it via the Browser.

There is also an API which you could maybe use!

You can connect to the GitLab internal Rails with executing gitlab-rails console. It might be slow at times, especially if you use a lot of Tab for complex structures, so be aware of this.

The following code listings are expected to be entered inside the gitlab-rails console.

Remove GitLab Labels from already closed Issues

p = Pr...

HowTo: let puppet agent run from different branch

This is not encouraged because this change is easily forgotten. We recommend using this only if you know what you are doing.

Should you be in need to run puppet from a branch (called environment in puppet land) other than production for a long time and can't or don't want to disable puppet you can do the following:

Add a new section to /etc/puppetlabs/puppet/puppet.conf on the server that should apply the environment. It usually looks like this:

[main]
< some stuff >

Add the agent section so it looks like this:

[m...

start a systemd service blocked by rate limiting

If a systemd service unit is blocked by rate limiting (see in documentation) you can reset the failcounter via:

systemctl reset-failed $UNIT

Afterwards you can try to start the service again:

systemctl start $UNIT