Repair broken etcd node
If one etcd node is no longer a member of the remaining etcd cluster or fails to connect you need to remove it from the cluster and then add it again:
- Stop etcd on the broken node :
sudo service etcd stop - delete the data on the broken node
sudo rm -r /var/lib/etcd/data/* - delete the wal data on the broken node:
sudo rm -r /var/lib/etcd/wal/* - Follow the instructions for etcd runtime-configuration Show archive.org snapshot , remove the broken node from the cluster, then re-add it again and update the etcd config on the broken node with the parameters printed by the add command.
- start etcd again
Even if etcd logging is configured to /var/log/etcd/etcd.log it can happen on new hosts (focal) that StandardOutput is only in journal (systemctl status etcd).
Related cards:
Elasticsearch unassigned replica shards on single node instances
If you have a single node elasticsearch instance and indices with replicas enabled your cluster state will be yellow. If you have replica shards they should be moved to a different node for high availability purposes. With a single node this can't...
HowTo: Rebalance Elasticsearch Shards
If you end up with an Elasticsearch cluster which has a very different disk usage on it's nodes you can use these steps to rebalance the shards.
Before we begin it's important to understand how Elasticsearch defines [balance](https://www.elastic....
How to prevent duplicate exported resources across a Puppet Infrastructure
There are cases where we have multiple exported resources that are identical. There are nodes that provide the same services for failover reasons. It is likely to have duplicate resources, when exporting nagios_check from these nodes. From the ...
Replacing exported resources with puppetdb queries
Instead of using Puppet exported resources you can use the puppetdb_query feature.
This can result...
HowTo Generate Nagios Config with puppet fast
Creating Nagios Config with puppet
Let's have a look at the classic way of managing Nagios configuration with exported Puppet resources.
Here is [a good article about the topic](https://www.ericholzbach.net/blog/automating-nagios-with-puppet-a...
Fix "A client error (MalformedCertificate) occurred: Invalid Private Key." at AWS SSL Certificate upload
I'm creating certificate requests with this command:
openssl req -new -out www.example.com.csr -keyout www.example.com.key -newkey rsa:2048 -nodes
When I try to upload the certificate to AWS IAM I get this error:
$ aws iam upload-se...
Test websocket connections
You can use wscat:
sudo apt-get install node-ws
#
wscat -c ws://echo.websocket.org
connected (press CTRL+C to quit)
> foo
< foo
> bar
< bar
>
HowTo: Downgrade a Port on FreeBSD
Example
collectd5 in version 5.9.0 is broken and you upgraded it everywhere. But because it's FreeBSD it does not restart running services. After every old version is removed from /var/cache/pkg you restarted collectd and it does not s...
netfilter's Connection Tracking system (nf_conntrack)
What is netfilter's Connection Tracking system?
The connection tracking system often referenced as nf_conntrack is part of the Netfilter framework. It allows the ...
Keepalived VRRP FAQ
How can I configure virtual IP's?
There are two parameter to set up virtual ips in Keepalived:
virtual_ipaddress
Addresses defined here are included into the VRRP Packages and are therefore limited in number, especially with IPv6.
Add...
