Updated: netfilter's Connection Tracking system (nf_conntrack)

Changes

# What is netfilter's Connection Tracking system?
The [connection tracking system](https://wiki.nftables.org/wiki-nftables/index.php/Connection_Tracking_System) often referenced as `nf_conntrack` is part of the Netfilter framework. It allows the Linux kernel to keep track of all logical network connections and sessions. In combination with `iptables` this feature is used to achieve a stateful firewall.
# Why to care about `nf_conntrack`?
All connections are stored in the connection tracking table. The size of the tracking table is based on the memory of the system. A node with 4 GB RAM will get a maximum table size of 64 KB.
On most systems the default settings are fine. However if you're running a VM host which has a lot of virtual machines running which by themselves have a lot of connections the connection tracking table can get filled. This could happen if you're running virtual load balancers on your VM hosts.
**If the table is full and there are still many new connections coming in the kernel will start to drop packages.** You might not be able to establish new connections and `dmesg` will output messages like:
```
2021-06-29T09:21:01,266251+02:00 nf_conntrack: nf_conntrack: table full, dropping packet
2021-06-29T09:21:01,267799+02:00 nf_conntrack: nf_conntrack: table full, dropping packet
2021-06-29T09:21:01,267806+02:00 nf_conntrack: nf_conntrack: table full, dropping packet
```
# Interact with `nf_conntrack`
It's possible to interact with `nf_conntrack` directly via the [/proc/sys/net/netfilter/nf_conntrack_* variables](https://www.kernel.org/doc/Documentation/networking/nf_conntrack-sysctl.txt) or through a userspace tool like `conntrack`.
## Get the maximum size of the connection tracking table
```
$ cat /proc/sys/net/netfilter/nf_conntrack_max
```
## Get the size of the currently allocated flow entries
```
$ cat /proc/sys/net/netfilter/nf_conntrack_count
```
## Raise the connection tracking table size to 512 KB
```
$ sudo sysctl -w net.nf_conntrack_max=524288
```
**Attention:** If you're using Proxmox the `sysctl` value will get restored after a short moment. Change the option `nf_conntrack_max` in the [host specific firewall configuration](https://pve.proxmox.com/wiki/Firewall) instead.
## List all connection tracking table entries
```
$ sudo conntrack -L
```
## Check for temporary entries
These entries [will be deleted](https://www.netfilter.org/documentation/FAQ/netfilter-faq-3.html#ss3.17) as soon as the system runs out of connection tracking entries.
```
$ sudo conntrack -L | grep 'UNREPLIED'
-```
+```
+
+Analyse why there are so many `UNREPLIED` entries in the conntrack table. E.g. if most of them come from the same IP address but there's different destination addresses, there might be a portscan in progress.
+
+After you have analysed or stored away the log, you should clean up unused entries to fix our monitoring, since these stale entries will not go away on their own and we won't see if another issue (e.g. an attack) will fill up our conntrack tables.
+
+Decide which option resolves the situation best with minimal impact on production:
+
+```shell
+$ sudo  conntrack -D -s 49.12.3.247 # example: delete all conntrack table entries from portscan02-prod.makandra
+$ sudo conntrack -D --state UNREPLIED # delete all UNREPLIEID entries in the conntrack table
+$ sudo conntrack -F # flush the whole table. ATTENTION: This will kill all connections!
+```
+
+
+