When configuring prometheus scrape_config Show archive.org snapshot s, you may use relabel_configs Show archive.org snapshot to filter your metrics or change some fields. Your config may look like this:
scrape_configs
- job_name: kubernetes-service-endpoints
sample_limit: 10000
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: Namespace
One common mistake is that you drop a label without thinking about it even if you need it in a later rule in the pipeline. For example: imagine you now want to introduce a configuration to drop metrics from a certain kubernetes namespace and add and extend your configuration like this:
scrape_configs
- job_name: kubernetes-service-endpoints
sample_limit: 10000
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: Namespace
### NEW drop RULE ###
- action: drop
regex: ".*foo.*"
source_labels:
- __meta_kubernetes_namespace
The drop will never occur, because meta_kubernetes_namespace will be removed due to a rule how prometheus processes the relabeling:
Quote
Labels starting with
__will be removed from the label set after target relabeling is completed.
See relabel_config Show archive.org snapshot for more information.
You should fix this by moving your drop action before the replace action and removing the __ from the source_labels:
- action: drop
regex: ${drop_namespace_regex}
source_labels:
- meta_kubernetes_namespace
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: Namespace
