When configuring prometheus scrape_config Show archive.org snapshot s, you may use relabel_configs Show archive.org snapshot to filter your metrics or change some fields. Your config may look like this:

scrape_configs
        - job_name: kubernetes-service-endpoints
          sample_limit: 10000
          kubernetes_sd_configs:
          - role: endpoints
          relabel_configs:
          - action: keep
            regex: true
            source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_scrape

          - action: replace
            source_labels:
              - __meta_kubernetes_namespace
            target_label: Namespace

Rails Long Term Support

Rails LTS provides security patches for old versions of Ruby on Rails (2.3, 3.2, 4.2 and 5.2)

• Prevents you from data breaches and liability risks
• Upgrade at your own pace
• Works with modern Rubies

Read more Show archive.org snapshot

One common mistake is that you drop a label without thinking about it even if you need it in a later rule in the pipeline. For example: imagine you now want to introduce a configuration to drop metrics from a certain kubernetes namespace and add and extend your configuration like this:

scrape_configs
        - job_name: kubernetes-service-endpoints
          sample_limit: 10000
          kubernetes_sd_configs:
          - role: endpoints
          relabel_configs:
          - action: keep
            regex: true
            source_labels:
              - __meta_kubernetes_service_annotation_prometheus_io_scrape
          - action: replace
            source_labels:
              - __meta_kubernetes_namespace
            target_label: Namespace
            ### NEW drop RULE ###
          - action: drop
            regex: ".*foo.*"
            source_labels:
            - __meta_kubernetes_namespace

The drop will never occur, because meta_kubernetes_namespace will be removed due to a rule how prometheus processes the relabeling:

Quote

Labels starting with __ will be removed from the label set after target relabeling is completed.

See relabel_config Show archive.org snapshot for more information.

You should fix this by moving your drop action before the replace action and removing the __ from the source_labels:

          - action: drop
            regex: ${drop_namespace_regex}
            source_labels:
            - meta_kubernetes_namespace
          - action: replace
            source_labels:
              - __meta_kubernetes_namespace
            target_label: Namespace