Which labels will Prometheus send to Alertmanager?

Like many users of Prometheus, at work we use labels with AlertManager's routing to make sure the right people get paged when something goes wrong. Prometheus allows you to provide their labels via a number of sources (if I missed any, please reach out and let me know!):

• Labels resulting from the evaluation of the query expression
• Labels on the alert rule
• Labels on the alert rule's group
• External labels defined in the configuration file

Last week I realized that I didn't really know which sources took precedence when two or more define the same label, so I ran a little experiment to find out! (Feel free to skip ahead to the summary if that's all you care about)

Setup

I created the following alert rules:

groups:
- name: labeled_group
  labels:
    label_core_zr_org_team: group_team
  rules:
  - alert: LabeledGroupUnlabeledRuleUnlabeledQuery
    expr: vector(1)
  - alert: LabeledGroupLabeledRuleUnlabeledQuery
    expr: vector(1)
    labels:
      label_core_zr_org_team: rule_team
  - alert: LabeledGroupUnlabeledRuleLabeledQuery
    expr: 'absent(fake_metric{label_core_zr_org_team="query_team"})'
  - alert: LabeledGroupLabeledRuleLabeledQuery
    expr: 'absent(fake_metric{label_core_zr_org_team="query_team"})'
    labels:
      label_core_zr_org_team: rule_team
- name: unlabeled_group
  rules:
  - alert: UnlabeledGroupUnlabeledRuleUnlabeledQuery
    expr: vector(1)
  - alert: UnlabeledGroupLabeledRuleUnlabeledQuery
    expr: vector(1)
    labels:
      label_core_zr_org_team: rule_team
  - alert: UnlabeledGroupUnlabeledRuleLabeledQuery
    expr: 'absent(fake_metric{label_core_zr_org_team="query_team"})'
  - alert: UnlabeledGroupLabeledRuleLabeledQuery
    expr: 'absent(fake_metric{label_core_zr_org_team="query_team"})'
    labels:
      label_core_zr_org_team: rule_team

...which I then used with the following configuration:

global:
  scrape_interval:     15s
  evaluation_interval: 15s
  external_labels:
    label_core_zr_org_team: 'global_team'

scrape_configs:
  - job_name: 'test-exporter'
    static_configs:
    - targets: ['localhost:9100']

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

rule_files:
- /tmp/tmp.PqWFRvfUO4/rules.yml

Results

After letting this run for a few minutes and checking AlertManager, I ended up with the following table:

Summary

...which can be summarized as follows:

• If the rule has a label, it is the one used
• Otherwise, if the rule's group has a label, that is used
• Otherwise, if running the query generates a label, that is used
• If none of these are the case, external labels are used