You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

  • the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
  • group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)
Answer from user2361830 on Stack Overflow
🌐
Yannick Pereira-Reis
ypereirareis.github.io › blog › 2020 › 02 › 21 › how-to-join-prometheus-metrics-by-label-with-promql
How to join Prometheus metrics by label with PromQL – Yannick Pereira-Reis
The result we want is something ... => 30887053824 node1 => 36352176166912 · sum(node_disk_bytes_read * on(instance) group_left(node_name) node_meta{}) by (node_name) on(instance) => this is how to JOIN on label insta...
Top answer
1 of 3
76

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

  • the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
  • group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)
2 of 3
15

It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.

The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.

For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:

consul_health_service_status
  * on(service_name, node) group_left(env)
consul_service_tags

Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:

consul_health_service_status{status="critical"}
  * on(service_name, node) group_left(env)
consul_service_tags
Discussions

Allow label_join to be used in aggregation operators
I have looked quite a bit and haven't found a way to do this but it seems simple so please let me know if I've missed something. I have a need to do capacity planning for failover events wh... More on github.com
🌐 github.com
12
November 23, 2021
Mechanism Search based on new label created through label_join function
Proposal As you already know using label_join() and label_replace() function can be used to add new label dynamically. But there is way by to do matching/search on top of this newly added label. So this new label is useless if it can't b... More on github.com
🌐 github.com
24
January 15, 2019
Using range with label_join - PromQL - Prometheus Monitoring System
I have a set of metrics that are being sent only when there is an issue. I want to create an expression that will capture the number of issues in the last 15 minutes. For efficiency I would like to use a regex with the metrics names (let’s asume that there will be no performance problems) . More on discuss.prometheus.io
🌐 discuss.prometheus.io
0
October 26, 2022
simpler syntax for joins on labels with different names
Background When doing joins, I usually encounter cases where I want to join through a label that has different names on each side. For example, suppose I have: A container_fs_reads_total metric wit... More on github.com
🌐 github.com
8
October 10, 2024
🌐
GitHub
github.com › prometheus › prometheus › issues › 2204
Joining different labels · Issue #2204 · prometheus/prometheus
November 18, 2016 - The label naming might not be 100% aligned, i.e. in Kubernetes we get container metrics with a pod_name label from cAdvisor and metrics with a pod label from kube-state-metrics. The values are the same pod identifying names. However, we have to use label_replace() to be able to join them properly, which is always somewhat awkward and hard to decipher.
Author: prometheus
🌐
Grafana
grafana.com › blog › 2021 › 08 › 04 › how-to-use-promql-joins-for-more-effective-queries-of-prometheus-metrics-at-scale
How to use PromQL joins for more effective queries of Prometheus metrics at scale | Grafana Labs
August 5, 2021 - PromQL supports the ability to join two metrics together: You can append a label set from one metric and append it to another at query time. This can be useful in Prometheus rule evaluations, since it lets you generate a new metric for a series by appending labels from another info metric.
🌐
GitHub
github.com › prometheus › prometheus › issues › 9850
Allow label_join to be used in aggregation operators · Issue #9850 · prometheus/prometheus
November 23, 2021 - {instance="datacenter0"} 1000 {instance="datacenter1"} 2000 sum by(label_join("instance", "datacenters", ",")) ( irate( tIPFilterParamsEgrHitByteCount{ instance=~"datacenter.*", }[10m] ) ) {datacenters="datacenter0,datacenter1"} 3000
Author: prometheus
🌐
Prometheus
prometheus.io › docs › prometheus › 1.8 › querying › functions
Query functions | Prometheus
For each timeseries in v, label_join(v instant-vector, dst_label string, separator string, src_label_1 string, src_label_2 string, ...) joins all the values of all the src_labels using separator and returns the timeseries with the label dst_label containing the joined value.
🌐
GitHub
github.com › prometheus › prometheus › issues › 5097
Mechanism Search based on new label created through label_join function · Issue #5097 · prometheus/prometheus
January 15, 2019 - label_join(sum(app:healthcheck:gauge{application="collectHealthies"} != bool 200) by (first_name, last_name), "full_name", " ", "first_name", "last_name") ... {first_name="Jack",full_name="Jack daniels",last_name="daniels"} 10 {first_name="...
Author: prometheus
🌐
O'Reilly
oreilly.com › library › view › hands-on-infrastructure-monitoring › 9781789612349 › 1d5afbc1-62fd-4ba2-ba88-b639ef3a7234.xhtml
label_join() and label_replace() - Hands-On Infrastructure Monitoring with Prometheus [Book]
May 31, 2019 - Content preview from Hands-On Infrastructure Monitoring with Prometheus · These functions are used to manipulate labels—they allow you to join labels to other ones, extract parts of label values, and even drop labels (though that particular operation is easier and more ergonomic to do with standard aggregation operators).
Authors: Joel BastosPedro Araújo
Published: 2019
Pages: 442
Find elsewhere
🌐
Prometheus
discuss.prometheus.io › promql
Using range with label_join - PromQL - Prometheus Monitoring System
October 26, 2022 - I have a set of metrics that are being sent only when there is an issue. I want to create an expression that will capture the number of issues in the last 15 minutes. For efficiency I would like to use a regex with the m…
🌐
SigNoz
signoz.io › blog › prometheus query tutorial with examples
Prometheus Query Tutorial with examples | SigNoz
August 1, 2022 - In this case, these are (a, b, c, z). There is no limit to the number of sources you can concatenate this way. The separator element should also be defined in the function. In this case, it is a simple comma separator (","). Prometheus ...
🌐
OneUptime
oneuptime.com › home › blog › how to join two metrics in prometheus query
How to Join Two Metrics in Prometheus Query
December 17, 2025 - # Create composite key for matching label_join( container_memory_usage_bytes, "pod_container", "_", "pod", "container" )
🌐
Grafana
grafana.com › blog › 2019 › 04 › 01 › how-were-using-prometheus-subqueries-at-grafana-labs.
How We're Using Prometheus Subqueries at Grafana Labs. | Grafana Labs
October 12, 2021 - quantile_over_time(0.95, sum by ... must never do a sum before a rate. For example, instead of rate((success+failure)[1m]) you must do rate(success[1m]) + rate(failure[1m]) ....
🌐
GitHub
github.com › prometheus › prometheus › issues › 15146
simpler syntax for joins on labels with different names · Issue #15146 · prometheus/prometheus
October 10, 2024 - Background When doing joins, I usually encounter cases where I want to join through a label that has different names on each side. For example, suppose I have: A container_fs_reads_total metric with a pod label A some_custom_pod_info met...
Author: prometheus
🌐
Iximiuz
iximiuz.com › en › posts › prometheus-vector-matching
Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)
November 30, 2021 - How to join metrics on matching labels? Vector matching: one-to-one, one-to-many and many-to-one, many-to-many. What on/ignoring clause is for? group_left and group_right explained. Logical/set operations explained.
🌐
Ac-programming
ac-programming.com › content › Prom › Functions › label.html
Prometheus Label Functions
The regular expression "(.*)" captures the value of "source_label" and assigns it to "new_label". ... label_join(my_metric{job="job1"}, "new_label", ",", "label1", "label2") This example joins the values of "label1" and "label2" with a comma (",") as the delimiter and assigns the result to ...
🌐
Promlabs
promlabs.com › promql-cheat-sheet
PromLabs | PromQL Cheat Sheet
Include any label sets that are either on the left or right side: up{job="prometheus"} or up{job="node"} Open in PromLens · Include any label sets that are present both on the left and right side: node_network_mtu_bytes and (node_network_a...
🌐
Medium
sean22492249.medium.com › prometheus-label-replace-label-join-label-values-002efbdec87e
Prometheus label_replace, label_join, label_values | by Kiwi lee | Medium
June 28, 2024 - label_join(up, "combination", ",", "job", "instance") #Before up{instance="localhost:9090", job="prometheus"} #After up{instance="localhost:9090", job="prometheus", combination="prometheus,localhost:9090"} 將 job, instance 合併後,再擷...
🌐
Robust Perception
robustperception.io › left-joins-in-promql
Left joins in PromQL – Robust Perception | Prometheus Monitoring Experts
December 30, 2019 - PromQL by default effectively does an inner join and requires that labels (except the metric name) on both sides of the expression to be exactly the same and match one-to-one. There's no way in PromQL to increase your cardinality (other than absent(), which can only go from 0 to 1).