It is possible using unless operator:

item_stock unless item_purchases

Results in a vector consisting of the elements of item_stock for which there are no elements in item_purchases with exactly matching label sets. All matching elements in both vectors are dropped.

Note that the metrics in question do not have 'exactly matching label sets' (name is common but count and in_stock are not). In this case you can use on to set the list of labels to match:

item_stock unless on(name) item_purchases
Answer from anemyte on Stack Overflow
🌐
GitHub
github.com › prometheus › prometheus › issues › 5841
Promql outer join support · Issue #5841 · prometheus/prometheus
August 7, 2019 - Proposal Outer join (especially left join & right join) are very useful. For example: We have one host level CPU alert (CPU > 70%), and we have another spatial aggregation metrics on pool level (host count in a pool). If we wants to get ...
Author: prometheus
Discussions

Prometheus PromQL Outer Join - Stack Overflow
Communities for your favorite technologies. Explore all Collectives · Stack Overflow for Teams is now called Stack Internal. Bring the best of human thought and AI automation together at your work More on stackoverflow.com
🌐 stackoverflow.com
August 23, 2020
prometheus - Is there a way to express a true "left join" in PromQL? - Stack Overflow
Note that similar-sounding question Is there a way to do a "left outer join" like query in PromQL? is actually asking for an anti-join, which is often expressed as a left join with a filter on IS NULL for the RHS. More on stackoverflow.com
🌐 stackoverflow.com
Proposal: PromQL arithmetic with default value, or outer join
Proposal If I have data in foo and data in bar, then foo + bar will add them up. If there are some gaps in foo or bar then foo + bar will have gaps in all of those places. I would like to add them ... More on github.com
🌐 github.com
46
February 21, 2024
Making joins simpler in PromQL
At yesterday's dev summit, and as part of the discussion around prometheus/snmp_exporter#180 , @brian-brazil also suggested making joins easier to write. This issue is to document possible chan... More on github.com
🌐 github.com
18
August 20, 2017
🌐
Stack Overflow
stackoverflow.com › questions › 63544806 › prometheus-promql-outer-join
Prometheus PromQL Outer Join - Stack Overflow
August 23, 2020 - I want to use the fields attnid and hhmm to match/join but I do not want to sum/count by hhmm at the end.
🌐
Robust Perception
robustperception.io › left-joins-in-promql
Left joins in PromQL – Robust Perception | Prometheus Monitoring Experts
December 30, 2019 - Another difference here is that PromQL binary operators work on expressions, whereas a SELECT statement works on tables. So PromQL's joins are more like doing a join over two SQL subqueries rather than than two tables so can have more expressive power.
Top answer
1 of 1
1

Best workaround: union with "or"

The closest I've come uses set operations to execute the query as two branches - one for the "inner join" side, and one that's an anti-join that returns only LHS rows that don't match the RHS. This is very verbose if the LHS is a nontrivial expression instead of a simple stand-alone metric, and it's really inefficient too.

(
  important_metric{}
  * on (join_labels)
    group_left(info_label)
    group by (join_labels, info_label) (
      workload_priorities_info{}
    )
)
# if there's no series on the LHS with the same labels as the RHS after ignoring
# the "info_label" label on each side, add the RHS series to the output, otherwise
# output the LHS series.
or ignoring (info_label)
important_metric{}

with inputs

important_metric{join_labels="1"} 100
important_metric{join_labels="2"} 64
important_metric{join_labels="3"} 5

workload_priorities_info{join_labels="1",info_label="highprio"}
workload_priorities_info{join_labels="3",info_label="lowprio"}

this produces

{join_labels="1",info_label="highprio"} 100
{join_labels="2"} 64
{join_labels="3",info_label="lowprio"} 5

Based on Proposal: PromQL arithmetic with default value, or outer join (issue#13625) this is probably the best workaround that is currently possible.

SQL analogues for comparison

The SQL translation of this would be something like

# inner join to find rows with matching info metrics
SELECT
  important_metric.value * workload_priorities_info.value,
  important_metric.join_labels,
  workload_priorities_info.info_label
FROM important_metric
INNER JOIN workload_priorities_info USING (join_labels)
# append two sets of rows
UNION ALL
# find rows that do NOT have any matching info-metric and append them to
# the result set
SELECT important_metric.value,
  important_metric.join_labels,
  NULL
FROM important_metric
LEFT JOIN INNER JOIN workload_priorities_info USING (join_labels)
WHERE workload_priorities_info.info_label IS NULL

where what we want is the equivalent of

SELECT
  important_metric.value * coalesce(workload_priorities_info.value, 1),
  important_metric.join_labels,
  workload_priorities_info.info_label
FROM important_metric
LEFT JOIN workload_priorities_info USING (join_labels)
🌐
Iximiuz
iximiuz.com › en › posts › prometheus-vector-matching
Prometheus Cheat Sheet - How to Join Multiple Metrics (Vector Matching)
November 30, 2021 - How to join metrics on matching labels? Vector matching: one-to-one, one-to-many and many-to-one, many-to-many. What on/ignoring clause is for? group_left and group_right explained. Logical/set operations explained.
Find elsewhere
🌐
Reddit
reddit.com › r › PrometheusMonitoring › comments › ehjyl0 › left_joins_in_promql
r/PrometheusMonitoring - Left joins in PromQL
December 30, 2019 - 5.3k members in the PrometheusMonitoring community. Prometheus Monitoring subreddit
🌐
The Mail Archive
mail-archive.com › prometheus-users@googlegroups.com › msg05033.html
[prometheus-users] Re: outer join
September 23, 2020 - https://www.robustperception.io/left-joins-in-promql · Build the query up in steps. I am guessing you want the "or on" condition inside the max by, not outside it. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
🌐
GitHub
github.com › prometheus › prometheus › issues › 13625
Proposal: PromQL arithmetic with default value, or outer join · Issue #13625 · prometheus/prometheus
February 21, 2024 - Proposal: PromQL arithmetic with default value, or outer join#13625 · #17644 · Copy link · Labels · component/promqlhelp wantedkind/featurenot-as-easy-as-it-lookspriority/P3 · bboreham · opened · on Feb 21, 2024 · Issue body actions · If I have data in foo and data in bar, then foo + bar will add them up.
Author: prometheus
🌐
GitHub
github.com › prometheus › prometheus › issues › 3094
Making joins simpler in PromQL · Issue #3094 · prometheus/prometheus
August 20, 2017 - At yesterday's dev summit, and as part of the discussion around prometheus/snmp_exporter#180 , @brian-brazil also suggested making joins easier to write. This issue is to document possible chan...
Author: prometheus
🌐
Felixhummel
blag.felixhummel.de › 25 › 08-01-til-prometheus-promql-left-join.html
TIL: Prometheus (PromQL) LEFT JOIN — Felix' Wobsite
August 1, 2025 - SELECT * FROM node_exporter_build_info LEFT JOIN node_os_info ON a.foo = b.foo AND a.bar = b.bar · group by (instance, version) (node_exporter_build_info) * on (instance) group_left(name, version_codename, version_id) node_os_info · This reduces rows to those matching labels in node_os_info. A table of node_exporter versions grouped by instance and version: SELECT * FROM node_exporter_build_info LEFT OUTER JOIN node_os_info ON a.foo = b.foo AND a.bar = b.bar ·
🌐
Google Groups
groups.google.com › g › prometheus-users › c › LgGTcR-jQvs
Join by field (outer join) with different names
October 12, 2020 - label_replace(lldpLocPortDesc, "ifIndex", "$1", "lldpLocPortNum", "(.+)") If that works, then you should be able to build a query on top of this which uses (instance, ifIndex) as the join labels.
🌐
Wavefront
docs.wavefront.com › query_language_series_joining.html
Combining Time Series With join() | VMware Aria Operations for Applications Documentation
When a new series is produced by combining a pair of matching input series, the data points in the new series are derived as for an inner join. When a new series (such as C6) is produced from a single right-hand input series, the data points in the new series are derived using the special syntax in the output data expression. The sample output data expression {ts1|0} / ts2 says to divide the right-hand series into 0 whenever there is no matching left-hand series, as is the case for C6. You must use the special syntax in a right outer join to provide alternate values for the left-hand series, which might be missing.
🌐
SigNoz
signoz.io › guides › how can i 'join' two metrics in a prometheus query? - joining metrics in promql
How can I 'join' two metrics in a Prometheus query? - Joining Metrics in PromQL | SigNoz
July 24, 2024 - Practice and experimentation are key to mastering metric joining in Prometheus · PromQL doesn't have explicit inner and outer join operations. Instead, it uses vector matching with modifiers like group_left and group_right to achieve similar ...
Top answer
1 of 3
76

You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):

(
  max(consul_health_service_status{status="critical"}) 
  by (service_name,status,node) == 1
)
   + on(service_name,node) group_left(env)
(
   0 * consul_service_tags
)

The important part here is the operation + on(service_name,node) group_left(env):

  • the + is "abused" as a join operator (fine since 0 * consul_service_tags always has the value 0)
  • group_left(env) is the modifier that includes the extra label env from the right (consul_service_tags)
2 of 3
15

It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.

The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.

For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:

consul_health_service_status
  * on(service_name, node) group_left(env)
consul_service_tags

Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:

consul_health_service_status{status="critical"}
  * on(service_name, node) group_left(env)
consul_service_tags
🌐
Stack Overflow
stackoverflow.com › questions › 61481671 › promql-multi-join
prometheus - PromQL multi join - Stack Overflow
April 28, 2020 - 2 left join prometheus metrics in grafana? 4 Nesting query in promQL · 0 Combine multiple Prometheus queries in Grafana · 5 Is there a way to do a "left outer join" like query in PromQL? 0 Join two prometheus queries · 0 How to Join two series in Prometheous using operator + 1 Grafana prometheus outer join for multiple values with same name ·