I've implemented by using label_replace for renaming label.
Answer from kolchanov on Stack Overflowmonitoring - How can I 'join' two metrics in a Prometheus query? - Stack Overflow
simpler syntax for joins on labels with different names
prometheus - PromQL join with identical label value but different label name - Stack Overflow
prometheus - PromQL join two metrics with identic label names but different label values - Stack Overflow
You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
- the
+is "abused" as a join operator (fine since0 * consul_service_tagsalways has the value 0) group_left(env)is the modifier that includes the extra labelenvfrom the right (consul_service_tags)
It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.
The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.
For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:
consul_health_service_status
* on(service_name, node) group_left(env)
consul_service_tags
Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:
consul_health_service_status{status="critical"}
* on(service_name, node) group_left(env)
consul_service_tags
The main roadblock you are probably hitting is the fact that the in both metrics you have the status label which is different AND the one you want to replicate. Prometheus won't be able resolve it while the label names collide.
The first thing you want to do is generate a new timeseries with a different label name for the second status. You can do this by using label_replace. For example:
label_replace(metric2{...}, "metric2_status", "$1", "status", "(.*)")
This will keep the status label but will also add a metric2_status with the value of the status label.
Now you can proceed to use on() and group_left(). These need to be used with an arithmetic or group function. If you are using this to add more labels to metric1, remember to use an arithmetic that will not alter the metric value.
For example - assuming metric2 has always a value of 1 - this is what you could use:
metric1{label_1="a", ...} *
on (label_a, label_b)
group_left(metric2_status)
label_replace(metric2{...}, "metric2_status", "$1", "status", "(.*)")
the resulting metric should have metric1's labels plus the metric2_status label.
When using group modifiers such as
ignoring()
on()
group_left()
group_right()
the label(s) used in on() modifier must be unique at least in one vector.
Consider metric1, metric2 vectors and common label_a=... label used in on(). These cases may occur
If metric1 has unique label_a=... and metric2 has unique label_a=... we have one-to-one match.
If metric1 has duplicate label_a=... and metric2 has unique label_a=... we have many-to-one match.
If metric1 has unique label_a=... and metric2 has duplicate label_a=... we have one-to-many match.
If metric1 has duplicate label_a=... and metric2 has duplicate label_a=... we have many-to-many match and vectors are not joinable.
Additionally the result vector must have unique set of label keys. That's why we are renaming the label in the right-hand side vector and joining it as a label with different name.
Working query is
( metric1{...} == 1 ) *
on(label_a)
group_left(metric2_status)
label_replace((metric2{...} == 1), "metric2_status", "$1", "status", "(.*)")
( metric1{...} == 1 )
Filter metric1 so that the left-hand side of the query contains only data with unique label_a=....
on(label_a)
join two metrics on label_a=....
group_left(metric2_status)
add metric2_status label from right-hand metric2.
label_replace((metric2{...} == 1), "metric2_status", "$1", "status", "(.*)")
create new label metric2_status in metrics2 from the status label.