You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
- the
+is "abused" as a join operator (fine since0 * consul_service_tagsalways has the value 0) group_left(env)is the modifier that includes the extra labelenvfrom the right (consul_service_tags)
monitoring - How can I 'join' two metrics in a Prometheus query? - Stack Overflow
Mechanism Search based on new label created through label_join function
simpler syntax for joins on labels with different names
Allow label_join to be used in aggregation operators
You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
- the
+is "abused" as a join operator (fine since0 * consul_service_tagsalways has the value 0) group_left(env)is the modifier that includes the extra labelenvfrom the right (consul_service_tags)
It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.
The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.
For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:
consul_health_service_status
* on(service_name, node) group_left(env)
consul_service_tags
Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:
consul_health_service_status{status="critical"}
* on(service_name, node) group_left(env)
consul_service_tags