For now the only solution I have is with query duplication:
(
(sum(kafka_consumergroup_lag[1d]) by (consumergroup) >= 100)
* on(consumergroup) group_left(team)
catalog_entities_info
)
or ignoring(team) (sum(kafka_consumergroup_lag[1d]) by (consumergroup) >= 100)
First half will add team label where possible, and second or half will add missed data (important to merge vectors while ignoring added team label, otherwise we will have duplicates).
What is PromQL?
How do I filter by label values in PromQL?
How do I predict future resource usage in PromQL?
You can use the argument list of group_left to include extra labels from the right operand (parentheses and indents for clarity):
(
max(consul_health_service_status{status="critical"})
by (service_name,status,node) == 1
)
+ on(service_name,node) group_left(env)
(
0 * consul_service_tags
)
The important part here is the operation + on(service_name,node) group_left(env):
- the
+is "abused" as a join operator (fine since0 * consul_service_tagsalways has the value 0) group_left(env)is the modifier that includes the extra labelenvfrom the right (consul_service_tags)
It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.
The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.
For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:
consul_health_service_status
* on(service_name, node) group_left(env)
consul_service_tags
Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:
consul_health_service_status{status="critical"}
* on(service_name, node) group_left(env)
consul_service_tags