It is a good practice in Prometheus ecosystem to expose additional labels, which can be joined to multiple metrics, via a separate info-like metric as explained in this article. For example, consul_service_tags metric exposes a set of tags, which can be joined to metrics via (service_name, node) labels.

The join is usually performed via on() and group_left() modifiers applied to * operation. The * doesn't modify values for time series on the left side because info-like metrics usually have constant 1 values. The on() modifier is used for limiting the labels used for finding matching time series on the left and the right side of *. The group_left() modifier is used for adding additional labels from time series on the right side of *. See these docs for details.

For example, the following PromQL query adds env label from consul_service_tags metric to consul_health_service_status metric with the same set of (service_name, node) labels:

consul_health_service_status
  * on(service_name, node) group_left(env)
consul_service_tags

Additional label filters can be added to consul_health_service_status if needed. For example, the following query returns only time series with status="critical" label:

consul_health_service_status{status="critical"}
  * on(service_name, node) group_left(env)
consul_service_tags

I got tired of all the fragmented documentation and I feel I provided a better answer in this post here: https://medium.com/@texasdave2/replace-and-remove-a-label-in-a-prometheus-query-9500faa302f0

Replace is not a true REPLACE

Your goal is to simply replace the old label name “old_job_id” with a new label name “new_task_id”. Prometheus label_replace will really “add” the new label name. It will preserve the old label name as well… So, that could be a problem, it’s not a true “replace in place”.

So if you want to “add” your new label name and “remove” the old label name, you need to do this:

sum without (old_job_id) (label_replace(metric, "new_task_id", "$1", "old_job_id", "(.*)"))

Here’s how this reads:

• sum without (old_job_id) will remove the old label name from the query output

• metric is your metric, like “node_filesystem_avail_bytes”

• “new_task_id” is where you would put your new label name

• “$1” is regex for using the string in new label name, don’t change this

• “old_job_id” is where you’ll put your old label, the one you want to get rid of (.*……. that mess is regex that will replace the whole label name