by(), without()
The max() aggregation (as well other aggregations) on its own discards all the labels and the metric name, and provides the aggregated result with no labels, since the max is selected across all the time series. It is no longer the sys_cpu_host_seconds_total{mode=sys} or sys_cpu_host_seconds_total{mode=user}, but the max value of them.
To control the labels in an aggregation you have 2 clauses to use:
- without() - removes the listed labels from the result vector, while all other labels are preserved in the output
F.e. if he metric has 2 labels mode and job and you want to preserve the job label while take the max among mode-s you could use:
max(sys_cpu_host_seconds_total)without(mode)
The result vector will return the max value preserving all the rest labels but the mode:
{job='demo-1'} 3065880.72
{job='demo-2'} 1760763.05
- by() - does the opposite and drops labels that are not listed in the by clause, even if their label values are identical between all elements of the vector
F.e. if he metric has 2 labels mode and job and you want to preserve the job label while take the max among job-s you could use:
max(sys_cpu_host_seconds_total)by(job)
The result vector will be the same, but it contains only the job label:
{job='demo-1'} 3065880.72
{job='demo-2'} 1760763.05
"__name__" label
The trick with the metric name is that internally it is a label with the special name of "__name__". So you could use it to preserve the metric name if you really need it:
max(sys_cpu_host_seconds_total)by(__name__, job)
The result vector will "preserve" the metric name, however it's just a trick since the result is no longer the metric you aggregate on, but the aggregation result:
sys_cpu_host_seconds_total{job='demo-1'} 3065880.72
sys_cpu_host_seconds_total{job='demo-2'} 1760763.05
topk(1, ...)
topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector, so you coul use this trick to preserve all the labels along with the name:
topk(1, sys_cpu_host_seconds_total)
The result vector will "preserve" all the labels along with the metric name:
sys_cpu_host_seconds_total{mode='sys', job='demo-1'} 3065880.72
by(), without()
The max() aggregation (as well other aggregations) on its own discards all the labels and the metric name, and provides the aggregated result with no labels, since the max is selected across all the time series. It is no longer the sys_cpu_host_seconds_total{mode=sys} or sys_cpu_host_seconds_total{mode=user}, but the max value of them.
To control the labels in an aggregation you have 2 clauses to use:
- without() - removes the listed labels from the result vector, while all other labels are preserved in the output
F.e. if he metric has 2 labels mode and job and you want to preserve the job label while take the max among mode-s you could use:
max(sys_cpu_host_seconds_total)without(mode)
The result vector will return the max value preserving all the rest labels but the mode:
{job='demo-1'} 3065880.72
{job='demo-2'} 1760763.05
- by() - does the opposite and drops labels that are not listed in the by clause, even if their label values are identical between all elements of the vector
F.e. if he metric has 2 labels mode and job and you want to preserve the job label while take the max among job-s you could use:
max(sys_cpu_host_seconds_total)by(job)
The result vector will be the same, but it contains only the job label:
{job='demo-1'} 3065880.72
{job='demo-2'} 1760763.05
"__name__" label
The trick with the metric name is that internally it is a label with the special name of "__name__". So you could use it to preserve the metric name if you really need it:
max(sys_cpu_host_seconds_total)by(__name__, job)
The result vector will "preserve" the metric name, however it's just a trick since the result is no longer the metric you aggregate on, but the aggregation result:
sys_cpu_host_seconds_total{job='demo-1'} 3065880.72
sys_cpu_host_seconds_total{job='demo-2'} 1760763.05
topk(1, ...)
topk and bottomk are different from other aggregators in that a subset of the input samples, including the original labels, are returned in the result vector, so you coul use this trick to preserve all the labels along with the name:
topk(1, sys_cpu_host_seconds_total)
The result vector will "preserve" all the labels along with the metric name:
sys_cpu_host_seconds_total{mode='sys', job='demo-1'} 3065880.72
You should use topk for selecting the metric with the maximum value:
topk(1,
sys_cpu_host_seconds_total{mode="sys"}
or sys_cpu_host_seconds_total{mode="user"}
)
Note that topk(k, q) query can return more than k time series when this query is used for building a graph in Grafana. This is because topk(k, q) independently selects top k series per each timestamp displayed on the graph. If you want up to k series with max malues to be displayed on the graph, then take a look at topk_max, topk_avg and other topk_* functions in MetricsQL - this is PromQL-compatible query language in Prometheus-like system I work on.
What is PromQL?
How do I predict future resource usage in PromQL?
How do I filter by label values in PromQL?
It is possible.
Example query:
max_over_time(
irate( messages_in_total[2m] )[1d:1m]
)
This will:
- take last 1 day
- For every 1 minute in that 1 day range it will execute
irate( messages_in_total[2m] ) - Combine that into range vector
- Call max_over_time on all results
See subquery documentation for more information!
While the answer returns the maximum per-second rate over the last 24 hours for messages_in_total metric, it has the following potential issues:
- It may skip a part of raw samples if the interval between them (aka
scrape_interval) is smaller than one minute. This can be fixed by reducing thestepvalue in square brackets after the colon, so it doesn't exceed thescrape_interval. - It may return an empty result or incomplete result if the scrape interval exceeds 2m (e.g. 2 minutes). This can be fixed by increasing the lookbehind window in the inner square brackets from
2mto the value exceeding 2xscrape_interval. - It may become very slow and resource hungry because of subquery overhead.
- Subqueries are easy to mis-use, so they would silently return unexpected results.
While Prometheus doesn't provide the reliable and easy to use solution for these issues, other Prometheus-like systems may have the solution. For example, the following MetricsQL query returns the maximum, the minimum and the average per-second increase rates for messages_in_total time series for the last 24 hours:
rollup_rate(messages_in_total[1d])
It uses rollup_rate function.
If you need only the maximum per-second rate, then the query can be wrapped into label_match function, which leaves only time series with rollup="max" label:
label_match(
rollup_rate(messages_in_total[1d]),
"rollup", "max"
)
Since version 2.7 (Jan 2019), Prometheus supports sub-queries:
max_over_time( sum(node_memory_MemFree_bytes{instance=~"foobar.*"})[1d:1h] )
(metric for the past 2 days, with a resolution of 1 hour.)
Read the documentation for more informations on using recording rules: https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery
However, do note the blog recommendation:
Epilogue
Though subqueries are very convenient to use in place of recording rules, using them unnecessarily has performance implications. Heavy subqueries should eventually be converted to recording rules for efficiency.
It is also not recommended to have subqueries inside a recording rule. Rather create more recording rules if you do need to use subqueries in a recording rule.
The use of recording rules is explained in brian brazil article: https://www.robustperception.io/composing-range-vector-functions-in-promql/
This isn't possible in one expression, you need to use a recording rule for the intermediate expression. See https://www.robustperception.io/composing-range-vector-functions-in-promql/