Difference between max and max_over_time
max returns maximum among time series returned by inner selector. Result is single time series.
max_over_time returns maximum value of every time series over specified range. Result contains number of values equal to number of inner time series.
Consider this demo.
Same logic applies to all <aggregation> vs. <aggregation>_over_time functions.
Maximum CPU usage
I want to know, what is the maximum CPU usage in last 7 days using promql for each instance.
First of all I'm assuming here you are using node_exporter's metrics, and the following message will be based on this assumption, though I don't know what service="x" means in your selector. Hopefully it is just static label applied to target.
Regarding calculation of CPU utilization everything is not as simple as you tried.
I believe, simplest query to get CPU utilization of an instance is this:
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100)
Taken from this answer.
So, to get maximum value over last week you could use this:
max_over_time((100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m])) * 100))[7d:])
[Question] Two different values for the same day when calculating max_over_time over two different time ranges
kubernetes - Calculate Max over time on sum function in Prometheus - Stack Overflow
prometheus - How to use rate to handle max_over_time - Stack Overflow
How to make a max_over_time([30d]) over a avg_over_time([1min])
I am tracking the number of jobs in a queue at specific time intervals using a gauge metric. Prometheus scrapes this every minute.
However, when I attempt to determine the highest number of jobs in the queue on a given day using the max_over_time query, I receive two distinct values for the same day based on different time ranges.
I am using the query max_over_time(job_count_by_service{service="ServiceA", tenant="TenantA"}[1d]). When I run this query for a 1-day time range (from 2023-08-19 00:00:00 to 2023-08-19 23:59:59), the value I get is 38. However, when I run the same query for a 5-day time range (from 2023-08-18 00:00:00 to 2023-08-22 23:59:59), the result for Aug 19th is 35.
https://i.stack.imgur.com/RSxCO.png https://i.stack.imgur.com/gmW3m.png
In Grafana I have configured the Min Step as 1d and Type as Range. I'm not sure whether that could affect the values in any way.
I assumed that max_over_time would pick the max value among all the values that fall in the range vector specified time period. For example, if on Day 1 the values are [1,2,7,6,5] and on Day 2 the values are [8,1,2,3,1] then the query would return 7 & 8 respectively for each day.
Since version 2.7 (Jan 2019), Prometheus supports sub-queries:
max_over_time( sum(node_memory_MemFree_bytes{instance=~"foobar.*"})[1d:1h] )
(metric for the past 2 days, with a resolution of 1 hour.)
Read the documentation for more informations on using recording rules: https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery
However, do note the blog recommendation:
Epilogue
Though subqueries are very convenient to use in place of recording rules, using them unnecessarily has performance implications. Heavy subqueries should eventually be converted to recording rules for efficiency.
It is also not recommended to have subqueries inside a recording rule. Rather create more recording rules if you do need to use subqueries in a recording rule.
The use of recording rules is explained in brian brazil article: https://www.robustperception.io/composing-range-vector-functions-in-promql/
This isn't possible in one expression, you need to use a recording rule for the intermediate expression. See https://www.robustperception.io/composing-range-vector-functions-in-promql/
Hello everyone I would like to get in a graph the maximum of a moving average, But simply nesting the two functions inside each other doesn't work.
max_over_time(avg_over_time(meter_amp{instance=~"$group_id-.*"}[1m])[30d])
Error Msg: : 1:89: parse error: ranges only allowed for vector selectorsIs there another solution for this?