Since version 2.7 (Jan 2019), Prometheus supports sub-queries:
max_over_time( sum(node_memory_MemFree_bytes{instance=~"foobar.*"})[1d:1h] )
(metric for the past 2 days, with a resolution of 1 hour.)
Read the documentation for more informations on using recording rules: https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery
However, do note the blog recommendation:
Epilogue
Though subqueries are very convenient to use in place of recording rules, using them unnecessarily has performance implications. Heavy subqueries should eventually be converted to recording rules for efficiency.
It is also not recommended to have subqueries inside a recording rule. Rather create more recording rules if you do need to use subqueries in a recording rule.
The use of recording rules is explained in brian brazil article: https://www.robustperception.io/composing-range-vector-functions-in-promql/
Answer from Franklin Piat on Stack OverflowSince version 2.7 (Jan 2019), Prometheus supports sub-queries:
max_over_time( sum(node_memory_MemFree_bytes{instance=~"foobar.*"})[1d:1h] )
(metric for the past 2 days, with a resolution of 1 hour.)
Read the documentation for more informations on using recording rules: https://prometheus.io/docs/prometheus/latest/querying/examples/#subquery
However, do note the blog recommendation:
Epilogue
Though subqueries are very convenient to use in place of recording rules, using them unnecessarily has performance implications. Heavy subqueries should eventually be converted to recording rules for efficiency.
It is also not recommended to have subqueries inside a recording rule. Rather create more recording rules if you do need to use subqueries in a recording rule.
The use of recording rules is explained in brian brazil article: https://www.robustperception.io/composing-range-vector-functions-in-promql/
This isn't possible in one expression, you need to use a recording rule for the intermediate expression. See https://www.robustperception.io/composing-range-vector-functions-in-promql/
prometheus - Promql difference between max and max_over_time - Stack Overflow
Actual timestamp() for max_over_time (and other agg functions)
[Question] Two different values for the same day when calculating max_over_time over two different time ranges
How to get maximum of a sum over a period of time
I am tracking the number of jobs in a queue at specific time intervals using a gauge metric. Prometheus scrapes this every minute.
However, when I attempt to determine the highest number of jobs in the queue on a given day using the max_over_time query, I receive two distinct values for the same day based on different time ranges.
I am using the query max_over_time(job_count_by_service{service="ServiceA", tenant="TenantA"}[1d]). When I run this query for a 1-day time range (from 2023-08-19 00:00:00 to 2023-08-19 23:59:59), the value I get is 38. However, when I run the same query for a 5-day time range (from 2023-08-18 00:00:00 to 2023-08-22 23:59:59), the result for Aug 19th is 35.
https://i.stack.imgur.com/RSxCO.png https://i.stack.imgur.com/gmW3m.png
In Grafana I have configured the Min Step as 1d and Type as Range. I'm not sure whether that could affect the values in any way.
I assumed that max_over_time would pick the max value among all the values that fall in the range vector specified time period. For example, if on Day 1 the values are [1,2,7,6,5] and on Day 2 the values are [8,1,2,3,1] then the query would return 7 & 8 respectively for each day.
Prometheus seems to hide metrics that happen between the scrape interval, and I can't find anyone who looked into this. Statsd has a type called "Timing" which is like a gauge, but also stores min, max, avg, and stddev of all values submitted to it, aggregated by time interval. This is useful when you want to measure latency of some process that happens faster than the scrape interval. They also get computed correctly when you plot them on graphs and zoom out, showing fewer points.
This also brings up the question about existing metrics like CPU and Memory usage. If the scrape interval is 30 seconds, is the CPU usage metric (say from k8s node exporter) the value sampled at that time only (so last), is it the average of CPU usage from the last scrape window, or something else? (I know that in this example the metric `node_cpu_seconds_total` is a counter, and from the start has no statistical data attached to it)
For some metrics, like CPU usage, looking at an average can be very misleading, as the data tends to be very spiky. This already happens when zooming out in Grafana. For these metrics when aggregating time intervals you almost always want to use max all the time. `max_over_time()` is incompatible with `rate()`.
The result is that zooming out in Grafana shows basically lies, as showing averages make it seem like everything is fine.