Implement PromQL offset modifier for range vector in LogQL
promql - Prometheus "negative" offset - Stack Overflow
prometheus - Difference between two metrics with time offset in PromQL - Stack Overflow
prometheus - Avoiding multiple subqueries in PromQL with offset to get historic averages - Stack Overflow
Hi All,
Some of our Prometheus monitoring uses 10-week rolling averages, which was set up a couple months ago, like so:
round((sum(increase(metric_name[5m])))
/
(
(sum(increase(metric_name[5m] offset 1w)) +
sum(increase(metric_name[5m] offset 2w)) +
sum(increase(metric_name[5m] offset 3w)) +
sum(increase(metric_name[5m] offset 4w)) +
sum(increase(metric_name[5m] offset 5w)) +
sum(increase(metric_name[5m] offset 6w)) +
sum(increase(metric_name[5m] offset 7w)) +
sum(increase(metric_name[5m] offset 8w)) +
sum(increase(metric_name[5m] offset 9w)) +
sum(increase(metric_name[5m] offset 10w))
)
/10), 0.01)This worked great, until US Daylight Saving Time rolled back, at which point the comparisons we are doing aren't accruate anymore. Now, after some fiddling around, I've figured out how to make a series of recording rules that spits out a DST-adjusted number of hours for the offset like so (derived from https://github.com/abhishekjiitr/prometheus-timezone-rules):
# Determines appropriate time offset (in hours) for 1 week ago, accounting for US Daylight Saving Time for the America/New_York time zone (vector(168) and (Time:AmericaNewYork:Is1wAgoDuringDST == 1 and Time:AmericaNewYork:IsNowDuringDST == 1)) # Normal value for when comparison time and the current time are both in DST or (vector(168) and (Time:AmericaNewYork:Is1wAgoDuringDST == 0 and Time:AmericaNewYork:IsNowDuringDST == 0)) # Normal value for when comparison time and the current time are both outside DST or (vector(167) and (Time:AmericaNewYork:Is1wAgoDuringDST == 0 and Time:AmericaNewYork:IsNowDuringDST == 1)) # Minus 1 hour for when time has "sprung forward" between the comparison time and the current time or (vector(169) and (Time:AmericaNewYork:Is1wAgoDuringDST == 1 and Time:AmericaNewYork:IsNowDuringDST == 0)) # Plus 1 hour for when time has "fallen back" between the comparison time and the current time
The problem is: I can't figure out a way to actually use this value with the offset modifier as in the first code block above.
Is anyone aware if such a thing is possible? I can fall back to making custom recording rules for averages for each metric we're alerting on this way, but that's obviously a lot of work.
time() % 86400 will give the number of seconds since midnight UTC.
When you are using grafana, you can tryout something like this: delta(your_counter[${__range_s}s]). There are several special variables in grafana: https://grafana.com/docs/grafana/latest/variables/variable-types/global-variables/