AFAICT the cause for the weird results is (1) the fact that your counter actually only increases once every minute, even though you collect it every 15 seconds combined with (2) Prometheus' rate() implementation discarding every 4th counter increase (in your particular setup).
More precisely, you appear to be computing a 1 minute rate, every 1 minute over a counter scraped at 15 second resolution, increasing every 1 minute (on average).
What this means essentially is that Prometheus will basically slice your 1 hour interval into disjoint 1 minute ranges and estimate the rate over each range. The first value will be the extrapolated rate of increase between points 0 and 3, the second will be the extrapolated rate between points 4 and 7 and so on. Because your counter only actually increases once a minute, you can run into 2 different situations:
- Your counter increases happen between point pairs 3-4, 7-8 etc. In this case Prometheus sees an increase rate of zero (because there is no increase between points 0 and 3, points 4 and 7 etc. This seems to be happening in the first half of your first graph.
- Your counter increases happen somewhere between points 0-3, 4-7 etc. In this case Prometheus takes the difference between the last and first points in each interval (your actual counter increase), divides it by the time difference between the 2 points (on average 45 seconds), then extrapolates that to 1 minute (essentially overestimating it by a factor of 1.(3) -- I'm eyeballing an increase of ~200k over ~50 minutes, so an average rate of about 67 QPS, whereas
rate()returns something closer to 90 QPS). This is what happens in the second half of your graph.
This is also why your graph looks wildly different across refreshes. The argument for the current implementation of rate() is that it is "correct on average". Which, if you look at the whole of your graph, across refreshes, is true. </sarcasm>
Essentially graphing a Prometheus rate() or increase() over a time range R with resolution R will result in aliasing, either overestimating (1.33x in your case) or underestimating (zero in your case) on anything but a smoothly increasing counter.
You can work around it by replacing your expression with:
rate(foo[75s]) / 75 * 60
This way you'll actually get the rate of increase between data points 1 minute apart (a 75 seconds range will almost always return exactly 5 points, so 4 counter increases) and reverse the extrapolation to 75 seconds that Prometheus does. There will be some noise in edge cases (e.g. if your evaluation is aligned with scraping times it's possible to get 6 points in one range and 4 in the next due to scrape interval jitter) but you're getting that anyway with rate().
BTW, you can see the aliasing by increasing the resolution of your graph to something like 1 second (anything 15 seconds or below should show it clearly).
Answer from Alin Sînpălean on Stack OverflowPrometheus rate functions and interval selections - Stack Overflow
what is the default grafana setting for $__rate_interval - Stack Overflow
How to set the rate interval in the PROMQL similar to $__rate_interval by grafana.
Prometheus Error Rate alert : interval range question - Stack Overflow
Can rate() be used with all types of Prometheus metrics?
How do you calculate request rates using the Prometheus rate function?
How does the Prometheus rate() function differ from increase()?
AFAICT the cause for the weird results is (1) the fact that your counter actually only increases once every minute, even though you collect it every 15 seconds combined with (2) Prometheus' rate() implementation discarding every 4th counter increase (in your particular setup).
More precisely, you appear to be computing a 1 minute rate, every 1 minute over a counter scraped at 15 second resolution, increasing every 1 minute (on average).
What this means essentially is that Prometheus will basically slice your 1 hour interval into disjoint 1 minute ranges and estimate the rate over each range. The first value will be the extrapolated rate of increase between points 0 and 3, the second will be the extrapolated rate between points 4 and 7 and so on. Because your counter only actually increases once a minute, you can run into 2 different situations:
- Your counter increases happen between point pairs 3-4, 7-8 etc. In this case Prometheus sees an increase rate of zero (because there is no increase between points 0 and 3, points 4 and 7 etc. This seems to be happening in the first half of your first graph.
- Your counter increases happen somewhere between points 0-3, 4-7 etc. In this case Prometheus takes the difference between the last and first points in each interval (your actual counter increase), divides it by the time difference between the 2 points (on average 45 seconds), then extrapolates that to 1 minute (essentially overestimating it by a factor of 1.(3) -- I'm eyeballing an increase of ~200k over ~50 minutes, so an average rate of about 67 QPS, whereas
rate()returns something closer to 90 QPS). This is what happens in the second half of your graph.
This is also why your graph looks wildly different across refreshes. The argument for the current implementation of rate() is that it is "correct on average". Which, if you look at the whole of your graph, across refreshes, is true. </sarcasm>
Essentially graphing a Prometheus rate() or increase() over a time range R with resolution R will result in aliasing, either overestimating (1.33x in your case) or underestimating (zero in your case) on anything but a smoothly increasing counter.
You can work around it by replacing your expression with:
rate(foo[75s]) / 75 * 60
This way you'll actually get the rate of increase between data points 1 minute apart (a 75 seconds range will almost always return exactly 5 points, so 4 counter increases) and reverse the extrapolation to 75 seconds that Prometheus does. There will be some noise in edge cases (e.g. if your evaluation is aligned with scraping times it's possible to get 6 points in one range and 4 in the next due to scrape interval jitter) but you're getting that anyway with rate().
BTW, you can see the aliasing by increasing the resolution of your graph to something like 1 second (anything 15 seconds or below should show it clearly).
What you say doesn't line up with the data, that raw data is only going up about once a minute. Are you sure you're scraping every 15s?
- See New in Grafana 7.2: $__rate_interval for Prometheus rate queries that just work.
- Rate is always per second. See Prometheus documentation for the rate function.
- Click on Query options, then click on the Info-Symbol. An explanation will be displayed.
- To get rate per minute, just multiply the rate with 60.
Edit: ($__rate_interval and $__interval)
Prometheus periodically fetches data from your application. Grafana periodically fetches Data from Prometheus. Grafana does not know, how often Prometheus polls your application for data. Grafana will estimate this time by looking at the configuration and assuming that every scrape gives us one data point. The $__interval variable then expands to the duration between two data points in the graph (Note that this is only true for small time ranges and high resolution as the intended use case for $__interval is reducing the number of data points when the time range is wide. See Approximate Calculation of $__interval.)
If the time-distance between every two data points in each series is 15 seconds, it does not make sense to use anything less than [15s] as interval in the rate function. The rate function works best with at least 4 data points. Therefore [1m] would be much better than anything betweeen [15s] and [1m]. This is what $__rate_interval tries to achieve: guessing a minimal sensible interval for the rate function.
Personally, I think, this does not always work if your application delivers sparse data (less than one data point per scrape). I prefer using fixed intervals like 10m or even 1h or 1d in these situations. The interval need to be great enough to get you enough data points for the metric to work with the rate function.
A different approach would be to use any of $__rate_interval and $__interval but also set the Min step parameter for the query in the Grafana UI to be big enough.
Just click button "Query inspector" and you will see detailed explanation for the query (Expr: section). In my case default value for $__rate_interval in Grafana is 1m0s.
Buried in the mass Prometheus docs, there is a paragraph for increase function:
increaseshould only be used with counters and native histograms where the components behave like counters. It is syntactic sugar forrate(v)multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability.
So answer your questions:
Is there a strong reason as why I should use
rateas opposed toincrease?Yes, use the
ratefunction.How valuable is to have the interval set for 5m?
Not so valuable. Since your RPS/QPS is very small - less than 10 per 5m, you may get some 5m time ranges with little or zero requests and others with much more requests. The alert rule will be too sensitive or just wrong in a wider time range view. 30m or 1h range might be better.
By the way, time series on each side of division operator should have matching labels to make the alert rule work.
It looks like you have e.g. slow-changing integer counter, which may increase by less than 100 during an hour. Prometheus can return unexpected results from increase() function when applied to slow-changing integer counters because of the following issues:
increase(m[d])may return fractional results over integer countermbecause of extrapolation. See this issue.increase(m[d])may miss counter increase between the last raw sample just before the lookbehind windowdand the first raw sample inside the lookbehind windowd. See this article for more details.increase(m[d])may miss the initial counter increase ifmtime series starts from value other than zero.
The same issues are applied to rate() as well, since increase() is a syntactic sugar over rate() in Prometheus, e.g. increase(m[d]) = rate(m[d]) * d.
It is recommended using longer lookbehind windows for rate() and increase() functions when they are applied to slow-changing counters, in order to minimize the significance of issues mentioned above. For example, to use 1h lookbehind window in square brackets instead of 5m, so the increased window catches non-zero counter increases.
As for the original query, it is better rewriting it to the following one:
(
sum(increase(errorMetric{service_name="someservice"}[1h]))
/
sum(increase(http_requests_count{service_name="someservice"}[1h]))
) > 0.05
This query contains the following changes comparing to the original query:
5mlookbehind window has been changed to1h- the
pathlabel filter has been removed from thehttp_requests_countmetric selector, so the number oferrorMetrictime series matches the number ofhttp_requests_counttime series. On the other hand, thepathlabel filter could be added toerrorMetricmetric selector instead.