In an ideal world (where your samples' timestamps are exactly on the second and your rule evaluation happens exactly on the second) rate(counter[1s]) would return exactly your ICH value and rate(counter[5s]) would return the average of that ICH and the previous 4. Except the ICH at second 1 is 0, not 1, because no one knows when your counter was zero: maybe it incremented right there, maybe it got incremented yesterday, and stayed at 1 since then. (This is the reason why you won't see an increase the first time a counter appears with a value of 1 -- because your code just created and incremented it.)
increase(counter[5s]) is exactly rate(counter[5s]) * 5 (and increase(counter[2s]) is exactly rate(counter[2s]) * 2).
Now what happens in the real world is that your samples are not collected exactly every second on the second and rule evaluation doesn't happen exactly on the second either. So if you have a bunch of samples that are (more or less) 1 second apart and you use Prometheus' rate(counter[1s]), you'll get no output. That's because what Prometheus does is it takes all the samples in the 1 second range [now() - 1s, now()] (which would be a single sample in the vast majority of cases), tries to compute a rate and fails.
If you query rate(counter[5s]) OTOH, Prometheus will pick all the samples in the range [now() - 5s, now] (5 samples, covering approximately 4 seconds on average, say [t1, v1], [t2, v2], [t3, v3], [t4, v4], [t5, v5]) and (assuming your counter doesn't reset within the interval) will return (v5 - v1) / (t5 - t1). I.e. it actually computes the rate of increase over ~4s rather than 5s.
increase(counter[5s]) will return (v5 - v1) / (t5 - t1) * 5, so the rate of increase over ~4 seconds, extrapolated to 5 seconds.
Due to the samples not being exactly spaced, both rate and increase will often return floating point values for integer counters (which makes obvious sense for rate, but not so much for increase).
In an ideal world (where your samples' timestamps are exactly on the second and your rule evaluation happens exactly on the second) rate(counter[1s]) would return exactly your ICH value and rate(counter[5s]) would return the average of that ICH and the previous 4. Except the ICH at second 1 is 0, not 1, because no one knows when your counter was zero: maybe it incremented right there, maybe it got incremented yesterday, and stayed at 1 since then. (This is the reason why you won't see an increase the first time a counter appears with a value of 1 -- because your code just created and incremented it.)
increase(counter[5s]) is exactly rate(counter[5s]) * 5 (and increase(counter[2s]) is exactly rate(counter[2s]) * 2).
Now what happens in the real world is that your samples are not collected exactly every second on the second and rule evaluation doesn't happen exactly on the second either. So if you have a bunch of samples that are (more or less) 1 second apart and you use Prometheus' rate(counter[1s]), you'll get no output. That's because what Prometheus does is it takes all the samples in the 1 second range [now() - 1s, now()] (which would be a single sample in the vast majority of cases), tries to compute a rate and fails.
If you query rate(counter[5s]) OTOH, Prometheus will pick all the samples in the range [now() - 5s, now] (5 samples, covering approximately 4 seconds on average, say [t1, v1], [t2, v2], [t3, v3], [t4, v4], [t5, v5]) and (assuming your counter doesn't reset within the interval) will return (v5 - v1) / (t5 - t1). I.e. it actually computes the rate of increase over ~4s rather than 5s.
increase(counter[5s]) will return (v5 - v1) / (t5 - t1) * 5, so the rate of increase over ~4 seconds, extrapolated to 5 seconds.
Due to the samples not being exactly spaced, both rate and increase will often return floating point values for integer counters (which makes obvious sense for rate, but not so much for increase).
Prometheus calculates rate(counter[d]) at timestamp t in the following way:
- It selects raw samples for the
countertime series on the time range(t-d ... t]. Note that thet-dtimestamp isn't included in the time range, whilettimestamp is included in the time range. If the selected time range contains less than two raw samples, then Prometheus returns an empty value (a gap) at the timestampt. - Then it calculates the increase of the selected raw samples. Usually it is calculated as the difference between the last selected sample and the first selected sample. Calculations become slightly complicated if the
counterwas reset to zero during the selected time range. Let's skip this for the sake of clarity. - Then the resulting increase can be extrapolated if timestamps for the first and/or the last raw samples are located too far from the bounds of the selected time range.
- Then the rate is calculated by dividing the extrapolated increase by
d.
Prometheus calculates increase(counter[d]) in the same way except the last step.
Let's look at a few examples applied to the original data:
second counter_value increase calculated by hand(call it ICH from now)
1 1 1
2 3 2
3 6 3
4 7 1
5 10 3
6 14 4
7 17 3
8 21 4
9 25 4
10 30 5
The
rate(counter[1s])will return nothing at any timestampt, since any time range(t-1s ... t]contains only a single raw sample, while Prometheus requires at least two samples for calculating bothrate()andincrease().The
rate(counter[2s])andincrease(counter[2])would return the following values per each timestamptwhen extrapolation isn't applied:
t counter_value rate(counter[2s]) increase(counter[2s])
1 1 - -
2 3 (3-1)/2=1.0 3-1=2
3 6 (6-3)/2=1.5 6-3=3
4 7 (7-6)/2=0.5 7-6=1
5 10 (10-7)/2=1.5 10-7=3
6 14 (14-10)/2=2 14-10=4
7 17 (17-14)/2=1.5 17-14=3
8 21 (21-17)/2=2 21-17=4
9 25 (25-21)/2=2 25-21=4
10 30 (30-25)/2=2.5 30-25=5
In reality Prometheus results for rate(counter[2s]) and increase(counter[2s]) may be slightly bigger because of extrapolation, since the first sample on the selected time range is located comparatively far from the start of the time range.
Such calculations have the following issues:
Prometheus can return fractional results from
increase()over time series, which contains only integer values. This is because of extrapolation. For example, Prometheus may return fractional results fromincrease(http_requests_total[5m]).Prometheus returns empty results (aka gaps) from
increase(counter[d])andrate(counter[d])when the lookbehind windowddoesn't cover at least two samples - seerate(counter[1s])andincrease(counter[1s])example above.Prometheus completely misses the increase between the raw sample just before the
(t-d ... t]interval and the first raw sample on this interval. This may result in inaccurate calculations. For example,increase(counter[1h])doesn't equal tosum_over_time(increase(counter[1m])[1h:1m]).
Prometheus developers are aware of these issues - see this link. These issues are addressed in the system I work on - VictoriaMetrics - more specifically, in MetricsQL query language - see this comment and this article for technical details.
irate() Vs rate() Functions in Prometheus
Let’s say you have a counter with these values observed at 1m intervals:
0 60 120 600 720 780
Now rate over 5m will be:
(780-0)/5/60 = 2.6/sec
And irate over 5m will be (only last two data points are used which happen to be only 1m apart)
(780-720)/1/60 = 1/sec
Increasing the resolution does not affect the irate function because the last two observed values do not change when you look further back.
More on reddit.comHelp with PromQL query (sum over time)
Counter reset after target restart
rate/sum confusion
How does the Prometheus rate() function differ from increase()?
Can rate() be used with all types of Prometheus metrics?
How do you calculate request rates using the Prometheus rate function?
Hi All,
I'm trying to understand how irate() & rate() functions work. Why does irate() produce a similar looking graph when the range / resolution is 24h or 5m ? While the difference in graph is clearly visible with rate() when using range as 24h (presents a smoothed out line) or 5m(more spikey).
In the below graph for irate() for 2 different resolutions the graph looks the same. As per prometheus docs irate() calculates the per second instant rate based on the last two data points. What does this mean if my range is 24h? Thank you.
https://preview.redd.it/08mc4pq2mye41.jpg?width=3206&format=pjpg&auto=webp&s=4aef0b681969c8b9831ec0f34310838511ea0350