Hi All,
I'm trying to understand how irate() & rate() functions work. Why does irate() produce a similar looking graph when the range / resolution is 24h or 5m ? While the difference in graph is clearly visible with rate() when using range as 24h (presents a smoothed out line) or 5m(more spikey).
In the below graph for irate() for 2 different resolutions the graph looks the same. As per prometheus docs irate() calculates the per second instant rate based on the last two data points. What does this mean if my range is 24h? Thank you.
https://preview.redd.it/08mc4pq2mye41.jpg?width=3206&format=pjpg&auto=webp&s=4aef0b681969c8b9831ec0f34310838511ea0350
Do I understand Prometheus's rate vs increase functions correctly? - Stack Overflow
Why is CPU utilization calculated using irate or rate in Prometheus? - Stack Overflow
Help with PromQL query (sum over time)
rate/sum confusion
How does the Prometheus rate() function differ from increase()?
Can rate() be used with all types of Prometheus metrics?
How do you calculate request rates using the Prometheus rate function?
In an ideal world (where your samples' timestamps are exactly on the second and your rule evaluation happens exactly on the second) rate(counter[1s]) would return exactly your ICH value and rate(counter[5s]) would return the average of that ICH and the previous 4. Except the ICH at second 1 is 0, not 1, because no one knows when your counter was zero: maybe it incremented right there, maybe it got incremented yesterday, and stayed at 1 since then. (This is the reason why you won't see an increase the first time a counter appears with a value of 1 -- because your code just created and incremented it.)
increase(counter[5s]) is exactly rate(counter[5s]) * 5 (and increase(counter[2s]) is exactly rate(counter[2s]) * 2).
Now what happens in the real world is that your samples are not collected exactly every second on the second and rule evaluation doesn't happen exactly on the second either. So if you have a bunch of samples that are (more or less) 1 second apart and you use Prometheus' rate(counter[1s]), you'll get no output. That's because what Prometheus does is it takes all the samples in the 1 second range [now() - 1s, now()] (which would be a single sample in the vast majority of cases), tries to compute a rate and fails.
If you query rate(counter[5s]) OTOH, Prometheus will pick all the samples in the range [now() - 5s, now] (5 samples, covering approximately 4 seconds on average, say [t1, v1], [t2, v2], [t3, v3], [t4, v4], [t5, v5]) and (assuming your counter doesn't reset within the interval) will return (v5 - v1) / (t5 - t1). I.e. it actually computes the rate of increase over ~4s rather than 5s.
increase(counter[5s]) will return (v5 - v1) / (t5 - t1) * 5, so the rate of increase over ~4 seconds, extrapolated to 5 seconds.
Due to the samples not being exactly spaced, both rate and increase will often return floating point values for integer counters (which makes obvious sense for rate, but not so much for increase).
Prometheus calculates rate(counter[d]) at timestamp t in the following way:
- It selects raw samples for the
countertime series on the time range(t-d ... t]. Note that thet-dtimestamp isn't included in the time range, whilettimestamp is included in the time range. If the selected time range contains less than two raw samples, then Prometheus returns an empty value (a gap) at the timestampt. - Then it calculates the increase of the selected raw samples. Usually it is calculated as the difference between the last selected sample and the first selected sample. Calculations become slightly complicated if the
counterwas reset to zero during the selected time range. Let's skip this for the sake of clarity. - Then the resulting increase can be extrapolated if timestamps for the first and/or the last raw samples are located too far from the bounds of the selected time range.
- Then the rate is calculated by dividing the extrapolated increase by
d.
Prometheus calculates increase(counter[d]) in the same way except the last step.
Let's look at a few examples applied to the original data:
second counter_value increase calculated by hand(call it ICH from now)
1 1 1
2 3 2
3 6 3
4 7 1
5 10 3
6 14 4
7 17 3
8 21 4
9 25 4
10 30 5
The
rate(counter[1s])will return nothing at any timestampt, since any time range(t-1s ... t]contains only a single raw sample, while Prometheus requires at least two samples for calculating bothrate()andincrease().The
rate(counter[2s])andincrease(counter[2])would return the following values per each timestamptwhen extrapolation isn't applied:
t counter_value rate(counter[2s]) increase(counter[2s])
1 1 - -
2 3 (3-1)/2=1.0 3-1=2
3 6 (6-3)/2=1.5 6-3=3
4 7 (7-6)/2=0.5 7-6=1
5 10 (10-7)/2=1.5 10-7=3
6 14 (14-10)/2=2 14-10=4
7 17 (17-14)/2=1.5 17-14=3
8 21 (21-17)/2=2 21-17=4
9 25 (25-21)/2=2 25-21=4
10 30 (30-25)/2=2.5 30-25=5
In reality Prometheus results for rate(counter[2s]) and increase(counter[2s]) may be slightly bigger because of extrapolation, since the first sample on the selected time range is located comparatively far from the start of the time range.
Such calculations have the following issues:
Prometheus can return fractional results from
increase()over time series, which contains only integer values. This is because of extrapolation. For example, Prometheus may return fractional results fromincrease(http_requests_total[5m]).Prometheus returns empty results (aka gaps) from
increase(counter[d])andrate(counter[d])when the lookbehind windowddoesn't cover at least two samples - seerate(counter[1s])andincrease(counter[1s])example above.Prometheus completely misses the increase between the raw sample just before the
(t-d ... t]interval and the first raw sample on this interval. This may result in inaccurate calculations. For example,increase(counter[1h])doesn't equal tosum_over_time(increase(counter[1m])[1h:1m]).
Prometheus developers are aware of these issues - see this link. These issues are addressed in the system I work on - VictoriaMetrics - more specifically, in MetricsQL query language - see this comment and this article for technical details.
There are a couple of things to unwrap here.
First, rate vs irate. Neither the linked question, nor the blog post address this (but Eitan's answer does touch on it). The difference is that rate estimates the average rate over the requested range (1 minute, in your case) while irate computes the rate based on the last 2 samples only. Leaving aside the "estimate" part (see this answer if you're curious) the practical difference between the 2 is that rate will smooth out the result, whereas irate will return a sampling of CPU usage, which is more likely to show extremes in CPU usage but also more prone to aliasing.
E.g. if you look at Prometheus' CPU usage, you'll notice that it's at a somewhat constant baseline, with a spike every time a large rule group is evaluated. Given a time range that was at least as long as Prometheus' evaluation interval, if you used rate you'd get a more or less constant CPU usage over time (i.e. a flat line). With irate (assuming a scrape interval of 5s) you'd get one of 2 things:
- if your resolution (i.e. step) was not aligned with Prometheus' evaluation interval (e.g. the resolution was
1mand the evaluation interval was13s) you'd get a random sampling of CPU usage and would hopefully see values close to both the highest and lowest CPU usage over time on a graph; - if your resolution was aligned with Prometheus' evaluation interval (e.g.
1mresolution and15sevaluation interval) then you'd either see the baseline CPU usage everywhere (because you happen to look at5sintervals set 1 minute apart, when no rule evaluation happens) or the peak CPU usage everywhere (because you happen to look at5sintervals 1 minute apart that each cover a rule evaluation).
Regarding the second point, the apparent confusion over what the node_cpu_seconds_total metric represents, it is a counter. Meaning it's a number that increments continuously and essentially measures the amount of time the CPU was idle since the exporter started. The absolute value is not all that useful (as it depends on when the exporter started and will drop to 0 on every restart). What's interesting about it is by how much it increased over a period of time: from that you can compute for a given period of time a rate of increase per second (average, with rate; instant, with irate) or an absolute increase (with increase). So both rate(node_cpu_seconds_total{mode="idle"}[1m]) and irate(node_cpu_seconds_total{mode="idle"}[1m]) will give you a ratio (between 0.0 and 1.0) of how much the CPU was idle (over the past minute, and respectively between the last 2 samples).
Looks like this is already answered here: Prometheus - Convert cpu_user_seconds to CPU Usage %? Looking at the provided link in the answers: https://www.robustperception.io/understanding-machine-cpu-usage you can see the explanation. Personally, I think that irate in this context makes more sense, as it will show you the average on the last active points (vs. rate which will average the entire sampled timeslot).