There are a couple of things to unwrap here.
First, rate vs irate. Neither the linked question, nor the blog post address this (but Eitan's answer does touch on it). The difference is that rate estimates the average rate over the requested range (1 minute, in your case) while irate computes the rate based on the last 2 samples only. Leaving aside the "estimate" part (see this answer if you're curious) the practical difference between the 2 is that rate will smooth out the result, whereas irate will return a sampling of CPU usage, which is more likely to show extremes in CPU usage but also more prone to aliasing.
E.g. if you look at Prometheus' CPU usage, you'll notice that it's at a somewhat constant baseline, with a spike every time a large rule group is evaluated. Given a time range that was at least as long as Prometheus' evaluation interval, if you used rate you'd get a more or less constant CPU usage over time (i.e. a flat line). With irate (assuming a scrape interval of 5s) you'd get one of 2 things:
- if your resolution (i.e. step) was not aligned with Prometheus' evaluation interval (e.g. the resolution was
1mand the evaluation interval was13s) you'd get a random sampling of CPU usage and would hopefully see values close to both the highest and lowest CPU usage over time on a graph; - if your resolution was aligned with Prometheus' evaluation interval (e.g.
1mresolution and15sevaluation interval) then you'd either see the baseline CPU usage everywhere (because you happen to look at5sintervals set 1 minute apart, when no rule evaluation happens) or the peak CPU usage everywhere (because you happen to look at5sintervals 1 minute apart that each cover a rule evaluation).
Regarding the second point, the apparent confusion over what the node_cpu_seconds_total metric represents, it is a counter. Meaning it's a number that increments continuously and essentially measures the amount of time the CPU was idle since the exporter started. The absolute value is not all that useful (as it depends on when the exporter started and will drop to 0 on every restart). What's interesting about it is by how much it increased over a period of time: from that you can compute for a given period of time a rate of increase per second (average, with rate; instant, with irate) or an absolute increase (with increase). So both rate(node_cpu_seconds_total{mode="idle"}[1m]) and irate(node_cpu_seconds_total{mode="idle"}[1m]) will give you a ratio (between 0.0 and 1.0) of how much the CPU was idle (over the past minute, and respectively between the last 2 samples).
Hi All,
I'm trying to understand how irate() & rate() functions work. Why does irate() produce a similar looking graph when the range / resolution is 24h or 5m ? While the difference in graph is clearly visible with rate() when using range as 24h (presents a smoothed out line) or 5m(more spikey).
In the below graph for irate() for 2 different resolutions the graph looks the same. As per prometheus docs irate() calculates the per second instant rate based on the last two data points. What does this mean if my range is 24h? Thank you.
https://preview.redd.it/08mc4pq2mye41.jpg?width=3206&format=pjpg&auto=webp&s=4aef0b681969c8b9831ec0f34310838511ea0350
Why is CPU utilization calculated using irate or rate in Prometheus? - Stack Overflow
Rate and irate display very different values
rate vs irate for container_cpu_usage_seconds_total
irate() Vs rate() Functions in Prometheus
Let’s say you have a counter with these values observed at 1m intervals:
0 60 120 600 720 780
Now rate over 5m will be:
(780-0)/5/60 = 2.6/sec
And irate over 5m will be (only last two data points are used which happen to be only 1m apart)
(780-720)/1/60 = 1/sec
Increasing the resolution does not affect the irate function because the last two observed values do not change when you look further back.
More on reddit.comHow does the Prometheus rate() function differ from increase()?
Can rate() be used with all types of Prometheus metrics?
How do you calculate request rates using the Prometheus rate function?
There are a couple of things to unwrap here.
First, rate vs irate. Neither the linked question, nor the blog post address this (but Eitan's answer does touch on it). The difference is that rate estimates the average rate over the requested range (1 minute, in your case) while irate computes the rate based on the last 2 samples only. Leaving aside the "estimate" part (see this answer if you're curious) the practical difference between the 2 is that rate will smooth out the result, whereas irate will return a sampling of CPU usage, which is more likely to show extremes in CPU usage but also more prone to aliasing.
E.g. if you look at Prometheus' CPU usage, you'll notice that it's at a somewhat constant baseline, with a spike every time a large rule group is evaluated. Given a time range that was at least as long as Prometheus' evaluation interval, if you used rate you'd get a more or less constant CPU usage over time (i.e. a flat line). With irate (assuming a scrape interval of 5s) you'd get one of 2 things:
- if your resolution (i.e. step) was not aligned with Prometheus' evaluation interval (e.g. the resolution was
1mand the evaluation interval was13s) you'd get a random sampling of CPU usage and would hopefully see values close to both the highest and lowest CPU usage over time on a graph; - if your resolution was aligned with Prometheus' evaluation interval (e.g.
1mresolution and15sevaluation interval) then you'd either see the baseline CPU usage everywhere (because you happen to look at5sintervals set 1 minute apart, when no rule evaluation happens) or the peak CPU usage everywhere (because you happen to look at5sintervals 1 minute apart that each cover a rule evaluation).
Regarding the second point, the apparent confusion over what the node_cpu_seconds_total metric represents, it is a counter. Meaning it's a number that increments continuously and essentially measures the amount of time the CPU was idle since the exporter started. The absolute value is not all that useful (as it depends on when the exporter started and will drop to 0 on every restart). What's interesting about it is by how much it increased over a period of time: from that you can compute for a given period of time a rate of increase per second (average, with rate; instant, with irate) or an absolute increase (with increase). So both rate(node_cpu_seconds_total{mode="idle"}[1m]) and irate(node_cpu_seconds_total{mode="idle"}[1m]) will give you a ratio (between 0.0 and 1.0) of how much the CPU was idle (over the past minute, and respectively between the last 2 samples).
Looks like this is already answered here: Prometheus - Convert cpu_user_seconds to CPU Usage %? Looking at the provided link in the answers: https://www.robustperception.io/understanding-machine-cpu-usage you can see the explanation. Personally, I think that irate in this context makes more sense, as it will show you the average on the last active points (vs. rate which will average the entire sampled timeslot).