The "increase" function calculates how much some counter has grown and the "rate" function calculates the amount per second the measure grows.
Analyzing your data I think you used [30s] for the "increase" and [1m] for the "rate" (the correct used values are important to the result).
Basically, for example, in time 2m we have:
increase[30s] = count at 2m - count at 1.5m = 4423 - 4402 = 21
rate[1m] = (count at 2m - count at 1m) / 60 = (4423 - 4381) / 60 = 0.7
Prometheus documentation: increase and rate.
Answer from Marcelo Ávila de Oliveira on Stack OverflowAre there approaches for capturing spikes with PromQL?
How do you calculate request rates using the Prometheus rate function?
Can rate() be used with all types of Prometheus metrics?
The "increase" function calculates how much some counter has grown and the "rate" function calculates the amount per second the measure grows.
Analyzing your data I think you used [30s] for the "increase" and [1m] for the "rate" (the correct used values are important to the result).
Basically, for example, in time 2m we have:
increase[30s] = count at 2m - count at 1.5m = 4423 - 4402 = 21
rate[1m] = (count at 2m - count at 1m) / 60 = (4423 - 4381) / 60 = 0.7
Prometheus documentation: increase and rate.
Prometheus calculates rate(count[d]) at timestamp t in the following way:
- It obtains raw samples per each time series with
countname on the time range(t-d ... t]. Note thatt-dtimestamp isn't included in the range, whilettimestamp is included in the range. For example, when calculatingrate(count[1m])at a timestampt=2mthe following raw samples are selected:4423 @ 2m, 4402 @ 1m45s, 4402 @ 1m30s, 4381 @ 1m15s. Note that the4381 @ 1msample isn't included in calculations. - Then it calculates the difference between the last and the first sample on the selected time range per each time series with the name
count. Prometheus can detect and remove time series resets to zero on the selected time range, but let's skip this for now for the sake of clarity. In the case above it calculates4423 @ 2m - 4381 @ 1m15s = 42. - Then it divides results from step 2 by the duration
din seconds per each time series with namecount. In the case above it calculates42 / 1m = 42 / 60s = 0.7.
The actual result for rate(count[1m]) @ 2m - 0.700023 - differs from the calculated result - 0.7 - because of extrapolation, which can be applied to results calculated at step 2 if timestamps for the first and/or the last raw sample are located too far from the selected time range bounds. See more details about the extrapolation in this issue.
Note also that Prometheus misses possible counter increase on the time range [1m ... 1m15s] when calculating both rate() and increase(). See more details about this issue here and here.
The rate(m[d]) function calculates the increase of a counter metric m over the given lookbehind window d in square brackets and then divides the increase by d. The calculation is performed independently per each matching time series m. For example, suppose there are http_requests_total metrics with url label:
http_requests_total{url="/foo"}
http_requests_total{url="/bar"}
If they have the following values at time t0:
http_requests_total{url="/foo"} 123
http_requests_total{url="/bar"} 456
... and the following values at time t0 + 5 minutes:
http_requests_total{url="/foo"} 345
http_requests_total{url="/bar"} 789
Then rate(http_requests_total[5m]) at time t0 + 5 minutes is calculated in the following way:
- To calculate increase for these metrics between
t0andt0 + 5 minutes:
increase(http_requests_total{url="/foo"}[5m]) = 345 - 123 = 222
increase(http_requests_total{url="/bar"}[5m]) = 789 - 456 = 333
- To divide the calculated increase by
5 minutesexpressed in seconds (5*60s = 300s):
rate(http_requests_total{url="/foo"}[5m]) = 222 / 300 = 0.74
rate(http_requests_total{url="/bar"}[5m]) = 333 / 300 = 1.11
So the end result of rate(http_requests_total[5m]) is a per-second average rps for the last 5 minutes, which is calculated individually per each time series with http_requests_total name.
A few notes:
Both rate() and increase() properly handle e.g.
counter resets, when the counter is reset to zero.Sometimes Prometheus can return unexpected results from
rate()andincrease()because of the chosen data model. See this issue. This issue is addressed in VictoriaMetrics - Prometheus-like monitoring system I work on - see this comment and this article.Some PromQL-compatible query engines such as MetricsQL allow skipping the lookbehind window in square brackets when using
rate()function, sorate(http_requests_total)is a valid MetricsQL query. In this case it automatically adds[$__interval]lookbehind window before query execution. See these docs for more details.
While I am not familiar with Micrometer Timer, the metric you're describing is of type Summary. It is counting the "events" in _count and summing the events magnitude, like duration, elapsed time and similar, in _sum.
If you now perform rate(metric_count[5m]), you'll get the 5m average per second rate of your events. And if you want to know the average duration of these events within 5m window, you do
rate(metric_sum[5m]) / rate(metric_count[5m]). If you try dividing metric_sum/metric_count, you'll get all time (since counter reset) average instead of 5m average at some point in time.
In a way, it looks a bit funny to use rate() for this. Using increase() seems more intuitive to me, but mathematically it's exactly the same as rate() is just an increase()/range and so these ranges cancel each other out in rate(metric_sum[5m]) / rate(metric_count[5m]).
