The "increase" function calculates how much some counter has grown and the "rate" function calculates the amount per second the measure grows.
Analyzing your data I think you used [30s] for the "increase" and [1m] for the "rate" (the correct used values are important to the result).
Basically, for example, in time 2m we have:
increase[30s] = count at 2m - count at 1.5m = 4423 - 4402 = 21
rate[1m] = (count at 2m - count at 1m) / 60 = (4423 - 4381) / 60 = 0.7
Prometheus documentation: increase and rate.
Answer from Marcelo Ávila de Oliveira on Stack OverflowThe "increase" function calculates how much some counter has grown and the "rate" function calculates the amount per second the measure grows.
Analyzing your data I think you used [30s] for the "increase" and [1m] for the "rate" (the correct used values are important to the result).
Basically, for example, in time 2m we have:
increase[30s] = count at 2m - count at 1.5m = 4423 - 4402 = 21
rate[1m] = (count at 2m - count at 1m) / 60 = (4423 - 4381) / 60 = 0.7
Prometheus documentation: increase and rate.
Prometheus calculates rate(count[d]) at timestamp t in the following way:
- It obtains raw samples per each time series with
countname on the time range(t-d ... t]. Note thatt-dtimestamp isn't included in the range, whilettimestamp is included in the range. For example, when calculatingrate(count[1m])at a timestampt=2mthe following raw samples are selected:4423 @ 2m, 4402 @ 1m45s, 4402 @ 1m30s, 4381 @ 1m15s. Note that the4381 @ 1msample isn't included in calculations. - Then it calculates the difference between the last and the first sample on the selected time range per each time series with the name
count. Prometheus can detect and remove time series resets to zero on the selected time range, but let's skip this for now for the sake of clarity. In the case above it calculates4423 @ 2m - 4381 @ 1m15s = 42. - Then it divides results from step 2 by the duration
din seconds per each time series with namecount. In the case above it calculates42 / 1m = 42 / 60s = 0.7.
The actual result for rate(count[1m]) @ 2m - 0.700023 - differs from the calculated result - 0.7 - because of extrapolation, which can be applied to results calculated at step 2 if timestamps for the first and/or the last raw sample are located too far from the selected time range bounds. See more details about the extrapolation in this issue.
Note also that Prometheus misses possible counter increase on the time range [1m ... 1m15s] when calculating both rate() and increase(). See more details about this issue here and here.
Do I understand Prometheus's rate vs increase functions correctly? - Stack Overflow
Sum rate with missing values?
Calculating the Avg with Gaps in Data
Min, max, avg, and stddev of values in between scrape interval
How does the Prometheus rate() function differ from increase()?
How do you calculate request rates using the Prometheus rate function?
Can rate() be used with all types of Prometheus metrics?
In an ideal world (where your samples' timestamps are exactly on the second and your rule evaluation happens exactly on the second) rate(counter[1s]) would return exactly your ICH value and rate(counter[5s]) would return the average of that ICH and the previous 4. Except the ICH at second 1 is 0, not 1, because no one knows when your counter was zero: maybe it incremented right there, maybe it got incremented yesterday, and stayed at 1 since then. (This is the reason why you won't see an increase the first time a counter appears with a value of 1 -- because your code just created and incremented it.)
increase(counter[5s]) is exactly rate(counter[5s]) * 5 (and increase(counter[2s]) is exactly rate(counter[2s]) * 2).
Now what happens in the real world is that your samples are not collected exactly every second on the second and rule evaluation doesn't happen exactly on the second either. So if you have a bunch of samples that are (more or less) 1 second apart and you use Prometheus' rate(counter[1s]), you'll get no output. That's because what Prometheus does is it takes all the samples in the 1 second range [now() - 1s, now()] (which would be a single sample in the vast majority of cases), tries to compute a rate and fails.
If you query rate(counter[5s]) OTOH, Prometheus will pick all the samples in the range [now() - 5s, now] (5 samples, covering approximately 4 seconds on average, say [t1, v1], [t2, v2], [t3, v3], [t4, v4], [t5, v5]) and (assuming your counter doesn't reset within the interval) will return (v5 - v1) / (t5 - t1). I.e. it actually computes the rate of increase over ~4s rather than 5s.
increase(counter[5s]) will return (v5 - v1) / (t5 - t1) * 5, so the rate of increase over ~4 seconds, extrapolated to 5 seconds.
Due to the samples not being exactly spaced, both rate and increase will often return floating point values for integer counters (which makes obvious sense for rate, but not so much for increase).
Prometheus calculates rate(counter[d]) at timestamp t in the following way:
- It selects raw samples for the
countertime series on the time range(t-d ... t]. Note that thet-dtimestamp isn't included in the time range, whilettimestamp is included in the time range. If the selected time range contains less than two raw samples, then Prometheus returns an empty value (a gap) at the timestampt. - Then it calculates the increase of the selected raw samples. Usually it is calculated as the difference between the last selected sample and the first selected sample. Calculations become slightly complicated if the
counterwas reset to zero during the selected time range. Let's skip this for the sake of clarity. - Then the resulting increase can be extrapolated if timestamps for the first and/or the last raw samples are located too far from the bounds of the selected time range.
- Then the rate is calculated by dividing the extrapolated increase by
d.
Prometheus calculates increase(counter[d]) in the same way except the last step.
Let's look at a few examples applied to the original data:
second counter_value increase calculated by hand(call it ICH from now)
1 1 1
2 3 2
3 6 3
4 7 1
5 10 3
6 14 4
7 17 3
8 21 4
9 25 4
10 30 5
The
rate(counter[1s])will return nothing at any timestampt, since any time range(t-1s ... t]contains only a single raw sample, while Prometheus requires at least two samples for calculating bothrate()andincrease().The
rate(counter[2s])andincrease(counter[2])would return the following values per each timestamptwhen extrapolation isn't applied:
t counter_value rate(counter[2s]) increase(counter[2s])
1 1 - -
2 3 (3-1)/2=1.0 3-1=2
3 6 (6-3)/2=1.5 6-3=3
4 7 (7-6)/2=0.5 7-6=1
5 10 (10-7)/2=1.5 10-7=3
6 14 (14-10)/2=2 14-10=4
7 17 (17-14)/2=1.5 17-14=3
8 21 (21-17)/2=2 21-17=4
9 25 (25-21)/2=2 25-21=4
10 30 (30-25)/2=2.5 30-25=5
In reality Prometheus results for rate(counter[2s]) and increase(counter[2s]) may be slightly bigger because of extrapolation, since the first sample on the selected time range is located comparatively far from the start of the time range.
Such calculations have the following issues:
Prometheus can return fractional results from
increase()over time series, which contains only integer values. This is because of extrapolation. For example, Prometheus may return fractional results fromincrease(http_requests_total[5m]).Prometheus returns empty results (aka gaps) from
increase(counter[d])andrate(counter[d])when the lookbehind windowddoesn't cover at least two samples - seerate(counter[1s])andincrease(counter[1s])example above.Prometheus completely misses the increase between the raw sample just before the
(t-d ... t]interval and the first raw sample on this interval. This may result in inaccurate calculations. For example,increase(counter[1h])doesn't equal tosum_over_time(increase(counter[1m])[1h:1m]).
Prometheus developers are aware of these issues - see this link. These issues are addressed in the system I work on - VictoriaMetrics - more specifically, in MetricsQL query language - see this comment and this article for technical details.