There are no interrelations between these two settings. For example, if rate(http_requests_total{job="api-server"}[5m]) is your PromQL query, it yields a single value when you execute it at a specific point in time. If you have a Grafana dashboard with a 2h time range, then Grafana just repeatedly executes this query at a series of point in times during the past 2 hours, and displays you the result of each query as e.g. a graph.
For example, if the time range of your Grafana dashboard is 2h and the interval is set to 1 minute, then Grafana executes your query 120 times during the time span ranging the past 2 hours. The results of all these queries form a graph and that's what you see in your Grafana dashboard.
The interval with which Grafana executes the query is determined by the Max data points and Min interval settings in the Query options of your Grafana dashboard:
See the explanations about these settings in the Grafana documentation.
Answer from weibeld on Stack OverflowNote: Grafana uses the
query_rangeendpoint of the Prometheus API to repeatedly execute the query over the given time range.
What is the exact fomula of the funtion rate?
How can I understand rate() ?
Applying a rate function to positive and negative changes
Grafana / prometheus: understand rate function - Stack Overflow
Hi everyone, I'm stupid. I can't figure out exactly how to interpret rate() in Prometheus.
node_network_transmit_bytes_total{} is just a counter. Use rate() for a rate. Got it.
rate(node_network_transmit_bytes_total{}[1m]) sounds like it should be interpreted as: "How many bytes were transmitted over a minute", but then there's the time range settings in the upper right. Would "Last 3 hours" mean, the average transmit rate per minute for the last 3 hours?
Basically, could someone explain what: rate(node_network_transmit_bytes_total{}[1m]) with a time range of "Last 3 hours" would indicate? None of the explanations I've found on blogs, etc make sense to my smooth brain. I'm basically just trying to see the transmit rate in Mbps, MB/s, or MB/h, for each node.
Note: In the screenshot, you are using irate instead of rate.
Anyway, both are functions to be used with monotonically incrementing counters and calculate how fast the counter increases. Memory usage is not a counter, it is a gauge, so using (i)rate is not useful. See doc on metric types.
Your graph shows how much the memory usage increases per second, not how much memory is used, what you probably want. You can see that you get a spike in irate each time the memory usage drops instead of increments. This drop is interpreted by (i)rate as a reset of the counter, so after the drop/reset, Prometheus thinks the counter immediately rockets from 0 to the actual memory usage in just a second.
See the docs for irate and the docs for rate
rate() is function to specify the time window for the quantile calculation.
https://prometheus.io/docs/prometheus/latest/querying/functions/#histogram_quantile
In an ideal world (where your samples' timestamps are exactly on the second and your rule evaluation happens exactly on the second) rate(counter[1s]) would return exactly your ICH value and rate(counter[5s]) would return the average of that ICH and the previous 4. Except the ICH at second 1 is 0, not 1, because no one knows when your counter was zero: maybe it incremented right there, maybe it got incremented yesterday, and stayed at 1 since then. (This is the reason why you won't see an increase the first time a counter appears with a value of 1 -- because your code just created and incremented it.)
increase(counter[5s]) is exactly rate(counter[5s]) * 5 (and increase(counter[2s]) is exactly rate(counter[2s]) * 2).
Now what happens in the real world is that your samples are not collected exactly every second on the second and rule evaluation doesn't happen exactly on the second either. So if you have a bunch of samples that are (more or less) 1 second apart and you use Prometheus' rate(counter[1s]), you'll get no output. That's because what Prometheus does is it takes all the samples in the 1 second range [now() - 1s, now()] (which would be a single sample in the vast majority of cases), tries to compute a rate and fails.
If you query rate(counter[5s]) OTOH, Prometheus will pick all the samples in the range [now() - 5s, now] (5 samples, covering approximately 4 seconds on average, say [t1, v1], [t2, v2], [t3, v3], [t4, v4], [t5, v5]) and (assuming your counter doesn't reset within the interval) will return (v5 - v1) / (t5 - t1). I.e. it actually computes the rate of increase over ~4s rather than 5s.
increase(counter[5s]) will return (v5 - v1) / (t5 - t1) * 5, so the rate of increase over ~4 seconds, extrapolated to 5 seconds.
Due to the samples not being exactly spaced, both rate and increase will often return floating point values for integer counters (which makes obvious sense for rate, but not so much for increase).
Prometheus calculates rate(counter[d]) at timestamp t in the following way:
- It selects raw samples for the
countertime series on the time range(t-d ... t]. Note that thet-dtimestamp isn't included in the time range, whilettimestamp is included in the time range. If the selected time range contains less than two raw samples, then Prometheus returns an empty value (a gap) at the timestampt. - Then it calculates the increase of the selected raw samples. Usually it is calculated as the difference between the last selected sample and the first selected sample. Calculations become slightly complicated if the
counterwas reset to zero during the selected time range. Let's skip this for the sake of clarity. - Then the resulting increase can be extrapolated if timestamps for the first and/or the last raw samples are located too far from the bounds of the selected time range.
- Then the rate is calculated by dividing the extrapolated increase by
d.
Prometheus calculates increase(counter[d]) in the same way except the last step.
Let's look at a few examples applied to the original data:
second counter_value increase calculated by hand(call it ICH from now)
1 1 1
2 3 2
3 6 3
4 7 1
5 10 3
6 14 4
7 17 3
8 21 4
9 25 4
10 30 5
The
rate(counter[1s])will return nothing at any timestampt, since any time range(t-1s ... t]contains only a single raw sample, while Prometheus requires at least two samples for calculating bothrate()andincrease().The
rate(counter[2s])andincrease(counter[2])would return the following values per each timestamptwhen extrapolation isn't applied:
t counter_value rate(counter[2s]) increase(counter[2s])
1 1 - -
2 3 (3-1)/2=1.0 3-1=2
3 6 (6-3)/2=1.5 6-3=3
4 7 (7-6)/2=0.5 7-6=1
5 10 (10-7)/2=1.5 10-7=3
6 14 (14-10)/2=2 14-10=4
7 17 (17-14)/2=1.5 17-14=3
8 21 (21-17)/2=2 21-17=4
9 25 (25-21)/2=2 25-21=4
10 30 (30-25)/2=2.5 30-25=5
In reality Prometheus results for rate(counter[2s]) and increase(counter[2s]) may be slightly bigger because of extrapolation, since the first sample on the selected time range is located comparatively far from the start of the time range.
Such calculations have the following issues:
Prometheus can return fractional results from
increase()over time series, which contains only integer values. This is because of extrapolation. For example, Prometheus may return fractional results fromincrease(http_requests_total[5m]).Prometheus returns empty results (aka gaps) from
increase(counter[d])andrate(counter[d])when the lookbehind windowddoesn't cover at least two samples - seerate(counter[1s])andincrease(counter[1s])example above.Prometheus completely misses the increase between the raw sample just before the
(t-d ... t]interval and the first raw sample on this interval. This may result in inaccurate calculations. For example,increase(counter[1h])doesn't equal tosum_over_time(increase(counter[1m])[1h:1m]).
Prometheus developers are aware of these issues - see this link. These issues are addressed in the system I work on - VictoriaMetrics - more specifically, in MetricsQL query language - see this comment and this article for technical details.