There's no increases there, the time series appear with the value 1. If they had increased from 0 to 1 rather than nothing to 1 then increase would show a non-zero value. I'd recommend initialising your metrics with the label values you know about.
rate()[1m] does not return any data
prometheus - Why is increase() showing only zero values when I can see the metric value increasing? - Stack Overflow
Do I understand Prometheus's rate vs increase functions correctly? - Stack Overflow
Prometheus query expression 0/0 and 1-0/0 returns 0
There's no increases there, the time series appear with the value 1. If they had increased from 0 to 1 rather than nothing to 1 then increase would show a non-zero value. I'd recommend initialising your metrics with the label values you know about.
I had the same problem, you need to get the occurrences in the range of 30 days or more, but when doing this it always returns 0, even if when doing sum(http_requests_received_total{job="TodoApi"}) it returned values.
The solution is the following, as each sample of the requests occurred in the prometheus Scrap Interval, you should then use the same interval to fetch that sample. But keeping this value in your hand, for example mine has 15s would not be feasible, since when increasing the range to 1d or more days the interval increases proportionally, this logic of the proportion I still don't understand, but there is a solution to get this value automatically using $__interval it takes the exact value, in this way the increase that previously brought a non-integer value now returns. Mine worked as follows:
increase( sum(http_requests_received_total{job="TodoApi"}) [$__interval:] )
Besides using $__interval also use : it will tell prometheus to look within the time interval obtained in $__interval:
enter image description here
In an ideal world (where your samples' timestamps are exactly on the second and your rule evaluation happens exactly on the second) rate(counter[1s]) would return exactly your ICH value and rate(counter[5s]) would return the average of that ICH and the previous 4. Except the ICH at second 1 is 0, not 1, because no one knows when your counter was zero: maybe it incremented right there, maybe it got incremented yesterday, and stayed at 1 since then. (This is the reason why you won't see an increase the first time a counter appears with a value of 1 -- because your code just created and incremented it.)
increase(counter[5s]) is exactly rate(counter[5s]) * 5 (and increase(counter[2s]) is exactly rate(counter[2s]) * 2).
Now what happens in the real world is that your samples are not collected exactly every second on the second and rule evaluation doesn't happen exactly on the second either. So if you have a bunch of samples that are (more or less) 1 second apart and you use Prometheus' rate(counter[1s]), you'll get no output. That's because what Prometheus does is it takes all the samples in the 1 second range [now() - 1s, now()] (which would be a single sample in the vast majority of cases), tries to compute a rate and fails.
If you query rate(counter[5s]) OTOH, Prometheus will pick all the samples in the range [now() - 5s, now] (5 samples, covering approximately 4 seconds on average, say [t1, v1], [t2, v2], [t3, v3], [t4, v4], [t5, v5]) and (assuming your counter doesn't reset within the interval) will return (v5 - v1) / (t5 - t1). I.e. it actually computes the rate of increase over ~4s rather than 5s.
increase(counter[5s]) will return (v5 - v1) / (t5 - t1) * 5, so the rate of increase over ~4 seconds, extrapolated to 5 seconds.
Due to the samples not being exactly spaced, both rate and increase will often return floating point values for integer counters (which makes obvious sense for rate, but not so much for increase).
Prometheus calculates rate(counter[d]) at timestamp t in the following way:
- It selects raw samples for the
countertime series on the time range(t-d ... t]. Note that thet-dtimestamp isn't included in the time range, whilettimestamp is included in the time range. If the selected time range contains less than two raw samples, then Prometheus returns an empty value (a gap) at the timestampt. - Then it calculates the increase of the selected raw samples. Usually it is calculated as the difference between the last selected sample and the first selected sample. Calculations become slightly complicated if the
counterwas reset to zero during the selected time range. Let's skip this for the sake of clarity. - Then the resulting increase can be extrapolated if timestamps for the first and/or the last raw samples are located too far from the bounds of the selected time range.
- Then the rate is calculated by dividing the extrapolated increase by
d.
Prometheus calculates increase(counter[d]) in the same way except the last step.
Let's look at a few examples applied to the original data:
second counter_value increase calculated by hand(call it ICH from now)
1 1 1
2 3 2
3 6 3
4 7 1
5 10 3
6 14 4
7 17 3
8 21 4
9 25 4
10 30 5
The
rate(counter[1s])will return nothing at any timestampt, since any time range(t-1s ... t]contains only a single raw sample, while Prometheus requires at least two samples for calculating bothrate()andincrease().The
rate(counter[2s])andincrease(counter[2])would return the following values per each timestamptwhen extrapolation isn't applied:
t counter_value rate(counter[2s]) increase(counter[2s])
1 1 - -
2 3 (3-1)/2=1.0 3-1=2
3 6 (6-3)/2=1.5 6-3=3
4 7 (7-6)/2=0.5 7-6=1
5 10 (10-7)/2=1.5 10-7=3
6 14 (14-10)/2=2 14-10=4
7 17 (17-14)/2=1.5 17-14=3
8 21 (21-17)/2=2 21-17=4
9 25 (25-21)/2=2 25-21=4
10 30 (30-25)/2=2.5 30-25=5
In reality Prometheus results for rate(counter[2s]) and increase(counter[2s]) may be slightly bigger because of extrapolation, since the first sample on the selected time range is located comparatively far from the start of the time range.
Such calculations have the following issues:
Prometheus can return fractional results from
increase()over time series, which contains only integer values. This is because of extrapolation. For example, Prometheus may return fractional results fromincrease(http_requests_total[5m]).Prometheus returns empty results (aka gaps) from
increase(counter[d])andrate(counter[d])when the lookbehind windowddoesn't cover at least two samples - seerate(counter[1s])andincrease(counter[1s])example above.Prometheus completely misses the increase between the raw sample just before the
(t-d ... t]interval and the first raw sample on this interval. This may result in inaccurate calculations. For example,increase(counter[1h])doesn't equal tosum_over_time(increase(counter[1m])[1h:1m]).
Prometheus developers are aware of these issues - see this link. These issues are addressed in the system I work on - VictoriaMetrics - more specifically, in MetricsQL query language - see this comment and this article for technical details.
AFAICT the cause for the weird results is (1) the fact that your counter actually only increases once every minute, even though you collect it every 15 seconds combined with (2) Prometheus' rate() implementation discarding every 4th counter increase (in your particular setup).
More precisely, you appear to be computing a 1 minute rate, every 1 minute over a counter scraped at 15 second resolution, increasing every 1 minute (on average).
What this means essentially is that Prometheus will basically slice your 1 hour interval into disjoint 1 minute ranges and estimate the rate over each range. The first value will be the extrapolated rate of increase between points 0 and 3, the second will be the extrapolated rate between points 4 and 7 and so on. Because your counter only actually increases once a minute, you can run into 2 different situations:
- Your counter increases happen between point pairs 3-4, 7-8 etc. In this case Prometheus sees an increase rate of zero (because there is no increase between points 0 and 3, points 4 and 7 etc. This seems to be happening in the first half of your first graph.
- Your counter increases happen somewhere between points 0-3, 4-7 etc. In this case Prometheus takes the difference between the last and first points in each interval (your actual counter increase), divides it by the time difference between the 2 points (on average 45 seconds), then extrapolates that to 1 minute (essentially overestimating it by a factor of 1.(3) -- I'm eyeballing an increase of ~200k over ~50 minutes, so an average rate of about 67 QPS, whereas
rate()returns something closer to 90 QPS). This is what happens in the second half of your graph.
This is also why your graph looks wildly different across refreshes. The argument for the current implementation of rate() is that it is "correct on average". Which, if you look at the whole of your graph, across refreshes, is true. </sarcasm>
Essentially graphing a Prometheus rate() or increase() over a time range R with resolution R will result in aliasing, either overestimating (1.33x in your case) or underestimating (zero in your case) on anything but a smoothly increasing counter.
You can work around it by replacing your expression with:
rate(foo[75s]) / 75 * 60
This way you'll actually get the rate of increase between data points 1 minute apart (a 75 seconds range will almost always return exactly 5 points, so 4 counter increases) and reverse the extrapolation to 75 seconds that Prometheus does. There will be some noise in edge cases (e.g. if your evaluation is aligned with scraping times it's possible to get 6 points in one range and 4 in the next due to scrape interval jitter) but you're getting that anyway with rate().
BTW, you can see the aliasing by increasing the resolution of your graph to something like 1 second (anything 15 seconds or below should show it clearly).
What you say doesn't line up with the data, that raw data is only going up about once a minute. Are you sure you're scraping every 15s?
If there is no activity during the specified time period, the rate() in the divider becomes 0 and the result of division becomes NaN.
This is the correct behaviour, NaN is what you want the result to be.
aggregations work OK.
You can't aggregate ratios. You need to aggregate the numerator and denominator separately and then divide.
So:
sum by (command_group, command_name)(rate(hystrix_command_latency_total_seconds_sum[5m]))
/
sum by (command_group, command_name)(rate(hystrix_command_latency_total_seconds_count[5m]))
Finally I have a solution for my specific problem:
Having a devision by zero leads to a NaN display - that is fine as a technical result and correct but not what the user wants to see (does not fulfil the business requirement).
So I searched a bit and found a "solution" for my problem in the grafana community:
Surround your problematic value with max(YOUR_PROLEMATIC_QUERY, or vector(-1)). An additional value mapping then leads to a useful output.
(Of course you have to adapt the solution to your problem... min/max... vector(42)/vector(101)/vector(...))
Update (1)
Okay. However. It seems to be a bit more tricky based on the query. For example I have another query that fails with NaN as a result of a devision by zero. The above solution does not work. I had to surround the query with brackets and added > 0 or on() vector(100).
I need to use the increase function, but the metric is very rare and when service restarts the counter is null for long time. The increase metric doesn’t play nicely with the null value and I can find a solution to consider null as zero.
Increase(my_counter[1h])
Works only if there are no null data points. Any other query I tried do not work like
Increase(my_metrics[1h] or vector(0))
I'm using mtail to track a job that logs to a file that gets transaction counts once every few days. Mtail keeps a running count of the transactions and while it's running i'm able to visualize/track rates of transactions. However, if mtail is restarted, no metrics are available until the next time the job runs. These missing metrics are preventing me from capturing the rate of change from the first job that runs after restart. Is there a way to default missing data to 0?
So that instead of:
_ _ _ 500
I get
0 0 0 500
and the rate, given the 4 data points is 500 between the last two instead of no data?