My question is this how rate will get calculated(my understanding is correct)?
Theoretically, it is correct (except in your example you have 6 datapoints, not 5). But the "magic" starts happening when you add start and end params to your query. In most cases, these two params will not exactly match t1 and t5 from your example, and Prometheus won't be able to capture one or more data points of yours:
In the example above, time window isn't perfectly aligned with real values. In this case, value of v1 will be missed, which would affect the result since increase will be calculated between v5 and v2. Knowing the avg interval between received values, Prometheus could assume that it didn't capture some of the values. So it will try to extrapolate the result by simply multiplying it by coefficient.
I'd recommend reading this great thread https://github.com/prometheus/prometheus/issues/3746.
I'd also recommend reading how alternative rate could work here https://medium.com/@romanhavronenko/victoriametrics-promql-compliance-d4318203f51e in "Better rate" section. Disclosure, I am one of the developers at VictoriaMetrics, so take it with a grain of salt.
Prometheus Counters - and how to deal with them
grafana - Rate and increase extrapolation in prometheus when service is redeploying - Stack Overflow
increase() in Prometheus sometimes doubles values: how to avoid? - Stack Overflow
PromQL(histograms): Prevent extrapolation below zero
This is known as aliasing and is a fundamental problem in signal processing. You can improve this a bit by increasing your sample rate, a 4m range is a bit short with a 2m range. Try a 10m range.
Here for example the query executed at 1515722220 only sees the [email protected] and [email protected] samples. That's an increase of 1 over 2 minutes, which extrapolated over 4 minutes is an increase of 2 - which is as expected.
Any metrics-based monitoring system will have similar artifacts, if you want 100% accuracy you need logs.
increase() will always (approximately) double the actual increase with your setup.
The reason is that (as currently implemented):
increase()is (as you observed) syntactic sugar forrate()i.e. it is the value that would be returned byrate()multiplied by the number of seconds in the range you specified. In your case, it israte() * 240.rate()uses extrapolation in its computation. In the vast majority of cases a 4 minute range will return exactly 2 data points, almost exactly 2 minutes apart. The rate is then computed as the difference between last and first (i.e. the 2 points in your case) divided by the time difference of the 2 points (around 120 seconds in 99.99% of cases) multiplied by the range you requested (exactly 240 seconds). So if the increase between the 2 points is zero, the rate is zero. If the increase between the 2 points is1.0, the computedrate()will be close to2.0 / 240and, as a result, theincrease()will be2.0.
This approach works mostly fine with counters that increase smoothly (e.g. if you have a more or less fixed number of signups every 2 minutes). But with a counter that rarely increases (as does your signups counter) or a spiky counter (like CPU usage) you get weird overestimates (like the increase of 2 you are seeing).
You can essentially reverse engineer Prometheus' implementation and get (something very close to) the actual increase by multiplying with (requested_range - scrape interval) and dividing by requested_range, essentially walking back the extrapolation that Prometheus does.
In your case, this would mean
increase(signups_count[4m]) * (240 - 120) / 240
or, more succinctly,
increase(signups_count[4m]) / 2
It requires you to be aware both of the length of the range and the scrape interval, but it will give you what you want: "ones for ones, and twos for twos, most of the time". Sometimes you'll get 1.01 instead of 1.0 because the scrapes were 119 seconds, not 120 seconds apart and sometimes, if your evaluation is closely aligned with the scrape some points right on the boundary might be included or not in a data point calculation, but it's still a better answer than 2.0.
What you need is the increase() function, that will calculate the difference between the counter values at the start and at the end of the specified time interval. It also correctly handles counter resets during that time period (if any).
increase(http_requests_total[24h])
If you have multiple counters http_requests_total (e.g. from multiple instances) and you need to get the cumulative count of requests, use the sum() operator:
sum(increase(http_requests_total[24h]))
See also my answer to that part of the question about using Grafana's time range selection in queries.
SO won't let me comment on Yoory's answer so I have to make a new one...
In Grafana 5.3, they introduced $__range for Prometheus that's easier to use:
sum(rate(http_requests_total[$__range]))
This variable represents the range for the current dashboard. It is calculated by to - from
http://docs.grafana.org/features/datasources/prometheus/