The challenge with calculating this number is that we only have a few data points inside a time range, and they tend not to be at the exact start and end of that time range (1 minute here). What do we do about the time between the start of the time range and the first data point, similarly the last data point and the end of the range?
We do a small bit of extrapolation to smooth this out and produce the correct result in aggregate. For very slow moving counters like this it can cause artifacts.
Answer from brian-brazil on Stack OverflowThe challenge with calculating this number is that we only have a few data points inside a time range, and they tend not to be at the exact start and end of that time range (1 minute here). What do we do about the time between the start of the time range and the first data point, similarly the last data point and the end of the range?
We do a small bit of extrapolation to smooth this out and produce the correct result in aggregate. For very slow moving counters like this it can cause artifacts.
Prometheus calculates increase(foo_requests_total[1m]) at a timestamp t in the following way:
- It selects all the raw samples per each time series with
foo_requests_totalname on the time range(t-1m ... t]. Note that samples at the timestampt-1maren't included in the selection, while samples at the timestamptare included in the selection. - It calculates the difference
dbetween the last and the first raw sample on the selected time range (Prometheus may also remove possible counter resets, but let's skip this step for the sake of clarity). - It extrapolates the calculated difference
dif the first and/or the last raw sample are located too far from the bounds of the selected time range.
The last step may result in fractional increase() values over integer counters as seen in the original question. See this issue for more details. Note also that increase() in Prometheus misses the difference between the first raw sample on the selected time range and the previous sample before the selected time range. This may result in smaller than expected increase() results.
Prometheus developers are going to fix these issues - see this design doc. In the mean time try VictoriaMetrics - its increase() function properly returns the expected integer result without any extrapolation over integer counters.
Surprising (incorrect?) behaviour of "increase" function
prometheus - Increase() vs changes() function for counters - Stack Overflow
Python "prometheus_client", how to get an int value?
Proposal for improving rate, increase, delta (based on xrate, xincrease)
Hello all,
I'm using Python prometheus_client and using Counter as my main way to print my results to http respond.
The only thing is.. the value is coming out as float, any way to make it int?
Here is an example of what I do:
value = 1
key = 'WooHoo_woo_hoo'
counter = Counter(key, '', ['name'])
counter.labels('MyName').inc(value)The result is always... float:
WooHoo_woo_hoo{name="MyName"} 1.0Any way to "fix" this? What am I doing wrong?
Thank you.
This is known as aliasing and is a fundamental problem in signal processing. You can improve this a bit by increasing your sample rate, a 4m range is a bit short with a 2m range. Try a 10m range.
Here for example the query executed at 1515722220 only sees the [email protected] and [email protected] samples. That's an increase of 1 over 2 minutes, which extrapolated over 4 minutes is an increase of 2 - which is as expected.
Any metrics-based monitoring system will have similar artifacts, if you want 100% accuracy you need logs.
increase() will always (approximately) double the actual increase with your setup.
The reason is that (as currently implemented):
increase()is (as you observed) syntactic sugar forrate()i.e. it is the value that would be returned byrate()multiplied by the number of seconds in the range you specified. In your case, it israte() * 240.rate()uses extrapolation in its computation. In the vast majority of cases a 4 minute range will return exactly 2 data points, almost exactly 2 minutes apart. The rate is then computed as the difference between last and first (i.e. the 2 points in your case) divided by the time difference of the 2 points (around 120 seconds in 99.99% of cases) multiplied by the range you requested (exactly 240 seconds). So if the increase between the 2 points is zero, the rate is zero. If the increase between the 2 points is1.0, the computedrate()will be close to2.0 / 240and, as a result, theincrease()will be2.0.
This approach works mostly fine with counters that increase smoothly (e.g. if you have a more or less fixed number of signups every 2 minutes). But with a counter that rarely increases (as does your signups counter) or a spiky counter (like CPU usage) you get weird overestimates (like the increase of 2 you are seeing).
You can essentially reverse engineer Prometheus' implementation and get (something very close to) the actual increase by multiplying with (requested_range - scrape interval) and dividing by requested_range, essentially walking back the extrapolation that Prometheus does.
In your case, this would mean
increase(signups_count[4m]) * (240 - 120) / 240
or, more succinctly,
increase(signups_count[4m]) / 2
It requires you to be aware both of the length of the range and the scrape interval, but it will give you what you want: "ones for ones, and twos for twos, most of the time". Sometimes you'll get 1.01 instead of 1.0 because the scrapes were 119 seconds, not 120 seconds apart and sometimes, if your evaluation is closely aligned with the scrape some points right on the boundary might be included or not in a data point calculation, but it's still a better answer than 2.0.
You can use the following functions to round a decimal number:
- round = Rounds to the nearest integer.
- ceil = Rounds UP to the nearest integer.
- floor = Rounds DOWN to the nearest integer.
See more details about the "round" function in the Prometheus documentation here.
In your case, you're using "rate" so you get the number of restarts per second * 60 * 5. Which is the same as the number of restarts every 5 minutes.
But, if you want to count the number of restarts (and not calculate the rate of restarts), maybe you should use the "increase" function instead.
See more details about the "increase" function in the Prometheus documentation here.
You probably don't want to use rate. I'm guessing you're looking for increase which will count how many restarts have occurred over a time period
increase(kube_pod_container_status_restarts_total{namespace=~"jenkins"}[10h])