The way I sorted this problem was doing the following prometheus query for Grafana. This query allows me to identify the total_processing_time consumed by an individual customer across all PODs in a K8s cluster within a given time range. The catch is that this query handles the following corner cases:
If the querying time_range encompasses only the last 1h but there are older timeseries that were started before that, the query below "offsets to zero" these timeseries by removing the value of the first measurement point within the time_range. This way these series are considered as if they have started within the time range.
If a new timeseries (due to a new POD) show up within the querying time_range, and the first measurement value is greater than 0, then the first measurement value of the new timeseries is added to the calculation to fulfill the gap from the
incrementfunction.
sum(
# Calculates the difference between the last measurement
# and the first measurement in the given time_range for each
# timeseries, this helps identify how much additional resources
# were consumed in the given time_range
increase(
total_processing_time{customer_id="$customer_id"}[$__range]
)
# Adds to the difference the minimum value of each timeseries,
# this helps to account for the timeseries that started within
# the given time_range with the first measurement greater than 0
# (not considered by the previous increase function)
+ min_over_time(
total_processing_time{customer_id="$customer_id"}[$__range]
)
)
# Removes the timeseries measurements being carried over from before
# the start of the time_range, this offsets the running timeseries
# as if they had started from zero within the time_range
- sum(
total_processing_time{customer_id="$customer_id"} @ start() or vector(0)
)
I need to use the increase function, but the metric is very rare and when service restarts the counter is null for long time. The increase metric doesn’t play nicely with the null value and I can find a solution to consider null as zero.
Increase(my_counter[1h])
Works only if there are no null data points. Any other query I tried do not work like
Increase(my_metrics[1h] or vector(0))
The way I sorted this problem was doing the following prometheus query for Grafana. This query allows me to identify the total_processing_time consumed by an individual customer across all PODs in a K8s cluster within a given time range. The catch is that this query handles the following corner cases:
If the querying time_range encompasses only the last 1h but there are older timeseries that were started before that, the query below "offsets to zero" these timeseries by removing the value of the first measurement point within the time_range. This way these series are considered as if they have started within the time range.
If a new timeseries (due to a new POD) show up within the querying time_range, and the first measurement value is greater than 0, then the first measurement value of the new timeseries is added to the calculation to fulfill the gap from the
incrementfunction.
sum(
# Calculates the difference between the last measurement
# and the first measurement in the given time_range for each
# timeseries, this helps identify how much additional resources
# were consumed in the given time_range
increase(
total_processing_time{customer_id="$customer_id"}[$__range]
)
# Adds to the difference the minimum value of each timeseries,
# this helps to account for the timeseries that started within
# the given time_range with the first measurement greater than 0
# (not considered by the previous increase function)
+ min_over_time(
total_processing_time{customer_id="$customer_id"}[$__range]
)
)
# Removes the timeseries measurements being carried over from before
# the start of the time_range, this offsets the running timeseries
# as if they had started from zero within the time_range
- sum(
total_processing_time{customer_id="$customer_id"} @ start() or vector(0)
)
If you have access to the code where the metric job_invocation_total is created and the cardinality of the metric labels is bound, you can initialize the counter with the value 0.
If this is not the case, you can create a recording rule that is either 0 or equal to the metric job_invocation_total.
increase() should consider creation of new timeseries as reset
kubernetes - How to provide default values for values in a Prometheus timeseries that are null - Stack Overflow
Time series: how to set nulls as zero?
PromQL won't handle null values during vector matching
I have metrics like these :
up{client=“clientA”, job=“admin”}
up{client=“clientA”, job=“customer”}
up{client=“clientB”, job=“admin”}
up{client=“clientB”, job=“customer”}
up{client=“clientC”, job=“admin”}
up{client=“clientD”, job=“admin”}
These metrics are from different jobs. I am trying to find clients who have no customer job. The job customer doesn't scrape any target for clientC and clientD hence there is no metric at all. Absent is not working in this case as the metric was not present at all.
There's no increases there, the time series appear with the value 1. If they had increased from 0 to 1 rather than nothing to 1 then increase would show a non-zero value. I'd recommend initialising your metrics with the label values you know about.
I had the same problem, you need to get the occurrences in the range of 30 days or more, but when doing this it always returns 0, even if when doing sum(http_requests_received_total{job="TodoApi"}) it returned values.
The solution is the following, as each sample of the requests occurred in the prometheus Scrap Interval, you should then use the same interval to fetch that sample. But keeping this value in your hand, for example mine has 15s would not be feasible, since when increasing the range to 1d or more days the interval increases proportionally, this logic of the proportion I still don't understand, but there is a solution to get this value automatically using $__interval it takes the exact value, in this way the increase that previously brought a non-integer value now returns. Mine worked as follows:
increase( sum(http_requests_received_total{job="TodoApi"}) [$__interval:] )
Besides using $__interval also use : it will tell prometheus to look within the time interval obtained in $__interval:
enter image description here