How do Counter metrics work in Prometheus?
How does the Prometheus counter metric type work?
What are Prometheus metrics?
My team switched from datadog to prometheus and counters have been the biggest pain-point. Things that just worked without thinking about it in datadog doesn't seem to have good solutions in prometheus. Surely we can't be the only ones hitting our head against the wall with these problems? How are you addressing them?
Specifically for use-cases around low-frequency counters where you want *reasonably* accurate counts. We use Created Timestamp and have dynamic labels on our counters (so pre-initializing counters to zero isn't viable or makes the data a lot less useful). That being said, these common scenarios have been a challenge:
-
Alerting on a counter increase when your counter doesn't start at zero. We use Created Timestamp gives us more confidence but it worries me that a bug/edge-case will cause us to miss an alert. Catching that would be difficult.
-
Calculating the total number of increments in a time period (ex: $__range). Sometimes short-lived series aren't counted towards the total.
-
Viewing the frequency of counter increments over time as a time series. Seems like aligning the rate and step helps but I'm still wary about the accuracy. It seems like for some time ranges it doesn't work correctly.
-
For calculating a success rate or SLI over some period of time. The approach of `sum(rate(success_total[30d])) / `sum(rate(overall_total[30d]))` doesn't always work if there are short-lived series within the query range. I see Grafana SLO feature uses recording rules, which I hope(?) improves this accuracy, but its hard to verify and is a lot of extra steps (i.e. `sum(sum_over_time((grafana_slo_success_rate_5m{})[28d:5m])) / sum(sum_over_time((grafana_slo_total_rate_5m{} )[28d:5m]))`
A lot of teams have started using logs instead of metrics for some of these scenarios. Its ambiguous when its okay to use metrics and when logs are needed, which undermines the credibility of our metrics' accuracy in general.
The frustrating thing is it seems like all the raw data is there to make these use-cases work better? Most of the time you can manually calculate the statistic you want by plotting the raw series. I'm likely over-simplifying things, and I know there's complicated edge-cases around counter-resets, missed scrapes, etc., however promql is more likely to understate the `rate`/`increase` to account for that. If anything, it would be better to overstate the `rate` since its safer to have a false positive than false negative for most monitoring use-cases. I rather have grafana widgets or promql that works for the majority of times you don't hit the complicated edge cases but overstates the rate/increase when that does happen.
I know this comes across as somewhat of a rant so I just want to say I know the prometheus maintainers put a lot of thought into their decisions and I appreciate their responsiveness to helping folks here and on slack.
Counters are allowed to be reset to 0, so there's no need to do anything special here to handle it. See http://www.robustperception.io/how-does-a-prometheus-counter-work/ for more detail.
It's recommended to use a client library which will handle all of this for you.
Also, by convention you should suffix counters with _total so that metric should be news_reads_total.
You generally don't want to look at the total of a counter the way that you are in your example, because it's not very meaningful once you actually try to use it analytically.
The idea is that you want to know increases over a period of time. For example, do you want to know the total amount of article views for the last 7 days, for this month so far, for the last 30 days, etc.
This answer and this article do an excellent job of explaining all this, but here are some examples. For demonstration purposes I use a counter called walks_started_total.
The problem
Query: `walks_started_total`
Solution 1
Seeing the total for the last week: `increase(walks_started_total[1w])`
Solution 2
Over a 1 minute period: `increase(walks_started_total[1m])`