My team switched from datadog to prometheus and counters have been the biggest pain-point. Things that just worked without thinking about it in datadog doesn't seem to have good solutions in prometheus. Surely we can't be the only ones hitting our head against the wall with these problems? How are you addressing them?

Specifically for use-cases around low-frequency counters where you want *reasonably* accurate counts. We use Created Timestamp and have dynamic labels on our counters (so pre-initializing counters to zero isn't viable or makes the data a lot less useful). That being said, these common scenarios have been a challenge:

• Alerting on a counter increase when your counter doesn't start at zero. We use Created Timestamp gives us more confidence but it worries me that a bug/edge-case will cause us to miss an alert. Catching that would be difficult.

• Calculating the total number of increments in a time period (ex: $__range). Sometimes short-lived series aren't counted towards the total.

• Viewing the frequency of counter increments over time as a time series. Seems like aligning the rate and step helps but I'm still wary about the accuracy. It seems like for some time ranges it doesn't work correctly.

• For calculating a success rate or SLI over some period of time. The approach of `sum(rate(success_total[30d])) / `sum(rate(overall_total[30d]))` doesn't always work if there are short-lived series within the query range. I see Grafana SLO feature uses recording rules, which I hope(?) improves this accuracy, but its hard to verify and is a lot of extra steps (i.e. `sum(sum_over_time((grafana_slo_success_rate_5m{})[28d:5m])) / sum(sum_over_time((grafana_slo_total_rate_5m{} )[28d:5m]))`

A lot of teams have started using logs instead of metrics for some of these scenarios. Its ambiguous when its okay to use metrics and when logs are needed, which undermines the credibility of our metrics' accuracy in general.

The frustrating thing is it seems like all the raw data is there to make these use-cases work better? Most of the time you can manually calculate the statistic you want by plotting the raw series. I'm likely over-simplifying things, and I know there's complicated edge-cases around counter-resets, missed scrapes, etc., however promql is more likely to understate the `rate`/`increase` to account for that. If anything, it would be better to overstate the `rate` since its safer to have a false positive than false negative for most monitoring use-cases. I rather have grafana widgets or promql that works for the majority of times you don't hit the complicated edge cases but overstates the rate/increase when that does happen.

I know this comes across as somewhat of a rant so I just want to say I know the prometheus maintainers put a lot of thought into their decisions and I appreciate their responsiveness to helping folks here and on slack.