The way you have it, it will alert if you have new errors every time it evaluates (default=1m) for 10 minutes and then trigger an alert. There is also a property in alertmanager called group_wait (default=30s) which after the first triggered alert waits and groups all triggered alerts in the past time into 1 notification. You can remove the for: 10m and set group_wait=10m if you want to send notification even if you have 1 error but just don't want to have 1000 notifications for every single error
Alerting on prometheus counter increments without false alarms from query timeouts?
I just use 'Keep last state", then rely on a separate check to ensure Grafana and prometheus are working.
More on reddit.comgrafana - Alert on increase in slow increasing prometheus counter not working due to resetarts - Stack Overflow
Prometheus alerts first counter value - Stack Overflow
Prometheus alerting rule not detecting first time metric increase - Stack Overflow
I had a similar issue with planetlabs/draino:
I wanted to be able to detect when it drained a node.
(Unfortunately, they carry over their minimalist logging policy, which makes sense for logging, over to metrics where it doesn't make sense...)
The draino_pod_ip:10002/metrics endpoint's webpage is completely empty... does not exist until the first drain occurs...
My needs were slightly more difficult to detect, I had to deal with metric does not exist when value = 0 (aka on pod reboot).
I had to detect the transition from does not exist -> 1, and from n -> n+1.
This is what I came up with, note the metric I was detecting is an integer, I'm not sure how this will worth with decimals, even if it needs tweaking for your needs I think it may help point you in the right direction:
(absent(draino_cordoned_nodes_total offset 1m) == 1 and count(draino_cordoned_nodes_total) > -1)
^ creates a blip of 1 when the metric switches from does not exist to exists
((draino_cordoned_nodes_total - draino_cordoned_nodes_total offset 1m) > 0)
^ creates a blip of 1 when it increases from n -> n+1
Combining the 2:
(absent(draino_cordoned_nodes_total offset 1m) == 1 and count(draino_cordoned_nodes_total) > -1) or ((draino_cordoned_nodes_total - draino_cordoned_nodes_total offset 1m) > 0)
^ or'ing them both together allowed me to detect changes as a single blip of 1 on a grafana graph, I think that's what you're after.
@neokyle has a great solution depending on the metrics you're using.
In my case I needed to solve a similar problem. The issue was that I also have labels that need to be included in the alert. And it was not feasible to use absent as that would mean generating an alert for every label. (I'm using Jsonnet so this is feasible, but still quite annoying!)
The key in my case was to use unless which is the complement operator. I wrote something that looks like this:
(my_metric unless my_metric offset 15m) > 0
This will result in a series after a metric goes from absent to non-absent, while also keeping all labels. The series will last for as long as offset is, so this would create a 15m blip. It's not super intuitive, but my understanding is that it's true when the series themselves are different. So this won't trigger when the value changes, for instance.
You could move on to adding or for (increase / delta) > 0 depending on what you're working with. This is a bit messy but to give an example:
(
my_metric
unless my_metric offset 15m
) > 0
or
(
delta(
my_metric[15m]
)
) > 0
I've noticed that transient errors executing alert rule queries cause engineers to get confusing alert notification that're false alarms. They're caused by thing like queries timing out when grafana cloud has blips in availability.
This usually happens because the engineers want an alert notification whenever a counter increments, so they set pending period to None. The default configuration for Alert state if execution error or timeout then causes the failed queries to immediately fire the alert
Suppressing all query errors is of course an option but then we risk suppressing alerts for real issues, like query logic/syntax errors and long-lasting grafana outages
Curious how others are handling this? I've found a solution that seems a bit complicated, which makes me suspicious that we're thinking about counter alerting in an unusual way.
The approach i've been suggesting is:
-
If you want to be alerted within 10 minutes of a counter increment, set the query evaluation interval to 5m and the pending period to 5m
-
Set Alert state if execution error or timeout to "Alerting"
-
Set the query range to >2x the pending period, so `increase(some_counter[610s])`. Not sure if setting it to exactly 2x (10m) risks missing an increment that happens around the time of query execution
That way there has to be two query failures in a row for the alert to fire which so far has been rare
Short answer: No.
Long answer: No, but you may get around this limitation by initializing your metrics with a value of 0 the moment you declare the respective metric in your code. Maybe there are other solutions as well I cannot think of right now.
The reason is that your metric app_error only starts to "exist" in Prometheus once it gets incremented the first time. Therefore increase will give you an increase of 0 (from nothing to 1 is... 0).
This is the ultimate solution I've come up with so far:
sum(increase(app_error[1h]) or vector(0))
+ sum((app_error unless app_error offset 1h) or vector(0))
The first part sum(increase(app_error[1h]) or vector(0)):
- if we have value 7 one hour ago and value 9 now then it will return 2
- it will sum the increases for all instances publishing this metric
- if one hour ago there were no values then
vector(0)will be returned
The second part sum((app_error unless app_error offset 1h) or vector(0)):
- if we have value 1 now and we had no value 1 hour ago - it will return 1
- it will sum the first counter values for all instances publishing this metric
- if we had a value 1 hour ago then
vector(0)will be returned
Conclusion: when you sum these 2 up you should detect either an increase or first counter increment.
I think I found a workaround for this.
For counters that existed before t, increase(_metric_[t]) is equivalent to _metric_ - _metric_ offset t. (it's not, but that is a different issue).
For counters that did not exist before t, the increase is simply the metrics value _metric_ - 0 = _metric_.
We can find out whether a metric existed at point t by querying it _metric_ offset t. And we can use that as a WHERE NOT EXISTS filter using the unless operator.
Putting it together, we get following query:
( _metric_ unless _metric offset 1d ) or ( _metric_ - _metric_ offset 1d )
^-----------new counters------------^ ^--------existing counters------^
Example
One event happens each timeframe, we want to measure the increase over 2 timeframes.
Expected:
- none for each query frame before the first occurrence
- one for the query frame on first occurrence
- 2 for each query frame beyond the first occurrence
t0 t1 t2 t3 t4 t5
_metric_ - - 1 2 3 4
_metric offset 2t - - - - 1 2
__ unless __ offset 2t - - 1 2 - -
__ <minus> __ offset 2t - - - - 2 2
=====================================================
() or () - - 1 2 2 2
Grafana example graph
total is the raw counter value, increase is the result of the query. It is still split in two series because the metric name is dropped on the - operation, but not on unless. But summing them up again works well, and is something you will probably do anyways.
Grafana graph with sum
It's really a shame prometheus makes it so hard for everyone who does not use it to display cpu temperature. This is one of the instances where my pride to have found a solution is only surpassed by my exasperation that it was necessary in the first place.
This is a "normal" behaviour. If the metric does not exist before and is then initialized with the value 1, this is not considered in functions like increase() or rate().
To catch the very first error, you need to make sure, that the metric exists from the beginning when your application starts having the initial value 0, then the first incrementatation will trigger your alert.
In your example, you should use delta function. Something similar to:
delta(node_network_receive_drop_total[5m]) > 0
That means, whenever you have a delta greater than 0 in the last 5 minutes, your alert will be triggered.
PS: consider using sum in case you have multiple instances.
In the Query tab a graph with a name something like node network drops and query something like this: increase(node_network_receive_drop_total[5m]).
When there are no drops, this graph should have a flat line at 0. When there is a drop the graph will show a line at 1 for 5 mins after the drop.
In the Alert tab create an alert with condition of WHEN max() OF query(A, 15m, now) IS ABOVE 0.
As you will only have one value returned by the query you could use max(), min(), or max() (they will all return the same value).
The A in query should match the letter on your query in the Query tab. If you have more than one query displayed on the graph, you may need to change this.
15m and now look at data from the last 15m.
I am quite new to Prometheus and need some help for a minor detail: I have modified an alerting dashboard in Grafana and I am missing a way to count all defined alert rules, basically a query to count all defines/active alert rules, ie 64 in my case:
I tried googling a solution without any meaningful result. Is there some easy query to count all defined rules?
My team switched from datadog to prometheus and counters have been the biggest pain-point. Things that just worked without thinking about it in datadog doesn't seem to have good solutions in prometheus. Surely we can't be the only ones hitting our head against the wall with these problems? How are you addressing them?
Specifically for use-cases around low-frequency counters where you want *reasonably* accurate counts. We use Created Timestamp and have dynamic labels on our counters (so pre-initializing counters to zero isn't viable or makes the data a lot less useful). That being said, these common scenarios have been a challenge:
-
Alerting on a counter increase when your counter doesn't start at zero. We use Created Timestamp gives us more confidence but it worries me that a bug/edge-case will cause us to miss an alert. Catching that would be difficult.
-
Calculating the total number of increments in a time period (ex: $__range). Sometimes short-lived series aren't counted towards the total.
-
Viewing the frequency of counter increments over time as a time series. Seems like aligning the rate and step helps but I'm still wary about the accuracy. It seems like for some time ranges it doesn't work correctly.
-
For calculating a success rate or SLI over some period of time. The approach of `sum(rate(success_total[30d])) / `sum(rate(overall_total[30d]))` doesn't always work if there are short-lived series within the query range. I see Grafana SLO feature uses recording rules, which I hope(?) improves this accuracy, but its hard to verify and is a lot of extra steps (i.e. `sum(sum_over_time((grafana_slo_success_rate_5m{})[28d:5m])) / sum(sum_over_time((grafana_slo_total_rate_5m{} )[28d:5m]))`
A lot of teams have started using logs instead of metrics for some of these scenarios. Its ambiguous when its okay to use metrics and when logs are needed, which undermines the credibility of our metrics' accuracy in general.
The frustrating thing is it seems like all the raw data is there to make these use-cases work better? Most of the time you can manually calculate the statistic you want by plotting the raw series. I'm likely over-simplifying things, and I know there's complicated edge-cases around counter-resets, missed scrapes, etc., however promql is more likely to understate the `rate`/`increase` to account for that. If anything, it would be better to overstate the `rate` since its safer to have a false positive than false negative for most monitoring use-cases. I rather have grafana widgets or promql that works for the majority of times you don't hit the complicated edge cases but overstates the rate/increase when that does happen.
I know this comes across as somewhat of a rant so I just want to say I know the prometheus maintainers put a lot of thought into their decisions and I appreciate their responsiveness to helping folks here and on slack.