See offset docs here: https://prometheus.io/docs/prometheus/latest/querying/basics/ I’m not sure if increase is reliable across long time windows like this, but it should handle resets in the counters, which the offset won’t handle. If you know that the metric is monotonic and never resets, then you could use the offset instead. Finally, you can take a look at recording rules, which allow to generate a new precalculated metric out of another metrics. More on reddit.com

So the alert you have written should test against 0, not 1 for the up metric. Alert are fired with the expression is true not false. Like in your test with just ‘up{job=“website”}==0’ up is fairly simple metric and in your case does the job. But their is a caveat, if your metrics are hosted in your website (like the website has a page with a /metrics that you configure Prometheus to scrape) and the website goes down, the missing metrics will not fire as there is no more up metric to test against. For that you have the ‘absent’ operator that you can setup like ‘absent(up{job=“website”})’ and that will be of value of 1 if the metrics doesn’exist (like mentioned website goes down with it’s metrics) Mind that the ‘absent’ operation will not have labels on it so if you compose the alert message with like “{{ $labels.instance }}’ that would not work, in my case I write the label that I expect directly in the label section of the alert. More on reddit.com

For alerts with big `for` value, like in your example, it is better to check ALERTS_FOR_STATE metric to see whether the condition has been triggered and for how long. It could be that condition was true up to 9min in a row and then once it wasn't - so all the progress will be reset. If you care to dig deeper, you can try using vmalert - a ruler component from VictoriaMetrics stack. You can feed it with the same alerting rule and configure it to query Prometheus for data. When you have that running you can check the alert's state in the vmalert UI. This would show you how many successful consecutive evaluations it had. More on reddit.com

My Prometheus instance is consuming a lot of memory over 13Gi, the node has a max 16Gb, so it's getting killed by k8s, how can I configure or should… More on reddit.com