PromQL itself does not support templating. You do have a few choices of doing this, though:
- Have whatever deployment tool you're using (Ansible, Chef, Puppet) populate that
$1with a regular expression that lists all the assets you're interested in (and use the=~matcher instead of=in your PromQL expression). Create another metric (either by pushing it to a Pushgateway or by defining it in a separate rule file) with an
asset_idlabel populated with all the asset IDs you're interested in, e.g.:should_alert{asset_id="123"} 1 should_alert{asset_id="124"} 1 should_alert{asset_id="125"} 1and then define your alert expression as:
expr: test_value >= 4 and on (asset_id) should_alert
Prometheus
prometheus.io › docs › prometheus › latest › configuration › alerting_rules
Alerting rules | Prometheus
There is also an optional keep_firing_for clause that tells Prometheus to keep this alert firing for the specified duration after the firing condition was last met. This can be used to prevent situations such as flapping alerts, false resolutions due to lack of data loss, etc. Alerting rules without the keep_firing_for clause will deactivate on the first evaluation where the condition is not met (assuming any optional for duration described above has been satisfied).
Samber
samber.github.io › home › rules
1162 Prometheus Alerting Rules for 94 Services | Awesome Prometheus Alerts
Awesome Prometheus AlertsAPA · ⌘K · Sponsored by · Search · 1162 alerting rules across 94 services and 13 categories. ⚠️ · Alert thresholds depend on the nature of your applications. Some queries may have arbitrary tolerance thresholds. Building an efficient monitoring platform takes time.
monitoring - Dynamic label values in Promethues alerting rules - Stack Overflow
I am a bit of a novice with the monitoring world. Here is my question. I want to fire an alert only for a set of assets based on asset-id. My metrics looks like the below. test_value{asset_id="1... More on stackoverflow.com
Dynamic Alert Rules
There was an error while loading. Please reload this page · So we can allow something like More on github.com
Dynamic Thresholds based on Query Value?
i guess i found a hacky way to do this Config from query results results in three of these transformations 1 Config Query C, apply to fields with type, apply to options number field max_capacity Threshold1 2 Config Query B, apply to fields with type, apply to options number field capacity use as max 2 Config Query D, apply to fields with type, apply to options number field delivery use as min More on reddit.com
Alert Manager - dynamic routing?
It kind of depends on the receiver, here is an example for email receiver: https://www.robustperception.io/using-labels-to-direct-email-notifications But doing the same for PagerDuty or Slack is going to be more challenging. More on reddit.com
GitHub
github.com › prometheus › alertmanager › issues › 2846
Make Alertmanager dynamic configuration possible · Issue #2846 · prometheus/alertmanager
March 13, 2022 - groups: - name: myalert rules: - alert: IsNotOK expr: (count by (instance, service, team, pagerduty_key) (probe_success{job="blackbox_probe"}==0 )) ==1 for: 10s labels: severity: critical origin: "{{ $labels.team }}" service: "{{ $labels.service }}" pagerduty_key: "{{ $labels.pagerduty_key }}" annotations: summary: "blabla..." I am able to setup only 1 alert in using variables here to cover all my targets. That's really nice and dynamically configurable!
Author: prometheus
Promlabs
promlabs.com › blog › 2024 › 04 › 04 › using-metrics-based-custom-thresholds-in-prometheus-alerting-rules
PromLabs | Blog - Using Metrics-Based Custom Thresholds in Prometheus Alerting Rules
April 4, 2024 - While hard-coded thresholds may be sufficient for many alerting rules, having the ability to parametrize a single generic alerting rule for many different label combinations can be very useful. You now know how to use a correlated metric to dynamically set individual thresholds for different label combinations, and how you can fall back to a default threshold if no custom threshold is defined for a given label combination.
Samber
samber.github.io › awesome-prometheus-alerts › rules.html
Awesome Prometheus alerts | Collection of alerting rules
Redirecting to /awesome-prometheus-alerts/rules/…
Faun
faun.pub › creating-universal-alerting-rules-with-prometheus-a46e53e684b4
How to create alerting rules with Prometheus | FAUN.dev() 🐾
September 20, 2021 - At first, let’s create a recording rule which will be our “dynamic” threshold variable depending on the count of CPU cores of the appropriate node. Example of recording rule for CPU core count · Then we can create our universal alerting rule by using already generated recording rules. Universal Prometheus alerting rule example ·
Prometheus
prometheus.io › docs › prometheus › latest › configuration › recording_rules
Defining recording rules | Prometheus
A limit for alerts produced by alerting rules and series produced recording rules can be configured per-group. When the limit is exceeded, all series produced by the rule are discarded, and if it's an alerting rule, all alerts for the rule, active, pending, or inactive, are cleared as well.
Google Groups
groups.google.com › g › prometheus-users › c › 4fV9qBXkfeI
How to dynamically add alerts rules in rules.conf and prometheus yml file via API or something
Is there any way I can edit the configuration files dynamically or programmically other than manually editing these files? On Friday, May 18, 2018 at 1:54:54 PM UTC+5:30, Simon Pasquier wrote: There's no API to create/update/delete alerting rules in Prometheus (the same for webhooks in ...
Java Code Geeks
javacodegeeks.com › home › software development
Prometheus Sample Alert Rules - Java Code Geeks
June 21, 2023 - Alerting and Notification: Prometheus has built-in support for defining alert rules based on metric conditions. It can generate alerts when certain thresholds are exceeded or specific conditions are met.
Adeptia
docs.adeptia.com › articles
Configuring rules and alerts in Prometheus
July 23, 2024 - Open in new window · Direct link to topic in this publication: