To send an alert to your gmail account, you need to setup the alertmanager configuration in a file say alertmanager.yaml:

cat <<EOF > alertmanager.yml
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me

receivers:
- name: email-me
  email_configs:
  - to: $GMAIL_ACCOUNT
    from: $GMAIL_ACCOUNT
    smarthost: smtp.gmail.com:587
    auth_username: "$GMAIL_ACCOUNT"
    auth_identity: "$GMAIL_ACCOUNT"
    auth_password: "$GMAIL_AUTH_TOKEN"
EOF

Now, as you're using kube-prometheus so you will have a secret named alertmanager-main that is default configuration for alertmanager. You need to create a secret alertmanager-main again with the new configuration using following command:

kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring

Now you're alertmanager is set to send an email whenever it receive alert from the prometheus.

Now you need to setup an alert on which your mail will be sent. You can set up DeadManSwitch alert which fires in every case and it is used to check your alerting pipeline

groups:
- name: meta
  rules:
    - alert: DeadMansSwitch
      expr: vector(1)
      labels:
        severity: critical
      annotations:
        description: This is a DeadMansSwitch meant to ensure that the entire Alerting
          pipeline is functional.
        summary: Alerting DeadMansSwitch

After that the DeadManSwitch alert will be fired and should send email to your mail.

Reference link:

https://coreos.com/tectonic/docs/latest/tectonic-prometheus-operator/user-guides/configuring-prometheus-alertmanager.html

EDIT:

The deadmanswitch alert should go in a config-map which your prometheus is reading. I will share the relevant snaps from my prometheus here:

"spec": {
        "alerting": {
            "alertmanagers": [
                {
                    "name": "alertmanager-main",
                    "namespace": "monitoring",
                    "port": "web"
                }
            ]
        },
        "baseImage": "quay.io/prometheus/prometheus",
        "replicas": 2,
        "resources": {
            "requests": {
                "memory": "400Mi"
            }
        },
        "ruleSelector": {
            "matchLabels": {
                "prometheus": "prafull",
                "role": "alert-rules"
            }
        },

The above config is of my prometheus.json file which have the name of alertmanager to use and the ruleSelector which will select the rules based on prometheus and role label. So I have my rule configmap like:

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-rules
  namespace: monitoring
  labels:
    role: alert-rules
    prometheus: prafull
data:
  alert-rules.yaml: |+
   groups:
   - name: alerting_rules
     rules:
       - alert: LoadAverage15m
         expr: node_load15 >= 0.50
         labels:
           severity: major
         annotations:
           summary: "Instance {{ $labels.instance }} - high load average"
           description: "{{ $labels.instance  }} (measured by {{ $labels.job }}) has high load average ({{ $value }}) over 15 minutes."

Replace the DeadManSwitch in above config map.

Answer from Prafull Ladha on Stack Overflow
🌐
Prometheus
prometheus.io › docs › prometheus › latest › configuration › alerting_rules
Alerting rules | Prometheus
To manually inspect which alerts are active (pending or firing), navigate to the "Alerts" tab of your Prometheus instance.
🌐
GitHub
gist.github.com › cherti › 61ec48deaaab7d288c9fcf17e700853a
send a dummy alert to prometheus-alertmanager · GitHub
#!/bin/bash # Set default values name=$RANDOM url='https://alertmanager.local/api/v1/alerts' summary='Testing summary!' instance="$name.example.net" default_severity='warning' # Function to send alert send_alert() { local status=$1 local custom_severity=$2 local current_severity=${custom_severity:-$default_severity} curl -XPOST $url -d "[ { \"status\": \"$status\", \"labels\": { \"alertname\": \"$name\", \"service\": \"my-service\", \"severity\":\"$current_severity\", \"instance\": \"$instance\" }, \"annotations\": { \"summary\": \"$summary\" }, \"generatorURL\": \"https://prometheus.local/<generating_expression>\" } ]" echo "" } # Main script echo "Firing up alert $name" send_alert "firing" "$1" read -p "Press enter to resolve alert" echo "Sending resolve" send_alert "resolved" "$1"
Discussions

Prometheus trigger script on alert
prometheus alert is firing I love the wordplay here, Wether it was intentional or not More on reddit.com
🌐 r/sysadmin
1
0
October 21, 2021
kubernetes - Triggering alerts on Prometheus dashboard - Stack Overflow
Is it possible to trigger some alerts on the Prometheus dashboard by manually stopping respective services on the Kubernetes cluster in order to verify that I'm receiving alert for issues on Promet... More on stackoverflow.com
🌐 stackoverflow.com
Manual alert for routes and receivers test
Now I must wait for real alert to test if I configured routes or receivers correctly. It would be great to add possibility of manual alert trigger to debug/test routes and receivers. It may be adde... More on github.com
🌐 github.com
10
July 21, 2016
Possible to alert if there has not been any change to a value for 48 hours?
I feel like maybe this is trying to fit a square peg in a round hole. Does the actual amount of time since last modified actually matter in a way where it needs to be collected and stored and charted out forever as a metric, or do you really just want to know about discrete events that would be better served by using logs? More on reddit.com
🌐 r/grafana
19
1
September 5, 2023
Top answer
1 of 2
5

To send an alert to your gmail account, you need to setup the alertmanager configuration in a file say alertmanager.yaml:

cat <<EOF > alertmanager.yml
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me

receivers:
- name: email-me
  email_configs:
  - to: $GMAIL_ACCOUNT
    from: $GMAIL_ACCOUNT
    smarthost: smtp.gmail.com:587
    auth_username: "$GMAIL_ACCOUNT"
    auth_identity: "$GMAIL_ACCOUNT"
    auth_password: "$GMAIL_AUTH_TOKEN"
EOF

Now, as you're using kube-prometheus so you will have a secret named alertmanager-main that is default configuration for alertmanager. You need to create a secret alertmanager-main again with the new configuration using following command:

kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring

Now you're alertmanager is set to send an email whenever it receive alert from the prometheus.

Now you need to setup an alert on which your mail will be sent. You can set up DeadManSwitch alert which fires in every case and it is used to check your alerting pipeline

groups:
- name: meta
  rules:
    - alert: DeadMansSwitch
      expr: vector(1)
      labels:
        severity: critical
      annotations:
        description: This is a DeadMansSwitch meant to ensure that the entire Alerting
          pipeline is functional.
        summary: Alerting DeadMansSwitch

After that the DeadManSwitch alert will be fired and should send email to your mail.

Reference link:

https://coreos.com/tectonic/docs/latest/tectonic-prometheus-operator/user-guides/configuring-prometheus-alertmanager.html

EDIT:

The deadmanswitch alert should go in a config-map which your prometheus is reading. I will share the relevant snaps from my prometheus here:

"spec": {
        "alerting": {
            "alertmanagers": [
                {
                    "name": "alertmanager-main",
                    "namespace": "monitoring",
                    "port": "web"
                }
            ]
        },
        "baseImage": "quay.io/prometheus/prometheus",
        "replicas": 2,
        "resources": {
            "requests": {
                "memory": "400Mi"
            }
        },
        "ruleSelector": {
            "matchLabels": {
                "prometheus": "prafull",
                "role": "alert-rules"
            }
        },

The above config is of my prometheus.json file which have the name of alertmanager to use and the ruleSelector which will select the rules based on prometheus and role label. So I have my rule configmap like:

kind: ConfigMap
apiVersion: v1
metadata:
  name: prometheus-rules
  namespace: monitoring
  labels:
    role: alert-rules
    prometheus: prafull
data:
  alert-rules.yaml: |+
   groups:
   - name: alerting_rules
     rules:
       - alert: LoadAverage15m
         expr: node_load15 >= 0.50
         labels:
           severity: major
         annotations:
           summary: "Instance {{ $labels.instance }} - high load average"
           description: "{{ $labels.instance  }} (measured by {{ $labels.job }}) has high load average ({{ $value }}) over 15 minutes."

Replace the DeadManSwitch in above config map.

2 of 2
0

If you are using kube-promehtheus, by default it have alertmanager-main secret and prometheus kind setup.

Step 1: You have to remove alertmanager-main secret

kubectl delete secret alertmanager-main -n monitoring

Step 2: As Prafull explained create secret with new change

cat <<EOF > alertmanager.yaml
route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me

receivers:
- name: email-me
  email_configs:
  - to: $GMAIL_ACCOUNT
    from: $GMAIL_ACCOUNT
    smarthost: smtp.gmail.com:587
    auth_username: "$GMAIL_ACCOUNT"
    auth_identity: "$GMAIL_ACCOUNT"
    auth_password: "$GMAIL_AUTH_TOKEN"
EOF

kubectl create secret generic alertmanager-main --from-file=alertmanager.yaml -n monitoring

Step3 : You have to add new prometheus rule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: null
  labels:
    prometheus: k8s
    role: alert-rules
  name: prometheus-podfail-rules
spec:
  groups:
  - name: ./podfail.rules
    rules:
    - alert: PodFailAlert
      expr: sum(kube_pod_container_status_restarts_total{container="ffmpeggpu"}) BY (container) > 10

NB : The role should be role: alert-rules which is specified in the rule selector prometheus kind. To check that is it used, use:

kubectl get prometheus k8s -n monitoring -o yaml
🌐
Medium
medium.com › devops-dudes › prometheus-alerting-with-alertmanager-e1bbba8e6a8e
Prometheus Alerting with AlertManager | by Sylia CHIBOUB | DevOps Dudes | Medium
November 9, 2022 - Prometheus server is going to track incoming time series data, once any of the rules defined in etc/prometheus/alert.rules.yml is satisfied, an alert is triggered to AlertManager service that notifies the client on Slack.
🌐
Google Groups
groups.google.com › g › prometheus-users › c › zHMigUUtwwg
Is there a way to get prometheus to trigger alerts right away?
October 25, 2017 - To answer your question, the correct thing to get an immediate alert is to remove the FOR condition. Any FOR condition will require that the alert fires for two consecutive evaluation intervals. ... -- You received this message because you are subscribed to the Google Groups "Prometheus Users" ...
🌐
Stackhero
stackhero.io › en-US › services › Prometheus › documentations › Alerts
Prometheus: Alerts
Stackhero handles the upgrade process for you, minimizing downtime and manual intervention. High performance and strong security are built in, thanks to your own private, dedicated infrastructure. Get up and running in about 5 minutes. Stackhero takes care of the setup so you can focus on monitoring, not maintenance. Try Prometheus cloud hosting on Stackhero to streamline your monitoring and alerting workflows. Prometheus can analyze your metrics and trigger alerts based on rules that you define.
🌐
OpsRamp
opsramp.com › home › guides › prometheus alerting
The Guide To Prometheus Alerting : OpsRamp
September 9, 2022 - Alertmanager is most powerful when combined with an automation tool that would accept Prometheus events as input and use them to trigger run-books that would automate the action-taking required to resolve the underlying cause of the problem.
Find elsewhere
🌐
Reddit
reddit.com › r/sysadmin › prometheus trigger script on alert
r/sysadmin on Reddit: Prometheus trigger script on alert
October 21, 2021 - Hi, I need to trigger a script when a particular prometheus alert is firing, what is the best way to achieve this?
🌐
Swiftorial
swiftorial.com › tutorials › prometheus › alerts › creating alerts
Creating Alerts | Alerts | Prometheus Tutorial
May 11, 2026 - After setting up your alerting ... can force an alert to trigger by simulating high CPU usage or by temporarily modifying the alerting rule to test different conditions. To check if the alerts are working as expected, you can use the Prometheus UI:...
🌐
Prometheus
prometheus.io › docs › alerting › latest › alertmanager
Alertmanager | Prometheus
The Alertmanager handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie.
🌐
Tech Tutorials
techtutorials.tv › sections › management-and-monitoring › prometheus-how-to-send-alerts
How to Send Alerts in Prometheus - Alertmanager | Tech Tutorials
September 29, 2023 - On the other hand maybe you want to trigger an alert if a metric goes above or below a certain in which case you’ll be using the > and < operators · And sometimes you’ll combine metrics together to return percentage values · We’ll run through some more examples, but a really good place to look for other alerts is here: https://samber.github.io/awesome-prometheus-alerts/
🌐
Medium
medium.com › @texasdave2 › create-a-test-alert-in-prometheus-for-kubernetes-7f359240f1bf
Create a test alert in your Prometheus pipeline for Kubernetes | by David O'Dell | Medium
July 15, 2019 - alert: TEST ALERT FROM PROMETHEUS PLEASE ACKNOWLEDGE expr: prometheus_build_info{instance="localhost:9090"} == 1 for: 10s labels: cluster: TEST severity: warning annotations: action: TESTING PLEASE ACKNOWLEDGE, NO FURTHER ACTION REQUIRED ONLY A TEST description: TEST ALERT FROM {{ $labels.instance }}
🌐
Fabian Lee
fabianlee.org › 2022 › 07 › 03 › prometheus-sending-a-test-alert-through-alertmanager
Prometheus: sending a test alert through AlertManager | Fabian Lee : Software Engineer
June 6, 2023 - Real alerts typically have scrape delays and then durations that must be met, so this is a way of getting almost immediate feedback on your routing and receivers. # use namespace where prometheus is installed my_ns=prom # port forward AlertManager pod to localhost:9093 kubectl port-forward statefulset/alertmanager-prom-stack-kube-prometheus-alertmanager -n $my_ns 9093 # send test alert to localhost:9093 $ curl -H 'Content-Type: application/json' -d '[{"labels":{"alertname":"myalert"}}]' http://127.0.0.1:9093/api/v1/alerts {"status":"success"}
🌐
Robusta
docs.robusta.dev › master › playbook-reference › triggers › prometheus.html
Prometheus and AlertManager - Robusta documentation
The node on which the command executes will be selected according to the alert labels. customPlaybooks: - triggers: - on_prometheus_alert: alert_name: HostHighCpuLoad scope: include: - labels: - "deployment=nginx" actions: - node_bash_enricher: bash_command: ps aux
🌐
Stack Overflow
stackoverflow.com › questions › 64427391 › triggering-alerts-on-prometheus-dashboard
kubernetes - Triggering alerts on Prometheus dashboard - Stack Overflow
As you said, triggering alerts on the Prometheus dashboard by manually stopping respective services on the Kubernetes cluster. This will enable you to verify alerts for issues on your Prometheus dashboard.
🌐
Better Stack
betterstack.com › community › guides › monitoring › prometheus-alertmanager
Effective Alerting with Prometheus Alertmanager | Better Stack Community
The rule_files line specifies that Prometheus can find alert rules in the /etc/prometheus/alerts.yml file, and the alerting section sends any triggered alerts to an Alertmanager instance running at alertmanager:9093.
🌐
Squadcast
squadcast.com › blog › prometheus-sample-alert-rules
Prometheus Alert Rules: Comprehensive Guide with Best Practices & Samples | Squadcast
April 17, 2023 - Prometheus uses the PromQL (Prometheus Query Language) to create alerting rules. The alert expression is the core of a Prometheus alert. You use PromQL to define the condition that triggers an alert.
🌐
GitHub
github.com › prometheus › alertmanager › issues › 437
Manual alert for routes and receivers test · Issue #437 · prometheus/alertmanager
July 21, 2016 - Now I must wait for real alert to test if I configured routes or receivers correctly. It would be great to add possibility of manual alert trigger to debug/test routes and receivers. It may be added on "Status" page as a form with fields for Status, Labels and Annotations (and maybe other fields from model.Alert).
Author: prometheus
🌐
Last9
last9.io › blog › prometheus-alerting-examples
Prometheus Alerting Examples for Developers | Last9
June 2, 2025 - To build smarter alerts or custom dashboards, this Prometheus API guide breaks down how to query and fetch metrics programmatically. Some alerts only make sense if the underlying service is running. For example, monitoring slow queries in a database is pointless if the database is down. By combining conditions, you can avoid alerts that don’t apply, keeping your alerts relevant and easier to act on. ... Here, the alert only triggers if the database is up and slow queries have been detected for 10 minutes.