🌐
Reddit
reddit.com › r/kubernetes › help me finding pods killed by oom
r/kubernetes on Reddit: Help me finding pods killed by OOM
September 20, 2024 -

Hello. Recently we are seeing many events like below. Cluster is running in version 1.27.16.
How can we find which POD has been killed? Without that information I don't really know on which pod we need to increase memory limits.
Sometimes in this place where we see "Java" is something different, so it's difficult for me sometimes to find "gulity" pod, as it's not the POD name but process name if I am not wrong.
Thanks in advance!

Warning  OOMKilling  43m    kernel-monitor  Memory cgroup out of memory: Killed process 662566 (java) total-vm:16311612kB, anon-rss:6252312kB, file-rss:18048kB, shmem-rss:0kB, UID:1001 pgtables:13056kB oom_score_adj:873

🌐
Blog
songrgg.github.io › operation › how-to-alert-for-Pod-Restart-OOMKilled-in-Kubernetes
How to alert for Pod Restart & OOMKilled in Kubernetes - Blog | Songrgg
July 14, 2021 - When the containers were killed because of OOMKilled, the container’s exit reason will be populated as OOMKilled and meanwhile it will emit a gauge kube_pod_container_status_last_terminated_reason { reason: "OOMKilled", container: "some-container" } ,
Discussions

Metric kube_pod_container_status_terminated_reason don't detect all events
What happened: The metric kube_pod_container_status_terminated_reason is still experiment but since a very long time. This metric can be very useful for monitoring alerts but is not detecting all "... More on github.com
🌐 github.com
10
August 18, 2023
Duration of kube_pod_container_status_terminated_reason metrics
I understand that this metrics is keeping track of pods which were terminated and the reason for it, but the metrics store data of how many days? Like the data will be of last two days or last 12 h... More on github.com
🌐 github.com
16
January 15, 2018
🌐
Netice9
netice9.com › blog › guide-to-oomkill-alerting-in-kubernetes-clusters
Guide to OOMKill Alerting in Kubernetes Clusters · NetIce9
November 23, 2020 - It turns out it has a metric called kube_pod_container_status_last_terminated_reason. The value of the metric is 1 when a container in a pod has terminated with an error. Based on the exit code, the reason label will be set to OOMKilled if the exit code was 137.
🌐
Medium
medium.com › @vissa.jyothi › observability-for-production-deployments-tracking-oomkilled-restarts-in-grafana-6f2fae4b922b
Observability for Production Deployments: Tracking OOMKilled Restarts in Grafana | by Jyothi Gundavarapu | Medium
October 28, 2024 - If you are a Grafana user then you should leverage it’s Alerting feature.For deployments on Friday evenings, use Grafana’s monitoring and alerting capabilities to observe pod restarts in production. By focusing on “OOMKilled” errors, your team gains improved observability into memory-related issues affecting pod stability over the weekend. I. PromQL query for Pod Restarts · sum by (pod, contaner, reason) (increase(kube_pod_container_status_last_terminated_reason{reason=”OOMKilled”}[30m])) > 0
🌐
Kubernetes
kubernetes.io › docs › tasks › debug › debug-application › determine-reason-pod-failure
Determine the Reason for Pod Failure | Kubernetes
January 22, 2024 - Kubernetes use the contents from the specified file to populate the Container's status message on both success and failure. The termination message is intended to be brief final status, such as an assertion failure message.
🌐
GitHub
github.com › kubernetes › kube-state-metrics › blob › main › docs › metrics › workload › pod-metrics.md
kube-state-metrics/docs/metrics/workload/pod-metrics.md at main · kubernetes/kube-state-metrics
The pod status reasons. Emitted only for the reason that is actually set; a missing series does not mean the reason is false. An unrecognized pod.status.reason is reported as Other; conditions and container-terminated reasons outside this list are not reported.
Author: kubernetes
🌐
OneUptime
oneuptime.com › home › blog › how to set up alerts for pod failures in kubernetes
How to Set Up Alerts for Pod Failures in Kubernetes
December 17, 2025 - groups: - name: pod-failures rules: - alert: PodCrashLoopBackOff expr: | max_over_time(kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}[5m]) == 1 for: 5m labels: severity: critical annotations: summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is in CrashLoopBackOff" description: "Container {{ $labels.container }} in pod {{ $labels.pod }} has been in CrashLoopBackOff for more than 5 minutes." runbook_url: "https://runbooks.example.com/pod-crashloop" Detect containers whose last termination reason was memory exhaustion: - alert: PodOOMKilled expr: | kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1 for: 0m labels: severity: warning annotations: summary: "Container {{ $labels.container }} OOMKilled in {{ $labels.namespace }}/{{ $labels.pod }}" description: "Container was last killed due to out of memory.
Find elsewhere
🌐
Medium
reddeppa-s.medium.com › kubernetes-ale-cb255f58338b
Kubernetes Alerts. Alarms for specific metric thresholds… | by Reddeppa S | Medium
January 6, 2023 - - alert: KubernetesContainerOomKiller expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) == 1 for: 0m labels: severity: warning annotations: summary: Kubernetes container oom killer (instance {{ $labels.instance }}) description: "Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}" Kubernetes Job failed ·
Top answer
1 of 3
4

There are a couple of things that you can use to get alerted on OOM and other issues, one is certainly container_oom_events_total which is provided by cadvisor but it has an issue when using it alongside Kubernetes see open GH issue here. You could also use kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} but then again it's not entirely accurate because it doesn't clear until a container is terminated for a different reason meaning it might be running and instead return such data. Last but not least you could use node_vmstat_oom_kill which as of today is working as expected, the downside of this one is that it works at node_level meaning it will not trigger only for a specific pod but for any pod having OOM issues (if you ask me this is good enough for alerting, regardless of it is a specific workload or not you want to get alerted when OOM its happening).

But if you need to use something that will give you the specific workload you could use a group_left between the kube_pod_container_status_restarts_total and the kube_pod_container_status_last_terminated_reason which should give you the restarts caused by OOM per workload... Something like the query below

sum by (namespace, pod) (kube_pod_container_status_restarts_total) * on(namespace, pod) group_left(reason) kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
2 of 3
3

As i know there is two metrics which allow you to monitor OOM. The first one is used for tracking OOMKilled status of your main process/pid. If it breach the limit pod will be restarted with this status.

kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}

And the second one for gathering total count of OOM events inside the container. So every time some child process or other process will breach the RAM limit they will be just killed and metric counter increased. But the container will be working as usual.

container_oom_events_total
🌐
GitHub
github.com › kubernetes › kube-state-metrics › issues › 2153
Metric kube_pod_container_status_terminated_reason don't detect all events · Issue #2153 · kubernetes/kube-state-metrics
August 18, 2023 - What happened: The metric ... since a very long time. This metric can be very useful for monitoring alerts but is not detecting all "Errors" or "OOMKilled" often the event is not collected....
Author: kubernetes
🌐
Kubernetes
kubernetes.io › docs › concepts › workloads › pods › pod-lifecycle
Pod Lifecycle | Kubernetes
June 20, 2026 - If a container has a preStop hook configured, this hook runs before the container enters the Terminated state. Kubernetes manages container failures within Pods using a restartPolicy defined in the Pod spec. This policy determines how Kubernetes reacts to containers exiting due to errors or other reasons, which falls in the following sequence:
🌐
GitHub
github.com › kubernetes › kube-state-metrics › blob › main › internal › store › pod_test.go
kube-state-metrics/internal/store/pod_test.go at main · kubernetes/kube-state-metrics
# HELP kube_pod_container_status_terminated_reason Describes the reason the container is currently in terminated state. # HELP kube_pod_container_status_waiting [STABLE] Describes whether the container is currently in waiting state.
Author: kubernetes
🌐
Sysdig
sysdig.com › blog › debug-kubernetes-crashloopbackoff
What is Kubernetes CrashLoopBackOff? And how to fix it | Sysdig
March 27, 2026 - · From the describe output, you can extract the following information: Current pod State is Waiting. Reason for the Waiting state is “CrashLoopBackOff”. Last (or previous) state was “Terminated”.
🌐
Unofficial Kubernetes
unofficial-kubernetes.readthedocs.io › en › latest › tasks › debug-application-cluster › determine-reason-pod-failure
Determine reason pod failure - Unofficial Kubernetes
apiVersion: v1 kind: Pod ... lastState: terminated: containerID: ... exitCode: 0 finishedAt: ... message: | Sleep expired ... Use a Go template to filter the output so that it includes only the termination message: {% raw %} kubectl get pod termination-demo -o go-template="{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"{% endraw %}
🌐
GitHub
github.com › kubernetes › kube-state-metrics › issues › 344
Duration of kube_pod_container_status_terminated_reason metrics · Issue #344 · kubernetes/kube-state-metrics
January 15, 2018 - I understand that this metrics is keeping track of pods which were terminated and the reason for it, but the metrics store data of how many days? Like the data will be of last two days or last 12 h...
Author: kubernetes
🌐
Baeldung
baeldung.com › home › kubernetes › alerts in kubernetes for pod failing
Alerts in Kubernetes for Pod Failing | Baeldung on Ops
March 18, 2025 - When a pod exceeds its assigned CPU or memory limits, it may be evicted or restarted. In particular, this can be caused by inefficient resource allocation, memory leaks, or unexpected traffic spikes.