Hello. Recently we are seeing many events like below. Cluster is running in version 1.27.16.
How can we find which POD has been killed? Without that information I don't really know on which pod we need to increase memory limits.
Sometimes in this place where we see "Java" is something different, so it's difficult for me sometimes to find "gulity" pod, as it's not the POD name but process name if I am not wrong.
Thanks in advance!
Warning OOMKilling 43m kernel-monitor Memory cgroup out of memory: Killed process 662566 (java) total-vm:16311612kB, anon-rss:6252312kB, file-rss:18048kB, shmem-rss:0kB, UID:1001 pgtables:13056kB oom_score_adj:873
Metric kube_pod_container_status_terminated_reason don't detect all events
Duration of kube_pod_container_status_terminated_reason metrics
There are a couple of things that you can use to get alerted on OOM and other issues, one is certainly container_oom_events_total which is provided by cadvisor but it has an issue when using it alongside Kubernetes see open GH issue here. You could also use kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} but then again it's not entirely accurate because it doesn't clear until a container is terminated for a different reason meaning it might be running and instead return such data. Last but not least you could use node_vmstat_oom_kill which as of today is working as expected, the downside of this one is that it works at node_level meaning it will not trigger only for a specific pod but for any pod having OOM issues (if you ask me this is good enough for alerting, regardless of it is a specific workload or not you want to get alerted when OOM its happening).
But if you need to use something that will give you the specific workload you could use a group_left between the kube_pod_container_status_restarts_total and the kube_pod_container_status_last_terminated_reason which should give you the restarts caused by OOM per workload... Something like the query below
sum by (namespace, pod) (kube_pod_container_status_restarts_total) * on(namespace, pod) group_left(reason) kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
As i know there is two metrics which allow you to monitor OOM. The first one is used for tracking OOMKilled status of your main process/pid. If it breach the limit pod will be restarted with this status.
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
And the second one for gathering total count of OOM events inside the container. So every time some child process or other process will breach the RAM limit they will be just killed and metric counter increased. But the container will be working as usual.
container_oom_events_total