There are a couple of things that you can use to get alerted on OOM and other issues, one is certainly container_oom_events_total which is provided by cadvisor but it has an issue when using it alongside Kubernetes see open GH issue here. You could also use kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} but then again it's not entirely accurate because it doesn't clear until a container is terminated for a different reason meaning it might be running and instead return such data. Last but not least you could use node_vmstat_oom_kill which as of today is working as expected, the downside of this one is that it works at node_level meaning it will not trigger only for a specific pod but for any pod having OOM issues (if you ask me this is good enough for alerting, regardless of it is a specific workload or not you want to get alerted when OOM its happening).
But if you need to use something that will give you the specific workload you could use a group_left between the kube_pod_container_status_restarts_total and the kube_pod_container_status_last_terminated_reason which should give you the restarts caused by OOM per workload... Something like the query below
sum by (namespace, pod) (kube_pod_container_status_restarts_total) * on(namespace, pod) group_left(reason) kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
There are a couple of things that you can use to get alerted on OOM and other issues, one is certainly container_oom_events_total which is provided by cadvisor but it has an issue when using it alongside Kubernetes see open GH issue here. You could also use kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} but then again it's not entirely accurate because it doesn't clear until a container is terminated for a different reason meaning it might be running and instead return such data. Last but not least you could use node_vmstat_oom_kill which as of today is working as expected, the downside of this one is that it works at node_level meaning it will not trigger only for a specific pod but for any pod having OOM issues (if you ask me this is good enough for alerting, regardless of it is a specific workload or not you want to get alerted when OOM its happening).
But if you need to use something that will give you the specific workload you could use a group_left between the kube_pod_container_status_restarts_total and the kube_pod_container_status_last_terminated_reason which should give you the restarts caused by OOM per workload... Something like the query below
sum by (namespace, pod) (kube_pod_container_status_restarts_total) * on(namespace, pod) group_left(reason) kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
As i know there is two metrics which allow you to monitor OOM. The first one is used for tracking OOMKilled status of your main process/pid. If it breach the limit pod will be restarted with this status.
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
And the second one for gathering total count of OOM events inside the container. So every time some child process or other process will breach the RAM limit they will be just killed and metric counter increased. But the container will be working as usual.
container_oom_events_total
Hello. Recently we are seeing many events like below. Cluster is running in version 1.27.16.
How can we find which POD has been killed? Without that information I don't really know on which pod we need to increase memory limits.
Sometimes in this place where we see "Java" is something different, so it's difficult for me sometimes to find "gulity" pod, as it's not the POD name but process name if I am not wrong.
Thanks in advance!
Warning OOMKilling 43m kernel-monitor Memory cgroup out of memory: Killed process 662566 (java) total-vm:16311612kB, anon-rss:6252312kB, file-rss:18048kB, shmem-rss:0kB, UID:1001 pgtables:13056kB oom_score_adj:873
Here are most likely reasons of prometheus eating memory:
- Overwhelming number of timeseries. Considering background this is most plausable. In prometheus datapoints are taking not much memory compared to unique timeseries. I couldn't find link now but AFAIR one datapoint takes around 4 bytes while timeseries w/o any datapoints takes around 1Kb. So having timeseries even w/o any datapoints will take space and might take memory. You can rule out this reason by comparing number of timeseries in prod and stage:
count({__name__=~".+"}). If there are significantly more timeseries in prod you'll have to figure out why and probably further reduce the number. - PromQL queries that load to much data into memory. If you have queries requesting long time period or huge amount of timeseries it could also be a reason since prometheus tries to load requested data into memory. Since you have OOM constantly reproducing you can test this assumption by blocking all queries to prometheus and see if it still hits OOM. It may be worth to look at query log too.
- Not enough memory on node. It could be just that other containers consume memory on node and prometheus is killed because it has lower QoS. Just make sure prometheus falls into guaranteed QoS.
The cause of my issue was a broken keycloak deployment in the keycloak namespace. An old keycloak setup was creating an high number of replicasets (around 36000), which caused the high cardinality for the replicaset-related queries in Prom.
The issue was not in staging since staging didn't mirror that configuration completely.
I had already tried the following relabeling to kube-state-metrics, dropping the queries before ingestion:
- regex: '(kube_replicaset_status_observed_generation|kube_replicaset_status_replicas|kube_replicaset_labels|kube_replicaset_created|kube_replicaset_annotations|kube_replicaset_status_ready_replicas|kube_replicaset_spec_replicas|kube_replicaset_owner|kube_replicaset_status_fully_labeled_replicas|kube_replicaset_metadata_generation)'
action: drop
sourceLabels: [__name__]
but it proved to be too conservative. After adding:
- regex: 'keycloak'
action: drop
sourceLabels: [namespace]
my instance became stable again.
Periodically I see the container Status: terminated - OOMKilled (exit code: 137)
But it's scheduled to the node with plenty of memory
As you may have already seen it's evident you are hitting more than the 1GB configured. The answer probably lies in how you are using Prometheus and what usage limits you are hitting for 1GB. Some of the things you can look at:
- Number of Time Series
- Average Labels Per Time Series
- Number of Unique Label Pairs
- Scrape Interval (s)
- Bytes per Sample
You can find a memory calculator for the usage above 👉 here.
✌️
The 1Gi memory limit for Prometheus pod is quite low for Kubernetes monitoring, where millions of metrics are scraped from thousands of targets (pods, nodes, endpoints, etc.).
The recommendation is to raise the memory limit for Prometheus pod until it stops crashing with out of memory error.
It is recommended setting up monitoring for the Prometheus itself - it exports own metrics at http://prometheus-host:9090/metrics url - see, for example, http://demo.robustperception.io:9090/metrics .
Prometheus memory usage can be decreased in the following ways:
- To increase
scrape_intervalin Prometheus configs, so it scrapes targets less frequently. This reduces Prometheus memory usage, since Prometheus stores recently scraped metrics in memory for up to 2 hours. - To filter out unneeded scrape targets via
relabel_configs. See https://www.robustperception.io/life-of-a-label . - To filter out unneeded metrics via
metric_relabel_configs. See https://www.robustperception.io/life-of-a-label .
P.S. There are alternative Prometheus-like solutions, which can use lower amounts of memory when scraping the same set of targets. See, for example, vmagent and VictoriaMetrics.