If memory is exhaustively used up by processes, to the extent which can possibly threaten the stability of the system, then the OOM killer comes into the picture.
NOTE: It is the task of the OOM Killer to continue killing processes until enough memory is freed for the smooth functioning of the rest of the process that the Kernel is attempting to run.
The OOM Killer has to select the best process(es) to kill. Best here refers to that process which will free up the maximum memory upon killing and is also the least important to the system.
The primary goal is to kill the least number of processes that minimizes the damage done and at the same time maximizing the amount of memory freed.
To facilitate this, the kernel maintains an oom_score for each of the processes. You can see the oom_score of each of the processes in the /proc filesystem under the pid directory.
$ cat /proc/10292/oom_score
The higher the value of oom_score of any process, the higher is its likelihood of getting killed by the OOM Killer in an out-of-memory situation.
How is the OOM_Score calculated?
In David's patch set, the old badness() heuristics are almost entirely gone. Instead, the calculation turns into a simple question of what percentage of the available memory is being used by the process. If the system as a whole is short of memory, then "available memory" is the sum of all RAM and swap space available to the system.
If instead, the OOM situation is caused by exhausting the memory allowed to a given cpuset/control group, then "available memory" is the total amount allocated to that control group. A similar calculation is made if limits imposed by a memory policy have been exceeded. In each case, the memory use of the process is deemed to be the sum of its resident set (the number of RAM pages it is using) and its swap usage.
This calculation produces a percent-times-ten number as a result; a process which is using every byte of the memory available to it will have a score of 1000, while a process using no memory at all will get a score of zero. There are very few heuristic tweaks to this score, but the code does still subtract a small amount (30) from the score of root-owned processes on the notion that they are slightly more valuable than user-owned processes.
One other tweak which is applied is to add the value stored in each process's oom_score_adj variable, which can be adjusted via /proc. This knob allows the adjustment of each process's attractiveness to the OOM killer in user space; setting it to -1000 will disable OOM kills entirely, while setting to +1000 is the equivalent of painting a large target on the associated process.
References
http://www.queryhome.com/15491/whats-happening-kernel-starting-killer-choose-which-process https://serverfault.com/a/571326
Answer from Ramesh on Stack ExchangeThe Linux kernel has a mechanism called “out-of-memory killer” (aka OOM killer) which is used to recover memory on a system. The OOM killer allows killing a single task (called also oom victim) while that task will terminate in a reasonable time and thus free up memory.
When OOM killer does its job we can find indications about that by searching the logs (like /var/log/messages and grepping for “Killed”). If you want to configure the “OOM killer” I suggest reading the following link https://www.oracle.com/technical-resources/articles/it-infrastructure/dev-oom-killer.html.
It is important to understand that the OOM killer chooses between processes based on the “oom_score”. If you want to see the value for a specific process we can just read “/proc/[PID]/oom_score” - as shown in the screenshot below. If we want to alter the score we can do it using “/proc/[PID]/oom_score_adj” - as shown also in the screenshot below. The valid range is from 0 (never kill) to 1000 (always kill), the lower the value is the lower is the probability the process will be killed. For more information please read https://man7.org/linux/man-pages/man5/proc.5.html.
In the next post I am going to elaborate about the kernel thread “oom_reaper”. See you in my next post ;-)
linux - How does the OOM killer decide which process to kill first? - Unix & Linux Stack Exchange
Linux - Out-of-Memory Killer (OOM killer)
Why is Linux so bad at handling OOM scenarios?
Is it possible to disable the OOM killer, constant "Device memory is nearly full."
Videos
The key to triggering the OOM killer quickly is to avoid getting bogged down by disk accesses. So:
Avoid swapping, unless your goal is specifically to test the behavior of OOM when swap is used. You can disable swap before the test, then re-enable it afterwards.
swapon -stells you what swaps are currently enabled.sudo swapoff -adisables all swaps;sudo swapon -ais usually sufficient to reenable them.Avoid interspersing memory accesses with non-swap disk accesses. That globbing-based method eventually uses up your available memory (given enough entries in your filesystem), but the reason it needs so much memory is to store information that it obtains by accessing your filesystem. Even with an SSD, it's likely that much of the time is spent reading from disk, even if swap is turned off. If your goal is specifically to test OOM behavior for memory accesses that are interspersed with disk accesses, that method is reasonable, perhaps even ideal. Otherwise, you can achieve your goal much faster.
Once you've disabled swap, any method that seldom reads from a physical disk should be quite fast. This includes the tail /dev/zero (found by falstaff, commented above by Doug Smythies). Although it reads from the character device /dev/zero, that "device" just generates null bytes (i.e., bytes of all zeros) and doesn't involve any physical disk access once the device node has been opened. That method works because tail looks for trailing lines in its input, but a stream of zeros contains no newline character, so it never gets any lines to discard.
If you're looking for a one-liner in an interpreted language that allocates and populates the memory algorithmically, you're in luck. In just about any general-purpose interpreted language, it's easy to allocate lots of memory and write to it without otherwise using it. Here's a Perl one-liner that seems to be about as fast as tail /dev/zero (though I haven't benchmarked it extensively):
perl -wE 'my @xs; for (1..2**20) { push @xs, q{a} x 2**20 }; say scalar @xs;'
With swap turned off on an old machine with 4 GiB of RAM, both that and tail /dev/zero took about ten seconds each time I ran them. Both should still work fine on newer machines with much more RAM than that. You can make that perl command much shorter, if your goal is brevity.
That Perl one-liner repeatedly generates (q{a} x 2**20) separate moderately long strings--about a million characters each--and keeps them all around by storing them in an array (@xs). You can adjust the numbers for testing. If you don't use all available memory, the one-liner outputs the total number of strings created. Assuming the OOM killer does kill perl--with the exact command shown above and no resource quotas to get in the way, I believe in practice it always will--then your shell should show you Killed. Then, as in any OOM situation, dmesg has the details.
Although I like that method, it does illustrate something useful about writing, compiling, and using a C program--like the one in Doug Smythies's answer. Allocating memory and accessing the memory don't feel like separate things in high-level interpreted languages, but in C you can notice and, if you choose, investigate those details.
Finally, you should always check that the OOM killer is actually what killed your program. One way to check is to inspect dmesg. Contrary to popular belief, it is actually possible for an attempt to allocate memory to fail fast, even on Linux. It's easy to make this happen with huge allocations that will obviously fail... but even those can happen unexpectedly. And seemingly reasonable allocations may fail fast. For example, on my test machine, perl -wE 'say length q{a} x 3_100_000_000;' succeeds, and perl -wE 'say length q{a} x 3_200_000_000;' prints:
Out of memory!
panic: fold_constants JMPENV_PUSH returned 2 at -e line 1.
Neither triggered the OOM killer. Speaking more generally:
- If your program precomputes how much memory is needed and asks for it in a single allocation, the allocation may succeed (and if it does, the OOM killer may or may not kill the program when enough of the memory is used), or the allocation may simply fail.
- Expanding an array to enormous length by adding many, many elements to it often triggers the OOM killer in actual practice, but making it do that reliably in testing is surprisingly tricky. The way this is almost always done--because it is the most efficient way to do it--is to make each new buffer with a capacity x times the capacity of the old buffer. Common values for x include 1.5 and 2 (and the technique is often called "table doubling"). This sometimes bridges the gap between how much memory can actually be allocated and used and how much the kernel knows is too much to even bother pretending to hand out.
- Memory allocations can fail for reasons that have little to do with the kernel or how much memory is actually available, and that doesn't trigger the OOM killer either. In particular, a program may fail fast on an allocation of any size after successfully performing a very large number of tiny allocations. This failure happens in the bookkeeping that is carried out by the program itself--usually through a library facility like
malloc(). I suspect this is what happened to me today when, during testing withbasharrays (which are actually implemented as doubly linked lists),bashquit with an error message saying an allocation of 9 bytes failed.
The OOM killer is much easier to trigger accidentally than to trigger intentionally.
In attempting to deliberately trigger the OOM killer, one way around these problems is to start by requesting too much memory, and go gradually smaller, as Doug Smythies's C program does. Another way is to allocate a whole bunch of moderately sized chunks of memory, which is what the Perl one-liner shown above does: none of the millionish-character strings (plus a bit of additional memory usage behind the scenes) is particularly taxing, but taken together, all the one-megabyte purchases add up.
This answer uses a C program to allocate as much memory as possible, then gradually actually uses it, resulting in "Killed" from the OOM protection.
/*****************************************************************************
*
* bla.c 2019.11.11 Smythies
* attempt to invoke OOM by asking for a rediculous amount of memory
* see: https://askubuntu.com/questions/1188024/how-to-test-oom-killer-from-command-line
* still do it slowly, in chunks, so it can be monitored.
* However simplify the original testm.c, for this example.
*
* testm.cpp 2013.01.06 Smythies
* added a couple more sleeps, in attempts to observe stuff on linux.
*
* testm.cpp 2010.12.14 Smythies
* attempt to compile on Ubuntu Linux.
*
* testm.cpp 2009:03:18 Smythies
* This is not the first edit, but I am just adding the history
* header.
* How much memory can this one program ask for and sucessfully get?
* Done in two calls, to more accurately simulate the program I
* and wondering about.
* This edit is a simple change to print the total.
* the sleep calls have changed (again) for MS C version 2008.
* Now they are more like they used to be (getting annoying).
* Smythies
*****************************************************************************/
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#define CR 13
int main(){
char *fptr;
long i, k;
i = 50000000000L;
do{
if(( fptr = (char *)malloc(i)) == NULL){
i = i - 1000;
}
}
while (( fptr == NULL) && (i > 0));
sleep(15); /* for time to observe */
for(k = 0; k < i; k++){ /* so that the memory really gets allocated and not just reserved */
fptr[k] = (char) (k & 255);
} /* endfor */
sleep(60); /* O.K. now you have 1 minute */
free(fptr); /* clean up, if we get here */
return(0);
}
The result:
doug@s15:~/c$ ./bla
Killed
doug@s15:~/c$ journalctl -xe | grep oom
Nov 11 16:08:24 s15 kernel: mysqld invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Nov 11 16:08:25 s15 kernel: oom_kill_process+0xeb/0x140
Nov 11 16:08:27 s15 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Nov 11 16:08:27 s15 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user/doug/0,task=bla,pid=24349,uid=1000
Nov 11 16:08:27 s15 kernel: Out of memory: Killed process 24349 (bla) total-vm:32638768kB, anon-rss:15430324kB, file-rss:952kB, shmem-rss:0kB, UID:1000 pgtables:61218816kB oom_score_adj:0
Nov 11 16:08:27 s15 kernel: oom_reaper: reaped process 24349 (bla), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
It still takes awhile to run, but on the order of minutes only.
The use of mlock in the C program might help, but I didn't try it.
My test computer is a server, so I use watch -d free -m to monitor progress.
Readers: Messing with OOM is somewhat dangerous. If you read all these answers and comments, you will note some collateral damage and inconsistencies. We can not control when other tasks might ask for a bit more memory, which could well be at just the wrong time. Proceed with caution, and recommend reboot of computer after these type of tests.
This appears to be problem in combination of two factors:
- Using a virtual machine.
- A possible kernel bug.
This is partly one of the lines which describes why this happens:
Mar 7 02:43:11 myhost kernel: memcheck-amd64- invoked oom-killer: gfp_mask=0x24002c2, order=0, oom_score_adj=0
The other line is this:
Mar 7 02:43:11 myhost kernel: 0 pages HighMem/MovableOnly
|The first line is the GFP mask assigned for the allocation. It basically describes what the kernel is allowed/not allowed to do to satify this request.
The mask indicates a bunch of standard flags. The last bit, '2' however indicates the memory allocation should come from the HighMem zone.
If you look closely at the OOM output, you'll see no HighMem/Normal zone actually exists.
Mar 7 02:43:11 myhost kernel: Node 0 DMA: 20*4kB (UM) 17*8kB (UM) 13*16kB (M) 14*32kB (UM) 8*64kB (UM) 4*128kB (M) 4*256kB (M) 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 3944kB
Mar 7 02:43:11 myhost kernel: Node 0 DMA32: 934*4kB (UM) 28*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3960kB
HighMem (generally also called Normal on x86_64) tends to map memory for zones outside of the standard 896MiB ranges directly kernel accessible on 32 bit systems. On x86_64 HighMem/Normal seems to cover all pages above 3GiB in size.
DMA32 contains a zone used for memory that would be accessible on 32-bit DMA devices, that is you can address them with 4 byte pointers. I believe DMA is for 16-bit DMA devices.
Generally speaking, on low memory systems Normal wouldn't exist, given that DMA32 covers all available virtual addresses already.
The reason you OOM kill is because there is a memory allocation for a HighMem zone with 0 pages available. Given the out of memory handler has absolutely no way to satisfy making this zone have pages to use by swapping, killing other processes or any other trick, OOM-killer just kills it.
I believe this is caused by the host VM ballooning on boot up. On KVM systems, there are two values you can set.
- The current memory.
- The available memory.
The way this works is that you can hot-add memory to your server up to the available memory. Your system however is actually given the current memory.
When a KVM VM boots up, it starts with the maximum allotment of memory possible to be given (the available memory). Gradually during the boot phase of the system KVM claws back this memory using its ballooning, leaving you instead with the current memory setting you have.
Its my belief thats what happened here. Linode allow you to expand the memory, giving you much more at system start.
This means that there is a Normal/HighMem zone at the beginning of the systems lifetime. When the hypervisor balloons it away, the normal zone rightly disappears from the memory manager. But, I suspect that the flag setting whether the said zone is available to allocate from is not cleared when it should. This leads the kernel to attempt to allocate from a zone that does not exist.
In terms of resolving this you have two options.
Bring this up on the kernel mailing lists to see if this really is a bug, behaviour expected or nothing at all to do with what I'm saying.
Request that linode set the 'available memory' on the system to be the same 1GiB assignment as the 'current memory'. Thus the system never balloons and never gets a Normal zone at boot, keeping the flag clear. Good luck getting them to do that!
You should be able to test that this is the case by setting up your own VM in KVM setting available to 6GiB, current to 1GiB and running your test using the same kernel to see if this behaviour you see above occurs. If it does, change the 'available' setting to equal the 1GiB current and repeat the test.
I'm making a bunch of educated guesses here and reading inbetween the lines somewhat to come up with this answer, but what I'm saying seems to fit the facts outlined already.
I suggest testing my hypothesis and letting us all know the outcome.
To answer your headline question, use oom_score_adj(kernel >=2.6.36) or for earlier kernels (>=2.6.11) oom_adj, see man proc
/proc/[pid]/oom_score_adj (since Linux 2.6.36) This file can be used to adjust the badness heuristic used to select which process gets killed in out-of-memory conditions...
/proc/[pid]/oom_adj (since Linux 2.6.11) This file can be used to adjust the score used to select which process should be killed in an out-of-memory (OOM) situation...
There's lots more to read but setting oom_score_adj to -1000 or oom_adj to -17 is going to achieve what you want.
The trouble is something else will be killed. Perhaps it would be better to determine why OOM is being invoked and deal with that.