If you want to kill all processes that are named java you can use following command:

killall -9 java

This command sends signals to processes identified by their name.

Maybe its ignoring the signal for some reason. Did you try kill -9?

But please note: kill -9 cannot be ignored or trapped. If a process sees signal 9, it has no choice but to die. It can't do anything else - not even gracefully clean up its files.

Assuming the problem is the OOM killer, then it has killed your process in a desperate attempt to keep the OS functioning in a severe memory shortage crisis.

I would conclude that:

• your JVM is actually using significantly more than 28Gb; i.e. you've got significant non-heap memory usage, and

• the OS is not configured with an adequate amount of swap space.

I'd try adding more swap space, so that the OS can swap out parts of your application in an emergency.

Alternatively, reduce the JVM's heap size.

Note that "-Xmx ..." sets the maximum heap size, not the maximum amount of memory that your JVM can use. The JVM puts some stuff outside the heap, including such things as the memory for thread stacks and memory-mapped files that your application is using.

The syslog confirms that it is the OOM killer at work.

In what way does the linked syslog say so?

It says this:

Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606133] Out of memory: kill process 6368 (run.sh) score 4747288 or a child
Nov 15 13:53:49 ip-10-71-94-36 kernel: [3707038.606146] Killed process 9359 (java)

The console says that java was killed, not that it quit.

Correct. It was killed by the operating system's OOM killer.

If it had run out of memory it would typically throw an OutOfMemory exception, which it didn't.

That is what would have happened if you had filled up the Java heap.

That is not what is going on here. The actual problem is that there is not enough physical RAM to hold the Java heap. The OOM killer deals with it ...

I'm running with such a huge heap because I need to store millions of objects each of which require several kilobytes of RAM.

Unfortunately, you are trying to use way more RAM than is available on the system. This is causing virtual memory to thrash, affecting the entire operating system.

When the system starts to thrash badly, the OOM killer (not the JVM) identifies your Java process as the cause of the problem. It then kills it (with a SIGKILL) to protect the rest of the system. If it didn't, there is a risk that the entire system would lock up completely and need to be hard rebooted.

Finally, you said:

My box has 35.84 GB of RAM ...

That is rather a strange value. 32 GiB is 34,359,738,368 bytes or 34.35 GB.

But based on that and the observed behavior, I suspect that that is the available virtual memory rather than physical RAM. Alternatively, your "box" could be a virtual machine with RAM overcommit enabled at the hypervisor level.

For killing a process that is associated with multiple processes, you need to kill that by using process id associated with that process.

To get the process id of that java process run

ps -A |grep java

output of this command will give the list of java processes running on your system. Note down Process ID (PID) of that process whom you want to kill and run

kill -9 PID

One likely reason is that your server is running out of memory and the process is being killed by the OOM killer. This is a mechanism in Linux that will kill low priority processes when the system is running out of memory.

If this is the case, you should be able to fix it by i) reducing the amount of memory you allow your java process to take, ii) increasing the process' priority (nice -15) or iii) increasing your available memory.

If you start the process from with in your Java application (ex. by calling Runtime.exec() or ProcessBuilder.start()) then you have a valid Process reference to it, and you can invoke the destroy() method in Process class to kill that particular process.

But be aware that if the process that you invoke creates new sub-processes, those may not be terminated (see https://bugs.openjdk.org/browse/JDK-4770092).

On the other hand, if you want to kill external processes (which you did not spawn from your Java app), then one thing you can do is to call O/S utilities which allow you to do that. For example, you can try a Runtime.exec() on kill command under Unix / Linux and check for return values to ensure that the application was killed or not (0 means success, -1 means error). But that of course will make your application platform dependent.

As waltinator suggested in the comment to you question, you can run your process with a shell script. That script can block waiting for your process to terminate, then inspect the exit status to get some idea of the failure mode.

Generally if a process is terminated by a signal, the exist status is 128 plus the signal number. For example, if someone did a kill -9 on the process, then the exit status would be 137 (128 + 9 = 137).

For example, here's a script that runs your command and prints a message if it terminated in response to a signal:

#!/bin/bash

$JAVA_HOME/bin/java \
    -verbose \
    -classpath '../../lib/*:.:./config' com.xxx.xxx.core.xxxxApp instance:$INSTANCE

rc=$? # The $? variable has the exit status of the previous command


if [[ ${rc} -gt 128 ]]; then
    printf "Process terminated with signal %d\n" $((rc - 128))
fi

Now, if I use kill -9 to terminate the java process, then I get:

Process terminated with signal 9

Short answer:

pkill java

This looks up a process (or processes) to kill by name. This will find any other java processes too, so be careful. It also accepts -9, but you should avoid using -9 unless something is really broken.

EDIT:

Based on updates, you may be able to specify the script names to pkill as well (I'm not positive). But, the more traditional way to handle this issue is to leave pid files around. After you start a new process and background it, its pid is available in $!. If you write that pid to a file, then it's easy to check if the process is still running and kill just the processes you mean to. There is some chance that the pid will be reused, however.

You can simply use pkill -f like this:

pkill -f 'java -jar'

EDIT: To kill a particular java process running your specific jar use this regex based pkill command:

pkill -f 'java.*lnwskInterface'

Zombie (defunct) processes are dead. All processes on unix hang around after they die until their parent process requests their return status. Most of the time, that happens instantly so you never see that the process is a zombie. But if the parent process is hung or stopped, it can't release its dead child processes so they hang around as zombies.

Using taskkill, you can kill a process based on window title using a filter.

taskkill /F /FI "WindowTitle eq Spotify" /T

• /F - force task kill
• /T - Kill child process
• /FI - Filter the tasks

If the window title has quotes in it, you can escape the nested quotes with a backslash (\).

You can use tasklist in a similar manner to search for a task based on its window title.

tasklist /V /FI "WindowTitle eq Spotify"

You can use the * as a wildcard to match a pattern

tasklist /V /FI "WindowTitle eq S*"