ulimit -t 1 limits the script's CPU time to 1 second. When the script has consumed all its CPU time it gets killed.

To limit the CPU time of just one command in your script you can use parentheses to start it in a subshell with its own limit, e.g.

(ulimit -t 1; LD_PRELOAD=../../EasySandbox/EasySandbox.so ./a.out < $testin) 

Enable core dump in your system, it is the most commonly used method for app crash analysis. I know it is a bit painful to gdb core file, but more or less you can find something out of it. Here is a reference link for you.(http://www.cyberciti.biz/tips/linux-core-dumps.html).

Another way to do this is tracing you script by "strace -p PID-X", note that it will slow down your system, espeically several hours in your case, but it can be the last resort.

Hope above helpful to you.

To improve, use killall, and also combine the commands:

ps -aux | grep script_name
killall script_name inotifywait

Or do everything in one line:

killall `ps -aux | grep script_name | grep -v grep | awk '{ print $1 }'` && killall inotifywait

The exit status of a killed command should be the signal number plus 128. So you can use the exit status to find out which signal killed you process.

I tested it like this on Linux in the shell:

print_exit_status_for_signal () {
  (
    sleep 1000
    echo Exit staus $? = signal ? - 128 ))
  ) &
  sleep 1
  killall "${1:+-$1}" sleep
}
print_exit_status_for_signal
print_exit_status_for_signal 15
print_exit_status_for_signal 9
print_exit_status_for_signal KILL
print_exit_status_for_signal TERM

EDIT: Note that a program can decide to exit with any¹ value (so you have to decide how far you trust the exit status to be the effect of a signal):

for i in $(seq 256); do
  sh -c "exit $i"
  echo Program exited with $?
done

Footnote 1: On my systems exit codes are represented as unsigned 8-bit numbers so they wrap at 256=0.

I would do something like this:

#!/bin/bash
trap : SIGTERM SIGINT

echo $$

find / >/dev/null 2>&1 &
FIND_PID=$!

wait $FIND_PID

if [[ $? -gt 128 ]]
then
    kill $FIND_PID
fi

Some explanation is in order, I guess. Out the gate, we need to change some of the default signal handling. : is a no-op command, since passing an empty string causes the shell to ignore the signal instead of doing something about it (the opposite of what we want to do).

Then, the find command is run in the background (from the script's perspective) and we call the wait builtin for it to finish. Since we gave a real command to trap above, when a signal is handled, wait will exit with a status greater than 128. If the process waited for completes, wait will return the exit status of that process.

Last, if the wait returns that error status, we want to kill the child process. Luckily we saved its PID. The advantage of this approach is that you can log some error message or otherwise identify that a signal caused the script to exit.

As others have mentioned, putting kill -- -$$ as your argument to trap is another option if you don't care about leaving any information around post-exit.

For trap to work the way you want, you do need to pair it up with wait - the bash man page says "If bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes." wait is the way around this hiccup.

You can extend it to more child processes if you want, as well. I didn't really exhaustively test this one out, but it seems to work here.

$ ./test-k.sh &
[1] 12810
12810
$ kill 12810
$ ps -ef | grep find
$

The process your cmd is supposed to be run in will be killed by the SIGHUP signal between the fork() and the exec(), and any nohup wrapper or other stuff will have no chance to run and have any effect. (You can check that with strace)

Instead of nohup, you should set SIGHUP to SIG_IGN (ignore) in the parent shell before executing your background command; if a signal handler is set to "ignore" or "default", that disposition will be inherited through fork() and exec(). Example:

#! /bin/sh
trap '' HUP    # ignore SIGHUP
xclock &
trap - HUP     # back to default

Or:

#! /bin/sh
(trap '' HUP; xclock &)

If you run this script with xfce4-terminal -H -x script.sh, the background command (xclock &) will not be killed by the SIGHUP sent when script.sh terminates.

When a session leader (a process that "owns" the controlling terminal, script.sh in your case) terminates, the kernel will send a SIGHUP to all processes from its foreground process group; but set -m will enable job control and commands started with & will be put in a background process group, and they won't signaled by SIGHUP.

If job control is not enabled (the default for a non-interactive script), commands started with & will be run in the same foreground process group, and the "background" mode will be faked by redirecting their input from /dev/null and letting them ignore SIGINT and SIGQUIT.

Processes started this way from a script which once run as a foreground job but which has already exited won't be signaled with SIGHUP either, since their process group (inherited from their dead parent) is no longer the foreground one on the terminal.

Extra notes:

The "hold mode" seems to be different between xterm and xfce4-terminal (and probably other vte-based terminals). While the former will keep the master side of the pty open, the latter will tear it off after the program run with -e or -x has exited, causing any write to the slave side to fail with EIO. xterm will also ignore WM_DELETE_WINDOW messages (ie it won't close) while there are still processes from the foreground process group running.

Every process has to have a parent process (to collect exit status, at the very least). see man 2 wait. If a parent process dies, its children are sent a SIGHUP signal (see man 7 signal). SIGHUP is a catchable signal, but if the child doesn't catch it, the default action is to kill the child process.

See man nohup to keep things going.

A few things that may cause a shell to exit (not exhaustive):

• calling the exit utility. Let's not forget about the obvious
• calling the return utility. In the case of bash that will return only if in a function or sourced file.
• exec cmd. That will execute cmd in the same process so in effect breaking out of that loop. The script will end when cmd exits.
• set -e/set -o errexit is enabled (see also the SHELLOPTS environment variable for bash) and a command exits with an error.
• set -u/set -o nounset is enabled and an unset variable is referenced.
• a DEBUG or ERR trap is defined that calls exit.
• Failing special builtins. Failure of special builtins (like set, :, eval...) causes the shell to exit. In the case of bash though, that only happens in POSIX mode (like when POSIXLY_CORRECT is in the environment or when invoked as sh...) and even then not for all special builtins. For instance : > / will cause the shell to exit.
• as mentioned by @schily, syntax error (like in code that is only reached conditionally).
• division by 0 (in $((1/x)) or ${array[1/x]}).
• internal bash error for instance because some limit is reached:
  • fails to allocate memory
  • fails to fork a process
  • stack size exceeded (for instance when using function recursion)
  • Some other limits in place via ulimit (which may also cause some signals to be sent).
• killed by a another process. Another process can call kill() to explicitly kill the interpreter of your script.
• killed by the system.
  • SIGINT/SIGQUIT. If you press ^C/^\.
  • SIGHUP. If the terminal is disconnected.
  • SIGSEGV/SIGBUS/SIGILL. The bash command does something wrong (a bug) or failing hardware (memory).
  • SIGPIPE: builtin (echo, printf) writing to a now-closed pipe or socket (could also happen for error messages if stderr is a pipe).

The first thing to check would be the error messages and the exit status.

You haven't written a script in your example; it's a function. You're therefore applying the kill to your interactive shell session. Interactive shells ignore most signals, including the SIGTERM that you're sending.

Put the function and its call in a script and run that. It's far more likely to do what you expect.

Depends on the signal. Bash allows you to trap with the trap command, but if your script is being killed with SIGKILL (9), it is uninterruptible and untrappable.

You need to kill the entire process group:

killall () {
    PID=$$
    kill -9 -$(ps -o pgid= $PID | grep -o '[0-9]*')
}

trap killall EXIT

# ...launch some child processes...

wait

Best way to kill all child processes

A process isn't "killed with SIGHUP" -- at least, not in the strict sense of the word. Rather, when the connection is dropped, the terminal's controlling process (in this case, Bash) is sent a hang-up signal*, which is commonly abbreviated the "HUP signal", or just SIGHUP.

Now, when a process receives a signal, it can handle it any way it wants**. The default for most signals (including HUP) is to exit immediately. However, the program is free to ignore the signal instead, or even to run some kind of signal handler function.

Bash chooses the last option. Its HUP signal handler checks to see if the "huponexit" option is true, and if so, sends SIGHUP to each of its child processes. Only once its finished with that does Bash exit.

Likewise, each child process is free to do whatever it wants when it receives the signal: leave it set to the default (i.e. die immediately), ignore it, or run a signal handler.

Nohup only changes the default action for the child process to "ignore". Once the child process is running, however, it's free change its own response to the signal.

This, I think, is why some programs die even though you ran them with nohup:

1. Nohup sets the default action to "ignore".
2. The program needs to do some kind of cleanup when it exits, so it installs a SIGHUP handler, incidentally overwriting the "ignore" flag.
3. When the SIGHUP arrives, the handler runs, cleaning up the program's data files (or whatever needed to be done) and exits the program.
4. The user doesn't know or care about the handler or cleanup, and just sees that the program exited despite nohup.

This is where "disown" comes in. A process that's been disowned by Bash is never sent the HUP signal, regardless of the huponexit option. So even if the program sets up its own signal handler, the signal is never actually sent, so the handler never runs. Note, however, that if the program tries to display some text to a user that's logged out, it will cause an I/O error, which could cause the program to exit anyway.

* And, yes, before you ask, the "hang-up" terminology is left over from UNIX's dialup mainframe days.

** Most signals, anyway. SIGKILL, for instance, always causes the program to terminate immediately, period.

I was able to test the prctl idea I posted about in a comment and it seems to work. You will need to compile this:

#include "sys/prctl.h"
#include "stdlib.h"
#include "string.h"
#include "unistd.h"

int main(int argc, char ** argv){

  prctl(PR_SET_PDEATHSIG, atoi(argv[1]),0,0,0);

  char * argv0 = strdup(argv[2]);

  char * slashptr = strrchr(argv0, '/');
  if(slashptr){
    argv0 = slashptr + 1;
  }

   return execvp(argv0, &(argv[2]));



}

Let's say you have compiled the above to an executable named "prun" and it is in your path. Let's say your script is called "foo.sh" and it is also in your path. Make a wrapper script that calls

prun 15 foo.sh

foo.sh should get SIGTERM when the wrapper script is terminated for any reason, even SIGKILL.

Note: this is a linux only solution and the c source code presented is without detailed checking of arguments