ulimit -t 1 limits the script's CPU time to 1 second. When the script has consumed all its CPU time it gets killed.

To limit the CPU time of just one command in your script you can use parentheses to start it in a subshell with its own limit, e.g.

(ulimit -t 1; LD_PRELOAD=../../EasySandbox/EasySandbox.so ./a.out < $testin) 

Often times when applications are being killed, it is always a good idea to take a quick look at your /var/log/messages file to see if the kernel is killing the process. The most common trigger (in my experience) has always been due to out-of-memory (OOM) errors, since my company primarily uses java applications, it is quite common for the devs to publish a bad code update that triggers an OOM event.

Scheduling tasks when your OS has the most available resources is probably why it is succeeding in the AM time slots and not in the PM when most people like to schedule taxing system jobs. Simple solutions to this are to either increase your systems resources, restrict resources allocated to your code or move around when your jobs are scheduled so they don't conflict.

I would do something like this:

#!/bin/bash
trap : SIGTERM SIGINT

echo $$

find / >/dev/null 2>&1 &
FIND_PID=$!

wait $FIND_PID

if [[ $? -gt 128 ]]
then
    kill $FIND_PID
fi

Some explanation is in order, I guess. Out the gate, we need to change some of the default signal handling. : is a no-op command, since passing an empty string causes the shell to ignore the signal instead of doing something about it (the opposite of what we want to do).

Then, the find command is run in the background (from the script's perspective) and we call the wait builtin for it to finish. Since we gave a real command to trap above, when a signal is handled, wait will exit with a status greater than 128. If the process waited for completes, wait will return the exit status of that process.

Last, if the wait returns that error status, we want to kill the child process. Luckily we saved its PID. The advantage of this approach is that you can log some error message or otherwise identify that a signal caused the script to exit.

As others have mentioned, putting kill -- -$$ as your argument to trap is another option if you don't care about leaving any information around post-exit.

For trap to work the way you want, you do need to pair it up with wait - the bash man page says "If bash is waiting for a command to complete and receives a signal for which a trap has been set, the trap will not be executed until the command completes." wait is the way around this hiccup.

You can extend it to more child processes if you want, as well. I didn't really exhaustively test this one out, but it seems to work here.

$ ./test-k.sh &
[1] 12810
12810
$ kill 12810
$ ps -ef | grep find
$

In order to silence the message, you must be redirecting stderr at the time the message is generated. Because the kill command sends a signal and doesn't wait for the target process to respond, redirecting stderr of the kill command does you no good. The bash builtin wait was made specifically for this purpose.

Here is very simple example that kills the most recent background command. (Learn more about $! here.)

kill $!
wait $! 2>/dev/null

Because both kill and wait accept multiple pids, you can also do batch kills. Here is an example that kills all background processes (of the current process/script of course).

kill $(jobs -rp)
wait $(jobs -rp) 2>/dev/null

I was led here from bash: silently kill background function process.

Every process has to have a parent process (to collect exit status, at the very least). see man 2 wait. If a parent process dies, its children are sent a SIGHUP signal (see man 7 signal). SIGHUP is a catchable signal, but if the child doesn't catch it, the default action is to kill the child process.

See man nohup to keep things going.

Enable core dump in your system, it is the most commonly used method for app crash analysis. I know it is a bit painful to gdb core file, but more or less you can find something out of it. Here is a reference link for you.(http://www.cyberciti.biz/tips/linux-core-dumps.html).

Another way to do this is tracing you script by "strace -p PID-X", note that it will slow down your system, espeically several hours in your case, but it can be the last resort.

Hope above helpful to you.

There really are only a couple of possibilities. Assuming you're just running this from the command line, you should see the message ... unless, of course, what you're doing puts the PID of your shell process in PIDS, in which case the kill would kill the (sub) shell running your command before you hit the echo.

Suggestion: echo $PIDS before you call kill and see what's there. In fact, I'd be tempted to comment out the kill and try the command, just to see what happens.

#!/bin/bash
PIDS=$(ps -e | grep $1 |grep -v grep| awk '{print $1}')
echo $PIDS
# kill -s SIGINT $PIDS
echo "Done sendings signal"

Of course, you can always run the script with bash -x to see everything.

The exit status of a killed command should be the signal number plus 128. So you can use the exit status to find out which signal killed you process.

I tested it like this on Linux in the shell:

print_exit_status_for_signal () {
  (
    sleep 1000
    echo Exit staus $? = signal $(( $? - 128 ))
  ) &
  sleep 1
  killall "${1:+-$1}" sleep
}
print_exit_status_for_signal
print_exit_status_for_signal 15
print_exit_status_for_signal 9
print_exit_status_for_signal KILL
print_exit_status_for_signal TERM

EDIT: Note that a program can decide to exit with any¹ value (so you have to decide how far you trust the exit status to be the effect of a signal):

for i in $(seq 256); do
  sh -c "exit $i"
  echo Program exited with $?
done

Footnote 1: On my systems exit codes are represented as unsigned 8-bit numbers so they wrap at 256=0.

With standard sh and bash, you can

set -e

It will

$ help set
...
        -e  Exit immediately if a command exits with a non-zero status.

It also works (from what I could gather) with zsh. It also should work for any Bourne shell descendant.

With csh/tcsh, you have to launch your script with #!/bin/csh -e

Your script is run on another shell and uses the -f option (follow) of tail which appends the output as the file grows. In short, the script has no exit point, so it's not supposed to end, even if you desire the script to do so.

Does the name of your script include MailSender? Try changing it, if so. Also, use pkill -9, it'll be cleaner:

#!/bin/bash

pkill -9 MailSender
echo starting
./MailSender

/bin/bash: line 2: 16577 Killed Means that the process with the PID 16577 got killed. A bash script ran this process, and the OS Killed it independently for some reason.

“Killed” indicates a SIGKILL, which can be generated by the kernel. One possible reason is if the process tries to go over a resource limit, or possibly if the system runs out of memory. It could also be something that UserLand generates for whatever reason, perhaps because the process is trying to use a system feature that UserLand doesn't provide. When attempting to compile GCC with only 2GB of RAM, the OOM killer (triggered by running out of RAM) is plausible.

Some of the possible reasons are listed by Gilles 'SO - stop being evil'

Use a negative PID which will cause a process group to be killed. The -- tells kill that the rest of the arguments are not option switches so the hyphen (minus) before the PID won't confuse it.

kill -- -12345

Also, -9 is a last resort. Don't use it until you've tried at least -15 (SIGTERM, which is the default) first. This gives a program the chance to do housekeeping before it exits. See When should I use kill -9 or Useless use of kill -9 or kill -9.

If you expect the signal to be delivered only to the running program and not to the shell running your script, then the basic synopsis might be:

#!/bin/bash
set -euo pipefail

sleep 1 &  # The original program
pid="$!"
kill -9 "$pid"  # Pick your lethal signal
wait -n "$pid" && status=0 || status="$?"
((status > 128)) && echo "${pid} got signal $((status - 128))" 1>&2 || :

Presumably, here^^^ we run the program in the background, so that we can send it the kill signal from the same snippet. In practice you would probably run it in the foreground and then check its $? return status instead of the status from wait -n.

If the killing signal is delivered to your entire process group, including the shell running your script, that is a different story. For the signal KILL (9) in particular, there is no way to mask it or report it. When the shell gets it, it dies. For other signals you could set up a trap command (see man bash for its syntax) to handle the signal gracefully in the script while still being able to detect and report the child process’ death from the signal.

Processes can call the _exit() system call (on Linux, see also exit_group()) with an integer argument to report an exit code to their parent. Though it's an integer, only the 8 least significant bits are available to the parent (exception to that is when using waitid() or handler on SIGCHLD in the parent to retrieve that code, though not on Linux).

The parent will typically do a wait() or waitpid() to get the status of their child as an integer (though waitid() with somewhat different semantics can be used as well).

On Linux and most Unices, if the process terminated normally, bits 8 to 15 of that status number will contain the exit code as passed to exit(). If not, then the 7 least significant bits (0 to 6) will contain the signal number and bit 7 will be set if a core was dumped.

perl's $? for instance contains that number as set by waitpid():

$ perl -e 'system q(kill $$); printf "%04x\n", $?'
000f # killed by signal 15
$ perl -e 'system q(kill -ILL $$); printf "%04x\n", $?'
0084 # killed by signal 4 and core dumped
$ perl -e 'system q(exit $((0xabc))); printf "%04x\n", $?'
bc00 # terminated normally, 0xbc the lowest 8 bits of the status

Bourne-like shells also make the exit status of the last run command in their own $? special parameter. However, it does not contain directly the number returned by waitpid(), but a transformation on it, and it's different between shells.

What's common between all shells is that $? contains the lowest 8 bits of the exit code (the number passed to exit()) if the process terminated normally.

Where it differs is when the process is terminated by a signal. In all cases, and that's required by POSIX, the number will be greater than 128. POSIX doesn't specify what the value may be. In practice though, in all Bourne-like shells that I know, the lowest 7 bits of $? will contain the signal number. But, where n is the signal number,

• in ash, zsh, pdksh, bash, the Bourne shell, $? is 128 + n. What that means is that in those shells, if you get a $? of 129, you don't know whether it's because the process exited with exit(129) or whether it was killed by the signal 1 (HUP on most systems). But the rationale is that shells, when they do exit themselves, by default return the exit status of the last exited command. By making sure $? is never greater than 255, that allows to have a consistent exit status:

  $ bash -c 'sh -c "kill \$\$"; printf "%x\n" "$?"'
  bash: line 1: 16720 Terminated              sh -c "kill \$\$"
  8f # 128 + 15
  $ bash -c 'sh -c "kill \$\$"; exit'; printf '%x\n' "$?"
  bash: line 1: 16726 Terminated              sh -c "kill \$\$"
  8f # here that 0x8f is from a exit(143) done by bash. Though it's
     # not from a killed process, that does tell us that probably
     # something was killed by a SIGTERM
  

• ksh93, $? is 256 + n. That means that from a value of $? you can differentiate between a killed and non-killed process. Newer versions of ksh, upon exit, if $? was greater than 255, kills itself with the same signal in order to be able to report the same exit status to its parent. While that sounds like a good idea, that means that ksh will generate an extra core dump (potentially overwriting the other one) if the process was killed by a core generating signal:

  $ ksh -c 'sh -c "kill \$\$"; printf "%x\n" "$?"'
  ksh: 16828: Terminated
  10f # 256 + 15
  $ ksh -c 'sh -c "kill -ILL \$\$"; exit'; printf '%x\n' "$?"
  ksh: 16816: Illegal instruction(coredump)
  Illegal instruction(coredump)
  104 # 256 + 15, ksh did indeed kill itself so as to report the same
      # exit status as sh. Older versions of `ksh93` would have returned
      # 4 instead.
  

  Where you could even say there's a bug is that ksh93 kills itself even if $? comes from a return 257 done by a function:

  $ ksh -c 'f() { return "$1"; }; f 257; exit'
  zsh: hangup     ksh -c 'f() { return "$1"; }; f 257; exit'
  # ksh kills itself with a SIGHUP so as to report a 257 exit status
  # to its parent
  

• yash. yash offers a compromise. It returns 256 + 128 + n. That means we can also differentiate between a killed process and one that terminated properly. And upon exiting, it will report 128 + n without having to suicide itself and the side effects it can have.

  $ yash -c 'sh -c "kill \$\$"; printf "%x\n" "$?"'
  18f # 256 + 128 + 15
  $ yash -c 'sh -c "kill \$\$"; exit'; printf '%x\n' "$?"
  8f  # that's from a exit(143), yash was not killed
  

To get the signal from the value of $?, the portable way is to use kill -l:

$ /bin/kill 0
Terminated
$ kill -l "$?"
TERM

(for portability, you should never use signal numbers, only signal names)

On the non-Bourne fronts:

• csh/tcsh and fish same as the Bourne shell except that the status is in $status instead of $? (note that zsh also sets $status for compatibility with csh (in addition to $?)).

• rc: the exit status is in $status as well, but when killed by a signal, that variable contains the name of the signal (like sigterm or sigill+core if a core was generated) instead of a number, which is yet another proof of the good design of that shell.

• es. the exit status is not a variable. If you care for it, you run the command as:

   status = <={cmd}
  

which will return a number or sigterm or sigsegv+core like in rc.

Maybe for completeness, we should mention zsh's $pipestatus and bash's $PIPESTATUS arrays that contain the exit status of the components of the last pipeline.

And also for completeness, when it comes to shell functions and sourced files, by default functions return with the exit status of the last command run, but can also set a return status explicitly with the return builtin. And we see some differences here:

• bash and mksh (since R41, a regression^Wchange apparently introduced intentionally) will truncate the number (positive or negative) to 8 bits. So for instance return 1234 will set $? to 210, return -- -1 will set $? to 255.
• zsh and pdksh (and derivatives other than mksh) allow any signed 32 bit decimal integer (-2³¹ to 2³¹-1) (and truncate the number to 32bits).
• ash and yash allow any positive integer from 0 to 2³¹-1 and return an error for any number out of that.
• ksh93 for return 0 to return 320 set $? as is, but for anything else, truncate to 8 bits. Beware as already mentioned that returning a number between 256 and 320 could cause ksh to kill itself upon exit.
• rc and es allow returning anything even lists.

Also note that some shells also use special values of $?/$status to report some error conditions that are not the exit status of a process, like 127 or 126 for command not found or not executable (or syntax error in a sourced file)...