watch -n0.1 --no-title cat /proc/interrupts

I'd recommend trying to search for an answer before asking a question. This is shamelessly copy/pasted from http://www.linuxjournal.com/content/watch-live-interrupts.

To see the interrupts occurring on your system, run the command:

watch -n1 "cat /proc/interrupts"

The watch command executes another command periodically, in this case "cat /proc/interrups". The -n1 option tells watch to execute the command every second.

Try using -d for fancy output with highlights.

Man page link for the watch command: http://linux.die.net/man/1/watch

Introduction to Linux Interrupts (describes what /proc/interrupts is all about):http://www.thegeekstuff.com/2014/01/linux-interrupts/

• The first Column is the IRQ number.
• The Second column says how many times the CPU core has been interrupted.
• For interrupt like rtc [Real time clock] CPU has not being interrupted. RTC are present in electronic devices to keep track of time. NMI and LOC are drivers used on system that are not accessible/configured by user.
• IRQ number determines the priority of the interrupt that needs to be handled by the CPU.

A small IRQ number value means higher priority.

For example if CPU receives interrupt from Keyboard and system clock simultaneously. CPU will serve System Clock first since it has IRQ number 0.

IRQ 0 — system timer (cannot be changed);

IRQ 1 — keyboard controller (cannot be changed)

IRQ 3 — serial port controller for serial port 2 (shared with serial port 4, if present);

IRQ 4 — serial port controller for serial port 1 (shared with serial port 3, if present);

IRQ 5 — parallel port 2 and 3 or sound card;

IRQ 6 — floppy disk controller;

IRQ 7 — parallel port 1. It is used for printers or for any parallel port if a printer is not present.

For Windows

Original Question: How can I find out what is causing interrupts on Windows?

There are a couple of answers there you may benefit from. Like Windows Process Explorer which shows how much processor time is spent serving interrupts, Windows Performance Analyzer (WPA), the xperf command, and The DPC/ISR Action

The keyboard handler itself is often a good interface to determine if the system is live. In the default mode, the tty line discipline will echo characters typed at the keyboard (this echoing is done by the kernel and isn't dependent on any user-space process getting scheduled). So if you're sitting at an idle command line on the system console, and you're not seeing the characters you typed echo onto the screen, there's a good chance your system is locked up. Note, however, that the mode can be changed by a program (such as vi/vim) and that can turn this echoing off (in that case you only see what the vim process prints to the screen).

A second test is to access the machine from the network. There are several things that should cause responses when sent from another machine. If you send a ping (an ICMP Echo Request), the kernel should automatically respond with an ICMP Echo Reply packet. Or if you know there's a server running on port 5000 (say), you should be able to create a connection to that port even if the server process itself never gets a chance to run (the TCP SYN-ACK sent in response to the SYN is done completely within the kernel). If these work, you know network interrupts are being processed, and the kernel's network soft-interrupt handler is being scheduled.

There are other things that would show you whether the keyboard interrupt is triggering and being handled, but those would generally require you to have a working command line link to the machine.

If you have such a link, cat /proc/interrupts will show you counts for each interrupt for each CPU. If the keyboard is interrupting, for example, one of those counts should increase on each interrupt.

You should never have a case where an interrupt is occurring but is not being serviced. All interrupts will be serviced before user-space processing is resumed. (Exactly what it means to service the interrupt is up to the driver and may vary by device / device type.)

If no interrupts are being serviced at all, your system will be completely locked up and unusable as no input or output at all will generally occur without interrupts.

In general, you canʼt find "table of all interrupts" without a real hardware start because it depends on ton of factors, including extension adapter set, exact chipset version, processor version, and so on.

Iʼd assume x86 as the context. It is defined by Intel that first 32 interrupt vectors (0-31) are for use by CPU itself - it can generate their invocation on internally defined exceptions. That would clash with old style (known from various IBM PC descriptions) that interrupts are assigned to 8-15, but, it is defined as OS task to reassign all conflicting interrupts when entering the protected mode. Then, interrupt controllers (nowadays, you can assume all them are at least APIC) are programmed to assign interrupt numbers of remained set to hardware that requires them. What numbers are assignable, depends on bus type and delivery manner:

• MSI (message signaled interrupt), MSI-X - the main techniques for PCI-E - are assigned by APIC programming, typically one number per device and role (some devices will emit multiple interrupt types);
• old line-based style (classic PCI) - up to 4 interrupt lines per bus; so there may be collision between numbers, and handlers shall iterate all possible devices. In classic designs of Pentium 1-3 times, they were assigned by BIOS to range 10-14 and then moved by OS to some upper range.

At the system I write this, interrupt numbers assigned to hardware are 36-62 with some gaps. 17 of them are used by xhci_hcd.

To sum up: for CPU interrupts, read the CPU doc. For others, assume dynamic assignment and find the current assignment in OS state using respective API.

I assume you don't have single socket system with 24C CPU. So it's probably NUMA system with 2x12C. In that case I'd suggest to make sure the program uses only one numa node (usually socket) and it's local half of RAM.

When you have 50G used, that means, numa locality can't be assured as it's more than half of the memory.

For checking of actual state, use numastat. If you're on the RHEL, you may use numad to handle memory locality automatically. Or you may use numactl --hardware will give you overview about your HW NUMA nodes. There is quite nice howto with examples:

http://fibrevillage.com/sysadmin/534-numactl-installation-and-examples

That way you may lock your program on desired CPUs.

And I'd suggest to check if you have irqbalance daemon running, otherwise you may have one core overloaded with interrupts.