The issue (-EINVAL) isn't being caused by the handler, as there's no way for the kernel to determine its signature at runtime.

Also, handler is never called while requesting an IRQ via request_irq() (unless CONFIG_DEBUG_SHIRQ_FIXME is defined, and [with the wrong signature] that would only lead to undefined behaviour AFTER parameter validation, not before, so it would be very unlikely to return an -EINVAL at this point).

There are 4 key points that are validated before an IRQ is properly setup:

• irqflags - You've none set, that passes the check.
• irq_to_desc() - Will perform a lookup over irq_desc_tree for irq. Will return -EINVAL if not found. Otherwise, returns a pointer to an irq descriptor structure (struct irq_desc *)
• irq_settings_can_request() - Check if the IRQ can be requested. You can easily rule out this by calling int can_request_irq(unsigned int irq, unsigned long irqflags) before request_irq().
• handler - Will check if handler is NULL. In your case, it is not.

So basically you've only two possible checks that may be causing the issue, and one of them you can rule out using can_request_irq() before calling request_irq(), but, unfortunately, this won't be possible to do under a module, because can_request_irq symbol isn't exported.

But if you've the patience and the time, you can build a new image with that code built-in, just see if can_request_irq() check passes. If it does, the only check that is causing the issue is irq_to_desc(), meaning that irq is invalid.

Can't extend this answer much more, as there's no more information I can work with from your question, but I hope it helps you to get in the right track.

By the way, just to point out the typedef of irq_handler_t in the case it may be useful:

include/linux/interrupt.h:92 (on 4.5 tree):

typedef irqreturn_t (*irq_handler_t)(int, void *);

You should define your handler as following and pass handler to request_irq().

irqreturn_t handler(int irq, void *data){ }

The second error of yours is because you are saying that handler receives 3 parameters while it's supposed to receive only two.

The first error is because that the definitions are wrong you say handler is returning irq_handler_t while it's supposed to return irqreturn_t which is a typedef for an enum. while irq_handler_t is typedef for a function pointer.

This is covered in chapter 10 of Linux Device Drivers, 3rd edition, by Corbet et al. It is available for free online, or you may toss some shekels O'Reilly's way for dead tree or ebook forms. The part relevant to your question begins on page 278 in the first link.

For what it's worth, here is my attempt to paraphrase those three pages, plus other bits I've Googled up:

• When you register a shared IRQ handler, the kernel checks that either:

  a. no other handler exists for that interrupt, or

  b. all of those previously registered also requested interrupt sharing

If either case applies, it then checks that your dev_id parameter is unique, so that the kernel can differentiate the multiple handlers, e.g. during handler removal.

• When a PCI¹ hardware device raises the IRQ line, the kernel's low-level interrupt handler is called, and it in turn calls all of the registered interrupt handlers, passing each back the dev_id you used to register the handler via request_irq().

The dev_id value needs to be machine-unique. The common way to do that is to pass a pointer to the per-device struct your driver uses to manage that device. Since this pointer must be within your driver's memory space for it to be useful to the driver, it is ipso facto unique to that driver.²

If there are multiple drivers registered for a given interrupt, they will all be called when any of the devices raises that shared interrupt line. If it wasn't your driver's device that did this, your driver's interrupt handler needs to detect this and return IRQ_NONE promptly.

Another case is that your driver is managing multiple devices. The driver's interrupt handler should use dev_id to decide which device to poll.

The example Corbet et al. give is that of a PC parallel port. When it asserts the interrupt line, it also sets the top bit in its first device register. (That is, inb(0x378) & 0x80 == true, assuming standard I/O port numbering.) When your handler detects this, it is supposed to do its work, then clear the IRQ by writing the value read from the I/O port back to the port with the top bit cleared.

I don't see any reason that particular mechanism is special. A different hardware device could choose a different mechanism. The only important thing is that for a device to allow shared interrupts, it has to have some way for the driver to read the interrupt status of the device, and some way to clear the interrupt. You'll have to read your device's datasheet or programming manual to find out what mechanism your particular device uses.

• When your interrupt handler tells the kernel it handled the interrupt, that doesn't stop the kernel from continuing to call any other handlers registered for that same interrupt. This is unavoidable if you are to share an interrupt line when using level-triggered interrupts.

Imagine two devices assert the same interrupt line at the same time. (Or at least, so close in time that the kernel doesn't have time to call an interrupt handler to clear the line and thereby see the second assertion as separate.) The kernel must call all handlers for that interrupt line, to give each a chance to query its associated hardware to see if it needs attention. It is quite possible for two different drivers to successfully handle an interrupt within the same pass through the handler list for a given interrupt.

Because of this, it is imperative that your driver tell the device it is managing to clear its interrupt assertion sometime before the interrupt handler returns. It's not clear to me what happens otherwise. The continuously-asserted interrupt line will either result in the kernel continuously calling the shared interrupt handlers, or it will mask the kernel's ability to see new interrupts so the handlers are never called. Either way, disaster.

Footnotes:

1. I specified PCI above because all of the above assumes level-triggered interrupts, as used in the original PCI spec. ISA used edge-triggered interrupts, which made sharing tricky at best, and possible even then only when supported by the hardware. PCIe uses message-signalled interrupts; the interrupt message contains a unique value the kernel can use to avoid the round-robin guessing game required with PCI interrupt sharing. PCIe may eliminate the very need for interrupt sharing. (I don't know if it actually does, just that it has the potential to.)

2. Linux kernel drivers all share the same memory space, but an unrelated driver isn't supposed to be mucking around in another's memory space. Unless you pass that pointer around, you can be pretty sure another driver isn't going to come up with that same value accidentally on its own.