The use-cases for Kafka and Amazon SQS/Amazon SNS are quite different.
Kafka, as you wrote, is a distributed publish-subscribe system. It is designed for very high throughput, processing thousands of messages per second. Of course you need to setup and cluster it for yourself. It supports multiple readers, which may "catch up" with the stream of messages at any point (well, as long as the messages are still on disk). You can use it both as a queue (using consumer groups) and as a topic.
An important characteristic is that you cannot selectively acknowledge messages as "processed"; the only option is acknowledging all messages up to a certain offset.
SQS/SNS on the other hand:
- no setup/no maintenance
- either a queue (SQS) or a topic (SNS)
- various limitations (on size, how long a message lives, etc)
- limited throughput: you can do batch and concurrent requests, but still achieving high throughputs would be expensive
- Regarding duplication of messages: SQS Standard queue guarantees at least-once delivery, however you can avoid retrieval of duplicate messages by setting visibility timeout on the queue. For SQS FIFO queue, message to be received is exactly-once and if you want to control sending same message again, you can do it using the deduplication id and the default time is 5 minutes before sending another one. SNS is designed to send same message across multiple consumers and it is similar to SQS Standard in case of replication of messages.
- SNS and SQS have the option to send failed messages to dead letter queues out of the box and you can redrive the messages out of the queue when ready.
- SNS has notifications for email, SMS, SQS, HTTP built-in. With Kafka, you would probably have to code it yourself
- no "message stream" concept
So overall I would say SQS/SNS are well suited for simpler tasks and workloads with a lower volume of messages.
Answer from adamw on Stack OverflowI am a big advocate of Kafka so it is a bit of a surprise when a company that I recently joined has a retrofitted solution of using SNS+SQS in their stack to achieve what Kafka offers out of the box. In your opinion what is the advantage of using that? AFAIK Kafka provides everything that the company needs and it is not that hard to setup these days with confluent cloud or MSK.
Videos
The use-cases for Kafka and Amazon SQS/Amazon SNS are quite different.
Kafka, as you wrote, is a distributed publish-subscribe system. It is designed for very high throughput, processing thousands of messages per second. Of course you need to setup and cluster it for yourself. It supports multiple readers, which may "catch up" with the stream of messages at any point (well, as long as the messages are still on disk). You can use it both as a queue (using consumer groups) and as a topic.
An important characteristic is that you cannot selectively acknowledge messages as "processed"; the only option is acknowledging all messages up to a certain offset.
SQS/SNS on the other hand:
- no setup/no maintenance
- either a queue (SQS) or a topic (SNS)
- various limitations (on size, how long a message lives, etc)
- limited throughput: you can do batch and concurrent requests, but still achieving high throughputs would be expensive
- Regarding duplication of messages: SQS Standard queue guarantees at least-once delivery, however you can avoid retrieval of duplicate messages by setting visibility timeout on the queue. For SQS FIFO queue, message to be received is exactly-once and if you want to control sending same message again, you can do it using the deduplication id and the default time is 5 minutes before sending another one. SNS is designed to send same message across multiple consumers and it is similar to SQS Standard in case of replication of messages.
- SNS and SQS have the option to send failed messages to dead letter queues out of the box and you can redrive the messages out of the queue when ready.
- SNS has notifications for email, SMS, SQS, HTTP built-in. With Kafka, you would probably have to code it yourself
- no "message stream" concept
So overall I would say SQS/SNS are well suited for simpler tasks and workloads with a lower volume of messages.
This is a classic trade-off:
AWS tools (SQS, SNS)
These will be easier for you to setup, and integrate with the rest of your architecture, especially if most of it is already running on AWS. It will also probably be cheaper at first, since they have a good pay as you go model, but the cost will not scale as well, so you have to think about that.
Apache Kafka
Here, you're using a highly popular (not trendy) distributed (this is important if you think you will scale a lot) PUB/SUB model. Nowadays, this model seems to be much preferred, since running analytics on the data going through the pipes is very common, and usually with an SOA architecture you can have a multitude of small services consuming the messages and doing their thing, without having the data be removed from the queue. You also get a lot of configuration options, so depending on your use case you can fine tune it to your needs. This means more work, but a more optimized service down the road.
Summary
This is a classic trade-off of speed of development and ease of development vs the best, very modular and personalized solution, that has more overhead for the first implementation but scales better.
Personal Advice
If you are prototyping something, favor speed of development, so AWS tools. If your requirements are frozen and require significant scale, definitely take the time to use kafka. I also am a big believer in using-open-source-makes-the-world-better, but that's not the biggest argument to use.