The main motive behind kafka is scaling.
MQTT is a protocol with a publicly available specification for easy communication between the client and the message, allowing you to publish / subscribe to exchanges. Many implementations of client libraries and brokers (Mosquitto, JoramMQ ...) exist and are almost compatible. MQTT simply indicates the transport and the indefinitely part of the application (that is, how data is processed and, possibly, stored, how clients are authorized ...) The specification is not clear if the data consumed by the topic is only in real time or possibly stored. The specification does not indicate anything about how a broker implementing MQTT can / should be scaled.
Apache Kafka, on the other hand, is a message broker based on an internal βcommit logβ: its main focus is on storing a huge amount of data on disk and allowing consumption in real time or later (while the data is still available on disk). It is designed to be deployed as a cluster of several nodes with good scalability properties. Kafka uses its own network protocol.
So, you compare two different things here: the standard pub / sub protocol (with several implementations) and special software for storing / distributing messages, vaguley of the same family with its own protocol.
I would say that if you need to store a huge number of messages to provide batch processing, look more at Kafka. If you have many clients / applications that exchange real-time messages on many topics with the extension, look more at the implementation of MQTT (or even AMQP) brokers.
source share