Kafka – As Messaging System

Kafka is designed for distributed high throughput systems. Kafka tends to work very well as a replacement for a more traditional message broker. In comparison to other messaging systems, Kafka has better throughput, built-in partitioning, replication and inherent fault-tolerance, which makes it a good fit for large-scale message processing applications.

What is a Messaging System?

A messaging system is a software solution that enables communication between distributed systems or applications by exchanging messages. These messages can contain data, commands, or events. Messaging systems are widely used in distributed systems, microservices architectures, and real-time data processing. They help decouple producers (senders) and consumers (receivers), allowing them to operate independently.

Types of Messaging Systems

1. Point-to-Point Messaging System
2. Publish-Subscribe Messaging System

---

1. Point-to-Point Messaging System

In a Point-to-Point (P2P) messaging system:

• A message is sent by a producer (sender) to a specific queue.
• A consumer (receiver) reads the message from the queue.
• Each message is consumed by only one consumer.
• Once a message is processed, it is removed from the queue.
• Ideal for task-based systems where each message represents a unit of work, and only one system needs to process it.

Example:

• A producer sends job tasks (like image processing) to a queue.
• Workers (consumers) fetch and process tasks from the queue.
• The typical example of this system is an Order Processing System, where each order will be processed by one Order Processor

---

2. Publish-Subscribe Messaging System

In a Publish-Subscribe (Pub-Sub) messaging system:

• Messages are published by a producer to a topic.
• Multiple consumers can subscribe to the same topic.
• Each subscribed consumer gets a copy of every message.
• Messages are not removed; instead, they are retained for a defined period or until all consumers acknowledge them.
• Ideal for broadcast systems where multiple systems need to react to the same event.

Example:

• A news service publishes breaking news to a topic.
• Subscribers like mobile apps, websites, and alerts systems receive the news updates.
• A real-life example is Dish TV, which publishes different channels like sports, movies, music, etc., and anyone can subscribe to their own set of channels and get them whenever their subscribed channels are available.

Apache Kafka is a distributed publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another. Kafka is suitable for both offline and online message consumption. Kafka messages are persisted on the disk and replicated within the cluster to prevent data loss. Kafka is built on top of the ZooKeeper synchronization service. It integrates very well with Apache Storm and Spark for real-time streaming data analysis.