Skip to content

Messaging Models

Three fundamental messaging models exist in distributed systems; Kafka uses the log-based model which subsumes both queue and pub-sub patterns.

Key Facts

  • Queue Model (Competing Consumers): messages go to a queue, exactly one consumer receives each message; consumed messages are removed; used for task distribution and load balancing
  • Pub-Sub Model (Fan-Out): messages published to a topic, every subscriber receives a copy; used for event notification where multiple systems need the same data
  • Log-Based Model (Kafka): messages appended to an immutable, ordered log; consumers maintain their own offset (position); messages retained by time/size policy, not by consumption
  • Log-based model enables: multiple independent consumer groups reading at their own pace, replay capability by resetting offset, no message loss on consumer failure
  • Traditional queues (RabbitMQ, ActiveMQ, SQS) focus on message delivery and task distribution; log-based streaming (Kafka, Pulsar, Redpanda) focuses on event storage and stream processing
  • Kafka is NOT a message broker - it is a distributed commit log (write-ahead log)
  • Key difference from traditional queues: reading a topic 10 times requires 10 consumer groups (10 x 8 bytes per partition overhead), not 10 copies of data

Patterns

Property Traditional Queue Log-Based Streaming
Message lifecycle Deleted after consumption Retained by policy
Ordering Per-queue (or none) Per-partition guaranteed
Replay Not possible Possible (reset offset)
Consumer groups Competing consumers Independent consumer groups
Throughput Moderate Very high (sequential I/O)
Use case Task distribution Event sourcing, analytics, CDC

Communication Patterns

Pattern Notation Description
Put/Take 1 -> 1 One-to-one, single consumer
Publish/Subscribe 1 -> * One-to-many, all subscribers receive
Request/Response 1 <-> 1 Synchronous, bidirectional

Protocols

AMQP, MQTT, STOMP, NATS, ZeroMQ, Kafka's custom binary protocol.

Gotchas

  • "Kafka is just another message broker" is false - Kafka's log-based architecture is fundamentally different from traditional queue-based brokers; it always persists messages regardless of consumption
  • Queue model cannot replay - once a message is consumed and acknowledged, it is gone; this is a fundamental architectural difference, not a configuration option
  • Pub-sub without retention means consumer must be online when the message is published; Kafka solves this by retaining messages for a configurable period

See Also