Message Queues & Event Streaming

Why This Comes Up in Every Design Interview

The moment you say "Service A needs to tell Service B something," the interviewer expects you to reason about:

• Sync vs async trade-offs
• Delivery guarantees
• Ordering requirements
• What happens when consumers are slower than producers
• How to handle poison messages

If you just say "I'll use Kafka" without reasoning through WHY and what delivery guarantees you need, you've lost points.

The Core Decision: Sync vs Async

When to choose async:

• Producer doesn't need immediate confirmation of processing
• Consumer is slower than producer (rate mismatch)
• You need to survive downstream failures
• Fan-out to multiple consumers
• You need replay capability

When to stay sync:

• User expects immediate response (login, payment confirmation)
• Simple request/response (read a user profile)
• Low latency critical (< 10ms)

Two Fundamental Models

Point-to-Point (Queue)

• Message consumed by exactly ONE consumer
• Once consumed, message is gone
• Use case: Job distribution, task processing
• Example: "Process this image," "Send this email"
• Systems: RabbitMQ, SQS, Celery

Publish/Subscribe (Topic)

• Message delivered to ALL subscribers
• Each subscriber gets its own copy
• Use case: Event broadcasting, multiple consumers need same event
• Example: "Order placed" → inventory service + notification service + analytics
• Systems: Kafka, SNS, Google Pub/Sub

Kafka's Consumer Groups: The Hybrid

Kafka combines both: within a consumer group, each partition goes to one consumer (point-to-point). Across groups, all groups get all messages (pub/sub).

Delivery Guarantees — The Most Important Trade-off

The practical answer for interviews:

"I'd use at-least-once delivery with idempotent consumers. Each message has a unique ID. Consumer checks if it's already processed (idempotency key in DB). This gives us effective exactly-once semantics without the complexity and throughput cost of true exactly-once."

How idempotent consumers work:

1. Message arrives with ID msg-abc-123
2. Consumer checks: "Have I processed msg-abc-123?"
3. If yes → ACK, skip. If no → process, record ID, ACK.
4. Safe to retry delivery — duplicate is harmless.

Ordering — When It Matters and When It Doesn't

Kafka's approach: Ordering guaranteed WITHIN a partition. Use partition key to co-locate related messages.

Partition key = user_id
→ All messages for user_123 go to partition 7
→ Processed in order by one consumer
→ But user_456's messages in partition 3 are processed in parallel

Back-of-envelope for partition count:

• Target throughput: 100K messages/sec
• Single consumer processes: 5K messages/sec
• Partitions needed: 100K / 5K = 20 partitions minimum
• With headroom (2x): 40 partitions

Kafka Architecture Deep Dive

Key design decisions:

• Log-based: Messages are APPENDED, never deleted (until retention expires). This enables replay.
• Pull-based consumers: Consumers pull at their own pace (vs push). Natural backpressure.
• Sequential I/O: Appending to log = sequential disk writes = incredibly fast (even faster than random SSD).

Why Kafka is fast (interview gold):

1. Sequential disk I/O (600MB/s vs 100MB/s random)
2. Zero-copy (sendfile syscall — data goes from disk to network without copying to user space)
3. Batch compression (many messages compressed together)
4. Page cache utilization (OS caches frequently read segments)

Back-of-Envelope: Kafka Capacity

Scenario: Design messaging for a social media platform, 500M daily active users, each generates ~20 events/day.

Handling Failures: Dead Letter Queues & Retry

What happens when processing fails?

Poison message problem: One bad message blocks the entire queue (head-of-line blocking). DLQ solves this — move it aside, continue processing.

Backpressure — When Producers Outrun Consumers

System Comparison: When to Use What

Decision framework:

• Need replay? → Kafka or Pulsar
• Simple job queue on AWS? → SQS
• Complex routing (priority, routing keys)? → RabbitMQ
• Multi-region, multi-tenant? → Pulsar
• Don't want to manage infrastructure? → SQS or managed Kafka (MSK, Confluent)

Event Sourcing & CQRS (Advanced Pattern)

Event Sourcing: Store every state change as an immutable event, not just current state.

CQRS (Command Query Responsibility Segregation): Separate write model (events/commands) from read model (materialized views). Write to Kafka → materialize into read-optimized DB.

Used by: Banking systems, audit trails, collaborative editing, gaming (replay systems).

Interview Framework

When asked "How do services communicate in your design?":

Step 1: "For this interaction, I need to decide sync vs async. [Service B] doesn't need to respond in real-time to [Service A], and we need to survive B being down, so I'd use async messaging."

Step 2: "I'd use [Kafka/SQS] with [at-least-once] delivery. Consumers are idempotent using a message ID stored in the processing DB."

Step 3: "For ordering, I'd partition by [entity_id] so all events for one entity are strictly ordered. With [N] partitions, I can parallelize to [N] consumers."

Step 4: "For failures — retry 3x with exponential backoff, then DLQ. We monitor DLQ depth and alert if it grows."

Step 5: "Back-of-envelope: [X] messages/sec, [Y] bytes each, [Z] retention = [W] storage needed."

Quick Recall