Batch vs Stream Processing

Why This Comes Up in Interviews

Nearly every large-scale system generates data that needs processing — analytics, recommendations, fraud detection, ETL. Interviewers want to hear:

• When batch is still the right answer (it often is)
• The Lambda vs Kappa architecture debate
• Exactly-once semantics and why it's hard
• Windowing strategies for stream aggregation
• Concrete tool choices and their trade-offs

Lambda Architecture

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
graph TD
    SOURCE[Data Source] --> BATCH[Batch Layer - Spark]
    SOURCE --> SPEED[Speed Layer - Flink/Kafka Streams]

    BATCH --> BATCHVIEW[(Batch Views - Complete, Accurate)]
    SPEED --> RTVIEW[(Real-time Views - Fast, Approximate)]

    BATCHVIEW --> SERVING[Serving Layer - Merge Results]
    RTVIEW --> SERVING
    SERVING --> QUERY[Query]

    style BATCH fill:#3498db,color:#fff
    style SPEED fill:#e74c3c,color:#fff
    style SERVING fill:#27ae60,color:#fff
    style SOURCE fill:#f39c12,color:#fff

The promise: Best of both worlds — accurate batch results AND real-time approximations.

The reality: You maintain TWO codebases computing the same thing differently. Bugs diverge. Testing doubles. Operational complexity explodes.

Kappa Architecture

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
graph TD
    SOURCE[Data Source] --> LOG[(Immutable Event Log - Kafka)]
    LOG --> STREAM[Stream Processor - Flink]
    STREAM --> VIEW[(Materialized Views)]
    VIEW --> QUERY[Query]

    LOG -->|Replay from offset 0| REPROCESS[Reprocess: Deploy new version, replay all events]
    REPROCESS --> NEWVIEW[(New Views - Replace old)]

    style LOG fill:#f39c12,color:#fff
    style STREAM fill:#3498db,color:#fff
    style VIEW fill:#27ae60,color:#fff
    style REPROCESS fill:#9b59b6,color:#fff

Kappa simplification: One processing path (streaming) handles both real-time AND historical reprocessing by replaying the log.

Processing Paradigm Comparison

Tool Landscape

Exactly-Once Semantics — The Hard Problem

Why exactly-once is hard: Network partitions mean you can never be 100% sure if a message was processed. The receiver might have processed it but the ack was lost.

How Flink achieves exactly-once:

Kafka exactly-once (idempotent producer + transactions):

• Producer ID + sequence number = broker deduplicates retries
• Transactions: atomically write to multiple partitions + commit consumer offsets
• End-to-end: consume → process → produce in a single atomic transaction

Windowing Strategies

Example — "Count clicks per user per 5-minute window":

Watermarks — Handling Late Data

The problem: Events arrive out of order. An event with timestamp 10:01 arrives at 10:05. How do you know when a window is "complete"?

Watermark: A declaration that "no events with timestamp <= W will arrive anymore."

Late data handling (Flink/Beam):

• Drop: Ignore events arriving after watermark (simplest)
• Side output: Route late events to a separate stream for reprocessing
• Allowed lateness: Keep window state open for N minutes past watermark; update results
• Accumulating + retracting: Emit updated result AND retraction of previous result

Back-of-Envelope: Streaming Pipeline Sizing

Scenario: Processing 1 million events/second (e-commerce clickstream)

When to Use What

Interview Framework

When data processing comes up in system design:

Step 1: "First, I'd clarify the latency requirement. If results can be minutes/hours stale, batch (Spark) is simpler, cheaper, and gives exact results. If sub-second freshness is needed, we need stream processing."

Step 2: "For streaming, I'd use Flink with Kafka as the message bus. Flink gives us exactly-once semantics via checkpointing and handles event-time processing with watermarks for out-of-order data."

Step 3: "For windowed aggregations, I'd use [tumbling/sliding/session] windows depending on the use case. I'd set watermarks with 5-minute allowed lateness and route truly late events to a side output for batch correction."

Step 4: "For architecture, I'd start with Kappa (stream-only) for simplicity — one codebase, replay from Kafka for reprocessing. If we later need heavy batch ML training, we might add a batch layer (Lambda) for that specific use case."

Step 5: "Exactly-once guarantee: Kafka idempotent producers + Flink checkpoints + transactional sinks ensure no duplicates or data loss end-to-end."

Quick Recall