Production Operations & Reliability

What separates "it works on my machine" from "it handles 10K requests/second at 3am on Black Friday" — the operations knowledge that distinguishes senior engineers.

Kubernetes Deployment Strategies

Strategy Comparison

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    subgraph Rolling["Rolling Update"]
        R1["v1 ✅"] & R2["v1 ✅"] & R3["v2 🔄"] & R4["v2 🔄"]
    end

    subgraph BlueGreen["Blue-Green"]
        B1["v1 (Blue) ✅"] -.->|"switch traffic"| B2["v2 (Green) ✅"]
    end

    subgraph Canary["Canary"]
        C1["v1 (90% traffic)"] & C2["v2 (10% traffic) 🐤"]
    end

    style R3 fill:#FEF3C7,stroke:#D97706,color:#000
    style R4 fill:#FEF3C7,stroke:#D97706,color:#000
    style B2 fill:#E8F5E9,stroke:#2E7D32,color:#000
    style C2 fill:#E3F2FD,stroke:#1565C0,color:#000

Canary Deployment with Istio

# VirtualService — route 10% to canary
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: order-service
spec:
  hosts:
    - order-service
  http:
    - route:
        - destination:
            host: order-service
            subset: stable
          weight: 90
        - destination:
            host: order-service
            subset: canary
          weight: 10
---
# DestinationRule — define subsets
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: order-service
spec:
  host: order-service
  subsets:
    - name: stable
      labels:
        version: v1
    - name: canary
      labels:
        version: v2

Zero-Downtime Deployments

The Critical Sequence

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
sequenceDiagram
    participant K as Kubernetes
    participant P as Pod (v1)
    participant LB as Service/LB
    participant N as New Pod (v2)

    Note over K,N: Zero-downtime deployment flow
    K->>N: 1. Create new pod (v2)
    N->>N: 2. Start application
    N->>K: 3. Readiness probe ✅
    K->>LB: 4. Add v2 to endpoints
    K->>P: 5. Send SIGTERM to v1
    K->>LB: 6. Remove v1 from endpoints
    P->>P: 7. Graceful shutdown<br/>(finish in-flight requests)
    Note over P: preStop hook sleep<br/>(wait for LB to drain)
    P->>K: 8. Pod terminated

    Note over K,N: ⚠️ Steps 5-6 race condition!<br/>preStop hook prevents dropped requests

Spring Boot Configuration

# application.yml
server:
  shutdown: graceful          # wait for in-flight requests

spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s   # max 30s to finish requests

# Kubernetes deployment
spec:
  containers:
    - name: app
      lifecycle:
        preStop:
          exec:
            command: ["sh", "-c", "sleep 5"]  # wait for LB to drain
      readinessProbe:
        httpGet:
          path: /actuator/health/readiness
          port: 8080
        initialDelaySeconds: 10
        periodSeconds: 5
        failureThreshold: 3
      livenessProbe:
        httpGet:
          path: /actuator/health/liveness
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 10
        failureThreshold: 5      # more tolerant — don't kill during GC pause

Database Migrations Without Downtime

The Expand-Contract pattern:

-- Phase 1: EXPAND (safe — additive change)
ALTER TABLE orders ADD COLUMN customer_email VARCHAR(255);  -- nullable!

-- Phase 2: MIGRATE (backfill)
UPDATE orders SET customer_email = (
    SELECT email FROM customers WHERE customers.id = orders.customer_id
);

-- Phase 3: CONTRACT (only after all code reads new column)
ALTER TABLE orders DROP COLUMN customer_id_legacy;

Chaos Engineering

The Process

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    SS["1. Define<br/>Steady State<br/>(normal metrics)"] --> H["2. Hypothesize<br/>'System will recover<br/>within 30s'"]
    H --> I["3. Inject Fault<br/>(kill pod, add latency,<br/>fill disk)"]
    I --> O["4. Observe<br/>(did metrics recover?<br/>did alerts fire?)"]
    O --> L["5. Learn<br/>(fix weaknesses,<br/>update runbooks)"]
    L -->|"next experiment"| SS

    style I fill:#FFCDD2,stroke:#C62828,color:#000
    style L fill:#E8F5E9,stroke:#2E7D32,color:#000

Types of Chaos Experiments

Chaos Monkey for Spring Boot

# Enable chaos in non-prod environments
chaos:
  monkey:
    enabled: true
    watcher:
      controller: true
      restController: true
      service: true
      repository: true
    assaults:
      level: 5                    # 1 in 5 requests affected
      latencyActive: true
      latencyRangeStart: 500      # add 500-1000ms latency
      latencyRangeEnd: 1000
      exceptionsActive: true
      killApplicationActive: false  # don't kill in shared environments!

Incident Response Playbook

Severity Levels

Incident Timeline

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    D["🔔 DETECT<br/>(alert fires)"] --> T["🔍 TRIAGE<br/>(assess severity,<br/>assign IC)"]
    T --> M["🛡️ MITIGATE<br/>(stop bleeding:<br/>rollback, scale, disable)"]
    M --> R["🔧 RESOLVE<br/>(root cause fix)"]
    R --> P["📝 POSTMORTEM<br/>(blameless, within 48h)"]

    style D fill:#FFCDD2,stroke:#C62828,color:#000
    style M fill:#FEF3C7,stroke:#D97706,color:#000
    style R fill:#E8F5E9,stroke:#2E7D32,color:#000

Common Failure Runbooks

Observability Stack

The Three Pillars + Correlation

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart TD
    subgraph Collection["Data Collection"]
        App["Application<br/>(OpenTelemetry SDK)"]
    end

    subgraph Pipeline["Processing Pipeline"]
        OTC["OTel Collector<br/>(filter, batch, export)"]
    end

    subgraph Backends["Storage & Query"]
        Traces["Jaeger/Tempo<br/>(distributed traces)"]
        Metrics["Prometheus<br/>(time-series metrics)"]
        Logs["Loki/Elasticsearch<br/>(structured logs)"]
    end

    subgraph Viz["Visualization"]
        Grafana["Grafana<br/>(unified dashboards)"]
    end

    App -->|"traces, metrics, logs"| OTC
    OTC --> Traces & Metrics & Logs
    Traces & Metrics & Logs --> Grafana

    style App fill:#E3F2FD,stroke:#1565C0,color:#000
    style OTC fill:#FEF3C7,stroke:#D97706,color:#000
    style Grafana fill:#E8F5E9,stroke:#2E7D32,color:#000

Distributed Tracing: Correlation IDs

// MDC propagation — every log line includes the trace ID
@Component
public class TraceFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res,
                                     FilterChain chain) throws ServletException, IOException {
        String traceId = req.getHeader("X-Trace-Id");
        if (traceId == null) traceId = UUID.randomUUID().toString();
        MDC.put("traceId", traceId);
        res.setHeader("X-Trace-Id", traceId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
    }
}

// logback-spring.xml pattern
// %d{ISO8601} [%thread] %-5level %logger - [traceId=%X{traceId}] %msg%n

SLI / SLO / SLA

Scaling Strategies

Horizontal Pod Autoscaler (HPA)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70    # scale when CPU > 70%
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "1000"      # scale when > 1000 RPS per pod
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30   # wait 30s before scaling up more
      policies:
        - type: Percent
          value: 50                     # add max 50% more pods at once
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300  # wait 5min before scaling down
      policies:
        - type: Pods
          value: 2                      # remove max 2 pods per minute
          periodSeconds: 60

KEDA — Event-Driven Autoscaling

# Scale based on Kafka consumer lag
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor
spec:
  scaleTargetRef:
    name: order-processor
  minReplicaCount: 1
  maxReplicaCount: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka:9092
        consumerGroup: order-processor
        topic: orders
        lagThreshold: "100"    # scale when lag > 100 messages per partition

Scaling Anti-Patterns

Production Readiness Checklist