Rate Limiting

The 30-Second Explanation

Rate limiting = controlling how many requests a user/service can make in a given time window.

The key insight: Rate limiting is NOT just about security. It's about service stability and fair resource allocation across tenants.

The Nightclub Analogy

You're a bouncer at a club with a 200-person capacity.

Algorithms at a Glance

Algorithms — What FAANG Actually Asks

Fixed Window Counter

Sliding Window Log

Sliding Window Counter

Token Bucket

Leaky Bucket

Where to Rate Limit

Distributed Rate Limiting — The Hard Part

Single server? Easy. In-memory counter.

100 servers behind a load balancer? Now you need shared state.

HTTP Response: What to Return

Real Systems

The 3 Mistakes That Get You Rejected

Interview Answer Template

"For [system], I'd implement rate limiting at [layer] using [algorithm] because [reason]. For distributed enforcement, I'd use [Redis/local+sync] with [token bucket/sliding window]. Key decisions: fail-open vs fail-closed on limiter failure, per-user vs per-IP vs per-API-key granularity, and returning proper 429 with Retry-After headers for good client behavior."

Rate Limiting vs Throttling — The Difference That Trips Up Seniors

The Core Distinction

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart TD
    REQ[Incoming Requests] --> RL{Rate Limiter}
    RL -->|Under limit| PASS[✓ Process normally]
    RL -->|Over limit| REJECT[✗ 429 Reject immediately]

    REQ2[Incoming Requests] --> TH{Throttler}
    TH -->|Under limit| PASS2[✓ Process normally]
    TH -->|Over limit| SLOW[⏳ Slow down / queue / degrade]

    style REJECT fill:#ef4444,color:#fff
    style SLOW fill:#f59e0b,color:#fff
    style PASS fill:#22c55e,color:#fff
    style PASS2 fill:#22c55e,color:#fff

Real-World Examples (All Levels)

For Junior Engineers: The Highway Analogy

Rate Limiting = Toll booth closes. Cars turned away. "Road full, take another route."
Throttling    = Speed limit drops from 100km/h to 60km/h. Everyone still moves, just slower.

For Senior Engineers: Production Patterns

For Architects: System-Level Patterns

Pattern 1: Adaptive Throttling (Netflix)

Normal load:     Full quality 4K stream (25 Mbps)
High load:       Downgrade to 1080p (5 Mbps)      ← Throttling (quality reduction)
Extreme load:    Downgrade to 720p (3 Mbps)       ← More throttling
Beyond capacity: "Service unavailable, try later"  ← Rate limiting kicks in

Pattern 2: Tiered Throttling (AWS)

Free tier:       Throttled to 1 req/sec sustained (burst to 5)
Standard tier:   Throttled to 10 req/sec sustained (burst to 100)
Enterprise tier: Throttled to 1000 req/sec sustained (burst to 5000)
Exceeded tier:   429 Too Many Requests             ← Rate limited

Pattern 3: Backpressure Throttling (Kafka Consumers)

Consumer lag < 1000:   Process at full speed
Consumer lag > 10000:  Throttle producer (slow accepts)
Consumer lag > 100000: Rate limit producer (reject new messages)

Where Each Applies in a Typical Architecture

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    subgraph "Client Side"
        DEB[Debounce/Throttle<br/>UI events]
    end

    subgraph "Edge / CDN"
        RL1[Rate Limit<br/>by IP · DDoS protection]
    end

    subgraph "API Gateway"
        RL2[Rate Limit<br/>by API key · per-plan]
        TH1[Throttle<br/>queue overflow requests]
    end

    subgraph "Application"
        TH2[Throttle<br/>degrade gracefully under load]
    end

    subgraph "Database"
        TH3[Throttle<br/>connection pool · query queue]
        RL3[Rate Limit<br/>max connections hard cap]
    end

    DEB --> RL1 --> RL2 --> TH1 --> TH2 --> TH3

    style RL1 fill:#ef4444,color:#fff
    style RL2 fill:#ef4444,color:#fff
    style RL3 fill:#ef4444,color:#fff
    style TH1 fill:#f59e0b,color:#fff
    style TH2 fill:#f59e0b,color:#fff
    style TH3 fill:#f59e0b,color:#fff
    style DEB fill:#3b82f6,color:#fff

Client-Side Throttling Patterns (JavaScript/Java)

// THROTTLE: Execute at most once per interval (steady rate)
// Use for: scroll events, resize, real-time search-as-you-type
function throttle(fn, limit) {
  let lastCall = 0;
  return (...args) => {
    const now = Date.now();
    if (now - lastCall >= limit) {
      lastCall = now;
      fn(...args);
    }
  };
}

// DEBOUNCE: Execute only after user STOPS for N ms
// Use for: search input (wait until user finishes typing)
function debounce(fn, delay) {
  let timer;
  return (...args) => {
    clearTimeout(timer);
    timer = setTimeout(() => fn(...args), delay);
  };
}

// Usage
searchInput.addEventListener('input', throttle(callAPI, 300));  // Max 1 call per 300ms
searchInput.addEventListener('input', debounce(callAPI, 500));  // Wait 500ms of silence

// RATE LIMITER: Hard reject after limit
RateLimiter rateLimiter = RateLimiter.of("api",
    RateLimiterConfig.custom()
        .limitForPeriod(100)           // 100 requests
        .limitRefreshPeriod(Duration.ofSeconds(1))  // per second
        .timeoutDuration(Duration.ZERO) // reject immediately (no waiting)
        .build());

// BULKHEAD (Throttle): Queue overflow, process at capacity
Bulkhead bulkhead = Bulkhead.of("db",
    BulkheadConfig.custom()
        .maxConcurrentCalls(25)         // only 25 concurrent DB calls
        .maxWaitDuration(Duration.ofMillis(500)) // queue for up to 500ms
        .build());

// RATE LIMITER: returns 429 immediately when exceeded
// BULKHEAD: queues up to 500ms, then rejects — this IS throttling

// Guava RateLimiter is actually a THROTTLER, not a rate limiter
// It blocks (slows down) callers instead of rejecting them
RateLimiter limiter = RateLimiter.create(10.0); // 10 permits per second

// This BLOCKS until a permit is available — throttling behavior
limiter.acquire(); // Waits if necessary (does not reject)
processRequest();

// vs. tryAcquire — this is rate limiting (reject if unavailable)
if (!limiter.tryAcquire()) {
    return ResponseEntity.status(429).body("Too many requests");
}
processRequest();

@Component
public class AdaptiveThrottleFilter extends OncePerRequestFilter {

    private final AtomicInteger activeRequests = new AtomicInteger(0);
    private static final int THROTTLE_THRESHOLD = 500;
    private static final int REJECT_THRESHOLD = 1000;

    @Override
    protected void doFilterInternal(HttpServletRequest request,
            HttpServletResponse response, FilterChain chain)
            throws ServletException, IOException {

        int current = activeRequests.incrementAndGet();
        try {
            if (current > REJECT_THRESHOLD) {
                // RATE LIMIT: hard reject
                response.setStatus(429);
                response.setHeader("Retry-After", "5");
                response.getWriter().write("Service overloaded");
                return;
            }
            if (current > THROTTLE_THRESHOLD) {
                // THROTTLE: degrade response
                request.setAttribute("degraded", true); // signal to return less data
            }
            chain.doFilter(request, response);
        } finally {
            activeRequests.decrementAndGet();
        }
    }
}

Throttling Strategies for Architects

The Combined Pattern: Rate Limit + Throttle Together

Most production systems use both — throttling first, rate limiting as the hard backstop:

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    R[Request] --> T{Current load?}
    T -->|"< 70% capacity"| FULL[Full Response<br/>200 OK]
    T -->|"70-90% capacity"| DEGRADE[Degraded Response<br/>200 OK · less data]
    T -->|"90-100% capacity"| QUEUE[Queue & Wait<br/>200 OK · delayed]
    T -->|"> 100% capacity"| REJECT[429 Rejected<br/>Retry-After: 5s]

    style FULL fill:#22c55e,color:#fff
    style DEGRADE fill:#84cc16,color:#fff
    style QUEUE fill:#f59e0b,color:#fff
    style REJECT fill:#ef4444,color:#fff

Real example — Salesforce API Governor Limits:

Real example — Stripe API:

Interview Answer: Rate Limiting vs Throttling

Common Mistakes in Interviews

Debounce vs Throttle vs Rate Limit — Final Comparison

Timeline (X = event, O = execution):

Events:    X X X X X - - - X X X X X - - - - -

Debounce:  - - - - - - - O - - - - - - - - O  (fires AFTER silence)
Throttle:  O - - - O - - - O - - - O - - - -  (fires every N ms)
Rate Limit: O O O O O ✗ ✗ ✗ O O O O O ✗ ✗ ✗  (allows N, rejects rest)

Quick Recall Card