Spring Boot Observability

Logging, Metrics, and Distributed Tracing — the three pillars that let you understand what your system is doing in production without attaching a debugger.

The Three Pillars of Observability

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    O(("Observability")) --> L["Logs<br/>What happened"]
    O --> M["Metrics<br/>How much / how fast"]
    O --> T["Traces<br/>Where did it go"]

    L --> LD["Structured JSON<br/>Correlation IDs"]
    M --> MD["Counters, Timers<br/>Gauges, Histograms"]
    T --> TD["Trace & Span IDs<br/>Propagation"]

    style O fill:#DBEAFE,stroke:#93C5FD,stroke-width:2px,color:#1E40AF
    style L fill:#D1FAE5,stroke:#6EE7B7,stroke-width:2px,color:#065F46
    style M fill:#FEF3C7,stroke:#FCD34D,stroke-width:2px,color:#92400E
    style T fill:#FEE2E2,stroke:#FCA5A5,stroke-width:2px,color:#991B1B
    style LD fill:#ECFDF5,stroke:#6EE7B7,color:#065F46
    style MD fill:#FFFBEB,stroke:#FCD34D,color:#92400E
    style TD fill:#FEF2F2,stroke:#FCA5A5,color:#991B1B

1. Structured Logging

Why Structured Logs?

Plain text logs (INFO: order processed) are impossible to query at scale. Structured JSON logs let you filter, aggregate, and correlate in Loki, Elasticsearch, or CloudWatch.

Logback JSON with logstash-logback-encoder

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>7.4</version>
</dependency>

logback-spring.xml:

<configuration>
    <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <includeMdcKeyName>traceId</includeMdcKeyName>
            <includeMdcKeyName>spanId</includeMdcKeyName>
            <includeMdcKeyName>correlationId</includeMdcKeyName>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="JSON"/>
    </root>
</configuration>

Output:

{
  "@timestamp": "2024-03-15T10:23:45.123Z",
  "level": "INFO",
  "logger_name": "com.example.OrderService",
  "message": "Order created",
  "traceId": "abc123def456",
  "spanId": "789ghi",
  "correlationId": "order-42",
  "orderId": "ORD-12345",
  "userId": "USR-789"
}

MDC for Correlation IDs

@Component
public class CorrelationFilter implements Filter {

    @Override
    public void doFilter(ServletRequest req, ServletResponse res,
                         FilterChain chain) throws IOException, ServletException {
        String correlationId = Optional
            .ofNullable(((HttpServletRequest) req).getHeader("X-Correlation-ID"))
            .orElse(UUID.randomUUID().toString());

        MDC.put("correlationId", correlationId);
        try {
            chain.doFilter(req, res);
        } finally {
            MDC.clear();
        }
    }
}

Spring Boot Auto-Configuration

Spring Boot 3+ automatically:

• Puts traceId and spanId into MDC (when Micrometer Tracing is on the classpath)
• Configures log correlation via logging.pattern.correlation
• Supports logging.structured.format.console=ecs or logstash (Boot 3.4+)

logging:
  structured:
    format:
      console: logstash
  pattern:
    correlation: "[${spring.application.name:},%mdc{traceId:-},%mdc{spanId:-}]"

2. Micrometer Metrics

Core Meter Types

Dimensional Tags

Every metric carries tags (key-value pairs) for slicing and filtering:

@Component
@RequiredArgsConstructor
public class OrderMetrics {

    private final MeterRegistry registry;

    public void recordOrderPlaced(String region, String paymentMethod) {
        registry.counter("orders.placed",
            "region", region,
            "payment_method", paymentMethod
        ).increment();
    }

    public void recordOrderLatency(String endpoint, Duration duration) {
        registry.timer("orders.processing.time",
            "endpoint", endpoint
        ).record(duration);
    }

    public void registerQueueSize(BlockingQueue<?> queue) {
        Gauge.builder("orders.queue.size", queue, BlockingQueue::size)
            .description("Pending orders in queue")
            .tag("type", "processing")
            .register(registry);
    }
}

Distribution Summary

DistributionSummary summary = DistributionSummary.builder("http.request.size")
    .description("Request payload size in bytes")
    .baseUnit("bytes")
    .tag("endpoint", "/api/orders")
    .publishPercentiles(0.5, 0.95, 0.99)
    .publishPercentileHistogram()
    .register(registry);

summary.record(requestBody.length());

Prometheus + Grafana Integration

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>

management:
  endpoints:
    web:
      exposure:
        include: health,metrics,prometheus
  prometheus:
    metrics:
      export:
        enabled: true

Prometheus scrapes /actuator/prometheus and Grafana visualizes it. Common auto-instrumented metrics:

• http.server.requests (timer) -- request duration by URI, method, status
• jvm.memory.used (gauge) -- heap/non-heap by area
• jvm.gc.pause (timer) -- GC pause duration
• system.cpu.usage (gauge) -- process CPU
• hikaricp.connections.active (gauge) -- DB pool usage

Custom Metric with @Timed

@Service
public class PaymentService {

    @Timed(value = "payment.process.time",
           description = "Time to process payment",
           extraTags = {"gateway", "stripe"})
    public PaymentResult processPayment(PaymentRequest request) {
        // business logic
    }
}

@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
    return new TimedAspect(registry);
}

3. Distributed Tracing

Micrometer Tracing (formerly Spring Cloud Sleuth)

Spring Cloud Sleuth is deprecated since Spring Boot 3. Micrometer Tracing is the replacement.

<!-- Bridge to Brave (Zipkin-compatible) -->
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>

<!-- Reporter: send spans to Zipkin/Tempo -->
<dependency>
    <groupId>io.zipkin.reporter2</groupId>
    <artifactId>zipkin-reporter-brave</artifactId>
</dependency>

OR for OpenTelemetry:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Trace and Span IDs

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    A["API Gateway<br/>traceId: abc123"] --> B["Order Service<br/>spanId: span-1"]
    B --> C["Payment Service<br/>spanId: span-2"]
    B --> D["Inventory Service<br/>spanId: span-3"]
    C --> E["Fraud Check<br/>spanId: span-4"]

    style A fill:#DBEAFE,stroke:#93C5FD,stroke-width:2px,color:#1E40AF
    style B fill:#D1FAE5,stroke:#6EE7B7,stroke-width:2px,color:#065F46
    style C fill:#FEF3C7,stroke:#FCD34D,stroke-width:2px,color:#92400E
    style D fill:#FEF3C7,stroke:#FCD34D,stroke-width:2px,color:#92400E
    style E fill:#FEE2E2,stroke:#FCA5A5,stroke-width:2px,color:#991B1B

• Trace ID -- single ID shared across ALL services for one request
• Span ID -- unique per operation within the trace
• Parent Span ID -- links spans into a tree

Context Propagation

Trace context is automatically propagated via HTTP headers:

Spring Boot auto-propagates through:

• RestTemplate / WebClient / RestClient (auto-instrumented)
• Kafka headers
• Spring Integration channels
• @Async thread boundaries

management:
  tracing:
    sampling:
      probability: 1.0  # 100% in dev, lower in prod (0.1 = 10%)
    propagation:
      type: w3c  # or b3 for Zipkin-native
  zipkin:
    tracing:
      endpoint: http://tempo:9411/api/v2/spans

Custom Spans

@Service
@RequiredArgsConstructor
public class OrderService {

    private final Tracer tracer;

    public Order createOrder(OrderRequest request) {
        Span span = tracer.nextSpan().name("create-order").start();
        try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
            span.tag("orderId", request.getOrderId());
            span.tag("userId", request.getUserId());

            Order order = processOrder(request);

            span.event("order-persisted");
            return order;
        } catch (Exception e) {
            span.error(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

4. Observation API (Spring 6)

The Observation API unifies metrics and tracing into a single abstraction. One observation produces both a timer metric AND a trace span.

How It Works

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    Code["Your Code"] --> OA["Observation API"]
    OA --> MH["MeterObservation<br/>Handler"]
    OA --> TH["TracingObservation<br/>Handler"]
    MH --> Met["Metrics<br/>(Micrometer)"]
    TH --> Tr["Traces<br/>(Zipkin/Tempo)"]

    style Code fill:#EFF6FF,stroke:#93C5FD,stroke-width:2px,color:#1E40AF
    style OA fill:#DBEAFE,stroke:#93C5FD,stroke-width:2px,color:#1E40AF
    style MH fill:#D1FAE5,stroke:#6EE7B7,stroke-width:2px,color:#065F46
    style TH fill:#FEF3C7,stroke:#FCD34D,stroke-width:2px,color:#92400E
    style Met fill:#ECFDF5,stroke:#6EE7B7,color:#065F46
    style Tr fill:#FFFBEB,stroke:#FCD34D,color:#92400E

@Observed Annotation

@Service
public class InventoryService {

    @Observed(name = "inventory.check",
              contextualName = "check-stock",
              lowCardinalityKeyValues = {"warehouse", "us-east"})
    public StockResult checkStock(String sku) {
        // This method automatically:
        // 1. Creates a Timer metric named "inventory.check"
        // 2. Creates a Trace span named "check-stock"
        return repository.findBySku(sku);
    }
}

@Bean
ObservedAspect observedAspect(ObservationRegistry registry) {
    return new ObservedAspect(registry);
}

Programmatic Observation

@Service
@RequiredArgsConstructor
public class NotificationService {

    private final ObservationRegistry registry;

    public void sendNotification(Notification notification) {
        Observation.createNotStarted("notification.send", registry)
            .lowCardinalityKeyValue("type", notification.getType())
            .lowCardinalityKeyValue("channel", notification.getChannel())
            .highCardinalityKeyValue("recipientId", notification.getRecipientId())
            .observe(() -> doSend(notification));
    }
}

Key concepts:

• lowCardinalityKeyValue -- finite set of values (used as metric tags AND span tags)
• highCardinalityKeyValue -- unbounded values like user IDs (span tags only, NOT metric tags)

5. Custom Health Indicators & Info Contributors

Custom Health Indicator

@Component
public class ExternalApiHealthIndicator implements HealthIndicator {

    private final RestClient restClient;

    @Override
    public Health health() {
        try {
            ResponseEntity<Void> response = restClient
                .get().uri("/health")
                .retrieve()
                .toBodilessEntity();

            if (response.getStatusCode().is2xxSuccessful()) {
                return Health.up()
                    .withDetail("responseTime", "45ms")
                    .build();
            }
            return Health.down()
                .withDetail("status", response.getStatusCode())
                .build();
        } catch (Exception e) {
            return Health.down(e).build();
        }
    }
}

Custom Info Contributor

@Component
public class ServiceInfoContributor implements InfoContributor {

    @Override
    public void contribute(Info.Builder builder) {
        builder.withDetail("deployment", Map.of(
            "region", "us-east-1",
            "cluster", "production-a",
            "version", "2.3.1",
            "lastDeploy", Instant.now().toString()
        ));
        builder.withDetail("features", Map.of(
            "newCheckout", true,
            "darkMode", false
        ));
    }
}

6. The Grafana Stack (Prometheus + Loki + Tempo)

%%{init: {'theme': 'base', 'themeVariables': {'fontSize': '13px', 'fontFamily': 'Inter, -apple-system, sans-serif'}, 'flowchart': {'nodeSpacing': 30, 'rankSpacing': 50, 'padding': 12, 'curve': 'basis'}, 'sequence': {'actorMargin': 60, 'messageMargin': 40}, 'class': {'padding': 12}}}%%
flowchart LR
    App["Spring Boot App"] --> Prom["Prometheus<br/>/actuator/prometheus"]
    App --> Loki["Loki<br/>JSON logs via Promtail"]
    App --> Tempo["Tempo<br/>OTLP/Zipkin spans"]

    Prom --> Graf["Grafana Dashboard"]
    Loki --> Graf
    Tempo --> Graf

    Graf --> Alert["Alertmanager<br/>PagerDuty/Slack"]

    style App fill:#DBEAFE,stroke:#93C5FD,stroke-width:2px,color:#1E40AF
    style Prom fill:#FEF3C7,stroke:#FCD34D,stroke-width:2px,color:#92400E
    style Loki fill:#D1FAE5,stroke:#6EE7B7,stroke-width:2px,color:#065F46
    style Tempo fill:#FEE2E2,stroke:#FCA5A5,stroke-width:2px,color:#991B1B
    style Graf fill:#EFF6FF,stroke:#93C5FD,stroke-width:2px,color:#1E40AF
    style Alert fill:#FFFBEB,stroke:#FCD34D,color:#92400E

Docker Compose (Local Dev)

services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

  loki:
    image: grafana/loki:latest
    ports: ["3100:3100"]

  tempo:
    image: grafana/tempo:latest
    ports: ["3200:3200", "9411:9411"]
    command: ["-config.file=/etc/tempo.yaml"]

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin

prometheus.yml:

scrape_configs:
  - job_name: 'spring-boot'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 15s
    static_configs:
      - targets: ['host.docker.internal:8080']

7. Configuration Properties Reference

Common Tags (applied to every metric)

management:
  metrics:
    tags:
      application: ${spring.application.name}
      environment: ${spring.profiles.active:default}
      region: us-east-1

Quick Recall

Interview Template