The three pillars are logs, metrics, and traces. They answer different questions: metrics tell you something is wrong, traces tell you where in the call graph the problem is, and logs tell you exactly what happened at that point. They're most powerful when connected — a metric alert leads you to a trace, and the trace leads you to the relevant log lines.
Logs are discrete events with context. A log line captures what happened at a specific moment. Structured logging (JSON) makes them machine-parseable and searchable. Aggregated in ELK (Elasticsearch + Logstash + Kibana) or Grafana Loki. Strength: rich detail. Weakness: high volume, expensive to store and query at scale.
Metrics are numerical measurements over time — counters, gauges, histograms. Low storage cost, fast to query. Used for dashboards, alerting, SLO tracking. Prometheus + Grafana is the standard stack. Strength: cheap, fast, great for alerting. Weakness: aggregated — you lose individual request detail.
Traces represent a single request's journey across services as a tree of spans. Each span captures start time, duration, and metadata for one operation. Used to understand latency distribution and find bottlenecks. Jaeger, Tempo, Zipkin. Strength: end-to-end visibility. Weakness: high data volume — requires sampling.
Metrics alert you there's a problem → P99 latency is 800ms, baseline is 50ms. Traces locate the problem → find slow traces from that window, identify which span is responsible (DB query, downstream call). Logs explain the problem → look at log lines from that specific span — slow query text, error messages, context.
Without traces, you're correlating log timestamps across multiple services manually — error-prone and time-consuming. Teams that skip traces regret it the first time they debug a cross-service latency issue.
Many teams have logs and metrics but no traces, often because traces are "the third thing" and get deprioritised. Adding traces retroactively to a large microservices system is hard: every HTTP client, Kafka producer/consumer, and DB connection needs instrumentation. The right time to add tracing is at the start of the project, not after the first production incident that requires it.
Structured logging means emitting log lines as key-value pairs (JSON) rather than unstructured strings. Every field is individually searchable in your log aggregator. MDC (Mapped Diagnostic Context) is a thread-local map that Logback/Log4j2 automatically appends to every log statement — set a traceId once on request entry and it appears in every log line for that request without any per-call code.
// Unstructured — a string, unparseable:
log.info("Order {} processed by user {} in {}ms", orderId, userId, duration);
// Structured — every field is independently queryable:
log.info("Order processed",
kv("orderId", orderId),
kv("userId", userId),
kv("durationMs", duration));
// Output: {"level":"INFO","message":"Order processed","orderId":"ORD-123",
// "userId":"U-456","durationMs":42,"traceId":"abc123",...}MDC filter — runs on every incoming HTTP request:
@Component
class TraceIdFilter extends OncePerRequestFilter {
@Override
protected void doFilterInternal(
HttpServletRequest req, HttpServletResponse res,
FilterChain chain) throws ServletException, IOException {
String traceId = Optional
.ofNullable(req.getHeader("X-Trace-Id")) // propagate from upstream
.orElse(UUID.randomUUID().toString());
MDC.put("traceId", traceId);
MDC.put("userId", extractUserId(req));
res.setHeader("X-Trace-Id", traceId); // echo back to caller
try {
chain.doFilter(req, res);
} finally {
MDC.clear(); // always clean up
}
}
}Also propagate traceId on outgoing HTTP calls so downstream services include it in their logs:
// RestTemplate interceptor — adds X-Trace-Id to every outgoing call
restTemplate.setInterceptors(List.of((request, body, execution) -> {
String traceId = MDC.get("traceId");
if (traceId != null) request.getHeaders().set("X-Trace-Id", traceId);
return execution.execute(request, body);
}));MDC is thread-local. When @Async or CompletableFuture spawns a new thread, that thread starts with an empty MDC — your traceId disappears from all async log lines. Fix: copy MDC in a TaskDecorator:
@Bean
public Executor asyncExecutor() {
ThreadPoolTaskExecutor exec = new ThreadPoolTaskExecutor();
exec.setTaskDecorator(runnable -> {
Map<String, String> mdc = MDC.getCopyOfContextMap();
return () -> {
try {
if (mdc != null) MDC.setContextMap(mdc);
runnable.run();
} finally { MDC.clear(); }
};
});
exec.initialize();
return exec;
}The decorator captures the MDC snapshot on the submitting thread, then restores it in the executing thread before the task runs.
A trace represents a single request end-to-end across multiple services, identified by a unique traceId. Each operation within the trace is a span — it has a spanId, a parentSpanId, start time, and duration. Spans form a tree. Context propagation means every service receives the traceId and spanId from its caller via HTTP headers, creates a child span, does its work, and passes the context to its own downstream calls.
The W3C traceparent header is the standard format: 00-{traceId}-{parentSpanId}-{flags}. Each service reads this header, extracts the traceId and parentSpanId, creates a new child span with the parentSpanId as its parent, and injects its own spanId into outgoing calls.
// Spring Boot 3 with Micrometer Tracing — automatic context propagation
// application.yml
management:
tracing:
sampling:
probability: 0.1 # sample 10% of requests
zipkin:
tracing:
endpoint: http://zipkin:9411/api/v2/spans
// pom.xml
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-brave</artifactId>
</dependency>
<dependency>
<groupId>io.zipkin.reporter2</groupId>
<artifactId>zipkin-reporter-brave</artifactId>
</dependency>With these dependencies, Spring Boot auto-instruments HTTP requests (both inbound and outbound via RestTemplate/WebClient), Kafka consumers/producers, and database calls. The traceId is also injected into MDC automatically — every log line gets the traceId without any extra code.
For custom spans around significant operations:
@Autowired Tracer tracer;
public Order processOrder(CreateOrderRequest req) {
Span span = tracer.nextSpan().name("process-order").start();
try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
span.tag("order.type", req.getType());
return orderService.create(req);
} catch (Exception e) {
span.error(e);
throw e;
} finally {
span.end();
}
}Recording every span for every request is expensive — a service handling 10,000 req/s with 20 spans per request generates 200,000 spans/second. Two sampling approaches:
Head-based sampling: decision made at the first service (e.g. sample 10% of requests). Cheap but you might miss the rare slow request in the 90% unsampled.
Tail-based sampling: decision made after the trace completes, prioritising slow and errored traces. Catches the outliers that matter most for debugging. Requires a collector that buffers completed traces before making the sampling decision (OpenTelemetry Collector supports this).
Micrometer Tracing replaced Spring Cloud Sleuth in Spring Boot 3. If your team is upgrading from Boot 2 to Boot 3, the migration is non-trivial: remove spring-cloud-sleuth, add micrometer-tracing-bridge-brave or the OTel bridge. Trace context is also lost across @Async boundaries unless you configure a ContextPropagatingTaskDecorator — the same problem as MDC, but for trace context.
Micrometer is Spring Boot's metrics facade — one API, pluggable backends (Prometheus, Datadog, CloudWatch). It ships with spring-boot-starter-actuator. Four core metric types cover almost all use cases: Counter, Gauge, Timer, and DistributionSummary. The most important decision beyond choosing a type is cardinality: every tag combination creates a separate time series.
Counter — monotonically increasing. For counting events. Query as a rate: rate(orders_processed_total[5m]).
Counter.builder("orders.processed")
.tag("status", "SUCCESS")
.description("Orders processed successfully")
.register(registry)
.increment();Gauge — current value, can go up or down. For measuring instantaneous state:
Gauge.builder("order.queue.size", orderQueue, Queue::size)
.description("Orders waiting to be processed")
.register(registry);Timer — measures duration AND count. Auto-computes count, total time, max, and optionally percentiles. The most useful type for latency:
Timer timer = Timer.builder("payment.charge")
.tag("provider", "stripe")
.publishPercentileHistogram(true) // for aggregatable fleet-wide percentiles
.register(registry);
Timer.Sample sample = Timer.start(registry);
try {
return stripeClient.charge(req);
} finally {
sample.stop(timer);
}DistributionSummary — like Timer but for non-time values (e.g. request payload size in bytes).
Rate: requests per second. Errors: error rate (%). Duration: latency percentiles (P50, P95, P99). Start here before adding domain-specific metrics. Spring Boot auto-instruments HTTP requests, JVM heap/GC, HikariCP pool size/active connections, Kafka consumer lag, and Spring Cache hits/misses — all without writing any code.
publishPercentiles(0.99) computes a pre-aggregated, non-aggregatable percentile client-side. You cannot average or sum P99 values across instances — the average of 5 P99s is not the fleet P99. It produces a per-pod P99 only.
publishPercentileHistogram(true) emits bucket counts (_bucket metrics). Prometheus histogram_quantile() recomputes the percentile from the combined buckets of all pods — this is the fleet-wide P99. Always use histogram for production dashboards:
# Prometheus query — fleet-wide P99 across all pods
histogram_quantile(0.99,
sum(rate(payment_charge_seconds_bucket[5m])) by (le)
)SLI (Service Level Indicator) is a measurement — a specific metric that captures how well you're serving users. SLO (Service Level Objective) is a target for that SLI — a promise you make to yourself about reliability. The error budget is the difference between perfect and your SLO, expressed as the amount of failure you're allowed. Burning the error budget too fast means slowing down deployments. Having budget to spare means you can move fast and take risks.
Good SLIs measure what users actually experience, not internal service health:
| Service | SLI | SLO target |
|---|---|---|
| API | % requests completing in <200ms | 99% over 30 days |
| API | % requests returning non-5xx | 99.9% over 30 days |
| Background job | % jobs completing within SLA window | 99.5% over 7 days |
| Kafka consumer | Consumer lag < 30s | 99% of time |
Error budget: 99.9% availability over 30 days = 0.1% allowed failure = ~43.2 minutes/month of downtime. This is the team's budget to spend on failures from any cause: bad deploys, infrastructure outages, dependency failures.
# Prometheus SLO burn rate alert (multi-window, from Google SRE book)
# Alert when you're burning the monthly budget too fast
- alert: HighErrorBudgetBurn
expr: |
(
job:slo_errors:ratio_rate1h{job="payment-service"} > (14.4 * 0.001)
and
job:slo_errors:ratio_rate5m{job="payment-service"} > (14.4 * 0.001)
)
for: 2m
annotations:
summary: "Burning error budget at 14.4x rate — will exhaust in 5 days"Error budget isn't just a monitoring concept — it drives engineering decisions. If you've consumed 80% of your monthly budget in week 1: freeze non-critical feature deployments, focus the team on reliability improvements, investigate what caused the burn. If you have plenty of budget: safe to ship fast, try risky experiments, accept more deployment risk.
This framing removes "reliability vs velocity" arguments: the error budget is objective. When it's gone, velocity stops. When it's full, move fast. Both sides of the tradeoff are satisfied.
Setting the SLO too tight on day one. If you set 99.99% and your service currently runs at 99.7%, you're immediately in a permanent incident. Good starting approach: measure your actual 30-day performance, set your initial SLO slightly below current performance to establish a baseline, then tighten quarterly as reliability improves. Don't set aspirational SLOs — set realistic ones you can actually meet.
Latency debugging is structured: metrics establish the scope, traces locate the bottleneck, and logs explain the root cause. Skipping ahead to logs without traces is the most common mistake — you end up grepping millions of lines without knowing which request to look at.
Step 1 — Scope the problem. Is it all endpoints or one specific endpoint? All users or a specific region/tenant? Did it start with a deployment? Check the RED dashboard: rate, error rate, P99 latency. Check if it's correlated with a deploy timestamp.
Step 2 — Find the slow traces. Open your tracing tool (Jaeger/Tempo), filter by the affected endpoint and time window, sort by duration descending. Find traces at the P99 boundary. Look at the flame chart — which span is taking the most time?
Step 3 — Check the DB first (it's the most common cause).
-- Active queries and their duration (PostgreSQL)
SELECT pid, now() - pg_stat_activity.query_start AS duration, query, state
FROM pg_stat_activity
WHERE state != 'idle' AND query_start IS NOT NULL
ORDER BY duration DESC;
-- Slowest queries by total time
SELECT query, calls, total_exec_time, mean_exec_time
FROM pg_stat_statements
ORDER BY mean_exec_time DESC LIMIT 20;Also check HikariCP pool: /actuator/metrics/hikaricp.connections.active — if it's at max, requests are queuing for a connection.
Step 4 — Check GC. /actuator/metrics/jvm.gc.pause — long GC pauses spike P99. Also check heap usage: jvm.memory.used. GC pauses of 500ms+ will appear as consistent P99 spikes across all endpoints simultaneously.
Step 5 — Check thread pool. /actuator/metrics/executor.active — if at max, requests are queueing. /actuator/threaddump to see what threads are blocked on (a lock, I/O, a downstream call).
Step 6 — Check downstream services. Did the slow span involve a downstream HTTP call? Check the downstream service's metrics. Check circuit breaker state. Check connection pool to downstream.
The latency spike that resolves itself — GC pauses look exactly like this. A full GC pause of 2 seconds every few minutes causes P99 spikes that look intermittent and mysterious. Find it in GC logs or jvm.gc.pause metrics. Root cause is usually heap pressure: memory leak, or a sudden burst of object allocation (processing a large payload). Fix: increase heap, tune GC settings, or find and fix the allocation source using async-profiler in allocation profiling mode.
Liveness and readiness answer different questions. Liveness: "Is this pod alive — should Kubernetes restart it?" Readiness: "Is this pod ready to serve traffic — should the load balancer send requests to it?" The critical distinction: liveness failure triggers a restart; readiness failure only removes the pod from load balancing rotation. Confusing them is a common source of restart loops in production.
Liveness probe — checks only that the JVM is alive and responsive. Never check downstream dependencies (DB, Redis, Kafka) in a liveness probe. If Redis goes down and your liveness probe fails, Kubernetes restarts your pod. The restart doesn't fix Redis, and now you've lost warm caches and in-flight requests. Restart loop begins.
Readiness probe — can check critical downstream dependencies. If the DB connection pool is exhausted or unavailable, the pod shouldn't receive traffic — it can't serve requests. But only check dependencies that are truly required to serve any request. Optional dependencies should not make the pod unready.
# application.yml
management:
endpoint:
health:
probes:
enabled: true
group:
liveness:
include: livenessState # only JVM liveness — never downstream
readiness:
include: readinessState, db, redis # check critical deps
health:
livenessstate:
enabled: true
readinessstate:
enabled: trueCustom readiness health indicator for Kafka:
@Component
class KafkaReadinessIndicator implements HealthIndicator {
@Override
public Health health() {
try {
kafkaAdmin.describeTopics(List.of("orders"));
return Health.up().build();
} catch (Exception e) {
return Health.down().withException(e).build();
}
}
}For services that take 30–60s to start (large Spring context, DB migration on startup), liveness probes can kill the pod before it's ready. Startup probes solve this: Kubernetes uses only the startup probe until it passes, then switches to liveness/readiness. This prevents premature restarts of slow-starting services:
# Kubernetes spec
startupProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
failureThreshold: 30 # 30 × 10s = 5 minutes to start
periodSeconds: 10
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
periodSeconds: 10Slow liveness probe causing a restart loop: the probe hits /actuator/health, which checks the DB, which is slow under load, which causes the probe to time out, which triggers a restart, which makes the DB load worse. The fix: liveness probe must be fast and dependency-free. If it takes more than 100ms to respond, something is wrong with the probe design, not the service.
OpenTelemetry (OTel) is a CNCF standard for observability — one SDK and API for logs, metrics, and traces, with pluggable exporters to any backend. The Java agent provides zero-code auto-instrumentation: attach it as a -javaagent flag and it automatically instruments Spring MVC, WebFlux, JDBC, Kafka, Redis, scheduling, and more. You get traces for every operation without writing a single line of instrumentation code.
# JVM args — attach the agent
JAVA_TOOL_OPTIONS=-javaagent:/path/to/opentelemetry-javaagent.jar
# Environment variables for configuration
OTEL_SERVICE_NAME=payment-service
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1 # sample 10%
OTEL_LOGS_EXPORTER=otlp # send structured logs to collector too
OTEL_METRICS_EXPORTER=otlpWhat you get automatically with the agent:
traceparent header injection/extraction on HTTP and Kafka message headers// Option 1: @WithSpan annotation (OTel annotation library)
@WithSpan("process-payment")
public PaymentResult processPayment(@SpanAttribute("order.id") String orderId) {
return doProcess(orderId);
}
// Option 2: programmatic span
Tracer tracer = GlobalOpenTelemetry.getTracer("payment-service");
Span span = tracer.spanBuilder("fraud-check").startSpan();
try (Scope scope = span.makeCurrent()) {
span.setAttribute("order.id", orderId);
span.setAttribute("order.amount", amount);
return fraudService.check(orderId);
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw e;
} finally {
span.end();
}Services don't export directly to Jaeger/Datadog/Tempo. They send to an OTel Collector, which receives telemetry, processes it (sample, enrich with k8s metadata, filter noisy spans), and exports to one or more backends. This decouples your services from your observability backend — changing from Jaeger to Tempo doesn't require touching application config.
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
timeout: 1s
tail_sampling:
policies:
- name: errors-policy
type: status_code
status_code: { status_codes: [ERROR] }
- name: slow-traces-policy
type: latency
latency: { threshold_ms: 1000 }
exporters:
jaeger:
endpoint: jaeger:14250
prometheus:
endpoint: "0.0.0.0:8889"The OTel Java agent instruments JDBC and logs the SQL statement by default. Verify it's sanitising bind parameters — you don't want WHERE password = 'hunter2' appearing in your traces. It sanitises by default in recent versions, but confirm with your specific agent version. Also: disable the agent in tests with OTEL_SDK_DISABLED=true to avoid spurious spans in test output and potential test interference.