The circuit breaker prevents your service from repeatedly hammering a failing downstream service. Without it, every request waits for a full timeout before failing, threads pile up, and the slowness cascades. With a circuit breaker, once failure rate crosses a threshold the circuit opens — subsequent calls fail immediately without touching the downstream service, giving it time to recover.
CLOSED — normal operation. Every call goes through. A sliding window tracks the failure rate. When the failure rate exceeds the threshold (e.g. 50% over the last 10 calls), the circuit transitions to OPEN.
OPEN — fast-fail mode. All calls immediately throw CallNotPermittedException without touching the downstream. After a configured wait duration (e.g. 30s), the circuit transitions to HALF_OPEN.
HALF_OPEN — probe mode. A limited number of calls are let through as test probes. If they succeed above the success rate threshold, the circuit closes. If they fail, it reopens and the wait duration resets.
// Resilience4j configuration
@Bean
public CircuitBreakerConfig paymentCbConfig() {
return CircuitBreakerConfig.custom()
.slidingWindowType(SlidingWindowType.COUNT_BASED)
.slidingWindowSize(10) // last 10 calls
.failureRateThreshold(50) // open at 50% failures
.minimumNumberOfCalls(5) // don't open on first failure
.waitDurationInOpenState(Duration.ofSeconds(30))
.permittedNumberOfCallsInHalfOpenState(3)
.slowCallRateThreshold(80) // also open on slow calls
.slowCallDurationThreshold(Duration.ofSeconds(2))
.build();
}
// Usage with annotation
@CircuitBreaker(name = "payment", fallbackMethod = "paymentFallback")
public PaymentResult charge(ChargeRequest req) {
return paymentClient.charge(req);
}
private PaymentResult paymentFallback(ChargeRequest req, Throwable t) {
log.warn("Payment CB open, falling back", t);
return PaymentResult.pending(req.getOrderId()); // queue for async retry
}Count-based: tracks the last N calls. Simple but can lag — if traffic drops to 1 call/minute, "last 10 calls" might span 10 minutes and miss a recent burst of failures. Time-based: tracks calls in the last N seconds. Reacts to recent failures regardless of traffic volume. I use time-based for high-traffic services.
Setting minimumNumberOfCalls too low — or not setting it — causes the circuit to open on the very first failure after a restart. Your service just deployed, the downstream hasn't responded to the first probe yet, and the circuit opens immediately. Always set a minimum of 5–10 calls before the failure rate calculation kicks in.
Also: circuit breaker state transitions should be alerted on, not just logged. A circuit opening in production is a signal that something is wrong downstream. Wire state transition events to your alerting system:
registry.circuitBreaker("payment")
.getEventPublisher()
.onStateTransition(e -> metrics.recordCbTransition(e.getStateTransition()))
.onCallNotPermitted(e -> metrics.incrementCbRejected());Naive retry — retry immediately on failure, N times — makes things worse. When a service is struggling, all failing clients retrying immediately flood it with the same load that caused the failure. The right approach is exponential backoff with jitter: each attempt waits longer than the last, and randomisation ensures clients don't all retry at the same instant.
Exponential backoff: wait = base × multiplier^attempt (e.g. 100ms, 200ms, 400ms, 800ms). Without jitter, all retrying clients still synchronise on those exact intervals — a wave at 100ms, a wave at 200ms, etc. Jitter randomises the wait within a range, spreading load across time.
# application.yml — correct retry config
resilience4j:
retry:
instances:
payment:
maxAttempts: 4
waitDuration: 100ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
enableRandomizedWait: true
randomizedWaitFactor: 0.5 # ±50% jitter → 50–150ms, 100–300ms, 200–600ms
retryExceptions:
- java.net.ConnectException
- java.net.SocketTimeoutException
- org.springframework.web.client.HttpServerErrorException$ServiceUnavailable
ignoreExceptions:
- com.example.BadRequestException # 400 — never retry client errors
- com.example.UnauthorizedException # 401 — retrying won't helpRetry is only safe on idempotent operations. A GET is idempotent — retrying it is always safe. A POST /orders that creates an order is not — a successful first call whose response was lost followed by a retry creates a duplicate order.
The fix: idempotency keys. The client generates a unique key (UUID) for each logical operation and sends it as a header (X-Idempotency-Key). The server stores processed keys and deduplicates:
@PostMapping("/orders")
public ResponseEntity<Order> createOrder(
@RequestHeader("X-Idempotency-Key") String idempotencyKey,
@RequestBody CreateOrderRequest req) {
// Check if already processed
Optional<Order> existing = idempotencyStore.find(idempotencyKey);
if (existing.isPresent()) return ResponseEntity.ok(existing.get());
Order order = orderService.create(req);
idempotencyStore.save(idempotencyKey, order, Duration.ofHours(24));
return ResponseEntity.status(201).body(order);
}The interaction between @Retry and @CircuitBreaker annotation order matters critically. If @Retry is the outermost annotation, it retries even when the circuit is OPEN — three pointless fast-fails before the fallback fires. The circuit breaker must be outermost:
// WRONG — retry wraps circuit breaker; retries fire even when circuit OPEN
@Retry(name = "payment")
@CircuitBreaker(name = "payment", fallbackMethod = "fallback")
public PaymentResult charge(ChargeRequest req) { ... }
// CORRECT — circuit breaker outermost; when OPEN, fails immediately, no retry
@CircuitBreaker(name = "payment", fallbackMethod = "fallback")
@Retry(name = "payment")
public PaymentResult charge(ChargeRequest req) { ... }The bulkhead pattern isolates resources so that a failure in one area can't exhaust resources for the whole system — named after the watertight compartments in a ship hull. In practice: allocate a separate, size-limited thread pool per downstream dependency. If the recommendation service becomes slow and blocks all 20 of its threads, your inventory and payment calls are completely unaffected — they have their own pools.
Thread-pool bulkhead: runs the call in a dedicated thread pool. The calling thread is free while waiting. Strong isolation — a blocked call doesn't tie up any application thread. Best for I/O-bound calls (HTTP, DB).
Semaphore bulkhead: limits concurrent calls using a semaphore on the same thread. No extra threads, lower overhead. If the semaphore is full, the call is immediately rejected. Best for quick calls or reactive stacks where thread-pool overhead is unacceptable.
# application.yml — thread-pool bulkhead
resilience4j:
thread-pool-bulkhead:
instances:
recommendations:
coreThreadPoolSize: 10
maxThreadPoolSize: 20
queueCapacity: 5 # beyond this, BulkheadFullException → fallback fires
keepAliveDuration: 20ms@Service
class ProductPageService {
// Recommendations: optional, can degrade. Runs in isolated pool.
@Bulkhead(name = "recommendations",
type = Bulkhead.Type.THREADPOOL,
fallbackMethod = "emptyRecommendations")
public CompletableFuture<List<Product>> getRecommendations(String productId) {
return CompletableFuture.completedFuture(recommendationClient.get(productId));
}
public CompletableFuture<List<Product>> emptyRecommendations(
String id, Throwable t) {
return CompletableFuture.completedFuture(List.of());
}
}Bulkhead sizing is not obvious and must be profiled under load. Too small → unnecessary rejections that degrade the user experience even when the downstream is healthy. Too large → insufficient protection; slow downstream still exhausts the pool and causes cascading slowness.
Sizing approach: measure the 99th percentile response time of the downstream under normal load. Estimate peak concurrent calls to that service. Pool size ≈ peakRPS × p99LatencySeconds (Little's Law). Add 20% headroom. If the downstream's p99 is 200ms and you have 50 req/s, you need roughly 10 threads (50 × 0.2 = 10). Set maxThreadPoolSize to 15 for headroom.
Every external call — HTTP, DB query, cache lookup — must have a timeout. Without one, a single slow response can hold a thread indefinitely. But timeouts alone are not enough: you need to propagate the remaining budget across service hops, not restart the timer at each hop. That's deadline propagation.
Two distinct timeouts to configure on every HTTP client:
// WebClient with timeout — recommended for new code
@Bean
public WebClient paymentWebClient() {
HttpClient httpClient = HttpClient.create()
.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 2000)
.responseTimeout(Duration.ofMillis(3000));
return WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(httpClient))
.baseUrl("https://payment-service")
.build();
}Resilience4j TimeLimiter wraps a CompletableFuture and cancels it if it exceeds the budget:
@TimeLimiter(name = "payment", fallbackMethod = "timeoutFallback")
@CircuitBreaker(name = "payment")
public CompletableFuture<PaymentResult> charge(ChargeRequest req) {
return CompletableFuture.supplyAsync(() -> paymentClient.charge(req));
}A timeout at each service hop is relative: "wait 5 seconds from now". If you have three serial hops each with a 5s timeout, the end-to-end latency can reach 15s even though your SLA is 10s.
A deadline is absolute: "the whole request must complete by T+500ms". Pass the remaining budget as a header. Each downstream service checks: if the deadline has already passed, fail immediately instead of doing any work.
// Outbound interceptor: propagate remaining deadline
public ClientHttpResponse intercept(HttpRequest request, ...) {
Instant deadline = DeadlineContext.current();
if (deadline != null) {
request.getHeaders().set("X-Request-Deadline",
String.valueOf(deadline.toEpochMilli()));
}
return execution.execute(request, body);
}
// Inbound filter: check deadline before processing
String deadlineHeader = request.getHeader("X-Request-Deadline");
if (deadlineHeader != null) {
Instant deadline = Instant.ofEpochMilli(Long.parseLong(deadlineHeader));
if (Instant.now().isAfter(deadline)) {
throw new DeadlineExceededException("Request deadline already passed");
}
}Read timeout without a circuit breaker: at 1,000 req/s with a 5s read timeout, you can accumulate 5,000 in-flight threads all waiting on a dead service. When the service comes back, all 5,000 complete at once — a thundering herd of responses that can overwhelm both services simultaneously. Always pair timeouts with circuit breakers.
Rate limiting protects your service from being overwhelmed and enforces fair usage quotas. The key design decisions are: which algorithm, where to enforce it (per-user, per-IP, global), and whether limits are per-instance or distributed across all pods. Forgetting that last point is the most common production mistake.
Token bucket (most common): a bucket holds N tokens, refilled at R tokens/second. Each request consumes one token. When empty, requests are rejected. Allows bursts up to bucket size. Most natural for API rate limiting.
Fixed window: count requests in a 1-second (or 1-minute) window, reset at the boundary. Simple but has a boundary burst problem: 100 requests in the last 50ms of window 1 + 100 requests in the first 50ms of window 2 = 200 requests in 100ms despite a "100/window" limit.
Sliding window: count requests in the last N seconds (continuously, not a fixed boundary). Eliminates the boundary burst. Slightly more expensive to compute — use a Redis Sorted Set with score = timestamp.
Leaky bucket: requests enter a queue, processed at a fixed output rate. No bursts allowed at all. Predictable throughput on the output side; useful for rate-smoothing rather than rate-limiting.
Resilience4j's built-in RateLimiter is in-memory — each pod has its own independent limit. With 10 pods each at 100 req/s, your effective limit is 1,000 req/s. For true distributed rate limiting, use Redis with an atomic Lua script:
-- Lua script: atomic sliding-window rate limit
local key = KEYS[1]
local now = tonumber(ARGV[1])
local window = tonumber(ARGV[2]) -- window in ms
local limit = tonumber(ARGV[3])
local windowStart = now - window
redis.call('ZREMRANGEBYSCORE', key, 0, windowStart)
local count = redis.call('ZCARD', key)
if count < limit then
redis.call('ZADD', key, now, now .. '-' .. math.random())
redis.call('EXPIRE', key, math.ceil(window / 1000) + 1)
return 1 -- allowed
end
return 0 -- rejectedOr use Bucket4j with Redis backend — a production-grade Java library that implements token bucket with distributed state:
@Bean
public BucketProxyManager<String> bucketManager(RedissonClient redisson) {
ProxyManager<String> proxyManager =
Bucket4jRedisson.casBasedBuilder(redisson).build();
BucketConfiguration config = BucketConfiguration.builder()
.addLimit(Bandwidth.classic(100, Refill.intervally(100, Duration.ofSeconds(1))))
.build();
return new BucketProxyManager<>(proxyManager, () -> config);
}
// Per-user rate limiting
public boolean tryConsume(String userId) {
return bucketManager.getBucket(userId).tryConsume(1);
}The in-memory vs distributed mistake is the most common. A team adds @RateLimiter from Resilience4j in a Kubernetes deployment with 15 pods, sets the limit to 50 req/s, and wonders why the downstream is still seeing 750 req/s. Always verify whether your rate limiter is per-pod or per-service before deploying to multi-instance environments.
A fallback provides an alternative response when a call fails, allowing your service to degrade gracefully rather than returning an error. But a fallback that silently hides failures, adds its own load, or masks on-call alerts is more dangerous than returning an explicit error. The rule: every fallback must emit metrics and trigger alerts — a fallback firing is a symptom, not a resolution.
Static default. Return a hardcoded value or empty collection. Simplest. Use when the feature is non-critical and an empty state is safe (e.g. empty recommendations list, default configuration).
Cached fallback. Return the last-known-good value from a local or Redis cache. Use for read APIs where slightly stale data is acceptable (e.g. product catalog, exchange rates).
Degraded response. Return partial data — skip the failing component but include everything else. Use when the failing part is supplementary (e.g. page loads without personalisation if the recommendation service is down).
Fail-fast with clear error. Return an immediate, meaningful error. Use when the failing component is critical and no alternative exists (payment processing, authentication). Don't pretend to succeed when you haven't.
@CircuitBreaker(name = "inventory", fallbackMethod = "cachedInventory")
public Inventory checkInventory(String productId) {
return inventoryClient.check(productId);
}
private Inventory cachedInventory(String productId, Throwable t) {
metrics.increment("inventory.fallback"); // always metric it
log.warn("Inventory CB open for {}, serving cached", productId);
Inventory cached = cache.get("inventory:" + productId);
if (cached != null) return cached;
// No cache available — fail explicitly rather than silently returning wrong data
throw new ServiceUnavailableException("Inventory service unavailable");
}Silent data loss. Fallback returns an empty list, calling code interprets "no items" as the real state and deletes records. This has caused real production data loss incidents.
Cascading fallback load. Your service calls B, which is slow. Fallback calls C instead. C is also under higher-than-normal load (everyone is falling back to it). Now you've amplified load on C, causing it to slow down too — a cascade you created by trying to avoid one.
Masking real failures. If your fallback always succeeds, on-call is never paged. The underlying service is down, users are getting degraded experience, and no one knows. This is why fallbacks must emit metrics and alerts even when they "succeed".
Resilience4j integrates with Spring Boot via resilience4j-spring-boot3 and Spring AOP. Configuration is in application.yml, annotations are applied to beans, and metrics are auto-exported to Micrometer/Prometheus. The three most important things to get right: AOP proxy awareness, annotation ordering, and wiring state transitions to alerts.
<!-- pom.xml -->
<dependency>
<groupId>io.github.resilience4j</groupId>
<artifactId>resilience4j-spring-boot3</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-aop</artifactId> <!-- REQUIRED -->
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency># application.yml — full config for a payment service call
resilience4j:
circuitbreaker:
instances:
payment:
slidingWindowSize: 10
minimumNumberOfCalls: 5
failureRateThreshold: 50
waitDurationInOpenState: 30s
permittedNumberOfCallsInHalfOpenState: 3
slowCallRateThreshold: 80
slowCallDurationThreshold: 2s
registerHealthIndicator: true # shows in /actuator/health
retry:
instances:
payment:
maxAttempts: 3
waitDuration: 200ms
enableExponentialBackoff: true
exponentialBackoffMultiplier: 2
enableRandomizedWait: true
randomizedWaitFactor: 0.5
timelimiter:
instances:
payment:
timeoutDuration: 3s// Wire state transitions to your alerting system
@PostConstruct
public void registerCbListeners() {
registry.circuitBreaker("payment").getEventPublisher()
.onStateTransition(event -> {
String transition = event.getStateTransition().name();
meterRegistry.counter("cb.state.transition",
"name", "payment", "transition", transition).increment();
if (event.getStateTransition().getToState() == State.OPEN) {
alerting.fire("Circuit breaker OPENED: payment service");
}
});
}Key Prometheus metrics auto-registered by Resilience4j:
resilience4j_circuitbreaker_state{name="payment"} — current state (0=CLOSED, 1=OPEN, 2=HALF_OPEN)resilience4j_circuitbreaker_calls_total{kind="failed|successful|not_permitted"}resilience4j_retry_calls_total{kind="successful_with_retry|failed_with_retry"}Actuator endpoints: /actuator/circuitbreakers, /actuator/retries, /actuator/health (if registerHealthIndicator: true).
Resilience4j annotations use Spring AOP — the same self-invocation problem as @Transactional. Calling a @CircuitBreaker method from within the same bean via this.method() bypasses the proxy and no circuit breaking occurs. The call goes directly to the real method. Fix: always call through the injected Spring proxy (use the autowired bean reference, not this).
This is a structured response that tests whether you think about resilience as a system property, not just a library configuration. The answer covers detection, containment, user impact, investigation, recovery, and prevention.
Step 1 — Detect. Alert fires on resilience4j_circuitbreaker_state{name="payment"} == 1 (circuit OPEN) or on rising error rate from your RED dashboard. If monitoring is missing, the first signal may be user reports — which means detection is too slow.
Step 2 — Contain. Circuit breaker fast-fails new requests to the failing service — no need to intervene. Bulkhead ensures other dependencies aren't starved. Check: is the circuit breaker actually open, or are requests still flowing through and timing out?
Step 3 — Assess user impact. Is this a critical path (payment, auth) or optional feature (recommendations, analytics)? If critical: escalate, open incident, communicate SLA impact. If optional: fallback is serving degraded content — acceptable temporarily.
Step 4 — Investigate. Use distributed traces (filter by high duration and error status in your tracing tool) to confirm which span is failing. Check the downstream service's own health endpoint, logs, and metrics. Common causes: DB connection pool exhaustion, memory pressure, bad deploy, dependency of a dependency failing.
Step 5 — Communicate. If the downstream is owned by another team: open incident, tag the owners, share trace IDs and error samples. Set realistic SLA expectations with users.
Step 6 — Recover. Circuit transitions to HALF_OPEN after wait duration. Monitor probe calls — watch resilience4j_circuitbreaker_state. If probe success rate recovers, circuit closes automatically. If not, it reopens and waits again.
Step 7 — Post-mortem. Was the timeout configured correctly? Did the bulkhead prevent cascade? Did the fallback behave correctly? Did alerts fire before users noticed? What would reduce MTTD/MTTR next time?