Q1 · Fundamentals
Explain ACID properties. What does each one actually guarantee at the DB level?
Atomicity
All operations in a transaction succeed or all are rolled back. No partial writes.
How it works: DB maintains a write-ahead log (WAL). On commit, changes are flushed to WAL first. On rollback or crash, WAL allows undoing partial work.
-- Transfer $100: both statements must succeed or both rolled back
BEGIN;
UPDATE account SET balance = balance - 100 WHERE id = 1;
UPDATE account SET balance = balance + 100 WHERE id = 2;
COMMIT;
-- If UPDATE #2 fails: UPDATE #1 is reversed. Balance never disappears.
Consistency
A transaction moves the database from one valid state to another. Constraints (NOT NULL, UNIQUE, CHECK, FK) are enforced at commit.
Note: Consistency is partly the DB's job (constraints) and partly the application's job (business rules). The DB can't enforce "balance ≥ 0" unless you add a CHECK constraint.
Isolation
Concurrent transactions don't see each other's intermediate states. The degree depends on the isolation level (see Q2).
Without isolation: two withdrawals running simultaneously could both read the same balance, both subtract, and result in insufficient funds going negative.
Durability
Once committed, data survives crashes. Guaranteed by flushing the WAL to disk before acknowledging the commit. Even if the server crashes a millisecond after COMMIT, the data is safe.
Interview drill-down — expect these follow-ups:
L1 → "What is BASE and how does it relate to ACID?"
BASE (Basically Available, Soft state, Eventually consistent) is the consistency model for many distributed NoSQL systems. Trade strong consistency for availability and partition tolerance. A Cassandra write is visible immediately to the writing node but may take milliseconds to replicate to others — reads during that window may return stale data. Fine for shopping carts, social feeds, recommendation counts. Absolutely not for financial transactions or inventory deduction where consistency violations mean real money lost.
L2 → "Durability says data survives crashes after COMMIT. But what if the disk has a delayed write and power fails between the WAL write and the actual data file write?"
PostgreSQL (and most RDBMS) use write-ahead logging precisely for this: the WAL is fsync'd to disk before the COMMIT is acknowledged to the client. On crash recovery, the DB replays the WAL to reconstruct any data file writes that didn't complete. This is why fsync=on is critical — with fsync=off the WAL write is asynchronous and data loss is possible. Never disable fsync in production, even for "performance".
L3 → "Your PostgreSQL DB is using synchronous_commit=off for performance. A developer says ACID durability is broken. Is he right?"
Partially right. synchronous_commit=off means PostgreSQL acknowledges the COMMIT before the WAL is flushed to disk. In a crash, the last few transactions (typically within ~200ms) may be lost. Atomicity, Consistency, and Isolation are still preserved — only Durability is weakened (it becomes "probably durable"). This is an explicit trade-off: acceptable for high-volume metrics or audit logs where losing 200ms of data is tolerable. Never for financial transactions or anything where losing a committed row would cause a business integrity violation.
Trap answer to avoid: "Consistency in ACID means the data is always consistent across all nodes." — That's distributed consistency (CAP theorem). ACID Consistency means a transaction transitions the DB from one valid state (satisfying all constraints) to another valid state. It's about constraint enforcement, not replication. Conflating the two shows confusion between single-node ACID and distributed system consistency models.
Q2 · Isolation
Explain transaction isolation levels. Which phenomena do they prevent, and when do you use each?
The three problems isolation solves
- Dirty read: reading uncommitted data from another transaction (that might roll back).
- Non-repeatable read: re-reading the same row gets different values (another transaction committed a change between reads).
- Phantom read: re-running a query returns different rows (another transaction inserted/deleted rows matching the WHERE clause).
Isolation levels
| Level | Dirty Read | Non-Repeatable | Phantom | Use when |
| READ UNCOMMITTED | Possible | Possible | Possible | Never in production |
| READ COMMITTED | Prevented | Possible | Possible | Default (PostgreSQL, Oracle) |
| REPEATABLE READ | Prevented | Prevented | Possible* | Reports, consistent multi-read |
| SERIALIZABLE | Prevented | Prevented | Prevented | Critical financial ops |
* PostgreSQL's REPEATABLE READ also prevents phantoms (uses MVCC snapshots).
Practical choices
- READ COMMITTED: most services — reads see only committed data, acceptable for normal queries.
- REPEATABLE READ: reports that read the same data multiple times and need consistency.
- SERIALIZABLE: financial transfers, inventory allocation — where phantom reads would cause real business errors. Use sparingly — lower throughput.
In JPA
@Transactional(isolation = Isolation.READ_COMMITTED) // default
@Transactional(isolation = Isolation.SERIALIZABLE) // for critical ops
Interview drill-down — expect these follow-ups:
L1 → "What is MVCC and how does it let readers and writers not block each other?"
Multi-Version Concurrency Control: instead of locking a row when writing, PostgreSQL creates a new version of the row (with a transaction ID stamp) while keeping the old version. Readers get a snapshot of the DB as of their transaction start time — they always read the correct old version even while a writer is updating. No read-lock needed. Writers don't block readers; readers don't block writers. The trade-off: old row versions accumulate (called "dead tuples") until VACUUM reclaims them.
L2 → "REPEATABLE READ in PostgreSQL prevents phantom reads too, unlike the SQL standard. How?"
PostgreSQL's REPEATABLE READ takes a consistent snapshot at the start of the transaction, not at each statement. Any rows inserted by other transactions after your snapshot point are invisible to you — even if they match your WHERE clause. This is stronger than the SQL standard's REPEATABLE READ, which only prevents re-reading the same row from changing, not new rows appearing. MySQL InnoDB achieves the same with gap locks, which is heavier.
L3 → "You need SERIALIZABLE isolation for a financial transfer but it's causing throughput to drop 60%. How do you reduce the cost?"
PostgreSQL's SSI (Serializable Snapshot Isolation) is optimistic — transactions proceed without blocking, but any serialization conflict causes one to abort and retry. To reduce overhead: (1) Keep transactions short — fewer conflict opportunities. (2) Use explicit pessimistic locking (SELECT FOR UPDATE) on just the rows you need instead of full SERIALIZABLE — gives serial behaviour for the specific rows at lower cost. (3) Structure the transaction to read before write in a single pass — avoids creating a serialization dependency. For a balance transfer, SELECT FOR UPDATE on both accounts in consistent order is often better than full SERIALIZABLE.
Trap answer to avoid: "I use SERIALIZABLE everywhere to be safe." — SERIALIZABLE causes transaction aborts and retries under contention (SSI) or significant lock overhead. Using it everywhere on a high-throughput system will cause cascading retries and degraded performance. Know which operations actually need it (financial writes, inventory allocation) and use READ COMMITTED for everything else.
Q3 · Indexing
Explain database indexing — types, composite index rules, and covering indexes.
What an index does
An index is a separate data structure (usually B-tree) that maps column values to row locations. Speeds up reads at the cost of write overhead (every INSERT/UPDATE/DELETE must maintain the index).
Index types
| Type | Use for |
| B-tree (default) | Range queries, equality, ORDER BY, <, >, BETWEEN |
| Hash | Equality only (=). Faster point lookups, no range support |
| Composite | Queries filtering on multiple columns |
| Partial/Filtered | Index a subset of rows (WHERE status = 'ACTIVE') |
| Covering | Includes all columns a query needs — avoids table lookup entirely |
Composite index — leftmost prefix rule
CREATE INDEX idx_order ON orders(customer_id, status, created_at);
-- USES the index (leftmost prefix matched):
WHERE customer_id = 5
WHERE customer_id = 5 AND status = 'PENDING'
WHERE customer_id = 5 AND status = 'PENDING' AND created_at > '2024-01-01'
-- Does NOT use the index (skips customer_id):
WHERE status = 'PENDING'
WHERE created_at > '2024-01-01'
Rule: Put high-cardinality, equality-filtered columns first. Range-filtered columns last.
Covering index
-- Query only needs customer_id and order_total:
SELECT customer_id, order_total FROM orders WHERE customer_id = 5;
-- Covering index: all needed columns are IN the index — no table access:
CREATE INDEX idx_covering ON orders(customer_id) INCLUDE (order_total);
-- "Index-only scan" — significantly faster on high-volume reads.
Interview drill-down — expect these follow-ups:
L1 → "You have a composite index on (customer_id, status, created_at). A query filters on customer_id and created_at but NOT status. Does it use the index?"
Partially. It uses the index on customer_id (leftmost prefix match) but cannot skip status to reach created_at in the B-tree structure. PostgreSQL may use the index for the customer_id filter, then do a filter (not an index condition) on created_at — meaning it scans all rows for that customer regardless of created_at. To fix: create a separate index on (customer_id, created_at), or use a partial index if status has low cardinality (e.g., only 'PENDING' rows need this query path).
L2 → "An index exists on the email column but the query SELECT * FROM users WHERE LOWER(email) = LOWER(:email) does a full table scan. Why?"
Functions applied to indexed columns prevent index use — the B-tree is built on the raw email value, not on LOWER(email). Fix: create a functional index: CREATE INDEX idx_email_lower ON users (LOWER(email)). Alternatively, enforce lowercase at write time so no function is needed at query time. This is a common gotcha with case-insensitive searches.
L3 → "Your orders table has 500 million rows. VACUUM can't keep up with dead tuple accumulation and table bloat is at 40%. What do you do?"
Short term: run VACUUM FULL (rewrites the table, reclaims all bloat) — but it takes an exclusive lock and blocks all access. Use pg_repack instead for online, lock-minimal bloat reduction. Long term: (1) Tune autovacuum aggressively: autovacuum_vacuum_scale_factor=0.01 (vacuum when 1% of rows are dead, not the default 20%). (2) Partition the table by date range — VACUUM operates per-partition, much faster. (3) Move to time-series storage or archival tables for old data if the write rate is so high that VACUUM can never catch up.
Trap answer to avoid: "More indexes = better performance." — Each index slows down INSERT, UPDATE, and DELETE because all indexes must be maintained. On write-heavy tables (audit logs, event streams), unnecessary indexes cause measurable write throughput degradation. Index only the columns your actual query patterns use, validated by EXPLAIN ANALYZE.
Q4 · Query Optimisation
How do you read and use an execution plan to fix a slow query?
EXPLAIN ANALYZE
EXPLAIN ANALYZE
SELECT o.id, c.name, o.total
FROM orders o
JOIN customers c ON o.customer_id = c.id
WHERE o.status = 'PENDING' AND o.created_at > NOW() - INTERVAL '7 days';
What to look for
| Symptom | What it means | Fix |
| Seq Scan on large table | No index used | Add index on filtered columns |
| Nested Loop with many rows | No join index | Index the join column |
| High "rows estimated" vs "actual" | Stale statistics | ANALYZE table_name |
| Sort on large dataset | No index for ORDER BY | Add index matching ORDER BY |
| Bitmap Heap Scan | Multiple indexes being combined | Often fine; composite index may help |
Common optimisation checklist
Interview drill-down — expect these follow-ups:
L1 → "What is a query hint and when would you use one?"
A directive to override the query planner's choice (e.g., force a specific index with /*+ INDEX(orders idx_status) */ in Oracle, or SET enable_seqscan=off in PostgreSQL). Use only as a last resort when: (1) statistics are accurate and fresh, (2) the right index exists, (3) you've verified with EXPLAIN ANALYZE that the planner is making a suboptimal choice. Query hints are fragile — they're hardcoded for a specific data distribution and will hurt performance if data characteristics change. Document why you added the hint and add it to your review checklist.
L2 → "EXPLAIN ANALYZE shows 'rows estimated: 1, actual: 50,000' — the planner picked a bad plan because of stale stats. How do you fix this permanently?"
Run ANALYZE on the table to update statistics. For tables with skewed data distributions (e.g., 90% of rows have status='CLOSED', 10% 'OPEN'), default statistics may miss the skew. Increase the statistics target for that column: ALTER TABLE orders ALTER COLUMN status SET STATISTICS 500 (default is 100). For columns with very high cardinality or skewed values, consider extended statistics: CREATE STATISTICS orders_status_customer ON status, customer_id FROM orders — teaches the planner about correlation between columns.
L3 → "You have a JOIN between orders (500M rows) and customers (1M rows). The planner chooses a Nested Loop but it's slow. What join strategy do you want and why?"
For large table joins, Hash Join is usually better: build a hash table from the smaller table (customers), then probe it for each row in the larger table (orders). O(N+M) vs Nested Loop's O(N×M). Nested Loop is only fast when the inner table has an index and the outer result set is small. If the planner chooses Nested Loop for a large outer set, it likely has bad row count estimates. Fix: ANALYZE both tables, check join column statistics. If still wrong, increase work_mem (more memory for hash joins) or use SET enable_nestloop=off for diagnosis.
Trap answer to avoid: "EXPLAIN shows the query is slow — I'll add an index on every WHERE column." — EXPLAIN (without ANALYZE) shows the planner's estimated plan, not the actual execution. Always use EXPLAIN ANALYZE to see real row counts and actual times. And adding indexes to every filtered column ignores composite index leftmost prefix rules — you may add an index that the planner never uses while slowing down writes.
Q5 · JPA / Hibernate
What is the N+1 query problem? How do you detect and fix it?
The problem
// 1 query to load 100 orders:
List<Order> orders = orderRepo.findAll();
// Then for EACH order, a separate query for its items (lazy loading):
orders.forEach(o -> o.getItems().size()); // triggers N more queries
// Result: 1 + 100 = 101 queries. With 1000 orders: 1001 queries.
Detection
- Enable Hibernate SQL logging:
spring.jpa.show-sql=true and spring.jpa.properties.hibernate.format_sql=true.
- Use Hibernate Statistics or p6spy to count queries per request.
- Symptom in production: fast single-object endpoints, very slow list endpoints.
Fix 1 — JOIN FETCH
@Query("SELECT o FROM Order o JOIN FETCH o.items WHERE o.status = :status")
List<Order> findWithItems(@Param("status") String status);
// 1 query with JOIN — all data fetched at once
Fix 2 — @EntityGraph
@EntityGraph(attributePaths = {"items", "customer"})
List<Order> findByStatus(String status);
// Declarative fetch plan — no JPQL needed
Fix 3 — DTO projection (best for read-only)
@Query("SELECT new com.example.OrderSummary(o.id, o.total, c.name) FROM Order o JOIN o.customer c")
List<OrderSummary> getOrderSummaries();
// Returns exactly what you need, no entity overhead, no lazy loading
Fix 4 — @BatchSize
@BatchSize(size = 50)
@OneToMany List<Item> items;
// Hibernate loads items in batches of 50 → 1 + ceil(N/50) queries instead of 1+N
Interview drill-down — expect these follow-ups:
L1 → "What is the Open Session in View anti-pattern and why is it dangerous?"
Spring's open-in-view (enabled by default) keeps the Hibernate session open through the HTTP request lifecycle — including the view/serialization layer. This means lazy loading works in @RestController @ResponseBody serialization. Dangerous because: (1) hidden N+1 queries fire silently during JSON serialization — hard to detect, (2) the DB connection is held for the full HTTP request duration including serialization time, causing connection pool exhaustion under load, (3) queries happen in the web layer, violating separation of concerns. Fix: spring.jpa.open-in-view=false and eager-load all needed data in the service layer.
L2 → "You disable open-in-view and now get LazyInitializationException on some endpoints. How do you fix these systematically?"
For each failing endpoint: (1) Identify the association being lazily loaded (the exception stack trace shows which getter). (2) Add JOIN FETCH or @EntityGraph to the repository query for that use case. (3) Better — use DTO projections that only fetch the fields the endpoint actually needs: @Query("SELECT new OrderDTO(o.id, o.total, c.name) FROM Order o JOIN o.customer c"). DTOs also prevent accidentally serializing sensitive fields. The LazyInitializationException failures are useful — they reveal exactly which associations were being silently loaded via open-in-view.
L3 → "Your service method calls a repository, gets 1000 entities, then calls a remote API per entity. This causes connection pool exhaustion. Why and how do you fix it?"
The @Transactional method holds the DB connection from start to finish. The remote API calls (potentially 90s total) happen with the connection held — 1000 concurrent requests × 90s = connection pool exhausted in seconds. Fix: split the transaction. Load the entities in a short @Transactional scope (commits + releases connection). Then make the remote API calls outside the transaction. Then write results in a second short @Transactional scope. This is the "fat transaction anti-pattern" — never make external calls inside a transaction that holds a DB connection.
Trap answer to avoid: "N+1 is fixed by using eager loading (FetchType.EAGER) on all associations." — EAGER loading fetches all associated entities for every query regardless of whether you need them. A findAll() on Orders with EAGER items now always loads all items for every order, even when you only need order IDs. This often creates worse performance than N+1 and causes OutOfMemoryErrors on large datasets. Use targeted JOIN FETCH or @EntityGraph on specific queries that need the data.
Q6 · Locking
Optimistic vs pessimistic locking — when do you use each?
Pessimistic locking
Lock the row immediately when reading. Other transactions must wait.
-- Database level:
SELECT * FROM account WHERE id = 1 FOR UPDATE; -- row locked until COMMIT
-- JPA:
Account acc = em.find(Account.class, id, LockModeType.PESSIMISTIC_WRITE);
Use when: high contention and cost of failure is high (balance deduction, seat booking, inventory reservation). Lock guarantees you won't conflict.
Optimistic locking
No lock held. Add a version column. At commit, check that version hasn't changed — if it has, another transaction beat you, retry.
@Entity
class Account {
@Version
private int version; // auto-incremented by Hibernate on every update
}
-- Hibernate generates:
UPDATE account SET balance = 900, version = 2 WHERE id = 1 AND version = 1;
-- If 0 rows updated → version changed → OptimisticLockException → retry
Use when: low contention and retry is cheap (profile updates, viewing data). High throughput — no blocking.
Decision guide
| Pessimistic | Optimistic |
| Contention | High | Low |
| Cost of conflict | High (money, seats) | Low (retry is cheap) |
| Throughput | Lower (blocking) | Higher (no locks) |
| Deadlock risk | Yes | No |
For microservices: in-process locks (synchronized) don't work across multiple pods. Use DB-level locking or a distributed lock (Redis Redlock) for cross-instance coordination.
Interview drill-down — expect these follow-ups:
L1 → "How do you handle an OptimisticLockException?"
Catch it, reload the entity with fresh data, re-apply the business logic, and retry. Spring Retry: @Retryable(retryFor = OptimisticLockingFailureException.class, maxAttempts = 3, backoff = @Backoff(delay = 50, multiplier = 2)). Cap retries to avoid infinite loops. Important: the entity must be reloaded — not re-used from the old reference, which still has the stale version. The retry logic should be in the service method, not the repository, since you need to re-execute the full business logic, not just the save.
L2 → "You use optimistic locking. Two users edit the same product simultaneously. User B submits first, User A submits a second later. User A gets an OptimisticLockException. What is the UX-correct response?"
Don't silently retry — User A's changes would silently overwrite User B's. Instead: catch the OptimisticLockException, reload the current state, diff it against User A's submitted changes, and present a merge conflict UI: "These fields were changed by another user while you were editing: [price: $50 → $45]. Please review and resubmit." Silent retries are correct for machine-generated updates; interactive user edits require human merge conflict resolution.
L3 → "You use pessimistic locking (SELECT FOR UPDATE). Under high concurrency, lock wait times spike to 5 seconds. How do you diagnose and reduce this?"
Diagnose: pg_stat_activity shows transactions waiting for locks; pg_locks shows who holds what. SET lock_timeout='500ms' to fail fast instead of waiting 5s. Root cause is usually: (1) transactions holding locks while doing slow work (external API calls inside the transaction — fix: move external calls outside), (2) lock granularity too coarse (locking the entire account when you only need one row), (3) unnecessary read locks (using PESSIMISTIC_WRITE when PESSIMISTIC_READ suffices for your use case). Consider advisory locks for coordinating at the application level without row-level contention.
Trap answer to avoid: "Optimistic locking is always better because it has no blocking." — Optimistic locking fails and retries under high contention. If 100 threads compete for the same entity, 99 of them will get OptimisticLockException and retry — potentially many times. Under high contention, optimistic locking can be worse than pessimistic because you do the work then throw it away. Choose based on actual contention: low contention → optimistic; high contention → pessimistic.
Q7 · Concurrency
What causes a database deadlock? How do you prevent and handle it?
What is a deadlock
Two (or more) transactions each hold a lock the other needs. Neither can proceed. The DB detects this via a wait-for graph and kills one transaction (the "victim") to break the cycle.
-- Transaction A: Transaction B:
BEGIN; BEGIN;
UPDATE account SET ... WHERE id=1; UPDATE account SET ... WHERE id=2;
-- (waits for B's lock on id=2) -- (waits for A's lock on id=1)
UPDATE account SET ... WHERE id=2; UPDATE account SET ... WHERE id=1;
-- DEADLOCK: DB kills one transaction
Prevention
- Consistent lock ordering: always acquire locks in the same order (e.g., lower account_id first). Eliminates cyclic dependency.
- Keep transactions short: acquire locks, do the work, release quickly. Don't hold locks across external calls or user input.
- Lock timeouts:
SET lock_timeout = '3s' — transaction fails fast instead of waiting indefinitely.
// Always lock in consistent order by account ID:
Long firstId = Math.min(fromId, toId);
Long secondId = Math.max(fromId, toId);
em.find(Account.class, firstId, LockModeType.PESSIMISTIC_WRITE);
em.find(Account.class, secondId, LockModeType.PESSIMISTIC_WRITE);
Handling
// Deadlock = transient error — safe to retry with backoff:
@Retryable(
retryFor = { DeadlockLoserDataAccessException.class },
maxAttempts = 3,
backoff = @Backoff(delay = 100, multiplier = 2)
)
@Transactional
public void transfer(Long from, Long to, BigDecimal amount) { ... }
Interview drill-down — expect these follow-ups:
L1 → "How do you find deadlocks in production PostgreSQL?"
PostgreSQL logs deadlocks at ERROR level: "deadlock detected" with the full wait-for cycle, query text, and PIDs of all involved transactions. Enable log_lock_waits=on and deadlock_timeout=1s to log any lock waits that exceed 1 second before they escalate to deadlocks — this gives you early warning. Use pg_stat_activity to see live waiting queries: SELECT * FROM pg_stat_activity WHERE wait_event_type = 'Lock'. Cross-reference with pg_locks to see which transaction holds what.
L2 → "Your application has a deadlock that only happens under load. Consistent lock ordering should prevent it, but it still fires. What else could cause it?"
Several subtle causes: (1) Foreign key constraint locks — updating a child row acquires a lock on the parent row to check the FK. If two transactions update different child rows whose parents cross-reference each other, a deadlock forms even if application code orders by ID. (2) Index page locks — B-tree page splits under concurrent inserts can cause lock cycles at the index level, not the row level. (3) Trigger side effects — a row update triggers a trigger that modifies another table in a different lock order than the application. Always examine the full lock chain in the PostgreSQL deadlock log, not just your application's SQL.
L3 → "You want zero-deadlock tolerance — if any deadlock occurs it means a bug. How do you enforce this in your CI pipeline?"
Integration tests that run concurrent transactions with known conflict patterns. Use CompletableFuture to execute two @Transactional operations concurrently against a real test database (not H2). Assert that no DeadlockLoserDataAccessException is thrown. Additionally: enable deadlock detection in your test DB with deadlock_timeout=100ms so it catches them quickly. In code review, enforce the rule: multiple-row operations must always acquire locks in the same order (enforce via code review checklist or custom ArchUnit rule that flags methods touching multiple entities in inconsistent order).
Trap answer to avoid: "I use @Transactional(timeout=5) to prevent deadlocks — the transaction will just cancel after 5 seconds." — Transaction timeout doesn't prevent deadlocks; the DB already detects and resolves deadlocks in milliseconds by killing one transaction. Timeout is for hung transactions, not deadlock prevention. Deadlock prevention requires consistent lock ordering. Conflating the two shows a misunderstanding of what deadlock detection vs timeout does.
Q8 · Performance
How does connection pooling work? How do you configure HikariCP for production?
Why connection pooling
Opening a DB connection is expensive (~50–200ms: TCP handshake, authentication, session setup). A pool pre-creates N connections and lends them to threads. Connection acquired in microseconds.
HikariCP key config
spring:
datasource:
hikari:
maximum-pool-size: 20 # max connections to DB
minimum-idle: 5 # keep 5 warm connections always
connection-timeout: 3000 # fail fast if no connection available (3s)
idle-timeout: 600000 # close idle connections after 10min
max-lifetime: 1800000 # recycle connections before DB kills them (30min)
keepalive-time: 30000 # send keepalive query to prevent idle timeout
How to size the pool
HikariCP's advice: pool_size = (core_count * 2) + effective_spindle_count. For most services: 10–20 connections per pod is the sweet spot.
- More connections ≠ more throughput. DB CPU/disk is the bottleneck, not connection count.
- With 10 pods × 20 connections = 200 connections to the DB. Stay within the DB's
max_connections limit.
- Use PgBouncer or RDS Proxy in front of the DB to multiplex connections at scale.
Connection pool exhaustion — a common production issue
// Symptoms: requests timing out, HikariCP timeout exceptions in logs
// Root cause: threads holding connections too long (slow queries, forgotten transactions)
// Debug: monitor pool metrics:
management.metrics.enable.hikari=true
// Grafana: hikaricp_connections_active, hikaricp_connections_pending
// If pending > 0 for more than a second → pool is too small OR queries are too slow
Interview drill-down — expect these follow-ups:
L1 → "What is a connection leak and how do you detect it?"
A code path acquires a DB connection (or opens a @Transactional method) but never releases it — an exception is thrown before the transaction closes, or a connection is obtained directly and not returned to the pool. Symptoms: hikaricp_connections_pending metric growing, eventually connection-timeout exceptions. Detection: set HikariCP leakDetectionThreshold=2000 — if any connection is held longer than 2 seconds without being returned, HikariCP logs a WARNING with the stack trace of where the connection was acquired. Fix the leak at the acquisition site.
L2 → "You have 10 pods each with maximum-pool-size=20. Your PostgreSQL max_connections is 100. What happens and how do you fix it?"
10 × 20 = 200 connection attempts, but PostgreSQL max_connections=100. Connections beyond 100 are rejected with "FATAL: sorry, too many clients already". HikariCP will fail to acquire a connection, and requests will fail with connection-timeout. Fix: (1) Reduce pool size per pod: max 8-9 per pod gives 80-90 total, within the limit. (2) Deploy PgBouncer (connection pooler) in front of PostgreSQL: your 200 app connections map to 100 DB connections — PgBouncer multiplexes them. (3) Use RDS Proxy for managed connection pooling on AWS. Always calculate: pods × pool_size < postgres_max_connections - 5 (reserve for admin connections).
L3 → "HikariCP connection-timeout fires at 3 seconds but your 95th percentile query time is 200ms. Why are connections not being released quickly enough?"
Connections aren't released until the @Transactional boundary closes. If a transaction does: (1) query (200ms) → (2) external HTTP call (2800ms) → (3) return — the connection is held for 3 seconds total. With pool size 20 and 20+ concurrent requests each making a 2.8s external call, the pool exhausts. Fix: never make external calls inside @Transactional. Restructure: query + close transaction, make HTTP call, open new transaction for write. Monitor with hikaricp_connections_active and hikaricp_connections_pending together — pending > 0 means you have a holding problem, not a pool-size problem.
Trap answer to avoid: "I'll increase maximum-pool-size to 200 to handle the load." — Larger pools don't automatically mean better performance. Beyond a certain point (the DB's CPU core count × 2), more connections increase DB server context-switching overhead and degrade throughput. More connections also means more memory pressure on the DB. The right answer is to understand why connections are held long (slow queries, transactions holding connections during external calls) and fix the root cause.
Q9 · Migrations
How do you manage schema migrations in production? What is the expand-contract pattern and how do you deploy schema changes without downtime?
Flyway vs Liquibase — choosing the right tool
| Aspect | Flyway | Liquibase |
| Migration format | SQL (or Java) files only | XML, YAML, JSON, or SQL |
| Rollback | Only in paid Flyway Teams; write undo scripts manually | Built-in rollback support (XML/YAML changesets) |
| Branching/merging | Linear versioning — filename order matters | Changeset IDs — merges cleanly, order is explicit |
| Checksum validation | Yes — changes to applied migrations break startup | Yes — MD5 checksums per changeset |
| Spring Boot integration | spring-boot-starter-flyway — auto-runs at startup | spring-boot-starter-liquibase — same |
| Learning curve | Lower — plain SQL, easy to reason about | Higher — XML/YAML abstraction layer |
| Best fit | Small-to-medium teams, SQL-first | Multi-DB support, complex branching, rollback |
In practice: Most Spring Boot shops use Flyway for its simplicity. Liquibase wins when you need rollback scripts, multiple database dialects, or complex branching workflows (e.g., multiple feature branches that must merge without conflicts).
Flyway fundamentals
-- File naming: V{version}__{description}.sql
-- V1__create_users_table.sql
-- V2__add_email_index.sql
-- R__seed_lookup_data.sql (repeatable — reruns when content changes)
-- Flyway tracks applied migrations in the schema_history table:
-- installed_rank | version | description | type | script | checksum | success
-- CRITICAL: Never modify an applied migration file.
-- Flyway recalculates the checksum at startup and throws:
-- FlywayException: Validate failed: Migration checksum mismatch
-- Fix: create a new migration that corrects the schema.
# application.yml — Spring Boot Flyway config
spring:
flyway:
enabled: true
locations: classpath:db/migration
baseline-on-migrate: false # true only for existing DBs first run
validate-on-migrate: true # checksum check at startup (keep true)
out-of-order: false # true allows V5 to run after V6 (careful)
schemas: public
# In multi-pod deployments, Flyway uses a DB advisory lock
# so only one pod runs migrations even if multiple start simultaneously
The expand-contract pattern — zero-downtime migrations
Zero-downtime deployment means the new application version and the old application version must be able to run against the same database simultaneously during rolling deployment. Most migrations break this if done naively.
Expand-Contract (aka Parallel Change) splits a breaking schema change into three safe phases:
| Phase | DB change | App version running |
| 1. Expand | Add the new column/table (nullable, no constraints yet) | Old app only — ignores new column |
| 2. Migrate | Backfill data into new column | New app writes to BOTH old and new columns |
| 3. Contract | Drop old column/constraint after all old app instances are gone | New app only — reads from new column |
-- Example: renaming column "username" to "display_name"
-- PHASE 1: Deploy V5 migration — add new column (nullable, backward compatible)
ALTER TABLE users ADD COLUMN display_name VARCHAR(255);
-- Old app: reads/writes username (doesn't see display_name)
-- New app: reads display_name if not null, else username (backward compat read)
-- writes to BOTH columns during transition
-- PHASE 2: Deploy V6 migration — backfill
UPDATE users SET display_name = username WHERE display_name IS NULL;
-- PHASE 3: Deploy V7 migration — after all old pods are gone
ALTER TABLE users ALTER COLUMN display_name SET NOT NULL;
ALTER TABLE users DROP COLUMN username;
Never do this: ALTER TABLE users RENAME COLUMN username TO display_name in a single migration during a rolling deploy. Old pods immediately break because they reference a column that no longer exists. Always expand first.
Common migration pitfalls
- Adding NOT NULL without a default — fails instantly on a table with existing rows. Always add the column as nullable, backfill, then add the constraint.
- Adding an index without CONCURRENTLY (PostgreSQL) —
CREATE INDEX takes a full table lock. Use CREATE INDEX CONCURRENTLY — it takes no exclusive lock but cannot run inside a transaction, so it needs a dedicated Flyway migration with -- flyway:nonTransactional.
- Large backfill in one UPDATE — locks rows for the duration, causes timeouts. Batch it:
UPDATE users SET x = y WHERE id BETWEEN 1 AND 10000, repeat in a loop.
- Modifying applied migrations — Flyway's checksum validation catches this at startup, which is good. Fix forward, never rewrite history.
-- Safe index creation in Flyway (PostgreSQL)
-- V8__add_users_email_idx.sql
-- flyway:nonTransactional -- tell Flyway not to wrap in a transaction
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_users_email ON users(email);
Interview drill-down — expect these follow-ups:
L1 → "Multiple pods start simultaneously. Two pods both try to run the same Flyway migration. What happens?"
Flyway prevents this with a database-level advisory lock (in PostgreSQL: pg_advisory_lock). The first pod acquires the lock, runs pending migrations, releases the lock. The second pod acquires the lock, sees that the migration is already applied (checksum matches), and skips it. This is why Flyway's startup lock behavior is important to understand — in slow deployments, other pods may time out waiting for the Flyway lock. Configure spring.flyway.lock-retry-count and lock-retry-interval-millis accordingly, or run migrations as a Kubernetes Job before rolling out the new pods.
L2 → "A Flyway migration ran and corrupted 10,000 rows. You need to roll it back. How?"
Flyway open-source has no automatic rollback. Steps: (1) If you have a recent snapshot/backup, restore — fastest if the table isn't heavily written. (2) Write a new migration (V{n+1}) that is the compensating transaction — reverses the data change. (3) If using Flyway Teams, write a paired undo script (U{n}__description.sql) that can be run with flyway undo. Key lesson: schema rollback is always harder than app rollback. Design migrations to be forward-only and safe (the expand-contract pattern prevents most rollback scenarios). Always test migrations on a production-sized staging database before prod deployment.
L3 → "You're merging two feature branches, each with a Flyway migration at version V7. Git merge succeeds. What's the deployment problem and how do you prevent it?"
Both branches created V7__feature_a.sql and V7__feature_b.sql. After merge, Flyway sees two migration files at the same version number and throws a MigrationException. Flyway requires strictly increasing version numbers with no gaps or conflicts. Prevention strategies: (1) Use timestamp-based versioning: V20260827143512__description.sql — timestamp collisions are practically impossible. (2) Use a migration registry in CI — a check that rejects PRs whose migration version already exists on main. (3) Switch to Liquibase: changeset IDs are globally unique strings (e.g., author:description), no linear ordering required. (4) Convention: never pre-assign version numbers locally — always look at what's already on main before creating a migration and take the next available.
Trap answer to avoid: "Just use spring.jpa.hibernate.ddl-auto=update in production." — Hibernate's DDL auto-update is unsafe for production: it can silently ignore column renames (leading to duplicate columns), cannot drop columns or tables (only adds), produces non-deterministic behavior, and leaves no audit trail of what changed. Flyway/Liquibase give you versioned, reproducible, auditable schema evolution. ddl-auto=validate is fine in production (validates the schema matches the entity model without modifying it), but update is not.