Staging proudly reported 27/27 services healthy while Kafka had been down for 14 minutes, because /health only probes the process and not downstream dependencies. Do you wire dependency liveness into health checks or keep a separate /ready probe to prevent this slapstick during deploys?
Split probes: liveness = process, readiness = deps, and make /ready fail if a lightweight Kafka check (produce to a health topic or query lag) doesn’t pass in about 2–3s, with a 10s cache to avoid flapping — ‘liveness ≠ readiness.’ Tiny caveat: keep Kafka out of /live or you’ll play restart pinball; we also gate deploys on /ready in CI/CD.
We hit the same “healthy while Kafka is dead” mess and fixed it by adding a 60s grace in /ready plus a small disk-backed spool for producers; if Kafka’s still unreachable after the grace, readiness flips, but short blips don’t flap deploys. We also stamp an X-Degraded: true header so downstreams know we’re limping instead of lying, which would’ve surfaced your “14 minutes” much sooner. Would a short grace + spool fit your deploy cadence, @OP?