Skip to content
← Back to Blog

My /health endpoint was a snitch.

·5 min read
#go#kubernetes#observability

My pod was healthy. Redis sneezed. My /health endpoint checked Redis, caught the timeout, and returned 503. The kubelet read that as “the process is broken” and restarted it. The process was not broken. It was standing next to something broken.

With ten replicas, all ten /health endpoints check Redis. Redis blips for two seconds and Kubernetes executes the whole fleet, one health poll at a time. That is not Kubernetes overreacting. That is your health endpoint, snitching.

So I built go-health — three probes that know the difference between being broken and standing next to something broken.

Three questions, three endpoints

Kubernetes asks your pod three questions. Liveness — “are you alive?” Readiness — “can you serve traffic?” Startup — “are you done booting?” Most codebases answer all three with the same handler, which is like answering “are you dead?” and “is the database up?” with the same shrug.

go-health splits them:

probe := health.New(injector,
    health.WithCriticalServices("database", "redis"),
    health.WithVersion("1.0.0"),
)

mux := http.NewServeMux()
probe.RegisterRoutes(mux, health.DefaultRoutes())

/healthz, /readyz, /startupz. Liveness never touches your dependencies — it returns in microseconds, always 200, because “am I alive” is not a question about Redis.

Readiness checks what you marked critical. Startup checks once, latches to 200, and never asks again — booting is a phase, not a lifestyle. And Kubernetes can poll as hard as it likes; your dependencies see one check per second, max.

Warn is not a death sentence

Here is the part I am proudest of. metrics-exporter dies. Nothing you serve depends on it. The old /health says 503, the pod restarts, and now the exporter is still broken and you are also missing a pod. A funeral for a component nobody uses.

go-health answers 200 with "status": "warn" instead:

{
  "status": "warn",
  "checks": {
    "database": { "status": "pass" },
    "metrics-exporter": { "status": "warn", "error": "connection refused" }
  }
}

The pod stays in rotation. Only services you explicitly marked critical are allowed to kill you. Your health endpoint goes from informant to diplomat.

Opinions, baked in

  • GET only. A HEAD request gets a 405 with the body health probes only accept GET. Not an error message. A fact.
  • Polite shutdown. Shutdown() flips readiness to 503 immediately, so load balancers stop sending traffic, while liveness stays 200, so Kubernetes does not restart you mid-drain. Dying with grace.
  • You can lie about your boot time. WithBootTime overrides the timestamp used to compute uptime. Useful for testing. Also useful for vanity.
  • The kubelet cannot hammer your dependencies. Responses are cached behind an atomic pointer and refreshed once a second. Ten polls a minute cost ten memory reads, not ten trips to Postgres.

What I deliberately did not build

  • Logging. The library imports no logging package. “A library must not make logging decisions for the host application.” Your logs, your rules.
  • Client-tunable timeouts. Someone will ask for ?timeout=50ms as a query parameter. That is a DoS amplifier with a query string. There is a design doc in the repo explaining why not; the short version is no.
  • A UI. go-health stays a single dependency (samber/do) and speaks JSON. Consumers who only want probes pay zero cost for pixels they will never see.

The status page your binary deserves

Because green JSON is not a personality, I also built go-health-dashboard. Same probe, two more lines, and your service ships a live status page:

dash := dashboard.New(probe,
    dashboard.WithTitle("My Service"),
)
dash.RegisterRoutes(mux)

Content negotiation does the split: the kubelet gets JSON, a browser gets HTML — green, yellow, and red cards grouped by severity, updating over SSE the moment a check flips. Trend sparklines, dark mode, a Prometheus endpoint, and a green heart for a favicon. The banner says “All Systems Operational”, and for once in your life it is not a lie.

The example app ships a fake redis that flaps every fifteen seconds, so you can watch the badge flip like a heart monitor with commitment issues.

One design note I stand behind: unknown statuses plot as fail. The trend line dips on anything that is not provably healthy. Optimism is for landing pages.

And yes — the dashboard runs on go-datastar and go-sse, from the last post. I am building a universe, one health endpoint at a time.

Where it stands

go-health is at v0.0.2 with one runtime dependency. The dashboard is at v0.3.1 and needs GOEXPERIMENT=jsonv2 to build — experimental Go, I know; I have made peace with it. The README still says “v0.0.1 alpha”, because I shipped v0.0.2 faster than I updated my own documentation — a library that reports the health of everything except itself. Both ship a FEATURES.md “generated by studying the actual code, not the marketing claims”, which is the only kind of feature list I know how to write.

Read the code. Your pods were always fine. Your health endpoint just had a big mouth.