The collection

All the tales by the fire

8 true observability horror stories. Filter by the kind of monster, or read them all and learn to recognize the omens.

Self-Healing

redacted Apr 17, 2026

Our auto-remediation killed unhealthy pods and restarted them. A bad config made every pod fail just after taking traffic, so it killed and restarted thousands of times a minute, all night, and never paged a soul.

read the tale →

#automation #remediation #feedback-loops

The Clock That Drifted

infra, graveyard shift Apr 2, 2026

Errors were happening in the future. One node's clock had drifted eleven minutes ahead, and its metrics arrived stamped from a time that had not occurred yet, quietly poisoning every aggregate.

read the tale →

#time #metrics #clock-skew

Cardinality

you know who you are Mar 19, 2026

Someone added one label to a counter so we could break traffic down per user. There were eight million users. Each became a time series, and the metrics backend fell over and took monitoring for the whole company with it.

read the tale →

#metrics #cardinality #monitoring-outage

The Silent Sampler

observability lead Mar 4, 2026

We turned on one percent trace sampling to save money. Three months later a rare bug appeared, and the traces that could have explained it had statistically never been recorded.

read the tale →

#tracing #sampling #cost-control

It Was the Logs All Along

platform team Feb 21, 2026

A slow service logged more to help us debug. The extra logs filled the volume, the agent buffered to memory, the OOM killer struck, and the restart logged even more. We built a machine that described its own death.

read the tale →

#logging #cascading-failure #resource-exhaustion

3:47 AM

the new hire Feb 8, 2026

A new hire's first on-call. A critical page at 3:47 AM. Forty-five minutes of cold-sweat investigation before learning the alert had fired every night for two years and everyone just slept through it.

read the tale →

#on-call #paging #alert-fatigue

The Alert That Never Fired

SRE, anonymous Jan 29, 2026

We had a disk-full alert. We had tested it. The night the database died, it stayed silent, because the metric pipeline that fed it died right along with the disk.

read the tale →

#alerting #thresholds #monitoring-gaps

The Dashboard That Lied

on-call, payments Jan 12, 2026

Every graph was green while checkout was completely down. The dashboard was not showing healthy traffic. It was showing the last thing it ever saw.

read the tale →

#dashboards #metrics #stale-data