True observability horror stories from on-call engineers: dashboards that lied, alerts that never fired, cardinality explosions, sampling blind spots, and the lessons each one taught. Read the tales and submit your own.https://o11yhorrors.com/en-ushttps://o11yhorrors.com/tales/self-healing/https://o11yhorrors.com/tales/self-healing/Our auto-remediation killed unhealthy pods and restarted them. A bad config made every pod fail just after taking traffic, so it killed and restarted thousands of times a minute, all night, and never paged a soul.Fri, 17 Apr 2026 00:00:00 GMTautomationremediationfeedback-loopshttps://o11yhorrors.com/tales/the-clock-that-drifted/https://o11yhorrors.com/tales/the-clock-that-drifted/Errors were happening in the future. One node's clock had drifted eleven minutes ahead, and its metrics arrived stamped from a time that had not occurred yet, quietly poisoning every aggregate.Thu, 02 Apr 2026 00:00:00 GMTtimemetricsclock-skewhttps://o11yhorrors.com/tales/the-cardinality-incident/https://o11yhorrors.com/tales/the-cardinality-incident/Someone added one label to a counter so we could break traffic down per user. There were eight million users. Each became a time series, and the metrics backend fell over and took monitoring for the whole company with it.Thu, 19 Mar 2026 00:00:00 GMTmetricscardinalitymonitoring-outagehttps://o11yhorrors.com/tales/the-silent-sampler/https://o11yhorrors.com/tales/the-silent-sampler/We turned on one percent trace sampling to save money. Three months later a rare bug appeared, and the traces that could have explained it had statistically never been recorded.Wed, 04 Mar 2026 00:00:00 GMTtracingsamplingcost-controlhttps://o11yhorrors.com/tales/it-was-the-logs-all-along/https://o11yhorrors.com/tales/it-was-the-logs-all-along/A slow service logged more to help us debug. The extra logs filled the volume, the agent buffered to memory, the OOM killer struck, and the restart logged even more. We built a machine that described its own death.Sat, 21 Feb 2026 00:00:00 GMTloggingcascading-failureresource-exhaustionhttps://o11yhorrors.com/tales/the-3-47-am-page/https://o11yhorrors.com/tales/the-3-47-am-page/A new hire's first on-call. A critical page at 3:47 AM. Forty-five minutes of cold-sweat investigation before learning the alert had fired every night for two years and everyone just slept through it.Sun, 08 Feb 2026 00:00:00 GMTon-callpagingalert-fatiguehttps://o11yhorrors.com/tales/the-alert-that-never-fired/https://o11yhorrors.com/tales/the-alert-that-never-fired/We had a disk-full alert. We had tested it. The night the database died, it stayed silent, because the metric pipeline that fed it died right along with the disk.Thu, 29 Jan 2026 00:00:00 GMTalertingthresholdsmonitoring-gapshttps://o11yhorrors.com/tales/the-dashboard-that-lied/https://o11yhorrors.com/tales/the-dashboard-that-lied/Every graph was green while checkout was completely down. The dashboard was not showing healthy traffic. It was showing the last thing it ever saw.Mon, 12 Jan 2026 00:00:00 GMTdashboardsmetricsstale-data