days without incident
last incident
Victoria and Rob broke prod
“A string prompt closed the Agent SDK control channel, silently scoring every agentic verifier 0.0”
hall of shame
2026-08-04 Victoria and Rob A string prompt closed the Agent SDK control channel, silently scoring every agentic verifier 0.0 2026-08-04 AWS Trust & Safety, reacting to us 27×-ing our own CodeBuild spend without telling anyone Legitimate Harbor traffic tripped AWS fraud detection, which suspended CodeBuild account-wide 2026-07-28 Scott Horizon production outage: migration lock queue behind a 42-hour runaway query 2026-07-21 Wyatt Mounted-bundle cutover silently dropped every seed file with a space in its name 2026-07-15 Wyatt Vercel workspace conversion experiment took prod down at midnight 2026-05-12 Jerry Wu `AND $1 IS NOT NULL` made pg's prepared-statement parse fail, silently emptying the Training tab 2026-03-25 Will Thunderbird pack hook `$HOME` resolved to `/root`, breaking email injection for all Horizon jobs 2026-02-25 Wyatt Bad in-band resources crashed the verifier service 2026-02-23 Wyatt Verifier standardization broke prod — then the fix broke it again