← back to the counter

Horizon production outage: migration lock queue behind a 42-hour runaway query

Summary

A production deploy at 6:14 PM PT ran the HAL-7101 phase-1 Prisma migration (20261218000000_add_fit_status_derivation, commit e90a7437). Its ALTER TABLE on horizon.people requested an ACCESS EXCLUSIVE lock that queued behind an ad-hoc analytics query that had been running against that table for 42 hours. In Postgres, a queued exclusive lock blocks all new readers, so every query touching horizon.people — which the root layout hits on every server render — stacked up behind the stalled migration. The connection pool exhausted within seconds and the site returned errors ("Something went wrong" boundary pages and API 500s) for roughly 8.5 minutes.

Recovery: the blocking backend was terminated with pg_terminate_backend(1157); the migration acquired its lock and completed within seconds (6:22:43 PM), the lock queue drained to zero, and page loads returned to ~0.4–0.6s. The migration was additive-only; no data was lost or at risk.

Impact

• All SSR routes and most APIs failed or hung for ~8.5 minutes — the root layout's session refresh reads horizon.people, so the block cascaded site-wide (/editor, /dashboard, /api/problems/*, /api/runs/*, ~55 routes total).
• Vercel error clusters show a dense storm of timeout exceeded when trying to connect (pg-pool via Prisma) confined to 6:14:09–6:19:59 PM, including two [RootLayout] failure clusters (~314 occurrences, ~48 distinct users) that exist only inside the outage window.
• Editor sessions in flight saw the Next.js error boundary (e.g. reference 3866094746); Yjs collaborative-editing auth (/api/collab/auth) also failed during the window.
• Secondary, pre-existing impact: the runaway query contributed to the prod DB (status-tracker-db, db.t3.large) running at 85–100% CPU with a CPU credit balance of zero for 12+ hours before the incident.

Timeline (PT)

How it happened

1. Why did users see error pages? Server renders and API calls timed out waiting for a database connection (pg-pool timeout exceeded when trying to connect).
2. Why did the pool exhaust? Every query touching horizon.people hung indefinitely, holding its connection; new requests could not get one.
3. Why did those queries hang? The migration's queued ACCESS EXCLUSIVE lock blocks all new lock acquisitions on the table — Postgres grants locks in order, so readers queue behind a waiting exclusive lock.
4. Why was the migration stuck waiting? A 42-hour-old ad-hoc query held a conflicting lock on horizon.people and never finished.
5. Why could this wedge production? Nothing bounds either side: Prisma migrations run with no lock_timeout, and the database has no statement_timeout or long-query alerting that would have killed or surfaced the runaway query during the preceding 42 hours.

Root cause: deploy-time DDL with unbounded lock waits, colliding with an unbounded ad-hoc query on a shared production credential. Either guard alone — a lock_timeout on migrations, or a statement/idle timeout on interactive sessions — would have prevented the outage.

Attribution

Not conclusively identified. What we know about the blocking session:

• Client library postgres.js (the porsager npm driver) — not used anywhere in this repo, so it was a personal script or external tool, not deployed application code.
• Source IP 73.189.218.13: Comcast residential, San Francisco. Not Wyatt's current IP. No connections from that IP remain.
• Connected Monday, Jul 27 at 12:17 AM PT as statusadmin — the single shared credential every engineer's PROD_DB_URL uses, so the database cannot distinguish who it was.
• The query analyzed training-cohort onboarding (people entering training after 7/14 without a Halluminate email), which suggests someone working on training/onboarding tooling.

Resolved the next day: it was Scott, who had kicked off the query from his laptop. Notably, the database never told us — attribution came from asking humans. The trail above is exactly as far as the system itself could take us, which remains the finding: with every engineer connecting as statusadmin, client_addr is the only identity signal we have (see action item 4).

Blameless framing: Scott did nothing unreasonable — a long analytical read on prod is normal work, and the query hanging for 42 hours was as invisible to him as to everyone else. The system allowed a routine query and a routine deploy to combine into a site-wide outage.

What surprised us

• A lock you're merely waiting for can take down the site. The migration never got its ACCESS EXCLUSIVE lock — it didn't need to. A queued exclusive lock blocks every new reader behind it, so the outage was caused by a statement that was doing literally nothing.
• A query ran on prod for 42 hours and nothing noticed. Not the CPU pegged at 85–100%, not the credit balance at zero, not the query itself. Our mental model was "something would tell us"; the reality was "nothing is configured to."
• The database couldn't answer "who ran this?" The client library (postgres.js) appears nowhere in our repo, and everyone connects as statusadmin, so attribution dead-ended at a residential IP. We assumed connection metadata identified people; it identifies a shared password. (It was Scott — but we learned that by asking around the next day, not from the system.)

What went well / what went poorly

Went well

• Diagnosis to recovery in ~3 minutes from first investigation: pg_blocking_pids() pinpointed the root blocker immediately.
• The migration was additive-only ("expand" phase), so unblocking it carried no data risk and it completed in seconds.
• Root-layout fail-open guards (from the HAL-5695 work) kept some pages rendering degraded rather than blank.

Went poorly

• A query ran for 42 hours on prod, burning a core the whole time, with zero alerting — on runtime, on CPU pegged at 85–100%, or on the CPU credit balance sitting at zero.
• Detection was a human hitting an error page, not monitoring.
• Migrations run unguarded during deploys: no lock_timeout, no retry, no pre-flight check for conflicting long-running locks.
• The error boundary exposed only an opaque reference number; correlating it to the server-side cause required log archaeology.

Action items

Evidence: RDS CloudWatch metrics (status-tracker-db), pg_stat_activity captures, Vercel runtime error clusters, _prisma_migrations records.