Vercel workspace conversion experiment took prod down at midnight
| Date | 2026-07-15 |
| Severity | SEV-1 — Horizon fully down, briefly |
| Outage window | ~12:28 – 1:17 AM PT (first user report → confirmed back up) |
| Duration | ~10 min hard-down per the operator; first report to all-clear spanned ~49 min |
| Detected by | User report — "Is horizon down?" in #vertical-data-sme, 12:28 AM |
| Prod breaker | Wyatt |
| Status | Resolved |
| Author | Wyatt Marshall |
Summary
While experimenting with Vercel's new workspace feature — consolidating the Horizon project and the verifier project under one workspace — the conversion was applied to the live Horizon project before it was meant to leave testing. Production serving broke; Horizon was fully down. Recovery was redeploying the last known-good production build (1:15 AM), with users confirmed back by 1:17 AM. The experiment was rolled back entirely within the hour.
Impact
- Horizon production fully down around midnight PT. Small mercy: the timing meant few users were on — but our contractor base spans timezones, and the people who were on (VD writers) hit a dead site and said so within minutes.
- Aftermath cleanup: the hour-long experiment left 16 zombie container images (~9.4 GB) in the Horizon project's container registry, dead weight that was identified and deleted in the VCR-storage-bloat cleanup two days later.
Timeline (PT)
| When | What happened |
|---|---|
| ~midnight | Experimenting with converting the Horizon Vercel project into a workspace with the verifier project. The conversion applies to the live project. Prod breaks. |
| 12:28 AM | "Is horizon down?" — first user report, #vertical-data-sme. |
| 12:54 AM | "Might be, checking" — operator (who broke it) begins investigating. |
| 1:07 AM | Second user confirms the outage. |
| 1:15 AM | Last known-good production build redeployed. |
| 1:17 AM | "back up now." Recovery confirmed. Experiment fully rolled back. |
How it happened
- Why was prod down? The workspace conversion changed how the live Horizon project built and served — the deployment record for that window shows duplicate production deployments of the same commits firing at the same second, consistent with the repo briefly being wired into the project twice.
- Why did an experiment touch the live project? Vercel project settings have no staging lane — there is no preview environment for project configuration. A conversion clicked through on the real project takes effect on the real project, immediately.
- Why was it clicked on the real project? The intent was "just looking at" the new feature; the conversion flow applied faster than the mental model of "I'm still exploring" caught up. Classic midnight-solo-operator territory.
Root cause: infrastructure configuration has no dry-run. Code goes through branch → preview → merge → prod; project settings go straight to prod. Exploring a new platform feature on the production project is production change management, even when it feels like window shopping.
What surprised us
- Project settings are deploys. Every guardrail we have (previews, CI, review) protects the code path; a dashboard toggle bypasses all of it.
- Vercel's instant-rollback/redeploy was genuinely the fast path — recovery was one redeploy of a prior good build, minutes once attempted.
- The experiment left billing residue: 16 orphaned container images quietly accruing storage cost, found only during a separate bill investigation.
What went well / what went poorly
Went well
- Recovery was fast and clean once engaged: redeploy last-good, confirm, roll the experiment back fully.
- The blast window was about the smallest possible for a full outage.
Went poorly
- Detection was a contractor at 12:28 AM, not monitoring — the operator learned prod was down from Slack, 26 minutes after the first report.
- No uptime alerting existed to catch "site fully down" (still true as of this write-up; see the 2026-07-28 incident's action items).
- Risky platform experimentation happened solo, at midnight, directly on the production project.
Action items
| # | Action | Priority | Ticket |
|---|---|---|---|
| 1 | Test platform/project-level changes on a scratch Vercel project, never the live one | P1 | practice, adopted |
| 2 | External uptime check on horizon.halluminate.ai (folds into the alerting gap from the 2026-07-28 incident) | P1 |
Written up 2026-07-28 from Slack history, the Vercel deployment record, and the VCR cleanup investigation.