Diff
2182be52 → head
before
after
1---1---
2title: Toby Incidents — How this works2title: Toby Incidents — How this works
3status: live3status: live
4last_updated: 2026-05-114last_updated: 2026-05-11
5---5---
6 6
7# Toby Incidents7# Toby Incidents
8 8
9This folder is the **warroom** for Toby bug reports and technical incidents. It's maintained by a team of four agents working as a workflow.9This folder is the **warroom** for Toby bug reports and technical incidents. It's maintained by a team of four agents working as a workflow.
10 10
11## The team11## The team
12 12
13| Agent | Role |13| Agent | Role |
14|---|---|14|---|---|
15| **`toby-incident-coordinator`** | Warroom commander. Receives the complaint, dispatches specialists, synthesises the incident doc. |15| **`toby-incident-coordinator`** | Warroom commander. Picks the work unit (inbox / labeled queue / discernment sweep of the open backlog), dispatches specialists, synthesises the incident doc, decides which transitions to record. |
16| **`toby-frontend-doctor`** | UI specialist. Reproduces via Playwright, reads `apps/extension` / `apps/landing` / `apps/mobile`. |16| **`toby-frontend-doctor`** | UI specialist. Reproduces via Playwright, reads `apps/extension` / `apps/landing` / `apps/mobile`. |
17| **`toby-backend-doctor`** | Go API specialist. Pulls GCP logs, queries Toby prod DB read-only, reads `apps/api`. |17| **`toby-backend-doctor`** | Go API specialist. Pulls GCP logs, queries Toby prod DB read-only, reads `apps/api`. |
18| **`toby-incident-validator`** | Quality gate. Re-checks evidence, applies `triple-check` (correctness / quality / security), returns a binding verdict before the incident closes. |18| **`toby-incident-validator`** | Quality gate. Re-checks evidence, applies `triple-check` (correctness / quality / security), returns a binding verdict + confidence before the incident closes. |
19| **`toby-incident-fix-shipper`** | Last-mile patcher. **Only runs when validator returned `validated` + high confidence.** Creates a fresh git worktree of `axiomzen/toby-mono-repo` under `/tmp/`, applies the proposed fix from the incident doc, runs the verify plan, pushes a `warroom/...` branch, and opens a PR. The user's primary checkout is never touched. Skipped automatically on conditional / rejected / non-high-confidence verdicts — humans review those first. |
19 20
21After all five sub-agents finish, the coordinator posts a single consolidated report to Slack `#C0B3FN70MEE` (Toby AZ Slack) summarising verdict, root cause, doctors used, ticket outcome, and PR URL (or decline reason). The Slack report is skipped on no-op runs (Wave 0 finds nothing to investigate).
22
20The workflow is recorded in AIOS as **Toby Incident Response** (id `9b78790f-2aea-4f65-876f-53d1a114c3ae`).23The workflow is recorded in AIOS as **Toby Incident Response** (id `9b78790f-2aea-4f65-876f-53d1a114c3ae`).
21 24
22## How to report an incident25## How to report an incident
23 26
24Drop a markdown file into `toby/incidents/_inbox/` with whatever you know. Filename = today's date + a slug, e.g.:27**The only way into the warroom is a ticket labeled `needs-warroom`.** Manual file drops into `_inbox/` are deprecated — that path got us into trouble (no provenance, no priority, no assignee, no audit trail when a complaint moved between mediums).
25
26```
27toby/incidents/_inbox/2026-05-11-blank-extension-page.md
28```
29
30Body shape (loose — agents adapt):
31
32```markdown
33## Symptom
34Extension tab stopped opening, displaying blank page on infinite load.
35
36## Reproduce
371. Open new tab (Cmd-T).
382. Expect Toby new-tab UI; get spinning blank for >10s.
39 28
40## When29The flow:
41Started ~2 hours ago. At least 3 user reports so far.
42 30
43## Anything I noticed311. **File a ticket** in the Toby project (via UI, MCP, or an agent like `toby-state-of-business`). For a user-facing bug, set `kind: "bug"` and add `labels: ["needs-warroom"]`. Body should describe symptom / reproduce / when / anything-noticed — same shape as the old inbox files.
44- Console: lots of 401 from /v3/me322. **The ticket→warroom bridge** (`lib/tickets.ts` → `bridgeWarroomIfNeeded`) sees the new label, writes an inbox file at `toby/incidents/_inbox/YYYY-MM-DD-<ticket-id>-<slug>.md` containing the ticket body + provenance, and stamps `warroom-bridged` on the ticket so it isn't double-written.
45- Restart of Chrome doesn't help333. **The warroom workflow** (`Toby Incident Response`, daily cron at 09:00 UTC) picks up the inbox file at its next tick, runs the 4-wave investigation, writes the canonical incident doc, and uses `aios_tickets_record_attempt` to transition the source ticket to `done` / `in_review` / `blocked` based on the validator's verdict.
46```
47 34
48The moment that file lands, the inbox watcher fires the coordinator agent. Wait 4–8 minutes; an incident doc appears at `toby/incidents/<YYYY-MM-DD>-<slug>.md`.35End-to-end latency is therefore "next 09:00 UTC tick", not "4–8 min". If a complaint genuinely can't wait until tomorrow, an operator can still tick the workflow manually from the Workflows app — but the standard path is the bridge.
49 36
50## What the warroom produces37## What the warroom produces
51 38
52Each closed incident becomes a single canonical doc:39Each closed incident becomes a single canonical doc:
53 40
54```41```
55toby/incidents/2026-05-11-blank-extension-page.md42toby/incidents/2026-05-11-blank-extension-page.md
56```43```
57 44
58With sections: symptom, reproduction, root cause, production impact, proposed fix (as a `diff`), verify plan, validator verdict + confidence, open questions, timeline of who-did-what-when, and links to the doctors' run artifacts for archeology.45With sections: symptom, reproduction, root cause, production impact, proposed fix (as a `diff`), verify plan, validator verdict + confidence, open questions, timeline of who-did-what-when, and links to the doctors' run artifacts for archeology.
59 46
60The agents NEVER apply patches to the Toby codebase themselves. The incident doc carries a fix proposal that **you** decide whether to ship.47The agents NEVER apply patches to the Toby codebase themselves. The incident doc carries a fix proposal that **you** decide whether to ship.
61 48
62## Triggering manually49## Triggering manually (operator-only escape hatch)
63
64If you want to run the warroom without dropping a file:
65
66```
67mcp__aios__aios_agents_run({
68 agent: "toby-incident-coordinator",
69 overrideOrders: "<paste the complaint here>"
70})
71```
72 50
73Or, from the Workflows app, tick **Toby Incident Response**.51If a complaint cannot wait until the next cron tick, an operator can run the workflow directly from the Workflows app: tick **Toby Incident Response**. The coordinator's Wave 0 will pull from inbox-first, then ticket queue. Prefer the labeled-ticket path — manual ticks bypass the audit trail and should be reserved for genuine emergencies.
74 52
75## Verdicts53## Verdicts
76 54
77The validator returns one of three states; the incident doc reflects it:55The validator returns one of three states; the incident doc reflects it:
78 56
79- **`closed`** → validator confirmed the fix would work; ready to ship after operator review.57- **`closed`** → validator confirmed the fix would work; ready to ship after operator review.
80- **`open`** → twice-rejected by the validator and the doctors couldn't satisfy its objections. Operator decides what to do next.58- **`open`** → twice-rejected by the validator and the doctors couldn't satisfy its objections. Operator decides what to do next.
81- **`conditional`** → fix is good IF a specific question is resolved first (e.g. "is this migration safe under live write traffic?"). Surfaced for operator decision.59- **`conditional`** → fix is good IF a specific question is resolved first (e.g. "is this migration safe under live write traffic?"). Surfaced for operator decision.
82 60
83## Folder structure61## Folder structure
84 62
85```63```
86toby/incidents/64toby/incidents/
87├── README.md ← this file65├── README.md ← this file
88├── _inbox/ ← drop complaints here (operator)66├── _inbox/ ← bridge-written; one file per warroom-bound ticket
89│ └── 2026-05-11-blank-page.md67│ └── 2026-05-12-toby-14-<slug>.md
90├── 2026-05-11-blank-extension-page.md ← canonical closed incident doc68├── 2026-05-11-blank-extension-page.md ← canonical closed incident doc
91└── ...69└── ...
92```70```
93 71
94The agents never touch `_inbox/` and never delete anything. You own that bucket.72The agents never apply patches to the codebase. The ticket→warroom bridge writes `_inbox/` files; the workflow consumes them and then leaves them alone (operator decides when to archive). Humans should not hand-drop files into `_inbox/` — use the labeled-ticket path so provenance lives in one system.
95 73