Backend finding — blank-extension-page (2026-05-11)

Reproduced (from backend side)?#

no — the backend is not implicated. The Go API has been deployed on the same commit since 2026-04-01 with a sub-0.002 % error rate, ~1M req/day of 2xx, and zero correlated spike at or after 2026-04-09. The SW boot path has no listener-after-await bug. The hang lives in the renderer (toby.html tab) — almost certainly Chrome's "extension context invalidated" state dropping chrome.storage.local.get callbacks, which is a Chromium MV3 platform behaviour, not anything we ship on the Go side.

Symptoms observed in logs / SW / API#

• Prod-api is stable. prod-api-00427-9p2 (revision commit-sha=4b0107858e706c904e6cf2841fbcbf81a1e2f94f) has been the active prod revision since 2026-04-01T20:08Z. The two before it (00425, 00426) ran the same SHA, so the last actual code change to deployed prod-api is older than 2026-02-02 — well before the user-complaint window the frontend doctor narrowed to post-2026-04-09.
• Error volume is rounding error. Last 24 h: 0 5xx in prod metrics (Cloud Run aggregated request_count, response_code_class=5xx); 23 ERROR-severity entries in stderr, of which 22 are expected 401s on stale-session PUT /v2/users/activity / GET /v2/states and 1 is a downstream toby-ai-api 500 on a cache-warm UpdateUserAICache call — none are on the user-hydration path.
• 503 noise is normal cold-edge churn, not a regression. 51 × Cloud-Run "malformed response or connection error" 503s in the last 36 h are spread across /v2/states, /v2/surveys, /v2/users, /v2/public/announcements — all post-hydration endpoints. They cannot cause the pre-hydration blank-page hang.
• No panics, fatals, or CRITICALs in the prod-api log stream since 2026-04-01.
• DB is healthy. users (Toby Prod, read-only) shows 41,578 last-active in the past 24 h, 59,654 in the past 7 d, 720 new signups in the past 7 d — i.e. the API is successfully serving the hot user-activity path right now.
• SW boot path is well-formed. Every chrome.*.addListener call in apps/extension/app/background/*.ts registers synchronously at module top level (or synchronously inside a non-awaiting init()). I could not find a listener-after-await MV3 boot-bug pattern.
• Min-instances=4 on prod-api → cold-start hangs are not a factor for the request volumes in play.

Root cause (best hypothesis, backend lens)#

Confidence: high that the proximate fault is not on the backend. The chrome.storage callback drop the frontend doctor identified is a Chromium MV3 renderer-side phenomenon ("extension context invalidated") that occurs in the toby.html tab, not in the SW or our API:

1. Chrome auto-updates the extension while the user has a long-lived new-tab open (or the user reloads the extension themselves). Chrome marks the old tab's extension context invalidated; subsequent calls to chrome.storage.local.get from that tab return without invoking the callback (lastError is set, but getUser() at apps/extension/app/state/accessors/user.tsx:45-50 never reads chrome.runtime.lastError, so the dropped callback is indistinguishable from "just slow").
2. The frontend's d68726b29 commit (2026-04-09) widened the AuthWrapper gate at apps/extension/app/containers/Toby.tsx:304 to depend on isUserHydrated, which is bound 1:1 to that one unbounded storage callback. The same chrome.storage drop now reliably produces a blank page where it used to either work or only flicker.
3. The Go API is structurally incapable of causing this drop: getUser() does not make a network call. It reads from chrome.storage. The very first network request to prod-api would happen only after isUserHydrated flips true. The hang precedes any HTTP request.

What the backend doctor can refute, with evidence:

• Not an API regression — code is from 2026-02-02 or earlier (commit SHA 4b0107858 has been deployed on three consecutive revisions back to revision 00425, 2026-02-02; the deploys since are config-only redeploys of the same SHA).
• Not a 5xx storm — 0 of either 500 or 5xx-class in last 24 h (response_code_class="5xx", hourly time-series, all zero).
• Not a CORS / cookie / CSP change — no commits to apps/api/middlewares/, apps/api/routes/, or any session handling code in 2026.
• Not a database collapse — 41k DAU, healthy hourly distribution, no row-count anomaly.
• Not a Cloud-Run cold-start race — min-instances=4 keeps the service permanently warm; the 503 "malformed response" blips are tiny (51 / 36 h) and don't touch the auth path.

Evidence#

GCP logs — error volume since 2026-05-04#

Filter (gcp-observability list_log_entries):

resource.type="cloud_run_revision"
  AND resource.labels.service_name="prod-api"
  AND severity>=ERROR
  AND timestamp>="2026-05-04T00:00:00Z"

Distinct error messages, count, all 23 entries:

   8  "401 PUT /v2/users/activity"
   8  "401 GET /v2/states"
   5  "failed to search nextItems"
   1  "failed to send AI request"            # downstream toby-ai-api 500
   1  "401 POST /v3/nexts/search"

All 401s. No panics. The one 500 is downstream toby-ai-api from services/ai/ai_service.go:171 UpdateUserCache — not on the auth or hydration path.

GCP metrics — 5xx by day (run.googleapis.com/request_count)#

2026-05-10 → 2026-05-11   500-class: 0     2xx: 1,108,508
2026-05-09 → 2026-05-10   500-class: 4     2xx:   560,996
2026-05-08 → 2026-05-09   500-class: 0     2xx:   718,076
2026-05-07 → 2026-05-08   500-class: 19    2xx: 1,186,238
2026-05-06 → 2026-05-07   500-class: 8     2xx: 1,265,714

Worst day = 19 / 1.18 M ≈ 0.0016 %. No spike pattern.

Cloud-Run revisions (gcloud run revisions list --service=prod-api)#

prod-api-00427-9p2  2026-04-01T20:08Z  commit 4b01078  ACTIVE
prod-api-00426-s6l  2026-04-01T17:58Z  commit 4b01078
prod-api-00425-x7f  2026-02-02T19:13Z  commit 4b01078
prod-api-00424-dpx  2026-01-28T20:16Z  commit 0b44adc

The deployed SHA hasn't actually changed since at least 2026-02-02. The 2026-04-01 deploys are config redeploys of the same SHA. Code on the wire today is older than the bug window.

Cloud-Run config (gcloud run services describe prod-api)#

autoscaling.knative.dev/minScale: "4"
autoscaling.knative.dev/maxScale: "20"
containerConcurrency: 900
run.googleapis.com/cpu-throttling: "false"
run.googleapis.com/startup-cpu-boost: "true"

Always 4 warm instances; cold-start cannot explain a recurring user-visible issue.

DB query — Toby Prod (Read Only) be55a66b-c905-4759-9ce1-a97785bb69e6#

SELECT
  COUNT(*) AS total_users,
  COUNT(*) FILTER (WHERE created_at  >= NOW() - INTERVAL '7 days') AS new_7d,
  COUNT(*) FILTER (WHERE created_at  >= NOW() - INTERVAL '1 day')  AS new_1d,
  COUNT(*) FILTER (WHERE last_active >= NOW() - INTERVAL '1 day')  AS active_1d,
  COUNT(*) FILTER (WHERE last_active >= NOW() - INTERVAL '7 days') AS active_7d
FROM users WHERE deleted_at IS NULL;

total_users  new_7d  new_1d  active_1d  active_7d
1,046,842    720     93      41,578     59,654

And hourly distribution (last 12 h):

2026-05-11 17:00Z   943
2026-05-11 16:00Z 2,427
2026-05-11 15:00Z 2,740
2026-05-11 14:00Z 3,015
2026-05-11 13:00Z 3,352   ← peak
2026-05-11 12:00Z 2,877
2026-05-11 11:00Z 2,072
2026-05-11 10:00Z 1,418
2026-05-11 09:00Z 1,443
2026-05-11 08:00Z 1,736
2026-05-11 07:00Z 2,018
2026-05-11 06:00Z 1,853

Healthy diurnal curve. No collapse, no flatline. The API is processing this traffic right now.

SW source — apps/extension/entrypoints/background.ts#

import { defineBackground } from 'wxt/utils/define-background';
import { persistQueryClientRestore } from '@tanstack/react-query-persist-client';
import { IDBpersister, queryClient } from '~/state/client';

// These register chrome listeners as side effects
import '~/background/contextMenus';
import '~/background/inject';
import '~/background/badge';
import '~/background/onInstall';
import '~/background/tobyLinks';
import '~/background/omnibox';

export default defineBackground(() => {
  persistQueryClientRestore({
    queryClient,
    persister: IDBpersister,
    maxAge: Infinity,
    hydrateOptions: { defaultOptions: { queries: { gcTime: Infinity } } },
  });
});

The side-effect imports happen before defineBackground is called, so they register listeners synchronously at module evaluation. No MV3 boot-bug.

SW listeners (all top-level / no await before addListener)#

• apps/extension/app/background/onInstall.ts:9 — chrome.runtime.onInstalled.addListener (sync, top-level).
• apps/extension/app/background/inject.ts:140,204 — chrome.tabs.onUpdated.addListener, chrome.runtime.onMessage.addListener (both sync, after only consts).
• apps/extension/app/background/badge.ts:4-5 — chrome.action.setBadge* (sync, top-level; no listeners).
• apps/extension/app/background/omnibox.ts:1 — chrome.omnibox.onInputEntered.addListener (sync, top-level).
• apps/extension/app/background/tobyLinks.ts:5-81 — chrome.webRequest.onBeforeRequest.addListener × 2, chrome.tabs.onUpdated.addListener (sync, all inside an if (isFireFox()) gate; Chrome users skip this entirely).
• apps/extension/app/background/contextMenus.ts:217-261 — declared async function init() but no await anywhere before the three .addListener calls on lines 218, 219, 231. Listener attachment is therefore synchronous. ✓

The pattern check confirms what the prod metrics already imply: even when SWs do die and restart, the listeners re-attach correctly. The MV3 boot path isn't the wound.

SW shortcomings I did find (defence-in-depth, not the root cause)#

• apps/extension/app/background/contextMenus.ts:145-163 — fetches ${env.API_URL_V3}/cards from the SW with no AbortController / no timeout. If the network stalls during a context-menu save and the SW idles past 30 s, the in-flight fetch keeps the SW alive until tcp resets — not a crash, but means SW lifetimes are jittery on bad networks.
• apps/extension/entrypoints/background.ts:14-19 — persistQueryClientRestore(...) inside defineBackground is fire-and-forget with no .catch(). If IDB throws on cold open, it's silently swallowed; the queryClient stays empty and any SW callsite reading from it gets stale-empty data. This is the same IDB-availability surface the frontend doctor flagged for useIsRestoring() — except in the SW it's even less visible because the SW has no UI.
• apps/extension/app/state/client.ts:24-29 — IDBpersister.restoreClient calls get(idbValidKey) from idb-keyval with no error path. If IDB is locked or quota-exceeded, the promise rejects deep inside persistQueryClientRestore. Surface ditto.

None of those cause the blank page. They explain why the "silent" failure mode is so silent.

Recent commits — Go API since 2026-01-01 (everything)#

b9bea18c  Post fallback Slack message when CWS review AI draft fails  (#10)
4666eefc  Merge pull request #4 (CWS review monitor)
cbc92a78  Make retention discount eligible for all cancellation reasons
ba247d9a  Add Chrome Web Store review monitor with AI-drafted responses
0727449d  Migrate landing page CI and remove dead workflows
5bd96126  Add Turborepo configuration
7effe8b6  Refine legacy user discount logic in subscription context  (#876)
81f1ab57  Refactor retention offer logic                              (#875)
a462d067  Enhance user email verification logic … token validation    (#874)
36fe2451  Refactor retention offer logic to utilize Stripe coupon     (#872)
52f51924  Enhance retention offer logic and legacy user support       (#871)
bd6f64a8  Add retention discount functionality                        (#869)
9fb43ddd  Fix build error from last commit                            (#870)
7c7a1cdf  Remove Legacy Discount from getCoupon                       (#868)

None of these touch apps/api/middlewares, apps/api/routes, session/cookie handling, or anything the extension hits on boot. a462d067 is the only one in user-verification territory but it predates the latest prod deploy (2026-04-01) and is on a Stripe verify path, not the SW-relevant chrome.storage seed.

Proposed fix (backend / SW side)#

There is no backend-side fix that resolves the user-visible bug, because the bug is not on the backend. The frontend doctor's Layer 1 (timeout the hydration promise) and Layer 2 (recovery UI) are the correct, sufficient fixes.

That said, three SW-side hardening items are worth shipping alongside the FE fix, because they remove ambiguity from future incidents of this shape:

1. Catch and log the persist-restore failure in the SW. apps/extension/entrypoints/background.ts:14:

    export default defineBackground(() => {
   -  persistQueryClientRestore({
   +  persistQueryClientRestore({
        queryClient,
        persister: IDBpersister,
        maxAge: Infinity,
        hydrateOptions: { defaultOptions: { queries: { gcTime: Infinity } } },
   -  });
   +  }).catch((err) => {
   +    console.error('[toby-sw] persistQueryClientRestore failed', err);
   +  });
    });
   

   Zero functional change; just makes IDB failures visible in chrome://extensions → service worker → console.

2. Bound the SW fetch(/v3/cards) and fetch(/lists) paths with an AbortController. apps/extension/app/background/contextMenus.ts:145-191. A 10 s timeout + abort. Prevents the SW from being kept alive by a stuck TCP socket on bad networks.

3. Promisify the SW's chrome.storage reads with a lastError check at the source. Build a single chromeStorageGet<T>(keys, { timeoutMs }) helper that:

   • reads chrome.runtime.lastError on every callback,
   • rejects on a timeout,
   • rejects if chrome.runtime.id is undefined (the canonical signal that the extension context is invalidated).

   Replace every raw chrome.storage.local.get(key, cb) in app/state/accessors/*.tsx and app/utils/chromeapi.ts:248 with that helper. This is the real defence — the FE's Layer 1 only handles the getUser site; this helper handles every site uniformly. (It belongs in the FE codebase; from the backend lens, I'm flagging it because the chrome.storage drop is a class of bug, not a one-off.)

I would NOT redeploy prod-api as part of this incident — the API code is innocent and a redeploy is a needless blast-radius increase.

Verify plan#

1. Confirm prod-api stability over the next 24 h. Re-run the 5xx and ERROR-severity log queries above; expect the same near-zero counts. No action if so.

2. After FE Layer 1 ships, watch for the new NewTabHangShown beacon the frontend doctor recommended (Layer 3). If it fires at non-trivial volume without a corresponding 5xx spike in prod-api, that confirms the chrome.storage drop is platform-side and we ship Layer 1+2 as-is.

3. If the beacon volume correlates with a Cloud-Run revision bounce (use gcloud run revisions list --service=prod-api to spot any 5xx jump on a new revision), re-open the backend investigation. With the current cadence (one prod-api code change in 4 months) this is unlikely.

4. Add a synthetic SW health probe. A Cloud Scheduler job that hits https://api2.gettoby.com/v2/states with a throwaway token every 5 min and alerts if 5xx > 2 % over a 15 min window. Right now we have no signal between "user complains in CWS review" and "Sentry/Amplitude burn". Out-of-scope for this incident; file as follow-up.

5. Optional repro for SW context invalidation. In a dev profile: open chrome-extension://<id>/toby.html → chrome://extensions → toggle the extension off and back on. The open tab should now have chrome.runtime.id === undefined and any chrome.storage.local.get from it returns without invoking the callback. This is the canonical scenario; if FE Layer 1 rescues the page after 5 s in this state, the fix is verified.

Confirm or refute frontend hypothesis#

Refine, then confirm. The frontend doctor wrote:

If the MV3 service worker is dead or the extension context is invalidated … chrome.storage.local.get can return without invoking its callback.

The "MV3 service worker is dead" half is unlikely to be the cause in isolation. chrome.storage.local.get reads are handled by the Chrome browser process, not by the extension SW — a dead SW alone does not drop callbacks; Chrome will quietly fulfill the storage read either way. What does drop callbacks is the renderer (the toby.html tab) being in the "extension context invalidated" state. That state is most commonly entered when:

• Chrome auto-updates the extension while a chrome_url_override new-tab is open (Toby owns the new-tab page → this happens to every Toby user, every time we ship an update);
• the user manually disables/re-enables or reloads the extension;
• the SW crashes during a critical phase that invalidates the handshake (rare, but possible — and the SW IS more crash-prone than it should be because of the unhandled persistQueryClientRestore rejection and the un-aborted fetches I flagged above).

So the chrome.storage callback drop is real, the FE-side fix is correct and sufficient, and the SW-hardening items in §"Proposed fix" reduce the frequency of the drop (by reducing renderer/SW invalidations) without changing the severity of any single occurrence. Ship FE Layer 1+2 regardless; backend SW hardening is a nice-to-have follow-up.

The frontend doctor's hypothesis that there was a "Manifest V3 service-worker boot regression" — proposed by toby-product-strategist artifact 388c1db4 — is not supported by the evidence. The SW boot code is well-formed (listeners synchronous, no await-before-addListener), and prod-api hasn't changed in months. The post-2026-04-09 sensitivity is purely the FE widening the gate; the underlying chrome.storage drop is an evergreen Chromium MV3 phenomenon that the FE used to tolerate accidentally.