artc · Release QA Harness
7-layer release gate for a live collaborative app
About
artc is Sapien's real-time collaborative document workspace (Rust backend, React frontend, live presence over WebSockets). As it approached private beta, releases were being judged by hand: someone clicked around the deploy and hoped. I designed and built the release gate that replaced that ritual, and shipped it through code review into the production monorepo.
The harness tests the deployed app, not the build. 7 layers fold into a single gated PASS/WARN/BLOCK verdict: API smoke, deterministic UI across 4 device classes, WCAG accessibility, visual regression, an exhaustive cold-path sweep, 2-user collaboration tests, and an LLM persona layer.
Process
The cold-path sweep
The bug class that motivated the harness: a control that breaks on hover or focus that no happy-path test ever touches. The sweep programmatically hovers and focuses every interactive control on every screen and flags anything that vanishes, loses contrast, throws a console error, or lacks an accessible name.
Collaboration without flake
2 concurrently authenticated browser contexts assert live presence and comment propagation over WebSocket. Nondeterminism (cursors, timestamps, avatars) is masked and seeded rather than retried, so a red result means a real regression.
Agentic personas with checklists
4 personas (a screen-reader user, a one-thumb mobile user, a first-timer, an adversarial stress tester) each drive the app against a fixed human-authored checklist. Designed around the research finding that an unguided LLM tops out near 30% F1 on web testing and reaches ~49% when handed a checklist.
The gate
Everything reduces to 1 machine-readable verdict consumed 2 ways: a nightly CI run and a /qa command on pull requests for humans, and a JSON verdict that agentic dev loops use to self-check before asking for review. Hardened through review: the gate cannot silently pass.
Outcome
Merged into the production monorepo through multi-reviewer code review, with measured per-screen consistency budgets (type, color, and touch-target floors), committed visual baselines, and a glossary that lets a non-expert read the verdict in 20 seconds. Release judgment went from someone's afternoon to a gate that guards every pull request.