ADR 0005: exo-verify — code verification as a platform serviceCopy Anchor Link

• Status: accepted (owner-approved 2026-09-27)
• Date: 2026-09-27
• Roadmap: exo-verify

ContextCopy Anchor Link

Exo provisions shared platform services to every app so apps do not spend time on basic plumbing. Verification is not one of them yet: each app carries ad-hoc unit tests, and staging failures slip through. Example: the AI router passed all unit tests and review but fell back 100% of the time on staging, because a fix never merged. Only a live behavioral check against the deployed environment catches that class of failure.

Two constraints shape the design. The owner is moving off GitHub to Entire as the forge, so GitHub Actions and GitHub-app PR bots are out. And most commits are made by agents (~300/month across the portfolio), so per-seat vendor pricing is the wrong unit.

DecisionCopy Anchor Link

Exo provides verification as a shared service, exo-verify, in four layers:

1. Standard verify contract. Every repo declares verify commands (typecheck, lint, test, smoke) in project.yaml. Exo runs what is declared; a missing command is a visible gap, not a silent pass.
2. Sandboxed runs on Railway workers (not GitHub Actions). Trigger order of preference: an Entire webhook if Entire exposes one; otherwise an exo verify CLI that agents run before pushing; otherwise Exo polling. In the sandbox an agent may also write targeted tests for the change and run them (TREX-style runtime validation).
3. Post-deploy live checks against staging: health, auth (Exo ID JWKS, OAuth callbacks), and 2–3 declared user journeys per app driven by Playwright or a browser agent.
4. AI review pass using an open-source engine (PR-Agent / Kodus-style) or our own prompts. All model calls go through the Exo AI broker, so cost is logged per app in exoAiBrokerJobs and the router keeps easy diffs on cheap models.

Results are stored in Convex as release gates, attached to the Entire trail, and gate staging → production promotion.

Unit economicsCopy Anchor Link

Cost per verified commit for the own service is roughly $0.02–0.10, recorded per app so it shows up in each app's cost-to-serve.

Alternatives consideredCopy Anchor Link

• Greptile — strongest vendor review and has runtime validation, but priced per seat plus credits and requires Enterprise for a non-GitHub/GitLab forge. Kept as a later option for the review layer (Greptile Enterprise) if our review quality falls short.
• CodeRabbit — the free CLI reviews a local diff, so agents can use it pre-push as an interim second opinion while layer 4 is built. Not the system of record.
• Graphite — GitHub-centric stacking/review; does not fit the Entire forge.
• GitHub Actions — rejected; the owner is leaving GitHub.

ConsequencesCopy Anchor Link

• Every app gains a verify block in project.yaml; onboarding an app includes declaring its 2–3 golden journeys.
• Vendor review engines are more tuned than ours at first. We track catch rate against false positives per layer and revisit buying the review layer if the gap persists.
• Live checks need a staging-only sign-in path for bots (seeded Exo ID test identities), because browser agents cannot pass Google's bot detection.
• Open question: does Entire expose webhooks or a checks API? Verify before building the trigger; the CLI and polling fallbacks keep the design unblocked either way.