exo-verify

Source: ADR 0005, accepted (owner-approved) on 2026-09-27. It is on main and not yet on staging: read it on main.

WhyCopy Anchor Link

Each App carries ad-hoc unit tests, and staging failures slip through. The motivating example: the AI router passed all unit tests and review but fell back 100% of the time on staging, because a fix never merged. Only a live behavioral check against the deployed environment catches that class of failure.

Two constraints shape the design:

• The owner is moving from GitHub to Entire, so GitHub Actions and GitHub-app bots are out.
• Most commits are made by agents (~300/month across the portfolio), so per-seat pricing is the wrong unit.

Four layersCopy Anchor Link

Results are stored in Convex as release gates, attached to the Entire trail, and gate the staging → production promotion.

Unit economicsCopy Anchor Link

Cost per verified commit is roughly $0.02–0.10, recorded per App so that it shows up in each App's cost to serve.

ConsequencesCopy Anchor Link

• Every App gains a verify block in project.yaml, and onboarding includes declaring its two or three golden journeys.
• Third-party review engines start out better tuned. Exo tracks catch rate against false positives per layer and will revisit buying the review layer if the gap persists.
• Live checks need a staging-only sign-in path for bots (seeded Exo Key test identities), because browser agents cannot pass Google's bot detection.
• Open question: does Entire expose webhooks or a checks API?

StatusCopy Anchor Link

Built in part on staging (2026-10-01). What runs today:

• Contract + CLI (packages/exo-verify, zero dependencies): a verify: block in project.yaml (typecheck, lint, a list of test commands, build, smoke checks per env, mode: enforce|report). exo-verify run [--gate] [--json <path>] and exo-verify smoke --env staging|production. A missing command counts as a gap, never a pass. Only enforce mode exits non-zero.
• Results: POST /v1/verify/runs (service token) stores rows in the verifyRuns Convex table. GET /v1/verify/status returns the latest status for each app.
• Build gates: Exo's and Observatory's Railway builds run exo-verify run --gate in enforce mode. Construct (12 existing lint errors) and Rampart (tests assume a root base path) run it in report mode. Apps use a vendored copy (scripts/exo-verify.mjs) from exo-verify vendor.
• Live checks: a Convex cron runs every 15 minutes. It makes HTTP GETs to each app's declared smoke routes on staging and production. For drift, it records whether production's /api/health commit differs from staging's and since when. It also checks production domains over DNS-over-HTTPS.
• Pre-push hook: exo-verify install turns on a versioned .githooks/ for one worktree. The hook runs the gate and then hands off to the existing .git/hooks (Entire). To skip once, set EXO_VERIFY_SKIP=1. The skip is always written to a local log. It is also posted as a skip run (best effort) when EXO_VERIFY_URL and EXO_VERIFY_TOKEN are set.
• AI review: exo-verify review --base <ref> runs PR-Agent (MIT, pinned pr-agent==0.46.0, plain-diff mode) through exo-router's OpenAI-compatible POST /v1/chat/completions, which logs per-app tokens and cost in exoAiBrokerJobs. Off by default: the Convex AI gateway is not enabled for the team, so the endpoint returns 503 exo_router_unavailable, records the failed job, and the CLI prints review skipped: <reason>.

Measured cost per run:

Not built yet: browser journeys (Playwright, Apache-2.0), Entire triggers, and sandboxed Railway workers. Vitest (MIT) is the test runner for Exo itself.