Ten years making sure simulation
software didn't fail. Then GenAI
rewrote the rules of testing.

Visual QA was the slowest gate on every release I'd ever owned — 15 to 30 minutes of a human clicking through screens looking for the kind of drift no test asserts on. The bug isn't in the diff, it's in the didn't-notice.

A Playwright + Claude-Vision loop renders every front-end change, scores it against a design system, and runs a bounded fix loop until the screen conforms. Deterministic pixel diff and design-token assertions are the hard pass/fail; the vision critic stays advisory. That hybrid keeps a non-deterministic model from ever wrongly failing a build.

Hold the cost line earlier. Prompt-caching the design system dropped per-run cost to ~$0.01 — I should have wired that in on day one, not after the first bill.

Most "AI for testing" tools generate a handful of cases and stop. The work that actually eats a senior QE's day — risk strategy, traceability, triaging which failures are flaky vs. real — gets handed straight back. I wanted the pipeline a QE lead would build if they had ten engineers each owning one stage.

Ten focused agents own the lifecycle end-to-end: Scout walks the repo, Strategist produces a risk matrix and equivalence-partition tables, Scribe writes Gherkin, Crafter writes Playwright, Deployer opens the PR, Runner dispatches CI, Triage classifies failures. A Gatekeeper scores every handoff ≥8.0/10 or retries upstream up to 3× before escalating. Every artefact records its parent, gate score, and attempt — fully auditable chain.

Specialisation beats one big agent. Sonnet for the synthesis stages, Haiku for the comparison-heavy ones with cacheable prompts. The gating loop is the part that makes non-deterministic models safe to put in a CI position.

Authoring API tests from an OpenAPI spec is mechanical work — and yet a senior engineer's whole day disappears into it. The schema already tells you what the happy-path, boundary, and auth-bypass cases look like. Why is anyone still typing them out?

Parses any OpenAPI 3.x spec, scores each endpoint by risk tier (Low / Med / High / Critical) with a LangChain agent, then generates four test categories per endpoint — happy path, boundary, auth bypass, malformed input. Pydantic contracts on every LLM output mean zero malformed cases reach CI. Cosine-similarity dedup at 0.92 threshold cuts test bloat 40%+ before the suite ever runs.

Schema-enforced LLM output is the difference between a demo and a tool you'd actually wire into CI. LangSmith made token cost a number I could explain in a review — that's what made the project legible to non-QE stakeholders.

Terraform PRs ship security regressions silently. Static scanners flag the same hundred findings on every run, so the signal that matters — what's new in this PR — gets buried in noise. The team stops looking.

An end-to-end pipeline that lives entirely inside GitHub Actions — no AWS credentials, no external infra. Checkov (CIS/NIST-mapped) + tfsec + Trivy scan the plan; findings diff against the main baseline to isolate true regressions; pytest invariants per domain gate the PR by severity; Claude Haiku triages each new finding with exploitability scoring and flags the auto-remediable ones; an auto-fix PR lands a minimal HCL patch back on the contributor's branch. A public dashboard publishes the posture-score delta every run.

Baselines are what turn raw scanner output into a working CI gate. Teams don't react to a number that's been red for a year — they react to a delta. The dashboard kept the right metric visible: not absolute findings, but regressions per PR.

Flaky locators are the single biggest tax on E2E suites. Every selector break is hours of debugging that doesn't ship value — and the team that pays it is the same team writing the next set of features.

The framework detects a broken Playwright locator at run time, queries an LLM with the live DOM, generates a repaired selector, and files an auto-PR — turning hours of investigation into zero human intervention per failure. A Next.js dashboard shows healed-selector history per run, across stable and breaking app versions side-by-side.

The dashboard mattered more than the model. Without the audit trail, self-healing feels like a black box and teams won't trust the auto-PRs — with it, they treat the healed selector like any other reviewable diff.