Verification-First Engineering

Rationale

Verification should be driven by explicit success criteria rather than trust in the initial output. The value lies in making the agent iterate against independent checks.

Common verification targets include:

• Syntactically correct but semantically wrong code
• Plausible-looking but incorrect solutions
• Code that works in isolation but breaks integration
• Hallucinated APIs or non-existent functions

The main issue is not only that agents can be wrong. It is that they are often wrong in ways that look locally plausible. Verification therefore needs to be designed as a mechanism for falsification rather than as a final sanity check.

How to Apply the Pattern

Verification should be specified before trust is granted.

1. Define objective success criteria for the task.
2. Choose the cheapest checks that could disconfirm the current output.
3. Run those checks immediately after each edit or step.
4. Review the actual diff and resulting complexity, not only the command output.
5. Continue only when the current slice is independently supported.

In practice, this means verification is not a single late-stage activity. It is the main control loop around execution.

Supporting Practices

Use Layered Verification

Different checks catch different classes of failure.

Design Verification to Catch Common Failure Modes

Keep the Verification Loop Tight

Agent Output
     │
     ▼
┌────────────┐   fail   ┌──────────────┐
│   Parse    │────────▶   Fix Syntax   
└─────┬──────┘          └──────┬───────┘
     │ pass                    │ retry
     ▼                         └──────────↺
┌────────────┐   fail   ┌──────────────┐
│    Lint    │────────▶   Fix Issues  
└─────┬──────┘          └──────┬───────┘
     │ pass                    │ retry
     ▼                         └──────────↺
┌────────────┐   fail   ┌──────────────┐
│    Test    │────────▶   Fix Logic   
└─────┬──────┘          └──────┬───────┘
     │ pass                    │ retry
     ▼                         └──────────↺
 Verified

Define Explicit Success Criteria

Use objective conditions to determine whether a task is complete.

## Success Criteria for This Task

- [ ] All existing tests pass
- [ ] New tests cover the added functionality  
- [ ] No linting errors
- [ ] No type errors
- [ ] Diff shows only intended changes (no dead code)
- [ ] Solution uses ≤ N lines / ≤ N files (complexity gate)

The task is complete only when all criteria are satisfied.

Guard Against Requirement Drift and Sycophantic Testing

AI agents often produce an initial result quickly, but the remaining work still includes edge cases, subtle bugs, security considerations, and architectural coherence. In some cases, the final portion of the task takes longer than expected because the developer must debug code they did not write and do not fully understand.

A related failure mode is sycophantic testing: agents write tests that match the implementation they just produced rather than tests that validate the requirements. The result may be a test suite with high coverage but low validation value.

Practical implications for verification:

• Write tests before asking the agent to implement (test-driven development prevents sycophantic tests)
• Build test suites that verify behaviour, not implementation (integration tests, property-based tests, scenario tests)
• When reviewing AI-generated tests, ask: does this test validate the requirement, or does it just confirm what the code currently does?

Review Failure Modes Systematically

Keep a Human Verification Checklist

Automated checks catch syntax and logic errors, but human review is the only mechanism that catches design issues, unintended scope changes, and the subtle failure modes listed above. Apply this checklist before accepting any non-trivial agent output:

Code that cannot be explained clearly should not be accepted without further review. Accepting code that is not understood creates a maintenance liability.

Practices

• Run static checks automatically after each agent edit
• Write tests before asking agents to implement
• Ask agents to verify their own work (“check if this compiles”)
• Use multiple verification methods in combination
• Automate verification in CI/CD
• Verify early so assumption propagation is caught before it compounds
• Check whether the solution could be simpler
• Review cleanup explicitly by checking the diff for unintended changes and dead code
• Use objective criteria such as tests and linters rather than subjective impressions

Anti-patterns

• Accepting agent code without running it
• Skipping tests because the code appears plausible
• Verifying only at the end of a large change
• Trusting claims that code “should work” without independent checks
• Not reviewing diffs for unintended changes
• Accepting complex solutions without questioning whether they can be simplified
• Relying on agent self-assessment

Indicators

• Share of tasks with explicit success criteria before implementation begins
• Time between an edit and its first independent verification step
• Frequency of defects caught by diff review, complexity review, or human review after tests pass
• Rate of regressions caused by changes that were never independently checked