Reviewing agent work and correcting specifications

An agreed contract can be ready for its next commitment and still be unusable for the agent receiving it. The document may be inaccessible, the required dependency may not run in the available environment or the assignment may leave a product decision implicit. Changing the product requirement would fix none of those failures.

Part 3 separated verification from validation and the oracle problem. This part follows the contract into implementation: can this agent use the agreement, does the bounded result conform and what happens when evidence challenges it? Conformance evidence can show that code follows a rule; it cannot show that the rule still serves its need. See Part 3’s discussion before treating a passing check as acceptance.

Give the agent a workable assignment, then review the result

Is the spec good enough for this agent?

Before work starts, state the work, repository state, supporting material, available checks and decisions the agent may take. Keep the current contract, historical reasons and proposed work distinguishable. Repository instructions describe conventions; design records carry agreed mechanisms; a task record carries questions and evidence. Part 2 discusses supplemental guidance, including the AGENTS.md evaluation.

Check that:

• Current rules and critical constraints are accessible, and the agent can distinguish governing references from superseded decisions.
• The relevant repository and tools can be used; unavailable dependencies and checks are identified.
• Scope, preservation requirements and implementation discretion are explicit, including product questions on which dependent work must wait.
• The assignment fits both the agent’s ability to retain relevant context and the reviewers’ ability to examine the result.
• Expected verification can be performed at the relevant boundary.

If a gap blocks the assignment, repair access or tools, revise verification or narrow the deliverable to supported work or an investigation. A narrower assignment says what this environment can establish; it does not make an unsupported guarantee acceptable.

Answer repository questions through inspection. For consequential or unfamiliar work, ask for a read-back of important outcomes, governing rules, materially different interpretations and intended checks. Use the read-back to find gaps, not to certify an agent because its paraphrase sounds convincing. A small trial can expose capability or environment problems the read-back leaves uncertain.

Build one slice across the relevant integration boundaries

A vertical slice should reach an observable result through the integration boundaries that matter to it. For an export, that might be one authorised request through to a complete downloadable file, including denial and failure cases. Synchronous or asynchronous, either must satisfy the contract.

Keep the increment small enough to examine implementation and evidence together. Choose boundaries, fixtures and failure paths before the agent hides them behind a convenient fake. Conformance review establishes whether checks reach those boundaries and detect claimed violations. Record what the slice exercised and which environment or fixture limits remain.

Does the code correctly implement the spec?

Start with an identified contract version and code revision or recorded workspace state. Attach the evidence to that code state, the relevant configuration and the checks performed. A result obtained before a change may no longer describe the reviewed implementation; rerun the checks whose evidence the change could invalidate.

The review has four parts:

1. Connect consequential obligations to the implementation paths responsible for them and to their evidence. Include preserved behaviour and shared invariants.
2. Follow those paths through success, rejection, partial failure and relevant interactions. Look for omissions, unintended side effects and behaviour introduced outside the agreed scope.
3. Inspect the checks. They need to reach the relevant boundary, use justified expectations and detect the claimed violation. Use a targeted negative control where necessary to establish that a critical check does useful work.
4. Record each assessed obligation as supported by stated evidence, violated or unverified. Explain exclusions and preserve the conditions and limits of the result.

Check for obligations the suite never exercises. Critical unverified obligations remain unresolved. A check that passes because its fixture, fake or expected output repeats the implementation’s premise is evidence about that setup, not necessarily the contract’s outcome.

Record who owns acceptance and what they decided: accept the bounded deliverable, return it for correction or defer or narrow it pending evidence. Acceptance requires resolving violations and unknowns that block that deliverable. Changing a requirement invokes the disagreement process; it is not an implicit escape from a failing check.

Take the small contract through acceptance

Return to the contact-deduplication contract in Part 2. Its v1 operation uses a non-empty contact_id with exact, case-sensitive identity, retains the first record in original order without mutation and rejects the entire operation on an invalid ID; email matching, field merging and persistent writes are outside its scope.

Suppose the next assignment implements that operation in an existing library. Supply contract version 1, the enclosing API’s signature and error convention, relevant repository paths and runnable checks.

Scroll horizontally to see all columns.

This is an implementation-conformance record. Acceptance also depends on the prior judgement that the contract’s purpose and rules are justified, as Part 3 explains. If the code or contract changes before merging, reassess affected evidence; no promise about every future contact feature is implied.

Diagnose disagreement before changing the contract

The specification states intended behaviour. Code, deployed configuration and observation supply evidence of actual behaviour. When they disagree, editing whichever is easier can hide the information needed to identify the fault. Classify the finding first.

Scroll horizontally to see all columns.

An explicit change in required behaviour can justify changing a test. Weakening the expectation solely to accommodate a defect destroys evidence. Approval cannot make an incorrect requirement correct, but discovering an error does not silently revoke a promise to consumers either.

What if the spec was faulty?

When a requirement is faulty, code that deviates from it may already deliver the intended behaviour. Assess actual behaviour and intended outcome separately.

The word corrected needs a basis. Suppose an agent is implementing contract version 1 and version 2 contains a corrected permission rule. If the policy already applied when version 1 was approved, the specification omitted an obligation. If adopted later, it is a changed requirement with transition obligations. If the environment no longer supplies an assumed guarantee, reassess that guarantee and recovery behaviour. New wording does not decide which case occurred.

When evidence undermines a requirement:

1. Preserve the counterexample, the applicable rule version and the evidence showing why the expected outcome is wrong, missing or impossible.
2. Revisit the domain decision with the relevant expertise. Establish whether the cause is an original mistake, a changed need or a failed environmental assumption.
3. Identify affected examples, checks, code, dependent work and consumers. For shipped behaviour, examine consequences for existing data and operations before choosing recovery steps.
4. Record the corrected decision and its reason, update the current rule and acceptance evidence, and reconcile dependent assignments. Change code where its behaviour conflicts with the corrected contract.
5. Check conformance to that contract and validate the disputed outcome using evidence beyond the old spec. Preserve unrelated decisions and the history of the correction.

After a correction, maintain a readable current contract. Mark cancelled requirements as cancelled, and give resumed agent sessions the active scope and decisions. Decide which completed change records become history and how accepted changes enter the current contract. Spec Kit leaves that maintenance strategy to teams; OpenSpec supports updating planning artefacts during implementation. The responsibility to reconcile them remains with the team.

Appendix A. Evidence and limitations

This appendix entry covers two studies relevant to contract history. They inform engineering judgement, not a universal format, context budget or workflow for changing product discovery. Part 2 covers clarification, ambiguity and repository instructions; Part 3 covers verification, validation and formal models; Part 5 covers workflow cost.

Does more context help?

Additional repository instructions and the history of a specification are different kinds of context. The AGENTS.md evaluation is discussed in Part 2.

SpecPath, August 2026 preprint

SpecPath held the final contract constant while varying its presentation history through duplication, overrides, cancellation and split instructions. Here, a run counts as successful when it passes every final-contract probe for the task. Averaging those success rates with equal weight for each of five source tasks gave 78.8% for direct instructions and 78.7% across the four alternative histories.

The study also reported 127 complete groups of runs, drawn from 210 eligible groups, holding the task, model deployment, agent scaffold and repeat number fixed across all five histories. Of those complete groups, 100 succeeded with direct instructions; 35 of those 100 failed under at least one equivalent history. These repeated runs still represent only five source tasks, and the study does not establish that missing results were random.

The practical concern is whether an agent incorporated the applicable contract after instructions changed. Supply a clear current contract and check the resulting behaviour. This is an engineering response; the study does not establish its effectiveness or one uniquely effective documentation strategy.

Execution began after the complete history was available. The study does not establish a single cause of the individual differences, and it did not evaluate ongoing product discovery. Its equivalent final contracts therefore do not answer what should happen when evidence changes the product requirement.

What happens when requirements arrive progressively?

When the Specification Emerges, March 2026 preprint

This study compared implementations derived from 20 machine-learning papers when a fixed target was supplied initially or disclosed progressively. On a composite fidelity scale of 0–1, progressive disclosure scored 0.116 lower on average for Claude Code and 0.071 lower for Codex. The metric combines faithfulness to the specified computation with structural integration, including reuse of earlier modules; it is not a count of correct outputs.

The semantic subscore, on a 0–4 scale, fell from 3.031 to 2.718 for Claude Code; Codex changed from 3.245 to 3.242. The lower composite scores therefore do not represent the same kind of fidelity difference for both platforms. The rubric can distinguish formulation and structural fidelity even where an implementation is functionally equivalent, so these scores do not establish defect rates.

The result motivates attention to retaining and reconciling requirements across long interactions. Its scope is limited to two coding platforms, synthetic interactions and partly model-based scoring. The target was fixed throughout: the study did not evaluate product discovery in which learning changes what should be built.

Reconcile the active assignment

Return to work already running. Suppose an agent is implementing contract version 1 when its assignment is updated to version 2 with a corrected permission rule. Establish the rule’s status using the evidence and policy timing described above, then identify and pause the affected work until the agent has the changed rule and can revise its plan. Independent work can continue.

At the next review, identify the version used and reconcile implementation and evidence with the applicable contract before acceptance. Updating a shared document does not establish that an active agent incorporated the change; a result correct against version 1 may no longer be acceptable. Preserve the counterexample and unrelated commitments. The invoice-export walkthrough in Part 5 applies this reconciliation to a changed access decision.