Reviewing agent work and correcting specifications

An agent cannot implement a contract it cannot retrieve, run a check whose dependency is unavailable, or safely settle a product question omitted from its assignment. Repair the assignment at the point where it fails. Rewriting a valid requirement would leave those problems intact.

Give the agent a workable assignment, then review the result

Is the spec good enough for this agent?

Before work starts, state the work, repository state, supporting material, available checks and decisions the agent may take. Keep the current contract, historical reasons and proposed work distinguishable. Repository instructions describe conventions; design records carry agreed mechanisms; a task record carries questions and evidence. Part 2 discusses supplemental guidance, including the AGENTS.md evaluation.

Check that:

• Current rules and critical constraints are accessible, and the agent can distinguish governing references from superseded decisions.
• The relevant repository and tools can be used; unavailable dependencies and checks are identified.
• Scope, preservation requirements and implementation discretion are explicit, including product questions on which dependent work must wait.
• The assignment fits both the agent’s ability to retain relevant context and the reviewers’ ability to examine the result.
• Expected verification can be performed at the relevant boundary.

If a gap blocks the assignment, repair access or tools, revise verification or narrow the deliverable to supported work or an investigation. A narrower assignment says what this environment can establish; it does not make an unsupported guarantee acceptable.

Answer repository questions through inspection. For consequential or unfamiliar work, ask for a read-back of important outcomes, governing rules, materially different interpretations and intended checks. Use the read-back to find gaps, not to certify an agent because its paraphrase sounds convincing. A small trial can expose capability or environment problems the read-back leaves uncertain.

Build one slice across the relevant integration boundaries

A vertical slice should reach an observable result through the integration boundaries that matter to it. For an export, that might be one authorised request through to a complete downloadable file, including denial and failure cases. Synchronous or asynchronous, either must satisfy the contract.

Keep the increment small enough to examine implementation and evidence together. Choose boundaries, fixtures and failure paths before the agent hides them behind a convenient fake. Conformance review establishes whether checks reach those boundaries and detect claimed violations. Record what the slice exercised and which environment or fixture limits remain.

Does the code correctly implement the spec?

Specification review asks whether the current requirements are justified, consistent, feasible and verifiable. Conformance review asks whether the implementation and its evidence satisfy the applicable requirements. Keep those conclusions separate: a conformance failure can require a code repair while the requirement remains settled, and a faulty requirement can require a contract decision even when the code follows it.

Start with an identified contract version and code revision or recorded workspace state. Attach the evidence to that code state, the relevant configuration and the checks performed. A result obtained before a change may no longer describe the reviewed implementation; rerun the checks whose evidence the change could invalidate.

The review has four parts:

1. Connect consequential obligations to the implementation paths responsible for them and to their evidence. Include preserved behaviour and shared invariants.
2. Follow those paths through success, rejection, partial failure and relevant interactions. Look for omissions, unintended side effects and behaviour introduced outside the agreed scope.
3. Inspect the checks. They need to reach the relevant boundary, use justified expectations and detect the claimed violation. Use a targeted negative control where necessary to establish that a critical check does useful work.
4. Record each assessed obligation as supported by stated evidence, violated or unverified. Explain exclusions and preserve the conditions and limits of the result.

Check for obligations the suite never exercises. Critical unverified obligations remain unresolved. A check that passes because its fixture, fake or expected output repeats the implementation’s premise is evidence about that setup, not necessarily the contract’s outcome.

Record who owns acceptance, what object they accepted and what they decided: accept the bounded deliverable against the named contract and code state, return it for correction or defer or narrow it pending evidence. Acceptance requires resolving violations and unknowns that block that deliverable. Changing a requirement invokes the disagreement process; it is not an implicit escape from a failing check.

Take the small contract through acceptance

Consider an assignment to implement DEDUP-01 through DEDUP-05 from the contact contract in Part 2 in an existing library. Supply contract version 1, the enclosing API’s signature and error convention, repository paths and runnable checks. The following record illustrates a conformance decision if those checks had run; it claims no actual execution or product validation.

Scroll horizontally to see all columns.

This is an implementation-conformance record. Acceptance also depends on the prior judgement that the contract’s purpose and rules are justified, as Part 3 explains. If the code or contract changes before merging, reassess affected evidence; no promise about every future contact feature is implied.

Repair ordinary defects before changing the contract

The specification states intended behaviour. Code, deployed configuration and observation supply evidence of actual behaviour. When they disagree, editing whichever is easier can hide the information needed to identify the fault. Classify the finding first.

For an ordinary defect, keep the applicable requirement fixed: reproduce the failure at the relevant boundary, trace the cause, repair the implementation or test setup, and rerun the affected checks against the repaired workspace. Review shared consumers and failure effects before acceptance. This path ends with a conformance decision; it does not require rewriting the contract or invoking every specification skill.

Scroll horizontally to see all columns.

An explicit change in required behaviour can justify changing a test. Weakening the expectation solely to accommodate a defect destroys evidence. Approval cannot make an incorrect requirement correct, but discovering an error does not silently revoke a promise to consumers either.

What if the spec was faulty?

When a requirement is faulty, code that deviates from it may already deliver the intended behaviour. Assess actual behaviour and intended outcome separately.

The word corrected needs a basis. Suppose an agent is implementing contract version 1 and version 2 contains a proposed revised permission rule. If the policy already applied when version 1 was approved, the specification omitted an obligation. If adopted later, it is a changed requirement with transition obligations. If the environment no longer supplies an assumed guarantee, reassess that guarantee and recovery behaviour. New wording does not decide which case occurred.

When evidence undermines a requirement:

1. Preserve the counterexample, the applicable rule version and the evidence showing why the expected outcome is wrong, missing or impossible.
2. Revisit the domain decision with the relevant expertise. Establish whether the cause is an original mistake, a changed need or a failed environmental assumption.
3. Identify affected examples, checks, code, dependent work and consumers. For shipped behaviour, examine consequences for existing data and operations before choosing recovery steps.
4. Record the decision and its reason once the basis is established, update the current rule and acceptance evidence, and reconcile dependent assignments. Change code where its behaviour conflicts with the revised contract.
5. Check conformance to that contract and validate the disputed outcome using evidence beyond the old spec. Preserve unrelated decisions and the history of the correction.

After a correction, maintain a readable current contract containing only obligations applicable to its scope and version. Remove withdrawn obligations from that contract and preserve their text, reason and timing in history; do not leave a cancelled rule looking current. Give resumed agent sessions the active scope and decisions. Decide which completed change records become history and how accepted changes enter the current contract. Spec Kit leaves that maintenance strategy to teams; OpenSpec supports updating planning artefacts during implementation. The responsibility to reconcile them remains with the team.

Reconcile the active assignment

Return to work already running. Suppose an agent is implementing contract version 1 when its assignment is updated to version 2 with a proposed revised permission rule. Establish whether the rule was already applicable or newly adopted using the evidence and policy timing described above, then identify and pause the affected work until the agent has the applicable rule and can revise its plan. Independent work can continue.

At the next review, identify the version used and reconcile implementation and evidence with the applicable contract before acceptance. Updating a shared document does not establish that an active agent incorporated the change; a result correct against version 1 may no longer be acceptable. Preserve the counterexample and unrelated commitments. The invoice-export walkthrough in Part 5 applies this reconciliation to a changed access decision.

Appendix A. Evidence and limitations

This appendix entry covers two studies relevant to contract history. They inform engineering judgement, not a universal format, context budget or workflow for changing product discovery. Part 2 covers clarification, ambiguity and repository instructions; Part 3 covers verification, validation and formal models; Part 7 covers workflow cost.

Does more context help?

Additional repository instructions and the history of a specification are different kinds of context. The AGENTS.md evaluation is discussed in Part 2.

SpecPath, August 2026 preprint

SpecPath held the final contract constant while varying its presentation history through duplication, overrides, cancellation and split instructions. Here, a run counts as successful when it passes every final-contract probe for the task. Averaging those success rates with equal weight for each of five source tasks gave 78.8% for direct instructions and 78.7% across the four alternative histories.

The study also reported 127 complete groups of runs, drawn from 210 eligible groups, holding the task, model deployment, agent scaffold and repeat number fixed across all five histories. Of those complete groups, 100 succeeded with direct instructions; 35 of those 100 failed under at least one equivalent history. These repeated runs still represent only five source tasks, and the study does not establish that missing results were random.

The practical concern is whether an agent incorporated the applicable contract after instructions changed. Supply a clear current contract and check the resulting behaviour. This is an engineering response; the study does not establish its effectiveness or one uniquely effective documentation strategy.

Execution began after the complete history was available. The study does not establish a single cause of the individual differences, and it did not evaluate ongoing product discovery. Its equivalent final contracts therefore do not answer what should happen when evidence changes the product requirement.

What happens when requirements arrive progressively?

When the Specification Emerges, March 2026 preprint

This study compared implementations derived from 20 machine-learning papers when a fixed target was supplied initially or disclosed progressively. On a composite fidelity scale of 0–1, progressive disclosure scored 0.116 lower on average for Claude Code and 0.071 lower for Codex. The metric combines faithfulness to the specified computation with structural integration, including reuse of earlier modules; it is not a count of correct outputs.

The semantic subscore, on a 0–4 scale, fell from 3.031 to 2.718 for Claude Code; Codex changed from 3.245 to 3.242. The lower composite scores therefore do not represent the same kind of fidelity difference for both platforms. The rubric can distinguish formulation and structural fidelity even where an implementation is functionally equivalent, so these scores do not establish defect rates.

The result motivates attention to retaining and reconciling requirements across long interactions. Its scope is limited to two coding platforms, synthetic interactions and partly model-based scoring. The target was fixed throughout: the study did not evaluate product discovery in which learning changes what should be built.