BDD and acceptance evidence for coding agents

Using behavioural examples and evidence to decide whether a specification is ready for the next commitment.

Verification examines conformance to the applicable contract; validation examines whether the requirements and delivered behaviour serve the intended need. An export containing every specified column may pass verification while leaving finance unable to distinguish two invoices. The result exposes a question about the contract itself.

Both judgements depend on how the expected result was chosen. The test oracle problem concerns how to determine what the correct result should be. A second agent handed the same mistaken premise may faithfully reproduce it.

An independent basis must support or challenge the disputed expectation without assuming it is correct. Seek it in domain examples grounded in observed work, an applicable external contract, a justified reference or a separately reasoned property. A property derived only from a disputed rule can assess conformance to that rule; it cannot justify the rule itself. Examples explain interpretation, properties generalise obligations and experiments challenge assumptions. Choose the combination that addresses the risk.

For the revoked-download example below, the expected denial should come from the account-access policy and a domain-reviewed case, not from the permission helper or a fixture that calls the same helper. The implementation and the check may still share assumptions about which account owns the export and where a download request begins; record those assumptions and test the boundary they could invalidate. Record the policy’s source and applicable version, who supplied the example, and which assumptions came from the rule or implementation under dispute. A second agent reading the same material can expose omissions; its agreement adds no independent basis for that expectation.

Use BDD to discover and express acceptance criteria

“Export the filtered invoices” can sound settled until finance expects a snapshot and the developer expects whatever the database returns when the worker runs. A concrete example makes that disagreement visible: an invoice changes after the request is accepted but before the file is assembled. Which value belongs in the file? The answer changes the contract and may change the design.

Behaviour-driven development gives that discussion a place in development. It predates coding agents: Dan North’s original account connects TDD with analysis and acceptance testing through behaviour and shared language, and recognises that a failing test may expose an outdated premise. Cucumber describes the practice as an iterative combination of discovery, formulation and automation, applied to small upcoming changes and revisited as questions arise.

BDD is optional. Where it helps, its examples can supply much of a specification’s behavioural content without adding another document layer. North’s and Cucumber’s accounts describe practices and conventions; they do not establish an effect size for introducing BDD into agent-assisted development. The choice is whether the examples improve the decision the team needs to make.

Let concrete cases expose disagreement

Example Mapping separates rules, examples, questions and deferred scope. The disputed invoice value becomes an explicit question to resolve before it can become an acceptance example. Agreement on the example records the current decision; it does not make the decision true without the domain basis described above.

That requires perspectives capable of challenging one another. Domain expertise explains the need, development examines feasibility and boundaries, and testing probes examples and omissions. These are contributions to continuing collaboration; they do not require exactly three people or a new approval board.

An agent can propose edge cases, expose inconsistent wording and retrieve repository evidence. Assigning agents the personas of business representative, developer and tester does not, by itself, supply independent evidence of user intent. On a solo project, observations and domain sources still have to challenge the author’s assumptions; simulated agreement cannot do that work for them.

Discuss detailed scenarios close to the relevant implementation work, so their assumptions can be checked promptly. Cucumber recommends discovery workshops near the start of the relevant story’s development. Project-level risks and expensive feasibility questions may need attention earlier. Treating the entire product’s scenarios as settled before building anything would preserve the waterfall problem, however readable the notation.

Make scenarios part of the contract

Agreed BDD scenarios belong among the acceptance criteria: they state concrete situations and the outcomes required for acceptance. A specification should contain or link to the examples needed to clarify consequential behaviour, alongside the general rules. It need not enumerate the entire test suite. Examples without a general rule leave the implementer to infer what happens in every unlisted case.

Scroll horizontally to see all columns.

Test code connects the agreed expectations to the system through setup, execution and assertions. An automated scenario can therefore be both a readable acceptance criterion and an executable check; the roles above do not require separate documents. Not every acceptance criterion or test needs Given/When/Then.

Further cases can be derived from settled rules and remain in test code without a new product decision for each test. But if choosing the expected result would decide behaviour the contract leaves unresolved, return that concrete question to the requirements discussion. A generated assertion must not turn an unanswered question into an obligation.

Give each rule and scenario one maintained home. If a feature file owns an example, reference it from the specification’s entry point. Refactoring test code must preserve the agreed outcome. If a scenario misstates a rule or the requirement itself needs revision, use the decision process in Part 4 to correct the affected expectations.

Choose the notation for the people who will use it

The EARS requirements in Part 2 state general obligations. Scenarios describe concrete cases and can expose decisions those obligations leave unresolved.

Gherkin provides a scenario notation, and Cucumber provides tools that connect scenarios to executable checks. Collaborative examples and ordinary tests can serve the same development practice without a Gherkin runner. In a Go library, domain-reviewed cases may become table-driven tests. Add the translation layer when the shared executable representation helps people review and maintain behaviour; if its only readers already work comfortably in test code, it may add little.

Where Gherkin is useful, describe observable outcomes and keep replaceable mechanics in the automation. The Gherkin reference distinguishes a business rule from the scenarios illustrating it and directs assertions towards observable results. An unchanged domain rule should usually survive a move from a web interface to an API. Browser interactions belong in the scenario when the interaction itself is required; elsewhere, selectors and clicks can conceal the obligation being tested.

A revoked download request

The following scenario is a hypothetical acceptance example for a permission boundary under investigation. It concerns a new request after access has been revoked. It does not decide what happens to a transfer that was already authorised, and it does not establish that the rule is feasible or that the whole export is ready for release. The proposed export rules and investigation remain in Part 5’s export walkthrough.

Feature: Invoice export downloads

  Rule: EXP-05 — Each download request requires current account authorisation

    Scenario: Access is revoked after an export is prepared
      Given an administrator has a completed export for account "A"
      And the administrator no longer has access to account "A"
      When the administrator starts a new download request for that export
      Then the download is denied
      And no invoice data is returned

The rule applies at the download boundary, including a direct storage URL if the design uses one. A queue, object store, helper function or database layout remains a design choice.

Keep the examples alive during implementation

Automate selected examples at the boundary they concern, and establish that the checks detect the relevant violation. A discovered case may send the work back to the domain rule; an agreed case can remain as regression evidence. TDD supplies a smaller loop of failing test, implementation and refactoring within this work. It is feedback inside an authorised increment, not a substitute for deciding whether the increment is ready.

For the fictional download example, the connection can be made explicit without claiming that a test has run:

Scroll horizontally to see all columns.

A requirement ID makes this connection searchable; attaching it to a test does not establish that the assertions assess the rule. Review in both directions: which important obligations lack suitable checks, and which expectations in the checks lack a basis in the contract? One rule may need several cases, and one check may exercise several rules. Private implementation tests do not all need product requirement IDs.

Ask what plausible violation each check would detect. For EXP-05, a check that never revokes access cannot reject an implementation that reuses the permission decision from export creation. A fake that always denies access would merely demonstrate that the fake denies access. The setup must reach the state change, and the assertions must inspect the protected result. A controlled fault can help establish whether a check detects the violation; mutation testing automates some such changes. PIT’s explanation distinguishes this from code coverage, which records executed code without establishing that assertions detect its faults. Neither a passing test nor a coverage percentage establishes that all edge cases are covered.

When a rule changes, review its linked examples and assertions against the new decision, and rerun the affected checks. Keep the result attached to the state it examined. An intact reference to EXP-05 cannot tell you whether its test still asserts the applicable requirement.

Adding Cucumber after implementation may produce useful regression tests, but examples guide development only when they participate in its decisions. If the contract is not ready, implementation feedback should return the question for investigation or revision rather than silently settle it.

Assess the spec and decide what evidence acceptance needs

Choose evidence while agreeing the contract, when an obligation can still be examined rather than merely handed to an implementer:

Scroll horizontally to see all columns.

An edge case does not always call for another requirement. If the expected outcome is unclear, clarify the behaviour before dependent implementation. If the rule already determines the outcome but no check exercises the case, strengthen the evidence. If the behaviour is outside the agreed scope, keep that boundary explicit and settle the responsibility before work that depends on it; an excluded case is not a covered case.

Probe the boundaries relevant to the commitment: empty or invalid inputs, limits, failures, permissions, retries, ordering, time and interacting state changes. Use distinguishing examples and broader properties to challenge the rules and their implementation. Record the consequential cases that remain unverified rather than treating one linked test per requirement as completeness.

For suitability, examine user and operational evidence: can finance complete its reconciliation with the exported data? Usability and accessibility may need both checks against stated criteria and observation of actual use. Those observations can challenge the criteria themselves. A scenario or component check covers selected executions and interactions; it does not establish every concurrent execution or a system-wide property.

Amazon’s formal-methods experience shows how precise models can expose distributed-system design defects beyond ordinary reviews and tests. The result remains about the represented system and stated properties; establishing correspondence with an implementation is further work.

Is the spec good enough for the next commitment?

Clarity is necessary, but it can make a wrong requirement exceptionally easy to implement. Review the draft against its purpose and evidence:

• Justified: Consequential rules and the intended outcome have a credible domain basis; reviewers can explain why they matter.
• Consistent: Rules agree with one another and with applicable external and shared contracts. Deliberate changes are identified.
• Sufficiently complete: The increment’s important outcomes, failure effects and boundaries are covered, with remaining questions explicit.
• Feasible: Critical assumptions have supporting evidence, or investigating them is the bounded next assignment.
• Verifiable: Acceptable and unacceptable outcomes can be distinguished, and suitable evidence can be obtained.

The review should end in a concrete choice: implement the bounded scope, investigate the uncertainty controlling it, or revise the contract. That choice is about the next commitment, not a verdict on every future capability.

An unresolved point blocks dependent implementation when different answers could change a promised outcome, break a consumer, violate a shared invariant or invalidate the work. Resolve it or narrow the assignment. Independent later capabilities can remain open, as can private implementation choices that preserve the obligations. When the decision is to implement, Part 4 supplies the agent and environment readiness checks; when it is to investigate or revise, preserve the question, evidence and owner so the next assignment cannot turn silence into authority.