Seconds after pasting a story, the AI produced a full Gherkin suite.

Coverage looked impressive. Tags were neat. The scenarios passed in CI.

Then review found the problem: a scenario had the right name but the wrong assertion. It checked that a message appeared. It never checked whether the refund was actually processed.

The problem was not that AI generated tests. The problem was that nobody inspected what those tests actually asserted.

This article is about what goes wrong when teams rely on AI-generated test cases without a discipline for critiquing the output.

The Better Question

Most teams start with: “Can AI generate tests from this story?”

Article 1 showed that the answer is yes, especially when you give it a structured decomposition and a clear coverage level. Article 5 showed how to teach AI your team's style so the output looks like it belongs in your suite. Article 6 showed how to validate requirements before generation.

By Article 7, the better question is: “What can go wrong in the tests AI generates, and how do we catch those failures before we trust them?”

AI is good at volume, consistency, and format. It is bad at judgment, domain nuance, and knowing your existing suite.

Output critique is the tester's craft for closing that gap.

The Coupon Story, Revisited

To keep continuity, we return to the same coupon checkout story used throughout this series:

As a checkout user,
I want to apply a discount coupon,
so that I pay less for my order.

In Article 1, we decomposed this into scenarios across coverage levels. In Article 6, we interrogated it with seven requirement gap questions.

Now assume the story has been cleaned up by specification linting, the team chose Standard coverage, and reference examples are in place. The AI generates a fresh set of coupon scenarios.

What can go wrong from here?

Failure 1: Hallucinated Assertions

AI can invent behavior that no requirement ever specified.

Scenario: Coupon applies to shipping
  Given I have a cart with items totaling $50
  And a valid coupon "SHIPFREE"
  When I apply the coupon
  Then my shipping cost should be $0
  And my cart total should remain $50

If your product has no concept of “coupon that only applies to shipping,” this scenario is a hallucination. It describes plausible behavior that does not exist in your domain.

Why it happens: The model imports patterns from training data that do not apply to your system. The story mentioned “discount coupon” but never defined which parts of the order can be discounted.

How to catch it: Compare each Then step against documented behavior. If an assertion cannot be traced to a requirement, acceptance criterion, or existing system capability, flag it. The scenario may be suggesting a feature that does not exist.

Failure 2: Inverted Logic

AI can get the condition backwards.

A scenario tests that a discount applies when the cart total exceeds $100. But the actual business rule is that the discount applies to orders up to $100, to encourage price-sensitive customers to complete checkout rather than abandon. The AI assumed the common pattern (rewards scale with spending) rather than the actual rule.

Scenario: High-value cart qualifies for loyalty discount
  Given I am a loyalty member
  And my cart total is $150
  When I proceed to checkout
  Then a 10% loyalty discount should be applied

The scenario is internally consistent. The logic is simply wrong.

Why it happens: The model infers business rules from patterns rather than reading them from the requirement. When the actual rule contradicts the common pattern, the model follows the pattern.

How to catch it: Review scenarios against the actual business logic, not just the requirement text. If a scenario's logic “makes sense” but contradicts what the system actually does, the AI inferred rather than read.

Failure 3: Implementation Leakage

AI tests how the system works instead of what it does.

Scenario: Apply coupon via checkout API
  Given I POST to /api/v2/cart/coupon with code "SAVE20"
  When the coupon_validation_service returns status 200
  Then the cart.discount_amount field should equal 20

This tests the API contract, not the user behavior. It couples the suite to implementation details. When the implementation changes, the test breaks even though the behavior is unchanged.

Why it happens: If reference examples or story text include implementation details, the model reproduces them. If the training data included technical specs, the model may default to that register.

How to catch it: Apply the readability test from Article 4: could someone unfamiliar with the codebase read this scenario and understand what behavior is being tested? If the answer requires knowledge of endpoints, field names, or internal services, the scenario has implementation leakage.

If a scenario asserts pure calculation logic, it may also belong at a different layer entirely. Article 3 covers where each test earns its cost.

Failure 4: Coverage Illusion

The AI generates many scenarios that test the same thing differently.

Scenario: Apply 10% coupon to $50 cart
Scenario: Apply 15% coupon to $75 cart
Scenario: Apply 20% coupon to $100 cart
Scenario: Apply flat $5 coupon to $30 cart
Scenario: Apply flat $10 coupon to $60 cart

Five scenarios, one behavior: valid coupon reduces total. The negative paths from Article 6 are missing. No expiration handling. No invalid code handling. No reuse prevention.

Why it happens: The model generates variations easily. Without explicit negative-path requirements, it defaults to what was specified: success cases.

How to catch it: After generation, categorize scenarios by the behavior they test, not by the inputs they use. If multiple scenarios collapse to the same assertion under different conditions, the coverage is narrower than it appears. Counts of passing scenarios are not coverage; they can create fake confidence if multiple tests assert the same behavior under different inputs.

Check specifically for L4 (missing negative path) coverage from Article 6.

Failure 5: Stale Pattern Drift

The AI reproduces patterns from outdated reference examples.

The team updated its Gherkin conventions six months ago. New scenarios use first person (“Given I am logged in”). But the reference examples still use third person (“Given the user is logged in”). The AI follows the examples, not the current standard.

# Current team convention
Given I have items in my cart

# Generated scenario (following stale examples)
Given the user has items in their cart

The scenario is valid. It just does not match how the team writes tests now.

Why it happens: Reference examples are few-shot training data. The model mirrors what it sees. If the examples are stale, the output is stale.

How to catch it: Review generated scenarios against current conventions, not historical ones. If the output matches the reference examples but not the team's current style, the examples need updating. This is the maintenance discipline from Article 5: reference examples are a living artifact.

What To Do Monday

Three moves you can run this week.

Sample and classify ten scenarios. Pick ten AI-generated scenarios from a recent story. For each one, tag it with one of the five failure modes: hallucinated assertion, inverted logic, implementation leakage, coverage illusion, or stale pattern drift. Or tag it “reduces real risk.” Two goals: see which failure modes show up most often, and build a shared vocabulary for talking about output quality.

Add three output critique questions to your review checklist. Extend your existing scenario review with: “Does this scenario assert behavior we actually decided?” “Is this the cheapest layer that can catch this risk?” “If this passes, what risk does it reduce?” Keep it short.

Decide where AI-generated tests live. Make an explicit decision: which stories will use AI-generated scenarios, where those scenarios live in your repo, and who owns reviewing and maintaining them. Document that decision in your test strategy. Without it, AI-generated tests remain an experiment rather than part of your practice.

Where Tooling Fits

You can run this critique process with a highlighter and a checklist. That works for a handful of stories. At scale, you need help.

Output critique automates part of this review. It compares generated scenarios against the original requirements and flags gaps, redundancies, missing negative paths, and untested edge cases.

But the review judgment remains human. The tool surfaces candidates for attention. The reviewer decides whether a flagged scenario is genuinely wrong or correctly extends beyond the explicit requirement.

The goal is not to eliminate review. The goal is to focus review on the scenarios most likely to be wrong.

The Trust Equation

Trust in AI-generated tests comes from layered verification:

Layer Article What It Checks
Validate Article 6 Are the requirements complete enough to generate from?
Calibrate Article 5 Does the output match the team's style?
Trust This article Is the generated content semantically correct?

Each layer catches different failures. Skipping a layer does not save time. It moves the failure downstream where it costs more to fix.

Summary

AI can generate test cases. That is not the same as generating the right test cases.

The five failure modes, hallucinated assertions, inverted logic, implementation leakage, coverage illusion, and stale pattern drift, are semantic errors that pass syntax checks. They require review, not just validation.

The discipline is not generating more tests. The discipline is catching the tests that look correct but are not.

Trust is earned by reviewing what AI produces, not by assuming that clean inputs guarantee correct outputs.