# AI Test Validator

Your test suite is green. 100% passing. Then production breaks and you realize the tests
were passing but they weren't testing anything.

**A test is only valuable if it can fail. If it can't detect broken code, it's worse than no
test at all — it's a lie.**

This skill does not read tests and opine. It **breaks the production code on purpose** and
checks whether the tests notice. Then it renders a verdict.

**Related skills** — this is step 3 of a 3-skill loop:
`vc-ai-first-tdd-planner` (plan the tests) → `vc-test-prompt-builder` (prompt for them) →
**`vc-ai-test-validator`** (the merge gate).

---

## Hard Rules

1. **The 80% mutation-score gate.** A suite below 80% mutation score does not merge. This is
   the number in `scripts/test-quality.yml` and it is not negotiable downward per-PR.
2. **The Nuclear Option thresholds.** Delete and rewrite from scratch — do not patch — if
   **any one** of these is true:
   - More than **50% of assertions are weak**
   - **No negative test cases at all**
   - **Heavy mocking that tests mocks, not code**
   - **Tests are all variations of the same scenario**
   - **You can't explain what the test actually validates**

   It's faster to rewrite than to fix fundamentally flawed tests.
3. **The one-line decision question.** For any test you're unsure about:
   **"Could this test pass with completely broken production code?"** If yes, delete it.
4. **The 6 Red Flags — delete these tests** (full list + rewrites in `references/red-flags.md`):
   - `assert result` — no specific value check
   - `assert result is not None` — truthy check only
   - Heavy mocking that eliminates failure paths
   - Test creates data then asserts on that same data
   - Tests private methods or attributes
   - No docstring explaining what scenario is tested
5. **A green test suite proves nothing if the tests can't fail.** "100% passing" is not the
   signal. Whether the suite fails when the code is broken is the signal.
6. **Never skip manual review.** Automated checks catch common issues but not all. Always
   manually review AI-generated tests, adversarially: *"If I were an attacker trying to break
   this code, would this test catch me?"*
7. **Emit a verdict.** Every run ends in a filled findings table with a per-file
   `KEEP` / `FIX` / `DELETE-AND-REWRITE`. A review with no verdict is not a review.

---

## Procedure

### Step 1 · Discover the stack — silently

**Silently.** Do not open with a questionnaire. Look first:

| Look at | To learn |
|---|---|
| `package.json`, `pyproject.toml` / `requirements.txt`, `pom.xml` / `build.gradle`, `go.mod` | language + test framework + whether a mutation tool is already installed |
| `git diff` / `git diff --stat HEAD~1` | which tests changed — validate those first |
| Glob `tests/`, `test/`, `spec/`, `**/*_test.py`, `**/*.test.ts`, `src/test/java/` | where the suite lives |
| `.github/workflows/` | whether a `test-quality.yml` already exists |
| The suite's mock density: grep for `mocker.patch`, `jest.mock`, `Mockito.mock` | over-mocking risk before you read a line |

Pick the mutation tool from what you found:

| Language | Tool | Command |
|---|---|---|
| Python | `mutmut` | `pip install mutmut && mutmut run` |
| JavaScript / TypeScript | `Stryker` | `npm install -g stryker && stryker run` |
| Java | `PITest` | `mvn org.pitest:pitest-maven:mutationCoverage` |

Ask only what the repo cannot tell you: the test command if it isn't in `package.json` /
`pyproject.toml`, and which module the user actually cares about if the suite is large.

### Step 2 · Run the assertion-strength greps

Run `scripts/assertion-strength-check.sh` (or the two greps directly):

```bash
grep -r "assert.*is not None$" tests/ && exit 1
grep -r "assert result$" tests/ && exit 1
```

Every hit is a weak assertion. Count them, and count total assertions — you need the ratio for
the **>50% weak** Nuclear Option threshold.

### Step 3 · Run mutation testing

Run the tool for the detected language. Capture two things:

- **the mutation score** (gate: **≥ 80%**)
- **which mutations survived** — a surviving mutant is a bug your suite cannot see

**Not comfortable in a terminal?** Delegate it. Tell your AI assistant, verbatim:

> "Set up mutation testing for this project and run it against our test suite. Tell me the
> mutation score and which mutations survived."

It will pick the right tool for your language (`mutmut` for Python, `Stryker` for JavaScript,
`PITest` for Java), install it, and report back what escaped detection.

### Step 4 · Apply the 3-mutation recipe by hand

Automated mutation testing is broad but shallow. For the **most important function under test**,
break it three ways yourself and confirm the test fails each time. The worked example
(`references/validation-patterns.md`, Pattern 1) uses `calculate_total(items, tax_rate)`:

```python
# Original implementation
def calculate_total(items, tax_rate):
    """Calculate total price with tax"""
    subtotal = sum(item.price for item in items)
    tax = subtotal * tax_rate
    return subtotal + tax

# AI-generated test
def test_calculate_total():
    items = [Item(price=10), Item(price=20)]
    total = calculate_total(items, 0.1)
    assert total == 33  # 30 + 3 tax = 33
```

The three mutations:

```python
# Mutation 1: Remove tax calculation
    return subtotal              # BUG: forgot to add tax
# Run test — does it fail? YES ✓

# Mutation 2: Double the tax
    tax = subtotal * tax_rate * 2   # BUG: double tax
# Run test — does it fail? YES ✓

# Mutation 3: Wrong operator
    return subtotal - tax        # BUG: subtraction instead of addition
# Run test — does it fail? YES ✓
```

**If the test fails for all three mutations, it's a good test. If it still passes with broken
code, the test is weak.** Restore the original code afterwards — always.

### Step 5 · Walk the 4 Validation Patterns

Full text, code, and rewrites in `references/validation-patterns.md`:

1. **The Mutation Test** — break the code on purpose, verify the test catches it (Step 4).
2. **The Assertion Strength Check** — does each assertion check the exact expected value, verify
   type not just truthiness, check all important fields, and would it fail if the function
   returned nonsense?
3. **The "Always Passes" Check** — find the tautologies. The three USELESS TEST exemplars are in
   `references/useless-test-exemplars.md`.
4. **The Negative Case Test** — for every success test, is there a corresponding failure test?
   AI rarely writes them without prompting.

Then the three common bad patterns (also in `references/validation-patterns.md`):
**The Brittle Mock** (BAD/BETTER), **The Redundant Test** (→ `@pytest.mark.parametrize`),
**The Implementation Test** (`assert cache._redis.get("cache:key")` → `assert cache.get("key") == "value"`).

### Step 6 · Run the 6-step Test Validation Checklist

For each test file:

1. **Mutation test:** Break code, verify test fails.
2. **Assertion check:** Are assertions specific?
3. **Negative cases:** Does it test error paths?
4. **Mock audit:** Could test pass with broken real code?
5. **Redundancy check:** Are tests meaningfully different?
6. **Public API test:** Does it avoid testing internals?

### Step 7 · Ask the 4 Manual Validation Questions

Automated checks are not enough. For each test that survived Steps 2–6:

- If I comment out a line of production code, does this test fail?
- If I swap return values, does this test catch it?
- Can I explain this test to a junior dev in one sentence?
- Would this test prevent a real bug I've seen before?

And the adversarial frame: *"If I were an attacker trying to break this code, would this test
catch me?"*

### Step 8 · Render the verdict

Emit the findings table (below). Every file gets **KEEP**, **FIX**, or **DELETE-AND-REWRITE**,
driven by the Nuclear Option thresholds — not by feel.

### Step 9 · Install the gate

Drop `scripts/test-quality.yml` into `.github/workflows/test-quality.yml` and
`scripts/assertion-strength-check.sh` into the repo. Now the 80% gate holds for the next PR too.

---

## Verdict rules

| Verdict | When |
|---|---|
| **KEEP** | Mutation score ≥ 80% for the file's target code, no weak assertions, at least one negative case, no mock-only tests. |
| **FIX** | Specific, bounded problems: some weak assertions (but ≤ 50%), a missing negative case, one brittle mock, redundant tests to parametrize. Name the exact edits. |
| **DELETE-AND-REWRITE** | **Any** Nuclear Option threshold trips: >50% of assertions weak · no negative cases at all · heavy mocking that tests mocks not code · all tests are variations of one scenario · you can't explain what the test validates. |

---

## Output contract

Writes to disk (when the user asks you to install the gate):

```
.github/workflows/test-quality.yml     the 80% mutation gate + assertion greps
scripts/assertion-strength-check.sh    the two greps, runnable
```

And emits **one filled findings table + verdict**, exactly this shape:

```
# AI Test Validation — payments/

Language: Python · Framework: pytest · Mutation tool: mutmut
Mutation score: 61%  (GATE: 80% — FAIL)
Surviving mutants: 14 of 36
Weak assertions: 9 of 15 (60%)  ← Nuclear Option threshold: >50%

## Findings

| Test file | Mutation score | Weak assertions | Negative cases | Mock-only | Verdict |
|---|---|---|---|---|---|
| tests/test_totals.py    | 94% | 0 of 6  | 2 | no  | KEEP |
| tests/test_user.py      | 71% | 3 of 5  | 0 | no  | FIX |
| tests/test_email.py     | 12% | 4 of 4  | 0 | yes | DELETE-AND-REWRITE |
| tests/test_add.py       | 88% | 2 of 6  | 0 | no  | FIX |

## Surviving mutants (the bugs your suite cannot see)

| File:line | Mutation | Why it survived |
|---|---|---|
| totals.py:14   | `subtotal + tax` → `subtotal - tax`  | only test asserts `total > 0` |
| user.py:31     | `if user.active:` → `if True:`       | no test for an inactive user |
| email.py:8     | body of `send_email` → `pass`        | smtplib is mocked; nothing observes the send |

## Required fixes before merge

1. tests/test_email.py — DELETE. `mocker.patch('smtplib.SMTP')` + `mock_smtp.assert_called_once()`
   asserts the mock was called, not that mail was sent. Rewrite against a `fake_smtp` fixture:
   assert `len(sent) == 1`, `sent[0].to == "test@example.com"`, `sent[0].subject == "Subject"`.
2. tests/test_user.py:12 — `assert user is not None` → `assert user.id == 123`,
   `assert user.email == "expected@example.com"`, `assert isinstance(user.created_at, datetime)`.
3. tests/test_user.py — add a negative case: `with pytest.raises(UserNotFound): get_user(-1)`.
4. tests/test_add.py — 3 near-identical tests collapse to one
   `@pytest.mark.parametrize("a,b,expected", [(2,3,5), (5,7,12), (-1,1,0), (0,0,0)])`.
5. Re-run `mutmut run`. Merge only at ≥ 80%.

## Verdict: BLOCKED — mutation score 61% < 80% gate; 1 file for deletion.
```

No section of that report may be empty. If a column has no finding, write `0` or `none` — not
a placeholder.

---

## Key takeaways

- **A green test suite proves nothing if the tests can't fail.** Tests that always pass give
  false confidence — worse than no tests.
- **Mutation testing is the gold standard.** Break the production code in three small ways; if
  the test doesn't fail for each, the test is weak.
- **Weak assertions are the #1 smell.** `assert result` and `assert result is not None` are red
  flags. Demand exact-value, type, and side-effect checks.
- **Heavy mocking erases the test.** If you mock the thing you're testing, you're asserting on
  the mock, not the behavior. Use fakes or integration tests instead.
- **For every success test, write a failure test.** Error paths are where production breaks.

---

## Files in this skill

| File | What it is |
|---|---|
| `references/validation-patterns.md` | The 4 Validation Patterns + the 3 common bad-test patterns, with all code |
| `references/red-flags.md` | The 6 Red Flags, the 6-step checklist, the 4 Manual Validation Questions, the Nuclear Option |
| `references/useless-test-exemplars.md` | The 3 USELESS TEST exemplars + the one-line decision question |
| `scripts/test-quality.yml` | `.github/workflows/test-quality.yml` — the 80% mutation gate |
| `scripts/assertion-strength-check.sh` | The two assertion-strength greps, runnable |
