# AI-First TDD

The test is the spec. You are not writing tests *for* the code — you are writing the only
unambiguous description of the feature that exists, and then making the agent satisfy it.

**But an unrun test is a wish, and a test that passes against an empty implementation is not a
spec.** That is the failure this skill exists to catch, and catching it requires running something.

## The loop, and what each step must PROVE

| Step | The claim | The proof (a command, not an opinion) |
|---|---|---|
| **1 · Detect** | We know how to run this repo's tests | The runner is read out of `package.json` / `pyproject.toml`, never assumed |
| **2 · Red** | The tests fail | Run the suite. **Exit 1** — tests ran and failed |
| **3 · Prove Red** | They fail for the **right reason** | Stub the implementation. Re-run. **Still exit 1, now on assertions** |
| **4 · AI-Green** | The agent satisfied the spec | Run the suite. **Exit 0** |
| **5 · Measure** | Coverage clears the target for this code type | Read `coverage.json` / `coverage-summary.json`. A number |
| **6 · Refactor** | The change was safe | Run the suite. **Exit 0**, unchanged |

If you catch yourself doing something that is not Red, AI-Green or Refactor, you have left the loop.

---

## Step 1 — Detect the runner. Do not assume Python.

The previous version of this skill hard-coded `pytest`. Half the readers do not have Python
installed. **Read the repo.**

```bash
# JS/TS: the test script and the runner that backs it
jq -r '.scripts.test // "—", (.devDependencies // {} | keys | join(" "))' package.json 2>/dev/null

# Python: pytest config lives in one of four places
rg -n --no-heading 'pytest|\[tool\.pytest|\[tool\.coverage|unittest' \
  pyproject.toml pytest.ini setup.cfg tox.ini 2>/dev/null

# everything else
ls go.mod Cargo.toml pom.xml build.gradle 2>/dev/null
```

Then read **2–3 existing test files** and match their naming, fixture style and import layout.
The tests you write must look like the tests already there, or nobody will maintain them.

Map what you found. **These are the four commands you will run for the rest of this skill** —
write them into `tdd-plan-<feature>.md` verbatim, because you will run each of them at least twice:

| Runner | Detected by | Run one file | Coverage command | Machine-readable report |
|---|---|---|---|---|
| **pytest** | `pyproject.toml` / `pytest.ini` / `setup.cfg` | `pytest tests/test_x.py -q` | `pytest --cov=src --cov-report=json --cov-report=term` | `coverage.json` |
| **vitest** | `vitest` in `devDependencies` | `npx vitest run src/x.test.ts` | `npx vitest run --coverage --coverage.reporter=json-summary` | `coverage/coverage-summary.json` |
| **jest** | `jest` in `devDependencies` | `npx jest src/x.test.ts` | `npx jest --coverage --coverageReporters=json-summary` | `coverage/coverage-summary.json` |
| **go test** | `go.mod` | `go test ./pkg/x` | `go test ./... -coverprofile=cover.out && go tool cover -func=cover.out` | `cover.out` |
| **cargo** | `Cargo.toml` | `cargo test x` | `cargo llvm-cov --json --output-path coverage.json` | `coverage.json` |

Verified 2026-07-14 against primary docs:
`--cov-report=json` writes `coverage.json` — https://pytest-cov.readthedocs.io/en/latest/reporting.html ·
vitest coverage needs `@vitest/coverage-v8` and defaults its output to `./coverage` —
https://vitest.dev/guide/coverage · both vitest and jest accept **any istanbul reporter**, and
`json-summary` is the istanbul reporter that writes `coverage-summary.json` —
https://jestjs.io/docs/configuration#coveragereporters-arraystring--string-options ·
https://istanbul.js.org/docs/advanced/alternative-reporters/

**Ask only what the repo could not tell you**, batched into one message:

- What behaviour should exist when this is done? (the spec, one paragraph)
- The public interface — the exact name and signature you want to call.
- Which **code type** is this? Business Logic / Public API / Data Processing / UI Component / Glue
  Code. *This picks the coverage target in Step 5 and nothing else does.*
- Any constraint the agent must honour (storage, encoding, error types, perf).

---

## Step 2 — Red. Write the failing test, then RUN it.

Write to the repo's **real** test path. Not a scratch file, not a code block in chat.

The seed set is four tests, minimum:

1. the **happy path**, with a specific assertion on value, shape, length and type
2. the **round-trip** / read-back case, if the interface implies one
3. an **idempotence / consistency** case, if the interface implies one
4. one **error case** that raises the *named* exception

Every test satisfies all five **Good Test Characteristics** (Hard Rules, below) and carries a
Given/When/Then docstring.

Now run it:

```bash
pytest tests/test_url_shortener.py -q; echo "exit=$?"
```

Read the exit code. It is a documented enum, and it is the cheapest signal in this entire skill —
https://docs.pytest.org/en/stable/reference/exit-codes.html (fetched 2026-07-14):

| Exit | pytest means | What you do |
|---|---|---|
| **0** | all collected tests passed | **STOP.** Your test passes with no implementation. It asserts nothing. Go back to Step 2. |
| **1** | tests ran and failed | This is Red. Continue to Step 3. |
| **5** | **no tests were collected** | Your file was never picked up — wrong path, wrong filename convention. Not Red. Broken. |

For vitest/jest, take the same three-way branch from `--reporter=json` / `--json` rather than the
exit code alone; the failure *messages* are what Step 3 reads.

---

## Step 3 — Prove Red. The step everyone skips, and the reason their tests are theatre.

Exit 1 is not enough. A test that fails with `ModuleNotFoundError` has proven **only that you
haven't written the file yet**. It has not proven it can tell right from wrong.

**Stand up the empty stub.** Every function in the interface exists, is importable, and returns
nothing:

```python
# src/url_shortener.py  — the STUB. Written by this skill, deleted by the agent in Step 4.
class NotFoundError(Exception): pass
class ValidationError(Exception): pass

class UrlShortener:
    def shorten(self, url): return None      # deliberately, correctly, useless
    def resolve(self, code): return None
```

Re-run the suite. **Two outcomes, and only one of them lets you continue:**

- **Still fails, and now the failures are `AssertionError` / `assert` diffs** → the tests describe
  a *behaviour*, not a filename. This is a real spec. Proceed.
- **Anything now passes** → that test is a tautology. It passed against a class that does nothing.
  It will pass against whatever the agent writes, too. **Delete it or strengthen the assertion.**
  `assert result` and `assert result is not None` are the usual culprits — pin the exact length,
  type, value and casing instead.

`scripts/red-proof.sh` — drop it in the repo, it is the whole gate in twenty lines:

```bash
#!/usr/bin/env bash
# red-proof — a test that passes against an empty implementation is not a spec.
# usage: scripts/red-proof.sh "<test-cmd>" <stub-file>
set -uo pipefail
TEST_CMD="$1"; STUB="$2"

echo "── 1. no implementation at all ──────────────────────────────"
$TEST_CMD; NAKED=$?
[ "$NAKED" -eq 0 ] && { echo "FAIL: suite is green with NO implementation. The tests assert nothing."; exit 1; }
[ "$NAKED" -eq 5 ] && { echo "FAIL: no tests collected. Wrong path or filename — this is broken, not Red."; exit 1; }

echo "── 2. empty stub in place ───────────────────────────────────"
[ -f "$STUB" ] || { echo "FAIL: write the stub at $STUB first."; exit 1; }
OUT=$($TEST_CMD 2>&1); STUBBED=$?
echo "$OUT" | tail -20

if [ "$STUBBED" -eq 0 ]; then
  echo "FAIL: the suite PASSES against a stub that returns None. Every one of these tests is a tautology."
  exit 1
fi
if echo "$OUT" | rg -q 'ModuleNotFoundError|ImportError|NameError|Cannot find module|is not defined'; then
  echo "FAIL: still failing on IMPORTS, not assertions. The stub does not match the interface the tests call."
  exit 1
fi
echo "PASS: fails on assertions against a working-but-empty implementation. That is a spec."
```

```bash
chmod +x scripts/red-proof.sh
scripts/red-proof.sh "pytest tests/test_url_shortener.py -q" src/url_shortener.py
```

> **This is not mutation testing.** Red-proof breaks the code by *never writing it* and asks whether
> the test notices. Mutation testing breaks **working** code in small, plausible ways and asks the
> same question of a suite that is already green. They catch different lies, and the second one is
> `vc-ai-test-validator`'s job — run it after Step 4, not instead of Step 3.

---

## Step 4 — AI-Green. The tests ARE the prompt.

Hand over the spec. Paste the **actual test file**, not a summary of it. Everything the agent needs
to satisfy the spec is in the file; everything else is context it can misread.

```text
Implement <Interface> so that these tests pass.

Context:     <one paragraph — what this thing is for>
Constraints: <storage / encoding / error types / perf — from Step 1>
Tests:       <the full contents of tests/test_url_shortener.py, verbatim>

Replace the stub at src/url_shortener.py. Do not modify the tests.
Do not add tests. Use <language/stdlib conventions detected in Step 1>.
```

Run the suite.

- **Green** → go to Step 5.
- **Red** → the spec was ambiguous. **Sharpen the test.** Do *not* hand-patch the implementation to
  make a fuzzy assertion go green — that buries the ambiguity in code that no longer has a spec, and
  it comes back at 2am.

**"Do not modify the tests" is load-bearing.** An agent given both files will happily satisfy the
spec by editing the spec.

---

## Step 5 — Coverage. A number, from a file, not a table you nodded at.

The old version of this skill printed a Coverage Goals table and left the reader to eyeball it.
**Measure it.**

```bash
pytest --cov=src/url_shortener --cov-report=json --cov-report=term
node scripts/coverage-gate.mjs
```

Pick the target per file from the code type you established in Step 1 — **one target per file, and
no file ships without one:**

| Code type | Target | Focus |
|---|---|---|
| Business Logic | **90%+** | all edge cases, error paths |
| Public API | **85%+** | contract tests, input validation |
| Data Processing | **80%+** | boundary conditions, null handling |
| UI Component | **60%+** | user interactions, error states |
| Glue Code | **40%+** | integration points only |

Write the assignment down so the gate can enforce it — `tdd-targets.json`:

```json
{
  "src/url_shortener.py": { "code_type": "business-logic", "target": 90 },
  "src/http/routes.py":   { "code_type": "glue-code",      "target": 40 }
}
```

`scripts/coverage-gate.mjs` — reads whichever report the detected runner emitted, joins it to the
targets, and **exits non-zero**, so it drops straight into CI:

```js
#!/usr/bin/env node
// coverage-gate — per-file targets from tdd-targets.json. Not a global average:
// a 90% business-logic file and a 40% glue file average to a lie.
import { readFile } from 'node:fs/promises';
import { existsSync } from 'node:fs';

const targets = JSON.parse(await readFile('tdd-targets.json', 'utf8'));
const pct = {};

if (existsSync('coverage.json')) {                       // pytest-cov
  const c = JSON.parse(await readFile('coverage.json', 'utf8'));
  for (const [f, d] of Object.entries(c.files)) pct[f] = d.summary.percent_covered;
} else if (existsSync('coverage/coverage-summary.json')) { // istanbul (vitest / jest)
  const c = JSON.parse(await readFile('coverage/coverage-summary.json', 'utf8'));
  for (const [f, d] of Object.entries(c)) {
    if (f !== 'total') pct[f.replace(`${process.cwd()}/`, '')] = d.lines.pct;
  }
} else {
  console.error('no coverage report found — run the coverage command from Step 1 first');
  process.exit(2);
}

let failed = 0;
for (const [file, { code_type, target }] of Object.entries(targets)) {
  const got = pct[file];
  if (got === undefined) { console.error(`✗ ${file}  NO COVERAGE DATA — is it even imported by a test?`); failed++; continue; }
  const ok = got >= target;
  if (!ok) failed++;
  console.log(`${ok ? '✓' : '✗'} ${file}  ${got.toFixed(1)}% / ${target}%  (${code_type})`);
}
console.log(failed ? `\n${failed} file(s) under target` : '\nall files clear their target');
process.exit(failed ? 1 : 0);
```

**Under target? The gap is missing edge cases — go back to Step 2 and write them.** Do not write
tests whose only purpose is to move the number; that is how you get a 95% suite that catches nothing,
and `vc-ai-test-validator` will find it and tell you to delete it.

Do not apply 90% to glue code. Do not accept 40% on business logic.

---

## Step 6 — Refactor, with the suite as the harness

Now the tests are worth something, and refactoring is finally cheap. Close every refactor prompt
with **"All existing tests must still pass."** Run the suite. Green means the refactor was safe. Red
means **revert the refactor, not the tests** — the tests are the spec, and the spec did not change.

---

## What lands on disk

```
tests/test_<feature>.<ext>     seed tests (Step 2) + edge tests, at the repo's real test path
src/<feature>.<ext>            the stub (Step 3), then the agent's implementation (Step 4)
tdd-targets.json               per-file code type + coverage target
scripts/red-proof.sh           the stub gate — exits non-zero if a test passes against nothing
scripts/coverage-gate.mjs      per-file coverage gate — exits non-zero under target. Drop in CI.
coverage.json | coverage/coverage-summary.json   the measured numbers
tdd-plan-<feature>.md          the four commands, the target, the prompt sent, each run's result
```

`tdd-plan-<feature>.md` is the reproducible record. Every line is either a file that exists or a
command that ran, with its actual exit code:

```markdown
# AI-First TDD — UrlShortener

Runner: pytest (detected: pyproject.toml [tool.pytest.ini_options])
Test:     pytest tests/test_url_shortener.py -q
Coverage: pytest --cov=src/url_shortener --cov-report=json --cov-report=term
Target:   business-logic → 90%

## Step 2 — Red (4 seed tests)
$ pytest tests/test_url_shortener.py -q   → exit 1 · 4 failed (ModuleNotFoundError)

## Step 3 — Prove Red
$ scripts/red-proof.sh "pytest tests/test_url_shortener.py -q" src/url_shortener.py
  1. no implementation  → exit 1 ✓
  2. empty stub         → exit 1, 4× AssertionError ✓
  PASS — the tests fail on assertions against a working-but-empty implementation.
  (First run FAILED here: test_shorten_url_returns_code asserted `assert code` and went GREEN
   against the stub. Replaced with `len(code) == 6 and code.isalnum()`.)

## Step 4 — AI-Green
Prompt: tests pasted verbatim + "replace the stub, do not modify the tests".
$ pytest tests/test_url_shortener.py -q   → exit 0 · 4 passed

## Step 5 — Coverage
$ node scripts/coverage-gate.mjs
  ✓ src/url_shortener.py  94.2% / 90%  (business-logic)

## Step 6 — Refactor (SHA-256 codes, type hints, extracted validation)
$ pytest -q   → exit 0 · 7 passed. Refactor safe.

## Handoff
$ mutmut run   (vc-ai-test-validator — the 80% mutation gate)
```

> The worked example above is **derived, not reported**: a URL shortener, in-memory dict, 6-char
> base62 codes, `NotFoundError` on an unknown code. It is here to show the shape of the record and
> the numbers are from that toy. Your numbers will differ; the *commands* will not.

---

## Hard Rules

1. **Detect the runner. Never assume pytest.** A skill that ships `pytest` to a TypeScript repo has
   already wasted the reader's afternoon.
2. **Run the test before you believe it.** Exit code 0 at Step 2 means the test asserts nothing.
3. **A test that passes against an empty stub is not a test.** Step 3 is non-negotiable and it is
   the whole reason this skill exists.
4. **Never ask the agent for code and tests in the same prompt.** Tests generated to match code test
   what the code *does*, not what it *should* do — they pass by construction, and they are the
   single most common way an AI-assisted suite ends up worthless. If the user asks for both, refuse
   and write the tests first.
5. **Never let the agent edit the tests to make them pass.** Say it in the prompt, every time.
6. **The Good Test Characteristics — all five, every test:** tests the public interface not private
   state · has a docstring naming the scenario · makes a specific assertion (never `assert result`)
   · tests one behaviour · fails for the right reason when the code is wrong.
7. **One coverage target per file, from the code type.** A repo-wide average hides exactly the file
   you should be worried about.
8. **Refuse to commit the three failure patterns:** tests coupled to implementation (asserting on
   `_url_map`) · assertions so loose the agent can return anything truthy · tests written after the
   implementation.

## What this skill will not do

- **It will not prove your existing suite can fail.** Red-proof shows a test notices an *absent*
  implementation. It does not show the test notices a *subtly wrong* one. That is mutation testing,
  the 80% mutation-score gate, and the per-file KEEP / FIX / DELETE-AND-REWRITE verdict — all of
  which are **`vc-ai-test-validator`**, and it is the merge gate that runs *after* Step 4. Do not
  reimplement it here; run it.
- **It will not generate the test cases from a spec you already have.** That is
  `vc-test-prompt-builder`. This skill owns the loop; that one owns the cases.
- **It will not pick your coverage number for you when the code type is ambiguous.** Ask. A wrong
  code type sets a wrong target and the gate then enforces the wrong thing forever.
- **It will not write the implementation.** The agent does that, in Step 4, from the spec you proved
  was a spec. If you find yourself hand-writing the implementation to satisfy your own test, the
  test was not clear enough — fix the test.
