The Closed Loop — How Tests Became the Source of Truth📋

The Open Loop Problem📋

The V1 system solved the hard problem: every acceptance criterion was a typed reference (keyof T), and every test method declared which AC it verified via @Implements<Feature>('acName'). The compiler caught typos, missing ACs, deleted features. That was real.

But the chain had a gap. The scanner could answer "which tests cover Feature X?" by reading decorators. It could not answer "which source files does Feature X claim?" without human help. The first attempt to close that gap lasted about ten to twenty minutes before we abandoned it for AST. But even that brief spike was instructive — it revealed three tempting manual approaches, all wrong:

1. .bindings.ts files — one per feature, where a developer declared "this AC verifies this symbol in this source file."
2. Feature.sourceFiles[] — an array on each Feature class listing the files it touches.
3. A regex-based scanner that grepped test files for decorator strings — the "just grep it" shortcut.

It took less than twenty minutes to see why none of these could work. The moment someone renamed a function, extracted a helper, or moved an import, the bindings would drift. And manual bindings don't just drift — they lie.

The False Binding📋

Consider the ACCENT feature's rightClickOpensPalette acceptance criterion. A developer wrote this binding file:

// requirements/bindings/accent.bindings.ts (deleted)
export const AccentBindings: FeatureBindings<AccentPaletteFeature> = {
  rightClickOpensPalette: [
    { file: 'src/lib/accent-palette-state.ts', symbol: 'createPaletteMachine' }
  ],
  // ...
};

This looks correct. Right-clicking opens the palette, the palette is created by createPaletteMachine, therefore the AC is implemented by that symbol. The compliance report would show a green checkmark. You'd move on.

But the actual test for that AC does something different:

@Verifies<AccentPaletteFeature>('rightClickOpensPalette')
'right-click on theme button opens palette'() {
  const result = routeThemeButtonClick('right', machine);
  expect(result.action).toBe('open-palette');
}

The test calls routeThemeButtonClick(), not createPaletteMachine(). The binding was conceptually true — right-clicking eventually involves the palette machine somewhere in the call chain — but factually false. The test verified the routing function, not the palette factory. Left in place, the compliance report would say "AC covered, binding to createPaletteMachine" — and it would be a lie. That's the kind of drift we spotted within minutes, and exactly why we dropped manual bindings on the spot.

The AST scanner fixed this in one pass. It walked the test body, found routeThemeButtonClick, resolved the import to src/lib/accent-palette-state.ts, and emitted the correct binding. The manifest entry now reads:

"rightClickOpensPalette": [
  { "file": "src/lib/accent-palette-state.ts", "symbol": "routeThemeButtonClick" }
]

No human declared this. The test did, by importing and calling routeThemeButtonClick. The AST does not lie about what the test actually calls — it cannot. It reads the syntax tree, not someone's mental model of what the test "should" be testing.

The AST Inference Breakthrough📋

The replacement is a single file: scripts/lib/test-bindings-scanner.ts. An AST walker that reads every *.test.ts and *.spec.ts file, extracts the decorators, follows the imports, and emits a BindingsManifest — the same data structure the manual binding files produced, but inferred mechanically from the test code itself.

The scanner is pure — no direct fs or child_process access. It takes a FileSystem port from src/lib/external.ts, making it unit-testable with an in-memory fake. The core types:

export interface SymbolTarget {
  file: string;   // repo-relative path: "src/lib/accent-palette-state.ts"
  symbol: string;  // exported name: "createPaletteMachine"
}

/** featureId -> acName -> [{file, symbol}] */
export type BindingsManifest = Record<string, Record<string, SymbolTarget[]>>;

The pipeline has seven stages:

For each test file, parseTestFile creates a TypeScript SourceFile, then:

1. collectImports builds a Map<localName, ImportEntry> — every import { x } from '...' with its module specifier and kind (named, default, namespace). Type-only imports are skipped.
2. extractFeatureTestArg finds the @FeatureTest(SomeFeature) class decorator and extracts the feature class name.
3. extractVerifiesArgs finds every @Verifies<Feature>('acName') method decorator and extracts the AC strings.
4. collectLocalHelpers and collectClassHelpers map top-level functions and class methods to their AST bodies.
5. For each @Verifies method, the scanner walks the method body collecting every identifier in value position — filtering out type positions, declaration left-hand sides, and property access right-hand sides.
6. Each collected name is resolved through the import map to a repo-relative file path. Names that don't appear in the import map (locals, parameters, vitest globals) are dropped. Names that resolve outside src/ or scripts/ are dropped.
7. aggregateTestBindings merges results from all test files into the final BindingsManifest, deduplicating by file::symbol.

The result: for every AC of every Feature, the manifest lists the exact source symbols the test actually calls. Not what a human declared — what the AST proves.

Peeling Expressions: getRootIdentifier📋

Step 5 in the pipeline — "walk the method body and collect identifiers" — hides a subtle problem. When the scanner encounters machine.toggle(), the CallExpression's callee is a PropertyAccessExpression (machine.toggle), not an Identifier. A naive scanner would try to resolve toggle — which is a method name, not an import. What we need is the root identifier: machine.

The getRootIdentifier function peels any expression down to its root:

export function getRootIdentifier(expr: ts.Expression): string | null {
  if (ts.isIdentifier(expr)) return expr.text;
  if (ts.isPropertyAccessExpression(expr)) return getRootIdentifier(expr.expression);
  if (ts.isElementAccessExpression(expr)) return getRootIdentifier(expr.expression);
  if (ts.isCallExpression(expr) || ts.isNewExpression(expr)) return getRootIdentifier(expr.expression);
  if (ts.isParenthesizedExpression(expr)) return getRootIdentifier(expr.expression);
  if (ts.isNonNullExpression(expr)) return getRootIdentifier(expr.expression);
  if (ts.isAsExpression(expr)) return getRootIdentifier(expr.expression);
  return null;
}

Ten lines, seven cases, recursive. It handles every expression shape that appears in real test code:

Once we have the root identifier, we check the import map. If machine was declared locally (const machine = setup()), it drops out. If createPaletteMachine was imported from src/lib/accent-palette-state.ts, it becomes a binding. The root identifier is the bridge between "what the code calls" and "what the import map can resolve."

What the Scanner Filters Out📋

The scanner does not collect every identifier in a method body. A naive approach — grab every Identifier node — would produce massive false positives. Consider a typical test:

const result: PaletteState = createPaletteMachine(color, opts);
expect(result.open).toBe(false);

A naive collector would emit: result, PaletteState, createPaletteMachine, color, opts, expect, open, toBe, false. Most of these are noise — type names, locals, property keys, assertion methods.

The collectReferencedNamesFromNode function applies five filters:

1. Type positions are skipped entirely. PaletteState is a TypeReference — it resolves to a type import, never a runtime call. The isTypePosition helper detects TypeReference, TypeLiteral, TypeQuery, and all TypeNode parents.

2. Declaration left-hand sides are skipped. const result declares result — it is not a reference to result. Variable declarations, function names, class names, parameter names, binding patterns — all filtered.

3. Property access right-hand sides are skipped. In result.open, the .open is a property key, not an identifier reference. Only the left-hand side (result) is collected.

4. Method/getter/setter/property-assignment keys are skipped. In { onStateChange: (state) => ... }, the key onStateChange is a property name, not a reference.

5. Vitest globals and locals naturally drop out. expect, toBe, false — none appear in the import map, so they are silently ignored during the resolution step.

The result: from the example above, only createPaletteMachine survives. It is imported, it is in scope (src/lib/), and it becomes a binding. Everything else is noise that the filters removed or that the import map did not match.

Transitive Resolution: Following the Call Graph📋

A naive scanner that only collects direct identifiers from method bodies misses too much. Real test files use helpers:

// test/unit/accent-palette-state.test.ts

function setup(initialColor: AccentColor = 'green') {
  const stateChanges: Array<{ state: PaletteState; prev: PaletteState }> = [];
  const colorChanges: AccentColor[] = [];
  const machine = createPaletteMachine(initialColor, {
    onStateChange: (state, prev) => stateChanges.push({ state, prev }),
    onColorChange: (color) => colorChanges.push(color),
  });
  return { machine, stateChanges, colorChanges };
}

@FeatureTest(AccentPaletteFeature)
class AccentPaletteTests {
  @Verifies<AccentPaletteFeature>('swatchChangesAccent')
  'starts closed with the given initial color'() {
    const { machine } = setup('blue');
    expect(machine.getState()).toEqual({ open: false, activeColor: 'blue' });
  }
}

The @Verifies method calls setup() — a local function, not an import. A direct-only scanner sees setup (local, filtered) and expect (vitest, filtered). It misses createPaletteMachine entirely.

The first version of the scanner had exactly this problem: 523 @Verifies methods resolved to zero symbols. That was the "empty methods" metric. It meant the scanner saw the decorator, confirmed the test existed, but could not trace which source code it actually verified.

The fix: collectReferencedNamesTransitive. When the scanner encounters a bare identifier that matches a local helper, it walks the helper's body recursively, collecting identifiers from that body too. Same for class helpers reached via this.someMethod(). A visited set prevents infinite recursion.

@Verifies method body
  ├── sees: setup (local helper → walk its body)
  │   ├── sees: createPaletteMachine (imported from src/lib/accent-palette-state.ts → BOUND)
  │   ├── sees: stateChanges, colorChanges (local variables → dropped)
  │   └── sees: onStateChange, onColorChange (callback params → dropped)
  ├── sees: expect (vitest global → dropped)
  └── sees: machine (local variable → dropped)

Result: swatchChangesAccent → [{ file: "src/lib/accent-palette-state.ts", symbol: "createPaletteMachine" }]

After transitive resolution: 523 empty methods dropped to 11. The remaining 11 are tests that inspect CSS output, raw HTML, or build artefacts without importing any TypeScript symbol from src/ — legitimate cases where there is nothing to bind.

Concrete Chain: ACCENT Feature End-to-End📋

To make this concrete, here is the complete traceability chain for the Accent Color Palette feature — from requirement to source file, with every link mechanically verified.

Step 1 — Feature class (requirements/features/accent.ts):

export abstract class AccentPaletteFeature extends Feature {
  readonly id = 'ACCENT';
  readonly title = 'Accent Color Palette';
  readonly priority = Priority.Medium;

  abstract swatchChangesAccent(): ACResult;
  abstract accentPersists(): ACResult;
  abstract toggleOpensAndCloses(): ACResult;
  // ... 11 more ACs, 14 total
}

Step 2 — Test file (test/unit/accent-palette-state.test.ts):

import { createPaletteMachine, isValidAccent, ACCENT_COLORS,
         routeThemeButtonClick, readStoredAccent } from '../../src/lib/accent-palette-state';
import { deriveAccentPalette } from '../../src/lib/accent-derive';

@FeatureTest(AccentPaletteFeature)
class AccentPaletteTests {
  @Verifies<AccentPaletteFeature>('swatchChangesAccent')
  'starts closed with the given initial color'() {
    const { machine } = setup('blue');  // setup calls createPaletteMachine
    expect(machine.getState()).toEqual({ open: false, activeColor: 'blue' });
  }
}

Step 3 — AST scanner infers the bindings:

Step 4 — BindingsManifest (excerpt from requirements-bindings.json):

"ACCENT": {
  "swatchChangesAccent": [
    { "file": "src/lib/accent-palette-state.ts", "symbol": "createPaletteMachine" },
    { "file": "src/lib/accent-preview-state.ts", "symbol": "createAccentPreviewMachine" }
  ],
  "accentAdaptsDarkLight": [
    { "file": "src/lib/accent-derive.ts", "symbol": "deriveAccentPalette" }
  ],
  "rightClickOpensPalette": [
    { "file": "src/lib/accent-palette-state.ts", "symbol": "routeThemeButtonClick" }
  ]
}

Step 5 — Derived source files: the compliance scanner reads the manifest and derives that ACCENT claims three files: src/lib/accent-palette-state.ts, src/lib/accent-preview-state.ts, src/lib/accent-derive.ts. No human declared this. The tests did, by importing from them.

Step 6 — Bidirectional queries (covered in Part II):

$ work trace feature ACCENT
→ src/lib/accent-palette-state.ts (11 ACs)
→ src/lib/accent-preview-state.ts (6 ACs)
→ src/lib/accent-derive.ts (1 AC)

$ work trace file src/lib/accent-derive.ts
→ Feature ACCENT (accentAdaptsDarkLight → deriveAccentPalette)

Every link is verified. Change the import, the binding changes. Delete the import, the AC becomes unbound. Rename the AC, keyof T catches it at compile time.

The Feature That Tracks Itself📋

The scanner is itself a feature: TEST-BINDINGS-INF (Test Bindings Inference) with 22 acceptance criteria — one for each facet of the AST walker:

export abstract class TestBindingsInferenceFeature extends Feature {
  readonly id = 'TEST-BINDINGS-INF';
  readonly title = 'Test Bindings Inference (AST Scanner)';
  readonly priority = Priority.High;

  abstract collectsAllImportKinds(): ACResult;
  abstract skipsTypeOnlyImports(): ACResult;
  abstract extractsFeatureTestArg(): ACResult;
  abstract extractsVerifiesArgs(): ACResult;
  abstract walksMethodBodyForCalls(): ACResult;
  abstract capturesBareIdentifierReferences(): ACResult;
  abstract skipsTypePositions(): ACResult;
  abstract skipsDeclarationLhs(): ACResult;
  abstract collectsLocalHelpers(): ACResult;
  abstract collectsClassHelpers(): ACResult;
  abstract transitiveThroughLocalHelpers(): ACResult;
  abstract transitiveThroughClassHelpers(): ACResult;
  // ... 10 more ACs
}

The test file imports 18 functions from scripts/lib/test-bindings-scanner.ts — the scanner's own code — and verifies them with @Verifies decorators:

import {
  parseTestFile, collectImports, resolveTarget, isInScope,
  extractFeatureTestArg, extractVerifiesArgs,
  collectCalleeNames, collectReferencedNames, collectReferencedNamesTransitive,
  collectLocalHelpers, collectClassHelpers, getRootIdentifier,
  aggregateTestBindings, sortManifest, diffManifests,
  collectTestFiles, scanTestBindings,
} from '../../scripts/lib/test-bindings-scanner';

@FeatureTest(TestBindingsInferenceFeature)
class ImportCollectionTests {
  @Verifies<TestBindingsInferenceFeature>('collectsAllImportKinds')
  'collects named, default, and namespace imports'() {
    const src = `import { foo } from './mod'; import bar from './b'; import * as ns from './n';`;
    const sf = ts.createSourceFile('test.ts', src, ts.ScriptTarget.Latest);
    const imports = collectImports(sf);
    expect(imports.size).toBe(3);
  }
}

When the AST scanner runs on these tests, it discovers that TEST-BINDINGS-INF claims scripts/lib/test-bindings-scanner.ts as its source file. The scanner inferred its own binding. Here is the actual manifest entry (abridged):

"TEST-BINDINGS-INF": {
  "collectsAllImportKinds": [
    { "file": "scripts/lib/test-bindings-scanner.ts", "symbol": "collectImports" }
  ],
  "transitiveThroughLocalHelpers": [
    { "file": "scripts/lib/test-bindings-scanner.ts", "symbol": "collectLocalHelpers" },
    { "file": "scripts/lib/test-bindings-scanner.ts", "symbol": "collectReferencedNamesTransitive" }
  ],
  "resolvesRootIdentifier": [
    { "file": "scripts/lib/test-bindings-scanner.ts", "symbol": "getRootIdentifier" },
    { "file": "scripts/lib/test-bindings-scanner.ts", "symbol": "collectCalleeNames" }
  ]
}

Every AC maps to functions inside the scanner itself. The scanner is reading its own tests, following its own imports, resolving its own symbols, and writing its own manifest entry. If the scanner's import resolution breaks, the tests for resolvesRelativeImportsToRepoPaths fail — and the manifest entry for that AC disappears, making the compliance report show the gap.

This is self-reference at two levels:

• REQ-TRACK tracks the tracking system (the compliance report, the quality gate, the decorators).
• TEST-BINDINGS-INF tracks the inference engine (the AST walker, the import resolver, the transitive helper follower).

If the scanner breaks, its own tests fail. If its own tests fail, its own binding disappears from the manifest. The immune system has an immune system. The recursion bottoms out at the TypeScript compiler — if @Verifies<TestBindingsInferenceFeature>('collectsAllImportKinds') does not compile, the chain is broken at the most fundamental level, and tsc refuses to build.

What the Scanner Cannot See📋

Eleven @Verifies methods still resolve to zero source symbols. These are not bugs — they are tests that verify behaviour without importing TypeScript code from src/ or scripts/:

• Tests that inspect generated CSS (regex match on a style string)
• Tests that parse raw HTML output (DOMParser on a rendered page)
• Tests that check build artefacts (file existence, JSON schema)
• E2E tests that use Playwright (page.goto, page.click) instead of imported functions

The compliance report distinguishes these as TU (test unit — symbol resolved to a source file) versus E2E (end-to-end — test exists but no symbol binding). Both are valid coverage — an AC verified by a Playwright test that clicks real UI elements is not less verified than one tested by a unit test. But the distinction matters for traceability:

The compliance report renders this as a table with one column per dimension:

Total = ACs declared on the Feature class. Covered = ACs with at least one @Verifies test (always = TU + E2E). TU = ACs whose tests resolve to source symbols. E2E = ACs verified by Playwright tests with no source binding. % = Covered / Total. src = vitest v8 line coverage on claimed files.

All three features are 100% covered — every AC has at least one @Verifies test. But the kind of coverage differs:

• Unit (TU) — the test imports a symbol from src/lib/, the scanner resolved the binding, and the manifest knows which function implements which AC. Full traceability: you can query "where is tocClickLoadsPage implemented?" and get a concrete answer.
• E2E — the test uses Playwright (page.goto, page.click), imports no source symbols. The AC is verified by a black-box test. The manifest cannot tell you which source file implements it — the test does not import source files.

THEME at 5/5 TU has maximal traceability: every AC maps to a function, every function maps to a file. NAV at 4/4 TU + 4/4 E2E is fully tested but half-opaque — the Playwright tests verify behaviour, but the manifest cannot link those 4 ACs to source code. VIS at 1/1 TU + 3/3 E2E is mostly opaque — visual screenshot comparisons are inherently E2E.

Neither is wrong. But the distinction matters: TU coverage feeds the manifest, the bidirectional queries, and the architecture X-ray. E2E coverage verifies correctness but is invisible to the traceability graph.

Next: Part II — Bidirectional Queries explores what happens when you can query the manifest in both directions — from feature to file, and from file to feature.