Part VI: The Compliance Scanner
Parse features. Scan tests. Cross-reference. Report. Fail. Five steps, 300 lines, and the system verifies itself.
Why a Scanner, Not the Runtime Registry
Part V showed the runtime registry — decorators push RequirementRef entries into an array at import time. Why not just read that array?
Because the scanner needs to work without running the tests. It runs as a standalone script:
npx tsx scripts/compliance-report.ts # Console report
npx tsx scripts/compliance-report.ts --json # JSON for automation
npx tsx scripts/compliance-report.ts --strict # Exit 1 if critical gapsThe scanner reads source code with regex patterns. It doesn't import test files, doesn't start Playwright, doesn't launch Vitest. It's fast (under 1 second) and dependency-free.
The Four Steps
Step 1: Read Feature Definitions
The scanner reads every .ts file in requirements/features/ and extracts:
- The class name (e.g.,
NavigationFeature) - The
id,title, andpriorityproperties - All abstract method names (the ACs)
function readFeatures(): FeatureDef[] {
for (const file of fs.readdirSync(FEATURES_DIR)) {
if (!file.endsWith('.ts')) continue;
const src = fs.readFileSync(path.join(FEATURES_DIR, file), 'utf8');
const classMatch = src.match(/export abstract class (\w+)/);
const idMatch = src.match(/readonly id = '([^']+)'/);
const titleMatch = src.match(/readonly title = '([^']+)'/);
const priorityMatch = src.match(/readonly priority = Priority\.(\w+)/);
const acMethods: string[] = [];
const methodRegex = /abstract (\w+)\(/g;
let m;
while ((m = methodRegex.exec(src)) !== null) {
acMethods.push(m[1]!);
}
features.push({ id, title, priority, acMethods, file });
}
}Simple regex parsing. No AST, no TypeScript compiler API. The feature files follow a strict format (Part IV), so regex is sufficient and fast.
Step 2: Scan Test Files
The scanner walks five test directories and detects three patterns:
const TEST_DIRS = [
'test/unit', 'test/e2e', 'test/a11y', 'test/visual', 'test/perf'
];Pattern 1: @Implements<Feature>('acName') — the primary pattern
const decoratorRegex = /@Implements<(\w+)>\s*\(\s*'(\w+)'\s*\)/g;Captures the feature class name and AC method name from every @Implements decorator in every test file.
Pattern 2: @FeatureTest(Feature) + coversACs — the batch pattern
For dynamic tests that can't use per-method decorators:
const featureTestRegex = /@FeatureTest\((\w+)\)\s*\n\s*class\s+(\w+)/g;
// Then find: static readonly coversACs = ['ac1', 'ac2', ...]Pattern 3: @FEAT:AC string tags — legacy fallback
For plain JavaScript test files that can't use TypeScript decorators. Not used in this project but supported for migration scenarios.
Step 3: Cross-Reference
The scanner builds a coverage matrix — for each feature, for each AC, which tests cover it:
function crossReference(features: FeatureDef[], refs: TestRef[]): FeatureCoverage[] {
return features.map(f => {
const acs = f.acMethods.map(method => {
const matchingTests = refs.filter(r =>
(r.featureId === f.id || classToId.get(r.feature) === f.id)
&& r.ac === method
);
return {
method,
tests: matchingTests.map(t => ({ file: t.testFile, name: t.testName })),
covered: matchingTests.length > 0,
};
});
const coveredCount = acs.filter(a => a.covered).length;
return {
id: f.id, title: f.title, priority: f.priority,
acs, coveredCount,
totalCount: acs.length,
percentage: Math.round((coveredCount / acs.length) * 100),
};
});
}Step 4: Report and Gate
Console output — colored, scannable:
── Feature Compliance Report ──
✓ NAV SPA Navigation + Deep Links 8/8 ACs (100%)
✓ SPY Scroll Spy 7/7 ACs (100%)
✓ BUILD Static Build Pipeline 13/13 ACs (100%)
✓ SEARCH Search 5/5 ACs (100%)
✓ A11Y Accessibility 5/5 ACs (100%)
✓ REQ-TRACK Requirements Tracking System 7/7 ACs (100%)
...
────────────────────────────────────────────────────────────
Coverage: 112/112 ACs (100%)
Critical uncovered: 0
Quality gate: PASSJSON output — for automation:
{
"timestamp": "2026-03-24T10:30:00.000Z",
"features": [
{
"id": "NAV",
"title": "SPA Navigation + Deep Links",
"priority": "critical",
"acs": [
{
"method": "tocClickLoadsPage",
"tests": [{ "file": "test/e2e/navigation.spec.ts", "name": "clicking TOC item..." }],
"covered": true
}
],
"coveredCount": 8,
"totalCount": 8,
"percentage": 100
}
],
"summary": {
"totalFeatures": 20,
"totalACs": 112,
"coveredACs": 112,
"percentage": 100,
"criticalUncovered": []
}
}Quality gate — with --strict, the scanner exits with code 1 if any critical feature has uncovered ACs:
if (FLAG_STRICT) {
const criticalUncovered = results
.filter(f => f.priority === 'critical')
.some(f => f.percentage < 100);
if (criticalUncovered) {
process.exit(1);
}
}The Self-Referential Loop
The requirements tracking system tracks itself. RequirementsTrackingFeature has 7 ACs:
export abstract class RequirementsTrackingFeature extends Feature {
readonly id = 'REQ-TRACK';
readonly title = 'Requirements Tracking System';
readonly priority = Priority.Critical;
abstract featureDefinitionsCompile(): ACResult;
abstract everyFeatureHasTest(): ACResult;
abstract scannerGeneratesJson(): ACResult;
abstract scannerGeneratesConsole(): ACResult;
abstract qualityGateFailsOnGap(): ACResult;
abstract eslintFlagsMissingDecorator(): ACResult;
abstract selfAppearsInReport(): ACResult;
}The test for selfAppearsInReport runs the compliance scanner and verifies that REQ-TRACK appears at 100%:
@Implements<RequirementsTrackingFeature>('selfAppearsInReport')
'REQ-TRACK is listed in compliance report at 100%'() {
const json = execSync(
'npx tsx scripts/compliance-report.ts --json',
{ cwd: ROOT, encoding: 'utf8', stdio: 'pipe' }
);
const report = JSON.parse(json);
const reqTrack = report.features.find((f: any) => f.id === 'REQ-TRACK');
expect(reqTrack).toBeDefined();
expect(reqTrack.percentage).toBe(100);
// This test IS one of the 7 ACs — closing the self-reference loop
}This test is both a test and one of the things it's testing. If you remove it, REQ-TRACK drops below 100%, which the scanner detects, which the quality gate catches. The system is self-enforcing.
Integration Into Your Workflow
The scanner is a standalone script. Integrate it wherever it fits your process:
# After writing tests, before pushing
npx tsx scripts/compliance-report.ts
# As part of the full test suite
npm run test:all && npx tsx scripts/compliance-report.ts --strict
# Generate a JSON snapshot for trend tracking
npx tsx scripts/compliance-report.ts --json > compliance-report.jsonOn this project, everything runs locally — build, test, scan, push. No cloud CI. The scanner is fast enough (under 1 second) to run after every test cycle.
What the Scanner Doesn't Do
The scanner verifies that every AC has at least one linked test. It doesn't verify:
- That the test actually passes (that's the test runner's job)
- That the test is meaningful (that's the developer's judgment)
- That the feature definition is complete (that's a design decision)
- That tests cover edge cases (that's what property-based testing is for)
The scanner is a coverage index, not a correctness oracle. It answers "is this AC tested?" not "is this AC tested well?"
Previous: Part V: The Decorator Chain Next: Part VII: V2 Roadmap — what's left to improve and how to get there.