← Automated Commissioning · Prompt Deck · Pictures

How do we make the feature matrix a measurement instrument instead of an opinion ledger?

Intent Contract​

Story Contract​

Stories are test contracts. Each row advances to ≥1 test. Tests must be RED before implementation starts. GREEN = value delivered.

S1 — Auto-update on merge​

Trigger: PR merged to main in engineering repo

Checklist:

• feature-matrix.json diff shows state changes within 5 minutes of merge
• Only features whose mapped tests were touched have state changes
• No feature reaches L3 when its test results are failing

Forbidden: States change for untouched features. Feature marked L3 with failing tests.

Evidence: integration — libs/commissioning/__tests__/

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S2 — Unmapped features visible​

Trigger: Commission script run completes

Checklist:

• Report shows unmapped count with specific feature IDs
• No unmapped feature silently stays at L0 without a flag
• Script does not report 100% coverage when mappings are missing

Forbidden: Unmapped features hidden. Coverage report lies.

Evidence: unit — libs/commissioning/__tests__/index-builder.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S3 — Safety Test blocks L3​

Trigger: Safety Test assertion fails in test run

Checklist:

• Feature state capped at L2 regardless of Success Test results
• Safety Test failure logged with the feature ID
• Escalate to commissioner if >5 features blocked simultaneously

Forbidden: Feature reaches L3 with active safety violation. Failures silently ignored.

Evidence: integration — libs/commissioning/__tests__/level-computer.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S4 — Regression triggers demotion​

Trigger: Previously-passing test starts failing

Checklist:

• Feature state moves from L3 to L2 (or lower) in next commission run
• Demotion logged with previous state, new state, and failing test file
• No one-way ratchet — states can go down

Forbidden: State stays at L3 after tests break. Demotion applied silently without logging.

Evidence: integration — libs/commissioning/__tests__/level-computer.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S5 — All three FAVV formats parsed​

Trigger: PRD uses FFO, FAVV v2.0, or FAVV v2.1 format

Checklist:

• Parser extracts feature-to-test mappings from FFO (6 cols), v2.0, and v2.1 (9 cols)
• v2.0 and FFO PRDs are not silently skipped
• Partial results returned when table is malformed — no crash, log warning

Forbidden: Crash on FFO. Silent skip of v2.0. Only v2.1 produces mappings.

Evidence: unit — libs/commissioning/__tests__/favv-parser.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S6 — UI features verified via Playwright​

Trigger: Commission script encounters feature with e2e spec

Checklist:

• Playwright runs spec headlessly and captures trace + screenshot
• Result feeds L-level computation alongside unit results
• Feature with passing unit tests but failing e2e is capped at L2
• Escalate to commissioner if Playwright CI adds >2 minutes per feature

Forbidden: Feature reaches L3 from unit tests alone when e2e spec exists but fails. Playwright runs headed in CI. Agent browser used instead of spec.

Evidence: e2e — libs/commissioning/__tests__/e2e-runner.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S7 — Mock routes flagged as counterfeit​

Trigger: Commission script analyzes e2e spec files before running

Checklist:

• AST scan flags page.route() and page.fulfill() usage
• Report lists counterfeit specs with file path and line number
• Scanner does not modify spec files
• No false positives on legitimate test setup (auth state, seed data)

Forbidden: Mock-route specs counted as passing evidence. Fixture data accepted as integration proof.

Evidence: static-analysis — libs/commissioning/__tests__/mock-detector.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S8 — Unit tests excluded from L3 evidence​

Trigger: Commission script categorizes test results

Checklist:

• Unit tests (mocked DB/server) logged but not used for L-level state changes
• Only integration (real DB) and e2e (real server) results feed computation
• Unit test exclusion does not affect unit test reporting elsewhere

Forbidden: Unit test with mocked DB counted as integration evidence. Hook test with stubbed API affects L-level.

Evidence: unit — libs/commissioning/__tests__/level-computer.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

S9 — Commissioning level appears in ProjectWithStats​

Trigger: Commission script completes a run against a project's features

Checklist:

• commissioning_results Convex table updated with { featureId, prdRef, level, updatedAt } per feature
• listProjectsWithStats returns commissioningLevel: number | null for each project (highest L-level across its features)
• Project with no commissioning run returns commissioningLevel: null — not 0
• Commission script does not write to Convex when --dry-run flag is active

Forbidden: Dashboard shows commissioningLevel: 0 when no commission run has occurred. Script writes to Convex in dry-run mode.

Evidence: integration — libs/commissioning/__tests__/convex-sync.spec.ts

Commission Result: ⬜ PASS / ⬜ FAIL Notes: (findings)

Build Contract​

Job 1: Parse PRD Specs into Feature-Test Index​

Job 2: Run Scoped Tests and Compute L-Levels​

Job 3: Write Results and Produce Evidence​

Job 4: E2E Verification via Playwright​

Job 5: Sync Results to Project Dashboard (N5 Bridge)​

Convex schema addition:

// commissioning_results table
defineTable({
  featureId: v.string(),    // e.g. "AUTH-001"
  prdRef: v.string(),       // e.g. "prd-identity-access"
  level: v.number(),        // 0-4
  updatedAt: v.number(),    // Unix timestamp (ms) — use Date.now()
})

Principles​

The Job​

Why Now​

• 210 features, 0 at L4. Manual tracking has hit its ceiling.
• Agent-driven development means agents need deterministic commissioning — they can't make judgment calls about L-levels.
• The FAVV v2.1 format now includes Artifact + Success Test + Safety Test columns — the mapping data finally exists in the PRD specs.
• Every new PRD adds more features to track manually.

Design Constraints​

Performance​

Priority Score​

Quality Targets​

Failure Budget​

Kill signal: Script produces states that contradict manual commissioner judgment on >20% of features after 3 runs. If the algorithm disagrees with reality that often, the mapping is wrong — fix the mapping before running again.

Platform​

Current State​

Build Ratio​

~50% composition (extend existing FAVV parser, use existing vitest, reuse Nx Playwright targets, write to existing JSON), ~50% new code (index builder, level computer, matrix writer, receipt generator, e2e discovery/runner/evidence).

Protocols​

Build Order​

Commissioning​

Agent-Facing Spec​

Commands:

# From engineering repo root — unit + integration commissioning
npx tsx tools/scripts/commission/commission-features.ts --dry-run    # preview
npx tsx tools/scripts/commission/commission-features.ts --feature=AUTH-001  # single
npx tsx tools/scripts/commission/commission-features.ts --all        # full matrix
npx tsx tools/scripts/commission/commission-features.ts --report     # unmapped report only

# E2e commissioning via Playwright (scoped to feature)
npx tsx tools/scripts/commission/commission-features.ts --feature=AUTH-001 --e2e  # includes Playwright
pnpm nx e2e web-e2e --grep="AUTH-001"                               # direct Nx target
pnpm exec playwright test --project=chromium --grep="AUTH-001"       # direct Playwright CLI

# Debugging and investigation
pnpm exec playwright codegen http://localhost:3000                   # generate selectors
pnpm exec playwright test apps/web-e2e/src/auth.spec.ts --debug     # step-through debug
pnpm nx e2e web-e2e --ui                                            # interactive UI mode

Boundaries:

• Always: read PRD specs, run mapped tests, write JSON, emit receipt
• Ask first: demoting a feature that was manually set to L3+
• Never: modify test files, change PRD specs, delete features from JSON

Test Contract:

Players​

Demand-Side Jobs​

Job 1: Commissioner Verifies Feature States​

Situation: After engineering ships a sprint, the commissioner needs to know which features actually advanced.

Features that serve this job: #5, #6, #7, #8, #9, #10

Job 2: Engineering Agent Updates Matrix After Merge​

Situation: CI pipeline merges code. Feature states should reflect the new reality.

Features that serve this job: #5, #8, #10

Role Definitions​

Relationship to Other PRDs​

Context​

• PRD Index — Automated Commissioning
• Prompt Deck — 5-card pitch
• Pictures — Pre-flight maps

Questions​

What breaks first when the script disagrees with a human commissioner?

• If 30% of features are unmapped, is the script's output trustworthy enough to replace manual edits?
• Should the script refuse to write results when unmapped percentage exceeds a threshold?
• When CRM gets a Story Contract, how many features jump from L0 to L3 in one run — and does that shock look like a bug?
• Is the Safety Test gate (blocking L3) too conservative — or not conservative enough?