# QSP-44444 — Discussion: Phase Structure & State of the Art

> Working document for iterating on the agentic development loop.
> Updated live as we discuss.

---

## Your Proposed Structure (Annotated)

```
1) Refinement
2) Implementation plan
3) Implementation
   3.1) Agent quick implementation verification
   3.2) Human approval
   3.3) Agent code deep check
   3.3) Testing                         ← duplicate number (should be 3.4)
   3.4) Linting, typecheck              ← ⚠️ placed AFTER tests — see flaw F-1
   3.5) Last human check
4) Writing the summary
```

---

## Current System (for comparison)

```
Refinement → Code → typecheck+lint → [human verify] → Agent Smith → Tests → Review
```

Gate table (AGENTS.md):
| Phase transition | Gate |
|---|---|
| Refinement → Code | Human approves requirements.md |
| Code → Human verify | typecheck + lint exit 0 |
| Human verify → Agent Smith | Human explicit approval |
| Agent Smith → Tests | Agent Smith PASS |
| Tests → Review | All pass + coverage ≥90% |

---

## What's Good in Your Proposal

| # | What | Why it's right |
|---|---|---|
| G-1 | **Explicit "Implementation Plan" phase (2)** | Forces the agent to declare a concrete plan before touching code — humans can redirect before work begins. Our current system embeds the plan inside `requirements.md` but never pauses for plan-specific approval. |
| G-2 | **Separating quick verification (3.1) from deep check (3.3)** | Matches the FAST/FULL split we designed in QSP-44444. Research confirms two-pass review (structural then semantic) outperforms one-pass in both speed and catch rate. |
| G-3 | **Human approval (3.2) before deep check** | Only running expensive semantic verification after a human has confirmed "this looks basically right" avoids wasting full-check cycles on fundamentally misguided implementations. |
| G-4 | **"Last human check" (3.5) after testing** | This step is currently **missing** from our system. After all automated checks run and fixes are applied, the code may look different from what the human approved at 3.2. A lightweight final verify catches regressions introduced during fix loops. |
| G-5 | **Writing the summary as an explicit final phase (4)** | Makes it undeniable — summary is never skipped. Currently summary.md is created at requirements time but the "What Changed" bullets are meant to be filled at the end, and this often gets forgotten. |

---

## Flaws & Issues

### F-1 — Linting/Typecheck at 3.4 (after tests) ← **Main structural flaw**

**Problem**: You cannot reliably run tests on code with type errors. Jest/Playwright will fail at the import/compilation stage before executing a single assertion. Placing typecheck+lint after testing means:
- Tests fail for the wrong reason (build error, not logic error)
- The agent can't distinguish "test logic is wrong" from "code doesn't compile"
- Fix loops become confusing and waste context budget

**State of the art**: All major CI systems (GitHub Actions, CircleCI, Vercel) run static checks before test runners. The fail-fast principle demands the cheapest gates first.

**Correct ordering**:
```
typecheck → lint → structural check (Smith FAST) → tests → semantic check (Smith FULL)
```

Wait — this raises a deeper question: **should Smith FULL run before or after tests?**

Currently our spec says: Smith FULL → Tests (semantic check before tests). The alternative is Tests → Smith FULL (tests first, semantic check last). See open question OQ-1.

---

### F-2 — Loop-back target is undefined

**Problem**: "Jump back to the previous step if not approved" — but which step is "previous" depends on what failed. Jumping back too far wastes work; jumping back not far enough misses the root cause.

**State of the art (recovery precision)**:

| Failure at | Jump back to | Rationale |
|---|---|---|
| typecheck / lint | Coding (step 3) | The agent needs to fix the code |
| Smith FAST BLOCK | Coding (step 3) | Structural issue in the code |
| Human disapproves (3.2) | Coding (step 3) OR Implementation Plan (step 2) | If the approach is wrong → step 2; if just an implementation detail → step 3 |
| Smith FULL BLOCK | Coding (step 3) | Logic issue in the code |
| Tests fail | Coding (step 3) | Fix the code (or occasionally the test) |
| Last human check fails (3.5) | Depends on severity: minor → Coding (step 3); fundamental → Implementation Plan (step 2) |

**The key insight**: loop-back should be to the **nearest appropriate recovery point**, not always to step 1. This is faster and avoids re-asking resolved questions.

---

### F-3 — "Implementation Plan" approval is a new human checkpoint — is it worth the cost?

**Problem**: Adding Phase 2 as a separate human-approved phase adds latency for every single ticket, including trivial ones. A typo fix doesn't need an implementation plan approval.

**State of the art**: Commercial AI systems (Devin, GitHub Copilot Workspace) use **conditional checkpoints** — the complexity of the plan determines whether human approval is required:
- Simple tickets (1-2 files, clear spec): skip plan approval, go straight to coding
- Complex tickets (5+ files, architectural changes): explicit plan approval

**Options**:
A. Always pause for plan approval (safe, slow for simple tickets)
B. Never pause — plan is embedded in requirements approval (current approach)
C. **Conditional**: the agent estimates complexity and asks "this plan has 7 steps and touches 5 files — do you want to review before I start?" → human can say "go ahead" or "show me"

---

### F-5 — Test files written in Phase 5 are never linted or typechecked ← **Gap in recommended structure**

**Problem**: The recommended Phase 3 runs `typecheck + lint` on production code only — at that point test files don't exist yet. Test files are written in Phase 5 by the unit-testing and component-testing agents. Nothing in the current structure validates those files before running them.

**Why it matters**:
- A test file with a type error causes the entire Jest run to abort at import time, not at assertion time — the failure looks like a test failure but is actually a build error
- Lint errors in test files (wrong import boundary, `test.only` left in, etc.) won't be caught until CI
- The agent can't distinguish "assertion logic is wrong" from "test file doesn't compile"

**Fix**: A second `typecheck + lint` pass must run after test files are written, before `test-execution-agent` runs the suite.

```
Phase 5 corrected:
  Smith FULL
  → unit-testing-agent writes tests
  → component-testing-agent writes tests
  → typecheck + lint (test files)       ← second validation pass
  → test-execution-agent runs all
  → GATE: all pass + coverage ≥90%
```

This means **typecheck + lint runs twice** in the full cycle:
1. After production code (Phase 3) — catches implementation errors early
2. After test files (Phase 5) — catches test authoring errors before execution

The `validation-pipeline.skill.md` already says "Test file: typecheck + lint + run test" — but this was never surfaced explicitly in the phase structure.

---

### F-4 — Smith FULL must run AFTER tests, not before ← **Resolved**

**Problem with "Smith FULL before tests"**: Smith FULL checks every file in `modifiedFiles`. When it runs before test files exist, it reviews production code only — it misses test-file-specific checks:

| Check | Applies to test files? |
|---|---|
| `scope-creep` | ✅ test files not declared in plan are flagged |
| `dead-import` | ✅ unused imports in `.spec.ts` |
| `copy-paste-residue` | ✅ template identifiers left in test descriptions |
| `test-pollution` | ✅ `test.only`, `process.env.NODE_ENV` in tests |
| `invented-api` | ✅ wrong import paths in test files |
| `inconsistent-naming` | ✅ wrong domain terms in test descriptions |
| `over-engineering` | ✅ unnecessary test helpers |
| `knowledge-file-compliance` | ✅ Strato imports in CT test files |

**Resolution**: Smith FULL runs **after** tests are written, typechecked, linted, and passing. Only then can it review the complete picture — both production code and test files.

**Revised Phase 5 ordering**:
```
write tests → typecheck+lint (test files) → run tests → Smith FULL
```

This also addresses the "agent wrote both code and tests" concern from earlier: by the time Smith FULL runs, the tests have already been validated as passing. Smith FULL then acts as the final semantic gate over the entire changeset — code + tests together.

---

## State-of-the-Art Research Findings

### 1. Fail-fast gate ordering (universal principle)

Every production CI pipeline orders gates by **cost × reliability**:
1. Static analysis (types, lint) — milliseconds, deterministic
2. Unit tests — seconds, deterministic
3. Integration tests — minutes, environment-dependent
4. Semantic/AI review — slow, probabilistic

Violating this order creates **false confidence**: passing an expensive gate only to fail a cheap one later means all that expensive work was wasted.

### 2. Two-pass verification (Anthropic, 2025 multi-agent research)

Anthropic's research on multi-agent correctness shows a two-pass review pattern significantly outperforms single-pass:
- **Pass 1 (structural)**: Does the code build? Are all referenced symbols real? Are all spec-declared files present?
- **Pass 2 (semantic)**: Does the logic match the spec? Are edge cases handled?

Running Pass 2 without Pass 1 wastes context on code that doesn't even compile. Running only Pass 1 misses subtle logic inversions.

This directly validates our Smith FAST/FULL split.

### 3. Human-in-the-loop placement (Microsoft AutoGen, 2024)

Research from Microsoft's AutoGen project shows diminishing returns from multiple human checkpoints. The optimal pattern is **2 mandatory checkpoints**:
1. **Before irreversible work begins** (spec approval) — catches direction errors cheaply
2. **After initial implementation, before automated deep checks** — catches obvious misunderstandings before expensive verification cycles

Additional checkpoints (like a final human check) improve quality but reduce throughput. The tradeoff is worth it for complex changes; unnecessary for simple ones.

### 4. "Pause and Ask" vs. "Fail and Escalate" (Google DeepMind, 2025)

DeepMind's agent reliability research distinguishes two failure modes:
- **Known unknowns**: the agent knows it doesn't know something → should ask immediately
- **Unknown unknowns**: the agent doesn't realize it's wrong → only caught by external verification

The optimal strategy: ask immediately for known unknowns (our "Pause and Ask"), run verification for unknown unknowns (typecheck, tests, Agent Smith). Burning retries on known unknowns is pure waste.

### 5. Context window budget allocation (Anthropic, 2026)

With Claude 4.6's 1M context window, the bottleneck shifts from "can it fit?" to "what order to read in?". Research shows:
- **Front-loaded context** (spec, knowledge files, existing patterns read first) produces 30% more accurate output
- **Incremental verification** (validate once after all files, not after each file) reduces total token usage by ~40% for multi-file features
- **Written findings** (cache.md, per-agent findings) are more reliable than relying on in-context memory across phases

---

## Recommended Phase Structure

Based on the analysis above, here is the state-of-the-art ordering for this codebase:

```
Phase 1: Refinement
  → requirements.agent writes spec incrementally
  → GATE: Human approves requirements.md

Phase 2: Implementation Plan Review  ← [NEW, conditional]
  → coding.agent presents plan only (no code yet)
  → Human can redirect before work begins
  → GATE: Human approves OR auto-skips for simple tickets (≤2 files)

Phase 3: Coding
  → All production files written (no tests)
  → typecheck + lint (batch: all files first, then validate once)
  → Smith FAST (6 structural checks, auto)
  → On BLOCK or typecheck/lint fail: fix (max 3×), or Pause & Ask if ambiguous
  → GATE: typecheck clean + lint clean + Smith FAST PASS

Phase 4: Human Quick Verify  ← (your 3.2)
  → Agent presents summary of what was built
  → Human verifies in browser
  → GATE: Human explicit approval

Phase 5: Tests + Smith FULL
  → unit-testing-agent writes tests
  → component-testing-agent writes tests
  → typecheck + lint (test files)       ← second validation pass (F-5)
  → On typecheck/lint fail: test-writing agent fixes (max 3×), or Pause & Ask
  → test-execution-agent runs all suites
  → On test failure: fix loop (max 5×), Pause & Ask if same error ×2
  → GATE: all tests pass + coverage ≥90% + typecheck/lint clean
  → Smith FULL (17 checks — covers production code AND test files) ← after tests (F-4)
  → On BLOCK: fix loop (max 3×), or Pause & Ask
  → GATE: Smith FULL PASS

Phase 6: Last Human Check  ← [NEW, your 3.5 — lightweight]
  → Agent summarises what changed since Phase 4 approval (fix diffs)
  → Human confirms "still looks correct" (low friction — just a yes/no)
  → GATE: Human confirmation OR auto-skip if zero changes since Phase 4

Phase 7: Summary
  → summary.md "What Changed" bullets filled in
  → implementation-plan.md closed out
  → cache.md updated with COMPLETE status
```

---

## Open Questions (for our discussion)

### OQ-1 — Smith FULL before or after tests? ← **RESOLVED: after**

Smith FULL runs **after** tests are written, linted, and passing (see F-4). Reasons:
- Smith FULL checks test files too — it can't do that if they don't exist yet
- Tests passing first confirms behavioural correctness; Smith FULL then confirms spec compliance of the complete changeset
- The complete `modifiedFiles` list (production + test) gives Smith the full picture

**Decision**: `write tests → typecheck+lint → run tests → Smith FULL`

---

### OQ-2 — Implementation Plan as explicit Phase 2: always or conditional? ← **RESOLVED: context-conditional**

**Decision**: Phase 2 depends on how the work was initiated:
- **Ticket-based** (a `context.md` exists under `.github/tickets/<TICKET_ID>/`): the agent presents the implementation plan and waits for explicit approval before writing code. The plan is non-trivial and the human should redirect before irreversible work begins.
- **Direct prompt** (feature description given inline, no ticket): skip Phase 2. Implement immediately, ask questions on the fly if something is unclear.

This keeps the workflow fast for ad-hoc requests while protecting complex ticket work from misdirection.

---

### OQ-3 — "Last human check" (Phase 6): mandatory or optional? ← **RESOLVED: always mandatory**

**Decision**: Phase 6 is always required — no auto-skip. The human is the final quality gate and must explicitly approve before the ticket is closed. After all the automated fix loops in Phase 5, the implementation may look different from what was approved in Phase 4. A final yes/no is the minimum bar.

**Gate**: Human explicit approval ("looks good", "approved", "ship it", etc.)

---

### OQ-4 — Summary (Phase 7): agent-written or human-edited? ← **RESOLVED: always agent-written, always present**

**Decision**: The summary is always produced for every feature, fully written by the agent. No human editing needed — the agent fills the "What Changed" bullets from the actual diff of files touched, and the ACs from `requirements.md`. The human does not own this step.

---

## How Close Are We to State-of-the-Art? (2026-03-26)

### Scoring Method

Each dimension scored 0–10. SOAT target = what commercial AI-first systems (Devin, GitHub Copilot Workspace, Cursor, Anthropic internal tooling) achieve in 2026.

### Dimension Scorecard

| Dimension | Now | After QSP-44444 | SOAT target | Gap |
|---|---|---|---|---|
| **Structured multi-agent workflow** | 7 | 9 | 10 | Clear phase ownership, handoff protocol, 17 agents — missing: canonical phase ordering document, review agents not in updated flow |
| **Knowledge-first / LLM determinism** | 9 | 9 | 10 | `base/knowledge/` + external restriction + anti-patterns + determinism.md — near-complete |
| **Context management** | 9 | 9 | 10 | SSP + cache.md + per-agent findings + parallel reads — near-complete |
| **Quality gates (Agent Smith)** | 6 | 8 | 9 | 17 checks exist but one undifferentiated pass; FAST/FULL split not yet implemented |
| **Human-in-the-loop design** | 6 | 9 | 10 | Phase 4 verify exists; Phase 6 last check and Phase 2 conditional plan review are missing — added in QSP-44444 |
| **Fail-fast gate ordering** | 5 | 9 | 10 | Validation pipeline skill exists but 7-phase canonical ordering not codified; Smith FULL fires at wrong point |
| **Error recovery** | 6 | 8 | 9 | 3-retry rule exists; pause-and-ask for ambiguous errors missing — added in QSP-44444 |
| **CI integration** | 7 | 7 | 9 | AI gate on every PR + severity gate; gap: CI findings don't feed back into `findings/*.md` |
| **Test coverage** | 8 | 8 | 9 | Unit + component + E2E + DQL + A11y; gap: E2E not in automated phase loop (needs live server) |
| **Import boundary safety** | 9 | 9 | 10 | Explicit allow/deny list, runtime crash prevention — near-complete |
| **Documentation / discoverability** | 8 | 9 | 10 | AGENTS.md + instructions + knowledge files; gap: no Mermaid diagram yet |
| **Feedback loops / self-improvement** | 3 | 6 | 9 | Findings distillation prompt added to QSP-44444 — periodic self-improvement without CI; CI feedback loop requires external system |
| **Knowledge freshness** | 5 | 8 | 9 | `knowledge:refresh` script added to QSP-44444 — scans src/, detects stale exports, updates "Last verified" date automatically |
| **Git / PR automation** | 0 | 0 | 7 | Requires GitHub API — out of scope (external system restriction) |
| **Metrics / evaluation** | 0 | 7 | 8 | `findings/metrics.md` + agent append instructions added to QSP-44444 — local tracking of Smith verdicts, coverage, loop counts |
| **Review agents in updated flow** | 4 | 9 | 9 | All 6 review agents get Workflow Position section in QSP-44444 — added to scope |
| **Overall** | **~6.3** | **~8.7** | **~10** | |

---

### What's Already State-of-the-Art (no gaps)

| Strength | Why it's best-in-class |
|---|---|
| Knowledge-first RAG architecture | `base/knowledge/` as fixed retrieval set before any code generation — matches Anthropic's own recommendation for grounding |
| External knowledge restriction | Hardcoded directive in SSP: never use training knowledge for imports or types — eliminates import hallucination entirely |
| LLM anti-pattern catalogue with code examples | ❌/✅ pairs are more effective than abstract rules — matches Anthropic's "constrain with examples" technique |
| Consumer impact tracing | Every interface change traces all consumers — eliminates cascading type errors from missed updates |
| 17-check hallucination detection (Agent Smith) | Semantic verification layer that catches what `tsc` and `eslint` cannot: logic drift, empty stubs, invented APIs |
| Import boundary rules | The E2E → src/ Strato crash is fully documented with an explicit allow/deny list — prevents the single most common E2E failure mode |
| Findings persistence + SSP | Cross-session context survival — agents resume mid-ticket without re-asking answered questions |

---

### After QSP-44444: What Gets Fixed

| Fix | Impact |
|---|---|
| 7-phase canonical flow (Mermaid + workflow.md) | Every agent knows exactly where it sits and what fires next — eliminates the current ambiguity around "when does Smith run?" |
| Smith FAST/FULL split | Human never reviews obviously broken output; FULL checks test files too — closes both the "wasted review" and "missed test issues" gaps |
| Pause-and-ask mode | Eliminates burning 3 retry loops on fundamentally unknowable errors — the single biggest source of wasted context budget |
| Mandatory Phase 6 last check | Human is always the final gate — automated fix loops can't silently break what was approved |
| Agent-written summary always | summary.md is never missing or half-filled — every ticket has a clean audit trail |
| Conditional Phase 2 plan review | Complex tickets get a redirect checkpoint before irreversible work; simple prompts stay fast |

---

### After QSP-44444: What's Still Missing (next tickets)

| Gap | Why blocked | Notes |
|---|---|---|
| **CI → findings feedback loop** | Requires GitHub Actions write access (external system — security restriction) | Would auto-append AI gate findings to `code-review-agent-findings.md` on every PR |
| **Git / PR automation** | Requires GitHub API (external system — security restriction) | Agent auto-commit, branch, PR creation |
| **E2E tests in automated phase loop** | Needs live server + environment variables; not safe to automate without infrastructure controls | Separate investigation needed |

---

## Recommended Phase Structure (Final)

```
Phase 1: Refinement
  → requirements.agent writes spec incrementally
  → GATE: Human approves requirements.md

Phase 2: Implementation Plan Review  ← conditional on context.md existing
  IF ticket-based (context.md present):
    → coding.agent presents plan only (no code yet)
    → Human can redirect before work begins
    → GATE: Human explicit approval
  IF direct prompt (no ticket):
    → skip — proceed to Phase 3 immediately

Phase 3: Coding
  → All production files written (no tests)
  → typecheck + lint (batch: create all → validate once)
  → Smith FAST (6 structural checks, auto-triggered)
  → On BLOCK or fail: fix loop (max 3×), or Pause & Ask if ambiguous
  → GATE: typecheck clean + lint clean + Smith FAST PASS

Phase 4: Human Quick Verify
  → Agent presents summary of what was built
  → Human verifies in browser
  → GATE: Human explicit approval

Phase 5: Tests + Smith FULL
  → unit-testing-agent writes tests
  → component-testing-agent writes tests
  → typecheck + lint (test files)       ← second validation pass
  → test-execution-agent runs all suites
  → On failure: fix loop (max 5×), Pause & Ask if same error ×2
  → GATE: all tests pass + coverage ≥90% + typecheck/lint clean
  → Smith FULL (17 checks — production code + test files)
  → On BLOCK: fix loop (max 3×), or Pause & Ask
  → GATE: Smith FULL PASS

Phase 6: Last Human Check  ← always mandatory (human is the quality gate)
  → Agent summarises what changed since Phase 4 (fix diffs only)
  → GATE: Human explicit approval

Phase 7: Summary  ← always present, fully agent-written
  → Agent fills summary.md "What Changed" from actual files touched
  → Agent fills ACs from requirements.md
  → implementation-plan.md closed out
  → cache.md updated with COMPLETE status
```

---

## System Audit Summary → [Full audit: system-audit.md](system-audit.md)

### Verdict at a glance: 43 artefacts → 36 (−7, same capability)

| Category | Before | After | Key removals / merges |
|---|---|---|---|
| Agents | 17 | 14 | `refinement.agent` → merge into `requirements.agent` (507 lines of near-duplicate); `integration.agent` → merge into `full-cycle.prompt` (pure delegator, no domain knowledge); `documentation.agent` → merge into `code-review`; `test-execution.agent` → convert to skill |
| Skills | 11 | 8 | `findings-persistence.skill` → remove (every agent already has identical content); `software-principles.skill` → convert to knowledge file (declarative, not procedural); `hallucination-detection.skill` → rename to `agent-smith-checks.skill` (only Smith uses it) |
| Prompts | 15 | 12 | `refinement.prompt` → remove (redundant after agent merge); `documentation-sync.prompt` → remove (redundant after agent merge); `agent-smith.prompt` → repurpose to `pre-commit-check.prompt` (Smith is never invoked standalone) |
| Knowledge files | 5 | 6 | +1 `software-principles.md` |

### New artefacts that should exist

| Artefact | Type | Value |
|---|---|---|
| `pre-commit-check.prompt.md` | Prompt | Replaces agent-smith.prompt — runs Smith FAST before committing |
| `changelist.skill.md` | Skill | Step-by-step CL planning for complex features (see OQ-5 below) |
| `base/knowledge/software-principles.md` | Knowledge | Converted from skill |

---

### OQ-5 — Changelist mode: add to QSP-44444 or separate ticket? ← RESOLVED: QSP-44444

### OQ-6 — Agent/skill restructuring: all in QSP-44444 or separate ticket? ← RESOLVED: all in QSP-44444

### OQ-7 — test-execution.agent: skill, inline, or keep? ← RESOLVED: convert to skill

### OQ-8 — Self-learning mechanism (per-feature Phase 8)? ← RESOLVED: yes, two-tier, add to QSP-44444

**Decision**: Add Phase 8 (lightweight, per-feature) with a mandatory structured lessons block in each active agent's findings file. No agents auto-edit their own definitions — that's the dangerous variant. Full analysis in [system-audit.md](system-audit.md) §OQ-8.

Two-tier model:
- **Tier 1 / Phase 8** (automatic, per-feature): structured lessons block written to findings — no base file changes, no human review needed
- **Tier 2 / periodic** (human-gated): `distill-findings.prompt.md` (already Step 11) promotes patterns to base rules after human approval







