The Eval Template

The eval carries commitment and memory, the two jobs a prototype cannot do. A document describes a rule. An eval enforces it. If it matters, it fails the build. The one feature that matters most: the rationale lives next to the check, in version control, read every single time the check fails. That is memory that cannot drift, because it moves with the code.

There are three kinds of eval. Together they cover the three things a wall of prose used to promise and never enforced.

1. Outcome evals — does the behavior actually move the success metric from the brief? This is how you catch drift from the bet at build time instead of in hindsight.
2. Guardrail evals — the commitments. Regulatory, SLA, contractual, and business rules. These fail the build.
3. Regression evals — the rules you already decided, so a future change cannot quietly break them.

---

Eval block (copy per rule)

# Eval: [what it checks, in five words]
# Kind: outcome | guardrail | regression
# Rule: [the invariant, in plain language]
# Rationale: [why this rule, and why this threshold. "N ms, because X."
#             This is the memory. Do NOT delete it. When this eval blocks a
#             change two years from now, this line is what the person staring
#             at the failure reads.]
# Owner / date:
# Source: [link to the brief line, ticket, contract, or regulation]
# Violation: [what specifically counts as failing]

test:
  given: [starting state]
  when:  [action]
  then:  [expected result or threshold]
  on_fail: block   # use "warn" for non-blocking guardrails

---

Worked example — a guardrail eval

# Eval: PII never leaves the EU region
# Kind: guardrail
# Rule: No request carrying customer PII is routed to a non-EU endpoint.
# Rationale: GDPR data-residency clause in the Acme MSA, section 7.3. Breach is
#            a contractual termination event, not just a fine. Do not relax this
#            to "warn" without legal sign-off. Added because the v1 prototype
#            happily called a US inference endpoint.
# Owner / date: falk / 2026-07-02
# Source: /briefs/eu-residency.md, Acme MSA §7.3
# Violation: any egress to a region tagged non-eu with a PII flag set

test:
  given: a request with pii=true and user.region=eu
  when:  the router selects an endpoint
  then:  endpoint.region == "eu"
  on_fail: block

Worked example — an outcome eval

# Eval: onboarding still lands users on first value
# Kind: outcome
# Rule: A new user reaches the "first insight" screen in <= 3 steps.
# Rationale: The bet in the brief is "faster time-to-value lifts week-1
#            retention." If step count creeps up, we are drifting from the bet.
#            Catch it here, not in next quarter's retention chart.
# Owner / date: falk / 2026-07-02
# Source: /briefs/onboarding.md — "The bet"
# Violation: the golden-path flow takes 4+ steps to first insight

test:
  given: a fresh account on the golden path
  when:  the user completes onboarding
  then:  steps_to_first_insight <= 3
  on_fail: block

---

Rules for this artifact

1. Rationale lives here, next to the check. Not in a doc three directories away that may or may not have been updated. Memory that moves with the code cannot drift.
2. Guardrails and regressions block the build. Outcome evals block or warn, depending on how load-bearing the bet is.
3. A complex domain gets more evals, not more prose. If your instinct is to write more, write more tests.
4. Every commitment in the brief should become an eval before the real build starts. If it cannot be encoded, it was never really a commitment, just a hope.