Evaluation plan: numbers before claims

A model trained here earns a place only with evidence: scores on a fixed evaluation set, recorded with the run that produced them, compared against a baseline. Most of the measuring tools are built; the frozen evaluation set and run records are next.

§ 3.1ADR 0010, 0013

Four metrics, one code path

All four metrics run on the same backend, so one install gives four numbers. The evaluators are small, in-house subset runners.

Evaluation metrics and their limits.
Metric	What it checks	How it is scored	Limit	State
Perplexity	How well the model predicts held-out text	Teacher-forced, plus a llama.cpp path for GGUF files	Only meaningful on a fixed text set	Built
MMLU subset	Broad knowledge, multiple choice	Log-probability of each answer letter	A subset, not the full benchmark	Built
TruthfulQA (MC1)	Resistance to common misconceptions	50 of the 817 validation questions by default, sampled with a seed	Not comparable with the widely quoted judge-model generation score	Built
Hallucination probes	Confident wrong answers on trap questions	About 20 probes: knowable facts, and fictional items where abstaining is correct	Does not measure free-form confabulation	Built

why a custom probe setPublic benchmark questions increasingly leak into training data. A small private set gives a cleaner signal, and anyone working on the project can extend it.

§ 3.2deferred, not rejected

Why not a standard evaluation harness yet

The widely used open evaluation harness expects Transformers-style backends, which would sideline the GGUF and Candle paths, and it carries far more machinery than three or four subset benchmarks need. The decision is to revisit it if the suite grows past about five benchmarks, if a needed benchmark exists only there, or if head-to-head numbers against a published model card are required.

§ 3.3ADR 0020, 0023

The grounding verifier

For answers built on retrieved documents, a model-free verifier checks each claim against the sources and names the best-supporting one. Its acceptance criterion is written as an executable test: it must flag at least 80 percent of deliberately seeded hallucinations. Separate span and figure checks, written with the standard library only, catch quotes and numbers that do not appear in the source.

Its decision record is plain about the limits: word-overlap scoring can be fooled by paraphrase or by borrowed words, and nothing here checks logical entailment. A trained detector can replace it behind the same interface if that proves necessary.

§ 3.4ADR 0021

A promotion gate that can fail

Moving up a tier is a ritual run by sultaan pretrain gate. Each check is HARD, soft or SKIPPED. A report passes only if every hard check that ran has passed, and SKIPPED is always printed, never shown as green.

The gate is itself tested: meta-tests corrupt checkpoints on purpose and assert that the specific check fails by name. Perplexity and throughput ratchets are soft checks that skip until a baselines file exists.

check grammar   HARD  -> must pass
                soft  -> reported, does not block
                SKIPPED -> always shown, never green
passes iff      every HARD check that ran passed

Summary of the gate’s rules, not real output.

§ 3.5planned

What comes next

Planned A frozen English evaluation set: a fixed slice of FineWeb-Edu chosen by document index, with a committed checksum and the data itself kept out of git.
Planned One run record per run: commit, configuration hash, data sources with provenance and licence, tokenizer, base model, hardware, duration, tokens seen, final losses, scores and outcome.
Planned A baseline ratchet: the gate reads the last promoted run record, and a checkpoint that scores worse fails.
Open Phase 3 exit: a model that reaches perplexity below 30 on the frozen set.
Open Fine-tunes: an adapter must beat its base on the probes and the frozen set. If it does not, it is rejected and recorded, so the experiment is never repeated by accident.

rule for this notebookA score appears here only with the run record behind it. Scores on our own subsets are not leaderboard numbers.