Training approach: from scratch, one rung at a time

The north star is a language model pretrained from scratch, starting small and climbing a ladder of sizes. Each rung must be measurably healthy before the next one is attempted. The infrastructure for that is built and proven on CPU. One small first-contact run has been trained on a laptop GPU; a run at tier T1 or above is still ahead.

§ 2.1ADR 0016

The model: a modern decoder in plain PyTorch

The architecture follows the current Llama and Qwen family recipe: pre-norm RMSNorm, rotary position embeddings (RoPE), SwiGLU feed-forward blocks, grouped-query attention, no bias terms and tied input and output embeddings.

Attention runs through PyTorch’s scaled_dot_product_attention. On a GPU it dispatches to fused flash or memory-efficient kernels automatically, so there is no separate flash-attention dependency and no custom kernel. On a CPU it falls back to a plain math kernel, which is why the smoke test runs anywhere.

design choiceThe project owns its pretraining loop for full control over sharding, packing and budgets. The Hugging Face Trainer is kept for fine-tuning only.

§ 2.2tier ladder

The ladder: what fits where

Training is limited by GPU memory, not system memory. With AdamW and mixed precision, each parameter costs roughly 16 bytes during training (weights, gradients, a full-precision master copy and two optimizer moments) plus activations; 8-bit AdamW brings that to roughly 10 bytes.

Tier presets in the trainer. Parameter counts assume the 152K-token Qwen2.5 vocabulary unless noted. Memory figures are the repository’s own estimates.
Tier	Parameters	GPU memory to train	Where it runs
T0	~20M	~1 GB	Any machine; used for the CPU smoke test
FC	~33M	Fits a 6 GB laptop GPU overnight	First-contact tier, with a custom 16K-token vocabulary
T1	~200M	~5–6 GB	Borderline on 6 GB (batch of 1, gradient checkpointing); otherwise one rented 12–24 GB GPU
T2	~460M	~12 GB	One 24 GB GPU
T3	~1.5B	~30–40 GB	One 40–80 GB data-centre GPU
T4+	≥3B	≥48 GB	Several GPUs with sharded training; a cloud job, not a single box

Figure 2.1. The fainter end of a bar shows the range of an estimate. The same numbers are in the table above.

why a custom vocabAt small sizes the tied embedding table (vocabulary × width) dominates the parameter count. A 16K vocabulary instead of 152K is the cheapest way to make a tiny model fit and train faster.

§ 2.3ADR 0021, 0022

Data: streamed, deduplicated, labelled

Streamed, not downloaded. FineWeb-Edu is read as a stream, tokenized and packed into fixed-length blocks, so no multi-terabyte download is needed.
A mixture with a bandit. A registry of sources (FineWeb-Edu, Wikipedia, OpenWebMath, a small slice of The Stack, a StackExchange set and Project Gutenberg English) can be mixed by a UCB1 bandit. A missing or unreachable source is skipped with a warning instead of crashing the run.
A local path. For reliable offline runs, the trainer can also read a plain text corpus from disk, such as a folder of public-domain books.
Near-duplicate filtering. A MinHash filter (128 permutations over 5-token shingles, fixed seed) removes near-duplicates deterministically. It is off by default and switched on per run.
Provenance on everything. Every source carries a class: clean-deterministic, open-weight, self-generated or tainted. Unknown sources default to tainted. Outputs of closed commercial models are never used as training data.

before any releaseA per-source data-licence register still has to be written, and a lawyer has to review it, before any weights leave the lab.

§ 2.4ADR 0021

Safety rails in the trainer

Long runs get interrupted, so the trainer assumes they will be:

Optimisation: AdamW or 8-bit AdamW, cosine schedule with linear warm-up, gradient accumulation and checkpointing, bf16 autocast and gradient clipping.
Atomic checkpoints: a checkpoint is written to a temporary directory, flushed to disk and only then renamed, so a half-written checkpoint is never picked up.
Safe resume: resuming refuses an incompatible checkpoint and names every mismatched field; it prefers the newest promoted checkpoint over the newest one.
Promotion: a checkpoint is promoted only with finite weights, finite optimizer state and a validation loss within a set band of its promoted ancestor.
Watchdogs: a NaN guard, a save on SIGTERM, wall-clock budgets, retries with back-off for streaming errors, and an opt-in health sentinel that escalates from NOMINAL to HEIGHTENED, ACTIVE and CRITICAL on loss spikes, gradient-norm shifts, throughput collapse or validation divergence.

uv run sultaan pretrain run --smoke
uv run sultaan pretrain preflight --config configs/pretrain/first-contact.yaml
uv run sultaan pretrain tokenizer --source fineweb-edu --docs 50000 --vocab 16000

CPU smoke test, hardware preflight, and the streamed 16K tokenizer recipe from the first-contact decision record (ADR 0026).

§ 2.5lab record

Runs recorded so far

T0 smoke test (CPU, synthetic data)

A 20-step run on synthetic data, used to prove the loop end to end: loss fell from 5.49 to 4.33, and two runs give identical results because everything is seeded.

First contact (16 July 2026)

Its decision record (ADR 0026) sets a modest goal: a model small enough to train overnight on a laptop and produce word-shaped output by morning. Success means “stopped emitting noise”, not a quality score.

Hardware: A laptop RTX 3050 with 6 GB, native Windows CUDA
Model: 33.9M parameters, custom 16K byte-level BPE vocabulary
Data: 500 Project Gutenberg books, 54.1M tokens
Result: Loss 9.78 → 3.89 over 20,000 steps, at about 26,000 tokens per second

Figure 2.2. The straight line is not the real curve. No held-out score was published for this run.

Caveats. These figures come from commit messages on a branch that is not yet merged into main. The checkpoint itself has not been located since, so the run is treated as reproducible from its configuration and data fetcher rather than as an artefact on hand. A loss figure is not a quality claim.

overnight supervisorA shell supervisor runs a preflight, halves the batch on an out-of-memory error and resumes, and gives up only after repeated rapid crashes, so one bad moment does not cost the night.

§ 2.6practical route

The practical route: LoRA on an open model

A QLoRA fine-tuning scaffold already exists (PEFT with optional 4-bit quantisation and experiment tracking in mlflow). What it lacks is chat-format supervised fine-tuning: a chat template, masking so that only the assistant’s tokens are learned, and a padding token separate from end-of-sequence.

The plan is to add that, train an adapter on owned hardware (an Apple-silicon laptop or the CUDA laptop), evaluate it against its base model, and keep or reject it with a written record either way. A rented-GPU run at tier T1 happens only if the owner decides to spend on it.