Our 125M legal base model, fine-tuned to answer questions closed-book: no context passage, it answers from its own weights and refuses when it does not know. Trained on 7,141 self-contained, faithfulness-judged, decontaminated Q&A pairs (10% refusals). It is tiny, so expect fluency over reliable recall.
A supervised fine-tune (SFT) of the 125M base on closed-book Q&A. Unlike a grounded/RAG model, no passage is supplied at inference: the model answers only from what it internalised during pretraining. At ~125M parameters (~32MB of usable knowledge) it speaks the legal register well but does not reliably recall specific facts, which is exactly the limitation the larger models and the alignment stages (DPO, RLAIF) explore.
Served scale-to-zero on Modal, so the first question may take ~20–30s while the model wakes.
| Stage | Cost | Share |
|---|---|---|
| Data pipelinecorpus clean + dedup + 16K tokenizer + tokenize (CPU) | $2.15 | 1.8% |
| Pretraining10 epochs on 8x H100, 21.9B tokens seen, incl. smoke runs | $118.64 | 97.7% |
| QA dataset (shared)1/9 share of the 7,141-pair closed-book QA set | $0.40 | 0.3% |
| Fine-tuning (QA SFT)full fine-tune on H100 | $0.29 | 0.2% |
| Total to assemble | $121.48 | 100% |
Pretraining, with the data pipeline that fed it, is 99.4% of the bill. Everything after it (fine-tuning) came to $0.69. Teaching a model to behave is cheap; teaching it to know is not.
Figures are actual invoiced Modal usage (GPU + CPU + memory) for this model’s lineage, not estimates. Shared inputs (the QA and preference datasets) are charged at this model’s share, and their generation also used ~$1–2 of external LLM API calls, included above. Serving is billed separately and scales to zero.