Tsinghua team trains 2B parameter LLM on consumer hardware

Summary

Researchers at Tsinghua University trained Puro-2B, a 2-billion parameter large language model (LLM), from scratch on consumer-grade RTX 5090 GPUs for a total cost of under $6,900. The team released the full training recipe, weights, and data under an Apache 2.0 license. The paper also introduces the 'Puro Cost Scaling Law,' which predicts that approximately $4,400 is sufficient to achieve performance levels comparable to Qwen2-1.5B.

Why it matters

The pre-training of LLMs has largely remained behind the curtain of industry-scale compute budgets, effectively eliminating the possibility of building models from scratch for most academic laboratories. By publishing a full, reproducible recipe on consumer hardware at a cost even lower than the GPU's own retail price, this work significantly lowers the barrier for independent research. The Apache 2.0 license further ensures that any lab or individual can work with these results without restriction.

Highlights

• Trained with up to 1.4 trillion tokens at FP8 precision on RTX 5090s; the actual cost was under $6,900, while the scaling law predicts approximately $4,400 for Qwen2-1.5B-level performance.
• Puro-2B approaches Qwen2.5-1.5B according to the team's evaluation protocol, whereas reproducing SmolLM3-3B is estimated to cost over $700,000.
• Includes controlled studies examining the impact of pre-training data curricula on post-training, adding methodological depth beyond the cost headline.

Critical take

The '$5,090' figure in the headline is actually a cost estimate derived from the scaling law, not the actual training expenditure (which was under $6,900); this is a subtle but significant confusion. Additionally, a 2B model approaching a 1.5B baseline is a modest achievement. Since the arXiv preprint has not yet undergone peer review, cost claims and performance comparisons should be considered provisional until independently replicated.