Fifty million parameters.
Build and train a language model capped at 50 million parameters. You design the architecture, curate the data and write the training loop — how much capability can you pull out of a deliberately small model?
Foundational LLM
Build and train a language model capped at 50 million parameters. You design the architecture, curate the data and write the training loop. We want to see how much capability you can pull out of a deliberately small model.
Example directionsWhat your build must meet.
- Parameter cap
- No more than 50,000,000 total trainable parameters — including token embeddings and the output head. Your repo must include a script or printed output showing the count, plus your model config.
- From scratch
- Trained from scratch: no initialising from pretrained weights, no fine-tuning an existing model, no distilling from a larger one.
- Allowed
- Standard frameworks and libraries (PyTorch, JAX, TensorFlow, Hugging Face transformers, tokenizer libraries…) and public datasets (FineWeb, TinyStories, The Pile…). “From scratch” means the weights and the training run, not reinventing your tooling. Credit everything.
- Not allowed
- A hosted inference API as the model itself — including the Featherless sponsor perk, which is for tooling, data processing and evaluation.
- Reporting
- Your README states the hardware you trained on, total training time and approximate compute. Training efficiency is scored and can’t be judged without it.
Report your numbers in the README and include the evaluation script you used.
Scored on these.
Judges score each project only against its own track’s criteria. Ties are broken by the first criterion listed.
Perplexity & Accuracy
Language-modelling quality on held-out validation sets. Lower perplexity and higher benchmark accuracy score higher.
Reasoning Performance
Performance on logic, math and commonsense reasoning tasks, evaluated through standardised benchmarks.
Training Efficiency
Compute efficiency, training speed and memory footprint. Efficient architectures and training strategies are rewarded.
Innovation
Novel architecture decisions, creative data strategies or unique training approaches.
Documentation & Demo
Clarity of the README; reporting of parameter count, training setup and benchmark results; how well the demo video explains what you built.