Homebrewed SLMs: Autoregressive and Masked Diffusion Language Models Built at Home for Portuguese
What it is
This is a tutorial and Python package for training small language models (11-12M parameters) from scratch on a single consumer GPU, specifically for Brazilian Portuguese. It provides notebooks and a tested package to train both autoregressive and masked diffusion models, along with a preregistered evaluation protocol to assess their performance on legal tasks.
The problem it solves
The high cost and resource intensity of training language models, which typically requires data centers, preventing individual researchers or small teams from building and evaluating custom models for specific domains or languages.
What you could use it for
- Training small language models for Brazilian Portuguese on consumer hardware
- Comparing autoregressive vs. masked diffusion generation capabilities
- Evaluating small models on legal text tasks like tamper detection and cross-reference filling
Why it matters
Democratizes the creation of domain-specific language models by showing that high-quality, specialized models can be built without data center resources, enabling researchers and developers to experiment with novel architectures like masked diffusion on limited hardware.
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.