AirLLM
What it is
AirLLM is an inference framework designed to significantly reduce the memory footprint of Large Language Models (LLMs). It enables running massive models (e.g., 70B to 2.8T parameters) on consumer-grade GPUs by leveraging per-expert streaming for sparse Mixture-of-Experts (MoE) models.
The problem it solves
High VRAM requirements for large language models, which typically necessitate expensive enterprise-grade GPUs or high-end clusters.
What you could use it for
- Running Llama 3.1 405B on a single 8GB GPU
- Inference of DeepSeek-V3 (671B) on ~12GB VRAM
- Deploying Kimi K3 (2.8T) on under 4GB VRAM
- Running Qwen3.8-27B dense VL models in 3.33GB VRAM
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.