turboquant-mlx
What it is
A Python library for Apple Silicon that implements extreme KV cache compression (1-3 bit) for LLMs using the MLX framework. It uses PolarQuant for unbiased dot-product estimation and asymmetric compression to reduce memory usage by up to 5x while preserving accuracy.
The problem it solves
Standard FP16 KV caching consumes excessive memory, limiting context length and model size for local LLM inference on Apple Silicon devices.
What you could use it for
- Running large LLMs like Llama 3 or DeepSeek R1 on Apple Silicon Macs with limited unified memory
- Extending context window length for local inference without hitting VRAM limits
- Deploying OpenAI-compatible local LLM servers with reduced memory footprint
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.