Cider: W8A8/W4A8 Inference and Optimized SDPA for Apple Silicon
What it is
Cider is a Python library built on MLX that enables W8A8 and W4A8 quantized inference on Apple M5+ chips by implementing custom INT8 TensorOps kernels. It also provides an optimized Scaled Dot-Product Attention (SDPA) kernel for faster autoregressive decoding, compatible with frameworks like mlx_lm and mlx_vlm.
The problem it solves
MLX lacks support for W8A8/W4A8 activation quantization and has suboptimal SDPA kernels for grouped-query attention (GQA) decode, limiting inference speed and efficiency on Apple Silicon.
What you could use it for
- Accelerating LLM prefill and decode speeds on Apple M5 Macs
- Running quantized LLMs with reduced memory footprint on Apple Silicon
- Optimizing inference performance for vision-language models (VLMs) like Qwen3-VL
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.