mzCache: On-Device LLM Memory Management under Multitasking
What it is
mzCache is an on-device LLM inference system designed for mobile multitasking environments. It uses restoration-oriented memory management to partition LLM memory into fine-grained shared buffers, allowing elastic eviction and concurrent CPU-side restoration while the GPU performs zero-wait inference.
The problem it solves
Mobile operating systems evict LLM memory (weights and KV cache) under multitasking pressure, forcing slow storage reads or full KV cache recomputation that severely degrades inference responsiveness.
What you could use it for
- Running LLMs on mobile devices while users switch between apps
- Reducing latency spikes caused by OS memory pressure during multitasking
- Optimizing Time-to-First-Token for on-device AI assistants
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.