EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures
What it is
EdgeAgent is a cross-layer inference system designed to optimize multi-agent LLM workloads on unified memory architectures (UMA). It combines zero-copy tensor parallelism to saturate CPU and GPU compute with dynamic scheduling that adjusts draft budgets and evicts stalled agents to maintain hardware utilization.
The problem it solves
Naive CPU-GPU co-execution on unified memory architectures suffers from severe bus contention during the memory-bound decode phase, and static batching fails to handle the high variance in drafting difficulty and tool-induced stalls in multi-agent workloads.
What you could use it for
- Running privacy-preserving multi-agent LLM systems on Apple Silicon devices
- Optimizing inference latency for agents with frequent tool invocations
- Improving hardware utilization for speculative decoding on UMA systems
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.