Adaptive KV Retention for LLM Agents at Human-Approval Timescales
What it is
This paper addresses the performance degradation in LLM serving systems when agent requests are suspended for long periods (minutes to hours) awaiting human approval. It proposes a tiered retention controller that balances the cost of keeping Key-Value (KV) cache in GPU memory against the cost of recomputing it, optimizing for active serving goodput.
The problem it solves
Existing LLM serving systems are optimized for seconds-scale pauses; long suspension times for human approval cause either massive GPU memory waste (if KV is retained) or high latency (if KV is evicted and recomputed), leading to poor goodput.
What you could use it for
- Serving LLM agents that require human-in-the-loop approval steps
- Optimizing GPU utilization in multi-tenant LLM inference clusters with variable request latencies
- Reducing resume latency for suspended agentic workflows without sacrificing throughput for active requests
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.