TokenRouter: Efficient Serving System for Token-Level LLM Routing
What it is
TokenRouter is a serving system designed for token-level LLM routing, which distributes inference work across different models at the token granularity. It uses a request-centric programming model for developers and a model-centric execution runtime that launches subservers for each LLM with a delayed-batching scheduler optimized via a mathematical throughput model.
The problem it solves
Existing LLM serving systems are built on single-LLM assumptions, causing severe step desynchronization, frequent batch admission delays, and high implementation complexity when handling fine-grained token-level routing.
What you could use it for
- Serving token-level routed LLM inference to improve cost-quality trade-offs
- Deploying multiple LLMs where specific tokens or steps are routed to specialized models
- Optimizing decoding throughput for fine-grained model routing workloads
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.