CoRun: Deterministic LLM Inference
What it is
CoRun is a scheduling-based system that achieves deterministic LLM inference by addressing batch-dependent GPU execution inconsistencies. It achieves this by using isolated prefill and fixed-shape batched decode stages, leveraging CUDA graphs to maintain performance while ensuring consistent output.
The problem it solves
Inconsistent LLM output caused by batch-dependent GPU execution, which complicates evaluation and training even when using fixed seeds.
What you could use it for
- Model evaluation requiring consistent outputs across multiple runs
- Reinforcement Learning (RL) where deterministic trajectories are required
- High-throughput LLM serving with deterministic output requirements
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.