xLLM
What it is
xLLM is a high-performance inference engine designed for Large Language Models (LLMs), Vision Language Models (VLMs), and Diffusion Transformers (DiT). It is optimized for Chinese AI accelerators and features a decoupled service-engine architecture for efficient enterprise-grade deployment.
The problem it solves
Lack of optimized inference frameworks for diverse AI accelerators and high costs/latency in enterprise-scale model deployment.
What you could use it for
- High-throughput LLM inference on Ascend NPU or Cambricon MLU hardware
- Deploying Vision-Language Models (VLM) with low latency
- Large-scale retail business model inference (e.g., JD.com)
- Hybrid KV cache management for long-context generation
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.