Continuous Batching
An inference-scheduling technique that continuously admits and retires generation requests so accelerator batches stay utilized despite variable sequence lengths.
Related Concepts
- Paged Attention
- KV Cache
- Throughput
- Tail Latency
An inference-scheduling technique that continuously admits and retires generation requests so accelerator batches stay utilized despite variable sequence lengths.