TokenFlow

TokenFlow is a responsive LLM serving system with buffer-aware preemptive scheduling built on SGLang. Its scheduler tracks each request’s buffered tokens and target consumption rate, pausing requests that are overproducing and resuming those running low to maintain responsive streaming under bursty workloads. Proactive KV cache migration between GPU and CPU memory overlaps cache movement with computation, improving effective throughput by up to 82.5% and reducing P99 time to first token by up to 80.2%.