Posts by Collection

portfolio

DynaSpa

Published: November 04, 2024

DynaSpa is a dynamic sparse GPU execution framework for convolution and attention. It selects custom GPU kernel variants for input-dependent sparsity patterns and uses sparsity-aware tiling to group irregular active regions into GPU-friendly work tiles, improving shared-memory/L1 reuse and reducing global-memory traffic. The system achieves up to 7.8× operator speedup.

Download Paper

DAF

Published: June 23, 2025

DAF is a memory-efficient DNN training runtime that dynamically adjusts activation precision according to runtime importance. Kernel fusion reduces quantization and memory-movement overhead, while a page-based memory manager supports variable-sized quantized activations and reduces training memory usage by up to 22.9×.

Download Paper

TokenFlow

Published: April 27, 2026

TokenFlow is a responsive LLM serving system with buffer-aware preemptive scheduling built on SGLang. Its scheduler tracks each request’s buffered tokens and target consumption rate, pausing requests that are overproducing and resuming those running low to maintain responsive streaming under bursty workloads. Proactive KV cache migration between GPU and CPU memory overlaps cache movement with computation, improving effective throughput by up to 82.5% and reducing P99 time to first token by up to 80.2%.

Download Paper

LeanStream

Published: September 01, 2026

LeanStream is a GPU-centric LLM inference runtime for models whose weights do not fit in GPU memory. It continuously refines weight priorities from partial GPU results, asynchronously streams high-priority weights from storage, and executes loaded weights as they arrive to overlap I/O with computation. On edge SoCs, fine-grained CPU-GPU coordination through unified memory and custom GPU kernels delivers up to 2.1× higher token-generation throughput and 7.5× lower memory usage. A server-side extension using persistent GPU kernels for remote weight streaming is in development.

Download Paper