Publications

Unified KV Pooling to Accelerate Long-Context LLM Serving

arXiv preprint arXiv:2606.14779, 2026.

Minchul Kang, Changyong Shin, Jinwoo Jeong, Jaerim Park, Woohyun Kim, Bonyul Gu, Dongwoo Kang, Gyeongsik Yang, Chuck Yoo

Download PDF

Abstract

Long-context LLM serving requires offloading KV caches to host-memory and SSDs, but existing mechanisms are not designed for such long contexts. We observe significant inefficiencies in current KV caching in long contexts: high serving latency ~30.7 s, exceeding the typical TTFT requirement of 10 s by more than 3x. Our in-depth analysis explains two major reasons: (1) retrieval is serialized through host-memory and SSD, leaving other host-memory modules and SSDs underutilized, and (2) SSD-based KV retrieval spends 84% of its time in the kernel filesystem rather than actual device access. To address the problems, we propose unified KV pooling, which aggregates multiple host-memory modules and SSDs into a single logical pool and distributes KV caches across devices based on their bandwidth. To eliminate the filesystem overhead, we design KV-passthrough, which bypasses the kernel filesystem and directly accesses SSD-resident KV caches from user space via SPDK. Across evaluations on LLaMA 3.1-8B, GPT-OSS-20B, and Qwen3-30B-A3B, unified KV pooling reduces TTFT in long-contexts ~4.1x over state-of-the-art techniques, all making under 10 s. It also reduces blocked I/O time by up to 23.2x by eliminating filesystem overhead.

Paper summary

Retrieve long-context caches in parallel

Using available memory and storage bandwidth together can shorten the wait before a long-context model begins answering.

Long contexts produce KV caches that extend beyond GPU memory into host memory and SSDs. The paper identifies two delays in retrieving them: traffic concentrated on a small subset of devices, and filesystem processing that dominates SSD access.

Pool the devices and simplify the access path

Unified KV pooling treats host-memory modules and SSDs as one logical pool. An orchestrator assigns cache data in proportion to each device’s measured bandwidth and records its location in a global lookup table.

The cache uses multiple devices together to distribute retrieval traffic.

KV-passthrough uses SPDK to access SSD-resident cache data from user space without the kernel filesystem. The design combines device-level parallelism with a shorter software path for those known cache objects.

The orchestrator distributes KV data across pool devices and tracks locations for parallel retrieval.View full-size figure
The orchestrator distributes KV data across pool devices and tracks locations for parallel retrieval.

Measure the time to the first token

The evaluation compares LMCache, layerwise pipelining and asynchronous loading variants on LLaMA-3.1-8B, GPT-OSS-20B and Qwen3-30B-A3B. At 128K context, unified KV pooling reports first-token latencies of 5.6, 5.2 and 9.8 seconds respectively.

For GPT-OSS-20B at 128K, this is approximately 4.1× lower latency than LMCache and 3.3× lower than the variant with both pipelining and asynchronous loading. An ablation separately enables pooling and passthrough; the combined design performs better than either component alone.

First-token latency with unified KV pooling at 128K context.
ModelLatency
LLaMA-3.1-8B5.6 s
GPT-OSS-20B5.2 s
Qwen3-30B-A3B9.8 s

Key takeaways

  1. KV offloading is limited by both device balance and software overhead. Bandwidth-weighted pooling and direct SSD access address different parts of the same data path.
  2. The mechanisms reinforce each other in the measured system. The ablation attributes lower first-token latency to pooling and passthrough together.