AI Inference

Intelligent Computing Center·AI Inference Acceleration

KV Cache offloading · GPU utilization increased to 94%

Project background

For large model inference scenarios, video memory has become a system bottleneck: KV Cache occupies a large amount of HBM, resulting in a limited context length that can be carried by a single card and low GPU utilization.

Solutions

Using the GP7000 G3.5 (CMX) storage platform, the KV Cache is migrated to the Ethernet flash cluster (JBOF/EBOF) through full hardware offloading, allowing the GPU to focus on token generation.

<20μs microsecond-level latency, close to memory access experience
GPU utilization increased to over 94%
Contextual recalculation ratio reduced from 80–90% to less than 10%
Natively compatible with Dynamo scheduling and NIXL transport library

Implementation value

While maintaining accuracy, it significantly improves single-machine concurrency and throughput, reduces reasoning costs per unit Token, and supports large-scale commercial deployment.