Actual measurement of 107 times KV Cache capacity, LUISUAN GP7000 prevents multiple rounds of long sessions from "collapse"
LUISUAN GP7000 SPDK KV Cache and pure DRAM solution · Layered comparison actual measurement
1. Background: KV Cache is eating up the memory budget of the inference cluster.
The KV Cache of multiple rounds of dialogue in large models will be accumulated. Taking the load of this test as an example, the cache increment in each round is "number of concurrencies × output token in each round × 88.1 KB". When the session lengthens to 20 or 50 rounds, the working set quickly crosses the single-machine DRAM boundary - and once the DRAM begins to be evicted, the vLLM chained block hash will collectively invalidate the reuse of the entire session.
We ran this question on a real cluster: whether NVMe-oF full flash can catch the part that DRAM cannot catch.
2. Test environment
| project | Configuration |
|---|---|
| Computing power server | There are two cx81/cx82 units, equipped with Intel Xeon 6767P (256 vCPU) and Xeon 6787P (344 vCPU) respectively, with 1007 GiB of memory. |
| GPU | 8 NVIDIA RTX 6000D per machine (Blackwell, sm_120, single card 83.6 GiB), no NVLink between cards, all PCIe Gen5 x16 |
| inference engine | vLLM Aggregation Mode (Dynamo Scheduling), 4 Engine Instances, TP=4, FP8 KV Cache, block=256 |
| Model | DeepSeek-V4-Flash, 156 GB / 48 shards, dual-machine local deployment |
| network | 200 Gb/s RoCEv2, Mooncake DRAM layer and NVMe-oF layer have separate physical network cards and do not compete for each other’s bandwidth; full GPUDirect (8 cards per machine are registered in RDMA) |
| storage | Green Plan GP7000 all-flash NVMe-oF storage array (dual PCIe 5.0 master / 6×BlueField-3 / 12×200 Gb/s RoCEv2 / 24×7.68 TB U.2 SSD / SPDK NVMe-oF Target) |
3. Comparison method: a deliberately unequal setting
| Configuration | KV drop point | Total number of DRAM segments | Effective capacity | Equivalent token capacity |
|---|---|---|---|---|
| DRAM KV Cache | Mooncake DRAM segment (pure memory) | 400 GiB | 429.5 GB | ≈ 4.8M token |
| LUISUAN GP7000 SPDK KV Cache | DRAM segments + NVMe-oF(SPDK) replica layer | 50 GiB | 46.09 TB | ≈ 1,092M token |
Key points: The DRAM segment of the SPDK solution is only 1/8 of the DRAM solution. It is not "the same memory and a layer of storage", but "cutting the DRAM to 1/8 and relying on NVMe-oF to make up the capacity." Each of the following performance conclusions is based on the premise that the SPDK solution first gives up 8 times the capacity on the DRAM side.
To ensure that the comparison is valid, we use machine criteria to verify the media identity: the total save_put difference between the two modes is +0.1%, and the load_get difference is +0.3%; the DRAM solution writes approximately 1.49 TB objects throughout the entire process, while the NoF layer allocation byte increment is always 0.0 B; both tiers of the SPDK solution are confirmed to be actually written.
4. Test results
4.1 Performance: Throughput equals, high concurrency surpasses
| Dimensions | DRAM KV Cache | GP7000 SPDK KV Cache | in conclusion |
|---|---|---|---|
| Pure read path throughput (f=1.0, C≤64) | benchmark | 0.97–1.03× | Flatten within the noise |
| 20-round steady-state RPS (osl=128, C=4…128) | benchmark | −0.1% ~ −4.9% | The difference is in the noise level |
| C=256 higher concurrency (20 rounds) | 24.27 RPS | 25.67 RPS | SPDK overtakes +5.5% |
| 20 rounds of steady-state TPOT (decoding single token) | benchmark | Out of 6 gears, 4th gear SPDK is slightly better | Decoding is medium independent |
In the comparison of C=256, the hit rate of the two modes is the same bit by bit (0.9750 vs 0.9752), and the store workload is the same (300.5 GB vs 300.3 GB), indicating that the saturation here comes from the engine side rather than the storage side——Simply expanding the DRAM segment from 50 GiB to 400 GiB (8×) without using NVMe-oF at all will not improve throughput.
4.2 Capacity: 107 times, and the payment method is feasible in engineering
Effective capacity increases from 429.5 GB to 46.09 TB, a 107-fold increase. Based on the actual measurement of the NoF pool of 46.09 TB and the estimated writing of approximately 21.5 GB per round, it can continuously support approximately 2,140 rounds of osl=5000 sessions before it is fully written.
4.3 Long sessions: From "collapse" to "hit rate monotonically increases"
Long output scenario (forced output of 5,000 tokens in each round, 20 rounds, C=112):
| Indicators (Round 20) | DRAM KV Cache | GP7000 SPDK KV Cache | difference |
|---|---|---|---|
| KV hit rate | 0.0085 | 0.9516 | 112× |
| single wheel wall clock | 727.3 s | 277.7 s | 2.62× |
| TTFT p50 | 246,236 ms | 16,852 ms | 14.6× |
| Recalculate per request excess | 99,754 token | 117 token | 853× |
| wall clock | 7,480 s | 5,036 s | 1.49× |
| Accumulated writing throughout the process | 5,557.3 GB | 1,063.5 GB | Write volume reduced to 1/5.2 |
The parsing upper limit of the 20th round is the ideal hit rate of 0.9527, SPDK measured 0.9516 = 99.88% of the upper limit; DRAM measured 0.0085 = 0.9% of the upper limit.
In 50 rounds of ultra-long sessions, the hit rate of the SPDK solution increased monotonically from 0.6580 to 0.9797, and the excess recalculation was maintained at 0.06%–0.2% of reusable prefixes for a long time. The whole process ran for 3.90 hours with 0 errors. The DRAM scheme runs out around round 20 under the same load, making a 50-round session arithmetically infeasible.
Working set exceeds DRAM capacity scenario (471 GB working set, DRAM solution has about 19% gap): The second round of wall clock is 121.0 s → 11.4 s (10.6×), the hit rate is 0.4821 → 0.9954, and the recalculation token difference is 113 times.
5. Mechanism: Why "which layer you fall on" is more important than "how fast you read"
DRAM failure is a cliff, not a slope. The vLLM prefix cache is a chained block hash (the k-th block hash contains the k-1th block), and the oldest block evicted by LRU is the chain head of each session; once the chain head is dropped, the entire session reuse becomes invalid and triggers "eviction → recalculation → rewrite → more evictions" positive feedback (the measured single-round write soared from 50 GB to 1,051.8 GB). The moment of collapse can be accurately predicted before testing.
Why can the read path be evened? A single-wheel wall clock consists of TTFT (25–28%) and decoding (70%) with almost no independent store waits; KV reads occur within the TTFT. The hit token supply is spread across three layers: GPU’s own prefix cache 87–91%, external store 8–12%, and local recomputation 2.5%. The media differences of the memory tiers are thus diluted.
in conclusion: The primary issue in KV Cache hierarchical design is not bandwidth, butIs there a drop point that will not be evicted?. NVMe-oF full flash provides exactly this point.
Applicable people
Teams that are deploying or expanding inference clusters for long context models such as DeepSeek, encounter KV Cache hit rates that collapse with session rounds, and need to increase the carrying capacity of multi-round sessions without increasing the memory budget.
Learn more
If you need the full text of the test report, or if you want to do capacity measurement based on the load form of your own cluster, please submit your request through the official website "Contact Us", or leave a message through the public account "LUISUAN Technology" to obtain it.