[Store] Does Mooncake store plan to support awareness of layers, chunk ordering, and TP?

[Hadigan]

It looks like the current Mooncake store does not have awareness of layer granularity, chunk order (earlier vs. later chunks), or TP ranks. This can cause problems during cache eviction:

1. The store might evict KVs for only some layers, forcing prefill to recompute all layer KVs for those tokens.
2. The store might evict the KVs corresponding to the earliest chunk(s) of a token sequence, forcing prefill to recompute KVs for all chunks/tokens.
3. The store might evict KVs produced by some TP ranks, forcing all TP ranks to recompute KVs.

I think the Mooncake store should be aware of layers, chunk ordering, and TP. On eviction, it should evict a token’s KV caches for all layers and for all TP ranks together (i.e., keep KV state complete per token).
From the chunk-order perspective, eviction policies should prefer removing KVs for later chunks first rather than evicting early-chunk KVs that make prefill expensive.

[stmatengss]

Good idea. Stay tuned! In sglang, we have a plan to support chunked/layers/TP awareness eviction strategies.