Faster Inference, From the First Token
Silk gives your GPU fleet a persistent, resilient KV cache tier over NVMe-oF block storage with RDMA. GPUs reuse context instead of recomputing it, and small cache reads don't hold up prefill.
Inference Hits the Memory Wall
Long contexts, multi-turn chat and agentic loops create KV cache faster than GPU memory can hold it. Once that cache is evicted, the GPU recomputes the prefill from scratch: users wait longer for the first token, and the most expensive hardware in your data center repeats work it's already done.
Local NVMe is stranded
Node-local SSD extends memory, but the cache is tied to one server. It can't be shared across the fleet, and it disappears when that node fails, restarts or is rescheduled.
File and object tiers add overhead
Shared file and object stores solve sharing, but every small cache read pays protocol and metadata cost. Those costs stack up at the exact moment prefill is waiting.
What Silk Changes
A shared, persistent KV cache layer built for the small, frequent I/O that inference generates.
Serving density
Offload KV cache to Silk instead of holding it all in HBM. Each GPU keeps more concurrent sessions, longer contexts and more models in flight.
Time to first token
Reload a cached prefix instead of recomputing it. Returning users, shared system prompts and agent loops get their first token faster, even at very long context lengths.
Accelerated KV cache
NVMe-oF block access over RDMA, with no file system metadata in the data path. Silk keeps its metadata fully resident in memory and avoids tree lookups, so small, frequent cache reads stay consistently fast.
AI framework ready
Works with the KV cache integrations in the inference stack you already run, including vLLM, LMCache, NVIDIA Dynamo and llm-d, with Kubernetes provisioning through the Silk CSI driver.
A Persistent KV Cache Tier for Every GPU
Silk adds the shared, durable tier underneath GPU, host and local memory.³
GPU HBM
Fastest and smallest. Holds the active working set.
Host DRAM
Larger, still volatile. Lost on restart.
Local NVMe
Large, but stranded on one server and lost on node failure.
Silk persistent KV cache tier
Shared across every GPU node, resilient to node failure and restarts, scaled independently of compute.
Why Block Wins for KV Cache
KV cache traffic is many small, latency-sensitive reads and writes. The protocol carrying them matters as much as the bandwidth behind it.
File and object paths
- Metadata lookups and open, close and lock calls on every request
- Per-request protocol overhead that grows as transfers get smaller
- Millions of small cache objects for the metadata service to track
- Latency that varies with namespace size and load
Silk block path
- NVMe over Fabrics with RDMA, direct block addressing, no file system in the data path
- Metadata fully resident in memory, no tree traversal
- Many parallel queues sized for small I/O
- Consistent latency as the cache and the fleet grow
Less software in the data path
NVMe over Fabrics with RDMA moves data between compute and storage without involving the processor, cache or operating system on either side. With no file or object layer in between, each small KV cache read takes the shortest path from storage to compute.
Time to first token at similar bandwidth
Independent research found two KV cache offload paths running at similar bandwidth, around 24 to 26 GB/s, delivered very different time to first token. The difference came from transfer granularity: roughly 21 large transfers per request versus roughly 640 small ones. Aggregate bandwidth alone doesn't predict serving latency.⁴
Cache That Survives
If the cache disappears every time a node restarts, you're back to recomputing prefill.
Persistent
Cache outlives GPU node restarts, upgrades and rescheduling, so you avoid a cold start and a wave of recomputed prefills.
Resilient
No single controller to fail. Silk's scale-out architecture keeps the cache tier serving through component failures.
Shared
Any GPU node can reuse any session's cache, which supports disaggregated prefill and decode and smarter request routing across the fleet.
Measured, With the Conditions Stated
From a single client
Duplex throughput over one 200GbE NIC, mixed read/write workload.¹
From a single client
Measured in public cloud. Full platform IOPS scales with client count.²
Better cost per token
Serving with a Silk KV cache tier versus serving with no KV cache.⁵
Explore Silk for AI Clouds
Make Every GPU Serve More
See how a persistent KV cache tier on Silk fits your inference stack, your hardware and your tenants.
- 1. Single client with one 200GbE NIC, mixed workload of roughly 65% reads / 35% writes. Figure is duplex (read plus write combined); 200GbE is full duplex.
- 2. Single client in public cloud. Full DataPod IOPS is tested across multiple clients and depends on client count and client CPU.
- 3. Tier names follow the G1 to G4 memory hierarchy used by NVIDIA Dynamo's KV block manager.
- 4. Kanichai, De Matteis and Trivedi, "Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs", Vrije Universiteit Amsterdam and IBM Research Zurich, arXiv 2609.11744, September 2026. The study compared LMCache with native vLLM offload to NVMe SSDs; it did not compare block and file protocols.
- 5. Silk analysis comparing inference with a KV cache tier against no KV cache.