llm-d 1.0 is a Kubernetes inference router. Google, the project, and IBM all dated it October 20.
Google, the llm-d project, and IBM all mark llm-d 1.0 on October 20. The router keeps a session on the replica that already holds its cache. The speed claim is Google's.
The short version
The llm-d 1.0 notes, dated October 20, 2026, describe a Kubernetes-native router that sends each request to a replica, prefers one that already holds that session's cache, and can shed load. Google's speed figures are Google's measurement, not a second lab's.
A router in front of the accelerators
Google's October 20 post says llm-d 1.0 is an open-source, Kubernetes-native stack for distributed inference, announced at the PyTorch Conference. It puts accelerator pools behind one endpoint: one router, one cache index, and one flow-control policy. Agents call a single API. The router picks the replica for each step. Adding a pool, or moving a model, is a platform change. The project blog the same day calls 1.0 the release where that stack meets day-two work: several tenants, upgrades, and agent sessions.when one region is the control plane.
Cache affinity is the routing rule
Google says the router tracks the replicas that hold each cached prefix and weighs load and predicted latency. When accelerator memory fills, the post says llm-d can move the cache to host memory and then to SSD or Managed Lustre, using connectors the serving stacks already have, rather than dropping the prefix. The point of the rule is that a long conversation should not start cold on a random replica. A plain Service, in Google's comparison, does not do that.Clairic returns typed values.
The number is Google's, on one workload
On an agentic coding workload on Ironwood, Google says prefix-aware routing plus host-memory offload cut median time to first token by more than 85 percent and raised throughput 2.4 times versus a plain Kubernetes Service, as of October 2026. That is a vendor measurement on a named chip and a named comparison. It is not a score this paper ran, and it is not a promise for every model. The project post lists other operators' figures the same way: each one stays with the operator who published it.what a process node measures.
Day-two features the project names
The GitHub release, as the project blog summarizes it, graduates the router configuration to a stable API and adds flow control that can shed load so a burst does not starve a latency-sensitive request. Disaggregated prefill and decode replicas can roll together. A session-state plugin keeps a multi-turn conversation near its cache. Those are the 1.0 claims. They are configuration and scheduling, not a new weight file. Asked directly, Does 1.0 mean every claim was remeasured here? No. Customer figures in the project post, including cache hit rates and tokens per second, belong to the organizations that reported them. This page does not rerun the fleet.what a license obligates.
Who says they already run it
The project post attributes production use to named companies: Google on its inference gateway, AWS on SageMaker HyperPod, IBM Research for coding agents, and others including Mistral, JetBrains, and Red Hat. Each sentence is that organization's account, quoted through the project. A reader who wants the measurement reads that organization's writeup. This page files the release, not a leaderboard. Asked directly, Whose 85 percent figure is that? Google's. The post says prefix-aware routing plus host-memory offload cut median time to first token by more than 85 percent on an Ironwood TPU coding workload, versus a plain Kubernetes Service, as of October 2026.an evaluation a second lab can run.
GKE is one way to run the router
Google says GKE Inference Gateway can run the llm-d router as a managed gateway, including across regions, and that mixed TPU and GPU pools can sit behind the same endpoint. Autoscaling on queue depth and predicted latency is described as a Kubernetes feature the stack can use. Running it on Google Cloud is an option the blog documents. The release itself is the upstream project. Asked directly, Is the project a company's private fork? The llm-d post calls it a CNCF Sandbox project with more than 500 contributors from over 25 organizations. Google, Red Hat, and IBM are named as founders. The code is the project's, not a single cloud's console.the distributed-cloud desk.
What 1.0 does not settle
A preview note in Google's own routing docs, updated earlier in October, still says a predicted-latency feature needs a warm-up and a homogeneous pool. That limit sits beside the 1.0 announcement. The release does not say every deployment is past preview, and it does not publish a default that this page should invent. Asked directly, Is llm-d 1.0 a new model? No. The October 20 notes describe a serving stack in front of models that vLLM or SGLang already run. The model weights are not what this release cuts.Commerce storefronts and WebMCP.
A fleet is not a region
The dependency is where a request lands when several accelerators already hold pieces of the same context. llm-d 1.0 answers with a router and a cache index. It does not move the model into the user's browser, and it does not replace the serving engine. The record is the October 20 posts from Google, the project, and IBM. Asked directly, What does the router remember? Google's post says the router tracks which replicas hold a cached prefix and weighs load and predicted latency when it picks one. A session stays near the copy of its context.what on-device inference keeps.
What stays on the record
October 20, 2026. llm-d 1.0. A CNCF Sandbox project. One endpoint in front of many accelerators. Prefix-aware routing. Google's Ironwood comparison, labeled as Google's. IBM's production note as IBM's. No second-lab rerun on this page. The short answer this page will stand on is this: The llm-d 1.0 notes, dated October 20, 2026, describe a Kubernetes-native router that sends each request to a replica, prefers one that already holds that session's cache, and can shed load. Google's speed figures are Google's measurement, not a second lab's.Futureweb.
Sources
The reports this brief is filing. Futureweb did not republish them.
Questions
Is llm-d 1.0 a new model?
No. The October 20 notes describe a serving stack in front of models that vLLM or SGLang already run. The model weights are not what this release cuts.
What does the router remember?
Google's post says the router tracks which replicas hold a cached prefix and weighs load and predicted latency when it picks one. A session stays near the copy of its context.
Whose 85 percent figure is that?
Google's. The post says prefix-aware routing plus host-memory offload cut median time to first token by more than 85 percent on an Ironwood TPU coding workload, versus a plain Kubernetes Service, as of October 2026.
Is the project a company's private fork?
The llm-d post calls it a CNCF Sandbox project with more than 500 contributors from over 25 organizations. Google, Red Hat, and IBM are named as founders. The code is the project's, not a single cloud's console.
Does 1.0 mean every claim was remeasured here?
No. Customer figures in the project post, including cache hit rates and tokens per second, belong to the organizations that reported them. This page does not rerun the fleet.