The makers of the llm-d project have announced the general availability of the release version of llm-d 1.0. Key open-source AI infrastructure milestone. The production-ready announcement of this new open-source LLM (large language model) inference solution, certified by the CNCF, offers enterprises a resilient, open-source platform to accelerate large language model inference to support massive multi-tenant, hybrid cloud deployments.
Launched in 2025 as a product of a partnership between Red Hat, IBM, Google Cloud, Core Weave and NVIDIA, llm-d is tackling one of the biggest bottlenecks in enterprise IT: the skyrocketing infrastructure and operating costs to run production AI at scale.
Solving the Operational Challenges of Production AI
The early AI enterprise emphasis was on model training, but for production workloads, inference management has become the predominant challenge – especially for agentic AI workloads. Autonomous agents experience transient traffic spikes and necessarily operate on long contexts with tremendous amounts of data about the world, and where just managing the context can require more compute than translating that context into final output tokens.
Designed as a co-designed collection of interconnected modules, llm-d 1.0 offers native hardware heterogeneity and distributed inference support across multiple model servers including vLLM and SGLang.
Below are some technical highlights of llm-d 1.0:
Multi-Tenant Quality of Service (QoS): Priority and fairness-aware flow control guarantees evenly sharing of GPU fleets for latency-sensitive user queries and bursty agent sessions.
Agent-Aware Workload Routing: Session-aware routing keeps multi-turn conversational exchanges close to the cached context, with turn-priority scheduling and native Responses API support cutting down on duplicate recomputation.
Zero-Downtime Infrastructure: High-availability router configurations with hot standby and coordinated prefill/decode replica management to support rolling updates without service outage.
Hardened Standard Foundation: Standardized v1 configuration API provides cross cloud, cross accelerator platform proven for Day 2 enterprise operations.
Also Read: Meta Unveils ‘Muse Image’: Next-Gen AI Image Generation Integrated Into Meta AI
Ecosystem Momentum and Track Record of Cost Savings
To date, the llm-d community has grown to include more than 500 active contributors from 25+ companies, such as cloud providers, frontier model developers and chipmakers.
Deployments across all production benchmarks show dramatic improvements in operational efficiency and cost management:
Quantitative research IBM Research showcased an implementation in which open models using llm-d reduced inference costs by a factor of 11.5, with a 98% prefix-cache hit rate and an average serving cost of $0.11 per million tokens.
Large-Scale Agentic Benchmarks: Running GLM 5.2-a 753b open-weight model on existing infrastructure of NVIDIA GPU for llm-d, we can deliver an output token of 6.6 million per minute at peak per throughput. We could serve up to 3,000 concurrent coding agents with 0 preempetions, resulting in 5-10x token cost saving than commercial API offerings.
Supporting Quotes
Chris Aniszczyk, CTO, CNCF
“AI inference is quickly becoming a core cloud native workload. llm-d 1.0 helps organizations meet this shift at scale by bringing together open source inference capabilities with the requirements that matter in scaling production to then meet the demands of agentic workloads. It’s an important step toward making high performance open source inference more operationally mature and accessible across the cloud native ecosystem.”
Brian Stevens, senior vice president and AI chief technology officer, Red Hat
“As organizations move beyond AI experimentation and into full-scale production, the bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads. Open source collaboration is essential to solving these day-two operational challenges at scale. With llm-d 1.0, the community is delivering a standardized, hardened core for distributed inference to empower enterprises to optimize hardware performance, address operational costs, and build high-density AI architectures on an open foundation.”
Priya Nagpurkar, vice president, AI native systems, IBM Research
“At IBM, we see firsthand that the challenge of enterprise AI is no longer just building powerful models but running them efficiently and reliably at scale. llm-d 1.0 represents an important milestone in creating an open foundation for distributed inference to help organizations get more value from their AI infrastructure and more control on the AI capabilities they depend on. As agentic workloads become more prevalent, open technologies like llm-d will be critical to improving performance, lowering costs, and accelerating innovation across the AI ecosystem.”
Mark Lohmeyer, VP & GM, AI & Computing Infrastructure, Google Cloud
“Agentic AI is unlocking tremendous value for enterprises and consumers across a broad range of use-cases. However, as these workloads scale rapidly in production, they place unique demands on the infrastructure and require new approaches to deliver the performance, reliability, and cost-effectiveness our customers need. The release of llm-d 1.0 is an important milestone for the industry to address these needs. As a founding contributor to llm-d, Google Cloud is integrating llm-d 1.0 into Google Kubernetes Engine, and enabling it across our Google TPU and NVIDIA GPU portfolio, enabling customers to scale multi-tenant environments, optimize their inference performance, and dramatically lower their overall serving costs on a hardened, open-source foundation.”
Shadi Saba, vice president of engineering, AI/ML at CoreWeave
“Every team running inference at scale hits the same problems: context that often costs more to manage than the output, tenants competing for the same fleet, and upgrades that can’t interrupt traffic. llm-d 1.0 gives the community a hardened, open base to solve them once instead of separately. CoreWeave helped found the project, and customers can run llm-d on CoreWeave Kubernetes Service alongside NVIDIA Dynamo, choosing the serving stack that fits their workload.”
Ujval Kapasi, vice president, Engineering AI Frameworks, NVIDIA
“The next wave of agentic AI demands inference that delivers performance and efficiency at scale. Bringing llm-d to NVIDIA Vera Rubin will help enterprises build and scale advanced AI applications on the next generation of NVIDIA accelerated infrastructure. Through our collaboration with the llm-d community and contributions from open source NVIDIA Dynamo, we’re helping enterprises get more from their AI infrastructure investments.”
Phil Guido, executive vice president and chief commercial officer at AMD
“With llm-d v1.0, we are advancing open, high-performance distributed inference on AMD Instinct™ GPUs. Agentic workloads require infrastructure that delivers responsiveness and efficiency. Working with the llm-d community, we’re delivering the latest inference innovation on an open foundation, giving developers greater flexibility and choice to deploy AI at scale.”
Acyr Locatelli, VP, Modeling, Cohere
“At Cohere, we face the dual challenge of training frontier language models while serving them efficiently at enterprise scale. llm-d helps us optimize for both, whether maximizing throughput for large-scale reinforcement learning workloads or delivering the responsiveness users expect in production. Its flexible, open-source architecture aligns closely with our commitment to sovereign AI, giving organizations greater control over how AI is deployed, operated, and scaled.”
Salman Paracha, SVP AI Engineering, DigitalOcean
“DigitalOcean is excited about the llm-d 1.0 release and relies on llm-d as foundational distributed inference software for DigitalOcean Serverless Inference and Managed Agents, serving trillions of tokens. Its Kubernetes-native design and integration with the Kubernetes Gateway API Inference Extensions provide robust orchestration and endpoint picking, while production-proven paths for dynamic flow control, inference-aware load balancing, and prefill/decode disaggregation help us operate reliably at scale. We collaborate closely with the llm-d community, contributing upstream features and feedback, and look forward to the continued development of llm-d.”
Vladislav Tankov, Director of AI, JetBrains
“JetBrains has been working with llm-d since its early releases, and it is great to see the project reach 1.0. llm-d now runs in production behind our new Junie Lite plan and serves a significant share of its traffic. Our traffic ranges from long agentic coding sessions to short requests, and predicted latency-based scheduling was straightforward to enable and keeps latency consistent across that mix. Junie Lite brings our AI coding agent to developers at no cost, and llm-d helps make that possible by letting us serve more requests per GPU at lower cost and with faster responses. We have enjoyed working with the llm-d team and congratulate them on this release.”
Mathis Felardos, Lucas Fernandes-Martins and Nikhil Thomas (Mistral, Inference Engineers and llm-d contributors)
“Serving frontier models across interactive applications and high-volume APIs means handling steep traffic spikes without letting bursty workloads degrade latency for priority customers. With llm-d’s prefix-cache-aware routing, flow control and disaggregated serving support we can enforce strict priority bands across our commercial tiers, automatically holding or shedding lower-priority requests when pools saturate so paid traffic meets our latency SLOs. Mistral is a proud supporter of open-source software, and we will continue to contribute back to llm-d and DisaggregatedSet projects.”
Nishchaie Khanna, Sr Director of Engineering, AI Platform, Roblox
“At Roblox, serving large language models efficiently across a mix of cloud and on-premises GPUs is one of our core infrastructure challenges. llm-d gives us a Kubernetes-native foundation for disaggregated serving and cache-aware routing on the hardware we already run, without tying us to a single inference engine. We’re excited to build on llm-d 1.0 and to contribute back to the community.”
