How Google Cloud runs AI inference at production scale – News

Enterprise technology investment continues to accelerate, but the friction point has shifted. The hard part is no longer training models or selecting architectures. It’s getting those models into production, keeping them responsive under real-world conditions and proving they deliver value once they’re live. AI inference is where initiatives either prove their value or grind to a halt.

That shift matters because inference behaves nothing like the environments where most AI experimentation begins. Latency expectations are unforgiving, and specialized accelerators introduce new cost and capacity constraints. Traditional infrastructure, designed for steady-state workloads, struggles under that combination. Cloud-native orchestration, built to absorb volatility and scale precisely when needed, has become central to making inference viable at enterprise scale.

“When it comes to containerization, Google Cloud has led for a long time with Google Kubernetes Engine and Cloud Run,” said theCUBE’s Savannah Peterson. “What’s interesting now is that with the surge in innovation velocity catalyzed by AI, their suite of tools is powering creatives, scientists, gamers and businesses around the world. Their ability to scale up or down and meet the end user where they are is one of the reasons they’re still at the top of the tech game.”

As enterprises shift from experimentation to execution, the question is no longer whether AI works in theory, but whether it holds up inside real production environments. Realizing practical value from AI requires infrastructure that supports AI inference at scale, reduces friction for developers and delivers measurable business outcomes. Google Cloud has positioned its container-first platforms to meet that moment, aligning orchestration, serverless compute and developer tooling around the practical demands of running AI systems in live environments.

This feature is part of News Media’s ongoing coverage of enterprise AI infrastructure, container platforms and production-scale deployment. (* Disclosure below.)

AI inference takes center stage

For many enterprises, building AI models is no longer the hardest part of the journey. The real challenge begins when those models must be served reliably, at scale and under unpredictable demand. AI inference is where performance, latency and cost converge, turning infrastructure decisions into business outcomes, according to Brandon Royal, product manager of AI infrastructure at Google Cloud.

“A model is only valuable until we can put it behind an [application programming interface] and make it available to do something interesting,” Royal said. “That’s really what inference is all about. It’s taking a model, whether it’s a large language model, a diffusion model for images or a simple model, and providing an endpoint by which we can expose that to users.”

GKE builds on that foundation by treating accelerators as first-class resources rather than exceptions. Models are packaged into containers that can move consistently from testing to production, giving teams a repeatable way to deploy inference services. That consistency becomes critical as organizations juggle multiple models, frameworks and environments, according to Poonam Lamba, senior product manager of GKE AI inference and stateful workloads at Google Cloud, and Eddie Villalba, outbound product manager at Google Cloud.

“Let’s say you have trained a model, now you will take that model, the configuration that you need to run that model — the libraries, the runtime environment, like TensorFlow or PyTorch or JAX — you will package all of these things into a container,” Lamba said. “Now this becomes a portable unit that you will take from your testing to production.”

As inference traffic grows more complex, load balancing falls short. To address that gap, Google introduced the GKE Inference Gateway, which routes requests based on model identity, priority and real-time performance signals, rather than treating inference as stateless web traffic.

“You can also specify if the incoming request is critical, standard or something that you can drop,” Lamba said. “But there’s more: It is also collecting real-time metrics from the KV-Cache utilization and the queuing that is happening at the model server level.”

Cost control remains inseparable from performance. Accelerators are expensive, and AI inference workloads rarely justify running them at full capacity around the clock. GKE features such as custom compute classes and the Dynamic Workload Scheduler allow organizations to prioritize capacity when it matters most, while balancing fairness and efficiency across workloads, according to Villalba.

“When I’m serving up something, I’m hitting an end user, and I need to make their experience happy,” Villalba said. “I need to make sure that the resources needed are available at all times.”

The developer experience dividend

As AI moves from experimentation into production, developer experience increasingly determines whether initiatives stumble or scale. Teams don’t slow down because they lack ideas, but because complexity compounds faster than they can manage it. Platform engineering provides developers with reliable ways to build and deploy while maintaining DevOps principles, according to Nick Eberts, product manager, Google Cloud at Google, and Ameenah Burhan, solutions architect for Google Cloud.

“I think that platform engineering is born out of a necessity,” Eberts said. “Why did DevOps pop off? Cloud, APIs. You could move fast. You had small businesses and big companies doing shadow IT really fast … It turns out moving fast actually delivers business value.”

One of the most tangible benefits of platform engineering is reduced cognitive load. By abstracting away infrastructure decisions, internal platforms give developers a more predictable starting point, allowing them to focus on business logic, Eberts noted. Research from theCUBE’s Paul Nashawaty confirms a measurable payoff: Ninety-two percent of developers say they need modern tools and platforms to innovate, and teams with high satisfaction and autonomy deploy 23% more frequently.

“Computers are tough. Kubernetes is tough,” Eberts said. “Your software engineer can maybe pick … a three-tier app … push the button and then get a scaffold of a golden path that represents that and gets started quicker. They can just work on writing business logic.”

Burhan also stressed that platforms only succeed when they’re treated as products, not side projects. That means clear ownership, roadmaps and feedback loops that align technical decisions with business goals. Even in organizations without large, dedicated platform teams, platform engineering enables site reliability engineers, security specialists and developers to work together.

“You probably have someone like that in your organization and just kind of collaborating with people,” Burhan said. “Lean on your [site reliability engineer] specialist or your security specialist to make sure that you’re making all of those best practices and golden paths for people.”

Bobby Allen, cloud therapist at Google, frames the shift in the developer experience as an analogy. Most organizations don’t want to invent AI capabilities; they want to apply intelligence to the products and processes they already have. Platform engineering lowers the barrier to participation without stripping away flexibility.

“AI’s not the dish. It’s the sauce that makes the dish better,” Allen told theCUBE. “Most people aren’t trying to create the sauce. They’re trying to use it in ways that make what they already serve better for the people at the table.”

Outcomes that scale

The real test of production AI comes when systems face uneven demand, real customers and financial pressure all at once. AI inference workloads amplify these stresses, forcing infrastructure to operate under conditions that are both predictable in timing and wildly unpredictable in intensity. Enterprises that succeed tend to design for variability first, treating scale and cost control as inseparable outcomes.

For Shopify Inc., that reality shows up most visibly during major retail moments, when traffic spikes collide with inference-driven services such as real-time personalization. Supporting those peaks requires infrastructure that can stretch quickly without becoming brittle — and partnerships that enable engineering teams to plan and react together, according to Farhan Thawar, vice president and head of engineering at Shopify, and Drew Bradstock, senior product director for Kubernetes & Serverless at Google.

“We work [with Google] to kind of handle these crazy sales that happen, especially like a Black Friday/Cyber Monday, which is for us all [is a] year planning event to make sure it all goes smoothly,” Thawar said.

That pressure has shaped how Shopify and Google Cloud collaborate at technical and operational levels. AI inference traffic differs from traditional workloads, requiring systems that can tolerate variability in latency and complexity without degrading user experience.

“You can use your same skills, but completely different because inference doesn’t behave the same way,” Bradstock said. “It really comes down to where it’s not predictable. It can be random in terms of the needs, the latency, [and] the more complex the answers that the merchant tasks want, it’s going to be really a variable.”

Flexibility at scale also shapes how organizations think about value creation. AI workloads don’t look the same across teams. Adaptability is increasingly central to how enterprises extract value from AI investments, according to Peterson.

“AI isn’t one size fits all,” Peterson said. “Google understands that and offers blueprints for best practices across use cases. They also let companies scale up and down as needed, optimizing both their spend and value-creation.”

The agentic horizon

As enterprises look beyond single-model deployments, attention is turning to agentic AI. While the terminology is relatively recent, the underlying patterns are familiar. Agentic systems still rely on AI inference at scale, but are distributed across more components and interactions.

At the architectural level, agentic workloads place a premium on elastic scaling and the ability to spin work up and down quickly. These characteristics map closely to serverless execution models, where compute is allocated only when needed and released just as fast. Cloud Run’s design makes it a natural fit for agent-driven workloads, according to Belinda Runkle, senior director of engineering, serverless, Google Cloud at Google, and Lisa Shen, product manager at Google Cloud.

“Where you have multiple agents, typically what’s happening is something needs to be spun off,” Runkle said. “I’m going to assign a task, a thinking task, a context task … and each of those is going to basically spin up a little unit of work that needs to get done. That ability to do fast fan out, fast scale up, scale to zero when it’s done … those things are already built into Cloud Run.”

From a systems perspective, agentic AI doesn’t replace existing AI inference pipelines so much as multiply them. Large language models still serve as the reasoning layer, while tools and APIs extend their reach into business systems. The complexity comes from coordination, not cognition.

“Large language models are the brain of an agent, and then the tools are really like the agent’s hands reaching out to the digital world,” Shen said. “You have agent plus tools, and then the large language models plus tools, then an AI agent to help you accomplish a specific task.”

Recent platform updates reflect that focus on practical integration. Managed MCP servers are designed to simplify how agents interact with cloud services by replacing bespoke integrations with standardized, agent-friendly interfaces. At the same time, Workspace Studio brings agent creation closer to everyday business users, lowering the barrier to experimentation while keeping execution within managed environments.

“Google’s recently announced Agent Sandbox is another example of de-risking AI initiatives, giving companies a safe place to test agents before integrating them across the enterprise,” Peterson noted.

(* Disclosure: TheCUBE is a paid media partner for the “Google Cloud: Passport to Containers” series. Neither Google Cloud, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or News.)

Image:  News/ChatGPT