AI agents are moving into production, making every generated token a direct business cost. This shift is forcing infrastructure providers to prioritize efficiency and the economics of running agentic systems at scale.
"When your product is an AI agent, every token you generate has a cost and business impact," said Chen Goldberg, EVP of product and engineering at CoreWeave. She highlighted Nvidia's Vera Rubin platform, which delivers 10 times better inference throughput per watt and one-tenth the cost per million tokens versus Blackwell.
The discussion, from the 'Scaling the Agentic Era' event, identified three key infrastructure insights:
First, agentic AI makes infrastructure an end-to-end systems engineering challenge. Performance now depends on the reliable flow of data, compute, and inference feedback through tightly coupled systems.
Second, continuous AI operations are becoming the standard model. Infrastructure must support coordinated environments for agents that plan, use skills, and solve problems, not just host single model requests.
Third, the rack is now the fundamental system unit. Production readiness requires validating the entire rack-integrating power, cooling, networking, and software-before deployment. "The rack is the new system now," noted Dell Technologies CTO Ihab Tarazi.
CoreWeave's rack-scale management tools, like its "Racky" manager, provide the centralized control needed to scale to hundreds of thousands of next-generation GPUs.