Proximity is Everything
By Jean S. Bozman, Cloud Architects LLC
Proximity is everything in AI infrastructure.
It holds true for the components within a server system or storage system – and it holds true within a data-center rack, housing dozens of GPUs in a single, scalable server cabinet.
This “proximity” of infrastructure goes beyond housing GPUs in one or more data-center racks – as layers of circuits within GPUs or CPUs are brought closer together by using extremely “thin” materials and higher levels of on-chip integration.
AI acceleration for real-world business results is quickly becoming a reality. Entire racks of clustered GPUs and CPUs are being woven together into highly dense – and highly diverse – infrastructure for AI enterprise, AI cloud and HPC + AI applications, supporting high performance with security and reliability.
The new proximity of compute, storage, networking and interconnects brings customers the benefits of accelerated AI processing with real-world business results much faster than before.
This exciting new infrastructure brings with it the challenges of providing enough power and cooling for these dense-packed racks – no matter where they reside geographically. In fact, large organizations are tapping data-center and cloud resources around the world – connecting their on-prem data-center resources in real-time via public and private (hybrid) clouds and hyperscalers’ worldwide clouds.
Keys to the Kingdom
AI acceleration for real-world business results is quickly becoming a reality, as customers and tech companies confirmed.
The “how” of designing, building and deploying dense, diverse and efficient infrastructure was the focus of the AI Infrastructure Summit conference in Santa Clara, California, (September 9 – September 11, 2025).
“Building AI infrastructure is a challenging and brave endeavor,” said Ian Buck, Vice President of Hyperscale and HPC at NVIDIA, in his keynote talk. “Building infrastructure that works – not only a single rack but an entire data center – with that much compute to keep all those researchers active and busy and humming along, is a challenge.”
Density and Diversity
Density and diversity are used inside data-center racks to bring multiple “layers” of compute, storage and networking components closer together. Faster is better – although it brings the challenge of reducing power/cooling requirements for the dense infrastructure.
The new proximity of compute, storage, networking and interconnects brings customers the benefits of accelerated AI processing with real-world business results much faster than before. But this exciting new infrastructure brings with it the challenges of providing enough power and cooling for these dense-packed racks – no matter where they reside.
Processor diversity is also become widely accepted in the AI infrastructure data center. To power AI, a variety of processors – GPUs, CPUs, TPUs (tensor processing units) and other specialized processors – are housed within the same data-center racks and enclosures.
Years ago, this would have made infrastructure look “lumpy”, requiring special interconnects and switches. Today’s AI infrastructure is built to support this diversity as a fact-of-life for enterprise datacenters and clouds.
Integration expertise is becoming increasingly important to customers who are installing AI infrastructure on-premises. It will shorten their learning curve, and it will give them the opportunity of leveraging techniques and tools that will let them scale up beyond 8 GPUs per server, beyond limited petabytes of storage, and beyond a formerly limited knowledge base to tightly cluster servers into far more scalable infrastructure designs that would not have been available just a few years ago.
The emerging brave-new-world forms of AI infrastructure must support a wide range of workloads, lower power/cooling and OPEX costs is gaining much higher visibility in IT organizations and CSPs.
Thin is “In”
It all comes down to physics.
Reducing the distance between compute elements – and between multiple GPUs within your data-center racks are both key to new-wave AI design.
Integration is increasingly “built in” to new systems, because it is needed to reduce the number of “hops” that would have been made between electrical components inside the individual computer frames – and between them.
Examples include:
- Dense forms of memory (3D DRAM and HBM high-bandwidth memory)
- Very large semiconductor wafers (GPUs) for on-chip density, avoiding “hops” from chip to chip when completing end-to-end processing tasks. Some of the largest tasks are waiting for memory, forcing a re-think of processor placement on AI system “real estate.”
- Highly integrated “vertical” stacking of processors, storage and networking elements of overall, end-to-end infrastructure.
- Faster networking interconnects – including optical interconnects – to reduce the “distance” between compute elements in state-of-the-art AI infrastructure deployments.
Highly Integrated Designs
Surely, we are already on the path to these highly integrated designs, helped by the highly dense data-center systems used by hyperscalers (e.g., AWS, Google Cloud and Microsoft Azure) with the biggest worldwide infrastructure “footprints” and a wide array of CSPs (cloud service providers) within large geographic regions and worldwide.
Mark Lohmeyer, director of Google’s AI Computing Infrastructure, told the conference: “We’re just getting started” on the next waves of infrastructure optimization, because “we need to deliver that kind of customer experience.”
He cited power and cooling as among the challenges to scaling rapidly with great performance, pointing out that “physical buildout of infrastructure is beginning to be limited by power efficiency.” That is true worldwide, impacting development of AI systems and driving up the cost of AI innovation.
However, Google Cloud is focused on reducing power/cooling “across the whole stack” for both scale-up AI models and AI inference purposes., Lohmeyer said. One of the best ways to reduce power/cooling demand is to “co-design across the entire stack.”
Having a relentless focus on efficiency and a simplified design process will “result in a shorter time to innovate” on future iterations of infrastructure, Lohmeyer said. He cited Google’s Deep Mind group and its work with other Google units, vendors, partners and customers as an example of the co-design process. He cited Toyota as an example of a Google Cloud customer using ease-of-use and speed of scaling.
Customers
AI infrastructure accelerates applications – and speeds delivery of application results. Large customers who are already using faster, highly integrated systems include cloud providers (e.g., AWS, Google, Microsoft Azure, Oracle OCI); customers (e.g., Bank of America, CVS Health, Pfizer). Speakers highlighted the challenges of providing enough power/cooling for operating dense infrastructure at extremely high speeds.
On a human level, AI adoption is transforming the business – and changing the way that organizations work. AI infrastructure accelerates applications – and speeds delivery of application results.
In one special session, two worldwide companies – Pfizer and CVS Health – said AI infrastructure is leading them to think differently about the ways they will ensure scalability, security, reliability of mission-critical operations on a worldwide IT landscape. One speaker said: “The new infrastructure designs are “forcing me to think about what will the next-gen datacenter look like?”
And, on a human level, companies and organizations will need to provide education and training to help their workforce leverage AI to gain business benefits. “We’re redefining what work looks like,” said Berta Rodriguez-Hervas, chief AI and analytics officer of Pfizer Inc. “You’re empowering people. . . and there’s a need for educating and providing the tools and data” that will allow a company’s workforce to leverage “AI to connect the dots.”
Gaining business benefits from AI acceleration and data analysis will likely lead IT professionals to consider hiring staffers from new sources. “We need to take a multi-prong approach to attracting talent,” said Alan Rosa, CISO and senior vice president of technology at CVS Health. “New hires may include staffers from non-traditional IT backgrounds, and in-house training programs will likely benefit from partnerships with cloud providers and AI specialists.
Key Takeaways
AI infrastructure is evolving, including both scaleable infrastructure for expansive data models in data centers and the Cloud – and a new wave of scale-out infrastructure for a constellation of scale-out workloads on the Edge, close to factories and offices across the network. This variety supports both model-training for scalable AI workloads, and inference training for scale-out AI workloads.
The AI infrastructure revolution underway worldwide is impacting system designs and IT deployments in all regions of the world. Processors, memory, storage and networking are all being affected in data-center and cloud installations. Integration services will be key to adopting new AI infrastructure quickly and with fewer growing pains associated with power/cooling and operational costs.
Faster networking is a key enabler of this next generation of AI infrastructure. Faster networking interconnects are providing more links between densely packed components and better access to memory for faster data transfers. In the process, the physical layers of networking are changing, too, with copper-based circuits replaced by fiber-optics and cooler, faster interconnects.
Presentations at the AI Infrastructure 2025 conference, many of which appear on YouTube, are confirming, in talk after talk, that the changes are happening quickly – and that they are being adopted widely. If the pace of change is accelerating too quickly for some, partnerships and services can help customers make the changes they’ll need to put AI “to work” for their organizations
