For about sixty years the unit of computing was the chip. Then for about twenty it was the server. As of the current generation it is the rack, and that is not a packaging convention. It is an architectural claim.
NVIDIA describes the GB200 NVL72 as seventy-two GPUs that behave as one accelerator, and the description is literally accurate rather than promotional. Seventy-two Blackwell GPUs and thirty-six Grace CPUs sit inside a single NVLink domain that the vendor describes as acting as one massive GPU, presenting 13.5 terabytes of unified HBM3e memory addressable as one pool, connected by a switch fabric delivering 130 terabytes per second of GPU-to-GPU bandwidth. A model does not run across seventy-two chips in that rack. It runs on one very large chip that happens to be assembled from seventy-two pieces and weighs about three thousand kilograms.
AI rack architecture is the study of why that assembly became necessary, and the answer runs through a constraint most coverage of this industry gets backwards. The binding limit is not how fast the silicon can calculate. It is how fast data can be moved to it, and every design decision in a modern rack is a response to that.
The claim is worth testing against the alternative reading, which is that this is a marketing construct and a rack is a rack. It is not, and the test is behavioural: a model too large for a single accelerator’s memory can be split across an NVL72 with a modest performance cost and across a networked cluster of equivalent chips with a severe one. Same silicon, same total memory, different result, and the difference is entirely in how the pieces are connected.
The memory wall, and why FLOPS is the wrong number
Every vendor datasheet leads with compute throughput, and for the dominant workload in 2026 it is close to irrelevant.
Language model inference during the decode phase is memory-bound rather than compute-bound. Generating each token requires reading the model weights out of memory, and the arithmetic performed on those weights is trivial relative to the cost of fetching them. Tokens per second therefore tracks memory bandwidth far more closely than it tracks floating-point capability, which means a chip with more bandwidth can outperform a chip with more compute on the workload that actually pays the bills.
The gap that produces this has been widening for decades. Processor throughput improved far faster than memory bandwidth across the whole history of computing, and the divergence is what engineers call the memory wall. Accelerators arrived at a point where they can calculate faster than anything can feed them.
The framework engineers use to determine which regime a workload sits in is the roofline model, which plots achievable performance against arithmetic intensity, meaning operations performed per byte fetched. Below a threshold the workload is bandwidth-limited and additional compute is idle. Above it the workload is compute-limited and additional bandwidth is idle. Training sits closer to the compute-limited side because batch sizes amortize weight fetches across many examples. Decode-phase inference sits firmly on the bandwidth-limited side because each token requires a full pass over the weights for a single sequence.
That distinction has a commercial consequence that the vendor comparisons obscure. As the industry shifts from training toward inference, which forecasts have becoming the primary driver of AI server demand toward the end of the decade, the metric that determines competitive position shifts with it, and a chip selected on compute benchmarks may be the wrong chip for the workload it ends up running.
High bandwidth memory is the response, and the mechanism is geometric rather than clever. HBM stacks DRAM dies vertically and connects them through the silicon with through-silicon vias, placing the memory immediately adjacent to the processor die on the same package. That produces a 1,024-bit interface per stack against sixty-four bits for a conventional DDR5 channel, sixteen times wider, with a much shorter path and correspondingly lower latency.
A B200 GPU carries 192 gigabytes of HBM3e at eight terabytes per second. The B300 in the GB300 generation carries 288, raising rack-level memory from 13.5 to 20.7 terabytes. HBM4, entering mass production in 2026, doubles the interface to 2,048 bits and targets around two terabytes per second per stack while maintaining transfer rates above eight gigabits per second.
That is the actual specification race, and it is being run on memory rather than on transistors.
HBM, and the three companies that gate the industry
The consequence of memory being the constraint is that the memory suppliers became the chokepoint, and the market structure is uncomfortable.
Three companies produce effectively all HBM: SK Hynix, Samsung, and Micron. SK Hynix holds roughly sixty-two percent share with NVIDIA accounting for something like ninety percent of its HBM output. HBM4 allocation is running roughly sixty to seventy percent SK Hynix, twenty-five to thirty percent Samsung, with Micron as the supplementary third source.
All three reported full capacity allocation through 2026. Micron confirmed its entire year’s HBM production sold out under binding volume and price agreements struck in December 2025, with orders locked more than twelve months ahead of delivery. That is not tightness. That is an industry operating on allocation.
The structural reason capacity cannot simply expand is that HBM shares fabrication lines with conventional DRAM. Diverting capacity to HBM tightens standard DRAM, which is why memory prices across the board have moved and why NVIDIA reportedly cut gaming GPU production substantially in the first half of 2026 on GDDR7 constraints. The AI buildout is consuming the memory industry’s output and the consumer market is absorbing the shortfall.
Demand growth compounds it. HBM consumption grew more than a hundred and thirty percent year over year based on 2025 shipments, with 2026 growth still projected above seventy percent as B300, GB300, and Rubin platforms ramp alongside Google TPU and AWS Trainium transitioning to HBM3e.
Which places a three-firm oligopoly, concentrated in a specific geography, at the base of the entire buildout. The same concentration pattern that governs critical materials applies with the same consequences, and the advanced packaging capacity that assembles HBM onto the processor die was itself the binding constraint on GPU supply until recently.
The yield problem underneath HBM explains why capacity does not respond quickly. Stacking twelve or sixteen DRAM dies vertically and connecting them with through-silicon vias means a defect anywhere in the stack can fail the whole assembly, so yields on the newest generations start low and improve slowly with process maturity. Reports of base-die issues on early HBM4 production, subsequently resolved before qualification, are the normal shape of that curve. A supplier cannot simply run more wafers to fix a yield problem, and the specialty materials and process gases involved have their own constrained supply.
Scale-up, scale-out, and why the distinction matters
The rack exists because two different kinds of communication have wildly different costs, and modern models require the expensive kind.
Scale-up means connecting accelerators tightly enough that they behave as one device, with shared memory addressing and very high bandwidth. Scale-out means connecting nodes over a network, which is far cheaper per unit and far slower.
The NVL72 is a scale-up domain. Each GPU has eighteen NVLink connections distributed across nine dedicated switch boards, delivering 1,800 gigabytes per second of bidirectional bandwidth per GPU into a non-blocking topology. Grace CPUs connect to their paired GPUs through NVLink chip-to-chip at 900 gigabytes per second, which allows unified memory addressing so a GPU can reach CPU memory as if it were local rather than traversing PCIe.
Scale-out uses conventional networking, with InfiniBand favoured for training clusters and Ethernet variants for multi-tenant environments, and both operate at a small fraction of NVLink bandwidth. The Ethernet camp has been closing the gap with AI-specific variants adding congestion control and lossless behaviour, which matters commercially because Ethernet has a vastly larger supplier base and the concentration risk in any single-vendor interconnect is exactly what large buyers try to avoid.
The reason the distinction is architectural rather than incremental shows up in specific model behaviours. Tensor parallelism splits a single layer’s computation across devices and requires all-to-all communication at every step, which is catastrophic across a network and tolerable across NVLink. Mixture-of-experts models route tokens to different experts, producing exactly the all-to-all pattern that creates communication hotspots on a networked cluster and resolves cleanly on a full-mesh fabric.
So the seventy-two-GPU domain is not a convenience. It is the boundary inside which a model can be split without paying a network penalty, and a model that exceeds a single accelerator’s memory has to be split somewhere.
There is a third tier now being discussed as scale-across, meaning communication between data centers rather than between racks, which arises when a training run exceeds what one facility can host. At that distance the bandwidth and latency penalties are severe enough to constrain what parallelism strategies work at all, and it is the reason multi-site training is a networking problem before it is a compute problem, with the fibre routes and latency budgets between facilities becoming a siting criterion in their own right.
What AI rack architecture actually contains
Enumerating the contents makes the engineering legible and explains where the money goes.
Compute trays hold two GB200 Grace Blackwell Superchips each, with each superchip containing two Blackwell GPUs and one Grace CPU. Each Blackwell GPU is itself two dies joined by a ten-terabyte-per-second chip-to-chip link, because a single die at that transistor count exceeds the reticle limit of the lithography process, which is a hard physical boundary set by the optics of the scanner rather than a design choice. Chiplet construction is therefore not an optimization; it is what happens once the ambition exceeds what one exposure can print, and it is why advanced packaging became the industry chokepoint rather than transistor fabrication. The interposer that carries the signals between chiplets and HBM stacks is itself a manufactured silicon component with its own capacity limits, which is one more layer of the supply chain that has to expand before anything else can. The GPU carries 208 billion transistors on a custom TSMC four-nanometer-class process. Each Grace CPU runs seventy-two Arm cores with up to 480 gigabytes of LPDDR5X, which functions less as a host processor than as a high-speed memory extension addressable by the GPUs, solving capacity rather than compute for models whose parameters exceed even the pooled HBM.
Switch trays hold the NVLink fabric, with industry estimates suggesting eight NVSwitch chips per board across nine boards, each operating at 14.4 terabytes per second aggregate.
Then the unglamorous majority: power distribution, busbars, the coolant distribution unit, manifolds, cold plates on every processor, networking interfaces for scale-out, and local storage. The magnets, specialty alloys, and minor metals distributed through the power and cooling equipment are a supply story of their own, and the copper alone across a large deployment is a commodity exposure most buyers never price.
The physical numbers constrain everything downstream. Roughly three thousand kilograms, one hundred twenty to one hundred forty kilowatts, mandatory liquid cooling, and a system price around three million dollars before networking and storage. That mass and that power density are why a data hall built to previous-generation assumptions cannot host one regardless of floor space.
The printed circuit board problem
One constraint deserves isolating because it is invisible in every specification sheet and is genuinely at the edge of manufacturability.
Moving 1,800 gigabytes per second per GPU across a rack requires signal integrity that pushes printed circuit board engineering to its commercial limits: extremely high layer counts, exotic low-loss dielectric materials, impedance control at tolerances that reject most of a production run, and connector systems that maintain signal quality across mechanical mating cycles.
That matters commercially because it narrows the supplier base. A component that only a few manufacturers can produce at yield becomes a scarcity, and the substrate and packaging materials involved have their own concentrated supply chains. A rack architecture that pushes bandwidth this hard is therefore not merely expensive because the silicon is expensive. It is expensive because several of the boring components are near the limit of what anybody can make. Cabling and connectors are the same story, since a rack carrying this much interconnect contains kilometres of copper in configurations that have to be assembled by hand and tested individually. That labour is skilled, the pool is small, and it is the same specialist workforce shortage that gates the buildings themselves.
Power delivery, which is becoming its own architecture
Getting one hundred forty kilowatts into a rack and distributing it to processors drawing over a kilowatt each is a problem conventional data center power design does not solve.
Traditional racks distribute alternating current to power supplies in each server. At current densities the conversion losses and the copper required for the currents involved become prohibitive, which is why the industry moved to higher-voltage direct current distribution within the rack and is now developing eight-hundred-volt DC architectures aimed at megawatt-class racks toward 2027.
Higher voltage means lower current for the same power, which means less copper, less resistive loss, and less heat generated by the distribution itself. It also means new safety practices, new component qualification, and a supply base that has to be built. Solid-state transformers, DC protection devices, and the busbar systems to carry those currents are components with small existing markets and long qualification cycles, which puts them in the same lead-time category as everything else electrical.
Power quality is the constraint nobody outside the industry discusses. An AI training run produces synchronized load steps as thousands of accelerators start and stop computation together, which creates transients the local grid has to absorb and which look nothing like the smooth baseload profile a data center historically presented. Operators now deploy energy storage inside the facility partly to smooth those steps, which means battery systems are becoming standard equipment for reasons having nothing to do with backup power.
Per-processor power tells the same story from the other end. An H100 runs around seven hundred watts. A B200 runs one thousand to twelve hundred. The B300 generation reaches roughly 1.4 kilowatts per GPU. Each increment tightens the thermal and electrical design simultaneously, and the electrical equipment required to deliver it is on multi-year lead times.
Numerical precision, which is the quiet efficiency story
The largest performance gains in the current generation came from arithmetic rather than from transistors, and this is underappreciated.
Blackwell added hardware support for four-bit floating point. Previous generations accelerated eight-bit. Halving the bits per value halves the memory required to store weights, halves the bandwidth required to move them, and roughly doubles the throughput of the arithmetic units.
Given that inference is memory-bound, a format that halves memory traffic is worth more than a proportional increase in compute would be. Much of the headline generational improvement is a precision result, and it only materializes for workloads quantized to the format the hardware accelerates. A team not quantizing captures a fraction of the advertised gain, which is a detail that rarely survives into procurement conversations. Quantization also costs accuracy on some workloads, which means the headline efficiency figure carries a quality assumption that has to be validated per model rather than accepted per datasheet.
The complementary software techniques operate on the same constraint. Paged attention manages the key-value cache more efficiently, prefix caching reuses computed cache entries across identical prompt prefixes, and continuous batching keeps the arithmetic units fed. Every one of them is a memory optimization rather than a compute optimization, which tells you where the bottleneck sits. Prefix caching in particular is close to free performance for workloads with repeated prompt structure, and it is the sort of efficiency gain that complicates any forecast built on compute demand scaling with usage.
Alternatives, and where the architecture is being challenged
The rack-scale NVLink approach is dominant and it is not the only design being funded, which matters for anyone assuming the current architecture is permanent.
Wafer-scale integration takes the opposite approach to the memory wall, putting an enormous number of cores on a single wafer with on-chip SRAM and eliminating external memory access entirely for models that fit. Reported results include throughput several times a Blackwell system on certain model sizes, achieved by removing the bandwidth constraint rather than by adding compute.
Processing-in-memory places compute elements inside the memory stack, attacking the same problem from the memory side. Major suppliers are developing variants.
Inference-specific accelerators trade generality for efficiency on the decode workload, which is a defensible bet given that the economics of serving tokens differ entirely from the economics of training, and NVIDIA’s own modular platform now accommodates third-party inference hardware in reference configurations, which is a notable concession from a company whose position rests on an integrated stack.
Custom silicon from the hyperscalers is the largest structural threat, and it is being pursued for the same reason any large buyer eventually integrates backward into a concentrated supplier. Google TPU and AWS Trainium are transitioning to HBM3e and represent internal demand that does not flow to NVIDIA, and every hyperscaler with a credible internal accelerator program has an incentive to reduce dependence on a single vendor whose gross margins are public. Those programs also compete for the same HBM allocation and the same packaging capacity, which means internal silicon relieves vendor concentration without relieving the physical constraint.
None of that displaces the current architecture in the near term. All of it is aimed at the same constraint, which is the strongest evidence that the constraint is correctly identified. When wafer-scale integration, processing-in-memory, inference-specific silicon, and custom hyperscaler accelerators are all attacking memory bandwidth from different directions, the diagnosis is not in dispute even where the treatment is.
The competitive dynamic worth watching is that AMD’s accelerators have carried a memory capacity advantage over comparable NVIDIA parts in several generations, which matters more operationally than the compute comparison implies given where the bottleneck sits. Whether that translates into share depends on software ecosystem maturity rather than on hardware, which is the moat that has held longest and which is the least physical constraint in this entire subject and therefore the most likely to erode.
The cadence, and what AI rack architecture does to a building
NVIDIA moved from a roughly two-year product cadence to an annual one, and the buildings have not.
The GB200 NVL72 was announced in March 2024, ramped through late 2024 and 2025, and is the primary frontier platform in 2026. GB300 entered production in the third quarter of 2025 with fifty percent more memory. Rubin follows in the second half of 2026, and the roadmap points at rack densities of several hundred kilowatts and eventually a megawatt.
A data hall is a thirty-year asset. Its electrical distribution, floor loading, and cooling plant are specified against a rack density assumption made years before the equipment exists. A facility designed around one hundred forty kilowatts per rack and commissioned in 2028 will be hosting hardware designed for considerably more, and retrofitting a live facility is expensive in a way that greenfield construction is not. That mismatch drives an observable behaviour: operators overbuild electrical and thermal capacity relative to the current generation, accepting stranded capital today to avoid stranded buildings later, which raises the cost per megawatt of everything being constructed and shows up in the construction cost escalation the sector has been reporting.
That mismatch is the reason the financing question about how long a GPU earns cannot be separated from the building question. The silicon has a four-to-six-year argument attached. The rack architecture that houses it changes annually. The building is committed for decades. Three clocks, no synchronization, and the fastest one setting the specification for the slowest.
The claims that do not hold up
An audit, because hardware specifications generate more misleading comparisons than almost any other technical domain.
More FLOPS means faster AI is wrong for the dominant workload. Decode-phase inference is memory-bandwidth-bound, and a compute comparison between accelerators frequently predicts the opposite of measured throughput.
The NVL72 is a rack of seventy-two GPUs is technically accurate and misses what makes it different, which is that they present as one device inside a coherent memory domain.
Bigger is always better ignores fit. A model that fits comfortably on a single accelerator gains nothing from a pooled seventy-two-GPU domain and pays for it, and the correct question is whether memory requirements exceed a single card.
GPUs are the bottleneck was true in 2023. Advanced packaging capacity expanded, and the constraints moved to HBM allocation, electrical equipment, and grid connections.
The performance gains are all from better chips understates the contribution of numerical precision and software, and the precision gains require the model to be quantized to capture them.
HBM shortages will resolve with more fabs understates the DRAM line-sharing problem, the stacking yield curve, and the multi-year cycle to add capacity, which is the same dynamic that governs every specialty materials shortage.
A rack is a rack is the assumption this whole subject exists to correct, since the difference between a networked cluster and a coherent domain of the same total capacity is a difference in what can run on it at all.
Vendor benchmark comparisons are directly comparable is rarely true, since published figures specify cluster size, precision format, latency target, and sequence lengths that differ between the compared systems.
What the rack is actually telling us
Step back from the specifications and AI rack architecture is a single argument stated in copper and silicon: the models outgrew the chips, so the chips had to be assembled into something larger that still behaves like one chip.
Every element of AI rack architecture follows from that. Unified memory addressing exists because a model needs one address space. The NVLink fabric exists because splitting a model across a network costs more than the model gains from being split. Liquid cooling exists because the density required to keep the interconnect short generates heat air cannot remove. Higher-voltage distribution exists because the currents involved otherwise waste too much copper. Four-bit arithmetic exists because halving memory traffic is worth more than doubling compute.
None of those is an independent innovation. They are consequences of a memory bandwidth constraint that has been widening for forty years and became binding at the point where model size passed accelerator memory.
Which produces the observation worth carrying into everything downstream. The scarce inputs in this industry are not the ones the coverage names. It is not transistors, which TSMC produces at extraordinary volume. It is stacked memory from three suppliers, advanced packaging capacity, printed circuit boards at the edge of manufacturability, and the electrical and thermal infrastructure to run any of it. Every one of those is a physical manufacturing constraint with a multi-year expansion cycle, sitting underneath a demand curve that changes annually.
That asymmetry is the whole shape of the industry at present. The critical minerals sector spent two decades learning the same lesson, which is that a concentrated supplier of a specialized input captures a disproportionate share of the value created downstream, and that the downstream participants generally do not notice until the supplier exercises the position.
The ten-lecture briefing on how AI data centers work runs the physics, the money, and the politics in the order the constraints actually bind, and the hardware sets the first one: the rack determines the thermal load, the thermal load determines the building, and the building determines everything anybody argues about afterward.
Seventy-two chips pretending to be one, drawing the power of a small neighbourhood, cooled by liquid because air cannot carry the heat away, waiting on memory produced by three companies. That is the unit of computing now, and AI rack architecture did not arrive at three tons because anybody wanted a three-ton rack. It became the unit because the models stopped fitting, and everything downstream in this subject is a consequence of that single fact arriving faster than the buildings, the grids, and the supply chains underneath them could respond.
Leave a Reply