OCI GPU Shapes: H100, A100, and When Each Fits

Choosing a GPU shape is the moment an AI project stops being a slide and starts being a bill. On OCI the choice is unusually consequential in both directions: the platform's bare metal design means you can get more from the same silicon than on virtualized rivals, and its pricing means the spread between the right shape and the wrong one is real money every hour. The good news is that the decision is more structured than the catalogue makes it look. Almost every workload sorts cleanly into one of a few patterns, and each pattern has an obvious home.

This article is part of our complete guide to AI on OCI. The comparative price question, what these shapes cost against their AWS and Azure equivalents, is handled separately in OCI GPU pricing vs AWS and Azure; here we focus on which shape fits which job.

The catalogue, from the top

OCI organizes GPU capacity into two families. Bare metal shapes hand you the entire physical node, eight accelerators, the full CPU complement, local NVMe, and the cluster network interface, with no hypervisor in the path. VM shapes carve smaller slices, typically one to four GPUs, for workloads that do not need a whole node. The flagship tiers are the NVIDIA H100 and A100 bare metal nodes, with newer generations arriving above them and older or smaller accelerators, such as the A10, serving the inference and graphics tier below.

Memory is the real constraint

The first filter is not speed, it is whether the model fits. GPU memory determines which models you can serve at all and at what batch size, and it is the reason the 80 GB H100 and A100 variants dominate language model work. A 7 billion parameter model in half precision wants roughly 14 GB before you add context and batching overhead; a 70 billion parameter model wants ten times that and spills across multiple accelerators. Inference servers, quantization, and tensor parallelism all bend these numbers, but the planning rule survives: count the memory first, then the GPUs, then the nodes. Jobs that fit in 24 GB belong on the A10 tier at a fraction of the cost. Jobs that need an aggregate memory pool belong on an 8 GPU bare metal node where NVLink makes the pool usable.

H100 versus A100: when the premium pays

The H100 is dramatically faster than the A100 for transformer workloads, particularly with the optimizations modern inference and training stacks apply automatically. But it also costs proportionally more per hour and is the shape everyone else wants too, which makes capacity scarcer, a constraint with its own playbook in GPU capacity planning on OCI. The honest decision rule is throughput per dollar on your actual job. Training runs and latency sensitive, high volume inference usually justify the H100 premium because time is the cost that dominates. Batch inference, embedding generation, experimentation, and most fine tuning work are frequently cheaper per unit of output on A100 nodes, which have become the value tier of serious AI infrastructure. Benchmarking both for a day costs less than running the wrong one for a quarter.

Bare metal changes the tuning conversation

OCI's distinctive offer at this layer is the absence of a hypervisor. On a bare metal node, the topology you tune is the topology you get: NVLink paths, NUMA boundaries, local NVMe for checkpoints and dataset cache, and the network interface your distributed job will saturate. Performance work transfers directly from NVIDIA's reference guidance, and the noisy neighbor variable simply does not exist. For single node workloads this is a convenience. For multi node training, where stragglers set the pace of the whole cluster, it is a structural advantage, and it is one of the reasons the supercluster tier described in building AI training clusters on OCI performs predictably at sizes where predictability is the entire game.

The supporting cast: CPU, NVMe, and the network interface

GPU shapes are sold by their accelerators but frequently bottlenecked by everything else on the node, and the spec sheet beyond the GPU column deserves a careful read. The CPU complement and system memory determine how fast your data pipeline can tokenize, decompress, and batch, and a data loader that cannot keep eight H100s fed turns the world's best training node into a very expensive space heater. Local NVMe matters twice: as dataset cache, where it removes object storage latency from the inner loop, and as the checkpoint landing zone, where its write bandwidth sets how much training time each checkpoint costs. The cluster network interface only matters when jobs span nodes, but then it matters totally. The practical habit is to benchmark the pipeline, not just the model: measure GPU utilization during a real run, and if it sits meaningfully below 90 percent, the money is leaking somewhere in the supporting cast rather than the silicon.

Sizing inference, the workload most people actually have

Training gets the headlines, but most enterprise GPU spend is inference, and inference sizing has its own logic. The variables are model memory footprint at serving precision, target latency, and concurrent request volume, and the modern serving stacks turn those into a shape recommendation fairly mechanically. The patterns that recur: quantized small and midsize models serve comfortably on the A10 tier, which is the cheapest viable home for most internal applications. Models in the 13 to 70 billion parameter class want one to several 80 GB accelerators, with batching policy deciding whether throughput or latency wins. And above that, multi GPU serving with tensor parallelism makes the 8 GPU bare metal node the natural unit. The cost discipline that matters most in inference is consolidation: many small underused endpoints cost far more than fewer shared ones, because every idle replica bills full rate. Measuring cost per thousand requests, rather than cost per hour, keeps the conversation anchored to the number the business actually buys.

A shape selection framework

The patterns we see in practice

A few recurring configurations cover most enterprise demand. A single A10 VM serves the data science team's development loop and the company's first production inference endpoint. A single A100 or H100 bare metal node carries fine tuning and serious inference for a midsize estate, and is the most common right answer in our engagements. A small cluster of H100 nodes appears when an organization commits to training or to inference volumes with real concurrency. And the supercluster tier remains the province of model builders, though enterprises increasingly rent slices of it for seasonal training campaigns. The mistake we correct most often is the aspirational H100 node running notebook experiments at single digit utilization, which is why every GPU estate we manage gets the same utilization scrutiny as the rest of the infrastructure through our OCI compute practice, and why GPU spend reviews fit naturally into our Optimization model, where the fee is a percentage of verified savings.

The shape catalogue will keep moving as new generations land. The method does not: count memory, benchmark the real job, buy the cheapest silicon that meets the number, and keep it busy.