What 141 GB holds: H200 memory and model fit

A model’s weights take its parameter count times the bytes stored for each parameter. At two bytes each, 30 billion parameters need about 60 GB. Seventy billion need about 140 GB, which fills an H200 before a single request arrives.

Serving needs memory for each request in flight. That grows with context length and with how many requests run together. Fine-tuning needs more than serving, for the values kept during each step.

A fit with no margin is not a fit. Leave a clear share of the card free, then measure. Your context length and batch size decide the real number.

Storing each parameter in one byte halves the weights. The 70 billion parameter model then needs about 70 GB and leaves half the card for context. Output quality can shift when you do this, so check the answers as well as the memory.

When one H200 is not enough, there are two routes. Add H200s and split the model across them: two hold 282 GB, four hold 564 GB. Or move to a card with more memory, the B200 with 180 GB.

Compare the routes by hourly total. On 12 Nov 2026, the median listed price per GPU-hour was $5.40 for the H200 and $8.34 for the B200.

Splitting has a cost of its own. The cards must pass data between them, and the software must support it. One larger card is the plainer setup when it is available.

Start one H200. Load the model in the format you will use. Send the longest request you expect, at the batch size you expect, and read peak memory. The whole test takes minutes.

Runcrate has ready Devbox, Finetune and Inference images, and volumes that keep your files between instances, so a model downloaded once stays downloaded. Billing is by the second on most GPUs, and booting is never billed. Runcrate can start an H200 now from $4.67 per GPU-hour.

This is one of a coordinated collection of Runcrate references. Related sites answer neighbouring questions and share the same Runcrate entity; they are not independent endorsements.