First choose what the two cards share
In classic data parallelism, each GPU keeps a copy of the model and receives part of the batch. For a model too large for a single card, parameters or layers must be distributed. The combined 360 GB therefore does not automatically constitute a single allocation. Write down your hypothesis before renting: increasing the global batch, reducing the time per step, or making a run possible.
A distribution test that produces a conclusion
Start with the same reduced dataset on one card, if the model fits, then on two. Record the memory peak of each GPU, the time per step after warm-up, and the cost of exchanges. One card nearly empty and the other saturated suggests an unbalanced distribution. Two independent experiments can be more useful than distributed training when your question concerns several hyperparameters.
Keep precision and the number of examples processed constant during this comparison. An improvement obtained by changing both the numeric format and the batch size would be difficult to attribute to the hardware.
Prepare the Blackwell stack and choose an alternative
Verify that your PyTorch version, your CUDA distribution and the compiled extensions correctly target Blackwell. A custom kernel compatible with another generation is not enough to establish that compatibility. Test the essential operations before the full campaign. If your model and its execution reserve fit within 141 GB, the H200 lets you study execution on a single GPU; the MI300X opens another path for a project prepared for ROCm.
Book a usable research window
Three days suit a pre-planned distribution validation. Seven days leave room to organize several variants and their evaluation. Thirty days correspond to a tracked campaign, with checkpoints and defined stopping criteria. Choose the number of two-card lots, the duration and the desired environment, then enter your contact details. After your crypto transfer, use "I've paid" in the order; then follow the payment and access preparation steps.