Which network does a GPU cluster need?
Both InfiniBand and RoCE give GPUs in different servers fast, low-latency access to each other's memory, which large training runs depend on. InfiniBand is a dedicated network built for this. RoCE (RDMA over Converged Ethernet) brings the same kind of direct memory access to Ethernet. Either can run a serious cluster; the choice usually follows the hardware on offer, the size of the job and the team that will operate it.
Why does the network matter so much?
When a model is trained across many servers, the GPUs exchange results constantly. If the network is slow, they wait instead of computing, and adding more GPUs adds less and less speed. Inference on a single server barely uses this network, so a cluster for serving models can often accept a simpler setup than one for training.
When is InfiniBand the safer choice?
InfiniBand has long been the default for large training clusters. It is designed for high bandwidth and low latency from the start, and many reference designs for GPU clusters assume it. The trade-off is a separate network with its own switches and skills to run it.
When does RoCE make sense?
RoCE runs over Ethernet, which most operators already know and run, commonly at 100 or 400 Gb/s per link. It needs careful configuration so the network does not drop or delay traffic under load, but it lets a cluster share tools and spare parts with the rest of the data hall.
What should you ask a provider?
- Which interconnect, at what speed per GPU or per server?
- Is the network non-blocking across the GPUs you rent, or shared and oversubscribed?
- Is storage on a separate network, and at what speed?
- Bare metal, or virtual machines on shared hardware?
A GPU request on AICapacity asks for the interconnect (InfiniBand, or RoCE at 400 or 100 Gb/s) and the storage network as standard fields. Our article on reserved and on-demand GPU contracts covers the commercial side.