The Invisible Backbone: Networking, Storage, and the Software Stack That Powers LLMs

Every blog in this series so far has been about hardware you can point at: a GPU, a server, a rack. This final post is about the parts that don't show up in a product photo — the interconnect fabric that lets thousands of GPUs behave like one machine, and the software stack that sits on top of it all. It's the densest topic in the series, so it's split cleanly into two halves: networking and storage first, then the software layer.

NVLink is a high-speed, direct GPU-to-GPU (and CPU-to-GPU) interconnect built specifically to get around the bandwidth bottleneck of PCIe. Depending on the generation, it delivers anywhere from 900GB/s up to 1.8TB/s — and critically, NVLink traffic never touches the PCIe bus at all. It's physically a separate link, just packaged in a familiar-looking slot.

NVSwitch takes this further: it's a dedicated switching chip that enables all-to-all GPU communication at full NVLink speed, within a single server. An HGX H200 8-GPU system, for example, ships with four NVSwitch devices to give every GPU a full-speed path to every other GPU.

For context: an 8-GPU DGX H100 gets 18 NVLink connections per GPU at 50 GT/s bidirectional (25 GT/s per direction) — which is where that 900GB/s per-GPU figure from Part 4 of this series actually comes from.

Once you scale beyond a single server, a different technology takes over.

InfiniBand (IB) is a networking standard purpose-built for HPC: very high throughput, very low latency, using a switch fabric topology rather than the tree-like topology of typical Ethernet. It has gone through several generational names — SDR, DDR, QDR, FDR, EDR, HDR, NDR, and now XDR — each faster than the last. As a baseline, InfiniBand starts at around 5 microsecond latency, and modern NDR-class fabrics can get down to 1–2 microseconds.

RDMA (Remote Direct Memory Access) is the technology that makes InfiniBand actually useful for GPU workloads: it lets one machine read from, or write directly into, another machine's memory — without involving the remote CPU at all, at very low latency and very high throughput. In practice, RDMA is almost never used without InfiniBand underneath it; treat the pairing as one concept.

RoCE (RDMA over Converged Ethernet) is the alternative path: it brings RDMA's benefits to standard Ethernet networks instead of requiring dedicated InfiniBand hardware. RoCE v1 operates at the Ethernet link layer; RoCE v2 operates at the internet layer, making it routable. RoCE can technically run over a standard TCP/IP stack, but that configuration gives up most of the latency benefit — expect 20–50 microsecond latency rather than InfiniBand's 1–2 microseconds.

NVIDIA's ConnectX series are the network interface cards (NICs and SmartNICs) that actually plug into servers, supporting up to 400Gb/s of Ethernet or InfiniBand connectivity, with newer generations pushing further: ConnectX-8 SuperNIC reaches 800Gb/s, and ConnectX-9 reaches 1.6Tb/s over PCIe Gen6.

BlueField DPUs (Data Processing Units) go a step further — NVIDIA actually coined the term "DPU" to describe a programmable processor that tightly integrates a general-purpose CPU with network interface hardware. Rather than just moving packets, a BlueField DPU can offload entire categories of work from the host CPU: storage protocol handling, RDMA operations, and built-in security functions like firewalling, intrusion detection, and encryption — freeing the host's CPU cores for actual compute instead of network plumbing.

NVMe (Non-Volatile Memory Express) is the high-performance storage protocol used by modern SSDs, connecting directly to the CPU over the PCIe bus. NVMe-oF (NVMe over Fabrics) extends that same protocol across a network — Ethernet, Fibre Channel, or InfiniBand — so a GPU server can read from storage that isn't physically inside it, at close to local-SSD speed. Tunneling NVMe commands through an RDMA fabric specifically is what makes this fast: high throughput, low latency, without round-tripping through a traditional network storage stack.

This is exactly the kind of workload BlueField DPUs are built to offload — handling NVMe-oF protocol translation and RDMA operations on the DPU's own ARM cores, rather than burning host CPU cycles on storage networking while your GPUs are trying to train.

NCCL (pronounced “Nickel,” the NVIDIA Collective Communication Library) is what lets you network multiple GPUs together for distributed training or real-time inference without manually tuning for specific hardware topology — it detects the topology automatically and exposes a straightforward C-based API.

The core concept is the rank: a unique identifier assigned to each process in a distributed computation. Every GPU gets its own rank, and when you perform a collective operation, every single rank has to call the same operation with matching parameters — they're synchronized by design.

If you only remember one of these, make it AllReduce — it's the operation running constantly during any multi-GPU training job, since every GPU's locally computed gradient has to be combined and synchronized before the next training step can begin.

Magnum IO is best understood as an umbrella, not a single product: it's NVIDIA's I/O subsystem for data centers, an abstraction layer that ties together most of the networking and storage technology covered above, with a consistent goal of bypassing the CPU wherever possible for better performance.

With the networking and storage layer in place, this second half covers the software that actually runs on top of it — from training a model to serving it in production.

Triton Inference Server

An open-source server purpose-built for serving models in production. Triton runs across cloud, data center, edge, and embedded deployments, on NVIDIA GPUs, x86/ARM CPUs, and even AWS Inferentia — and it can deploy models straight from TensorRT, PyTorch, ONNX, OpenVINO, plain Python, or RAPIDS FIL. It's part of the broader NVIDIA AI Enterprise suite.

TensorRT-LLM and Quantization

TensorRT is NVIDIA's inference optimization ecosystem, tuning a trained model specifically for the target hardware it'll run on. TensorRT-LLM extends this specifically for large language models: convert your model checkpoint into a TensorRT-LLM checkpoint format, build an optimized inference engine from it, then run inference on that engine.

Quantization is the technique that makes this genuinely fast: reducing a model’s numerical precision (say, from BF16 down to FP8) to cut memory footprint and computational cost. TensorRT-LLM supports a range of quantization recipes — FP4, FP8 (per-tensor, block-scaling, or row-wise), and 4-bit weight schemes like GPTQ and AWQ. The H100’s Transformer Engine, covered in Part 4 of this series, is purpose-built for exactly this kind of FP8 inference — a 70B-parameter model like Llama 70B is commonly quantized to FP8 specifically to run efficiently on it.

RAPIDS and cuDF

RAPIDS is NVIDIA's GPU-accelerated data science framework — a whole collection of libraries (cuDF, cuML, cuGraph, and others) for running entire data pipelines on the GPU instead of the CPU. cuDF (pronounced “KOO-dee-eff”) is the one most people touch directly: a Python GPU DataFrame library, built on the Apache Arrow columnar format, that mirrors pandas for loading, joining, filtering, and aggregating data — just running on the GPU.

NeMo Framework

NVIDIA's development platform for building custom generative AI models, covering language models, multimodal and computer vision models, automatic speech recognition, NLP, and text-to-speech. NeMo isn't just a training library — it covers the full lifecycle of building, monitoring, and optimizing AI agents.

ONNX and Netron

ONNX (Open Neural Network Exchange) is an open, framework-agnostic model format that most major ML frameworks can export to — a common language any framework can use to describe a model's architecture and math, independent of what trained it. A saved model in this format is usually called an "ONNX graph." Netron is the standard tool for actually looking inside one: a free visualizer (browser-based or local) that supports ONNX plus a long list of other formats — PyTorch, TensorFlow Lite, Keras, Safetensors, and more — letting you inspect a model's structure without writing any code.

NVIDIA Container Toolkit

The set of libraries and utilities that make GPU-accelerated Docker containers actually work — including the NVIDIA Container Runtime and the nvidia-ctk CLI. The modern, recommended approach is to use CDI (Container Device Interface), a standardized spec for exposing devices like GPUs to containers:

# Run with all GPUs visible, using CDI (recommended)
docker run --device nvidia.com/gpu=all -it my-training-image

# Or select a specific GPU
docker run --gpus device=0 -it my-training-image

Slurm

Slurm (Simple Linux Utility for Resource Management) is a free, open-source job scheduler used by a large share of the world's supercomputers and GPU clusters, built for traditional HPC-style batch scheduling. In practice, you queue a training job as a script and let Slurm handle allocating the requested GPUs and nodes:

#!/bin/bash
#SBATCH --job-name=llm-train
#SBATCH --gpus=8
#SBATCH --time=24:00:00

srun python train.py --config configs/llama-70b.yaml

Out of Memory (OOM) errors are one of the most common failure modes in GPU training, and the memory math surprises a lot of people the first time. Here's a concrete example:

One more infrastructure detail worth knowing: an Out-of-Band (OOB) network is a separate, dedicated management network — physically or logically isolated from your production network — used to monitor and troubleshoot servers and switches independently. If your main GPU cluster network goes down entirely, an OOB network is what still lets you log in and diagnose the problem rather than being locked out along with everyone else.