Cluster Test & Verification
33%Proving a cluster is ready: test each layer from one GPU up to the whole cluster and compare the results with a known-good baseline.
- Single-node stress tests and HPL
- Single-node NCCL tests, including NVLink Switch
- Cable signal quality and correct cabling
- Firmware and software on switches, BlueField-3 and transceivers
- ClusterKit node assessment
- NCCL east-west fabric bandwidth
- NCCL, HPL and NeMo burn-in
- Storage testing
Lessons that cover this domain
- The Validation Ladder: Proving a Cluster Is Ready
- GPU Health with DCGM Diagnostics
- Validating Cables and the Fabric
- NCCL Tests: What Bandwidth Should I Expect?
- HPL: The Whole-System Stress Test
- ClusterKit: Assessing Every Node at Once
- Storage Validation: Can Storage Feed the GPUs?
- Burn-in: Faults That Only Appear Under Heat
- Cluster Lab: Slow Under Load
- Lab 2: Validate One 8-GPU Node, Then Two