More than 11% of Meta’s Prometheus GPUs are reportedly idle or reserved
TechnologyUnverifiedUpdated:
Meta says more than 11% of Prometheus GPUs are failed or held in reserve for failures. It also reports a large reduction in training interruptions over a year, alongside the spare-capacity strategy.
Eight of seventy-two GPUs held as replacement spares
According to Meta, Prometheus in New Albany, Ohio uses GB200 and GB300 systems with 72 GPUs per NVLink domain, but only 64 active at a time. The remaining eight are hot spares intended to replace failed GPUs. Meta also describes data-parallel training, checkpointing, telemetry, and diagnostics intended to limit the effect of failures. It says the cluster achieved more than 85 percent goodput for pretraining and more than 90 percent for reinforcement learning on Muse Wonder.
Meta says that during the first phase of Prometheus it saw roughly 1,000 training interruptions per day for every 8,000 GPUs.
Meta says it reduced the rate to about 11 daily interruptions per 8,000 GPUs over the following year, while reserving eight of the 72 GPUs in each NVLink domain as hot spares.
The account illustrates the redundancy cost of keeping large distributed AI-training runs operating: a substantial share of installed GPUs can be unavailable for ordinary work because they are failed or reserved for continuity.
Sources
- The RegisterTens of thousands of Nvidia GPUs sit idle in Meta's Prometheus supercluster, just waiting for a failureRead source →