Scaling Pre-Training in Practice: A Hierarchical Approach — Aleph Alpha

Taking a 30B-A3B MoE model from 16 to 512 B200 GPUs while achieving near-linear scaling and 35.3% MFU: how we shrink the search space at small scale before increasing the GPU count.

saved by

• Falah Rajput