Crash in NCCL when running with distributed GPU · Issue #13559 · google/jax
Description (could be a cluster issue, but unclear. This is on an HPC where I don't have sudo, and there are a lot of CUDA installations floating around, so I haven't ruled out that it's not some d...
Crash in NCCL when running with distributed GPU · Issue #13559 · jax-ml/jax · GitHub Skip to content You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} Uh oh! There was an error while loading. Please reload this page . jax-ml / jax Public Notifications You must be signed in to change notification settings Fork 3.7k Star 36k Crash in NCCL when running with distributed GPU #13559 New issue Copy
related reading
- From bare metal to a 70B model: infrastructure set-up and scripts - Imbueimbue.com
- Massively Scale Your Deep Learning Training with NCCL 2.4 | NVIDIA Technical Blogdeveloper.nvidia.com
- Inside NVIDIA GPUs: Anatomy of high performance matmul kernels - Aleksa Gordićaleksagordic.com
- CUDA C++ Programming Guide (Legacy) — CUDA C++ Programming Guidedocs.nvidia.com
- 2410.21680arxiv.org
- What happens when you run a CUDA kernelfergusfinn.com
- GitHub - adam-maj/tiny-gpu: A minimal GPU design in Verilog to learn how GPUs work from the ground up · GitHubgithub.com
- the bug that taught me more about PyTorch than years of using it | Elana Simonelanapearl.github.io
- From Single-Node to Multi-GPU Clusters: How Discord Made Distributed Compute Easy for ML Engineersdiscord.com
- Mixture-of-Kittens: our open-source MoE megakernel for NVL72scursor.com
- How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
- ClusterMAX™ 2.0: The Industry Standard GPU Cloud Rating Systemnewsletter.semianalysis.com