Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization

N/A

Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization Colin Unger, Stanford University; Zhihao Jia, Carnegie Mellon University and Meta; Wei Wu, Los Alamos National Laboratory and NVIDIA; Sina Lin, Microsoft; Mandeep Baines and Carlos Efrain Quintero Narvaez, Meta; Vinay Ramakrishnaiah, Nirmal Prajapati, Pat McCormick, and Jamaludin Mohd-Yusof, Los Alamos National Lab; Xi Luo, SLAC National Accelerator Laboratory; Dheevatsa Mudigere, Jongsoo Park, and Misha Smelyanskiy, Meta; Alex Aiken,…

related reading

• How To Scale Your Modeljax-ml.github.io
• Making Deep Learning go Brrrr From First Principleshorace.io
• How to Parallelize a Transformer for Training — an explorable explanationezyang.github.io
• Deriving Muonjeremybernste.in
• CVPR2023_eff_tutorial_molchanov.pdfnvlabs.github.io
• Transformer Math 101 | EleutherAI Blogblog.eleuther.ai
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Paradigms of Parallelism | Colossal-AIcolossalai.org
• Pipeline-Parallelism: Distributed Training via Model Partitioningsiboehm.com
• Parallelism in Distributed Deep Learning · Better Tomorrow with Computer Scienceinsujang.github.io
• Muon: An optimizer for hidden layers in neural networks | Keller Jordan blogkellerjordan.github.io
• The Little Book of Deep Learningfleuret.org