Josh Yan

on the atlas — 2

• Modular Manifolds - Thinking Machines Lab16 savers
• Curius / Onboarding2668 savers

highlights — 2

• When we train large neural networks, we need to keep them healthy. We do not want the tensors in the network—either the weights, activations or gradients—to grow too large or too small. Very small and very large tensors cause a variety of problems not just limited to numerical underflow and overflow. For example, weight matrices changing size during training makes it harder to design training algorithms—since the relative size of updates to weights has a significant impact on the speed of learning.

  Modular Manifolds - Thinking Machines Lab

• problem of weight norms exploding.

  Modular Manifolds - Thinking Machines Lab