Growing Bonsai Networks with RNNs - Casey Primozic's Homepage

A summary of my research and experiments on growing sparse computational graphs by training small RNNs. This post describes the architecture, training process, and pruning method used to create the graphs and then examines some of the learned solutions to a variety of objectives.

Growing Bonsai Networks with RNNs - Casey Primozic's Homepage cprimozic.net | @ameobea10 cprimozic.net @ameobea10 • Portfolio • Contact • Blog • Professional Experience Growing Bonsai Networks with RNNs 2023-07-23 Subscribe to Blog via RSS Background Bonsai Networks Overview Custom Neural Network Components Activation Function RNN Architecture Sparsity-Promoting Regularizer Trainable Initial States Tinygrad Training Grokking Constructing the Graphs Pruning Results y[n] = x[n-2] Understanding the Visualization [-1, -1, 1] loop Balanced Parenthesis Replace 1 -> 111 Complex Gated Loop with Condit

related reading

• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• A Gentle Introduction to Graph Neural Networksdistill.pub
• Ilya 30u30arc.net
• A Recipe for Training Neural Networkskarpathy.github.io
• Distill — Latest articles about machine learningdistill.pub
• Weight-Sparse Circuits May Be Interpretable Yet Unfaithful — LessWronglesswrong.com
• Neural networks and deep learningneuralnetworksanddeeplearning.com
• A Recipe for Training Neural Networkskarpathy.github.io
• Understanding Convolutions on Graphsdistill.pub
• The Decade of Deep Learning | Leo Gaobmk.sh
• The Little Book of Deep Learningfleuret.org