Stochastic Spectral Descent for Restricted Boltzmann Machines

Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated w...

Stochastic Spectral Descent for Restricted Boltzmann Machines David Carlson, Volkan Cevher, Lawrence Carin Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics , PMLR 38:111-119, 2015. Abstract Restricted Boltzmann Machines (RBMs) are widely used as building blocks for deep learning models. Learning typically proceeds by using stochastic gradient descent, and the gradients are estimated with sampling methods. However, the gradient estimation is a computational bottleneck, so better use of the gradients will speed up the descent algorithm. To this end

related reading

• Stochastic gradient descent - Wikipediaen.m.wikipedia.org
• Why Momentum Really Worksdistill.pub
• The Little Book of Deep Learningfleuret.org
• David Duvenaudcs.toronto.edu
• NeurIPS-2021-understanding-end-to-end-model-based-reinforcement-learning-methods-as-implicit-parameterization-Supplemental.pdflis.csail.mit.edu
• A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
• [1512.04202] Preconditioned Stochastic Gradient Descentarxiv.org
• [2101.12176] On the Origin of Implicit Regularization in Stochastic Gradient Descentarxiv.org
• [2503.22478] Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descentarxiv.org
• Keep the gradient flowingfa.bianp.net
• [2507.13181] Spectral Bellman Method: Unifying Representation and Exploration in RLarxiv.org
• A Free Lunch from the Noise:Provable and Practical Exploration for Representation Learningarxiv.org