Tangentially / A Machine Learning Blog | Aaron Defazio, Machine Learning Researcher & Data Scientist
Konstantin and I were honored with an ICML Outstanding paper award for our paper “Learning-Rate-Free Learning by D-Adaptation”. Besides the award, our implementation has already been downloaded nearly a million times over the last year (most before the award), and is seeing wide-spread use in training and fine-tuning deep learning models, particularly LoRAs. Read on if you would like to know more about our work and the impact it is already having as a practical solution that removes the need to tune the baseline learning rate. Conference awards often elevate papers that take interesting new approaches to old problems, and this is certainly the case here. Our approach sounds like it wouldn’t work: we take an upper bound that would normally be used to show convergence, and we invert it, to instead give a lower bound 𝑑 𝑘 𝑑 𝑘 on a key quantity 𝐷=∥ 𝑥 0 − 𝑥 ∗ ∥ 𝐷 = ∥ 𝑥 0 − 𝑥 ∗ ∥ . Since 𝐷 𝐷 plays the key role of the numerator of the step size in convex Lipscthiz optimization