flâneur

A minimizer Far, Far Away – Parameter-free Learning and Optimization Algorithms

parameterfree.com · 1,329 words · saved by 1 readers

A few recent Arxiv papers and some recent conversations during my lectures made me realize that some optimization people might not be fully aware of important details on SGD when used on functions where the minimizer can be arbitrarily far from the initialization or even in the case when the minimizer does not exist. So, let’s talk about it. First of all, when does this happen? Well, in machine learning it is very common. For example, if you run logistic regression on a separable dataset, or SGD with universal kernels and no repeated data points, or even in deep learning when we assume the so-called interpolation assumption and use cross-entropy + softmax. In all these cases, the minimizer does not exist because we can make the training error arbitrarily close to by increasing the norm of and keeping its direction fixed. In more intuitive terms, we can say that the minimizer is “at infinity”. So, is this a weak or a strong assumption? I’ll let you decide, but for sure if you believe

on Feb 14 , 2024 February 15, 2024 A few recent Arxiv papers and some recent conversations during my lectures made me realize that some optimization people might not be fully aware of important details on SGD when used on functions where the minimizer can be arbitrarily far from the initialization or even in the case when the minimizer does not exist. So, let’s talk about it. First of all, when does this happen? Well, in machine learning it is very common. For example, if you run logistic regression on a separable dataset, or SGD with universal kernels and no repeated data points, or eve

related reading