flâneur

RMSprop — PyTorch 2.4 documentation

pytorch.org · 1,921 words · saved by 1 readers

For further details regarding the algorithm we refer to lecture notes by G. Hinton. and centered version Generating Sequences With Recurrent Neural Networks. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus 𝛾 / ( 𝑣 + 𝜖 ) γ/( v ​ +ϵ) where 𝛾 γ is the scheduled learning rate and 𝑣 v is the weighted moving average of the squared gradient. params (iterable) – iterable of parameters to optimize or dicts defining parameter groups lr (float, optional) – learning rate (default: 1e-2) momentum (float, optional) – momentum factor (default: 0) alpha (float, optional) – smoothing constant (default: 0.99) eps (float, optional) – term added to the denominator to improve numerical stability (default: 1e-8) centered (bool, optional) – if True, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay (float, opti

class torch.optim.RMSprop(params, lr=0.01, alpha=0.99, eps=1e-08, weight_decay=0, momentum=0, centered=False, capturable=False, foreach=None, maximize=False, differentiable=False)[source]# Implements RMSprop algorithm. input:α (alpha), γ (lr), θ0 (params), f(θ) (objective)λ (weight decay), μ (momentum), centered, ϵ (epsilon)initialize:v0←0 (square average), b0←0 (buffer), g0ave←0for t=1 to … dogt←∇θft(θt−1)if λ≠0gt←gt+λθt−1vt←αvt−1+(1−α)gt2vt~←vtif centeredgtave←gt−1aveα+(1−α)gtvt~←vt~−(gtave)2if μ>0bt←μbt−1+gt/(vt~+ϵ)θt←θt−1−γbtelseθt←θt−1−γgt/(vt~+ϵ)return θt\begin{aligned}…

related reading