flâneur

Choosing Gradient Norm Clip Value? [D] : r/MachineLearning

reddit.com · 105 words · saved by 1 readers

How do I choose the max value to use for global gradient norm clipping? The value must somehow depend on the number of parameters because more parameters means the parameter gradient vector has more numbers in it and higher dimensional vectors have bigger norms than lower dimensional ones. I can not find anything online giving advice on choosing the max norm value. Can anyone give me advice that improves over "just try loads of values". (Am training transformers) [sidequest] Legends are told about weird results when adam is combined with gradient clipping. We don't have compute and I have cannot leave my sick brother to go on an adventure. If only there was someone who could look into that old myth ... 😂 😂 😂 I wrote a paper a few months back about adapting the clipping value based on the history of gradient norms using percentiles. I use it all the time now for every model I train (with p=10): https://github.com/pseeth/autoclip. Give it a shot! For my use cases (training audio netwo

Common values are 1, 3, 5, 8, 10. Sometimes I saw something like 0.01 with a lot of epochs (and I mean a lot). I would recommend you to train your network without any clipping for one (or two) epoch, than inspect some layers (in the beginning, in the middle and in the end) and check their norms and abs values of the weights - it will give you some ideas how to move forward. I didn't dig too deep into clipnorm and clipvalue, but I never saw any theoretical proofs on why some values are good/bad. It's more of a empirical thing I guess.

related reading