Dr. Maxim Kodryan
23 Jan 2024On January 23 the defense of Maxim Kodryan's PhD thesis on "Training Dynamics and Loss Landscape of Neural Networks with Scale-Invariant Parameters" took place.
Scale invariance is one of the key properties inherent in the parameters of most modern neural network architectures. Provided by the ubiquitous presence of layers of normalization of intermediate activations and/or weights, scale invariance, as the name implies, consists in the invariance of the function implemented by the neural network when its parameters are multiplied by an arbitrary positive scalar. In his work, Maxim investigates the effects of this property on the training dynamics of neural network models, as well as its influence on the intrinsic structure of the loss landscape.