Hyperparameters and Regularization Shape ReLU Network Loss.

This study delves into the theoretical properties of two-layer ReLU networks, particularly focusing on how width-dependent hyperparameters and L2-regularization influence their loss landscape. Understanding these dynamics is crucial for advancing deep neural network theory. The researchers derived a sufficient condition under which global minima can collapse to the zero solution, a phenomenon that can hinder effective learning. Interestingly, their experiments revealed a distinction between optimizers: AdamW was observed to prevent this collapse of learned parameters, while SGD did not. This finding offers a potential explanation for AdamW's widespread success in deep learning training. Furthermore, for networks with a single input dimension, the study provides an analytical solution for the globally optimal parameter sets. It demonstrates that L2-regularization has a width-invariant effect on connectivity but its dimensionality-reducing impact intensifies with increasing network width. These results offer valuable insights into the intricate interplay between network architecture, regularization, and optimization strategies.