Samuel K. Ainsworth writes with Jonathan Hayase and Siddhartha Srinivasa that neural network loss landscapes often contain nearly a single basin after accounting for permutation symmetries of hidden units, and they introduce three algorithms to permute one independently trained model into alignment with a reference model so the two can be merged in weight space with near-convex interpolation. Experiments across MLPs, CNNs and ResNets on MNIST, CIFAR-10/100 and ImageNet show the single-basin phenomenon, including the first zero-barrier linear mode connectivity between independently trained ResNets on CIFAR-10, and reveal how mode connectivity emerges with training time and model width. The authors also provide a counterexample showing linear mode connectivity is not universal and discuss limits of the hypothesis.
- Permutation symmetries for a 3-layer 512-width MLP are ~10^3498, far exceeding atoms in the observable universe.
- Weight matching uses coordinate descent on the sum of bilinear assignments problem and runs in seconds without accessing training data.
- Activation matching reduces to a linear assignment problem per layer via ordinary least squares regression on unit activations.
- Straight-through estimator learning of permutations gives the best barriers but is much more expensive than weight or activation matching.
- Merging models trained on disjoint CIFAR-100 splits with weight matching improves test loss and calibration over both inputs while using the compute/memory of one model.
- Permutation symmetries for a 3-layer 512-width MLP are ~10^3498, far exceeding atoms in the observable universe.
- Weight matching uses coordinate descent on the sum of bilinear assignments problem and runs in seconds without accessing training data.
- Activation matching reduces to a linear assignment problem per layer via ordinary least squares regression on unit activations.
- Straight-through estimator learning of permutations gives the best barriers but is much more expensive than weight or activation matching.
- Merging models trained on disjoint CIFAR-100 splits with weight matching improves test loss and calibration over both inputs while using the compute/memory of one model.
