AutoML25 best paper award.

Our paper “Overtuning in Hyperparameter Optimization” (with Lennart Schneider & Bernd Bischl) won the Best Paper Award at the 4th AutoML Conference.
In this paper, we formally define and study overtuning, a phenomenon where hyperparameter optimization improves validation scores but can hurt generalization to new data. In addition, we relate it to (meta-)overfitting and conduct a thorough survey of existing methods from the literature. Our large-scale analysis shows overtuning is more common than we expected (although most of the time it does not hurt too much), occasionally severe, and influenced by factors like resampling strategy, dataset size, and model class. A better understanding of overtuning will help the AutoML community design more reliable optimization and selection strategies, especially in small-data regimes where robustness is critical.
