Adaptive Hybrid Bayesian Optimization and Dimension Based Blocking for Hyperparameter Tuning

Abstract

The exponential growth in machine learning model complexity has rendered traditional hyper- parameter optimization methods, such as grid and random search, computationally intractable due to the curse of dimensionality. Bayesian optimization offers a data-efficient alternative, whereas standard approaches often rely on static surrogate models and heuristic batch selection strategies that require manual tuning. This dissertation improves upon the state of Bayesian optimization through two primary contributions. First, we introduce an Adaptive Hybrid Bayesian Optimization framework. By utilizing Student-t Process surrogates with a novel sample-size-dependent degree of freedom schedule, this method automatically transitions from heavy-tailed exploration in early stages to Gaussian-like exploitation in later stages, eliminating the need for manual exploration parame- ter tuning. Second, to address the inefficiencies of parallel evaluation, we propose Dimension Based Blocking, a metric-free batch selection strategy. Unlike distance based baselines that suffer in high-dimensional spaces, this approach enforces spatial diversity through dynamic, axis-aligned exclusion regions. Comprehensive empirical analysis on synthetic benchmarks and real-world datasets (MNIST, Fashion-MNIST, CIFAR-100) demonstrates that these methods significantly outperform estab- lished baselines. Specifically, the Adaptive Hybrid framework achieves superior consistency and variance control on difficult optimization landscapes, while Dimension Based Blocking maintains higher batch diversity, enabling more efficient parallelization.