Why it matters

Hyperparameter search is expensive. μP + Bayesian help. Grid search is dead.

Advertisement

The architecture

μP for transfer from small.

Bayesian optimization.

Population-based training.

Hyperparameter searchμP transfersmall → largeBayesian optsample efficientPBTpopulation-basedμP (Yang 2022); Optuna, W&B Sweeps for Bayesian; PBT for pop-based
Hyperparameter search.
Advertisement

How it works end to end

μP transfer (Yang 2022).

Optuna, W&B Sweeps.

PBT for population.