Algorithm
Start with baseline. For each iteration: compute gradient at current predictions, fit tree to gradient, add tree × learning rate.
Advertisement
XGBoost innovations
Regularized loss (L1+L2 on leaf weights). Sparse-aware split finding. Weighted quantile sketch. Cache-aware algorithm.
Advertisement
LightGBM
Histogram-based splits (bin features). Leaf-wise (not level-wise) tree growth. 10x faster than XGBoost.
CatBoost
Symmetric (oblivious) trees. Ordered boosting for target leakage. Native categorical support.