Gradient Boosting

Definition

Gradient Boosting is an ensemble learning technique in machine learning. The core idea: Instead of training a single complex model, many simple models are built sequentially. Each new model corrects the errors of the previous one. The result is a "committee" of weak learners that together achieve very high prediction accuracy.

The name combines "Gradient" (the mathematical direction of steepest descent of an error function) and "Boosting" (the iterative strengthening of models). Rather than climbing a mountain, the algorithm goes down the mountain -- step by step in the direction where the error decreases fastest.

The technique was formalized by Jerome Friedman in 1999. Modern implementations like XGBoost, LightGBM, and CatBoost are standard in data science competitions and production systems. Ben Kraiem et al. (2023) achieved 94% accuracy with Gradient Boosting when predicting the optimal project management methodology -- the best performance of all tested models.

Why it matters

Gradient Boosting is characterized by three properties that make it particularly suitable for complex prediction problems:

• High accuracy -- Through iterative error correction, Gradient Boosting often achieves better results than single complex models like Neural Networks.
• Feature importance -- The model transparently shows which variables have the greatest influence on the prediction. In Kraiem et al., these were team expertise and organizational culture.
• Robustness to missing values -- Unlike many other algorithms, Gradient Boosting can natively handle incomplete data.

The main disadvantage: Gradient Boosting is susceptible to overfitting if too many iterations are trained. The solution is regularization techniques (L1/L2 penalties) and early stopping.

Related terms