Property and casualty ratemaking has gone through a quiet but profound transformation. From tables and factors built almost by hand, we moved to formal statistical models and, in recent years, to machine learning algorithms capable of capturing interactions that no actuary would code manually. Understanding that arc helps us use each tool where it belongs.
A brief history of ratemaking
In its early days, ratemaking was largely an exercise in cell-based segmentation: risks were grouped by observable characteristics and an average premium was computed for each group. The method was transparent, but it struggled when cells ran thin or when several factors interacted. The need for a framework that could estimate the simultaneous effect of several variables opened the door to formal statistics.
The GLM as the standard
The generalized linear model (GLM) became the industry standard for good reasons. It can model claim frequency, typically with a Poisson distribution, and severity, with a Gamma distribution, through a link function that connects the linear predictor to the mean. The general form is $g(\mu) = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p$, where $g$ is the link function.
The log link is especially convenient in insurance because it turns coefficients into multiplicative factors: each variable adjusts the base premium by a factor, which is intuitive and easy to communicate to underwriters and regulators. The GLM offers interpretable coefficients, confidence intervals and solid statistical theory. For years the Casualty Actuarial Society (CAS) has promoted literature and exams that made the GLM the common language of modern ratemaking.
Regularization: Ridge and Lasso
As the number of variables grows, the classical GLM can overfit or become unstable in the presence of collinearity. Penalization emerged as the remedy. Ridge regression adds a penalty on the sum of squared coefficients, of the form $\lambda \sum_j \beta_j^2$, which shrinks the coefficients and stabilizes the estimates. Lasso regression penalizes the sum of absolute values, $\lambda \sum_j |\beta_j|$, with the added virtue of driving some coefficients exactly to zero, which performs automatic variable selection. Both preserve the interpretability of the GLM while controlling complexity.
Trees and ensembles
Decision trees follow a different logic. Instead of imposing a functional form, they split the risk space into regions through successive cuts on the variables. A single tree is intuitive but unstable and prone to overfitting. The answer was ensembles: combining many trees to obtain more robust predictions.
Two families dominate. Bagging, of which the random forest is the best-known example, trains many trees on resampled data and averages their results, reducing variance. Boosting, by contrast, builds trees sequentially, with each new tree correcting the errors of the ensemble so far.
Gradient boosting
Gradient boosting takes that idea to its most powerful form. The technique fits each new tree to the gradient of a loss function with respect to the current predictions, moving step by step toward the minimum. Implementations such as XGBoost and LightGBM made it fast, scalable and able to handle large volumes of data with many variables.
For ratemaking its appeal is obvious: it captures interactions and nonlinearities that a GLM models only if the actuary specifies them by hand. In many comparative studies, a well-tuned gradient boosting model outperforms a GLM in predictive accuracy, particularly when there are complex relationships among risk characteristics.
How they fit into property and casualty insurance
In practice, adoption is rarely a wholesale replacement. Hybrid and complementary approaches have emerged.
- Structure discovery. Many teams use gradient boosting to detect relevant interactions and then feed those signals into the GLM, keeping an interpretable model in production.
- Combined models. The GLM captures the main effects and a boosting model fits the residual, aiming for the best of both worlds.
- Direct production use. Some portfolios deploy gradient boosting as the rating model, where the regulatory and governance framework allows it.
The price of power: interpretability
The predictive power of gradient boosting comes at a cost: opacity. An ensemble of hundreds of trees offers no clean coefficients to explain to a regulator, or to a policyholder asking why their premium went up. In insurance, where fairness and nondiscrimination are legal and ethical requirements, that opacity is a real risk.
That is why the field of interpretability has flourished. Shapley values, drawn from game theory, attribute to each variable its contribution to an individual prediction. Partial dependence plots show the marginal effect of a variable on the premium. These tools make it possible to audit a complex model without giving up its performance.
An evolution, not a rupture
The path from the GLM to gradient boosting is not the story of one technique dethroning another, but of a toolkit that keeps growing. The GLM remains irreplaceable for its transparency and regulatory fit; gradient boosting adds precision where complexity justifies it. A good pricing actuary does not pick a side: they pick the right tool for each problem and always answer for the fairness and solvency behind the number.