Penalised regression

Machine Learning in Finance · Session 6

Last time: choosing between candidates

0.740
one column
0.772
two columns
0.747
three columns

Cross-validation compared three candidates and kept the one-column model. Each candidate either contained a column or did not.

Three columns give eight possible subsets. A real table has far more columns than three.

This session

  • a table with nineteen columns, and what plain regression does with it
  • a penalty on the size of the coefficients: ridge regression
  • why the penalty needs every column on the same scale, and a pipeline
  • choosing the penalty’s strength with the folds: hyperparameter tuning
  • the lasso, which sets coefficients to exactly zero, and the elastic net
  • what the arguments of these scikit-learn objects do, and when to change them

1 · Many columns

Part 1

A wider table, and what ordinary least squares does with it.

The question

How volatile will the whole market be over the next twenty trading days?

Why it matters the risk of almost any portfolio moves with the index, so its volatility sets margins, limits and option prices

What you know today the index’s own history over several windows, and how volatile each stock in it has been lately

The table

One row per trading day, built from prices.csv with rolling and shift as last time. The index is SPY, the S&P 500 fund in the data.

Nineteen columns, one target

vol_5d … vol_120d ret_5d … ret_60d AAPL_vol … DIS_vol vol_next the index’s volatility, six windows its return, three windows ten stocks, 20-day volatility the next twenty days 2,376 rows, returns in percent, every column looking back from today

Building this table from the price file is one of the exercises.

Splitting by date

table.columns[:-1] is every column except the last, which is the target.

A helper for the error

The RMSE is computed many times this session. A two-line function keeps every later cell short.

One column, as before

This is the same model as last time, on the index. A test error of 0.237 is the number to beat.

All nineteen columns

The training error fell and the test error rose. Eighteen extra columns made the forecast worse.

The nineteen coefficients

Higher volatility at J&J and JPMorgan lowers the forecast, and the index’s own volatility raises it at 60 days and lowers it at 120. Those signs are not believable.

The columns are near-copies of each other

vol_40d and vol_60d correlate at 0.93. Two columns with the same information can take +5 and −4.8 and fit exactly as well as +0.1 and +0.1. The fit is fine; the coefficients are noise.

The coefficients depend on the rows

Each column has five dots, one per block of training rows. vol_120d is +1.02 on one block and −0.94 on another. A coefficient that changes sign with the sample cannot be used for next year.

From half a million fits to one

choose a subset of the 19 columns 524,288 candidates to cross-validate and each column is all in or all out keep every column, charge for size 1 fit every coefficient pulled towards zero a little for useful columns, a lot for noise the right-hand side is penalised regression, also called regularised

Subset selection asks, for each column, in or out. A penalty asks a softer question: how large may its coefficient be?

2 · Ridge regression

Part 2

A penalty on the squared coefficients, and one new argument.

What ordinary least squares minimises

\text{RSS} = \sum_{i=1}^{n}\left(y_i - \hat{y}_i\right)^2

The residual sum of squares on the training rows, and nothing else. A coefficient of +5 is accepted if it lowers the RSS by a little, which is how +5 and −4.8 on two near-copies come about.

The ridge objective

\text{RSS} + \alpha \sum_{j=1}^{p} \beta_j^2

Ridge adds a charge for the size of the slopes. A coefficient of 5 now costs 25 times alpha, and is kept only if it lowers the RSS by more than that.

The sum runs over the slopes and leaves the intercept alone.

What alpha does

alpha = 0 alpha in between alpha very large no penalty at all ordinary least squares every slope pulled towards zero but none reaches it every slope is zero the forecast is the training average

alpha is the price of coefficient size. It is set before .fit() runs, and the rows cannot choose it, because every value gives a fitted model.

Ridge in code

Only the first line changed: Ridge fits and predicts exactly like LinearRegression. 0.228, against 0.248 unpenalised and 0.237 for one column.

Your turn 1

Fit the same nineteen columns with a much smaller penalty and a much larger one. Which of the earlier numbers does each land near?

Try 1 for the small one and 100000 for the large one.

Your turn 1: answer

small = Ridge(alpha=1)         # 0.247
large = Ridge(alpha=100000)    # 0.282

alpha=1 is ordinary least squares to three decimals: a penalty of 1 is nothing next to an RSS over 1,894 rows. alpha=100000 is worse than repeating this month, and at ten million the error is 0.291, the same as guessing the average.

Too little penalty leaves the overfitting in. Too much removes the signal with the noise.

Why smaller coefficients forecast better

These are the same four columns on the same five blocks. The ridge coefficients are small and they stay put, so a coefficient fitted on the past holds on the years to come. That is a little bias in exchange for far less variance.

3 · The penalty and the units

Part 3

The penalty charges by coefficient size, and size depends on units.

The same column, in two units

typical values the OLS coefficient its ridge penalty, β² vol_20d in percent 0.98 1.21 0.90 0.019 0.0004 vol_20d in decimals 0.0098 0.0121 0.0090 1.913 3.66 the same column, the same forecast, and a penalty ten thousand times larger

Ridge shrinks the column in small units first, whatever it contains. The penalty is judging columns by their units, not by their usefulness.

Measured: OLS

The coefficient is exactly one hundred times larger and the test error is exactly the same.

Measured: ridge

The coefficient on vol_20d falls from 0.031 to 0.0003. Ridge has dropped the column, and nothing about its information changed.

Put every column on the same scale

z = \frac{x - \bar{x}}{s}

Subtract each column’s mean and divide by its standard deviation. Every column then has mean 0 and standard deviation 1, whatever units it arrived in, and one coefficient is comparable with another.

The standardising formula from two sessions ago, where it was noted that penalised regression needs it.

StandardScaler

.fit() learns each column’s mean and standard deviation from the training rows. .transform() applies them, to the test rows as well: their own mean is information from the future.

Ridge on standardised columns

With every column in standard deviations, the same model and the same alpha score 0.218, down from 0.228.

Reading standardised coefficients

The bars are comparable now. The last five and ten days matter most, a market that fell in the last week is more volatile in the month after, and the other stocks each add a little.

Two steps that belong together

training rows test rows StandardScaler fit, then transform StandardScaler transform only, training mean and sd Ridge fit Ridge predict

The training rows are scaled and then fitted; the test rows are scaled the same way and then predicted. Cross-validation repeats the pair on five fitting blocks, so the scaler has to be refitted inside every fold with the model.

Pipeline

A pipeline is a list of named steps. .fit() fits the scaler, transforms, and fits the ridge; .predict() transforms with the fitted scaler and predicts. The names are yours to choose.

A pipeline is a model

cross_val_score takes the pipeline where it took LinearRegression() last time, and refits the scaler inside each fold. The third fold still contains March 2020.

4 · Choosing alpha

Part 4

A setting the data cannot pick, and the tool that searches over it.

Parameters and hyperparameters

Parameters

The intercept and the nineteen slopes. .fit() computes them from the training rows and stores them with a trailing underscore.

Hyperparameters

alpha. Passed in before .fit() and never changed by it. Every value gives a fitted model, so the training rows cannot say which value is right.

A hyperparameter is chosen the way candidates were chosen last time: fit with each value, score on held-out rows, keep the best.

Try several

The cross-validated error falls to 1000 and rises again. The folds locate the point between too little penalty and too much.

The validation curve

This is the same loop over a finer grid. High on the left where the penalty is too weak, high on the right where it is too strong, lowest at a few thousand.

Why the grid goes in factors of ten

Each line is one coefficient as alpha grows. Between 1 and 10 almost nothing happens; between 1,000 and 10,000 everything does. The grid steps by a factor, not by a fixed amount.

GridSearchCV

fold 1fold 2fold 3fold 4fold 5mean
alpha 10.5540.6531.0780.8900.4140.718
alpha 100.5370.6281.0770.8230.4080.694
alpha 1000.5400.5341.0820.5470.3920.619
alpha 10000.4760.4721.1270.3270.3970.560
alpha 100000.4720.4811.2360.2830.4630.587
bestalpha = 1000 has the lowest mean. It is refitted on all 1,894 training rows, and that model is the result.

Five values times five folds is twenty-five fits, run by one call.

GridSearchCV in code

The grid is a dictionary of settings and the values to try. "ridge__alpha" is the step name, two underscores, then the argument: that is how a setting inside a pipeline is addressed.

What the search holds afterwards

cv_results_ is the whole grid as a table: every value tried, its mean score over the folds, the spread, and its rank.

The arguments of GridSearchCV

argument what it changes when to change it
estimator the model or pipeline being tuned always a pipeline when scaling is involved
param_grid the settings and the values to try keep each list short; several keys multiply
cv how the folds are made always set it: the default ignores time order
scoring the score, larger is better match it to the error you will report
refit refit the best on all training rows leave True, so .predict() works
n_jobs how many folds run at once -1 on your own machine, to use every core
return_train_score also record the training score to see how large the overfitting gap is

Your turn 2

The best alpha was 1000, with 100 and 10000 on either side. Search a finer grid around it and see whether the score improves.

Five values between 300 and 4000, stepping by roughly a factor of two.

Your turn 2: answer

fine = {"ridge__alpha": [300, 600, 1000, 2000, 4000]}

The best is 2000, at 0.5583 against 0.5599 for 1000. On the test rows the two give 0.217 and 0.218.

The gain is in the third decimal, and the fold scores range from 0.33 to 1.13. One significant figure of alpha is enough.

5 · Lasso

Part 5

A different penalty, which sets coefficients to exactly zero.

One change to the penalty

\text{ridge:}\quad \text{RSS} + \alpha \sum_{j=1}^{p} \beta_j^2

\text{lasso:}\quad \text{RSS} + \alpha \sum_{j=1}^{p} |\beta_j|

The squares become absolute values. Everything else is the same: a price on size, set by alpha, chosen by cross-validation.

Why absolute values give exact zeros

With two columns the fit can be drawn. The grey rings are equal RSS around the OLS point, and the penalised fit is the point of the shaded region on the smallest ring it reaches. A diamond has corners on the axes, so the ring usually meets it where one coefficient is exactly zero.

The two paths

Ridge shrinks every coefficient smoothly and none reaches zero. Lasso drives them to zero one at a time. The alpha scales differ too: ridge’s useful range is in the thousands here, lasso’s is between 0.01 and 1.

Lasso in code

The same pipeline with Lasso in the second step scores 0.217, level with ridge.

Which columns survived

Fourteen of the nineteen coefficients are exactly zero. named_steps["lasso"] reaches the fitted Lasso inside the pipeline, which has no coef_ of its own.

Choosing lasso’s alpha

This is the same search with a grid on lasso’s scale. Alpha 0.1 has the lowest cross-validated error, 0.580 against 0.560 for ridge.

The zeros are not a verdict

On the early blocks the lasso keeps XOM_vol and DIS_vol; on the later ones, vol_5d and AAPL_vol. Among near-copies it keeps one, and which one depends on the rows.

A zero means the column was not needed given the others, on these rows. It does not mean the column carries no information.

Ridge or lasso

Ridge

Every column stays in with a small coefficient. Suits a table where many columns each carry a little and are correlated: the weight is spread across near-copies instead of landing on one.

Lasso

A short list of columns and zeros elsewhere. Suits a table where a few columns matter, or where a compact forecast is wanted. Among near-copies it keeps one, chosen by the rows.

Cross-validate both and compare. Here ridge is slightly ahead, and the two are within the fold spread of each other.

Elastic net

\text{RSS} + \alpha\left[\, \rho \sum_{j} |\beta_j| + \tfrac{1 - \rho}{2} \sum_{j} \beta_j^2 \right]

The elastic net uses both penalties at once. alpha sets the total and l1_ratio, the \rho above, sets the mix: 1 is the lasso, 0 is ridge. It is used when the columns are correlated and a compact model is still wanted.

Searching two settings at once

Two keys in the grid give three times three combinations, and five folds make forty-five fits. The best mix keeps seven columns.

6 · Reading the arguments

Part 6

Which settings matter, which do not, and how to find out.

Every setting has a default

get_params() lists every argument of an object and its current value. Ridge(alpha=1000) changes one and leaves the rest. help(Ridge) prints the documentation for all of them.

When to change what

Always alpha, the penalty’s strength: by cross-validation on a grid in factors of ten. cv: TimeSeriesSplit for any time series. scoring: the error you will report.

Sometimes max_iter, the passes the lasso solver may take: raise it when the warning appears. l1_ratio, the mix: for the elastic net. n_jobs=-1, folds in parallel: on your own machine.

Rarely fit_intercept, an unpenalised intercept: leave it on. positive, coefficients of zero or above: when the sign is known. tol, when the solver counts as finished: leave it.

Why only lasso has max_iter Ridge has a closed-form solution. Lasso and ElasticNet are solved by repeated passes over the columns, and the passes can run out.

The convergence warning

ConvergenceWarning: Objective did not converge. You might want to increase
the number of iterations, check the scale of the features or consider
increasing regularisation.

The lasso solver ran out of passes before settling. The result is returned anyway, and it is slightly wrong.

Standardise first columns on wildly different scales are the usual cause, and the pipeline already handles it

Raise max_iter Lasso(alpha=0.001, max_iter=10000). A very small alpha needs more passes than the default 1,000

7 · What the coefficients mean now

Part 7

Penalised coefficients are for forecasting, and what that rules out.

Prediction, not inference

Ordinary least squares

Unbiased coefficients with standard errors, so a t-test can ask whether a column has an effect. The econometrics toolkit applies.

Ridge and lasso

Coefficients shrunk towards zero on purpose. They are biased, they have no standard errors, and there is no summary(). They answer one question: what forecast.

To test whether a column matters, fit OLS on a few columns chosen for a reason and read the standard errors. To forecast, penalise.

What can still be said

A standardised ridge coefficient of 0.079 on vol_5d means: one standard deviation more volatility over the last five days raises the forecast by 0.079 percentage points, with the other eighteen columns held where they are.

Among near-copies the split is arbitrary: 0.079 on vol_5d and 0.061 on vol_10d could as well be the other way round. Read related columns together.

Six forecasts of the same thing

0.291
guess the average
0.260
repeat this month
0.237
one column
0.248
nineteen columns, OLS
0.218
nineteen columns, ridge
0.217
nineteen columns, lasso

All six are test RMSEs in percentage points. Nineteen columns were worse than one until the coefficients were held small; then they were the first to beat it, with alpha chosen without the test rows.

The workflow now

1. seal the test rows 2. scaler and model in a pipeline 3. grid of alphas times the folds 4. keep the best alpha 5. refit on all training rows 6. score, once steps 2 to 5 are one call: GridSearchCV(pipeline, grid, cv=folds, scoring=…).fit(train) choosing happens in here reporting

The workflow from last time, with the candidates replaced by a grid of alpha values and the model replaced by a pipeline.

What stays with you

Many columns overfit Nineteen columns fitted the training rows better than one and forecast worse. Near-copies get large, opposite coefficients that are noise.

A penalty holds coefficients small Ridge adds \alpha \sum \beta_j^2, lasso adds \alpha \sum |\beta_j|. Ridge(alpha=...) fits and predicts like LinearRegression.

The penalty needs standardised columns It charges by coefficient size, and size depends on units. StandardScaler fitted on the training rows, then a Pipeline.

Alpha is a hyperparameter Set before .fit(), never changed by it. Chosen by cross-validation on the training rows, on a grid in factors of ten.

GridSearchCV runs the whole search Every value times every fold, the best mean kept, refitted on all training rows. best_params_, best_score_, cv_results_, .predict().

Lasso gives exact zeros A compact model, and the kept set changes with the rows. Ridge spreads weight across correlated columns.

The scales are not the same Ridge’s useful alpha was in the thousands, lasso’s near 0.1. Read the validation curve rather than reusing a grid.

Forecasting, not inference Penalised coefficients are biased on purpose and have no standard errors. Fit OLS on a few chosen columns to test an effect.

Where this goes next

beforefitting a model, and choosing between candidates with cross-validationlinear regression
nowholding coefficients small, and choosing how small with a grid searchridge, lasso, elastic net
nextclassification, where the same penalty appears under the name Clogistic regression
thentrees and ensembles, with more hyperparameters and the same searchthe same workflow

Every later model has settings chosen the way alpha was chosen today: a pipeline, a grid, the folds, and the test rows opened once at the end.

Before the next session case

  • Exercises: building the wide table from the price file, ridge and lasso, standardising inside a pipeline, and reading a grid search.
  • Case Part 6: the risk report’s model gets every column the desk can offer, and a penalty chosen by cross-validation.

Stuck? Ask a friend, ask an AI for a hint rather than an answer, or email jobo@econ.au.dk.