Generated from the lectures

Revision

What each session should leave you with, as its last slide puts it, then the routine every model went through, and a glossary of the course's terms. Each session links back to its slide.

Session 1 · Beginning Python for financial data

the slide · the lecture · its PDF

Runtime state Cells share a memory; “correct” means correct top-to-bottom from a fresh start.

Variables = assigns a name to a value; assigning again overwrites.

Types int, float, str, bool; and "150" is not 150.

Formatting f"{r:.2%}" and f"{p:.2f}": the two recipes we reuse.

Functions & methods round(x, 2) versus ticker.upper(): the dot-grammar runs the course.

Lists Index from 0, [-1] is the end, slice [a:b] excludes b.

Errors Read the last line first. Nothing is broken.

Simple return (p1 - p0) / p0: the object this course is about.

Session 2 · Functions, loops and dictionaries

the slide · the lecture · its PDF

For loops for x in items: runs the body once per item.

While loops while condition: repeats until the condition is false.

if / elif / else Run a block only when a condition holds.

Functions def name(args): with return, plus defaults and a docstring.

Dictionaries {key: value}, looked up by key, not position.

Volatility The standard deviation of the daily returns.

Session 3 · Packages: NumPy, pandas and matplotlib

the slide · the lecture · its PDF

Your own machine Install once, import in every notebook.

Looking things up Hover, or help. It is on your machine, not online.

NumPy Arrays do arithmetic all at once. @ for matrices.

pandas read_csv, filter, new columns, groupby, pivot.

matplotlib fig, ax = plt.subplots(), then label everything.

Judgment The form, the scale and the baseline decide what a reader sees.

Session 4 · Foundations of machine learning

the slide · the lecture · its PDF

One table Rows are observations, columns are features, one column is the target.

Y = f(X) + \varepsilon You estimate f. You never remove \varepsilon.

Supervised or not A target column decides it. Number means regression, label means classification.

Cleaning is the work Missing, extreme, wrong scale, not numeric. In that order, most days.

Nothing from the future Not in a fill, not in a scaler, not in a split.

Pick the metric first MAE, RMSE and R² answer different questions. So do precision and recall.

Training error always falls Which is exactly why it cannot be used to choose a model.

Bias against variance Too simple misses the shape. Too flexible learns the noise.

Session 5 · Model selection and cross-validation

the slide · the lecture · its PDF

Fitting a model LinearRegression(), .fit(X, y), .predict(X). Later models change only the first line.

X is a table, y is a column train[["vol_20d"]] with two brackets, train["vol_next"] with one.

Always report a baseline Compare the model against the training average and against a simple rule that uses no model.

Training error always prefers more columns Adding a column can never worsen the fit, so training error cannot say whether the column helps.

Choosing uses up data Rows used to compare candidates can no longer give an honest score for the winner.

One split is one measurement Moving the cut date can change the winner. Cross-validation averages over several cuts.

Read the folds, not only the mean A fold much worse than the others points to a period the model handles badly.

Folds must follow the calendar Use TimeSeriesSplit for a time series. KFold shuffles, and here it chose the worst of the three models.

Session 6 · Penalised regression

the slide · the lecture · its PDF

Many columns overfit Nineteen columns fitted the training rows better than one and forecast worse. Near-copies get large, opposite coefficients that are noise.

A penalty holds coefficients small Ridge adds \alpha \sum \beta_j^2, lasso adds \alpha \sum |\beta_j|. Ridge(alpha=...) fits and predicts like LinearRegression.

The penalty needs standardised columns It charges by coefficient size, and size depends on units. StandardScaler fitted on the training rows, then a Pipeline.

Alpha is a hyperparameter Set before .fit(), never changed by it. Chosen by cross-validation on the training rows, on a grid in factors of ten.

GridSearchCV runs the whole search Every value times every fold, the best mean kept, refitted on all training rows. best_params_, best_score_, cv_results_, .predict().

Lasso gives exact zeros A compact model, and the kept set changes with the rows. Ridge spreads weight across correlated columns.

The scales are not the same Ridge’s useful alpha was in the thousands, lasso’s near 0.1. Read the validation curve rather than reusing a grid.

Forecasting, not inference Penalised coefficients are biased on purpose and have no standard errors. Fit OLS on a few chosen columns to test an effect.

Session 8 · Classification with logistic regression

the slide · the lecture · its PDF

A label from a comparison (table["vol_next"] > table["vol_20d"]).astype(int). Predicting the majority class on every row is the share a model has to beat.

A straight line cannot give a probability Fitted to ones and zeros it runs below 0 and above 1. The sigmoid keeps the curve between them.

Logistic regression fits the log-odds LogisticRegression(), .fit(X, y). A coefficient adds to the log-odds and np.exp of it multiplies the odds; the sign is what to read.

predict_proba, then a threshold .predict_proba() gives a column per class, in the order of classes_. .predict() compares the second column with one half; any other threshold is a comparison you write.

The confusion matrix confusion_matrix, with rows for what happened and columns for the prediction. Accuracy, precision and recall are built from its four counts, and the threshold trades precision against recall.

AUC needs no threshold roc_auc_score(y, p) with the probabilities, never the predicted labels. 0.5 is a random ranking; scoring="roc_auc" on the folds.

C is one over alpha The penalty is ridge’s by default, and a small C is a strong penalty. Chosen by GridSearchCV in a pipeline with a scaler. penalty="l1" for exact zeros.

Fitted by iteration Log-loss has no formula for the coefficients, so .fit() improves them step by step. Raise max_iter when the warning appears.

Session 9 · Classification in practice

the slide · the lecture · its PDF

Accuracy hides a rare class 77.9 percent of borrowers did not default, so predicting that nobody defaults scores 0.779. Report recall beside accuracy, and read the confusion matrix.

The threshold is a decision .predict() uses one half. At a cost of 5 for a miss and 1 for a false alarm, one sixth is cheapest, and saved a quarter of the cost.

Rows that are not a time series train_test_split(..., stratify=y, random_state=0) for a cross-section, TimeSeriesSplit for days. stratify keeps the rare class in both halves.

class_weight moves the cut, not the ranking class_weight="balanced" raised recall from 0.311 to 0.550 and left AUC at 0.748. It also broke the probabilities.

Calibration: said against happened Bucket with pd.cut and compare each bucket with what happened. brier_score_loss is the one number, CalibratedClassifierCV the repair.

More than two classes classes_ is alphabetical, coef_ gets a row per class and predict_proba a column. The softmax turns the scores into probabilities.

Macro or weighted Macro treats the classes equally, weighted by their size. When the class you care about is the small one, read macro.

k-nearest neighbours KNeighborsClassifier(n_neighbors=k), scaled, with k on the folds. A large k is a simple model. Logistic regression won here.

Session 11 · Trees and forests

the slide · the lecture · its PDF

Linear regression has one slope When volatility was high, linear regression on vol_20d was 0.18 too low after a fall and 0.14 too high after a rise.

A tree is questions, boxes and a formula \hat f(\mathbf{x}) = \sum_m c_m \mathbf{1}\{\mathbf{x} \in R_m\}: each leaf forecasts the mean of its training days.

Grown greedily, one cut at a time Every column and every cut, the smallest RSS, then again inside each box; a stump is one step. The best number for a box is its mean.

plot_tree shows it plot_tree(model, feature_names=...), with filled, impurity=False, precision and label="root". Each box shows its question, error, days and forecast.

Depth is the setting Without a limit: one leaf per training day, an error of 0 on them and 0.466 on the test days. Choose the size of a tree on TimeSeriesSplit folds.

Bagging and forests, for variance Averaging 20 bootstrap trees took the error from 0.410 to 0.320. Drawing columns at random makes the trees differ, and a tuned forest reached 0.226.

Tuning, cost and size Tuning moved the forest by 0.070 and ridge by 0.030. Each setting multiplies a grid, and 500 full trees forecast worse than ridge’s 20 coefficients.

A tree for a label The same cuts, judged by Gini. On the credit table the forest beat logistic regression; on the volatility table linear regression and ridge beat every tree.

Session 12 · Boosting and reading a model

the slide · the lecture · its PDF

Averaging cannot remove bias 100 averaged stumps err 0.577 on the training days, against 0.600 for one. 100 stumps added one after another err 0.457.

Boosting fits what is left \hat f_m = \hat f_{m-1} + \nu\, h_m: each small tree h_m is fitted to the residuals, the negative gradient of the loss.

More trees can overfit At \nu = 0.1 the test error is lowest near 144 stumps, 0.221, and 0.265 at 3,000. Deeper trees turn up sooner.

Stop on a later block XGBoost stopped on 2022 at 244 stumps: 0.225. Stopped on the test days, it reported 0.204, a score those days chose.

XGBoost and LightGBM Bins, a penalty on each tree, growth leaf by leaf: on the credit table the same trees fit about 25 times faster than in GradientBoosting.

Boosting a label Each tree is fitted to y_i - \hat p_i on the log-odds scale. On the credit table, 0.777: level with the forest.

Importance, two ways Counts of cuts ranked a column of random numbers first. Shuffles on the test rows ranked it last, at zero.

Correlated columns share importance Shuffled together, late_now and months_late cost 0.206 of AUC; a refit without either one loses under 0.02.

The routine behind every model

From Session 13: the slide

A baseline first The training average, the majority class or persistence. Every score goes next to the baseline’s.

A split that respects time When the rows are days or hours, the test rows come after the training rows.

Settings on validation rows Folds or a later block choose alpha, C, k, the depth and the number of trees.

The test rows once They score the model already chosen, and the folds are reported beside them.

A model read Its coefficients, its importance, or what it does to one row.

A model priced The time to fit and to predict, and who has to understand it.

Glossary

The course's terms, each with the session that introduced it.

Accuracy Session 4, Session 8
The share of predictions that match the label. Misleading when one class is rare: always predicting the majority class can score well.
AUC Session 8
The area under the ROC curve: the chance that a randomly chosen 1 gets a higher score than a randomly chosen 0. 0.5 is a random ranking and 1 a perfect one; no threshold is involved.
Bagging Session 11
Grow many deep trees, each on a bootstrap sample of the training rows, and average their forecasts. The average cancels much of what one tree gets wrong by chance.
Baseline Session 4, Session 5
The simplest rule a model has to beat: the training average, the majority class, or persistence. A score means little until it sits next to the baseline's.
Bias and variance Session 4
Bias is the error of a model too simple to follow the truth; variance the error of a model so flexible it follows the noise of its training rows. More flexibility trades the first for the second.
Boosting Session 12
Start from the mean and add small trees one at a time, each fitted to what the trees so far still get wrong and shrunk by the learning rate.
Bootstrap sample Session 11
As many rows as the data, drawn at random with replacement, so some rows come twice and about a third not at all.
Calibration Session 9
Whether a probability means what it says: of the days given 0.3, about 30 percent should turn out to be 1s.
Classification Session 4, Session 8
Predicting a label, such as whether next month will be busier than this one, rather than a number.
Confusion matrix Session 4, Session 8
The four counts behind every score of a classifier: true positives, false positives, false negatives and true negatives.
Cross-validation Session 5
Choosing between models with the training rows only: fit on some folds, score on the next, and average. Time-series folds always score rows that come after the ones fitted on.
Data leakage Session 4
Information the forecast could not have had at the time, slipping into a feature or into the fit, such as a scaler fitted on the test rows. It makes a model look better than it is.
Decision tree Session 11
A model that asks one question of one column at a time and forecasts the mean, or the class shares, of the training rows in each leaf.
Early stopping Session 12
Add trees until the score on later rows stops improving, and keep the number of trees at which it was best.
Elastic net Session 6
The ridge and lasso penalties at once, in a mix set by l1_ratio.
F1 Session 4, Session 9
The harmonic mean of precision and recall, which sits close to the smaller of the two.
Feature Session 4
A column used to predict, built only from what was known when the forecast is made.
Feature importance Session 12
How much a model leans on each column: from its cuts, measured on the training rows, or by permutation, on rows it did not fit.
Fold Session 5
One of the blocks the training rows are cut into for cross-validation.
Gini impurity Session 11
How mixed the labels in a box are: 2p(1 - p) for two classes, and zero when the box is pure. A classification tree cuts where it falls most.
Gradient Session 12
The direction in which a loss rises fastest. For squared error, minus the gradient is the residual, which is why boosting fits its trees to residuals.
Trying every value of a setting on the folds and keeping the best; GridSearchCV then refits it on all the training rows.
Hyperparameter Session 6
A setting chosen before fitting, such as alpha, C, k, the depth of a tree or the number of trees. Chosen on validation rows, never on the test rows.
k-nearest neighbours Session 9
Classify a row by a vote among the k training rows closest to it, on standardised columns.
Lasso Session 6
Linear regression with a charge on the sum of the absolute coefficients. It sets some coefficients to exactly zero.
Learning rate Session 12
The share of each new tree that boosting adds. A smaller rate needs more trees for the same fit.
Log loss Session 8
What logistic regression minimises: minus the log of the probability given to what happened. A confident wrong probability costs the most.
Logistic regression Session 8
A linear model for a label: a weighted sum of the columns, put through the sigmoid to become a probability.
MAE, MSE and RMSE Session 4
The mean absolute, mean squared and root mean squared error of a forecast of a number. The RMSE is in the target's own units.
Majority class Session 8, Session 9
Predicting the most common label everywhere: the baseline for accuracy.
Mean reversion Session 5
A value far from normal tends to be followed by one closer to normal, such as a calm month after a busy one.
Monotone constraint Session 13
Forcing a model's forecast to rise, or to fall, as one column rises, everything else unchanged.
Out-of-bag Session 11
Scoring each training row with the trees whose bootstrap sample left it out. Fair for separate cases; too kind on days, whose neighbours were in the sample.
Overfitting Session 4
Learning the noise of the training rows along with their pattern: a lower training error and a higher test error.
Penalty Session 6
A charge on the size of the coefficients, added to what the fit minimises, with a strength set by alpha, or by 1/C in logistic regression. Also called regularisation.
Permutation importance Session 12
How far a model's score falls when one column is shuffled, on rows the model did not fit.
Persistence Session 4, Session 5
Forecasting that the next value equals the latest one: next month's volatility is this month's. Hard to beat in finance.
Pipeline Session 6
Steps chained into one model, such as a scaler and then ridge, so cross-validation refits every step inside every fold.
Precision Session 4, Session 8
Of the times the model said yes, the share that were right.
Random forest Session 11
Bagging in which each question chooses among a few columns drawn at random, so the trees differ more and their average is steadier.
Recall Session 4, Session 8
Of the real yeses, the share the model caught.
Recursive binary splitting Session 11
How a tree grows: the best single cut, then the best cut inside each new box, never revising an earlier one.
Regression Session 4, Session 5
Predicting a number, such as next month's volatility.
Residual Session 5, Session 12
What happened minus what was forecast.
Return Session 1, Session 4
The relative change in a price, (p1 - p0) / p0. The log return, log(p1 / p0), adds up over time.
Ridge Session 6
Linear regression with a charge on the sum of the squared coefficients: every coefficient shrinks towards zero, none to exactly zero.
ROC curve Session 8
The recall against the false positive rate, at every threshold.
Sigmoid Session 8
1 / (1 + exp(-z)): turns any number into a probability between 0 and 1.
Softmax Session 9
The sigmoid for more than two classes: exponentiate each class's score and divide by the total.
Standardise Session 4, Session 6
Subtract a column's mean and divide by its standard deviation, both learned on the training rows.
Stump Session 11, Session 12
A tree with one question and two leaves.
Supervised learning Session 4
Learning a rule from rows whose answer is known. Unsupervised learning looks for structure with no answer column.
Target Session 4
The column to predict, set on the row of the features it is predicted from.
Test rows Session 4
Rows held back until the end and used once, to score the model that was chosen.
Threshold Session 8, Session 9
The probability from which a classifier says 1. The default is 0.5; when the two mistakes cost different amounts, another threshold is cheaper.
Training rows Session 4
The rows a model is fitted on.
Validation Session 5
Scoring candidates on rows they were not fitted on, before the test rows: the folds, or a later block.
Volatility Session 2, Session 3
The standard deviation of returns: how much a price moves. Annualised by the square root of 252 trading days.
XGBoost and LightGBM Session 12
Fast boosting libraries that cut each column only between bins. LightGBM grows its trees leaf by leaf.