One row per function: the call, with its arguments, and what it does.
Grouped by topic, with the imports a topic needs at the top of its card and the
session that introduced each row at its edge.
Allowed at the exam, as a copy saved on your laptop.Download it as a PDF
before the exam, since the site cannot be opened during it. Ctrl+P also works: the menu
and the page furniture drop out when printing.
A notebook shows the value of the last line of a cell. Anything else you want to see, you have to print.
print(value, other)
Show one or more values. Separate several with commas.Session 1
# comment
Everything after # on a line is a note for humans, not code.Session 1
value
The last line of a cell is displayed automatically, with no print.Session 1
Numbers
+ · - · * · /
The usual arithmetic. Division always gives a float.Session 1
x ** n
To the power of. Compound growth, and square roots as ** 0.5.Session 1
round(number, ndigits)
Round to a number of decimals.Session 1
abs(number)
Distance from zero, sign discarded.Session 2
x // n · x % n
Whole division and the remainder: 7 // 5 is 1 and 7 % 5 is 2. They turn a position counted along a grid's rows into its row and its column.Session 11
Text
"text"
A string. Single or double quotes, as long as they match.Session 1
"AA" + "PL"
Glue strings together. Both sides must be strings.Session 1
f"{value}"
An f-string: put a value inside text.Session 1
f"{r:.2%}"
As a percentage with 2 decimals. The recipe for returns.Session 1
f"{x:.2f}"
As a plain number with 2 decimals. The recipe for prices.Session 1
f"{name:<6}" · f"{x:>7.1%}"
A field width: pad left-aligned text, or right-align a number, so a column of output lines up.Session 2
text.upper() · text.lower()
A copy of the string in one case. Tickers arrive in every mixture of both.Session 1
text.strip()
A copy with the spaces at each end removed. The first thing you do to text from a file.Session 2
text.count(part)
How many times a piece of text appears inside a string.Session 1
text.startswith(part) · text.endswith(part)
True when the text begins, or ends, with that piece. [c for c in columns if c.startswith("vol_")] keeps the columns whose names begin with vol_.Session 3
True and false
== · !=
Equal, and not equal. Two equals signs, because one assigns.Session 1
< · > · <= · >=
The usual comparisons. The answer is True or False.Session 1
and · or · not
Combine conditions. In pandas the same job is done by &, | and ~.Session 2
Types
type(value)
What kind of thing this is.Session 1
int(x) · float(x) · str(x)
Convert between whole number, decimal and text.Session 1
None
The absence of a value. What a function returns when it has no return.Session 1
Built-in functions
Always available, no import needed.
len(items)
How many items.Session 1
sum(numbers)
Add them all up.Session 1
min(values) · max(values)
Smallest and largest.Session 1
sorted(values)
A new list, in order. The original is untouched.Session 2
range(start, stop)
Whole numbers from start up to but NOT including stop.Session 2
enumerate(items)
The position and the value together, so you need no counter of your own.Session 2
zip(a, b)
Walk two lists in step, with no indexing. Stops at the shorter one.Session 2
set(items)
Each distinct item once, in no fixed order. len(set(items)) counts the distinct items.Session 9
help(thing)
Print the documentation. Works offline.Session 2
dir(thing)
List everything an object can do. Useful when you half-remember a name.Session 3
Lists
An ordered box of values. Counting starts at 0.
[a, b, c]
Make a list.Session 1
items[i]
The item at that position. [0] is first.Session 1
items[-1]
Counting from the end. [-1] is last.Session 1
items[a:b]
A slice: from a up to but NOT including b.Session 1
items[:n] · items[n:]
Everything before n, everything from n.Session 1
items.append(x)
Add to the end. Changes the list itself.Session 1
items.index(x)
The position of the first x.Session 1
[]
An empty list, ready for a loop to fill.Session 1
[x for x in items if condition]
A list built in one line: every item for which the condition holds. [c for c in frame.columns if c != "price"] is every column but the target. Without the if it keeps them all, and x * 2 in place of the first x changes each one.Session 3
Loops
Doing something once per item.
for item in items:
Run the indented body once for every item.Session 2
for i in range(n):
Loop over positions rather than values.Session 2
for i in range(1, len(items)):
Positions 1 onwards: how you reach item i and i-1 together.Session 2
total += x
Add to a variable and store it back. Same as total = total + x.Session 2
while condition:
Repeat for as long as the condition holds. Something inside must change it.Session 2
Making choices
if condition:
Run the block only when the condition is True.Session 2
elif condition: · else:
Further cases, checked in order, and a catch-all.Session 2
Your own functions
def name(parameters):
Create a function. The indented body is what it does.Session 2
return value
Hand a value back and stop there. Without it you get None.Session 2
def name(x, rate=0.05):
A default: used when the caller does not pass it.Session 2
"""What it does."""
A docstring, the first line of the body. Editors show it on hover, and help() prints it.Session 2
Dictionaries
Values stored by name instead of by position.
{"AAPL": 0.36, "KO": 0.12}
Make one. Keys are usually strings.Session 2
data[key]
Read the value under that key.Session 2
data[key] = value
Add a new key, or overwrite an existing one.Session 2
for key in data:
Loop over the keys. Read each value with data[key].Session 2
data.keys() · data.values()
The names on their own, or the numbers on their own. max, min, sum and for all accept them.Session 2
for key, value in data.items():
The key and the value together, so you need no lookup.Session 2
{}
An empty dictionary, ready for a loop to fill. The counterpart of [].Session 2
min(data, key=data.get)
The KEY with the smallest value, not the value itself. max works the same way.Session 5
sorted(data, key=data.get)
All the keys, ordered by their values. Largest first with reverse=True.Session 5
{key: round(value, 3) for key, value in data.items()}
A dictionary built in one line, one entry per item: here the same keys with every value rounded.Session 3
name(**settings)
Call a function, or create a model, with the keys of a dictionary as the argument names: XGBRegressor(**{"max_depth": 3}) is XGBRegressor(max_depth=3). One dictionary of settings then reaches every fit that needs it.Session 12
Packages
import numpy as np
Bring a package in under a short name. np, pd and plt are conventions everybody uses.Session 3
%pip install pandas
Install a package. Once per machine, in a notebook cell.Session 3
NumPy arrays
A list that does arithmetic on every element at once. Indexing and slicing work exactly as they do for lists.
import numpy as np
On this card: values is a one-dimensional array, mask an array of True and False.
np.array(list)
Turn a list of numbers into an array.Session 3
values * 2
Arithmetic applies to every element. No loop.Session 3
a - b
Two arrays, element by element.Session 3
(v[1:] - v[:-1]) / v[:-1]
Every daily return in one line: each day against the day before.Session 3
values.mean()
The average.Session 3
values.std()
The standard deviation. This is what volatility is.Session 3
values.min() · values.max() · values.sum()
The usual summaries.Session 3
np.sqrt(x)
Square root. np.sqrt(252) annualises a daily volatility.Session 3
values > 0
A question asked of every element: True or False for each.Session 3
(values > 0).sum()
Count the Trues, because True counts as 1.Session 3
(values > 0).mean()
The share that are True.Session 3
values[values > 0]
Keep only the elements where the condition holds.Session 3
values.shape
How big it is. For a 1-D array it prints as (n,).Session 3
values.argmax() · values.argmin()
The POSITION of the largest or smallest element. Feed it to an index to get the label.Session 5
(a == b).all()
True only when every element of a comparison is True: the vectorised way to check that two arrays or columns agree everywhere.Session 8
np.allclose(a, b) · np.isclose(a, b)
Equal up to rounding error: one True or False for the whole array, or one per element. Two computations of the same number rarely agree to the last decimal, so == is the wrong test for floats.Session 3
float(values.mean())
A plain Python number from a NumPy one. From NumPy 2 on, a NumPy number stored in a list or a dictionary prints as np.float64(0.2); after float() it prints as 0.2.Session 5
np.diff(values)
The change from each element to the next, so n values give n - 1 changes. (np.diff(values) < 0).all() is True when the values fall at every step.Session 12
np.where(mask, a, b)
An array that takes a where the mask is True and b elsewhere. Keeps one group's values and blanks the rest before argmin or argmax.Session 8
np.zeros(n, dtype=int)
An array of n zeros, as integers; without dtype, as floats. A prediction of 0 on every row is np.zeros(len(test), dtype=int).Session 8
np.arange(start, stop, step)
Values from start up to but not including stop, a fixed step apart. np.linspace fixes the count instead of the step.Session 8
np.logspace(0, 4, 5)
Values spaced by a constant factor: 1, 10, 100, 1000, 10000. The right shape for a grid of alphas.Session 6
np.unique(values)
The distinct values, sorted. A cut on a column can only fall between two neighbouring ones.Session 11
np.quantile(values, np.linspace(0.02, 0.98, 60))
Sixty cut points spread evenly through the data, from the 2nd to the 98th percentile. s.quantile does the same for a Series.Session 11
np.asarray(values, dtype=float)
An array of floats from a list, an array or a Series, so one function accepts all three.Session 11
np.mean(arrays, axis=0)
Average several arrays of the same length, position by position: many trees' forecasts made into one.Session 11
values.cumsum(axis=0)
Running totals down the rows of a 2-D array. Divide row n by n + 1 for the running average of the first rows: how a forest's error moves as trees are added.Session 11
np.random.default_rng(0)
A random number generator with a fixed seed, so the same code gives the same numbers every time.Session 4
rng.normal(0, 1, size=n)
n draws from a normal distribution with mean 0 and standard deviation 1, from rng = np.random.default_rng(0). As a column of a table: pure noise.Session 4
n whole numbers from 0 up to but not including 2, a coin for each row; and the values given, in a random order. Shuffling one column of the test rows is how permutation importance breaks it.Session 12
NumPy matrices
A matrix is a 2-D array: rows first, then columns.
import numpy as np
np.array([[1, 2], [3, 4]])
A matrix, written as a list of rows.Session 3
X.shape
Rows and columns.Session 3
X[i, j] · X[i, :] · X[:, j]
One element, a whole row, a whole column. The colon means all.Session 3
X.T
The transpose: rows become columns.Session 3
X.sum(axis=1)
Sum along each row, one total per row; axis=0 sums down each column instead. The softmax divides every row of scores by its own total, so each row of probabilities adds to one.Session 9
A @ B
Matrix multiplication. For two vectors this is the dot product.Session 3
np.ones(n)
A vector of ones. The intercept column of a regression.Session 3
np.column_stack([a, b])
Glue 1-D arrays together as the columns of a matrix.Session 3
np.linalg.inv(A)
The inverse: the matrix that undoes A. Fails if a column is redundant.Session 3
np.linalg.solve(A, b)
Solve A x = b. Safer and faster than inverting. OLS in one line: solve(X.T @ X, X.T @ y).Session 3
pandas: one column
A Series is values with a label on each one: a list and a dictionary at once.
import pandas as pd
On this card: s is a Series.
pd.Series(a_dict)
Build one from a dictionary. The keys become the labels.Session 3
s["AAPL"]
Look a value up by its label.Session 3
s * 2
Maths applies to every value, and the labels come along.Session 3
s.mean() · s.std() · s.min() · s.max()
A Series summarises itself, like an array.Session 3
s.median()
The middle value. Compare it with the mean: a gap between them means a skewed tail.Session 4
s.sort_values(ascending=False)
In order. Leave ascending out for smallest first.Session 3
s.value_counts(normalize=True)
The share of each distinct value, largest first; leave normalize out for the counts. The largest share is what the majority rule scores.Session 9
s.dropna()
Throw away the missing values.Session 3
s.unique()
The distinct values, in the order they first appear. sorted() puts them in order.Session 3
s.nunique()
How many distinct values there are. On the index of a bootstrap sample: how many different days it drew.Session 11
s.sort_index()
Order by the labels instead of the values. After value_counts, the counts in order of the value counted.Session 3
s.isin(values)
True where the value appears in values. frame.index.isin(sample.index) marks the days a sample drew, and ~ the days it left out.Session 11
s.value_counts().reindex(index, fill_value=0)
A count for every label in index, with 0 for labels that never appeared: how many times a bootstrap sample drew each day.Session 11
s.abs()
The size of every value, sign dropped. abs() on its own does the same for one number.Session 4
s.idxmin() · s.idxmax()
The LABEL of the smallest or largest value, where argmin gives its position: on a Series of errors indexed by alpha, the alpha with the lowest error.Session 4
s.to_dict()
The Series as a plain dictionary, label to value. s.value_counts().to_dict() keeps the counts in a form any report can store.Session 3
pandas: a table
A DataFrame is several Series side by side, sharing one index.
import numpy as np
import pandas as pd
On this card: frame is a DataFrame, mask a column of True and False.
pd.read_csv(path, parse_dates=["date"])
Read a CSV. parse_dates turns a text column into real dates.Session 3
pd.DataFrame({name: values, ...})
Build a table from a dictionary of columns.Session 3
frame.head(n) · frame.tail(n)
The first or last n rows, 5 by default.Session 3
frame.shape
Rows and columns.Session 3
frame.dtypes
What each column holds. A price column that arrived as text is a silent bug.Session 3
frame.columns
The column names. list(frame.columns) for a plain list.Session 3
frame["close"]
One column, as a Series.Session 3
frame[["date", "close"]]
Several columns, as a smaller table. Note the two sets of brackets.Session 3
frame[frame["ticker"] == "AAPL"]
Keep only the rows where the condition is True.Session 3
(mask_a) & (mask_b) · (mask_a) | (mask_b)
And, or. Each condition needs its own brackets.Session 3
~mask
Flip a mask: every True becomes False. How you get the other half without writing the condition twice.Session 4
frame.copy()
Take your own copy before modifying a selection, so pandas knows what you meant.Session 3
frame["new"] = values
Assign to a name that does not exist yet, and the column appears.Session 3
s.pct_change()
The percentage change from each row to the next. The first is NaN.Session 3
frame.sort_values("col")
Sort the rows by a column.Session 3
frame.loc["2024-01-02":"2024-01-31"]
Rows by label. With a date index you can slice with dates, and both ends are included.Session 3
frame.iloc[0] · frame.iloc[-1]
Rows by position: the first row, the last row.Session 3
frame.iloc[[-1]]
The last row, kept as a one-row table by the second pair of brackets, so a model's predict accepts it.Session 12
frame.values · frame.to_numpy()
The plain numbers underneath, as a NumPy array, labels dropped.Session 3
frame.set_index("date")
Move a column into the index, so .loc can slice by it.Session 4
A random half of the rows, drawn without replacement, and the other half: the table without the rows whose labels are given.Session 11
frame.dropna()
Drop every row with a missing value in any column, such as the first rows a shift leaves empty. Compare the number of rows before and after.Session 10
frame.dropna(subset=columns)
Drop a row only where one of these columns is missing. Rows missing nothing but the target, such as the latest days, stay in.Session 12
frame.join(other)
Add the columns of another table with the same index, each row matched by its label. frame.join(pd.get_dummies(frame["weekday"], prefix="weekday", dtype=int)) adds the weekday columns in one step.Session 13
frame.idxmin(axis=1)
For every row, the name of the column holding the smallest value: on a table of errors with one column per model, the best model on each row.Session 12
for col in frame:
Looping over a table gives its column names, one at a time: this is how the columns from get_dummies are copied into another table.Session 10
pd.cut(s, [0, 0.2, 0.4, 1.0])
Sort every value into one of the ranges given, and return the range it fell in. Group by the result to summarise each range.Session 9
pd.cut(s, edges, labels=["low", "mid", "high"])
The same, with a name for each range instead of the range itself. This is how a number becomes a label with more than two values.Session 9
pd.cut(s, [-np.inf, -50, 50, np.inf])
Ranges open at both ends, so every value lands in one: down by more than 50, less than 50 either way, up by more than 50.Session 10
pandas: groups and shapes
import pandas as pd
On this card: frame is a DataFrame, s a Series, g a grouped table such as frame.groupby("ticker").
frame.groupby("ticker").size()
Split by a column, then count the rows in each group.Session 3
frame.groupby("ticker")["close"].mean()
Split, compute inside each group, put the answers back together.Session 3
frame.groupby("ticker")["close"].pct_change()
A return WITHIN each stock. Group first, or you take a return across two companies.Session 3
Group by the ranges pd.cut made, keeping only ranges that have rows: how many rows fell in each range, and the share of ones among them. This is how a calibration table is built.Session 9
frame.groupby("hour")["price"].shift(1)
The value one row earlier WITHIN each group. With one row per hour, grouped by the clock hour, that is the same hour the day before; .shift(2) reaches two days back. The first row of each group is NaN.Session 10
Write text next to a point, 5 points to the right of it and 4 above, so the text does not cover the marker.Session 12
ax.set_title(text, loc="left")
Say what the reader is looking at.Session 3
ax.set_ylabel(text) · ax.set_xlabel(text)
Label the axes, with units. "price (USD)", not "price".Session 3
ax.plot(x, y, label="AAPL") · ax.legend()
Name each line, then show the key. Needed as soon as there are two.Session 3
ax.set_ylim(a, b)
Fix the vertical range. For bars, always start at zero.Session 3
ax.axhline(y)
A horizontal reference line: zero on an error plot, or an average across bars. ax.axvline(x) is the vertical one.Session 4
ax.plot(x, y, drawstyle="steps-post")
A staircase: each value held flat until the next one. How a forecast that takes only a few values looks over time.Session 11
ax.set_xscale("log")
A logarithmic axis, so factors of ten are evenly spaced. For a validation curve over alpha or C.Session 6
fig.tight_layout()
Shrink the gaps so the titles and labels of several axes do not overlap. The line before plt.show().Session 3
plt.show()
Display the figure. The last line of a plotting cell.Session 3
plt.figure(figsize=(11, 3))
A figure with no axes of your own. plot_tree draws on the current figure, so make one of the right size first.Session 11
fig.savefig("name.png", dpi=200)
Save it to a file. Use .pdf for something that stays sharp at any size.Session 3
Features and a target
One row per observation, one column per feature, and one column you are trying to predict. Every model assumes it.
On this card: s is a Series, X the feature table.
s.shift(-1)
Move a column UP, so tomorrow's value sits on today's row. This is how a target is built.Session 4
s.rolling(20).std()
A feature over the last twenty rows. It only ever looks backwards, which is what makes it legal.Session 4
(s > 0).astype(int)
Turn a number into a 0/1 label, which turns a regression into a classification.Session 4
s.corr(other)
How strongly two columns move together, between -1 and +1. A feature that correlates almost perfectly with the target is a leak, not a discovery.Session 4
s.diff()
The difference between each row and the one before it. On logged prices this is the log return.Session 4
frame["x"] - frame.groupby("hour")["x"].shift(1)
A change as a feature: this hour's value minus the same hour the day before, such as tomorrow's wind forecast minus today's. Often more telling than the level itself.Session 10
frame["change"].abs()
The size of a change, sign dropped. A logistic regression cannot build this shape from the change itself, so a big move either way needs the column.Session 10
n, p = X.shape
The two numbers that describe a learning problem: observations, and feature columns. The target is not a feature.Session 4
Missing values
Real tables have holes in them. What you do about the holes changes your answer, so it is a decision rather than a formality.
import pandas as pd
pd.date_range(start, end, freq="D")
Every calendar day between two dates.Session 4
frame.reindex(index)
Force a table onto different row labels. Rows that did not exist arrive empty.Session 4
frame.isna().sum()
Count the missing values in each column.Session 4
s.ffill()
Carry the last known value forward. It only looks backwards, so a forecast may use it.Session 4
s.fillna(value)
Fill with something you choose. Filling with the column mean shrinks the variance, so be deliberate.Session 4
s.interpolate()
Draw a straight line across the gap. It uses the value AFTER the gap, so never in a forecast.Session 4
s.isna().astype(int)
Record that a value was missing. Sometimes that fact is itself the signal.Session 4
Extremes, and changing the scale
import numpy as np
import pandas as pd
s.quantile([0.25, 0.75])
The percentiles. Q1 and Q3 are the edges of the middle half of the data.Session 4
(s < q1 - 1.5 * iqr) | (s > q3 + 1.5 * iqr)
The IQR rule, with iqr = q3 - q1: flag anything outside Q1 - 1.5 IQR to Q3 + 1.5 IQR. On returns it flags far more than it should.Session 4
np.log(values)
The natural log. Log returns add up over time, and a log tames a long right tail.Session 4
np.exp(model.predict(X))
A model fitted to np.log(price) forecasts the log of the price; np.exp turns the forecast back into kroner. np.exp of the difference between two such forecasts is the factor between them: 1.147 is 14.7 percent more.Session 7
(s - s.mean()) / s.std()
Standardise: mean 0, standard deviation 1. Compute those two numbers on the training rows only.Session 4
One 0/1 column per category. This is how text gets into a feature table.Session 4
pd.get_dummies(s, prefix="weekday", dtype=int)
One 0/1 column per value of a single column, named weekday_0, weekday_1 and so on. A logistic regression then gives each weekday its own level, where the weekday as one number could only give it one slope across the week.Session 10
Scoring a prediction of a number
Choose the metric before you see the answer, and always report it beside a baseline.
import numpy as np
On this card: y is what happened and p the prediction, two arrays of the same length.
np.abs(y - p).mean()
MAE, mean absolute error: the average size of a mistake, in the units of the target.Session 4
((y - p) ** 2).mean()
MSE, mean squared error: one large miss dominates it, which is sometimes exactly what you want.Session 4
np.sqrt(((y - p) ** 2).mean())
RMSE, the root of the MSE, back in the units of the target. The one to report.Session 4
Accuracy: the share called correctly. Meaningless when one class is rare.Session 4
tp / (tp + fp)
Precision: of the times the model said yes, how often was it right? Low precision means false alarms.Session 4
tp / (tp + fn)
Recall: of the times it really happened, how many were caught? Low recall means misses.Session 4
2 * precision * recall / (precision + recall)
F1: the harmonic mean of the two, which sits close to the smaller one.Session 4
cost_miss * fn + cost_false * fp
What a set of decisions costs when the two mistakes are not worth the same. Sweep the threshold and keep the cheapest, on training rows only.Session 9
cost_false / (cost_false + cost_miss)
The cheapest threshold, written down rather than swept for. It only works on probabilities that are calibrated.Session 9
Training rows and test rows
A score computed on the rows a model learned from is not evidence.
import numpy as np
from sklearn.model_selection import train_test_split
frame.loc[:"2022-12-31"]
Split by date, never at random: the rows up to a date for training, frame.loc["2023-01-01":] for testing. A shuffled time series lets a model train on the future.Session 4
Shuffle the rows and cut them in two. Right for a cross-section where rows are separate cases, wrong for a time series. stratify keeps a label's share equal in both halves, which matters when it is rare.Session 9
np.polyfit(x, y, degree)
Fit a polynomial of that degree. Raising the degree always lowers training error.Session 4
np.polyval(coef, x)
Evaluate a fitted polynomial, so you can score it on rows it never saw.Session 4
Fitting a model
Four lines: import, create, fit, predict. Every model in scikit-learn takes them, so the only thing that changes later is the first one.
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
On this card: X is the feature table, always with two brackets, y the target column, model a fitted model, folds a TimeSeriesSplit.
frame[["col"]]
A LIST of column names, so the result stays a table. X always has to be two-dimensional.Session 5
model = LinearRegression()
Create the model. It installs as scikit-learn and imports as sklearn.Session 5
model.fit(X, y)
Read the training rows and compute the coefficients. Features first, target second.Session 5
model.predict(X)
One prediction per row of X, in the same order, as a plain NumPy array.Session 5
model.intercept_ · model.coef_
What the fit found. A trailing underscore means it came from the data rather than from you.Session 5
mean_squared_error(y, p)
The truth goes first. Take np.sqrt of it to get back into the units of the target.Session 5
Choosing between models
Rows used to compare candidates can no longer give the winner an honest score. That is what the folds are for.
from sklearn.model_selection import TimeSeriesSplit, KFold, cross_val_score
On this card: X is the feature table, always with two brackets, y the target column, model a fitted model, folds a TimeSeriesSplit.
folds = TimeSeriesSplit(n_splits=5)
Folds that move forward in time, so every scored block comes after the rows fitted on.Session 5
TimeSeriesSplit(n_splits=5, max_train_size=500)
A rolling window instead of an expanding one: the oldest rows drop out.Session 5
TimeSeriesSplit(n_splits=5, gap=20)
Drop the rows whose target reaches into the block about to be scored.Session 5
KFold(n_splits=5, shuffle=True, random_state=0)
The textbook folds. They assume the rows are interchangeable, which a time series is not.Session 5
for fit_rows, score_rows in folds.split(X):
The row positions in each fold, as pairs. cross_val_score loops over this for you.Session 5
cross_val_score(model, X, y, cv=folds, scoring="neg_root_mean_squared_error")
Fit and score once per fold. Errors come back negative, because every score is reported so that larger is better; put a minus in front.Session 5
Penalised regression
Linear regression with a charge on the size of the coefficients, the scaling that charge needs, and a search over how strong it should be.
from sklearn.linear_model import Ridge, Lasso, ElasticNet
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
On this card: X is the feature table, always with two brackets, y the target column, model a fitted model, folds a TimeSeriesSplit. scaler is a fitted StandardScaler, pipe a fitted Pipeline, search a fitted GridSearchCV.
Ridge(alpha=1000)
Linear regression plus alpha times the sum of squared slopes. alpha=0 is OLS; larger alpha pulls every slope towards zero.Session 6
Lasso(alpha=0.1)
The same with a charge on the sum of absolute slopes. Sets some coefficients to exactly zero.Session 6
ElasticNet(alpha=0.1, l1_ratio=0.5)
Both penalties at once. l1_ratio=1 is the lasso, l1_ratio=0 is ridge.Session 6
Lasso(alpha=0.001, max_iter=10000)
More passes for the solver. Raise it when the ConvergenceWarning appears; standardise first, which is the usual cause.Session 6
scaler = StandardScaler().fit(X_train)
Learn each column's mean and standard deviation from the TRAINING rows. The penalty charges by coefficient size, and size depends on units.Session 6
scaler.transform(X)
Subtract the learned mean and divide by the learned standard deviation. Returns a NumPy array; use it on the test rows too, with the training numbers.Session 6
scaler.mean_ · scaler.scale_
The numbers the scaler learned, one per column, in column order.Session 6
Two steps as one model: .fit scales then fits, .predict scales with the fitted scaler then predicts. Cross-validation refits the scaler inside every fold.Session 6
pipe.named_steps["ridge"].coef_
Reach the fitted object inside a step. The pipeline itself has no coef_.Session 6
Fit every value on every fold, keep the best mean score, and refit it on all the training rows. Always pass cv; the default ignores time order.Session 6
Run the search; then the winning setting, and its mean score over the folds.Session 6
search.best_estimator_ · search.predict(X)
The winning pipeline, refitted on every training row, and predictions from it. Open the test rows once, here.Session 6
pd.DataFrame(search.cv_results_)
The whole grid as a table: every value tried, its mean score over the folds, the spread, and its rank.Session 6
model.get_params()
Every argument of an object and its current value. help(Ridge) prints the documentation.Session 6
Classification
A label as the target, logistic regression, the probabilities it gives, and the scores that judge a classifier.
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.calibration import CalibratedClassifierCV
from sklearn.model_selection import cross_val_score, TimeSeriesSplit, GridSearchCV
from sklearn.metrics import (accuracy_score, confusion_matrix, precision_score,
recall_score, roc_curve, roc_auc_score, log_loss, f1_score,
average_precision_score, precision_recall_curve, brier_score_loss,
classification_report)
On this card: X is the feature table, y the label column of ones and zeros, model a fitted classifier, p the probability column model.predict_proba(X)[:, 1], predicted the 0/1 predictions.
(a > b).astype(int)
A label from a comparison: True and False become 1 and 0. Its mean is the share of ones, and predicting the majority class everywhere is the accuracy a model has to beat.Session 8
The classifier, shown with its defaults: a ridge penalty of strength 1/C, solved in at most 100 steps. Create it, then .fit(X, y), as for every model.Session 8
model.intercept_ · model.coef_
The fitted numbers, on the log-odds scale. coef_ has one row per boundary between classes, so coef_[0, 0] is the first slope.Session 8
-model.intercept_[0] / model.coef_[0, 0]
With one column: the value of the column at which the probability is one half, where .predict() switches from 1 to 0.Session 8
np.exp(model.coef_)
The odds factor: one unit more of a column multiplies the odds of a 1 by this. A coefficient adds to the log-odds; its exponential multiplies the odds.Session 8
model.classes_
The classes, in the order predict_proba uses for its columns.Session 8
model.predict_proba(X)
One row per observation, one column per class, each row summing to one. [:, 1] is the probability of a 1.Session 8
model.predict(X)
The 0/1 predictions: 1 where the probability of a 1 is at least one half.Session 8
(p >= threshold).astype(int)
Predictions at any other threshold, from the probability column. The model is unchanged; only the comparison changed.Session 8
model.n_iter_
How many steps the solver took. Log-loss has no formula for the coefficients, so .fit() improves them step by step; raise max_iter when the ConvergenceWarning appears, or standardise first.Session 8
log_loss(y, p)
The objective, averaged over the rows: minus the log of the probability given to what happened. A confident wrong probability costs the most.Session 8
accuracy_score(y, predicted)
The share of predictions that match the label. True labels first, predictions second, like every metric. Compare it with the majority rule.Session 8
confusion_matrix(y, predicted)
A 2 by 2 array: rows are what happened, columns are the prediction, both in the order of classes_. Top left true negatives, bottom right true positives.Session 8
precision_score(y, predicted)
The share of predicted 1s that were 1. Falls when the threshold falls.Session 8
recall_score(y, predicted)
The share of real 1s that were predicted. Falls when the threshold rises.Session 8
roc_curve(y, p)
Returns fpr, tpr, thresholds: the false positive rate and the recall at every threshold where a point moves. Joined up, the ROC curve.Session 8
roc_auc_score(y, p)
The area under the ROC curve: 0.5 for a random ranking, 1 for a perfect one, no threshold involved. Pass the probabilities, never the 0/1 predictions.Session 8
cross_val_score(model, X, y, cv=folds, scoring="roc_auc")
One score per fold; larger is better, so no minus sign. Other strings: "accuracy", "neg_log_loss", "precision", "recall".Session 8
LogisticRegression(C=0.01)
The strength of the penalty, upside down: C = 1/alpha, so a small C is a strong penalty. Standardise in a pipeline first and choose C on a grid in factors of ten.Session 8
{"logit__C": [0.0001, 0.001, 0.01, 0.1, 1, 10]}
A grid for GridSearchCV: the step name, two underscores, the argument. Then GridSearchCV(pipe, grid, cv=folds, scoring="roc_auc").fit(X, y), and best_params_, best_score_, predict_proba as before.Session 8
The lasso's penalty on a classifier, which sets coefficients to exactly zero. It needs a solver that can handle absolute values; the default lbfgs cannot. scikit-learn 1.8 and later retire penalty: there it is LogisticRegression(l1_ratio=1, solver="liblinear", C=0.01).Session 8
confusion_matrix(y, predicted).ravel()
The 2 by 2 array flattened into tn, fp, fn, tp, in reading order. The four counts every other score is built from.Session 9
LogisticRegression(class_weight="balanced")
Count each class in inverse proportion to its size while fitting. Raises recall on a rare class and lowers accuracy, leaves the AUC alone, and breaks the probabilities.Session 9
average_precision_score(y, p)
The whole precision-recall curve in one number. Its no-information line is the share of ones, not 0.5, so report the two together. More informative than AUC when the class is rare.Session 9
precision_recall_curve(y, p)
Returns precision, recall and the thresholds, for drawing the curve or reading the precision at a recall you care about.Session 9
brier_score_loss(y, p)
The mean squared error of the probabilities, with what happened as 1 or 0. Smaller is better. It separates two models that the AUC cannot tell apart.Session 9
Wrap a model and learn a second, small model mapping its numbers onto probabilities, on folds. It repairs a distorted level; it cannot repair a base rate that has changed since fitting.Session 9
Precision, recall and F1 for every class, with the macro and weighted averages underneath. labels= sets the order.Session 9
f1_score(y, predicted, average="macro")
One F1 per class, averaged equally. "weighted" averages by class size instead, and None returns the scores themselves. Read macro when the class you care about is the small one.Session 9
confusion_matrix(y, predicted, labels=order)
The matrix with the rows and columns in an order you choose rather than alphabetically. Without it the middle row of a three-class matrix is rarely the class you expect.Session 9
model.decision_function(X)
The raw score per class, before the exponential and the division that turn scores into probabilities.Session 9
np.exp(scores) / np.exp(scores).sum()
The softmax: make every score positive, then divide by the total so they add to one. With two classes it is the sigmoid.Session 9
A bet on the direction, in money: long where the probability of a rise is at least t, short where it is at most 1 - t, every trade paying a cost. The best t rises with the cost; choose it on the training rows.Session 10
k-nearest neighbours
A classifier with no coefficients: it stores the training rows and lets the nearest ones vote.
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
On this card: X is the feature table, always with two brackets, y the target column, model a fitted model, folds a TimeSeriesSplit.
KNeighborsClassifier(n_neighbors=k)
Classify a row by a vote among the k training rows closest to it. Works for any number of classes with no change.Session 9
The scaler is not optional here: distance is dominated by whichever column has the largest spread, so an unscaled model reads only that one.Session 9
k, as the dial
A small k fits the training rows and little else; k of 1 scores a perfect training AUC. A large k is a simple model, the way a small C was.Session 9
{"knn__n_neighbors": [1, 5, 15, 51, 151, 301]}
A grid for GridSearchCV. k cannot exceed the SMALLEST fold, not the training size; above that the search returns nan and warns.Session 9
model.predict_proba(X)
The share of the k neighbours in each class, so the probabilities are multiples of 1/k. A small k gives coarse ones.Session 9
["wind_change", "weekday", "hour"]
k-NN can take a category as one plain number: once the columns are scaled, the nearest rows to a Monday at 18:00 are Mondays near 18:00. A logistic regression needs the weekday as columns instead.Session 10
Trees and forests
Models that ask questions of one column at a time instead of fitting slopes: one tree, many trees averaged, and the settings that decide how big they grow.
import time
from sklearn.tree import DecisionTreeRegressor, DecisionTreeClassifier, plot_tree
from sklearn.ensemble import BaggingRegressor, RandomForestRegressor, RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
On this card: X is the feature table, y the target, columns the list of feature names, tree a fitted tree, forest a fitted forest, model the model a search tunes, grid a dictionary of settings to try, folds a TimeSeriesSplit.
A tree for a number. Each question compares one column with a cut, and each leaf forecasts the mean of its training rows. max_depth caps the questions on any path; without it the tree splits until it fits the training rows almost exactly.Session 11
DecisionTreeRegressor(min_samples_leaf=100)
Another way to keep a tree small: no leaf may hold fewer training rows than this. Choose it, or max_depth, on the folds.Session 11
DecisionTreeRegressor(ccp_alpha=0.02)
Grow the tree, then prune it: a question stays only if it lowers the training error by at least ccp_alpha for each leaf it adds. The default, 0, keeps every question.Session 11
A tree for a label. The same cuts, chosen by Gini, 2p(1 - p) for two classes, which is twice the squared error of the 0s and 1s per row. Each leaf holds the share of each class, which predict_proba returns.Session 11
Draw a fitted tree with matplotlib: every box shows its question, the rows that reached it and its forecast. filled shades the boxes by their forecast, impurity=False drops the error line, and label="root" names the fields in the top box only.Session 11
For a classification tree: name the classes in the order of classes_, and show each box's shares instead of its counts.Session 11
tree.get_depth() · tree.get_n_leaves()
The questions on the longest path, and the number of leaves. A tree grown without a limit ends with about one leaf per training row.Session 11
tree.tree_.feature[0] · tree.tree_.threshold[0]
Inside a fitted tree: the column number and the cut of the first question, so columns[tree.tree_.feature[0]] is its name. The boxes are numbered depth first, and a leaf has feature -2.Session 11
tree.apply(X)
The number of the leaf each row lands in. Group any column by it to see what each leaf holds.Session 11
Bagging: grow a deep tree on each of 25 bootstrap samples of the training rows and average their forecasts, which cancels much of what one tree gets wrong by chance.Session 11
forest.estimators_
The fitted trees inside a bagging model or a forest, as a list, each a tree like any other. They were fitted on plain arrays, so give their predict X.to_numpy().Session 11
A forest: bagging in which each question chooses among a few columns drawn at random, here the square root of their number, so the trees differ. The default for a number, max_features=1.0, is plain bagging; RandomForestClassifier's default is already "sqrt".Session 11
Out-of-bag: each training row is forecast by the trees whose sample left it out, so the forest scores itself without folds. Fair on rows that are separate cases; too kind on days, whose neighbours were in the sample.Session 11
The forest for a label. forest.oob_decision_function_[:, 1] is each training row's out-of-bag probability of a 1.Session 11
RandomForestRegressor(n_jobs=-1)
Grow the trees on every core at once. GridSearchCV takes n_jobs too, and then fits the folds side by side instead.Session 11
{"max_depth": [1, 2, 3, 4, 6, 8]}
A grid for a model on its own, outside a pipeline: the argument name with no step name in front. GridSearchCV(DecisionTreeRegressor(random_state=0), grid, cv=folds, ...) as before.Session 11
Try n_iter combinations drawn at random from the grid instead of every one; best_params_ and best_score_ as for GridSearchCV. For grids too big to run in full.Session 11
len(grid["a"]) * len(grid["b"]) * 5 + 1
The fits a search will make: one per combination on each of five folds, and one refit on all the training rows. Time one fit and multiply before you press run.Session 11
start = time.perf_counter()
Read a clock, in seconds. time.perf_counter() - start after a fit is how long it took.Session 11
Boosted trees
Small trees added one at a time, each fitted to what the trees before it still get wrong, the libraries that do this fast, and two ways to read which columns a model uses.
from sklearn.ensemble import (GradientBoostingRegressor, GradientBoostingClassifier,
HistGradientBoostingRegressor, HistGradientBoostingClassifier)
from sklearn.inspection import permutation_importance
from xgboost import XGBRegressor, XGBClassifier
from lightgbm import LGBMRegressor, LGBMClassifier, early_stopping
On this card: X and y are the rows a model is fitted on, X_stop and y_stop later rows held back to stop it, X_test and y_test the test rows, columns the list of feature names, boost a fitted boosting model, result what permutation_importance returns.
Boosting: start from the mean of y, then add n_estimators small trees one at a time, each fitted to the residuals of the forecast so far and shrunk by learning_rate. max_depth=1 makes every tree a stump; the default is 3. A smaller learning_rate needs more trees for the same fit.Session 12
The same for a label. The trees are added on the log-odds scale, each fitted to y minus the probability so far, and predict_proba turns the sum into probabilities.Session 12
list(boost.staged_predict(X))
The forecast after 1, 2, 3, ... trees, up to n_estimators. staged_predict hands them out one at a time, so list() collects them: [49] is the forecast after 50 trees.Session 12
Early stopping inside scikit-learn: hold back a random tenth of the training rows and stop once 10 more trees have not helped on them. boost.n_estimators_ is the number of trees it grew. A random tenth suits separate cases and is too kind on days.Session 12
scikit-learn's fast boosting: each column is first sorted into at most 255 bins, and cuts fall only between bins. max_iter is the number of trees. Above 10,000 rows it stops early on its own, on a random tenth of them, unless early_stopping=False. It has no feature_importances_.Session 12
XGBoost, with the methods every scikit-learn model has: fit, predict, and predict_proba on XGBClassifier. It cuts each column only between 256 bins and puts a light ridge penalty on the leaf values. Its own defaults are learning_rate=0.3 and max_depth=6, so set both.Session 12
Early stopping in XGBoost: score every new tree on the rows given to fit as eval_set, and stop 50 trees after the best score. n_estimators is then only a ceiling.Session 12
boost.fit(X, y, eval_set=[(X_stop, y_stop)], verbose=False)
The rows that stop it, as a list of (columns, target) pairs: rows later than those fitted on, never the test rows. verbose=False keeps it from printing the score of every tree.Session 12
boost.best_iteration · boost.best_score
Where the score on eval_set was best, counting the first tree as 0, and that score. predict then uses the first best_iteration + 1 trees.Session 12
The number of trees early stopping found, refitted on all the training rows, the block that chose it included. The refit takes no eval_set and no early_stopping_rounds.Session 13
boost.evals_result()["validation_0"]["rmse"]
XGBoost's record of the score after every tree, for each eval_set entry, under validation_0, validation_1, ... in the order given. Plot it against the number of trees to see the curve early stopping read.Session 12
reg_lambda is the ridge penalty on the leaf values. subsample and colsample_bytree give each tree a random share of the rows and of the columns, as a forest does. Settings to try after the depth and the number of trees, not before.Session 12
boost.predict(X, output_margin=True)
XGBClassifier: the sum of the trees before the sigmoid, on the log-odds scale. 1 / (1 + np.exp(-margin)) gives back predict_proba(X)[:, 1].Session 12
XGBRegressor(monotone_constraints={"area": 1})
Never let the forecast fall as a column rises, everything else unchanged: 1 for a column that may only raise the forecast, -1 for one that may only lower it, by its name in the table given to fit.Session 13
LightGBM. It grows each tree leaf by leaf, splitting whichever leaf lowers the error most, so num_leaves sets the size of a tree: 2 makes a stump, and the default is 31. verbose=-1 silences its messages.Session 12
boost.fit(X, y, eval_set=[(X_stop, y_stop)], eval_metric="auc", callbacks=[early_stopping(50, verbose=False)])
Early stopping in LightGBM: a callback passed to fit rather than a setting of the model. boost.best_iteration_ counts the first tree as 1, where XGBoost's best_iteration counts it as 0.Session 12
boost.best_score_["valid_0"]["l2"] ** 0.5
LightGBM's best score, by data set and metric. For a number the metric is the squared error, l2, so its square root is the RMSE.Session 12
How much the model used each column, one number per column in column order. scikit-learn's trees sum each column's fall in the error and scale the total to one, XGBoost averages the fall per cut, and LightGBM counts the cuts. Measured on the training rows, so a column of noise can still collect many cuts.Session 12
LGBMClassifier(importance_type="gain")
Make LightGBM sum the fall in the loss on each column instead of counting its cuts. XGBoost takes importance_type too, where "weight" counts the cuts.Session 12
For any model: shuffle one column at a time on rows the model did not fit, and record how far the score falls, n_repeats times for each column. A column of noise falls by about zero.Session 12
result.importances_mean · result.importances_std
Per column, in column order: the mean fall in the score over the repeats, and its spread. pd.Series(result.importances_mean, index=columns) names them.Session 12
Reading error messages
The last line names the problem. Read it before you change anything: it is almost always telling you the truth.
NameError
A name Python has never seen. Usually a typo, or a cell you have not run yet.Session 1
TypeError
The right operation on the wrong kind of thing. Often text where a number belongs, or a placeholder ... left in a cell.Session 1
IndexError
A position that does not exist. A list of 5 stops at index 4.Session 1
KeyError
A dictionary key, or a column name, that is not there.Session 2
SyntaxError
Python could not read the line at all. Look for a missing bracket, quote or colon.Session 1
ZeroDivisionError
Dividing by zero. Often an empty list you thought had something in it.Session 1
ModuleNotFoundError
The package is not installed, or you are running a different Python.Session 3
FileNotFoundError
The path is wrong relative to where you are running from.Session 3
ValueError, on shapes
Two arrays whose dimensions do not fit. Check .shape on both.Session 3
LinAlgError
A matrix with no inverse: one column carries nothing the others do not. Perfect multicollinearity.Session 3
ValueError, on feature names
The columns given to predict are not the ones fit saw, in the same order. Select them with the same list both times.Session 11
ValueError, on early stopping
Must have at least 1 validation dataset: XGBoost was given early_stopping_rounds but no eval_set to score the trees on. Pass eval_set to fit, or leave early_stopping_rounds out of a refit.Session 12
ConvergenceWarning
Not an error: the solver ran out of steps before the coefficients settled. Raise max_iter, or standardise the columns first.Session 6