Build an ML Model with Pen and Paper
A gentle introduction to ML and the underlying math
Preface
Since this is the first in the series, it might be worth explaining the motivation behind it. However, if you want to dive right in, scroll to Overview.
First and foremost, I wanted to be exceptional at hardware and machine learning engineering. My MS EE advisor said that one way to validate my knowledge is by explaining it well to others. In addition, throughout my academic journey, I’ve experienced a lot of frustration with how content is organized. I struggle to find the connection between high-level math and engineering because they’re often taught separately and without context. Later, they branch out into almost entirely self-contained, convoluted methodologies without addressing the intricate nuances that connect them. If I ever need to look back, I want there to be a resource that fits my learning preferences.
The content is organized in a way that first showcases the overarching concept, then explains the relevant concepts in math and engineering—a top-down approach. While this is not intended to replace math or engineering lectures, it tries to spark curiosity and draw connections between concepts so that readers can build intuition before diving deeper on their own.
Overview
A deep learning model takes data as input and outputs a prediction: identifying the mushroom in a picture, picking the next word in a sentence, or estimating a house’s price.
To make a prediction, it needs two critical pieces:
- Input — your data (say, an image), represented as numbers
- Weights — learned numbers that determine how strongly different inputs affect the result
At its core, much of this computation is matrix multiplication, interspersed with nonlinear functions.
To make accurate predictions, however, the model needs to be trained. The process introduces a few more moving parts:
- Target — the correct prediction
- Loss function — measure of how far the model’s prediction is from the target
- Backpropagation — a method for calculating how much each weight contributed to the error
- Optimizer — an algorithm that uses those calculations to adjust the weights to reduce the error
Jeremy Howard from Fastai said that people don’t need to be good at math to understand these concepts, but without an understanding of linear algebra, calculus, and some probability, training a model feels like laying bricks without a blueprint, a routine procedure that might build a castle but usually ends in a shed.
Linear Algebra & Calculus Review
The part of linear algebra that’s relevant to deep learning is matrix operations.
NOTE: GPUs became a critical part of deep learning because they pack thousands of small cores that run arithmetic in parallel, which is exactly what matrix multiplication needs.
Even though modern tools such as PyTorch and TensorFlow are relatively forgiving for engineers without a strong math background, it’s still worth reviewing the basics to build intuition.
Matrix Multiplication
The building block is the dot product of two vectors. The result is the sum of element-wise products between a horizontal (row) vector and a vertical (column) vector:
\[ \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1 \cdot 4 + 2 \cdot 5 + 3 \cdot 6 = 32 \]
Note what happened to the shape, two vectors of length 3 collapse into one scalar (a number). The vectors must be the same length, since every element needs a partner.
A vector has just one dimension, but real-world data is often multi-dimensional, and such data is represented as a matrix.
Matrix multiplication is like a dot product done many times. Each entry in the result is the dot product of a row from the left and a column from the right:
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1\cdot 5 + 2\cdot 7 & 1\cdot 6 + 2\cdot 8 \\ 3\cdot 5 + 4\cdot 7 & 3\cdot 6 + 4\cdot 8 \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix} \]
Matrix Addition
In addition, matrices are added element-wise, so each entry in the result is the sum of the two entries in the same position:
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1+5 & 2+6 \\ 3+7 & 4+8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \\ 10 & 12 \end{bmatrix} \]
Both matrices must have the same shape, and the result keeps that shape. Subtraction works identically, element by element.
Derivatives
The core tool from calculus is the derivative - the instantaneous rate of change. It’s “rise over run”, the difference in \(y\) divided by the difference in \(x\), measured as the run shrinks to nothing.
Take a vehicle whose position (y-axis) over time (x-axis) is: \[ f(t) = t^2 + 4 \] Its velocity at a given moment, say \(t = 4\), is the derivative \(f'(t)\). Remember high-school physics? \(\text{velocity} = \text{distance}/\text{time}\). The derivative is that same ratio, but measured over a vanishingly small step \(h\) (\(h < 0.00000{\dots}0001\)). Shrinking \(h\) turns the average velocity into the instantaneous velocity at that moment:
\[ f'(t) = \lim_{h \to 0} \frac{f(t+h) - f(t)}{h} \]
Plugging the equation into the formula, we get: \[ f'(t) = \lim_{h \to 0} \frac{(t+h)^2+4-(t^2+4)}{h} = 2t \]
NOTE: Derivatives can be written in a few ways: the Leibniz form \(\frac{d}{dx}f(x)\) and Lagrange (prime) form \(f'(x)\) mean exactly the same thing.
So the velocity at time \(t = 4\) is \(f'(4) = 2(4) = 8\). In practice you rarely take the limit by hand. The power rule (\(\frac{d}{dx}x^n = nx^{n-1}\)) gives the same answer instantly, \(t^2 \to 2t\), and the constant \(4 \to 0\). Alongside the power rule, a handful of other rules cover the common cases, such as products, quotients, and compositions.
This serves as a glimpse into the math that’s relevant to building a simple machine learning model. Additional resources with external links for more in-depth explanations can be found at the end.
Deep Learning - the Essence
You’ve probably seen versions of this network illustration passed around the internet as a symbol of how complex these algorithms are. But strip away the visual and it’s essentially just a long sequence of dot products. Each layer multiplies its inputs by a matrix of weights and passes the result on. Run an input all the way through and you get a prediction — that’s a single forward pass.
In mathematics, each neuron in this network boils down to a single formula:
\[ z = b + \sum_{i=1}^{m} x_i w_i \]
- \(x\) is the input vector.
- \(w\) is the weight vector, initialized with random values. Sometimes it’s also referred to as a neuron.
- \(i\) indexes each input, running from 1 to \(m\) (the total number of inputs).
- \(\sum\) represents the summation \(x_1 w_1 + x_2 w_2 + \dots + x_m w_m\).
- \(b\) is another arbitrary value called the bias. Like weights, it’s learned during training.
The presence of \(b\) might be confusing. It’s there because any weight multiplied by an input of 0 gives 0. Without a bias, an all-zero input would force the output to 0 no matter what the weights learn. The bias shifts the output so the neuron can still contribute to the prediction.
Since the weights start out arbitrary, the network outputs nonsense at first. Training fixes this over many iterations of backpropagation. The weights and bias get updated, and the output gradually converges toward the expected result.
Calculation by Hand
A good way to gain intuition is to train a model by hand, which consists of one forward pass and one round of backpropagation. The XOR logic gate neural network is a classic example for demonstrating how data flows through the pipeline.
Step 1 - Understand the Data
An XOR logic gate has two inputs, \(x_1\) and \(x_2\). Each can be 1 or 0 independently, and the output is also either 1 or 0.
Target — the XOR truth table:
| \(x_1\) | \(x_2\) | \(y\) - XOR Result |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
As the truth table shows, the XOR result is 1 when \(x_1\) and \(x_2\) differ, and 0 otherwise. The goal of the network is to predict the XOR Result from the input values of \(x_1\) and \(x_2\).
For this example, the network will be trained on the input \(x_1 = 1\) and \(x_2 = 0\).
Step 2 - Create Arbitrary Weights
Plotting the four inputs against their targets shows that no single straight line can separate the 1s from the 0s. That’s why the XOR network needs at least two sets of weights (neurons) to make a prediction.
\[ \text{hidden neuron 1} = \mathbf{w}_1 = \begin{bmatrix} 0.5 \\ 0.5 \end{bmatrix}, \quad \text{hidden neuron 2} = \mathbf{w}_2 = \begin{bmatrix} -0.5 \\ 0.5 \end{bmatrix} \]
\[ \text{hidden weights} = \mathbf{W}_1 = \begin{bmatrix} \mathbf{w}_1 & \mathbf{w}_2 \end{bmatrix}, \quad \text{hidden bias} = \mathbf{b}_1 = \begin{bmatrix} 0.2 & -0.4 \end{bmatrix} \]
The final prediction has to be a scalar (a single number), so we need one more weight vector to collapse the hidden layer’s output.
\[ \text{output weights} = \mathbf{W}_2 = \begin{bmatrix} 0.3 \\ -0.4 \end{bmatrix}, \quad \text{output bias} = \mathbf{b}_2 = 0.1 \]
Step 3 - Select Activation Function
This is often referred to as applying non-linearity. A dot product plus bias is linear, meaning it’s a straight line or flat plane. Real data is rarely that clean. An activation function bends the output so the network can curve where it needs to. There are many activation functions to choose from. In this case, sigmoid is a natural fit for probability:
[TODO: introduce probability]
\(\hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}}\)
Sigmoid squashes any input into the range 0 to 1, which is useful for predicting the probability \(\hat{y}\).
Step 4 - Run Forward Pass for Input (1, 0)
The forward pass should feel familiar by now. For the hidden layer output, the formula is: \(\mathbf{a}_1 = \sigma([x_1 \quad x_2] \cdot \mathbf{W}_1 + \mathbf{b}_1)\).
\[ \mathbf{z}_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 0.5 & -0.5 \\ 0.5 & 0.5 \end{bmatrix} + \begin{bmatrix} 0.2 & -0.4 \end{bmatrix} = \begin{bmatrix} 0.7 & -0.9 \end{bmatrix} \]
\[ \mathbf{a}_1 = \sigma(\mathbf{z}_1) = \begin{bmatrix} \frac{1}{1 + e^{-0.7}} & \frac{1}{1 + e^{0.9}} \end{bmatrix} = \begin{bmatrix} 0.66818 & 0.28905 \end{bmatrix} \]
Similarly, for the final prediction output, the formula is: \(\hat{y} = \sigma(\mathbf{a}_1 \cdot \mathbf{W}_2 + \mathbf{b}_2)\).
\[ \mathbf{z}_2 = \begin{bmatrix} 0.66818 & 0.28905 \end{bmatrix} \begin{bmatrix} 0.3 \\ -0.4 \end{bmatrix} + 0.1 = 0.18483 \]
\[ \hat{y} = \sigma(\mathbf{z}_2) = \frac{1}{1 + e^{-0.18483}} = 0.54607 \]
Step 5 - Calculate Loss
Like the activation function, there are many loss functions to pick from, each measuring how far the prediction \(\hat{y}\) lands from the true label \(y\). Since sigmoid gives a probability and the result is binary, we can pick binary cross-entropy:
\[ L = -\frac{1}{n} \sum_{i=1}^{n} \left[ y_i \ln(\hat{y}_i) + (1 - y_i) \ln(1 - \hat{y}_i) \right] \]
Here \(i\) runs over all \(n\) training examples. The loss spikes when the model is confident and wrong, and drops near 0 when it’s confident and right.
\[ L = -\frac{1}{1} \left[ (1) \ln(0.54607) + (1 - 1) \ln(1 - 0.54607) \right] = 0.60501 \]
Chain Rule Review
We just pushed data from input to prediction. Now we need to work backwards to adjust the arbitrary weights we created, so the next prediction becomes more accurate.
This is where the derivative shines. The derivative of the loss with respect to a weight, \(\frac{\partial L}{\partial w}\), tells us how the loss shifts as we nudge that weight. Notice how it’s \(\partial\) instead of \(d\). This is a partial derivative — the same idea as a normal derivative, except it measures the slope for one weight while holding the rest fixed. A neural network can have thousands of weights (\(f(w_1, w_2 \dots w_n)\)), and we want to know how each one affects the model’s loss.
Collect all of those partial derivatives together and you get the gradient — the vector that tells the algorithm which direction to nudge every weight to bring the loss down.
Like the power rule mentioned earlier, there’s another shortcut worth introducing called the chain rule. It is used to take derivatives of nested functions:
\[ f(x) = g(h(l(x))) \quad \rightarrow \quad f'(x) = g'\big(h(l(x))\big) \cdot h'\big(l(x)\big) \cdot l'(x) \]
As we can see, it’s recursive, multiply the derivative of the outer function by the derivative of the inner one, and repeat until you run out of nesting. For loss over weight in deep learning, the chain looks like this:
\[ \frac{\partial L}{\partial w_i} = \underbrace{\frac{\partial L}{\partial \hat{y}}}_{\text{loss function}} \cdot \underbrace{\frac{\partial \hat{y}}{\partial z}}_{\text{activation function}} \cdot \underbrace{\frac{\partial z}{\partial w_i}}_{\text{dot product}} \]
Output Neuron Backpropagation
Since we’re working backwards, we start with how \(\mathbf{W}_2\) affects the loss. That means computing \(\frac{\partial L}{\partial \mathbf{W}_2}\), which can be expressed as:
\[ \frac{\partial L}{\partial \mathbf{W}_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial \mathbf{z}_2} \cdot \frac{\partial \mathbf{z}_2}{\partial \mathbf{W}_2} = L'( \sigma(\mathbf{a}_1\mathbf{W}_2 + \mathbf{b}_2)) \cdot \sigma'(\mathbf{a}_1\mathbf{W}_2 + \mathbf{b}_2) \cdot (\mathbf{a}_1\mathbf{W}_2 + \mathbf{b}_2 )' \]
After the terms cancel, the result becomes:
\[ \frac{\partial L}{\partial \mathbf{W}_2} = (\hat{y} - y) \cdot \mathbf{a}_1 = \begin{bmatrix} (0.54607 - 1) \cdot 0.66818 \\ (0.54607 - 1) \cdot 0.28905 \end{bmatrix} = \begin{bmatrix} \mathbf{-0.30331} \\ \mathbf{-0.13121} \end{bmatrix} \]
Now we calculate the partial derivative for the bias as well, since it’s also a parameter that gets updated. The process is the same, we just treat \(\mathbf{b}_2\) as the variable instead of \(\mathbf{W}_2\):
\[ \frac{\partial L}{\partial \mathbf{b}_2} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial \mathbf{z}_2} \cdot \frac{\partial \mathbf{z}_2}{\partial \mathbf{b}_2} = L'( \sigma(\mathbf{a}_1\mathbf{W}_2 + \mathbf{b}_2)) \cdot \sigma'(\mathbf{a}_1\mathbf{W}_2 + \mathbf{b}_2) \cdot (\mathbf{a}_1\mathbf{W}_2 + \mathbf{b}_2 )' \]
After solving the derivative and plugging in the constants (\(\frac{\partial \mathbf{z}_2}{\partial \mathbf{b}_2} = 1\)), we get: \(-0.45393 \times 1 = \mathbf{-0.45393}\)
Updating the Parameters
Remember the earlier velocity example? As the curve approaches the minimum, the lowest point on the parabola, the slope flattens to 0, indicating the vehicle is stationary. In deep learning, we want to step downhill the same way until the loss stops shrinking. This is gradient descent: \(w \leftarrow w - \eta \frac{\partial L}{\partial w}\). Here \(\eta\) is the learning rate, which sets the size of the step we take toward the minimum. A larger \(\eta\) means a bigger jump in the weights each training cycle. For this example, we use \(\eta = 0.1\) to update every parameter:
\[ \mathbf{W}_1 \leftarrow \begin{bmatrix} 0.5 & -0.5 \\ 0.5 & 0.5 \end{bmatrix} - 0.1 \begin{bmatrix} -0.03019 & 0.03731 \\ 0 & 0 \end{bmatrix} = \begin{bmatrix} 0.50302 & -0.50373 \\ 0.5 & 0.5 \end{bmatrix} \]
\[ \mathbf{W}_2 \leftarrow \begin{bmatrix} 0.3 \\ -0.4 \end{bmatrix} - 0.1 \begin{bmatrix} -0.30331 \\ -0.13121 \end{bmatrix} = \begin{bmatrix} 0.33033 \\ -0.38688 \end{bmatrix} \]
\[ \mathbf{b}_1 \leftarrow \begin{bmatrix} 0.2 & -0.4 \end{bmatrix} - 0.1 \begin{bmatrix} -0.03019 & 0.03731 \end{bmatrix} = \begin{bmatrix} 0.20302 & -0.40373 \end{bmatrix} \]
\[ \mathbf{b}_2 \leftarrow 0.1 - 0.1(-0.45393) = 0.14539 \]
Verify Effectiveness
One backward pass should have moved the prediction closer to the target. Running the same input \((1, 0)\) through the updated weights:
Hidden Layer: \[ \mathbf{z}_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 0.50302 & -0.50373 \\ 0.5 & 0.5 \end{bmatrix} + \begin{bmatrix} 0.20302 & -0.40373 \end{bmatrix} = \begin{bmatrix} 0.70604 & -0.90746 \end{bmatrix} \]
\[ \mathbf{a}_1 = \sigma(\mathbf{z}_1) = \begin{bmatrix} \frac{1}{1 + e^{-0.70604}} & \frac{1}{1 + e^{0.90746}} \end{bmatrix} = \begin{bmatrix} 0.66953 & 0.28752 \end{bmatrix} \]
Output Layer:
\[ \mathbf{z}_2 = \begin{bmatrix} 0.66953 & 0.28752 \end{bmatrix} \begin{bmatrix} 0.33033 \\ -0.38688 \end{bmatrix} + 0.14539 = 0.25532 \]
\[ \hat{y} = \sigma(\mathbf{z}_2) = \frac{1}{1 + e^{-0.25532}} = 0.56349 \]
The prediction moved from 0.54607 up to 0.56349, toward the target of 1. One backward pass indeed brought a small improvement to the final output.
That’s a single iteration for a single example. Real training repeats the whole loop for many epochs. Run it long enough and the four predictions settle onto 0, 1, 1, 0, showing the network has learned XOR.
Food for Thought
Everything above is the smallest possible version of deep learning. Scaling it up introduces new problems worth considering:
- Stochastic Gradient Descent — With a million data points, is there a more efficient way to train than averaging the gradient over every single example before each update?
- Mini-batches — Between one example and the full dataset sits a compromise: random batches of 32, 64, or 128 examples. Why do these sizes pair so well with GPUs?
- Overfitting — What goes wrong when a model learns its training data too well, and how would you even detect it?
- Dropout — Why does forcing the network to work short-handed make it generalize better?
- Early Stopping — When would quitting early beat training all the way to the bottom of the loss curve?