Build an ML Model with Pen and Paper
A gentle introduction to ML and the underlying math
Overview
A deep learning model takes data as input and outputs a prediction: identifying the mushroom in a picture, picking the next word in a sentence, or estimating a house’s price. It all comes down to three critical pieces:
- Input – your data (say, an image), converted into a matrix
- Weights – another matrix that gets multiplied with the input to form a prediction
- Loss function – how far that prediction lands from the truth
It sounds complex, but it’s mostly matrix multiplication with a little calculus mixed in.
Linear Algebra & Calculus Review
The part of linear algebra relevant to deep learning is matrix operations.
NOTE: GPUs became a critical part of deep learning because they pack thousands of small cores that run arithmetic in parallel – exactly what matrix multiplication needs.
Even though modern tools such as PyTorch and TensorFlow are relatively forgiving for engineers without a strong math background, it’s still worth reviewing the basics to build intuition.
Matrix Multiplication
The building block is the dot product of two vectors. The result is the sum of element-wise products between a horizontal (row) vector and a vertical (column) vector:
\[ \begin{bmatrix} 1 & 2 & 3 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1\cdot 4 + 2\cdot 5 + 3\cdot 6 = 32 \]
Note what happened to the shape, two vectors of length 3 collapse into one scalar (a number). The vectors must be the same length, since every element needs a partner.
A vector has just one dimension, but real-world data is often multi-dimensional.
Matrix multiplication is just this dot product done many times. Each entry in the result is the dot product of a row from the left and a column from the right:
\[ \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1\cdot 5 + 2\cdot 7 & 1\cdot 6 + 2\cdot 8 \\ 3\cdot 5 + 4\cdot 7 & 3\cdot 6 + 4\cdot 8 \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix} \]
Derivatives
The core tool from calculus is the derivative – the instantaneous rate of change. It’s “rise over run”, the difference in \(y\) divided by the difference in \(x\), measured as the run shrinks to nothing.
Take a vehicle whose position (y-axis) over time (x-axis) is \(f(t) = t^2 + 4\). Its velocity at a given moment, say \(t = 4\), is the derivative \(f'(t)\). Remember high-school physics? Velocity = distance ÷ time. The derivative is that same ratio, but measured over a vanishingly small step \(h\) (\(h < 0.00000{\dots}0001\)). Shrinking \(h\) turns the average velocity into the instantaneous velocity at that moment:
\[ f'(t) = \lim_{h \to 0} \frac{f(t+h) - f(t)}{h} \]
Expanding \(f(t+h) = t^2 + 2th + h^2 + 4\) and subtracting \(f(t)\) leaves \(2th + h^2\). Dividing by \(h\) gives \(2t + h\), and as \(h \to 0\) the lone \(h\) vanishes:
\[ f'(t) = 2t \]
NOTE: Derivatives can be written a few ways: the Leibniz form \(\frac{d}{dx}f(x)\) and Lagrange (prime) form \(f'(x)\) mean exactly the same thing.
So the velocity at time \(t = 4\) is \(f'(4) = 2(4) = 8\). In practice you rarely take the limit by hand. The power rule (\(\frac{d}{dx}x^n = nx^{n-1}\)) gives the same answer instantly, \(t^2 \to 2t\), and the constant \(4 \to 0\). Alongside the power rule, a handful of other rules cover the common cases, such as products, quotients, and compositions.
Deep Learning - the Essence
You’ve probably seen versions of this network illustration passed around the internet as a symbol of how complex these algorithms are. But strip away the visual and it’s essentially just a long sequence of dot products. Each layer multiplies its inputs by a matrix of weights and passes the result on. Run an input all the way through and you get a prediction — that’s a single forward pass.
In mathematics, each neuron in this network boils down to a single formula:
\[ z = b + \sum_{i=1}^{m} x_i w_i \]
- \(x\) is the input vector.
- \(w\) is the weight vector, initialized with random values.
- \(i\) indexes each input, running from 1 to \(m\) (the total number of inputs).
- \(\sum\) represents the summation \(x_1 w_1 + x_2 w_2 + \dots + x_m w_m\).
- \(b\) is another arbitrary value called the bias. Like weights, it’s learned during training.
The presence of \(b\) might be confusing. It’s there because any weight multiplied by an input of 0 gives 0. Without a bias, an all-zero input would force the output to 0 no matter what the weights learn. The bias shifts the output so the neuron can still contribute to the prediction.
Since the weights start out arbitrary, the network outputs nonsense at first. Training fixes this over many iterations of backpropagation. As the weights and bias get updated, and the output gradually converges toward the expected result.
Calculation by Hand
A good way to gain intuition is to train a model by hand, which consist of one forward pass, one round of backpropagation. The XOR logic gate neural network is a classic example for demonstrating how data flows through the pipeline.
Step 1 - Understand the Data
An XOR logic gate has two inputs, \(x_1\) and \(x_2\). Each can be 1 or 0 independently, and the output is also either 1 or 0.
Target — the XOR truth table:
| \(x_1\) | \(x_2\) | \(y\) - XOR Result |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |
As the truth table shows, the XOR result is 1 when \(x_1\) and \(x_2\) differ, and 0 otherwise. The goal of the network is to predict the XOR Result from the input values of \(x_1\) and \(x_2\).
For this example, the network will be trained on the input \(x_1 = 1\) and \(x_2 = 0\).
Step 2 - Create Arbitrary Weights
Plotting the four inputs against their targets reveals the problem, which is no single straight line can separate the 1s from the 0s. That’s why the XOR network needs at least two sets of weights (neurons) to make a prediction.
\[ W_1 = \begin{bmatrix} 0.5 \\ 0.5 \end{bmatrix}, \quad W_2 = \begin{bmatrix} -0.5 \\ 0.5 \end{bmatrix}, \quad \mathbf{b}_1 = \begin{bmatrix} 0.2 & -0.4 \end{bmatrix} \]
Weights for the hidden layer output
The final prediction has to be a scalar (a single number), so we need one more weight vector to collapse the hidden layer’s output.
\[ W_3 = \begin{bmatrix} 0.3 \\ -0.4 \end{bmatrix}, \quad \mathbf{b}_2 = 0.1 \]
Weights for the prediction output
Step 3 - Select Activation Function
This is often referred to as applying non-linearity. A dot product plus bias is linear, meaning it’s a straight line or flat plane. Real data is rarely that clean. An activation function bends the output so the network can curve where it needs to. There are many activation functions to choose from. In this case, sigmoid is a natural fit for probability:
\(\hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}}\)
Sigmoid squashes any input into the range 0 to 1, which is useful for predicting the probability \(\hat{y}\).
Step 4 - Run Forward Pass for Input (1, 0)
The forward pass should feel familiar by now. For the hidden layer output, the formula is: \(\mathbf{a}_1 = \sigma([x_1 \quad x_2] \cdot [W_1 \quad W_2] + \mathbf{b}_1)\).
\[ \mathbf{z}_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 0.5 & -0.5 \\ 0.5 & 0.5 \end{bmatrix} + \begin{bmatrix} 0.2 & -0.4 \end{bmatrix} = \begin{bmatrix} 0.7 & -0.9 \end{bmatrix} \]
\[ \mathbf{a}_1 = \sigma(\mathbf{z}_1) = \begin{bmatrix} \frac{1}{1 + e^{-0.7}} & \frac{1}{1 + e^{0.9}} \end{bmatrix} = \begin{bmatrix} 0.66818 & 0.28905 \end{bmatrix} \]
Similarly, for the final prediction output, the formula is: \(\mathbf{\hat{y}} = \sigma(\mathbf{a}_1 \cdot W_3 + \mathbf{b}_2)\).
\[ \mathbf{z}_2 = \begin{bmatrix} 0.66818 & 0.28905 \end{bmatrix} \begin{bmatrix} 0.3 \\ -0.4 \end{bmatrix} + 0.1 = 0.18483 \]
\[ \mathbf{\hat{y}} = \sigma(\mathbf{z}_2) = \frac{1}{1 + e^{-0.18483}} = 0.54607 \]
Step 5 - Calculate Loss
Like the activation function, there are many loss functions to pick from, each measuring how far the prediction \(\hat{y}\) lands from the true label \(y\). Since sigmoid gives a probability and the result is binary, we can pick binary cross-entropy:
\[ L = -\frac{1}{n} \sum_{j=1}^{n} \left[ y_j \ln(\hat{y}_j) + (1 - y_j) \ln(1 - \hat{y}_j) \right] \]
Here \(j\) runs over all \(n\) training examples. The loss spikes when the model is confident and wrong, and drops near 0 when it’s confident and right.
\[ L = -\frac{1}{1} \left[ (1) \ln(0.54607) + (1 - 1) \ln(1 - 0.54607) \right] = 0.60501 \]
Step 6 - Making the Network Learn
We just pushed data from input to prediction. Now we need to work backwards to adjust the arbitrary weights we created, so the next prediction becomes more accurate.
This is where the derivative shines. The derivative of the loss with respect to a weight, \(\frac{\partial L}{\partial w}\), tells us how the loss shifts as we nudge that weight. Notice how it’s \(\partial\) instead of \(d\). This is a partial derivative — the same idea as a normal derivative, except it measures the slope for one weight while holding the rest fixed. A neural network can have thousands of weights (\(f(w_1, w_2 \dots w_n)\)), and we want to know how each one affects the model’s loss.
Collect all of those partial derivatives together and you get the gradient — the vector that tells the algorithm which direction to nudge every weight to bring the loss down.
Like the power rule mentioned earlier, there’s another shortcut worth introducing called the chain rule. It is used to take derivatives of nested functions:
\[ f(x) = g(h(l(x))) \quad \rightarrow \quad f'(x) = g'\big(h(l(x))\big) \cdot h'\big(l(x)\big) \cdot l'(x) \]
As we can see, it’s recursive, multiply the derivative of the outer function by the derivative of the inner one, and repeat until you run out of nesting. For loss over weight in deep learning, the chain looks like this:
\[ \frac{\partial L}{\partial w_i} = \underbrace{\frac{\partial L}{\partial \hat{y}}}_{\text{loss function}} \cdot \underbrace{\frac{\partial \hat{y}}{\partial z}}_{\text{activation function}} \cdot \underbrace{\frac{\partial z}{\partial w_i}}_{\text{dot product}} \]
Output Neuron Backpropagation
Since we’re working backwards, we start with how \(W_3\) affects the loss. That means computing \(\frac{\partial L}{\partial W_3}\), which can be expressed as:
\[ \frac{\partial L}{\partial W_3} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_2} \cdot \frac{\partial z_2}{\partial W_3} = L'(\sigma(a_1W_3 + \mathbf{b_2})) \cdot \sigma'(a_1W_3 + \mathbf{b_2}) \cdot (a_1W_3 + \mathbf{b_2})' \]
After the terms cancel, the result becomes:
\[ \frac{\partial L}{\partial W_3} = (\hat{y} - y) \cdot a_1 = \begin{bmatrix} (0.54607 - 1) \cdot 0.66818 \\ (0.54607 - 1) \cdot 0.28905 \end{bmatrix} = \begin{bmatrix} \mathbf{-0.30331} \\ \mathbf{-0.13121} \end{bmatrix} \]
Now we calculate the partial derivative for the bias as well, since it’s also a parameter that gets updated. The process is the same, we just treat \(\mathbf{b}_2\) as the variable instead of \(W_3\): \[ \frac{\partial L}{\partial \mathbf{b_2}} = \frac{\partial L}{\partial \hat{y}} \cdot \frac{\partial \hat{y}}{\partial z_2} \cdot \frac{\partial z_2}{\partial \mathbf{b_2}} = L'(\sigma(a_1W_3 + \mathbf{b_2})) \cdot \sigma'(a_1W_3 + \mathbf{b_2}) \cdot (a_1W_3 + \mathbf{b_2})' \]
After solving the derivative and plugging in the constants (\(\frac{\partial z_2}{\partial \mathbf{b_2}} = 1\)), we get: \(-0.45393 \times 1 = \mathbf{-0.45393}\)
Updating the Parameters
Remember the earlier velocity example? As the curve approaches the minimum, the lowest point on the parabola, the slope flattens to 0 indicating vehical is stationary. In deep learning, we want to step downhill the same way until the loss stops shrinking. This is gradient descent: \(w \leftarrow w - \eta \frac{\partial L}{\partial w}\). Here \(\eta\) is the learning rate, which sets size of the step we take toward the minimum. A larger \(\eta\) means a bigger jump in the weights each training cycle. For this example, we use \(\eta = 0.1\) to update every parameter:
\[ W_1 \leftarrow \begin{bmatrix} 0.5 \\ 0.5 \end{bmatrix} - 0.1 \begin{bmatrix} -0.03019 \\ 0 \end{bmatrix} = \begin{bmatrix} 0.50302 \\ 0.5 \end{bmatrix} \]
\[ W_2 \leftarrow \begin{bmatrix} -0.5 \\ 0.5 \end{bmatrix} - 0.1 \begin{bmatrix} 0.03731 \\ 0 \end{bmatrix} = \begin{bmatrix} -0.50373 \\ 0.5 \end{bmatrix} \]
\[ W_3 \leftarrow \begin{bmatrix} 0.3 \\ -0.4 \end{bmatrix} - 0.1 \begin{bmatrix} -0.30331 \\ -0.13121 \end{bmatrix} = \begin{bmatrix} 0.33033 \\ -0.38688 \end{bmatrix} \]
\[ \mathbf{b_1} \leftarrow \begin{bmatrix} 0.2 & -0.4 \end{bmatrix} - 0.1 \begin{bmatrix} -0.03019 & 0.03731 \end{bmatrix} = \begin{bmatrix} 0.20302 & -0.40373 \end{bmatrix} \]
\[ \mathbf{b_2} \leftarrow 0.1 - 0.1(-0.45393) = 0.14539 \]
Verify Effectiveness
One backward pass should have moved the prediction closer to the target. Running the same input \((1, 0)\) through the updated weights:
Hidden Layer: \[ \mathbf{z}_1 = \begin{bmatrix} 1 & 0 \end{bmatrix} \begin{bmatrix} 0.50302 & -0.50373 \\ 0.5 & 0.5 \end{bmatrix} + \begin{bmatrix} 0.20302 & -0.40373 \end{bmatrix} = \begin{bmatrix} 0.70604 & -0.90746 \end{bmatrix} \]
\[ \mathbf{a}_1 = \sigma(\mathbf{z}_1) = \begin{bmatrix} \frac{1}{1 + e^{-0.70604}} & \frac{1}{1 + e^{0.90746}} \end{bmatrix} = \begin{bmatrix} 0.66953 & 0.28752 \end{bmatrix} \]
Output Layer:
\[ \mathbf{z}_2 = \begin{bmatrix} 0.66953 & 0.28752 \end{bmatrix} \begin{bmatrix} 0.33033 \\ -0.38688 \end{bmatrix} + 0.14539 = 0.25532 \]
\[ \mathbf{\hat{y}} = \sigma(\mathbf{z}_2) = \frac{1}{1 + e^{-0.25532}} = 0.56349 \]
The prediction moved from 0.54607 up to 0.56349, toward the target of 1. One backward pass indeed brought a small improvement to the final output.
That’s a single iteration for a single example. Real training repeats the whole loop for many epochs. Run it long enough and the four predictions settle onto 0, 1, 1, 0, showing the network has learned XOR.
Food for Thought
Everything above is the smallest possible version of deep learning. Scaling it up introduces new problems worth considering:
- Stochastic Gradient Descent — With a million data points, is there a more efficient way to train than averaging the gradient over every single example before each update?
- Mini-batches — Between one example and the full dataset sits a compromise: random batches of 32, 64, or 128 examples. Why do these sizes pair so well with GPUs?
- Overfitting — What goes wrong when a model learns its training data too well, and how would you even detect it?
- Dropout — Why does forcing the network to work short-handed make it generalize better?
- Early Stopping — When would quitting early beat training all the way to the bottom of the loss curve?