Steps in Designing a Learning System
A learning system improves its performance through experience. Before we can
build one, we need a precise, engineering-level definition of what
“learning” actually means — and Tom Mitchell gave us exactly that.
This post walks through the five steps used to design any machine learning
system, from defining the training experience all the way to measuring
performance.
Tom Mitchell’s Definition of Machine Learning
This is the most widely quoted definition in machine learning, and it’s the
starting point for every learning system we design.
A\ computer\ program\ is\ said\ to\ learn\ from\ experience\ E
\ with\ respect\ to\ some\ task\ T
\ and\ performance\ measure\ P,
\ if\ its\ performance\ at\ T,
\ as\ measured\ by\ P,
\ improves\ with\ experience\ E.
$$
Notice the three components hiding inside this sentence: Experience
(E), Task (T), and Performance measure
(P). Every learning system we design has to explicitly define all
three. The five steps below turn this abstract definition into a concrete
system.
Step 1: Training Experience (E)
Training experience consists of the training examples the system learns
from — pairs of inputs and their known target outputs.
E=\left\{(x_1,y_1),(x_2,y_2),\ldots,(x_n,y_n)\right\}
$$
where
x_i=\text{Input}
$$
$$
y_i=\text{Target Output}
$$
$$
(\text{Email},\text{Spam})
$$
An email is the input, and whether it’s spam is the target output.
Step 2: Target Function
The target function is the “true” underlying mapping from input to output
that we’re trying to approximate.
f:X\rightarrow Y
$$
For spam detection, this looks like:
f(\text{Email})=
\begin{cases}
1,&\text{Spam}\\
0,&\text{Not Spam}
\end{cases}
$$
Step 3: Representation (Choosing a Hypothesis)
Since we can’t know the true target function f directly, we choose
a hypothesis h that approximates it.
h(x)\approx f(x)
$$
Common representations include:
h(x)=\beta_0+\beta_1x
$$
— a linear model, or —
$$
h(x)=\sigma(\beta_0+\beta_1x)
$$
where the sigmoid function is defined as
\sigma(z)=\frac{1}{1+e^{-z}}
$$
Step 4: Learning Algorithm
Once we’ve picked a representation, the learning algorithm’s job is to find
the parameters that make h(x) fit the data as closely as possible.
Linear Regression Cost Function
J(\theta)=\frac1{2m}\sum_{i=1}^{m}(h_\theta(x_i)-y_i)^2
$$
Gradient Descent
The parameters are updated iteratively to minimize the cost:
\theta_j:=\theta_j-\alpha\frac{\partial J(\theta)}{\partial\theta_j}
$$
Step 5: Performance Measure (P)
Finally, we need a way to quantify how well the system is doing. The
right metric depends on whether the task is classification or regression.
For Classification
Accuracy
Accuracy=
\frac{TP+TN}
{TP+TN+FP+FN}
$$
Precision
Precision=
\frac{TP}{TP+FP}
$$
Recall
Recall=
\frac{TP}{TP+FN}
$$
F1 Score
F_1=
\frac{2\times Precision\times Recall}
{Precision+Recall}
$$
For Regression
Mean Squared Error
MSE=
\frac1n
\sum_{i=1}^{n}(y_i-\hat y_i)^2
$$
Putting It All Together: The Complete Learning System
The five steps flow into each other, forming a single pipeline:
E
\longrightarrow
f
\longrightarrow
R
\longrightarrow
A
\longrightarrow
P
$$
where
\begin{aligned}
E &= \text{Training Experience} \\
f &= \text{Target Function} \\
R &= \text{Representation} \\
A &= \text{Learning Algorithm} \\
P &= \text{Performance Measure}
\end{aligned}
$$
The Learning Model
Tying this back to Mitchell’s definition, learning itself can be expressed
as a function of experience, task, and performance measure:
Learning=f(E,T,P)
$$
where
\begin{aligned}
E &= \text{Experience}\\
T &= \text{Task}\\
P &= \text{Performance Measure}
\end{aligned}
$$
Summary
\boxed{
Training\ Experience
\rightarrow
Target\ Function
\rightarrow
Representation
\rightarrow
Learning\ Algorithm
\rightarrow
Performance\ Measure
}
$$
Every machine learning system you’ll ever build — from a simple spam
filter to a deep neural network — follows this same five-step
blueprint. Get comfortable with these steps, and the rest of ML starts to
feel a lot less like magic and more like engineering.