How Machines Learn: Features, Training and Testing

The anatomy of learning

Before building models, you need to understand how a machine actually learns from data. The vocabulary and process here are universal — they apply to every ML model you'll ever build. Get these concepts clear and everything else falls into place.

Features and labels

Machine learning data has two parts. The features are the input information used to make a prediction — also called the independent variables, conventionally named X. The label (or target) is the answer you're trying to predict — the dependent variable, conventionally named y:

# Predicting house prices:
# FEATURES (X) — the inputs:        LABEL (y) — the answer:
#   size, bedrooms, location           price
#   1200, 3,        "suburb"   -->      4500000
#   2000, 4,        "city"     -->      8000000

# In code, with pandas:
X = df[["size", "bedrooms", "location"]]   # features
y = df["price"]                            # label

The model's job is to learn the relationship between the features (X) and the label (y), so that given new features, it can predict the label. Choosing good features — information actually relevant to what you're predicting — is one of the most important parts of ML, often more important than the algorithm you pick.

Training: how the model learns

Training (also called fitting) is the process where the model studies the examples and adjusts itself to capture the pattern connecting features to labels. You show it many rows of X with their known y, and it tunes its internal parameters to predict y as accurately as possible:

from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(X_train, y_train)   # this is training — the model learns here

That one line, model.fit(), is where the learning happens. Internally, the model adjusts itself to minimise its prediction errors on the training examples — much like the dart-throwing analogy from learning any skill: try, see the error, adjust, repeat. After fit(), the model has learned a pattern and can make predictions.

The critical idea: train/test split

Here's the single most important concept for doing ML correctly. You must never test your model on the same data it learned from. Why? Because a model can simply memorise the training data and look perfect on it, while being useless on new data. So we split our data: most of it for training, and a held-back portion the model never sees during training, for testing:

from sklearn.model_selection import train_test_split

# Hold back 20% of the data for testing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model.fit(X_train, y_train)         # learn from the training set
score = model.score(X_test, y_test) # test on UNSEEN data

The train_test_split function does exactly this — typically holding back 20% for testing. We train on the 80% and evaluate on the held-out 20%. This is the only honest way to know if your model actually generalises — if it works on data it's never seen, which is the whole point. Testing on training data is like giving students the exam answers in advance: a perfect score that means nothing.

Overfitting: the central danger

The train/test split exists to catch the most common ML problem: overfitting. An overfit model has essentially memorised the training data — including its random noise — instead of learning the real underlying pattern. It scores brilliantly on training data but poorly on new data:

# A telltale sign of overfitting:
print(model.score(X_train, y_train))   # 0.99 -- near-perfect on training!
print(model.score(X_test, y_test))     # 0.62 -- much worse on test data
# This big gap means the model memorised rather than learned.

Think of a student who memorises practice questions word-for-word but can't answer a slightly different exam question — they overfit to the practice set. The opposite problem, underfitting, is a model too simple to capture the pattern at all (poor on both training and test). The goal is the balance in between: a model that learns the genuine pattern and performs well on new data. Watching the gap between training and test scores is how you detect overfitting, and managing it is a core ML skill.

The complete ML process

Putting it together, here's the workflow every ML project follows — your reliable map:

1. Prepare the data — clean it and choose features (your data science skills).
2. Split — separate into training and test sets.
3. Choose and train a model — fit it on the training data.
4. Evaluate — test it on the unseen test data.
5. Improve — try better features, different models, or tuning.
6. Predict — use the final model on genuinely new data.

This process repeats for every model you build, whether predicting prices or classifying images. With these foundations — features and labels, training, the train/test split, and overfitting — you understand how machines learn. Now let's build your first real model: linear regression, which predicts numbers.