2.3.2 Probability and Statistics

Why probability and statistics matter

Machine learning is built from data, and data is rarely complete or perfectly predictable. A model does not simply say what will happen; it learns patterns from observations and often estimates what is likely to happen.

Probability gives us a language for uncertainty. It lets us describe possible outcomes, combine uncertain events and update a belief when new evidence arrives.

Statistics helps us understand observed data. It lets us summarise a dataset, compare groups, measure relationships and judge whether a pattern is meaningful or merely noise.

Together, they appear throughout AI:

• A classifier may return a probability for each possible class.
• A training set is a sample from a larger population we care about.
• Evaluation metrics are statistics calculated from model predictions.
• Bayes' theorem connects prior knowledge with new evidence.
• Means, variances and percentiles help us inspect data before modelling.
• Correlation can reveal a useful relationship, but it cannot by itself prove cause and effect.

Learn from these first

Use these two resources in order. You do not need to complete every advanced statistics unit before continuing with AI.

1. Khan Academy: Statistics and probability

Start with Khan Academy's Statistics and probability course. It combines short videos with practice questions and gives the gentlest introduction to the vocabulary on this page.

Focus on the parts covering:

• summarising quantitative data;
• mean, median, variance and percentiles;
• probability and compound events;
• independent, dependent and conditional events; and
• relationships between two numerical variables.

2. Dive into Deep Learning: Probability and Statistics

Then read Dive into Deep Learning: Probability and Statistics. It connects the same ideas directly to machine learning and includes executable notebooks for several deep-learning frameworks.

Pay particular attention to:

• probabilities as a description of uncertainty;
• random variables and distributions;
• conditional probability and independence;
• Bayes' theorem; and
• the distinction between a probability and a statistic calculated from data.

The connection between the topics

Probability basics

A random experiment is a process whose exact result is uncertain. An outcome is one possible result, the sample space is the set of all possible outcomes, and an event is a collection of outcomes we care about.

A probability lies between $0$ and $1$:

• $P(A)=0$ means event $A$ is impossible under the model.
• $P(A)=1$ means event $A$ is certain under the model.
• A larger value of $P(A)$ means that $A$ is considered more likely.

The complement $A^c$ means that $A$ does not occur, so

$$P(A^c)=1-P(A).$$

The union $A\cup B$ means that $A$, $B$ or both occur. The intersection $A\cap B$ means that both occur.

Conditional probability

The conditional probability $P(A\mid B)$ is the probability of $A$ after learning that $B$ has occurred:

$$P(A\mid B)=\frac{P(A\cap B)}{P(B)}, \qquad P(B)>0.$$

The condition changes the group being considered. This is the mathematical form of asking, “Among cases where $B$ is true, how often is $A$ also true?”

Independent and dependent events

Events $A$ and $B$ are independent when learning that one occurred does not change the probability of the other:

$$P(A\mid B)=P(A).$$

For independent events,

$$P(A\cap B)=P(A)P(B).$$

Events are dependent when knowing one event changes the probability of the other. In AI, dependencies are often useful signals: if an image feature changes the probability of a class, a model may be able to learn from that relationship.

Bayes' theorem

Bayes' theorem reverses a conditional probability:

$$P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}.$$

Its parts are often described as:

Bayes' theorem matters because the probability we want is often the reverse of the probability we can measure directly.

Practical applications

• Medical testing: a test's accuracy is not the same as the probability that a person has the condition after a positive result; the condition's prevalence also matters.
• Spam filtering: the words in a message act as evidence that updates the probability that the message is spam.
• Classification: observed features can update the probability assigned to each possible class.
• Fault detection: a sensor reading can update the probability that a machine or system has failed.

Descriptive statistics

Descriptive statistics reduce a dataset to a few values that reveal its centre, spread and position. They are usually the first things to inspect before choosing or training a model.

Measures of centre

The mean and median answer different questions, so neither is automatically the “correct” average.

Measures of spread and position

The $50$th percentile is the median. Percentiles are especially useful when a distribution is skewed or when a few extreme values make the mean misleading.

Variables in a dataset

A variable is a quantity or category recorded for each observation. In a supervised-learning table, variables usually play one of two roles:

The terms describe a modelling role, not a guarantee of causation. Calling a column an independent variable does not prove that changing it would change the target.

Correlation and causation

Correlation means that two variables tend to vary together. A positive correlation means they tend to move in the same direction; a negative correlation means they tend to move in opposite directions.

A correlation coefficient is commonly between $-1$ and $1$:

• values near $1$ indicate a strong positive linear relationship;
• values near $-1$ indicate a strong negative linear relationship; and
• values near $0$ indicate little linear relationship.

A value near $0$ does not rule out a curved or otherwise non-linear relationship.

Causation means that changing one variable produces a change in another. Correlation alone cannot establish this because the relationship may be explained by:

• a confounding variable that influences both quantities;
• reverse causality, where the assumed outcome actually influences the assumed cause;
• selection or measurement bias; or
• coincidence in a small or noisy dataset.

Controlled experiments, careful study design and causal assumptions are needed to support a causal claim. A predictive model can be highly accurate while learning correlations that are not causal.

How the ideas fit into an AI workflow

1. Describe the data. Use counts, means, medians, variances and percentiles to understand each variable.
2. Inspect relationships. Look for correlations and dependencies that might help prediction or reveal leakage and redundancy.
3. Separate inputs from targets. Decide which variables are features and which outcome the model should predict.
4. Represent uncertainty. Use probabilities rather than pretending every prediction is certain.
5. Update with evidence. Use conditional probability and Bayes' theorem to understand how observations change a belief.
6. Interpret carefully. Treat statistical association as evidence of a relationship, not automatic proof of causation.

What you should know before moving on

You should be able to:

• explain the difference between probability and statistics;
• recognise independent, dependent and conditional events;
• describe what Bayes' theorem does and why the prior matters;
• choose between mean, median and mode as a summary of centre;
• explain variance, standard deviation and percentiles;
• identify independent and dependent variables in a supervised-learning problem; and
• explain why correlation does not by itself prove causation.

This foundation is enough for the first classical machine-learning modules. Later topics will add probability distributions, sampling, inference and model-specific uses of these ideas when they are needed.