5.1.2 PyTorch Basics

What PyTorch is for

PyTorch is the library you use to build and train neural networks. At IOAI it is the deep learning library: the 2026 rules list torch and scikit-learn as the core libraries and say that TensorFlow and Keras are not available. So learn PyTorch, and skip courses built on TensorFlow or Keras.

This page is only a short overview. It lists what you need to be able to read and change the training code in a task, shows each idea briefly, and points to the courses that teach it properly. The layers, activations and losses themselves come in Deep Learning Fundamentals.

You should know NumPy Basics first, and the PyTorch part of Classes and Inheritance helps too.

What you need for IOAI

• Tensors: PyTorch's arrays. They work almost exactly like NumPy arrays, with shape and dtype.
• Moving between NumPy and PyTorch, and making sure the dtype is float32.
• The GPU: the contest machines have GPUs, and you move data and models there with .to(device).
• Autograd: PyTorch works out gradients for you with backward(). You need the idea, not the maths, at this level.
• Models as classes that inherit from nn.Module.
• The training loop: predict, measure the loss, compute gradients, update the weights, repeat.
• Evaluation: model.eval() and torch.no_grad() when you only want predictions.
• Dataset and DataLoader, for feeding data in batches; see Dataset and DataLoader.

Where to learn it

Start with PyTorch's own Learn the Basics. It is short and every page runs as a notebook. For this module, the pages that matter are:

If you want more explanation and exercises, Daniel Bourke's free Learn PyTorch for Deep Learning is a well-known beginner course. Do 00. PyTorch Fundamentals and 01. PyTorch Workflow, including the exercises at the end of each. The same course is on YouTube as one long video.

Later, once training code feels familiar, Andrej Karpathy's Neural Networks: Zero to Hero builds everything from scratch, which is a great way to understand what backward() really does.

The basics in code

Tensors

import torch

t = torch.tensor([[1.0, 2.0],
                  [3.0, 4.0]])
t * 2

tensor([[2., 4.],
        [6., 8.]])

t.shape gives torch.Size([2, 2]), and t @ t multiplies matrices, as in NumPy. Converting in both directions is one call:

import numpy as np

torch.from_numpy(np.array([1.0, 2.0, 3.0]))

tensor([1., 2., 3.], dtype=torch.float64)

Note the float64: NumPy uses 64-bit numbers by default, but PyTorch models expect float32. Add .float() when you convert data for a model. Going the other way, t.numpy() gives a NumPy array.

The GPU

device = "cuda" if torch.cuda.is_available() else "cpu"

On a computer without an NVIDIA GPU this gives "cpu"; on the contest machines it will be "cuda". The model and every tensor you give it must be on the same device: model.to(device) and X.to(device).

Gradients

For y = x² + 2x, the slope at x = 3 is 2·3 + 2 = 8. PyTorch works it out for you:

x = torch.tensor(3.0, requires_grad=True)
y = x ** 2 + 2 * x
y.backward()
x.grad

tensor(8.)

Training a network is this, repeated for millions of weights: compute the loss, call backward(), and nudge each weight against its gradient. Gradient Descent and Backpropagation explain why that works.

A whole training loop

This trains the simplest possible model, a straight line, on points from y = 2x + 1:

from torch import nn

torch.manual_seed(0)
X = torch.tensor([[0.0], [1.0], [2.0], [3.0]])
y = 2 * X + 1

model = nn.Linear(1, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.05)

for epoch in range(500):
    pred = model(X)            # 1. predict
    loss = loss_fn(pred, y)    # 2. measure how wrong
    optimizer.zero_grad()      # 3. clear old gradients
    loss.backward()            # 4. compute new gradients
    optimizer.step()           # 5. update the weights

print(round(model.weight.item(), 3), round(model.bias.item(), 3))

2.0 1.0

It found the weight 2 and the bias 1. Almost every training loop you will see in an IOAI task has these same five steps; a real one just uses a bigger model and loops over batches from a DataLoader. To use the trained model:

with torch.no_grad():
    print(model(torch.tensor([[10.0]])))

tensor([[21.0000]])

torch.no_grad() tells PyTorch not to track gradients, which saves memory when you only want predictions. For models with layers such as dropout, also call model.eval() first.

Common mistakes

As with NumPy, print .shape and .dtype first when something breaks.

What comes next

• Multi-Layer Perceptrons: the first real neural network.
• Dataset and DataLoader and A Simple MLP in PyTorch: the full training pipeline.