3.1.1 Pandas

Why pandas?

Almost every IOAI task gives you data as a table: a file such as train.csv, with one row per example and one column per feature. Before we can train a model, we need to load that table, look at what is inside and select the parts the model needs. pandas is the Python library we will use for this, and it is available in the contest.

You do not need to know all of pandas. Most tasks begin with the same small set of actions: load a table, inspect it, select rows and columns, add a column and group related rows. That is all we will focus on here; the rest can wait until you actually need it.

If you have not completed Python Fundamentals and Jupyter Notebook Interface and Basics yet, do those first. We will type everything below into a notebook and inspect each result as we go.

DataFrames and Series

• A DataFrame is the whole table: rows and named columns, much like a spreadsheet.
• A Series is one column from that table: a sequence of values with a name.

Each row also has a label called the index. When we load an ordinary file, pandas usually starts with the simple labels 0, 1, 2 and so on. You do not need to do anything special with the index yet; just learn to recognise it when it appears.

Pick one main course

As with Python Fundamentals, do not collect several courses and half-finish all of them. Pick one and do its exercises.

Our pick: Kaggle Learn Pandas

Kaggle Learn Pandas has six short lessons in the same style as the Kaggle Python course. Each lesson comes with an exercise notebook that checks your answers. Work through them in order; if one moves too quickly, the last column shows where you can slow down and read more.

Other good options

• Official tutorials: the pandas getting started tutorials answer one question per page, with clear pictures.
• A free book: chapter 3 of Jake VanderPlas's Python Data Science Handbook goes deeper.
• Videos: Corey Schafer's pandas series, or Keith Galli's complete tutorial in one sitting.

A quick tour

The rest of this page is a quick tour of the pandas operations you will use in everyday tasks. Our data is made up and deliberately small, so you can check every result by eye instead of trusting the code blindly.

Load a table

In a task, the data comes as a CSV file. The examples here use students.csv. Download it and put it in the same folder as your notebook:

Download students.csv

It holds this table:

Once we have imported pandas, loading the file takes one line:

import pandas as pd

df = pd.read_csv("students.csv")
df

Because df is the last line of the cell, the notebook displays the table for us. The extra column on the left is the index, numbering the rows from 0 to 5. import pandas as pd is the standard import; when you see pd in other people's code, it means pandas.

Look at the data

Before changing a new dataset, look at it. These four checks should become your usual starting point:

df.shape

(6, 4)

This tells us that the table has 6 rows and 4 columns. df.head() shows the first 5 rows, which is much more useful when the full file contains 100,000. df.info() then lists every column, its type and how many values are not missing:

df.info()

<class 'pandas.DataFrame'>
RangeIndex: 6 entries, 0 to 5
Data columns (total 4 columns):
 #   Column   Non-Null Count  Dtype
---  ------   --------------  -----
 0   name     6 non-null      str
 1   city     6 non-null      str
 2   maths    6 non-null      int64
 3   science  6 non-null      int64
dtypes: int64(2), str(2)
memory usage: 324.0 bytes

int64 means whole numbers and str means text. If you expected a numeric column but see str, something in that column is not being read as a number. Do not worry about fixing it yet; Data Cleaning will show you how.

df.describe() summarises every number column:

df.describe()

count, mean, min and max mean what their names suggest. 50% is the median, and std tells us how spread out the values are. For a text or category column, value_counts gives us a useful summary instead:

df["city"].value_counts()

city
Delhi     3
Mumbai    2
Pune      1
Name: count, dtype: int64

In a classification task, run value_counts on the label column. You will immediately see whether the classes are balanced or whether one class is much more common than the others.

Pick out columns and rows

Use one pair of brackets to select one column as a Series:

df["maths"]

0    91
1    64
2    78
3    85
4    58
5    72
Name: maths, dtype: int64

Use double brackets to select several columns as a smaller DataFrame. The inner brackets hold a Python list of column names:

df[["name", "maths"]]

We can also keep only the rows that pass a test. df["maths"] > 75 produces a Series of True and False values; placing that test inside df[...] keeps the rows where it is True:

df[df["maths"] > 75]

The index keeps each row's original number, so we can see that rows 1, 4 and 5 were left out. That is normal: filtering does not automatically renumber the rows.

To combine two tests, use & for "and" and | for "or". Remember to put each test in brackets—this is easy to forget at first:

df[(df["maths"] > 70) & (df["city"] == "Delhi")]

When you need one value, give loc a row label and a column name:

df.loc[2, "science"]

np.int64(95)

The result is still just the number 95. pandas stores numbers with NumPy, another library, so you may see the NumPy type printed around a single value. You can use int(...) to turn it into an ordinary Python integer if you need to, but most of the time you can leave it alone.

Add a column and sort

We add a new column with an ordinary assignment. Arithmetic then works on whole columns at once, row by row, so we do not need to write a loop:

df["total"] = df["maths"] + df["science"]
df.sort_values("total", ascending=False).head(3)

Group and summarise

groupby follows a useful pattern: split the rows into groups, calculate something for each group and put the answers back together:

df.groupby("city")["maths"].mean()

city
Delhi     80.333333
Mumbai    61.000000
Pune      85.000000
Name: maths, dtype: float64

Read the code from left to right: "group the rows by city, take the maths column, then average it inside each group." Long pandas expressions become much less intimidating when you read them one step at a time.

A real dataset

Now let us use the same tools on real data. The Iris dataset comes with scikit-learn, so you can load it without internet. It contains 150 flowers, four measurements for each flower and a species number in target.

from sklearn.datasets import load_iris

iris = load_iris(as_frame=True).frame
iris.shape

(150, 5)

Next we analyze the mean length of different targets

iris.groupby("target")["petal length (cm)"].mean().round(1)

target
0    1.5
1    4.3
2    5.6
Name: petal length (cm), dtype: float64

One line has already told us something useful: the three species have very different average petal lengths, so this measurement should help separate them. This is why we inspect data before choosing a model. Terminology and Approaching a Typical Problem build on this habit.

From pandas to a model

When we train a model, we give it the feature columns and label column separately. By convention, we call them X and y:

X = df[["maths", "science"]]
y = df["city"]
X.shape, y.shape

((6, 2), (6,))

scikit-learn accepts these pandas objects directly, so you do not need to convert them first. We will use this pattern throughout Scikit-learn Basics.

Common mistakes

The last mistake catches almost everyone at least once. pandas methods often return a new table instead of changing the old one. If you want to keep the result, assign it to a name.

Check yourself

Use the df from the tour, including the total column. Try each exercise before opening its answer; if you are unsure, inspect the intermediate table in your notebook.

More practice

CodeChef's Learn Pandas course gives you small problems after each lesson, and its pandas practice path has more once you finish. You do not need to complete everything at once; use them when you want more repetition with a particular operation.

The best practice is still a real dataset. Open the data from any past IOAI task and try the four "look" checks on it. Ask yourself what you know about the table before you change anything.

What comes next

If you can load a table, inspect it, select rows and columns, add a column and group values, you know enough pandas to continue. We will pick up the less common operations when a later task needs them.

• Matplotlib: draw your data, because a picture often shows what a table hides.
• Data Cleaning: what to do when the data has gaps, typos and wrong types.