Split, Apply, Combine
Overview
The shape of every group-by, named once so the rest makes sense.
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Name the three steps of a group-by.
- Say why
df.groupby("x")on its own computes nothing. - Read the result, and get the key back as a column.
- Report the group size beside the group summary.
Why This Matters
Every question of the form "per region", "by month", "for each subject" is this one operation. Naming its three steps once means the rest of the unit is variations rather than new material — and it makes the error messages legible, because they nearly always come from the apply step.
Step 1: Three steps in one call
Split the table into groups by a key, apply a summary to each group, combine the answers into one result. pandas does all three in one expression, which is why it helps to be able to name them separately.
Nothing happens until you name the summary
The grouped object is a plan, not a result. Printing it tells you almost nothing, which is confusing until you know it is deliberate.
import pandas as pd
df = pd.DataFrame({"subject": ["maths", "art", "maths"], "score": [90, 70, 80]})
grouped = df.groupby("subject")
print(grouped)
print(grouped["score"].mean())The three steps, visible
You will not write it this way, but seeing the split makes the rest concrete.
import pandas as pd
df = pd.DataFrame({"subject": ["maths", "art", "maths"], "score": [90, 70, 80]})
for key, piece in df.groupby("subject"):
print(key, "->", piece["score"].tolist(), "-> mean", piece["score"].mean())Mini Practice #1: Watch the split
Try It YourselfLoop over the groups and print each key with the rows it holds, then get the same answer in one expression.
Step 2: The key becomes the index
The result is labelled by the group key, which is a Series indexed by whatever you grouped on.
Getting the key back as a column
Sometimes you want a plain table rather than an indexed Series. reset_index turns the key back into a column, and as_index=False asks for that from the start.
import pandas as pd
df = pd.DataFrame({"subject": ["maths", "art", "maths"], "score": [90, 70, 80]})
print(df.groupby("subject")["score"].mean().reset_index())
print(df.groupby("subject", as_index=False)["score"].mean())Always report the size
A mean over two rows and a mean over two thousand are not comparable, and the group with the striking average is very often the group with three rows in it. Putting the count beside the summary is the cheapest honesty there is.
import pandas as pd
df = pd.DataFrame({"subject": ["maths", "art", "maths"], "score": [90, 70, 80]})
out = df.groupby("subject")["score"].agg(["mean", "size"])
print(out)Where rows with a missing key go
A row whose key is NaN is dropped from the result by default, so your groups can silently fail to account for the whole table. dropna=False keeps them as their own group.
import pandas as pd
import numpy as np
df = pd.DataFrame({"subject": ["maths", np.nan], "score": [90, 70]})
print(df.groupby("subject")["score"].size().sum(), "of", len(df))
print(df.groupby("subject", dropna=False)["score"].size().sum(), "of", len(df))Mini Practice #2: Mean and size together
Try It YourselfProduce the mean and the row count per city in one result, and notice which group you would be wary of.
Step 3: When to Use It
Any question with “per” in it
Mean per team, total per month, count per category: group, summarise, and pandas combines.
df.groupby("team")["score"].mean()Comparing groups
A grouped result is one row per group, ready to sort, chart or put in a table.
When Not to Use It
Do not loop over the unique values and filter each one by hand. It repeats the scan per group and is easy to get subtly wrong; groupby does it in one pass.
End-of-Lesson Exercises
Exercise 1: Mean score per subject
Write mean_by_subject(df) returning a dictionary of subject to mean score, each mean rounded to one decimal place.
groupby the subject, take the score column, ask for the mean, round it, and to_dict() at the end gives plain keys and values.
Exercise 2: The mean, with the size beside it
Write summary_by(df) returning a dictionary of subject to a two-element list of [mean score rounded to one place, number of rows]. Reporting the size is what stops a group of two being read like a group of two thousand.
Group once and take both the mean and the size off the same grouped object, then build the dictionary with round() and int() so the values are plain Python.