transform versus aggregate
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Say what shape an aggregate returns and what shape a transform returns.
- Put a group summary beside every row.
- Compare a row with its own group rather than with the table.
- Choose between the two from the question being asked.
Why This Matters
Questions like "which rows beat their own group average" cannot be answered by an aggregate, because an aggregate throws away the rows. transform is what keeps the shape, and it is the piece people are missing when they fall back to a loop over the groups.
Step 1: aggregate collapses
One value per group. The rows are gone, which is usually what you wanted.
The shape of an aggregate
Four rows in, two groups out. Nothing here lines up with the original table any more.
import pandas as pd
df = pd.DataFrame({"k": ["a", "a", "b", "b"], "v": [1, 3, 10, 20]})
means = df.groupby("k")["v"].mean()
print(means)
print(len(df), "rows ->", len(means), "groups")Mini Practice #1: Two shapes
Try It YourselfPrint the length of the aggregate and the length of the transform, next to the length of the table.
Step 2: transform keeps the shape
One value per row — each row gets its own group's answer. The result lines up with the table, so you can put it straight into a column or compare against it.
The group answer, beside every row
Same length as the table, same index, so it can sit alongside the data it describes.
import pandas as pd
df = pd.DataFrame({"k": ["a", "a", "b", "b"], "v": [1, 3, 10, 20]})
df["group_mean"] = df.groupby("k")["v"].transform("mean")
print(df)Comparing a row with its own group
This is the question an aggregate cannot answer. Note how different it is from comparing against the whole table's mean.
import pandas as pd
df = pd.DataFrame({"k": ["low", "low", "high"], "v": [1, 2, 100]})
group_mean = df.groupby("k")["v"].transform("mean")
print("above own group:", int((df["v"] > group_mean).sum()))
print("above table mean:", int((df["v"] > df["v"].mean()).sum()))The row with 100 is above the table mean and is exactly its own group's mean, because it is alone in its group. Which of those two numbers you report is the difference between two quite different claims.
Standardising within a group
The same idea with two transforms: each value's distance from its group mean, in units of its group's spread.
import pandas as pd
df = pd.DataFrame({"k": ["a", "a", "a", "b"], "v": [10.0, 20.0, 30.0, 5.0]})
g = df.groupby("k")["v"]
df["z"] = (df["v"] - g.transform("mean")) / g.transform("std")
print(df)Mini Practice #2: Beat your own group
Try It YourselfAdd a column holding each row's group mean, then count the rows above it. Compare that with the count above the whole table's mean.
Step 3: When to Use It
Comparing each row to its group
transform returns one value per row, aligned to the original table.
df["vs_team"] = df["score"] - df.groupby("team")["score"].transform("mean")Filling gaps with a group value
Fill missing values with the mean of the row’s own group rather than the whole column.
When Not to Use It
Do not use agg when you need a value on every row, or transform when you want one row per group. The shape of the answer decides which.
End-of-Lesson Exercises
Exercise 1: Rows above their own group
Write above_group_mean(df) returning the number of rows whose v is above the mean of their own group.
transform("mean") gives each row its group average, which lines up with the table so you can compare row by row. An aggregate would collapse the groups and leave you nothing to compare against.
Exercise 2: Each row as a share of its group
Write share_of_group(df) returning a list of each row's v as a percentage of its own group's total, rounded to one decimal place, in the original row order.
transform("sum") gives each row its group total, lined up with the table. Divide, multiply by 100, round, and tolist().