Unit 7 • Lesson 2

Counting Groups

Overview

How many of each, and the difference between size and count.

Intermediate 20–25 min

← Previous Next: agg with Several Summaries

What You Will Learn in This Lesson

By the end of this lesson, you will be able to:

Count the distinct values in a column with value_counts.
Say the difference between size and count.
Turn counts into proportions.
Choose which one answers the question you were asked.

Why This Matters

"How many of each" is the most common question anyone asks of a table, and there are two right answers to it depending on whether you mean rows or values. On a column with gaps in it the two differ, and picking the wrong one gives a number that is plausible and wrong.

Step 1: value_counts for one column

For a single column this is the shortest path: distinct values and how often each appears, already sorted by frequency.

Counts, and shares

normalize=True turns the counts into proportions of the whole, which is usually what a reader wants beside them.

import pandas as pd

city = pd.Series(["L", "Y", "L", "L"])
print(city.value_counts())
print(city.value_counts(normalize=True))

Missing values are not counted by default

So the counts can sum to less than the length of the column. dropna=False makes the gaps a category of their own.

import pandas as pd
import numpy as np

city = pd.Series(["L", np.nan, "Y"])
print(city.value_counts().sum(), "of", len(city))
print(city.value_counts(dropna=False))

Mini Practice #1: Counts and shares

Try It Yourself

Print the counts for this column and then the same thing as proportions.

import pandas as pd

city = pd.Series(["L", "Y", "L", "L", "Y", "B"])

print(city.value_counts())
print(None)   # as proportions

Press Run to see output

Step 2: size counts rows, count counts values

On a group, size is how many rows are in it. count is how many non-missing values a column has in it. They differ exactly where the data has holes.

The difference, shown

Ten rows with three missing scores give a size of 10 and a count of 7. Which one you want depends on whether you are asking "how many students" or "how many marked papers".

import pandas as pd
import numpy as np

df = pd.DataFrame({"k": ["a", "a", "a"], "v": [1, np.nan, 3]})
g = df.groupby("k")
print("size:", g.size().to_dict())
print("count:", g["v"].count().to_dict())

Both at once

Asking for both is usually the honest answer, because the gap between them is itself information about the data.

import pandas as pd
import numpy as np

df = pd.DataFrame({"k": ["a", "a", "b"], "v": [1, np.nan, 3]})
print(df.groupby("k")["v"].agg(["size", "count", "mean"]))

Note the mean: it is computed over the count, not the size. A group of three rows with one missing value has its mean taken over two.

Mini Practice #2: Where they differ

Try It Yourself

Print size, count and mean for each group and explain to yourself why the mean is what it is.

import pandas as pd
import numpy as np

df = pd.DataFrame({"k": ["a", "a", "a", "b"], "v": [10, np.nan, 20, 5]})

print(df.groupby("k")["v"].agg(["size", "count", "mean"]))

Press Run to see output

Step 3: When to Use It

Rows per group

size counts every row, including those with missing values.

df.groupby("team").size()

Values per group

count counts the non-missing values in a column — how many actually have a score.

When Not to Use It

Do not use size and count interchangeably. When a column has gaps they give different answers, and the difference is the number of missing values.

End-of-Lesson Exercises

Exercise 1: How many of each

Write counts_by(df, column) returning a dictionary of each distinct value in that column to how many rows have it.

value_counts on the column gives exactly this. to_dict() makes the keys and values plain, though the counts may need int() to shed their numpy type.

import pandas as pd

def counts_by(df, column):
    # {value: how many rows}
    return {}

print(counts_by(pd.DataFrame({"s": ["a", "a", "b"]}), "s"))

Press Run to try it, then Check my work to mark it.

Exercise 2: Rows against marked papers

Write size_and_count(df) returning a dictionary of each key to a two-element list of [rows in the group, rows whose v is not missing]. Where a group has gaps the two numbers differ, and that difference is the point.

Group once, then ask the grouped object for .size() and the column for .count(). size counts rows; count counts values that are not missing.

import pandas as pd

def size_and_count(df):
    # {key: [rows, non-missing values]}
    return {}

print(size_and_count(pd.DataFrame({"k": ["a", "a"], "v": [1, None]})))

Press Run to try it, then Check my work to mark it.

← Prev Next