Filtering with Masks
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Filter a table with a condition on one of its columns.
- Combine conditions with
&and|, bracketed. - Say what happens to rows whose value is missing.
- Notice how many rows a filter dropped.
Why This Matters
Filtering is the operation you will reach for more than any other, and it has exactly two traps: and does not work, and rows with NaN fall out of both a condition and its opposite. Both are worth meeting on purpose here rather than in an analysis whose totals do not add up.
Step 1: A condition on a column selects rows
The mask is one True or False per row, and putting it in the brackets keeps the rows it marked — the whole row, not just the column you tested.
Mask, then select
Exactly the numpy idea from unit 3, with the difference that the unit of selection is a row.
import pandas as pd
df = pd.DataFrame({"name": ["Ada", "Grace", "Alan"], "score": [92, 88, 79]})
mask = df["score"] > 85
print(mask)
print(df[mask])Count what you dropped
A filter that leaves three rows out of five hundred is either the answer or a bug, and the only way to tell is to look at the number.
import pandas as pd
df = pd.DataFrame({"score": [92, 88, 79, 60]})
kept = df[df["score"] > 85]
print(f"{len(kept)} of {len(df)} rows kept")Mini Practice #1: How many survived
Try It YourselfFilter to scores above 70 and print how many rows you kept out of how many you started with.
Step 2: Combining conditions
Use & and | rather than and and or, and bracket each condition.
Two conditions
The brackets are required because & binds more tightly than the comparison operators.
import pandas as pd
df = pd.DataFrame({"score": [92, 88, 79, 60], "city": ["L", "Y", "L", "Y"]})
print(df[(df["score"] > 70) & (df["city"] == "L")])Negating one
~ flips a mask. It is the readable way to say "everything except", and it is easier to check than rewriting the condition backwards.
import pandas as pd
df = pd.DataFrame({"city": ["L", "Y", "L"]})
print(df[~(df["city"] == "L")])Where the missing rows go
A comparison with NaN is never True, so a row with a missing value falls out of the filter and out of its opposite. The two halves do not add up to the whole table, and nothing says so.
import pandas as pd
import numpy as np
df = pd.DataFrame({"score": [92, np.nan, 60]})
high = df[df["score"] > 80]
low = df[df["score"] <= 80]
print(len(high), "+", len(low), "=", len(high) + len(low), "of", len(df))If the two halves have to account for every row, filter on .notna() first and report the missing ones separately, rather than letting them vanish.
Mini Practice #2: The rows that vanish
Try It YourselfRun this and check whether the two halves add up to the whole table. Then fix it so they do.
Step 3: When to Use It
Keeping rows by a condition
A boolean Series selects whole rows, with every column coming along.
df[(df["score"] >= 60) & (df["subject"] == "maths")]Counting what matched
mask.sum() counts the rows a condition keeps, which is the check after every filter.
When Not to Use It
Do not assume a filter and its opposite cover every row. Rows with NaN fail both, so count the missing ones separately.
End-of-Lesson Exercises
Exercise 1: The names that passed
Write passing_names(df, mark) returning a list of names whose score is at least mark, in the order they appear.
Mask the table, then take the name column of what survives and turn it into a list with .tolist().
Exercise 2: Account for every row
Write split_counts(df, mark) returning a three-element list of [at or above the mark, below it, missing a score]. The three must add up to the number of rows.
Count the missing ones with .isna().sum() first, then the two comparisons. A comparison is never True for NaN, so those rows fall out of both and have to be counted separately.