Filtering Rows
Overview
Selecting the rows you meant, and noticing how many you dropped.
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Write a filter as a comprehension with a condition.
- Combine conditions with
and,orandnotwithout changing their meaning. - Get the boundary right:
>=versus>. - Check how many rows a filter kept, and name the rule in one place.
Why This Matters
Filtering decides who is in the analysis, so it changes the answer more than any calculation after it. A pass mark written as > instead of >= fails everyone exactly on the line, and marks cluster on the line. The boolean masks you meet in numpy (unit 3) and pandas (unit 6) are the same conditions applied to a whole column at once — the logic, and the traps, carry straight over.
Step 1: A filter is a comprehension with a test
Keep the rows for which a condition is true.
One condition
The if at the end of a comprehension keeps a row only when the test holds. The rows themselves come through whole.
rows = [{"name": "Ada", "score": 92}, {"name": "Alan", "score": 58}, {"name": "Grace", "score": 88}]
passing = [r for r in rows if r["score"] >= 60]
print([r["name"] for r in passing])The boundary
“At least 60” includes 60. > 60 silently excludes everyone who scored exactly 60, and grades pile up at pass marks because people are marked up to them.
scores = [59, 60, 60, 61, 75]
print(len([s for s in scores if s >= 60]))
print(len([s for s in scores if s > 60]))Mini Practice #1: Where the boundary bites
Try It YourselfCount how many students pass with >= 50 and with > 50. How many students does the wrong operator fail?
Step 2: Combining conditions
Real filters are rarely one test.
and, or, not
and needs both; or needs either; not flips a test. Parentheses make the intent visible and stop precedence surprises — and binds tighter than or.
rows = [
{"name": "Ada", "subject": "maths", "score": 92},
{"name": "Alan", "subject": "maths", "score": 58},
{"name": "Grace", "subject": "art", "score": 88},
]
print([r["name"] for r in rows if r["subject"] == "maths" and r["score"] >= 60])
print([r["name"] for r in rows if r["subject"] == "art" or r["score"] < 60])A range
Python allows a chained comparison, 60 <= s < 70, which reads like the maths and avoids repeating the value.
scores = [55, 60, 65, 70, 90]
print([s for s in scores if 60 <= s < 70])Mini Practice #2: Two conditions
Try It YourselfPrint the names of students who study computing and scored between 70 and 89 inclusive.
Step 3: Name the rule, count the result
Two habits that stop filters quietly going wrong.
Say the condition once
If “passing” is written in three places, one of them will be edited and the others will not. Put the rule in a function and use it everywhere.
PASS_MARK = 60
def passed(row):
return row["score"] >= PASS_MARK
rows = [{"score": 92}, {"score": 58}]
print(len([r for r in rows if passed(r)]))
print(len([r for r in rows if not passed(r)]))Check the counts add up
A filter and its opposite should account for every row. If they do not, something in the data — a missing score, say — is falling through both.
rows = [{"score": 92}, {"score": 58}, {"score": None}]
kept = [r for r in rows if r["score"] is not None and r["score"] >= 60]
dropped = [r for r in rows if r["score"] is not None and r["score"] < 60]
print(len(kept), "+", len(dropped), "of", len(rows))Key idea: Printing before-and-after counts is the cheapest check in data work. A filter that keeps 3 rows of 900 is almost always a mistake in the condition, not a finding.
Step 4: When to Use It
Selecting who is in the analysis
Pass marks, date ranges, one category. Put the rule in one function and use it everywhere.
passing = [r for r in rows if r["score"] >= PASS_MARK]Splitting a table into parts
A filter and its opposite, with a check that the counts add up to the whole.
When Not to Use It
Do not test for a missing value with “if not score”: a real 0 is falsy too. Use “is None”, and print the before-and-after row counts.
End-of-Lesson Exercises
Exercise 1: Rows at or above the mark
Write passing(rows, mark) that returns the rows whose score is at least mark.
At least the mark means the mark passes. A plain greater-than silently fails everyone sitting exactly on the boundary, which is where marks cluster.
Exercise 2: Pass, fail and unknown
Write split_results(rows, mark) returning a tuple (passed, failed, unknown) of counts. A row passes when its score is at least mark, fails when it is below, and is unknown when the score is None. The three counts must add up to the number of rows.
Test for None first, with "is None", so that 0 is not mistaken for missing. Only then compare with >= mark.