Missing Values
Overview
Find them, then decide between dropping and filling on purpose.
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Count missing values, including markers like
"n/a", before deciding anything. - Drop incomplete rows into a new list without touching the original.
- Fill missing values, and say what filling does to the summary.
- Choose between dropping and filling, and report which you chose.
Why This Matters
Every real data set has gaps, and every choice about them changes the answer. Dropping loses rows and can bias what is left; filling keeps the rows and invents values. Neither is wrong, but doing either without noticing is. pandas’ dropna and fillna in unit 5 are these exact decisions, so they are worth understanding as decisions first.
Step 1: Find them first
You cannot decide what to do about a gap you have not counted.
Blanks and markers
A missing value is not only "". Files use "n/a", "NA", "-" and more. Put every marker you know about in a set, and test membership after stripping spaces.
MISSING = {"", "n/a", "na", "-"}
scores = ["92", "", "N/A", "88", " - "]
missing = [s for s in scores if s.strip().lower() in MISSING]
print(len(missing), "of", len(scores), "missing")How many changes the decision
Two gaps in three hundred rows can be dropped without much thought. A column that is mostly empty is not a cleaning problem — it is a finding, and it should be reported.
scores = ["92", "", "", "", "88"]
share = scores.count("") / len(scores)
print(f"{share:.0%} of scores are missing")Mini Practice #1: All the markers
Try It YourselfCount how many readings are missing, treating blanks, "n/a" in any case, and "-" as missing.
Step 2: Dropping
Keep only the rows that have what you need.
Build a new list
A comprehension with a condition builds the complete rows into a new list. The original stays as it was, which matters when another part of the analysis still needs it.
rows = [{"name": "Ada", "score": "92"}, {"name": "Grace", "score": ""}, {"name": "Alan", "score": "79"}]
complete = [r for r in rows if r["score"] != ""]
print(len(rows), "->", len(complete))Never remove while looping
Removing from a list while a for loop walks it shifts every later item one place left, and the loop steps over the item that moved into the gap. Two blanks in a row and one survives.
scores = ["92", "", "", "88"]
for s in scores:
if s == "":
scores.remove(s)
print(scores) # one blank survivedStep 3: Filling, and what it costs
Filling keeps every row by putting a value where the gap was. The value you choose is a claim.
Fill with zero
Zero says the student sat the test and scored nothing. If that is not true, it drags the average down.
scores = [92, None, 88]
filled = [s if s is not None else 0 for s in scores]
print(filled, sum(filled) / len(filled))Fill with the mean
Filling with the mean of the known values leaves the mean unchanged — but every filled row sits exactly on it, so the data looks less spread out than it is.
scores = [92, None, 88, None]
known = [s for s in scores if s is not None]
mean = sum(known) / len(known)
filled = [s if s is not None else mean for s in scores]
print(filled)
print(sum(filled) / len(filled))Key idea: Whichever you choose, write it down next to the result: “2 of 30 scores were missing and were left out.” A reader cannot judge the number otherwise.
Mini Practice #2: Zero versus mean
Try It YourselfFill the gaps with 0, then with the mean of the known scores. Print the average each way and the lowest value each way.
Step 4: When to Use It
Drop when the gaps are few
A handful of incomplete rows in a large table, in the column you are analysing.
complete = [r for r in rows if r["score"] != ""]Fill when the fill value is true
Zero when zero really happened; a known default when the source documents one.
When Not to Use It
Do not fill with the mean just to keep a row count, and never remove items from a list while looping over it. Build a new list, and say how many rows you dropped.
End-of-Lesson Exercises
Exercise 1: Drop the incomplete rows
Write drop_missing(rows) that returns a new list containing only the rows whose score is not an empty string. Leave the original list alone.
Build a new list rather than removing from the one you were given. Removing while looping skips rows, and a caller who still needs their data will not thank you.
Exercise 2: Fill with the mean
Write fill_with_mean(scores) for a list of numbers and Nones. Return a new list where each None is replaced by the mean of the known values, rounded to one decimal place. If there are no known values, return the list unchanged.
known = [s for s in scores if s is not None]. If known is empty, return the list. Otherwise compute mean = round(sum(known) / len(known), 1) and build [mean if s is None else s for s in scores].