What Dirty Data Looks Like

The four problems you will meet in almost every real file.

Four recurring problems

Missing values, wrong types, duplicates, and the same thing written several ways. Every one of them is quiet: nothing raises, the summary just comes out wrong.

rows = [
    {"name": "Ada", "score": "92"},
    {"name": "Grace", "score": ""},
    {"name": "ada", "score": "92"},
]
print(len(rows))

Look before you summarise

Printing a few rows and the set of values in a column costs nothing and catches most of it. An average computed over dirty data is a confident wrong answer.

scores = ["92", "", "88"]
print(set(scores))
print([s for s in scores if s == ""])

Exercise

Try It Yourself

Write count_missing(rows) that returns how many rows have an empty string for score.

def count_missing(rows):
    # How many rows have "" for score?
    return 0

print(count_missing([{"score": "92"}, {"score": ""}]))

Press Run to see output

Check Your Understanding

← Previous Next