Unit 5 • Lesson 6

Checking What You Loaded

Overview

Four questions to ask a new table before you trust a number from it.

Intermediate 20–25 min

← Previous Unit 5 test

What You Will Learn in This Lesson

By the end of this lesson, you will be able to:

Run the same four checks on any table you did not build.
Use the distinct values of a category column as a fifth check.
Say what each check would have caught.
Report what you found, rather than only acting on it.

Why This Matters

Every silent failure in this unit is caught by one of these checks, and together they take about four lines. The reason to make them a habit rather than a debugging step is that they are cheapest before you have built an analysis on top of the table, and most expensive after you have published one.

Step 1: The four questions

How many rows, what are the columns, what type is each, and how much is missing. In that order, because each one can make the next pointless.

All four, in four lines

Shape first — a surprise here means stop and fix the load. Then dtypes, then the gaps.

import pandas as pd
from io import StringIO

text = "name,score,city\nAda,92,Leeds\nGrace,,York\nAlan,79,Leeds\n"
df = pd.read_csv(StringIO(text))

print(df.shape)
print(list(df.columns))
print(df.dtypes.to_dict())
print(df.isna().sum().to_dict())

What each one would have caught

Shape catches a wrong separator and a lost header row. dtypes catches an unnamed missing marker and a numeric column read as text. The missing counts catch a column that is mostly empty — which is a fact worth reporting rather than a problem to fix silently.

import pandas as pd
from io import StringIO

# A wrong separator: one column, and the name is the whole header line.
print(pd.read_csv(StringIO("a;b\n1;2\n")).shape)

# An unnamed marker: the column is text, not numbers.
print(pd.read_csv(StringIO("v\n1\nn/a\n"))["v"].dtype)

Mini Practice #1: All four on one table

Try It Yourself

Run all four checks on this table and say what each tells you. One of the columns is not the type it should be.

import pandas as pd
from io import StringIO

text = "name,score\nAda,92\nGrace,n/a\n"
df = pd.read_csv(StringIO(text))

print(df.shape)
print(list(df.columns))
print(df.dtypes.to_dict())
print(df.isna().sum().to_dict())

Press Run to see output

Step 2: Distinct values catch the rest

For a column that should hold a handful of categories, the set of what is actually in it is the fastest way to find the fifth problem: the same thing spelled two ways.

What is actually in there

value_counts gives the distinct values and how common each is. Two spellings of one city show up immediately, and a group-by would otherwise have split them without a word.

import pandas as pd

city = pd.Series(["Leeds", "leeds", "York", "Leeds "])
print(city.value_counts())
print(city.nunique())

Three of those four are the same city. Stripping whitespace and lowercasing before grouping is usually the fix, and knowing it was needed is the point of looking.

A range check on the numbers

For a numeric column, min and max are the cheap version of the same question: are these values possible? A score of 999 or a negative age is a sentinel or an error, and both are easier to see here than in a mean.

import pandas as pd

scores = pd.Series([92, 88, 999])
print(scores.min(), scores.max(), round(scores.mean(), 2))

Mini Practice #2: Find the duplicate spelling

Try It Yourself

Count the distinct cities, then count them again after stripping and lowercasing. The two numbers should differ.

import pandas as pd

city = pd.Series(["Leeds", "leeds", "York", "Leeds "])

print(city.nunique())
print(None)   # after .str.strip().str.lower()

Press Run to see output

Step 3: When to Use It

Every time a file is read

Shape, dtypes, missing counts and a look at the first rows — before any number is trusted.

print(df.shape, df.isna().sum())

When the data is updated

The same checks catch a new column, a changed marker or a doubled file.

When Not to Use It

Do not skip the checks because the file loaded last time. Files change; the checks are cheap and the wrong answer is not.

End-of-Lesson Exercises

Exercise 1: A load report

Write load_report(path) that reads a CSV and returns a tuple of the row count, the sorted column names, and how many values are missing in the whole table.

df.isna().sum() gives a count per column; summing that again gives the total. int() at the end keeps it a plain number.

import pandas as pd

def load_report(path):
    # (rows, [columns], missing values)
    return None

print(load_report("scores.csv"))

Press Run to try it, then Check my work to mark it.

Exercise 2: How many distinct, really

Write distinct_cities(path) that reads a file with a city column and returns how many distinct cities it holds, treating values that differ only by surrounding spaces or capitals as the same.

Read the column, then .str.strip().str.lower() before .nunique(). Counting without normalising first is the mistake the first hidden case checks.

import pandas as pd

def distinct_cities(path):
    # Distinct cities, ignoring case and surrounding spaces.
    return 0

print(distinct_cities("places.csv"))

Press Run to try it, then Check my work to mark it.

← Prev Unit 5 test