Selecting What You Need
Overview
Read the columns you want, and stop carrying the ones you do not.
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Read only the columns you need with
usecols. - Tidy shouty or spaced column names in one pass.
- Say what a smaller table buys beyond memory.
- Keep an id column as text so it does not lose its leading zeros.
Why This Matters
A real export is forty columns and you want six. Carrying the other thirty-four costs memory, but more importantly it costs attention: every one of them is a column that can surprise you later, and none of them is a column you checked.
Step 1: usecols on the way in
A wide export is mostly columns you will never touch. Naming the ones you want keeps the table small and the checks short.
Ask for what you need
usecols takes the names, and the resulting table has only those. The rest of the file is never turned into columns at all.
import pandas as pd
from io import StringIO
text = "name,city,score,notes\nAda,Leeds,92,ignore me\n"
df = pd.read_csv(StringIO(text), usecols=["name", "score"])
print(df)
print(df.shape)The type of an id
An id made of digits will be read as a number, and a number has no leading zeros. dtype= keeps it as text so 007 stays 007.
import pandas as pd
from io import StringIO
text = "id,score\n007,92\n"
print(pd.read_csv(StringIO(text))["id"].iloc[0])
print(pd.read_csv(StringIO(text), dtype={"id": str})["id"].iloc[0])This also matters for joining. If one file keeps "007" as text and another reads it as the number 7, merging them raises an error in unit 8 — and converting the number back with astype(str) gives "7", which matches nothing and raises nothing.
Mini Practice #1: Two of six
Try It YourselfRead only the name and score columns from this wide file, and confirm the shape.
Step 2: Renaming as you go
Column names from real systems are often shouty, spaced, or both. rename takes a dictionary from old name to new.
One name at a time
A small mapping for the handful you actually care about.
import pandas as pd
df = pd.DataFrame({"Full Name": ["Ada"], "SCORE": [92]})
print(list(df.rename(columns={"Full Name": "name"}).columns))A rule for all of them
When every name needs the same treatment, build the mapping with a comprehension rather than typing forty pairs.
import pandas as pd
df = pd.DataFrame({"Full Name": ["Ada"], "SCORE": [92]})
tidy = df.rename(columns={c: c.lower().replace(" ", "_") for c in df.columns})
print(list(tidy.columns))Mini Practice #2: Tidy every name
Try It YourselfLowercase every column name and turn spaces into underscores, in one rename.
Step 3: When to Use It
Wide files
usecols reads only the columns the analysis needs, which is faster and easier to check.
pd.read_csv(path, usecols=["id", "score"])Large files
nrows reads a sample first, so you can inspect the shape before loading everything.
When Not to Use It
Do not drop columns you might need to explain a result, such as the one that says where a row came from. Reading less is for columns you are sure about.
End-of-Lesson Exercises
Exercise 1: Tidy the column names
Write tidy_columns(df) that returns a new table whose column names are all lowercase with spaces turned into underscores.
Build a dictionary from each existing name to its tidy version, then hand it to rename. str.lower() and str.replace(" ", "_") do the work.
Exercise 2: Read only what you need
A wide file wide.csv is supplied. Write narrow(path, keep) that reads only the columns named in keep and returns the resulting shape as a tuple. Read the columns you want rather than reading everything and dropping the rest.
pd.read_csv(path, usecols=keep) then .shape. Reading everything and then selecting gives the same shape but does all the work you were trying to avoid.