Unit 4 • Lesson 5

dtypes in pandas

Overview

What each column is holding, and how to change it on purpose.

Intermediate 20–25 min

← Previous Next: Reading a CSV into pandas

What You Will Learn in This Lesson

By the end of this lesson, you will be able to:

Read dtypes and say what each answer means.
Explain why one bad value turns a numeric column into object.
Convert with astype, and with to_numeric when the data is messy.
Find the values that would not convert, rather than guessing.

Why This Matters

A numeric column read as text sorts alphabetically, sums by concatenating and compares wrongly — and none of that raises. The dtype is the single cheapest thing to check after a load, and this lesson is the one that makes the rest of the course's numbers trustworthy.

Step 1: Ask before you calculate

dtypes tells you what every column is. A column of numbers read from a file is very often text, and nothing about printing it will show you that.

What the answers mean

int64 and float64 are numbers. object means Python objects, which for a data file almost always means strings. bool and datetime64 say what they look like.

import pandas as pd

df = pd.DataFrame({"score": ["92", "88"], "n": [1, 2]})
print(df.dtypes)
print(df)

Printed, the two columns look identical. Only the dtypes line tells you one of them is text.

What text does to arithmetic

Summing a text column concatenates it. Sorting it puts "100" before "20". Neither raises, and both are wrong.

import pandas as pd

text = pd.Series(["92", "88"])
nums = pd.Series([92, 88])
print(text.sum())
print(nums.sum())
print(sorted(["100", "20"]))

Why one bad value spoils the column

A column has one dtype. If a single value cannot be a number, the whole column falls back to object — so one stray "n/a" in ten thousand rows costs you the entire column.

import pandas as pd

print(pd.Series([1, 2, 3]).dtype)
print(pd.Series([1, 2, "n/a"]).dtype)

Mini Practice #1: Two columns that print the same

Try It Yourself

These two columns look identical printed. Show that they are not, three different ways.

import pandas as pd

df = pd.DataFrame({"a": ["1", "2"], "b": [1, 2]})

print(df)
print(df.dtypes)
print(df["a"].sum())
print(df["b"].sum())

Press Run to see output

Step 2: Converting a column

astype changes a column's type when every value can make the trip. to_numeric with errors="coerce" is for when they cannot.

astype, when the data is clean

Direct and strict. If anything in the column cannot convert, it raises — which is the right behaviour when you believe the column is clean and want to be told if it is not.

import pandas as pd

df = pd.DataFrame({"score": ["92", "88"]})
df["score"] = df["score"].astype(int)
print(df["score"].sum())

to_numeric, when it is not

errors="coerce" turns anything unconvertible into NaN instead of raising. You then decide what the NaN means — dropped, or filled — rather than having the decision made for you.

import pandas as pd

messy = pd.Series(["10", "n/a", "5"])
numbers = pd.to_numeric(messy, errors="coerce")
print(numbers)
print(numbers.sum())        # NaN is skipped by sum
print(numbers.fillna(0).sum())

Find what would not convert

Before filling anything, look at what failed. Coerce, then mask for the new NaNs against the original column — that is the list of values worth reading with your own eyes.

import pandas as pd

messy = pd.Series(["10", "n/a", "5", ""])
bad = messy[pd.to_numeric(messy, errors="coerce").isna()]
print(bad.tolist())

Usually this is a short list of two or three markers — "n/a", "-", a stray header repeated mid-file — and knowing them lets you name them at load time with na_values instead.

Mini Practice #2: Which values failed

Try It Yourself

Coerce this column to numbers and print the values that did not survive, before deciding what to do about them.

import pandas as pd

messy = pd.Series(["10", "n/a", "5", "-", "7"])
numbers = pd.to_numeric(messy, errors="coerce")

print(numbers.tolist())
print(None)   # the original values that became NaN

Press Run to see output

Step 3: When to Use It

Right after loading

Check df.dtypes before any maths: a numeric column showing object means text got in.

df["score"] = pd.to_numeric(df["score"], errors="coerce")

Categories with few values

Converting repeated labels to category saves memory and makes the allowed values explicit.

When Not to Use It

Do not use astype(int) on a column that may contain junk or missing values — it raises. Use to_numeric with errors="coerce", then decide what to do with the NaNs.

End-of-Lesson Exercises

Exercise 1: Total a messy column

Write numeric_total(df) that returns the sum of the score column, where the scores arrived as text and anything that is not a number counts as zero.

to_numeric with errors="coerce" turns the junk into missing values, and fillna(0) then makes them zero before you sum.

import pandas as pd

def numeric_total(df):
    # Sum the score column, treating junk as 0.
    return 0

print(numeric_total(pd.DataFrame({"score": ["92", "88"]})))

Press Run to try it, then Check my work to mark it.

Exercise 2: Report what would not convert

Write unconvertible(df) that returns a list of the values in the score column that cannot be read as numbers, in the order they appear. Report them rather than silently replacing them.

Coerce the column, ask which results are NaN with .isna(), and use that mask on the original column rather than on the converted one.

import pandas as pd

def unconvertible(df):
    # The original values that fail to become numbers.
    return []

print(unconvertible(pd.DataFrame({"score": ["10", "n/a", "5"]})))

Press Run to try it, then Check my work to mark it.

← Prev Next