Parsing Dates
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Convert a column of date text into real datetimes.
- Parse at load time rather than afterwards.
- Handle a value that will not parse.
- Say why day-first and month-first matter.
Why This Matters
Text that looks like a date sorts alphabetically and compares alphabetically, so "02/2024" comes before "1/2023" and every ordering you build on it is wrong. Worse, an ambiguous format can be parsed the wrong way round and still succeed, giving you a year of data where every day up to the twelfth is a different date than you think.
Step 1: to_datetime turns text into dates
Until it does, sorting is alphabetical and "greater than" means something other than "later".
What text does to order
Run this once. The text ordering is not an approximation of the date ordering; it is unrelated to it.
import pandas as pd
text = pd.Series(["10/01/2024", "02/01/2024", "1/03/2023"])
print(sorted(text.tolist()))
print(pd.to_datetime(text, dayfirst=True).sort_values().dt.strftime("%Y-%m-%d").tolist())The conversion
Once converted, the column has a datetime dtype and every comparison means what you expect.
import pandas as pd
days = pd.to_datetime(pd.Series(["2024-03-01", "2024-01-15"]))
print(days.dtype)
print(days.min(), days.max())
print(days.max() - days.min())Mini Practice #1: Text order against date order
Try It YourselfSort these as text and as dates, and compare the two answers.
Step 2: Parsing on the way in
parse_dates on read_csv does it at load time, which is one fewer step to forget.
At load time
Cheaper than converting afterwards, and it means the column is never text long enough for you to sort it by accident.
import pandas as pd
from io import StringIO
text = "day,n\n2024-01-05,1\n2024-02-01,2\n"
df = pd.read_csv(StringIO(text), parse_dates=["day"])
print(df.dtypes)When a value will not parse
errors="coerce" gives NaT — the datetime version of NaN — instead of raising. Counting them tells you how bad the column is before you decide what to do.
import pandas as pd
messy = pd.Series(["2024-01-05", "not a date", "2024-02-01"])
days = pd.to_datetime(messy, errors="coerce")
print(days.tolist())
print("unparseable:", int(days.isna().sum()))The ambiguity that does not raise
03/04/2024 is the third of April or the fourth of March, and both readings succeed. If the file is British, say so — otherwise every date whose day is 12 or under is silently wrong.
import pandas as pd
value = pd.Series(["03/04/2024"])
print(pd.to_datetime(value, dayfirst=False).dt.strftime("%d %B %Y").iloc[0])
print(pd.to_datetime(value, dayfirst=True).dt.strftime("%d %B %Y").iloc[0])Mini Practice #2: Count what will not parse
Try It YourselfCoerce this column and report how many values failed, before deciding anything.
Step 3: When to Use It
Any date column from a file
to_datetime turns text into real dates that sort, compare and subtract correctly.
df["date"] = pd.to_datetime(df["date"])Known formats
Pass format= when you know it; it is faster and refuses dates that do not fit.
When Not to Use It
Do not let an ambiguous date like 03/04/2024 be guessed. Say the format or dayfirst=True, or March and April swap silently.
End-of-Lesson Exercises
Exercise 1: The earliest date
Write earliest(dates) that takes a Series of date strings and returns the earliest as a string in YYYY-MM-DD form.
Parse first, then take the minimum, then format it back. On ISO-formatted text the alphabetical answer happens to agree, which is exactly why the habit matters when the format changes.
Exercise 2: How many dates are usable
Write usable_dates(values) that takes a Series of date strings and returns a two-element list of [how many parsed, how many did not]. Report the bad ones rather than dropping them quietly.
to_datetime with errors="coerce" turns the failures into NaT instead of raising. notna() counts the good ones and isna() the bad.