Change Over Time
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Compare a row with the one before it.
- Compute a growth rate and read it correctly.
- Say what happens to the first row.
- Recognise when a percentage change is undefined.
Why This Matters
Change is what most time-series questions are actually about, and both ways of measuring it have a row that cannot be answered — the first one, which has nothing before it. Filling that row with zero is the commonest quiet lie in a growth table.
Step 1: shift compares a row with its neighbour
diff is the difference from the previous row and pct_change is that difference as a proportion. Both are built on shift, which simply moves the column down by one.
The three of them
Seeing shift beside diff makes it obvious where the missing first row comes from.
import pandas as pd
s = pd.Series([100.0, 120.0, 90.0])
print(s.shift().tolist())
print(s.diff().tolist())
print(s.pct_change().tolist())A proportion, not a percentage
pct_change returns 0.2 for a rise of a fifth. Multiply by 100 only when you format it for a reader, not while you are still calculating.
import pandas as pd
s = pd.Series([100.0, 120.0])
change = s.pct_change()
print(change.tolist())
print([f"{x:.1%}" for x in change.dropna()])Mini Practice #1: Three views of the same change
Try It YourselfPrint the shift, the diff and the percentage change, and line up where each one is missing.
Step 2: A percentage needs a base
Growth from zero is undefined, and pandas says so with an infinity rather than an error.
Division by zero
inf is not a number you can put in a report. It means "grew from nothing", which is a sentence, not a percentage.
import pandas as pd
s = pd.Series([0.0, 5.0])
print(s.pct_change().tolist())The first row, and what to do with it
It has no predecessor, so it has no change. Drop it or report it as unknown; filling it with 0 claims a flat first period that was never measured.
import pandas as pd
s = pd.Series([100.0, 120.0, 90.0])
changes = s.diff()
print(changes.tolist())
print(changes.dropna().tolist())
print(changes.fillna(0).tolist(), "<- claims no change in period one")Counting the rises
A useful summary that sidesteps the whole problem: a missing first difference is not greater than zero, so it excludes itself.
import pandas as pd
s = pd.Series([1, 2, 1, 3])
print(s.diff().tolist())
print("rises:", int((s.diff() > 0).sum()))Mini Practice #2: The undefined row
Try It YourselfRun this and look at both awkward values — the missing first one and the infinite second.
Step 3: When to Use It
Step-by-step change
diff gives the change from one row to the next; pct_change the growth rate.
df["growth"] = df["sales"].pct_change()Comparing with a year ago
shift(12) on monthly data lines each month up with the same month last year.
When Not to Use It
Do not report a percentage change from zero or near zero. It is infinite or huge, and a sentence about the absolute change says more.
End-of-Lesson Exercises
Exercise 1: Count the rises
Write rises(s) that returns how many times the value went up compared with the row before it.
diff() gives the change at each step, with the first one missing. Count how many of those are above zero — a missing value is not, so it takes care of itself.
Exercise 2: Growth, honestly reported
Write growth(s) returning the percentage change at each step as a list rounded to one decimal place, with the undefined first step dropped rather than filled. The values are percentages, so 0.2 is reported as 20.0.
pct_change() gives proportions with the first row missing. Multiply by 100, round to one place, dropna() and tolist(). Filling the first row with 0 would claim a flat period nobody measured.