PyPath
Start learning
Unit 2 • Lesson 4

Removing Duplicates

Overview

Deciding what counts as the same row is most of the work.

Beginner 20–25 min
← Previous Next: Filtering Rows

What You Will Learn in This Lesson

By the end of this lesson, you will be able to:

  • Remove exact repeats while keeping the original order.
  • Normalise text with strip and lower so the same value is recognised.
  • Keep the first spelling of each value while comparing a normalised key.
  • Deduplicate whole records on the fields that identify them.

Why This Matters

A student entered twice is counted twice, in every count and every average, and nothing warns you. Removing duplicates is easy; the real work is deciding what “the same” means — the same name in a different case, the same person with a different score — and that decision is the one pandas’ drop_duplicates(subset=...) will ask you to make explicitly in unit 6.

Step 1: Exact duplicates

Start with the easy case: values that are character-for-character identical.

1

set() removes repeats and the order

A set keeps one of each value, which answers “how many distinct?” instantly. It has no order, so it is the wrong tool when the order of the rows means something.

names = ["Grace", "Ada", "Grace", "Alan"]
print(len(names), "rows,", len(set(names)), "distinct")
print(set(names))
2

A seen-set keeps the order

Walk the list, remember what you have already kept, and keep a value only the first time. The set is for fast membership tests; the list is the result.

names = ["Grace", "Ada", "Grace", "Alan"]

seen = set()
unique = []
for n in names:
    if n not in seen:
        seen.add(n)
        unique.append(n)
print(unique)

Mini Practice #1: Two ways to count

Try It Yourself

Print how many distinct cities there are with set, then print the distinct cities in their original order.

Press Run to see output

Step 2: Same value, written differently

“Ada”, “ada” and “ Ada ” are one person to a human and three strings to Python.

1

Normalise to compare

strip() removes surrounding spaces and lower() removes case. The normalised form is a key for deciding sameness.

names = ["Ada", "ada", " Ada ", "GRACE"]
keys = [n.strip().lower() for n in names]
print(keys)
print(len(set(keys)), "people")
2

Compare the key, keep the original

The output should still say "Ada", not "ada". Use the key for the seen-set and append the original text.

names = ["Ada", " ada ", "Grace", "GRACE"]

seen = set()
kept = []
for n in names:
    key = n.strip().lower()
    if key not in seen:
        seen.add(key)
        kept.append(n)
print(kept)

Mini Practice #2: Messy subjects

Try It Yourself

Count the subjects as a human would, ignoring case and spaces.

Press Run to see output

Step 3: Duplicate records

For whole rows, decide which fields identify a record. Two rows can share a name and be different people, or differ in a score and be the same entry typed twice.

1

A tuple of fields as the key

Dictionaries cannot go in a set, but a tuple of their identifying values can. Choose the fields that define “the same record” — here, name and date.

rows = [
    {"name": "Ada", "date": "2024-01-05", "score": 92},
    {"name": "Ada", "date": "2024-01-05", "score": 92},
    {"name": "Ada", "date": "2024-02-01", "score": 95},
]

seen = set()
unique = []
for r in rows:
    key = (r["name"], r["date"])
    if key not in seen:
        seen.add(key)
        unique.append(r)
print(len(rows), "->", len(unique))
2

Report what you removed

The number of duplicates removed is part of the result. A few is housekeeping; a third of the file means something upstream is wrong.

names = ["Ada", "Grace", "Ada", "Ada"]
unique = list(dict.fromkeys(names))
print("removed", len(names) - len(unique), "duplicates of", len(names), "rows")

Key idea: dict.fromkeys(values) is a one-line order-keeping deduplication for exact matches — dictionary keys are unique and keep insertion order. It cannot normalise, so the loop is still the tool when case and spaces matter.

Step 4: When to Use It

Files that were appended to

Exports run twice, forms submitted twice. Deduplicate on the fields that identify a record.

key = (r["name"], r["date"])

Text typed by people

Normalise case and spaces to see that “Ada” and “ ada ” are one value, while keeping the original spelling.

When Not to Use It

Do not deduplicate on every field when two real records can share them, and do not use set() when order matters. Decide what “the same” means first.

End-of-Lesson Exercises

Exercise 1: Unique names, first spelling kept

Write unique_names(names) that returns the names with duplicates removed, ignoring case and surrounding spaces, keeping the first spelling of each and the original order.

Two things at once: a normalised key for deciding sameness, and the original text for the output. Keep a set of keys you have already seen.

Press Run to try it, then Check my work to mark it.

Exercise 2: Duplicate records by key

Write dedupe(rows, fields) that returns a new list of the rows (dictionaries) with repeats removed, where two rows are the same if they agree on every field named in fields. Keep the first of each and the original order.

For each row build key = tuple(row[f] for f in fields). Keep the row only if key is not already in a seen set, then add it.

Press Run to try it, then Check my work to mark it.
← Prev Next