What Data Analysis Actually Is
Overview
Four steps that turn a file of rows into an answer you can defend.
What You Will Learn in This Lesson
By the end of this lesson, you will be able to:
- Name the four steps every analysis goes through, in order.
- Turn a vague goal into a question a table can answer.
- Compute an answer from the data rather than reading it off and typing it.
- Say why the cleaning step is where most of the time and most of the mistakes go.
Why This Matters
Every tool in this course — the csv module, numpy, pandas — is a faster way of doing one of four steps. Knowing the steps first is what stops the tools feeling like a pile of unrelated functions, and it is what lets you notice when an analysis skipped one. The most common bad result is not a wrong calculation; it is a correct calculation over data nobody looked at.
Step 1: The shape of the work
Every analysis, from a class spreadsheet to a national survey, makes the same four moves. The tools change; the moves do not.
Get, clean, summarise, explain
Get the data into your program. Clean it until it means what you think it means. Summarise it into a few numbers. Explain what those numbers say — and what they do not. This unit covers getting and summarising with plain Python; unit 2 is entirely about cleaning.
rows = [
["Ada", 92],
["Grace", 88],
["Alan", 79],
]
# Get: the rows are here.
# Summarise: how many, and the best score.
print("students:", len(rows))
print("best:", max(r[1] for r in rows))Where the time actually goes
The summary is often three lines. Getting the data fit to summarise — finding the blank score, the name typed twice, the number stored as text — is most of the job. It is also where mistakes hide, because nothing raises when a column is dirty; the average simply comes out wrong.
scores = ["92", "88", ""]
# One blank value, and a sum over the column fails outright.
# Worse cases do not fail at all -- that is unit 2.
print(len(scores), "values, one of them empty:", scores.count(""))Step 2: An answer needs a question first
A table on its own says nothing. It answers questions, and only the ones precise enough to be computed.
Vague goals and computable questions
"Is this class doing well?" cannot be computed until you decide what well means. "What is the average score, and how many students are below 60?" can. Writing the question down first is also what stops you fishing — a table always contains some pattern if you look long enough.
scores = [92, 88, 79, 55]
print("average:", sum(scores) / len(scores))
print("below 60:", len([s for s in scores if s < 60]))The question decides the summary
"Who scored highest?" wants max. "How spread out are the scores?" wants the smallest and largest together. "How many passed?" wants a count. Choosing the summary is the second half of asking the question.
scores = [92, 88, 79, 55]
print("highest:", max(scores))
print("range:", min(scores), "to", max(scores))
print("passed:", len([s for s in scores if s >= 60]))Mini Practice #1: Pick the summary
Try It YourselfEach comment is a question. Replace each None with the expression that answers it from scores.
Step 3: Compute it, do not copy it
The single habit that separates analysis code from a calculator: the answer is derived from the data every time the code runs.
A typed answer is right once
print(3) is correct today. Add a fourth student and it is silently wrong, and nothing in the code tells you. print(len(rows)) is correct for every version of the data.
rows = [["Ada", 92], ["Grace", 88], ["Alan", 79], ["Katherine", 99]]
print(3) # stale the moment the data changed
print(len(rows)) # still rightReading a column out of rows
To take the largest score you need the scores on their own. A generator over the rows pulls one position out of each — the same idea the next lesson calls a column.
rows = [["Ada", 92], ["Grace", 88], ["Alan", 79]]
print(max(r[1] for r in rows))
print(min(r[1] for r in rows))Key idea: Later, pandas will do df["score"].max(). It is the same move: select a column, then summarise it. Nothing in this unit is thrown away.
Mini Practice #2: Break the typed answer
Try It YourselfRun it once. Then add a fifth row to rows and run it again. Which line went stale?
Predict, then check: A boundary can change the answer
Before you run this, predict both lists. Does a score of exactly 60 count as a pass?
scores = [59, 60, 61]
print([score for score in scores if score > 60])
print([score for score in scores if score >= 60])Show the output and explanation
[61] [60, 61]
The first condition excludes 60; the second includes it. Translate “60 or more” into >= before choosing a function. A calculation can run perfectly and still answer the wrong question.
Try one change: Change the passing threshold to 61. Predict which list becomes empty before running again.
Go deeper: Define the denominator before calculating
A percentage is a count divided by a population. Define that population first: all enrolled students, students who submitted, and students with valid scores can give different answers. A correct calculation is useful only when the labels explain exactly what was counted.
Trace the example
Predict the output before opening the explanation.
scores = [40, 60, 80, 100]
passed = sum(score >= 60 for score in scores)
print(passed)
print(100 * passed / len(scores))Show the output and trace
3 75.0
Three scores meet the inclusive boundary >= 60. Dividing 3 by 4 gives 0.75, then multiplying by 100 gives 75%. Boolean conditions contribute 1 for True and 0 for False when summed.
A mistake to watch for
If missing scores were removed earlier, this denominator represents recorded scores, not necessarily all enrolled students. Do not describe a subset as the entire population, and do not divide by len(scores) when the list is empty.
Transfer challenge
Change the threshold to 80 and report both the passing count and percentage. Write a sentence naming the population and threshold.
Change the example below to solve the challenge. Predict the result, run your code, and compare. This practice is ungraded.
Compare with a worked solution
scores = [40, 60, 80, 100]
passed = sum(score >= 80 for score in scores)
print(passed)
print(100 * passed / len(scores))Run both versions. Explain any difference in their output, then change one input and predict the new result.
Step 4: When to Use It
A question with a computable answer
When you can say what number would settle it — a count, an average, a highest — the four steps apply directly.
print(len([s for s in scores if s < 60]))Before you trust a number someone quotes
Walking back through get, clean, summarise and explain is how you check someone else’s figure as well as your own.
When Not to Use It
Do not start summarising before you have written the question down. A table always offers some pattern, and a pattern found by browsing is not a finding.
End-of-Lesson Exercises
Exercise 1: Count and top score
Three students and their scores are in rows. Print how many students there are, then the highest score.
print(3) is right today and wrong the moment a fourth student arrives. Ask len() and max() for the answers so the code still holds when the data changes.
Exercise 2: Answer three questions
Write quick_look(scores) that returns a tuple (count, lowest, passed): how many scores there are, the lowest one, and how many are 60 or more. The input is a non-empty list of numbers. Count zero scores as observations, and include exactly 60 as a pass. Return the tuple; do not just print it.
len(scores) and min(scores) give the first two. For the third, count the scores where s >= 60 — a comprehension inside len() does it in one line.