Read failures

What you will be able to do

Find the cause of two real failures in ticket-cleaner. One stops the program with a traceback. The other gives a wrong result with no error at all. The method is the same for every bug. Collect facts, find where expected and actual differ, and make the failure small enough to show its cause.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has both investigations, the task and its check, with the instructions. Its first cells make the ticket-cleaner folder as at the start of this lesson. They also install httpx and download the data files. So the notebook works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Environment variables and the settings module, from Configuration and secrets.
• read_rows and split_records, and text encodings, from Reading and writing files.
• Interactive mode (>>>), from Basic HTTP client use: Worked example.

What you need on your computer (only for the local route; the notebook needs none of it)

• ticket-cleaner reads its source from TICKETS_SOURCE: ticket_cleaner/config.py and the new count_tickets.py. You made them in Configuration and secrets: Guided practice. If they are missing or different, do that section again. You can also copy the files from the setup cell of this lesson’s notebook. It has every file as at the start of this lesson.
• TICKETS_SOURCE is not set in your terminal. Remove it with unset TICKETS_SOURCE (macOS and Linux) or Remove-Item Env:TICKETS_SOURCE (PowerShell), as in Configuration and secrets: Environment variables.
• data/tickets.csv and data/tickets.json. You downloaded them in Reading and writing files: Get the data file. If one is missing or changed, download it again with the same curl command. The checksums are in the dataset card. This lesson also downloads data/tickets-excel.csv, in Failure 2.
• A terminal in ~/projects/ticket-cleaner. You made the folder in Files and terminal: Guided practice.
• The project’s environment .venv, active in your terminal (the prompt starts with (.venv)). You made it in Isolate dependencies: Guided practice. If it is missing or broken, delete it and make it again from the lock file, as in Make the environment again from nothing.

Tested with Python 3.14.6 on macOS. The line numbers in your tracebacks can be different from the ones here, if your files have more or fewer lines. The notebook was tested in Jupyter.

A method, not luck

When a program fails, it is tempting to change something and run it again, until it works. Sometimes that works, but you do not learn the cause, and the bug often comes back. A better method has a fixed order:

From symptom to cause: a crash and a wrong result start differently, but both end with a failure that you can repeat on purpose.

Text description of the diagram

Start with the symptom. If the program stops with a traceback, read the last line for the error type and message. Then find the last frame that is in your own code. If the program gives a wrong result, write down the expected and the actual output. Then find the first step where they differ, with print or a breakpoint. Both paths then ask: can you make the failure happen again on purpose? If not, collect more facts (input, settings, versions) and ask again. If yes, shrink the input until only the cause is left. That is a minimal failing example. Fix the bug, and keep the example as a regression test.

Failure 1: a traceback

A teammate sends a JSON file from a different system. Make it in your project as data/broken.json:

data/broken.json

[
  {"id": "T-3001", "status": "open", "category": "billing", "priority": 2},
  null,
  {"id": "T-3003", "status": "closed", "category": "login", "priority": 3}
]

Run the program with it. On macOS and Linux, you can set the variable for one command:

macOS

TICKETS_SOURCE=data/broken.json python count_tickets.py

Windows

$env:TICKETS_SOURCE = "data/broken.json"
python count_tickets.py

Linux

TICKETS_SOURCE=data/broken.json python count_tickets.py

Output

Source: data/broken.json
Traceback (most recent call last):
  File "/Users/amira/projects/ticket-cleaner/count_tickets.py", line 34, in <module>
    main()
    ~~~~^^
  File "/Users/amira/projects/ticket-cleaner/count_tickets.py", line 23, in main
    tickets, rejected = split_records(rows)
                        ~~~~~~~~~~~~~^^^^^^
  File "/Users/amira/projects/ticket-cleaner/ticket_cleaner/parsing.py", line 50, in split_records
    problems = find_problems(raw)
  File "/Users/amira/projects/ticket-cleaner/ticket_cleaner/parsing.py", line 21, in find_problems
    if clean_text(raw.get(field)) == "":
                  ^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

A traceback is the path of function calls that led to the error. Each call is a frame: a file, a line number, the function name, and the line of code. The first frame is the start of the program, and the last frame is where the error occurred. So read it from the bottom up:

Read	What it tells you
The last line	The error type, AttributeError, and the message: a value of type NoneType has no .get. The only value of type NoneType is None.
The last frame	In find_problems, the line raw.get(field). The ^^^^ marks show the exact part: raw.get. So raw is None.
The frame above it	split_records called find_problems(raw), with one of the rows.
Higher frames	How the program got there: main() called split_records. Read these only if you need them.

The facts so far: one row is None. Where does it come from? json.load turns JSON null into None, and the second element of the file is null. So the cause is in the input: a record that is not a dictionary. find_problems assumed that every row is a dictionary, and its contract did not say what to do otherwise.

This is invalid input, not a defect in the program’s logic. Input from outside can contain anything, so the program must reject it with a reason, like any other bad record. Add this check at the start of find_problems, after the docstring:

ticket_cleaner/parsing.py (in find_problems)

    if not isinstance(raw, dict):
        return ["not a record"]

isinstance(raw, dict) is True if raw is a dictionary. The early return stops before any raw.get. Run the program again:

Output (first lines)

Source: data/broken.json
Row 2 rejected: not a record
3 rows: 2 valid, 1 rejected.

Note

When the last frame is in a module that you did not write, for example in httpx or json, do not start there. Go up to the last frame that is in your files. That is where your code gave the library a value that it could not use.

Failure 2: a wrong result with no error

The help desk exports its tickets from a spreadsheet program. Download that file:

macOS

curl -o data/tickets-excel.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets-excel.csv

Windows

curl.exe -o data/tickets-excel.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets-excel.csv

Linux

curl -o data/tickets-excel.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets-excel.csv

Open it in VS Code. It looks the same as tickets.csv. Run the program with it (set TICKETS_SOURCE=data/tickets-excel.csv as before):

Output (last lines)

Row 14 rejected: missing id; missing category
Row 15 rejected: missing id
15 rows: 0 valid, 15 rejected.
Report: reports/summary.json

There is no traceback. The program says that every row is missing its ID, but you can see the IDs in the file. Follow the right side of the decision tree.

Expected and actual. Expected: 9 valid, 6 rejected, as for tickets.csv. Actual: 0 valid, every row missing id.

Find the first step where they differ. find_problems says “missing id”, so look at what it receives. You can add print(raw), but a breakpoint is often faster. It stops the program at a line, and you can look at every variable before it continues. Add this line as the first line of find_problems, after the docstring:

ticket_cleaner/parsing.py (temporary)

    breakpoint()

Run the program again. It stops, and shows the debugger prompt (Pdb). Type a command after the prompt, and press Enter:

Debugger session

> /Users/amira/projects/ticket-cleaner/ticket_cleaner/parsing.py(19)find_problems()
-> breakpoint()
(Pdb) p raw
{'\ufeffid': 'T-1001', 'status': 'open', 'category': 'billing', 'priority': '2', 'subject': 'Charged twice for March'}
(Pdb) p list(raw)
['\ufeffid', 'status', 'category', 'priority', 'subject']
(Pdb) p raw.get("id")
None
(Pdb) q

Command	Does
p expression	Prints the value of any expression, with the variables at this point.
l	Lists the code around the current line.
n	Runs the next line, then stops again.
c	Continues to the next breakpoint, or to the end.
q	Quits the program. If Python asks Quit anyway? [y/n], type y.

There it is: the first key is not 'id', but '\ufeffid'. There is an invisible character before id. Remove the breakpoint() line now. A breakpoint that you forget stops the program for every user.

Make it small. Is the problem the file, or the code? A minimal failing example is the smallest input that still shows the failure. Make one in interactive Python (python, then type each line):

Interactive Python

>>> import csv
>>> with open("tiny.csv", "w", encoding="utf-8-sig") as file:
...     file.write("id,status\nT-1,open\n")
...
19
>>> with open("tiny.csv", encoding="utf-8", newline="") as file:
...     print(list(csv.DictReader(file)))
...
[{'\ufeffid': 'T-1', 'status': 'open'}]

(19 is the number of characters that write wrote. Interactive mode shows it.) Two lines of data, and the failure is still there. The only special thing is encoding="utf-8-sig" when the file was written. That is how spreadsheet programs save CSV. They put three bytes at the start of the file, to say “this file is UTF-8”. These bytes are a byte order mark (BOM). Reading with encoding="utf-8" keeps the mark as the character \ufeff, which becomes part of the first column name.

Fix. The encoding utf-8-sig reads UTF-8 and removes a byte order mark if there is one. A file without a mark reads the same as before. Change the CSV line in read_rows:

ticket_cleaner/files.py (in read_rows)

        with open(path, encoding="utf-8-sig", newline="") as file:

Run the program again with data/tickets-excel.csv. The result is 15 rows: 9 valid, 6 rejected., the same as for tickets.csv. Delete tiny.csv.

Guided practice: both failures

Do both investigations on your own computer, in order.

1. Make data/broken.json, and run the program with it. Read the traceback from the bottom up. Write down the error type and the last frame in your own code.
2. Add the isinstance check to find_problems. Run again. Row 2 is rejected as not a record.
3. Download data/tickets-excel.csv, and run the program with it. Write down expected and actual.
4. Add breakpoint() to find_problems. Run, and use p raw and p list(raw). Quit with q.
5. Remove the breakpoint() line.
6. Make the minimal failing example in interactive Python.
7. Change the encoding to utf-8-sig, and run again. The result is 9 valid and 6 rejected.
8. Remove TICKETS_SOURCE (unset TICKETS_SOURCE, or Remove-Item Env:TICKETS_SOURCE), and run once more with the default file, to check that it still works.

Your turn: semicolons

In some countries, spreadsheet programs separate values with ; instead of ,. Make data/tickets-semicolon.csv with this text:

data/tickets-semicolon.csv

id;status;category;priority;subject
T-4001;open;billing;2;Wrong invoice total
T-4002;closed;login;3;Locked out

Run the program with it. Use the decision tree: write down expected and actual, find the first step where they differ, and make a minimal failing example. You do not need to fix it. Write two sentences: the cause, and what a fix would need to do.

Show the cause

Both rows are rejected with missing id; missing status; missing category; missing priority. p list(raw) at a breakpoint shows one key: 'id;status;category;priority;subject'. csv.DictReader splits on , by default, so it reads each line as one column. A fix must tell the reader the separator: csv.DictReader(file, delimiter=";"). A program could also detect it, but then it needs a rule for files that contain both characters. For ticket-cleaner, rejecting the file with a clear message is enough, and every row is already reported.

Common mistake

Changing code until the error goes away

Symptom: after many small changes, the error is gone, but you do not know why. Or a different error appears in a place that worked before.

Cause: the changes were guesses, not answers to facts. Some changes only hide the symptom, for example raw = raw or {} for the None row. The error goes away, but a clear failure becomes a misleading result. The row is now rejected as missing id; missing status; missing category; missing priority, and the real problem, a null in the file, is hidden.

Fix:

1. Undo the guesses. If you use Git, git diff shows them, and git restore removes them.
2. Go back to the decision tree, and collect one fact at a time. For example: the last line of the traceback, the value of one variable, or the smallest input.
3. Change the code only when you can say the cause in one sentence.

Check your understanding

A traceback has six frames. The last two are in httpx. Where do you start to look for the cause?

Show the answer

At the last frame in your own code, the third from the bottom. That line called httpx with a value or at a time that httpx could not handle, for example a URL with a typing mistake. Read the last line of the traceback first for the error type and message, then go up to your code.

Check your understanding

Why make a minimal failing example, when you can already see the problem in the full file?

Show the answer

It proves the cause. With 15 rows and 5 modules, many things could be the cause. With two lines of CSV and one open, only the encoding is left. It also becomes the regression test in the next lessons: a small input that fails before the fix, and passes after it.

Recap

• Read a traceback from the bottom up: the last line gives the error, and the last frame in your code shows where.
• For a wrong result without an error, compare expected and actual, and find the first step where they differ.
• breakpoint() stops the program. p prints a value, n runs the next line, c continues, q quits. Remove it after use.
• A minimal failing example removes everything that is not the cause.
• Input from outside can be anything. Reject a bad record with a reason, do not hide it.

Next, you make the program handle expected errors with clear messages and exit codes, and record what it does with logging.

Next: Exceptions and logging