Exceptions and logging

What you will be able to do

Make ticket-cleaner fail well. A missing file or a server that is down must give one clear line and exit code 1, not a long traceback. A bug in your own code must still show its full traceback, so you can find it. And every rejected record goes to a log that you can turn up or down without a change to the code.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every step, task and check of this lesson, with the instructions. Its first cells make the ticket-cleaner folder as at the start of this lesson. They also install httpx and download the data files. So the notebook works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• How to read a traceback, from Read failures: Failure 1.
• Exit codes, from Files and terminal.
• Status codes and raise_for_status(), from Basic HTTP client use.

What you need on your computer (only for the local route; the notebook and the boxes on this page need none of it)

• The two fixes from Read failures: Guided practice: find_problems rejects a value that is not a dictionary (Failure 1), and read_rows reads CSV with utf-8-sig (Failure 2). If they are missing, do that section again. You can also copy the files from the setup cell of this lesson’s notebook. It has every file as at the start of this lesson.
• TICKETS_SOURCE is not set in your terminal. Remove it with unset TICKETS_SOURCE (macOS and Linux) or Remove-Item Env:TICKETS_SOURCE (PowerShell), as in Configuration and secrets: Environment variables.
• data/tickets.csv and data/tickets.json. You downloaded them in Reading and writing files: Get the data file. If one is missing or changed, download it again with the same curl command. The checksums are in the dataset card.
• The project’s environment .venv, active in your terminal (the prompt starts with (.venv)). You made it in Isolate dependencies: Guided practice. If it is missing or broken, delete it and make it again from the lock file, as in Make the environment again from nothing.
• httpx 0.28.1 in that environment. You installed it in Isolate dependencies: Worked example. Check it with python -c "import httpx; print(httpx.__version__)". If it is missing, run python -m pip install -r requirements-lock.txt.

Tested with Python 3.14.6 and httpx 0.28.1 on macOS, and Pyodide 314.0 (browser). The notebook was tested in Jupyter.

Two kinds of error

An exception is Python’s way to report an error: an object that describes what went wrong. When code raises an exception, Python stops the normal path and goes back up through the callers. If no caller handles it, the program stops and shows the traceback. You have seen many: KeyError, ValueError, FileNotFoundError.

Not all errors are equal. The question is: whose fault is it, and can the program do something useful about it?

	Expected errors	Programming defects (bugs)
Examples	The file does not exist. The server is down. The JSON is cut short. A record has a bad priority.	A typing mistake in a field name. A function gets a list where it expects a ticket.
Cause	The world outside the program.	The program itself.
Can happen in correct code?	Yes, at any time.	No. Correct code does not have them.
What the program does	Handle it: reject the record, or stop with a clear message and exit code 1.	Nothing. Let it crash with the full traceback, then fix the code.

ticket-cleaner already handles one kind of expected error without exceptions: an invalid record becomes a Rejected entry. This lesson handles the others: errors that make it impossible to read the input at all.

try and except

A try block runs code that can raise an exception. An except block after it says which exception types to handle, and what to do:

try:
    rows = read_rows(Path("data/missing.csv"))
except FileNotFoundError as error:
    print("Cannot read the file:", error)

If read_rows raises a FileNotFoundError, Python jumps to the except block, and error is the exception object. Its text is the message. If no error is raised, the except block is skipped. If a different exception is raised, the except block does not catch it, and it goes up to the caller as before.

Exception types form families. A more general type covers the specific types under it:

Catch this	It also covers
OSError	FileNotFoundError, PermissionError (no right to read the file) and other errors from the operating system.
ValueError	json.JSONDecodeError (the JSON text is not valid), and the ValueError that read_rows raises for a .txt file.
httpx.HTTPError	httpx.HTTPStatusError (a 4xx or 5xx code), httpx.ConnectTimeout, httpx.ConnectError.

To catch more than one type, put them in a tuple: except (OSError, ValueError) as error:.

Run Python in your browser

Catch only what you expect

"2" works and "high" is caught. None raises a TypeError, which the except does not name, so the program stops. Is a None here an expected error or a defect? Think about where the value comes from, then decide whether to add TypeError.

Python code (you can change it)def to_priority(text: str) -> int: return int(text) for text in ["2", "high", None]: try: print(text, "->", to_priority(text)) except ValueError as error: print(text, "-> rejected:", error)

Catch at the edge, not everywhere

Where should the try go? Not in read_rows or fetch_rows. Those functions cannot know what the user wants: stop, try a different file, or try again later. They raise, and let the caller decide.

The right place is the edge of the program: main(), which knows the whole situation. It can say what failed, with which source, and return an exit code. Everything between the edge and the cause stays simple, with no error handling.

In Files and terminal, the exit code told other programs if a command worked. Now main() returns the exit code, and the last line passes it to Python:

if __name__ == "__main__":
    raise SystemExit(main())

SystemExit is a special exception that ends the program with the given exit code: 0 for success, 1 for an error.

Logging instead of print

Until now, the program used print for everything. A log is better for messages about what the program does, as opposed to its result:

• Each message has a level: DEBUG (details for a developer), INFO (normal progress), WARNING (something unusual, and the program continues), ERROR (something failed).
• You choose the lowest level to show when the program starts. With WARNING, the DEBUG and INFO messages are hidden. With INFO, you see more. The code does not change.
• Log messages go to standard error (stderr), a separate output stream from print, which goes to standard output (stdout). Another program, or a file, can take the result without the log messages.

import logging

logger = logging.getLogger("ticket_cleaner")
logging.basicConfig(level="WARNING", format="%(levelname)s %(name)s: %(message)s")

logger.info("reading tickets from %s", "data/tickets.csv")   # hidden at WARNING
logger.warning("row %d rejected: %s", 7, "unknown status 'opne'")

Output

WARNING ticket_cleaner: row 7 rejected: unknown status 'opne'

getLogger("ticket_cleaner") gets a logger with a name, which appears in each message. basicConfig sets the level and the format once, at the start of the program. The values after the message replace %s (text) and %d (a whole number), in order.

A good log message has context: enough facts to act on it without the code. row 7 rejected: unknown status 'opne' says which record and why. invalid input does not. Never log a secret: the same rules as for printing apply.

Worked example: count_tickets.py fails well

Amira adds log_level to the settings, so LOG_LEVEL can change the level without a change to the code:

ticket_cleaner/config.py (changes)

@dataclass(frozen=True)
class Settings:
    source: str
    tickets_url: str
    api_token: str | None = field(repr=False)
    log_level: str


def load_settings() -> Settings:
    """Read the settings from environment variables, with safe defaults."""
    return Settings(
        source=os.environ.get("TICKETS_SOURCE", "data/tickets.csv"),
        tickets_url=os.environ.get("TICKETS_URL", DEFAULT_URL),
        api_token=os.environ.get("TICKETS_API_TOKEN") or None,
        log_level=os.environ.get("LOG_LEVEL", "WARNING").upper(),
    )

The script catches the three families of expected errors in one place, logs the rejected records, and returns an exit code:

count_tickets.py

import logging
from pathlib import Path

import httpx

from ticket_cleaner.config import Settings, load_settings
from ticket_cleaner.files import read_rows, write_json
from ticket_cleaner.parsing import split_records
from ticket_cleaner.remote import fetch_rows
from ticket_cleaner.report import build_summary

OUTPUT_FILE = Path("reports/summary.json")

logger = logging.getLogger("ticket_cleaner")


def load_rows(settings: Settings) -> list[dict]:
    """Return raw records from a file, or from the web service for "remote"."""
    if settings.source == "remote":
        return fetch_rows(settings.tickets_url, token=settings.api_token)
    return read_rows(Path(settings.source))


def main() -> int:
    settings = load_settings()
    logging.basicConfig(
        level=settings.log_level, format="%(levelname)s %(name)s: %(message)s"
    )
    logger.info("reading tickets from %s", settings.source)

    try:
        rows = load_rows(settings)
    except (OSError, ValueError, httpx.HTTPError) as error:
        logger.error("cannot read the tickets from %s: %s", settings.source, error)
        return 1

    tickets, rejected = split_records(rows)
    for record in rejected:
        logger.warning("row %d rejected: %s", record.row, "; ".join(record.problems))

    summary = build_summary(tickets, rejected, status="open")
    write_json(summary, OUTPUT_FILE)
    logger.info("wrote %s", OUTPUT_FILE)
    print(f"{len(rows)} rows: {len(tickets)} valid, {len(rejected)} rejected.")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

The try block holds one line: the line that touches the outside world. split_records, build_summary and write_json are outside it. If one of them raises an exception, it is not an input problem, and the traceback must show it.

Here is the result for four situations:

Situation	Output	Exit code
Default (data/tickets.csv)	6 WARNING lines, then 15 rows: 9 valid, 6 rejected.	0
LOG_LEVEL=info	Also INFO ticket_cleaner: reading tickets from data/tickets.csv and wrote reports/summary.json	0
TICKETS_SOURCE=data/missing.csv	ERROR ticket_cleaner: cannot read the tickets from data/missing.csv: [Errno 2] No such file or directory: 'data/missing.csv'	1
TICKETS_SOURCE=remote and a wrong TICKETS_URL	ERROR ticket_cleaner: cannot read the tickets from remote: Client error '404 Not Found' for url '...'	1

Guided practice: make it fail well

1. Update config.py and count_tickets.py as in the worked example.

2. Run python count_tickets.py, then show the exit code (echo $?, or $LASTEXITCODE in PowerShell). The output has 6 warnings and the counts. The exit code is 0.

3. Set LOG_LEVEL to info, and run again. Two INFO lines appear. Remove LOG_LEVEL again.

4. Set TICKETS_SOURCE to data/missing.csv, and run. You see one ERROR line and no traceback. The exit code is 1.

5. See the two output streams. Run the program with its log hidden, then with its result hidden. 2> sends stderr to a file, and > sends stdout. /dev/null ($null in PowerShell) is a place that throws away everything:

   macOS

   python count_tickets.py 2>/dev/null
   python count_tickets.py >/dev/null

   Windows

   python count_tickets.py 2>$null
   python count_tickets.py >$null

   Linux

   python count_tickets.py 2>/dev/null
   python count_tickets.py >/dev/null

   The first command shows only 15 rows: 9 valid, 6 rejected. The second shows only the warnings.

6. Remove TICKETS_SOURCE.

Your turn: predict, then run

For each situation, predict the last line of the output and the exit code. Then make the situation, run the program, and compare. Undo each change before the next one.

1. A JSON file that is cut short. Make data/cut.json with the text [{"id": 1 (no end), and set TICKETS_SOURCE to it.
2. A bug: in report.py, change ticket.category in count_by_category to ticket.categry.
3. LOG_LEVEL=error with the normal data file.

Show the answers

1. ERROR ticket_cleaner: cannot read the tickets from data/cut.json: Expecting ',' delimiter: line 1 column 10 (char 9), exit code 1. json.JSONDecodeError is a ValueError, so the except catches it. This is an expected error: a file can be damaged at any time.
2. A full traceback that ends with AttributeError: 'Ticket' object has no attribute 'categry'. Did you mean: 'category'?, exit code 1. The except does not catch it, because split_records and build_summary are outside the try block, and AttributeError is not in the tuple. That is correct: it is a defect, and the traceback shows you the line to fix.
3. Only 15 rows: 9 valid, 6 rejected., exit code 0. The warnings are hidden, because WARNING is below ERROR. The result is still printed, because print is not part of the log.

Common mistake

A catch-all except that hides a bug

Symptom: the program says cannot read the tickets, but the file is there and is correct. Or the program prints nothing useful, and the report is old.

try:
    rows = load_rows(settings)
    tickets, rejected = split_records(rows)
    summary = build_summary(tickets, rejected, status="open")
except Exception:
    logger.error("cannot read the tickets")
    return 1

Cause: except Exception: catches almost every exception, including defects such as an AttributeError from a typing mistake in build_summary. The message blames the input, and the traceback, which would show the real line, is gone. With except Exception: pass, the error disappears completely.

Fix:

1. Put only the lines that touch the outside world in the try block.
2. Name the exception types that you expect, as in the worked example.
3. Sometimes you must catch everything at the very top of a program, for example to log it before it ends. Then use logger.exception("..."), and end with exit code 1. It writes the full traceback into the log.

Check your understanding

Which of these should main() catch: FileNotFoundError, KeyError from a dictionary in report.py, httpx.ConnectTimeout, TypeError from a wrong argument in your code?

Show the answer

FileNotFoundError and httpx.ConnectTimeout: both come from the world outside the program and can happen in correct code. The KeyError and the TypeError are defects in your code. Do not catch them. Let the traceback show where they are, and fix the code.

Check your understanding

Why does the program print the counts with print, but the rejected rows with logger.warning?

Show the answer

The counts are the result that the user asked for, so they go to standard output. The rejected rows are messages about the run, with a level, so they go to the log (standard error). Then a user can hide the log with LOG_LEVEL, or send it to a different file, and the result stays the same.

Recap

• Expected errors come from outside the program. Handle them. Defects are bugs in your code. Let them crash, and fix them.
• try / except SomeError as error: handles one family of errors. Catch the specific types that you expect, never everything.
• Catch at the edge, in main(), around the lines that touch the outside world. Return 1 for an error, and end with raise SystemExit(main()).
• Use logging for messages about the run: a level, a name, and enough context to act. Results go to stdout, and logs to stderr.

Next, you turn the checks from this module into automatic tests with pytest, including a test that proves the encoding bug stays fixed.

Next: Focused tests