Reading and writing files

What you will be able to do

Read the tickets from a real data file. Keep every valid ticket, list every rejected record with its reason, and save the result as a report file. Nothing is lost without a trace, and a failed run never leaves a damaged report.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every step, task and check of this lesson, with the instructions. Its first cells make the ticket-cleaner folder as at the start of this lesson, and download the data files. So the notebook works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The Ticket dataclass, find_problems and to_ticket, from Structured records.
• Relative paths and the working directory, from Files and terminal and Install and verify Python: Your turn.

What you need on your computer (only for the local route; the notebook and the boxes on this page need none of it)

• ticket_cleaner/records.py, parsing.py and the typed report.py, and count_tickets.py with the sample rows. You made them in Structured records: Guided practice. If they are missing or different, do that section again. You can also copy the files from the setup cell of this lesson’s notebook. It has every file as at the start of this lesson.
• The data folder in the project, from Files and terminal: Guided practice. If it is missing, run mkdir data.
• curl and an internet connection, to download a data file (1 KB) in Get the data file.
• A terminal in ~/projects/ticket-cleaner. You made the folder in Files and terminal: Guided practice.
• The project’s environment .venv, active in your terminal (the prompt starts with (.venv)). You made it in Isolate dependencies: Guided practice. If it is missing or broken, delete it and make it again from the lock file, as in Make the environment again from nothing.

Tested with Python 3.14.6 on macOS and Pyodide 314.0 (browser). The notebook was tested in Jupyter.

Get the data file

The course data is in the public labs repository. It has 15 synthetic tickets: 9 valid, and 6 with one problem each. Download it into a data folder in your project.

1. Make sure that the data folder exists. You made it in Module 1. If not, run mkdir data.

2. Download the file:

   macOS

   curl -o data/tickets.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets.csv

   Windows

   curl.exe -o data/tickets.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets.csv

   Type curl.exe, not curl. In Windows PowerShell, curl is a different command.

   Linux

   curl -o data/tickets.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets.csv

   If the output is curl: command not found, run sudo apt install curl (or sudo dnf install curl) first.

   If you cannot use curl, open the link in your browser, and save the page as tickets.csv in the data folder.

3. Do the same for the JSON version: in the command, change tickets.csv to tickets.json in both places.

4. Open data/tickets.csv in VS Code, and read it. It is short enough to read completely.

data/tickets.csv (first lines)

id,status,category,priority,subject
T-1001,open,billing,2,Charged twice for March
T-1002,closed,login,3,Password reset email is late
T-1003,open,login,1,Cannot sign in after update

Text files, encodings and with

A file on disk is a series of bytes. To read it as text, Python must know the encoding: the rule that turns bytes into letters. The standard encoding for text today is UTF-8. It covers every language, and the course data uses it.

If you do not give an encoding, Python uses the default of the computer. On macOS and Linux that is UTF-8. On Windows, it is often an older encoding, and every letter outside English appears wrong. So always give the encoding:

with open("data/tickets.csv", encoding="utf-8") as file:
    text = file.read()

The with statement opens the file, and gives it the name file for the indented block. It closes the file when the block ends, even if an error stops the block. A file that stays open can keep your changes in memory instead of on disk, or block another program from using it. The with statement is a context manager: it sets something up before a block, and always cleans up after it.

Paths with pathlib

The pathlib module gives you a Path type for file paths. A Path knows the parts of a path, and joins paths correctly on every system:

Code	Result	Meaning
Path("data") / "tickets.csv"	data/tickets.csv	Join with /, even on Windows. Python uses \ there for you.
Path("data/tickets.csv").suffix	".csv"	The file extension.
Path("data/tickets.csv").name	"tickets.csv"	The file name.
Path("reports/summary.json").parent	reports	The folder that contains it.
path.exists()	True or False	Is there a file or folder at this path?

A relative path such as data/tickets.csv starts from the working directory, as in Files and terminal. So you run the script from the project folder.

CSV and JSON

A CSV file (“comma-separated values”) is a table as text: one line per row, with commas between the values. The first line usually has the column names. The csv module reads it. csv.DictReader gives each row as a dictionary, with the column names as keys:

import csv

with open("data/tickets.csv", encoding="utf-8", newline="") as file:
    rows = list(csv.DictReader(file))

print(rows[0])

Output

{'id': 'T-1001', 'status': 'open', 'category': 'billing', 'priority': '2', 'subject': 'Charged twice for March'}

Every value is a string, also '2'. A CSV file has no types. That is why you wrote find_problems and to_ticket in Module 3. The newline="" argument lets the csv module handle line endings itself, because a value in quotes can contain a line break.

A JSON file holds the same kinds of values as Python: objects (like dictionaries), lists, strings, numbers, true, false and null. The json module turns JSON text into Python values:

import json

with open("data/tickets.json", encoding="utf-8") as file:
    rows = json.load(file)

print(rows[0])

Output

{'id': 'T-1001', 'status': 'open', 'category': 'billing', 'priority': 2, 'subject': 'Charged twice for March'}

Here the priority is the number 2, not the string '2'. find_problems accepts both, because clean_text turns any value into text first. JSON null becomes None, true becomes True.

Run Python in your browser

Read a CSV file

"w" opens a file for writing, and makes it new. \n is a line break. Run it, then add a third row, T-1003,pending,1, and run it again.

Python code (you can change it)import csv # The box has its own small disk. Write a file to it first. with open("tickets.csv", "w", encoding="utf-8") as file: file.write("id,status,priority\n") file.write("T-1001,open,2\n") file.write("T-1002,Closed ,3\n") with open("tickets.csv", encoding="utf-8", newline="") as file: rows = list(csv.DictReader(file)) print(len(rows), "rows") for row in rows: print(row) print("Type of priority:", type(rows[0]["priority"]))

Worked example: read, split, report, write

The program has four steps. Each one is a function with one job.

1. Read the rows. A new module, files.py, reads a CSV or a JSON file into a list of dictionaries. The file extension decides which reader to use:

ticket_cleaner/files.py

import csv
import json
import os
from pathlib import Path


def read_rows(path: Path) -> list[dict]:
    """Read raw records from a .csv or .json file."""
    if path.suffix == ".csv":
        with open(path, encoding="utf-8", newline="") as file:
            return list(csv.DictReader(file))
    if path.suffix == ".json":
        with open(path, encoding="utf-8") as file:
            return json.load(file)
    raise ValueError(f"cannot read {path.name}: use a .csv or .json file")

The last line uses raise. It stops the function with an error and a message, because there is no correct list to return for a .txt file. Module 5 shows how a program catches such an error.

2. Split the valid tickets from the rejected records. A rejected record needs its own small type, with the row number and the problems. Add it to records.py:

ticket_cleaner/records.py (add at the end)

@dataclass(frozen=True)
class Rejected:
    """An input record that is not a valid ticket, and why."""

    row: int
    problems: list[str]

Then add split_records to parsing.py. It also finds a ticket ID that appears twice. Add Rejected to the import line at the top of parsing.py.

ticket_cleaner/parsing.py (add at the end)

def split_records(rows: list[dict]) -> tuple[list[Ticket], list[Rejected]]:
    """Separate valid tickets from rejected records. Nothing is dropped silently."""
    tickets = []
    rejected = []
    seen_ids = set()
    for row_number, raw in enumerate(rows, start=1):
        problems = find_problems(raw)
        if not problems:
            ticket = to_ticket(raw)
            if ticket.id in seen_ids:
                problems.append(f"duplicate id {ticket.id}")
        if problems:
            rejected.append(Rejected(row=row_number, problems=problems))
        else:
            tickets.append(ticket)
            seen_ids.add(ticket.id)
    return tickets, rejected

enumerate(rows, start=1) gives each row with its number: 1 for the first record, 2 for the second. The function returns a tuple of two lists, and the caller unpacks it: tickets, rejected = split_records(rows). Every input row ends in exactly one of the two lists. That is the promise “nothing is dropped silently”.

3. Build the report. Add build_summary to report.py. It returns a dictionary that has only types that JSON can store: strings, numbers, lists, dictionaries and None. Add Rejected to the import line at the top of report.py.

ticket_cleaner/report.py (add at the end)

def build_summary(tickets: list[Ticket], rejected: list[Rejected], status: str) -> dict:
    """Return the report as a dictionary that can be saved as JSON."""
    selected = filter_by_status(tickets, status)
    return {
        "status": status,
        "valid_records": len(tickets),
        "selected": len(selected),
        "by_category": count_by_category(selected),
        "average_priority": average_priority(selected),
        "rejected": [{"row": r.row, "problems": r.problems} for r in rejected],
    }

4. Write the report safely. Add write_json to files.py. Two rules make output safe. First, never write over the input file. Second, never leave half a file: if the program stops in the middle of writing, the old report must stay complete.

ticket_cleaner/files.py (add at the end)

def write_json(data: dict, path: Path) -> None:
    """Save data as JSON. Write a temporary file first, so a failure never leaves half a file."""
    path.parent.mkdir(parents=True, exist_ok=True)
    temporary = path.with_name(path.name + ".tmp")
    with open(temporary, "w", encoding="utf-8") as file:
        json.dump(data, file, indent=2, ensure_ascii=False)
        file.write("\n")
    os.replace(temporary, path)

Line	Why
path.parent.mkdir(parents=True, exist_ok=True)	Makes the reports folder if it does not exist. exist_ok=True means “no error if it is already there”.
temporary = ...	The data goes to summary.json.tmp first.
json.dump(..., indent=2, ensure_ascii=False)	Writes readable JSON, with letters such as ü as they are.
os.replace(temporary, path)	Renames the finished file to summary.json in one step. A reader sees the old complete file or the new complete file, never a mix.

The script now reads the file instead of the sample rows:

count_tickets.py

from pathlib import Path

from ticket_cleaner.files import read_rows, write_json
from ticket_cleaner.parsing import split_records
from ticket_cleaner.report import build_summary

INPUT_FILE = Path("data/tickets.csv")
OUTPUT_FILE = Path("reports/summary.json")


def main() -> None:
    rows = read_rows(INPUT_FILE)
    tickets, rejected = split_records(rows)
    for record in rejected:
        print(f"Row {record.row} rejected: {'; '.join(record.problems)}")

    summary = build_summary(tickets, rejected, status="open")
    write_json(summary, OUTPUT_FILE)
    print(f"{len(rows)} rows: {len(tickets)} valid, {len(rejected)} rejected.")
    print(f"Report: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()

Guided practice: the first real report

1. Make ticket_cleaner/files.py with read_rows and write_json.

2. Add Rejected to records.py, split_records to parsing.py, and build_summary to report.py. Update the import lines at the top of parsing.py and report.py.

3. Replace count_tickets.py with the new version.

4. Run it:

   python count_tickets.py

   Output

   Row 7 rejected: unknown status 'opne'
   Row 8 rejected: missing id
   Row 9 rejected: priority must be 1, 2 or 3, not 'high'
   Row 11 rejected: priority must be 1, 2 or 3, not '5'
   Row 13 rejected: duplicate id T-1003
   Row 14 rejected: missing category
   15 rows: 9 valid, 6 rejected.
   Report: reports/summary.json

5. Open reports/summary.json in VS Code. Check that selected is 4, and that the rejected list has 6 entries.

6. Check that nothing is lost: 9 valid + 6 rejected = 15 rows. Open data/tickets.csv, and find row 7. It is on line 8 of the file, because line 1 has the column names.

Your turn: JSON input, a second report

Change the script so that it reads data/tickets.json, and writes the report for the pending tickets to reports/pending.json. Change only the constants and one argument. Predict the numbers before you run it.

Show the expected result

Set INPUT_FILE = Path("data/tickets.json"), OUTPUT_FILE = Path("reports/pending.json"), and status="pending". The output still shows 9 valid and 6 rejected, because the JSON file has the same records. reports/pending.json has "selected": 2, "by_category": {"billing": 1, "login": 1} and "average_priority": 1.5. Row 10 (LOGIN) counts as login, because to_ticket cleaned it.

reports/summary.json did not change: each run writes only its own output file. Change the constants back to the CSV file and summary.json before the next lesson.

Common mistake

Strange letters such as Müller

Symptom: the program runs, but a subject in the output or the report shows Müller instead of Müller. Or the program stops with UnicodeDecodeError.

Cause: Python read the file with a different encoding than the one that it was saved in. The file is UTF-8, where ü is two bytes. An older encoding reads those two bytes as two letters, ü. This happens on Windows when open() has no encoding argument, because the default there is often not UTF-8.

Fix:

1. Find every open( in your code. Each one must have encoding="utf-8".
2. Run the program again.
3. If a report was written with the wrong letters, run the program again to write it correctly. Do not try to repair the letters by hand.

FileNotFoundError

If the output is FileNotFoundError: [Errno 2] No such file or directory: 'data/tickets.csv', the path does not lead to the file from the working directory. Run pwd and ls data, as in Module 1. Check that the terminal is in ticket-cleaner, and that the file name is exactly tickets.csv.

Check your understanding

Why does count_tickets.py print the rejected rows and save them in the report, instead of only counting the valid tickets?

Show the answer

So that no record is lost without a trace. If invalid rows were only skipped, the report would show 9 tickets, and nobody would know that 6 more existed. With the row numbers and the reasons, a person can find each record in the file and correct it. The check is simple: valid + rejected = all rows.

Check your understanding

The program stops with an error while json.dump writes the report. What is on disk afterwards: the old report, half a new report, or nothing?

Show the answer

The old report, complete. The new data goes to summary.json.tmp, and only os.replace at the end gives it the real name. If the program stops before that line, summary.json is still the last complete report. A summary.json.tmp file can stay behind, and you can delete it.

Recap

• Always open text files with encoding="utf-8", inside a with block, so the file closes even after an error.
• pathlib.Path joins paths and gives their parts. Relative paths start at the working directory.
• csv.DictReader gives every value as a string. json.load keeps numbers, true, false and null.
• Split the input into valid and rejected records. Every row ends in exactly one list, with a reason for each rejection.
• Write output to a temporary file, then rename it with os.replace. Never write over the input.

Next, the same records come from a web service instead of a file.

Next: Basic HTTP client use