Deliver a reusable dataset

What you will be able to do

Deliver the dataset as a project that anyone can run again from a clean start with one command. The same input must give the same files, and a changed input must stop the run with a clear message.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook writes check_input.py and run_pipeline.py. It runs the pipeline twice from a clean snapshot and compares the SHA-256 checksums. Then it points the pipeline at the August export, and does the module practice in a fresh copy. Its first cells install pandas, download the practice data and rebuild the files of the earlier lessons, so it works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The data contract and exit codes, from Validation rules.
• Checksums, the lineage records and the dataset card, from Track lineage.
• For a refresh on a handoff with a fresh copy, see Quality and handoff: Prove it: a fresh copy in Python for Practical AI Engineering.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project ~/projects/ticket-data with get_data.py and the raw files in data/raw/, from Set up a practice database: Build the database. This lesson rebuilds larkfield.db with python get_data.py --reset, as in The database is protected, and you can reset it.
• The virtual environment .venv, from Profile before changing: The same profile in pandas. requirements.txt lists pandas==3.0.6 and scikit-learn==1.9.1, from Fit transformations correctly: Install scikit-learn.
• The scripts of the earlier lessons:
  • scripts/extract.py and the two query files in queries/, from Reproducible extraction: Build extract.py.
  • scripts/clean.py, from Handle messy values: Finish clean.py.
  • scripts/build_features.py, from Features and target timing: build_features.py.
  • scripts/split.py, from Choose a split: Make and check the split.
  • scripts/validate.py, from Validation rules: The contract in validate.py.
• DATASET.md in the project folder, from Track lineage: The dataset card.
• If a script is missing or broken, copy the course’s finished version from C04/pipeline into the same place in your project. You do not need to repair the outputs by hand: run_pipeline.py deletes and rebuilds them.
• An internet connection, to download one small file. There is no account and no cost.

Tested with Python 3.14.6 and pandas 3.0.6 on macOS. The Windows commands are not tested. The notebook was tested in Jupyter.

What “reusable” means

Priya does not want only the files. She wants to make them again: next month, on her computer, after a fix. Mei gives three promises:

1. One command. The whole preparation runs with one command, from a clean snapshot. Nobody needs to remember the order of the steps.
2. Same input, same output. Two runs on the same snapshot give the same files, byte for byte. A process with this property is reproducible.
3. Changed input, clear failure. When the export changes in a way the scripts do not expect, the run stops and says what changed.

How do you prove promise 2? You cannot compare 1,369 rows by eye. You compare checksums. A small example in Python’s interactive mode:

Interactive Python

>>> import hashlib
>>> hashlib.sha256(b"T-1,email,2\n").hexdigest()
'aba761b1f15fa31acb1313ea20118288337bfaba145be01e9ead1e03c7e47ee2'
>>> hashlib.sha256(b"T-1,email,2\n").hexdigest()
'aba761b1f15fa31acb1313ea20118288337bfaba145be01e9ead1e03c7e47ee2'
>>> hashlib.sha256(b"T-1,Email,2\n").hexdigest()
'ebd66142a4ce9df959615da1256552e7679bbe07d069ed9017a31eabfac4b550'

The same bytes give the same checksum. One capital letter gives a completely different one. So if two runs give the same checksum for train.csv, the two files are identical.

The project, organised for handoff

Mei’s project now has everything that Priya needs, in a fixed place:

ticket-data/
  run_pipeline.py      the one command
  DATASET.md           the dataset card (Track lineage)
  requirements.txt     pinned versions
  get_data.py, q.py    from the labs repository
  queries/             extract_tickets.sql, extract_outcomes.sql, 60-contract-checks.sql
  scripts/             check_input.py   the input schema (new in this lesson)
                       extract.py, clean.py, build_features.py, split.py
                       validate.py      the output contract (Validation rules)
  data/raw/            the export files
  data/extract/  data/clean/  data/model/    written by the pipeline
  reports/             cleaning.json and validation.json: the quality report

Priya asks	The file that answers
How was the data selected?	queries/*.sql, scripts/extract.py, data/extract/manifest.json
What schema does the export need?	scripts/check_input.py
What can I rely on in the delivered files?	scripts/validate.py, reports/validation.json
How was the split made?	scripts/split.py, data/model/split_manifest.json
What was removed or corrected?	reports/cleaning.json, data/clean/quarantine.csv
Are my files the same as yours?	data/model/MANIFEST.json (this lesson)

Two new files are missing: the input check and the one command.

The input check

validate.py checks the output. The new export can also break the input: a renamed column, or numbers written as text. The pipeline must find that before it loads the data, because a loaded file can turn into quiet mistakes. The check is a separate script, scripts/check_input.py. It uses only the Python standard library.

scripts/check_input.py

"""Check the raw export files before anything loads them: each file must have
the expected columns, and each value must be readable as its column's type.

    python scripts/check_input.py
    python scripts/check_input.py --tickets data/raw/tickets-2026-08-01.csv

This check looks at the shape of the data only. Messy values (spellings,
duplicates, sentinels) are the cleaning step's job. Exit code 1 on failure.
"""

import argparse
import csv
import sys
from datetime import datetime
from pathlib import Path

# The schema of the export of 2026-07-01: column name -> type.
SCHEMAS = {
    "customers": {"customer_id": "text", "segment": "text", "region": "text", "joined_on": "date"},
    "tickets": {
        "ticket_id": "text", "customer_id": "text", "created_at": "datetime", "channel": "text",
        "team": "text", "priority": "integer", "order_value": "number", "word_count": "integer",
        "first_reply_minutes": "integer", "priority_now": "integer", "closed_at": "datetime",
    },
    "outcomes": {
        "outcome_id": "text", "ticket_id": "text", "recorded_at": "datetime", "status": "text",
        "resolution_code": "text", "csat": "integer",
    },
}

# Each type has a function that raises ValueError for a value it cannot read.
READERS = {
    "text": str,
    "integer": int,
    "number": float,
    "date": lambda v: datetime.strptime(v, "%Y-%m-%d"),
    "datetime": lambda v: datetime.strptime(v, "%Y-%m-%d %H:%M:%S"),
}


def check_file(path, schema):
    """Return a list of problems. An empty list means the file is OK."""
    if not path.exists():
        return [f"file not found: {path}"]
    with open(path, newline="", encoding="utf-8") as f:
        reader = csv.reader(f)
        header = next(reader, [])
        rows = list(reader)

    problems = []
    missing = [c for c in schema if c not in header]
    extra = [c for c in header if c not in schema]
    for column in missing:
        problems.append(f"missing column: {column}")
    for column in extra:
        hint = " (was a column renamed?)" if missing else ""
        problems.append(f"unexpected column: {column}{hint}")

    # An empty value is a missing value. It is allowed here; only values that
    # are present must have the right type.
    for column, kind in schema.items():
        if column not in header or kind == "text":
            continue
        i = header.index(column)
        bad = []
        for line, row in enumerate(rows, start=2):  # line 1 is the header
            value = row[i]
            if value == "":
                continue
            try:
                READERS[kind](value)
            except ValueError:
                bad.append((line, value))
        if bad:
            line, value = bad[0]
            present = sum(1 for row in rows if row[i] != "")
            problems.append(f"{column}: {len(bad)} of {present} values are not a valid {kind}"
                            f" (first: line {line}, {value!r})")
    return problems


def main():
    parser = argparse.ArgumentParser()
    for name in SCHEMAS:
        parser.add_argument(f"--{name}", default=f"data/raw/{name}.csv", type=Path)
    args = parser.parse_args()

    failed = False
    for name, schema in SCHEMAS.items():
        path = getattr(args, name)
        problems = check_file(path, schema)
        print(f"{'PASS' if not problems else 'FAIL'}  {path}")
        for problem in problems:
            print(f"      {problem}")
        failed = failed or bool(problems)

    if failed:
        print("The input does not match the expected schema. Do not load it.")
        sys.exit(1)
    print("The input matches the expected schema.")


if __name__ == "__main__":
    main()

Part	Why
SCHEMAS	The expected schema, written by hand, as in validate.py. It is the contract with the help-desk export.
READERS	One function per type. float("$10.85") raises ValueError, so the check finds numbers written as text.
--tickets and the other options	By default, the script checks data/raw/. With an option, it checks a new file before you use it.
Only header and types	Spellings such as Email are not schema problems. clean.py handles them, and validate.py checks the result. Each check has one job.

The one command

run_pipeline.py runs the six steps in order. It stops at the first step that fails. At the end, it writes data/model/MANIFEST.json, with a checksum and a row count for every output file.

run_pipeline.py

"""Run the whole preparation, in order, from larkfield.db to a validated
dataset. Stop at the first step that fails. At the end, write
data/model/MANIFEST.json with a SHA-256 checksum of every output file.

    python get_data.py --reset     start from a clean snapshot
    python run_pipeline.py

    python run_pipeline.py --tickets data/raw/tickets-2026-08-01.csv
                                   check a new export before you load it
"""

import argparse
import csv
import hashlib
import json
import shutil
import sqlite3
import subprocess
import sys
from datetime import datetime, timezone
from importlib.metadata import version
from pathlib import Path

STEPS = [
    "scripts/check_input.py",
    "scripts/extract.py",
    "scripts/clean.py",
    "scripts/build_features.py",
    "scripts/split.py",
    "scripts/validate.py",
]
# Folders that only the pipeline writes. They are deleted before a run, so an
# old file can never look like the output of this run.
OUTPUT_FOLDERS = [Path("data/extract"), Path("data/clean"), Path("data/model")]
OUTPUTS = [
    "data/extract/tickets.csv", "data/extract/outcomes.csv",
    "data/clean/tickets.csv", "data/clean/quarantine.csv", "data/clean/outcomes.csv",
    "data/model/tickets_model.csv", "data/model/train.csv", "data/model/valid.csv", "data/model/test.csv",
    "data/model/build_summary.json", "data/model/split_manifest.json",
    "reports/cleaning.json", "reports/validation.json",
]
MANIFEST = Path("data/model/MANIFEST.json")


def sha256(path):
    return hashlib.sha256(Path(path).read_bytes()).hexdigest()


def describe(path):
    """Checksum, and the number of data rows for a CSV file."""
    entry = {"sha256": sha256(path)}
    if path.endswith(".csv"):
        with open(path, newline="", encoding="utf-8") as f:
            entry["rows"] = sum(1 for _ in csv.reader(f)) - 1  # minus the header
    return entry


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--tickets", default="data/raw/tickets.csv",
                        help="the raw tickets file for the input check")
    args = parser.parse_args()

    if not Path("larkfield.db").exists():
        sys.exit("larkfield.db not found. Run: python get_data.py")
    for folder in OUTPUT_FOLDERS:
        shutil.rmtree(folder, ignore_errors=True)

    started = datetime.now(timezone.utc)
    for number, step in enumerate(STEPS, start=1):
        command = [sys.executable, step]  # the same Python that runs this script
        if step == "scripts/check_input.py":
            command += ["--tickets", args.tickets]
        name = Path(step).stem
        result = subprocess.run(command, capture_output=True, text=True)
        if result.returncode != 0:
            print(f"{number}/{len(STEPS)} {name}: FAILED\n")
            print((result.stdout + result.stderr).rstrip())
            print(f"\nStopped at step {number} ({name}). No MANIFEST.json was written: this run delivered no dataset.")
            sys.exit(1)
        print(f"{number}/{len(STEPS)} {name}: ok")

    # The extraction manifest says which snapshot and which selection the
    # dataset comes from. Its time stamp goes in "run", with this run's.
    extract = json.loads(Path("data/extract/manifest.json").read_text(encoding="utf-8"))
    outputs = {path: describe(path) for path in OUTPUTS}
    manifest = {
        "dataset": "Larkfield ticket escalation dataset",
        "source": extract["source"],
        "selection": extract["selection"],
        "queries": {f["query"]: f["query_sha256"] for f in extract["files"]},
        "steps": STEPS,
        "outputs": outputs,
        # One checksum for all outputs together: equal outputs give an
        # equal fingerprint, whatever the time of the run.
        "outputs_fingerprint": hashlib.sha256(json.dumps(outputs, sort_keys=True).encode()).hexdigest(),
        "run": {
            "started_at": started.isoformat(timespec="seconds"),
            "extracted_at": extract["extracted_at"],
            "python": sys.version.split()[0],
            "sqlite": sqlite3.sqlite_version,
            "pandas": version("pandas"),
        },
    }
    MANIFEST.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
    model = outputs["data/model/tickets_model.csv"]["rows"]
    parts = ", ".join(f"{p} {outputs[f'data/model/{p}.csv']['rows']}" for p in ["train", "valid", "test"])
    print(f"Done: {model} rows ({parts}).")
    print(f"Wrote {MANIFEST}. Outputs fingerprint: {manifest['outputs_fingerprint']}")


if __name__ == "__main__":
    main()

Four decisions make the pipeline safe to run again:

• Delete old outputs first. If a step fails, its old file from yesterday must not look like today’s result. So the run starts with empty output folders.
• Each step is a separate program. subprocess.run starts each script with the same Python (sys.executable) and reads its exit code. That is why validate.py and check_input.py end with sys.exit(1) on failure. A step that fails stops the run, and its own message is printed.
• Content apart from time. The checksums are in outputs. The run time and the time of the extraction are in run. extracted_at changes on every run, so manifest.json itself is not in outputs; its content is copied into the manifest.
• One fingerprint. outputs_fingerprint is a checksum of all the output checksums together. To compare two runs, you compare one line.

Worked example: run it twice from a clean snapshot

Mei starts from a clean snapshot. get_data.py --reset deletes the database and builds it again from the checked CSV files:

python get_data.py --reset

Deleted larkfield.db.
customers    240 rows
tickets     1417 rows
outcomes    1874 rows
Built larkfield.db. Check: customers 240, tickets 1417, outcomes 1874.

python run_pipeline.py

1/6 check_input: ok
2/6 extract: ok
3/6 clean: ok
4/6 build_features: ok
5/6 split: ok
6/6 validate: ok
Done: 1369 rows (train 927, valid 227, test 215).
Wrote data/model/MANIFEST.json. Outputs fingerprint: 2b2460321543ffb73fe9d5b04fd8f5a9d77ee990fff37823b0855b911ba4b855

Part of the manifest:

data/model/MANIFEST.json (part)

  "outputs": {
    "data/extract/tickets.csv": {
      "sha256": "96786af9e2ebb2492219867ac302f57fabdefb4c53fb60a1071d9412e70e6f39",
      "rows": 1416
    },
    ...
    "data/model/train.csv": {
      "sha256": "e83c4a056c6e26595777760918e49d9f9b9b39be7bb561a20aefd25dd4c177b8",
      "rows": 927
    },
    ...
  },
  "outputs_fingerprint": "2b2460321543ffb73fe9d5b04fd8f5a9d77ee990fff37823b0855b911ba4b855",
  "run": {
    "started_at": "2026-10-07T03:13:22+00:00",
    "extracted_at": "2026-10-07T03:13:22+00:00",
    "python": "3.14.6",
    "sqlite": "3.50.4",
    "pandas": "3.0.6"
  }

She keeps a copy of the manifest, runs the pipeline again, and compares. The fingerprint is the same. Then she compares the two files line by line with diff, a standard tool on macOS and Linux:

diff first-run.json data/model/MANIFEST.json

77,78c77,78
<     "started_at": "2026-10-07T03:13:22+00:00",
<     "extracted_at": "2026-10-07T03:13:22+00:00",
---
>     "started_at": "2026-10-07T03:13:25+00:00",
>     "extracted_at": "2026-10-07T03:13:25+00:00",

Only the two times are different. All 13 output files are identical. The pipeline is reproducible.

Your fingerprint and Mei's

Your fingerprint matches Mei’s only if your scripts write exactly the same bytes. A different column order or one more decimal changes it. That is not a failure: the promise is that your two runs match. On Windows, pandas and Python write a different line ending, so the checksums differ from the macOS ones even with the same scripts. Compare two runs on the same computer.

Mei adds the output checksums to DATASET.md, so that Priya can check the files she receives. She also adds check_input.py as step 1 in the Transformation order table, and the command to the top of it: “Run python get_data.py --reset, then python run_pipeline.py.”

DATASET.md (new section)

## Output checksums (SHA-256)

| File | Rows | SHA-256 |
|---|---|---|
| `data/model/tickets_model.csv` | 1,369 | cb6e783f28cb17f45a7ec1412a3714a49d4ec10a292f8077dcc7bafb7421235b |
| `data/model/train.csv` | 927 | e83c4a056c6e26595777760918e49d9f9b9b39be7bb561a20aefd25dd4c177b8 |
| `data/model/valid.csv` | 227 | 5a01365d2294d32db58a55a48a8f5b4c5958e0dc381ae06c9fdc33c5fb0e88cd |
| `data/model/test.csv` | 215 | 1c0aa441b17db00d66f5f5df9f604c38d99787fd69c9894f987a7ca548d04d1b |

Outputs fingerprint (all 13 output files): `2b2460321543ffb73fe9d5b04fd8f5a9d77ee990fff37823b0855b911ba4b855`. Two runs gave the same fingerprint. On Windows, pandas writes CSV files with different line endings, so the checksums differ from these; compare two runs on the same computer.

Guided practice: the one command, twice

1. Make scripts/check_input.py and run_pipeline.py with the code above. run_pipeline.py goes in the project folder, not in scripts/.

2. In the terminal, go to your project folder and check that the environment is active.

3. Start from a clean snapshot: python get_data.py --reset. The last line is Built larkfield.db. Check: customers 240, tickets 1417, outcomes 1874.

4. Run python run_pipeline.py. You should see six ok lines and Done: 1369 rows (train 927, valid 227, test 215). Note the fingerprint.

5. Keep a copy of the manifest:

   macOS

   Terminal

   cp data/model/MANIFEST.json first-run.json

   Windows

   PowerShell

   Copy-Item data\model\MANIFEST.json first-run.json

6. Run python run_pipeline.py again. The fingerprint is the same as in step 4.

7. Compare the outputs of the two manifests with Python:

   python -c "import json; a = json.load(open('first-run.json')); b = json.load(open('data/model/MANIFEST.json')); print('same outputs' if a['outputs'] == b['outputs'] else 'different outputs')"

   Output

   same outputs

8. Copy the four checksums of data/model/ from MANIFEST.json into a new section of DATASET.md.

Your turn: point the pipeline at the August export

The help desk was updated in July. The next export, of 2026-08-01, has 40 new tickets. Before Mei loads it, she checks it.

1. Download the file into data/raw/:

   macOS

   Terminal

   curl -o data/raw/tickets-2026-08-01.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C04/data/tickets-2026-08-01.csv

   Windows

   PowerShell

   curl.exe -o data/raw/tickets-2026-08-01.csv https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C04/data/tickets-2026-08-01.csv

2. Predict: which step stops, and which problems does the message name?

3. Run python run_pipeline.py --tickets data/raw/tickets-2026-08-01.csv.

4. Look in data/model/. Is there a MANIFEST.json?

Show the expected result

python run_pipeline.py --tickets data/raw/tickets-2026-08-01.csv

1/6 check_input: FAILED

PASS  data/raw/customers.csv
FAIL  data/raw/tickets-2026-08-01.csv
      missing column: channel
      unexpected column: contact_channel (was a column renamed?)
      order_value: 34 of 34 values are not a valid number (first: line 2, '$10.85')
PASS  data/raw/outcomes.csv
The input does not match the expected schema. Do not load it.

Stopped at step 1 (check_input). No MANIFEST.json was written: this run delivered no dataset.

The first step stops the run. The message names both changes: channel is now contact_channel, and order_value has a $ sign. 34 of 40 tickets have an order value; the other 6 are account tickets with no order, so their empty value is allowed. There is no MANIFEST.json, and no data/model/ folder: the run deleted the old outputs and delivered nothing.

The fix is not in the check. Mei asks Grace about the update, then changes clean.py (rename the column, remove the $) and the schema in check_input.py. The result is a new dataset version, recorded in the Changes table of DATASET.md.

To build the July 1 dataset again, run python run_pipeline.py with no option.

Common mistake

A changed export with no input check

Symptom: a script that worked last month gives strange results or an error far from the cause. For example, the count of “unknown” order values (−1) is suddenly 0, or the script stops with KeyError: 'channel'.

Cause: the export’s schema changed, and nothing checked it before the data was used. pandas reads the August file with no warning:

Interactive Python

>>> import pandas as pd
>>> df = pd.read_csv("data/raw/tickets-2026-08-01.csv")
>>> df["order_value"].head(3).tolist()
['$10.85', '$25.58', '$91.70']
>>> df["order_value"].dtype
<StringDtype(na_value=nan)>
>>> (df["order_value"] == -1).sum()
np.int64(0)
>>> df["channel"]
Traceback (most recent call last):
  ...
KeyError: 'channel'
>>> pd.read_csv("data/raw/tickets-2026-08-01.csv", dtype={"order_value": "Float64"})
Traceback (most recent call last):
  ...
ValueError: Unable to parse string "$10.85" at position 0

order_value is now text. The comparison with −1 does not fail; it is false for every row, so the “unknown” rule finds nothing. The KeyError names a column, but not the reason. The ValueError names a value, but not the column or the file.

Fix:

1. Run scripts/check_input.py on every new export before you load it. It names the file, the missing and the unexpected column, and the column with the bad values.
2. Keep the check as step 1 of run_pipeline.py, so it cannot be forgotten.
3. When the check fails, change the schema and the cleaning on purpose, and make a new dataset version.

In AI work: a model in production gets its input from systems that change. A renamed column can turn a feature into “all missing”, and the model keeps making predictions, only worse. An input check turns that into a clear stop.

Module practice

The module practice: run the preparation pipeline from a clean input snapshot. First, a small worked example.

Worked example: the steps by hand, with their counts

Before she trusts the one command, Mei runs each step by hand once, from a clean snapshot, and writes down what each step prints. Each count must match the lineage records of the last lesson.

Command	What it prints (main lines)	Exit code
python get_data.py --reset	Built larkfield.db. Check: customers 240, tickets 1417, outcomes 1874.	0
python scripts/check_input.py	3 × PASS, The input matches the expected schema.	0
python scripts/extract.py	data/extract/tickets.csv: 1416 rows, data/extract/outcomes.csv: 1869 rows, Not selected: 1 ticket row(s)	0
python scripts/clean.py	"exact_duplicates_removed": 14, "quarantined": 4, "clean_rows": 1398; outcomes "clean_rows": 1834	0
python scripts/build_features.py	"too_recent_for_label": 29, "rows": 1369, "escalated": 188	0
python scripts/split.py	train "rows": 927, valid "rows": 227, test "rows": 215	0
python scripts/validate.py	All 13 rules passed.	0

Every number follows from the one before it: 1,417 − 1 = 1,416; 1,416 − 14 − 4 = 1,398; 1,398 − 29 = 1,369; 927 + 227 + 215 = 1,369. run_pipeline.py runs the same steps, so it must give the same counts.

Your task: a fresh copy

Prove that the project runs anywhere, not only in your folder with its old files and its old environment.

1. Make a new folder, for example ~/projects/ticket-data-fresh.
2. Copy into it only the code and the raw data: get_data.py, q.py, run_pipeline.py, requirements.txt, DATASET.md, the folders queries/ and scripts/, and data/raw/. Do not copy larkfield.db, data/extract/, data/clean/, data/model/, reports/ or .venv/.
3. In the new folder, make a new virtual environment, activate it, and install requirements.txt, as in the Python course.
4. Run python get_data.py.
5. Run python run_pipeline.py.
6. Compare the fingerprint with the one from your project folder.
7. Run the pipeline with the August export, and check that it stops at step 1.
8. Write two sentences in DATASET.md, under Changes or in a new section Reproducibility: the date, the fingerprint, and “two environments gave the same fingerprint”.

Show the model answer

Mei’s fresh copy, in a different folder with a different virtual environment (pandas 3.0.6):

python run_pipeline.py (fresh copy)

1/6 check_input: ok
2/6 extract: ok
3/6 clean: ok
4/6 build_features: ok
5/6 split: ok
6/6 validate: ok
Done: 1369 rows (train 927, valid 227, test 215).
Wrote data/model/MANIFEST.json. Outputs fingerprint: 2b2460321543ffb73fe9d5b04fd8f5a9d77ee990fff37823b0855b911ba4b855

The fingerprint is the same as in the project folder. With --tickets data/raw/tickets-2026-08-01.csv, the run stops at step 1 and names channel, contact_channel and order_value, as in “Your turn”.

The note in DATASET.md: “2026-10-06: a fresh copy with a new virtual environment gave the outputs fingerprint 2b2460321543ffb73fe9d5b04fd8f5a9d77ee990fff37823b0855b911ba4b855, the same as the project folder. The August export stops at the input check.”

If your fingerprints are different, compare the outputs sections of the two manifests to find the first file that differs. Then look at the step that writes it. Common causes: a file that you forgot to copy, a script edited in one folder only, or a different pandas version.

Module check

Check your understanding

Retrieval: name the four kinds of validation rule in a data contract, with one Larkfield example of each.

Show the answer

Required values: ticket_id, created_at, channel and the target must not be missing. Allowed values and ranges: channel is one of email, chat, phone, web_form, unknown; order_value is 0 to 2,000. Unique keys: each ticket_id appears once, and no ticket is in two splits. Expected distributions: the escalation rate is between 5% and 30% overall and in each split.

Check your understanding

Predict: Mei raises MAX_WORDS in clean.py from 10,000 to 50,000 and runs run_pipeline.py. Which step stops, and why? Which checksums would change if the step did not stop?

Show the answer

validate.py (step 6) stops the run. T-10361, with 48,210 words, is no longer quarantined, so it reaches the modelling table. The rule “numbers and dates are in their allowed ranges” fails: word_count: outside 0 to 10000 in 1 row (first: T-10361, 48210). No MANIFEST.json is written. Without the rule, the two extract files would keep their checksums, because extraction comes before cleaning. Every file from clean.py onward would change, so the fingerprint would change too. The cleaning rule and the contract must agree, or the contract stops the run.

Check your understanding

Completion check: show that your preparation is reproducible and that it fails in a meaningful way when the schema changes. What evidence do you give Priya?

Show the answer

Two pieces of evidence, both from your own runs:

1. Reproducible outputs. Two runs of python run_pipeline.py from a clean snapshot (python get_data.py --reset), with the same outputs_fingerprint. The outputs sections of the two manifests are equal; only started_at and extracted_at differ. A run in a fresh copy with a new environment makes the evidence stronger.
2. A meaningful failure. The output of python run_pipeline.py --tickets data/raw/tickets-2026-08-01.csv: it stops at step 1, names channel as missing, contact_channel as unexpected and order_value as not a number with the example $10.85, and writes no MANIFEST.json.

The check is demonstrated when both are shown. It is partially demonstrated in two cases: the changed export is not tested, or the failure message does not name the columns. It is not yet demonstrated if the steps must be run by hand, or if the fingerprints differ without an explanation.

See it end to endPrepare a leak-free table for city repair requestsCase study

The repair-requests case study delivers a pipeline that anyone can run again, with a dataset card and a manifest.

Recap

In this module you turned a set of scripts into a dataset that another person can trust and run again.

• Validation rules: a data contract lists required values, allowed values and ranges, unique keys and expected distributions. validate.py checks 13 rules on the delivered files, writes reports/validation.json and exits with code 1 on failure. A rule must be tested on data that breaks it.
• Lineage: the snapshot table, manifest.json, cleaning.json, quarantine.csv, build_summary.json and split_manifest.json record every step. DATASET.md gives the provenance (synthetic, generate.py, seed 2026, CC0), the transformation order, the versions and the limitations.
• Delivery: python get_data.py --reset and python run_pipeline.py build the dataset from a clean snapshot with one command. MANIFEST.json holds a SHA-256 checksum of every output, apart from the run time, and two runs give the same fingerprint. check_input.py stops a changed export before it is loaded, with a message that names the problem.

Next come three case studies on real data. Complete at least one of them for the certificate. Then the final assignment: you prepare the dataset from the start and deliver it with the evidence for each step.

Next: Explain an online shop's sales