Track lineage

What you will be able to do

Record where the dataset came from and every step that changed it, in order. Then Priya can trust it, repeat it, and answer the question “why is this value different from the export?”

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook follows ticket T-10013 through the files, reads every lineage record, checks the provenance, writes DATASET.md and scripts/trace.py, and compares the card with the records. Its first cells install pandas, download the practice data and rebuild the files of the earlier lessons, so it works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• What a data contract is, and what validate.py checks, from Validation rules.
• What each step of the pipeline does:
  • extraction, from Reproducible extraction;
  • cleaning, from Handle messy values;
  • the target, from Features and target timing;
  • the split, from Choose a split.

What you need on your computer (only for the local route; the notebook and the box on this page need none of it)

• The project ~/projects/ticket-data with larkfield.db and get_data.py, from Set up a practice database: Build the database. If the database is missing or changed, reset the database with python get_data.py --reset.
• The virtual environment .venv with pandas 3.0.6, from Profile before changing: The same profile in pandas. Activate it before you run a script.
• The lineage records of the earlier lessons:
  • data/extract/manifest.json, from Reproducible extraction: Build extract.py.
  • reports/cleaning.json and data/clean/quarantine.csv, from Handle messy values: Finish clean.py.
  • data/model/build_summary.json, from Features and target timing: build_features.py.
  • data/model/split_manifest.json, from Choose a split: Make and check the split.
  • scripts/validate.py and reports/validation.json, from Validation rules: The contract in validate.py. python scripts/validate.py must pass all 13 rules.
• If a record is missing or does not match the numbers on this page, run the scripts again in this order: extract.py, clean.py, build_features.py, split.py, validate.py. If a script is missing or broken, first copy the course’s finished version from C04/pipeline into the same place in your project.

Tested with Python 3.14.6 and pandas 3.0.6 on macOS. The notebook was tested in Jupyter.

One ticket’s journey

Priya opens train.csv and finds ticket T-10013 with an order value of 32.6. Grace says: “In the help desk, that ticket shows 3260.” Who is right? Mei follows the ticket through the files:

Where	channel	order_value	What explains the change
tickets table in larkfield.db	webform	3260.0	The export of 2026-07-01 06:00.
data/extract/tickets.csv	webform	3260.0	Selected: created on 2026-01-06, inside the selection in manifest.json.
data/clean/tickets.csv	web_form	32.6	clean.py: respelled channel; cents ÷ 100. The row keeps channel_raw = webform, order_value_raw = 3260.0 and order_value_flag = converted_from_cents. cleaning.json counts 24 such rows.
data/model/train.csv	web_form	32.6	Created before 2026-05-01, so in train (split_manifest.json).

Both are right. Grace sees the raw value, in cents from the old web form. Priya sees the corrected value in dollars. Mei can show each step, and the file that records it. That is data lineage: the record of where each piece of data came from, and of every step that changed it, in order.

Lineage starts with provenance: the origin of the data. Who made it, when, how, and with what permission. Provenance answers “where did it start?” Lineage answers “what happened to it since then?”

The lineage of the delivered dataset

The diagram shows the whole path, from the export files to the delivered dataset. Each step writes a small file that records what it did. The diamonds are gates: checks that stop the path when something is wrong.

The dataset contract and its lineage: every step leaves a record, and three gates stop the pipeline before a broken dataset is delivered.

Text description of the diagram

The diagram has three parts, from top to bottom.

1. Source and input gates. The export files (customers, tickets, outcomes) were made by generate.py with seed 2026, under the CC0 licence. Two gates check them. get_data.py checks that each file’s checksum matches; if yes, it builds larkfield.db, the snapshot of 2026-07-01 06:00 with 1,417 ticket rows. check_input.py checks that the schema is as expected. If either gate says no, the pipeline stops, and the message names the problem.
2. Preparation. When both gates pass, four steps run in order, and each writes a lineage record. extract.py: 1,416 tickets and manifest.json. clean.py: 1,398 tickets, 4 quarantined, and cleaning.json. build_features.py: 1,369 rows and build_summary.json. split.py: train 927, valid 227, test 215, and split_manifest.json.
3. Output gate and delivery. validate.py checks the 13 rules of the contract. If they pass, the delivered dataset is the train, valid and test files, with MANIFEST.json and DATASET.md. If they fail, the pipeline stops, and no dataset is delivered.

You already wrote most of the lineage. Each module added a record:

Record	Written by	It answers
snapshot table in larkfield.db	get_data.py	Which export is this, and which data version?
data/extract/manifest.json	extract.py (Module 3)	Which rows were selected, with which query, and how many were left out?
reports/cleaning.json	clean.py (Module 4)	How many rows were removed, corrected or quarantined, and why?
data/clean/quarantine.csv	clean.py (Module 4)	Which tickets were set aside, each with its reason?
data/model/build_summary.json	build_features.py (Module 5)	How many tickets were too recent for a final target?
data/model/split_manifest.json	split.py (Module 5)	Which rule made the split, and how big is each part?
reports/validation.json	validate.py (this module)	Did the contract pass?

The first record is in the database itself:

Run SQL in your browser

Try it: the snapshot record

Run the query. You should see 3 rows: the source, exported_at (2026-07-01 06:00:00) and data_version (1.0). Every later record starts from these three values.

SQL query (you can change it)SELECT name, value FROM snapshot;

extract.py copies these values into its manifest, with the selection and a checksum of each query:

data/extract/manifest.json

{
  "extracted_at": "2026-10-07T03:05:39+00:00",
  "source": {
    "source": "Larkfield help-desk export (synthetic, Nextia Learning C04)",
    "exported_at": "2026-07-01 06:00:00",
    "data_version": "1.0"
  },
  "selection": {
    "created_from": "2026-01-01",
    "created_before": "2026-07-01",
    "rule": "created_from <= created_at < created_before"
  },
  "ticket_rows_in_source": 1417,
  "ticket_rows_not_selected": 1,
  "files": [
    {
      "query": "queries/extract_tickets.sql",
      "query_sha256": "28a0267e280062b0e02f9727cb5a95d2c8b60309e5fc992a578e44ba61b54338",
      "file": "data/extract/tickets.csv",
      "rows": 1416
    },
    {
      "query": "queries/extract_outcomes.sql",
      "query_sha256": "533e9573c06975a463369aea503c4218c806640b99067d240607c1cea6fdc864",
      "file": "data/extract/outcomes.csv",
      "rows": 1869
    }
  ]
}

Your extracted_at is the time of your own run, so it is different. Every other value must be the same as here.

clean.py writes the counts of Module 4. They add up: 1,416 rows in, 14 duplicates removed, 4 quarantined, 1,398 clean.

reports/cleaning.json

{
  "tickets": {
    "input_rows": 1416,
    "exact_duplicates_removed": 14,
    "channel_respelled": 39,
    "order_value_ok": 1179,
    "order_value_not_applicable": 188,
    "order_value_converted_from_cents": 24,
    "order_value_unknown": 11,
    "quarantined": 4,
    "clean_rows": 1398
  },
  "outcomes": {
    "input_rows": 1869,
    "repeated_events_removed": 30,
    "events_of_quarantined_tickets": 5,
    "clean_rows": 1834
  }
}

Provenance: where the export came from

The provenance of this dataset has three parts.

Synthetic generation. Synthetic data is data that a program makes, instead of data that a real system records. The Larkfield export comes from C04/generate.py in the nextia-learning-labs repository. It uses only the Python standard library. It adds each problem on purpose and in a known amount: the 14 duplicates, the cents, the leaky columns. The data is made up for this course; no real people are in it.

The seed. The generator uses random numbers. A seed is the start value of a random number generator. With the same seed, the generator makes the same numbers in the same order. generate.py uses seed 2026, so every run writes the same files.

Checksums. How do you know that your tickets.csv is the file that the generator wrote? A checksum is a short fingerprint of a file’s bytes. SHA-256 is a common method. If one byte changes, the checksum changes completely. get_data.py stores the expected checksum of each file, and refuses a file that does not match. You check it yourself in the guided practice.

Licence. A licence says what you may do with data. The Larkfield files are under CC0 1.0: a public-domain dedication. You may copy, change and share them for any purpose, also commercial, without permission and without attribution. The labs repository has a dataset card for the export, C04/data/dataset.md, with the licence, the checksums and the list of problems.

Real data needs permission

With real data, the licence and the permission come first. A dataset that you can download is not always a dataset that you may train a model on. Customer tickets can contain personal data, and their use can need a legal basis or the customer’s consent. Record the licence and the permission in the provenance before you use the data.

Dataset versions

The export has a version, data_version 1.0. Mei’s delivered dataset needs a version too, because Priya must say which data trained a model. Which changes make a new version?

Change	New version?	Why
A new export, for example August	Yes	Different rows.
A changed rule in clean.py, for example a new limit for word_count	Yes	The same export gives different rows.
A changed selection, for example tickets from February only	Yes	Different rows.
The same scripts run again on the same snapshot	No	The output must be identical. The next lesson proves it with checksums.

A version number alone does not prove anything. The checksums of the output files do. The next lesson adds them.

Worked example: the dataset card

A dataset card (also called a datasheet) is a short document that travels with a dataset. It says what the data is, where it came from, how it was made, and what it must not be used for. Mei writes DATASET.md in the project folder. She copies every number from a lineage record, not from memory.

DATASET.md

# Dataset card: Larkfield ticket escalation dataset

Prepared by Mei (data engineer) for Priya (data scientist). Version 1.0 of the dataset, made from data version 1.0 of the export.

## What it is

One row per support ticket of Larkfield, a fictional online shop for home and garden products. The target, `escalated_72h`, is 1 if a senior agent took the ticket within 72 hours of its creation. Every feature is known when the ticket is created, so a model can use it at that moment.

| File | Rows | What one row is |
|---|---|---|
| `data/model/train.csv` | 927 | a ticket created before 2026-05-01 |
| `data/model/valid.csv` | 227 | a ticket created in May 2026 |
| `data/model/test.csv` | 215 | a ticket created from 2026-06-01 to 2026-06-27 |
| `data/model/tickets_model.csv` | 1,369 | all three parts together |

Escalation rate: 13.7% overall (188 of 1,369); train 11.7%, valid 19.8%, test 16.3%.

## Provenance

| Field | Value |
|---|---|
| Source | The Larkfield help-desk export: `customers.csv`, `tickets.csv`, `outcomes.csv` |
| Snapshot | Exported 2026-07-01 06:00:00, data version 1.0 (the `snapshot` table in `larkfield.db`) |
| How the source was made | **Synthetic.** `C04/generate.py` in `nextia-learning-labs`, Python standard library only, seed 2026. No real system, customer or agent. |
| Licence | CC0 1.0 (public domain). No attribution needed. |
| Raw file checksums (SHA-256) | `customers.csv` 7b437642f0095524f8311f67a7caeb17100d23eed92618b9cf012cd3d3b79993<br>`tickets.csv` 9c47e08cf2ec5977733d32cc21eb97d0dd01b021d1f5fbf7ec31fef6279b1a28<br>`outcomes.csv` 67b4d9d9275b32061a8888050c351df7314054c11ee1ce29e856c1d5403d0653 |
| Selection | Tickets with `2026-01-01 <= created_at < 2026-07-01` |

## Transformation order

| Step | Script | Rows in → out | Lineage record |
|---|---|---|---|
| 0 | `get_data.py` | 3 CSV files → `larkfield.db` (tickets 1,417 rows) | checksums of the raw files |
| 1 | `scripts/extract.py` | 1,417 → 1,416 ticket rows (1 row created in 2036 is outside the selection); 1,874 → 1,869 outcome events | `data/extract/manifest.json` |
| 2 | `scripts/clean.py` | 1,416 → 1,398 tickets (14 exact duplicates removed, 4 quarantined); 1,869 → 1,834 events | `reports/cleaning.json`, `data/clean/quarantine.csv` |
| 3 | `scripts/build_features.py` | 1,398 → 1,369 (29 tickets are too recent for a final target) | `data/model/build_summary.json` |
| 4 | `scripts/split.py` | 1,369 → train 927, valid 227, test 215 | `data/model/split_manifest.json` |
| 5 | `scripts/validate.py` | 13 rules, no change to the data | `reports/validation.json` |

## Columns

| Column | Type | Missing allowed | Meaning |
|---|---|---|---|
| `ticket_id` | text | no | unique, for example `T-10001` |
| `customer_id` | text | yes (guests) | for example `C-0001` |
| `created_at` | datetime | no | `YYYY-MM-DD HH:MM:SS`, 2026-01-05 to 2026-06-27 |
| `channel` | text | no | `email`, `chat`, `phone`, `web_form`, `unknown` (was missing) |
| `team` | text | no | `delivery`, `returns`, `payment`, `warranty`, `account` |
| `priority` | integer | no | 1 (most urgent) to 3, set at creation |
| `segment` | text | no | `home`, `trade`, `business`, `guest` (no customer), `unknown` (deleted customer) |
| `region` | text | no | `north`, `south`, `east`, `west`, `unknown` |
| `order_value` | number | yes | dollars, 0 to 2,000. Missing for account tickets (no order) and where the order system said "unknown" (−1). January web-form values were cents and are divided by 100. |
| `word_count` | integer | no | words in the first message, 0 to 10,000 |
| `customer_tenure_days` | number | yes (guests, deleted customers) | days from `joined_on` to `created_at` |
| `prior_tickets_90d` | integer | no | the same customer's tickets in the 90 days before this one |
| `created_hour` | integer | no | 0 to 23 |
| `escalated_72h` | integer | no | the target: 0 or 1 |

Never in the dataset: `priority_now`, `first_reply_minutes`, `closed_at`, and anything from the outcome events after creation. They are known only after the ticket is created (leakage).

## Tool versions

Tested on macOS: Python 3.14.6 (SQLite 3.50.4), pandas 3.0.6.

## Known limitations

- **Synthetic data.** The relations between columns are simple and were chosen for teaching. Do not draw conclusions about real help desks or real customers from this data.
- **The escalation rate changes over time.** It rises after 2026-05-01, when a new warranty policy started (Grace, support lead). Train has 11.7%, valid 19.8%. A model trained on train only sees the old policy.
- **Small test set.** 215 tickets, 35 of them escalated. A difference of a few tickets changes a score noticeably.
- **Customers repeat across parts.** 77 of the 89 customers in test also have tickets in train. Results on test say how the model does for known customers; for new customers they can be worse.
- **Corrected values.** 24 order values were converted from cents. 11 "unknown" values (−1) are now missing. The originals are in `data/clean/tickets.csv` (`order_value_raw`, `order_value_flag`).
- **Excluded rows.** 4 tickets are in quarantine (3 created before the customer joined, 1 with 48,210 words). 29 tickets from the last 72 hours before the snapshot have no final target and are not in the dataset.
- **What not to conclude.** The dataset is for learning data preparation. A model trained on it is a demonstration, not a tool for routing real tickets.

## Changes

| Version | Date | Change |
|---|---|---|
| 1.0 | 2026-10-06 | First delivery, from the export of 2026-07-01 06:00 (data version 1.0). |

Three sections matter most to Priya:

• Transformation order. The steps are not interchangeable. For example, build_features.py counts prior_tickets_90d from the clean tickets, so the 14 duplicates must be gone first. If the order changes, the values change.
• Known limitations. These are the facts that a score cannot show: the policy change, the small test set, the customers that appear in train and test. The 77 of 89 comes from split_manifest.json.
• What not to conclude. A dataset card protects the data from wrong use, not only from wrong values.

The next lesson adds a section with the checksums of the output files.

Guided practice: check the provenance and trace a ticket

1. In the terminal, go to your project folder and check that the environment is active.

2. Start Python’s interactive mode with python. Calculate the checksum of tickets.csv:

   Interactive Python

   >>> import hashlib
   >>> from pathlib import Path
   >>> hashlib.sha256(Path("data/raw/tickets.csv").read_bytes()).hexdigest()
   '9c47e08cf2ec5977733d32cc21eb97d0dd01b021d1f5fbf7ec31fef6279b1a28'

3. Open get_data.py and find FILES. Compare the value for tickets.csv with your result. They must be equal.

4. Type exit() to leave interactive mode.

5. Make scripts/trace.py. It shows in which files a ticket appears:

   scripts/trace.py

   """Show where one ticket is in each file of the pipeline."""
   
   import sys
   from pathlib import Path
   
   FILES = ["data/extract/tickets.csv", "data/clean/tickets.csv", "data/clean/quarantine.csv",
            "data/model/train.csv", "data/model/valid.csv", "data/model/test.csv"]
   
   ticket = sys.argv[1]
   for path in FILES:
       lines = Path(path).read_text(encoding="utf-8").splitlines()
       found = [line for line in lines if line.startswith(ticket + ",")]
       print(f"{path:<26} {len(found)} row(s)")

6. Run it for T-10013:

   python scripts/trace.py T-10013

   data/extract/tickets.csv   1 row(s)
   data/clean/tickets.csv     1 row(s)
   data/clean/quarantine.csv  0 row(s)
   data/model/train.csv       1 row(s)
   data/model/valid.csv       0 row(s)
   data/model/test.csv        0 row(s)

7. Make DATASET.md in the project folder, with the sections of Mei’s card. Copy each number from the record that the table above names.

Your turn: three tickets, three records

Trace three more tickets with scripts/trace.py: T-10031, T-10361 and T-10147. Predict the result first. For each ticket, name the lineage record that explains where it went.

Show the answer

python scripts/trace.py T-10031

data/extract/tickets.csv   2 row(s)
data/clean/tickets.csv     1 row(s)
data/clean/quarantine.csv  0 row(s)
data/model/train.csv       1 row(s)
data/model/valid.csv       0 row(s)
data/model/test.csv        0 row(s)

T-10031 is one of the 14 exact duplicates: 2 rows in the extract, 1 after cleaning. The record is exact_duplicates_removed: 14 in reports/cleaning.json.

python scripts/trace.py T-10361

data/extract/tickets.csv   1 row(s)
data/clean/tickets.csv     0 row(s)
data/clean/quarantine.csv  1 row(s)
data/model/train.csv       0 row(s)
data/model/valid.csv       0 row(s)
data/model/test.csv        0 row(s)

T-10361 is in quarantine, with the reason word_count_above_10000 in its row of data/clean/quarantine.csv. cleaning.json counts it in quarantined: 4.

python scripts/trace.py T-10147

data/extract/tickets.csv   0 row(s)
data/clean/tickets.csv     0 row(s)
data/clean/quarantine.csv  0 row(s)
data/model/train.csv       0 row(s)
data/model/valid.csv       0 row(s)
data/model/test.csv        0 row(s)

T-10147 is in no file. It is the ticket with the year 2036, so it is outside the selection. The record is ticket_rows_not_selected: 1 in data/extract/manifest.json. A ticket that is in no file still has lineage: the manifest says why it is not there.

Common mistake

The rules pass, but the card is wrong

Symptom: Priya counts 1,159 rows in tickets_model.csv. DATASET.md says 1,369. validate.py passes all 13 rules.

Cause: Mei ran the extraction again with a different selection, python scripts/extract.py --created-from 2026-02-01, and then the other steps. The new files are valid: 1,204 tickets extracted, 1,188 clean, 1,159 in the modelling table, 717 in train. But nobody changed the card. The contract checks the data, not the document about the data.

Fix:

1. Compare the card with the records. The selection in data/extract/manifest.json says "created_from": "2026-02-01". The card says 2026-01-01.
2. Decide: is the new selection wanted? If not, run the steps again with the default selection.
3. If yes, it is a new dataset version. Update every number in the card from the new records, and add a row to the Changes table.

In AI work: a model card or a paper that quotes the wrong dataset is very hard to correct later. Results cannot be repeated, and nobody knows which data trained the model. The next lesson makes this check automatic: MANIFEST.json records the selection and a checksum of every output in each run.

Check your understanding

What is the difference between provenance and lineage? Give one Larkfield example of each.

Show the answer

Provenance is the origin: who made the data, when, how, and under which licence. Example: the export of 2026-07-01 06:00, made by generate.py with seed 2026, under CC0. Lineage includes the provenance and adds every step after it, in order. Example: T-10013’s order value went from 3260.0 to 32.6 in clean.py, and cleaning.json counts 24 such conversions.

Check your understanding

Predict: somebody runs generate.py with seed 2027 and puts the new tickets.csv in data/raw/. What does get_data.py do?

Show the answer

A different seed makes different random numbers, so the file has different bytes and a different SHA-256 checksum. get_data.py sees that the checksum does not match FILES. It downloads the original file again and checks it, so the database is built from the real 2026 export. The checksum is the gate: a file that is not the expected file never becomes the snapshot.

Check your understanding

Omar asks: 'Can we publish this dataset on our website, and can we train a model for real tickets on it?' Answer both questions.

Show the answer

Publish: yes. The export is under CC0 1.0, so Larkfield may copy and share it for any purpose, without permission or attribution. Train for real tickets: no. The data is synthetic, and its relations were chosen for teaching. The card says so under “What not to conclude”. The licence answers “may we use it?”. The limitations answer “should we use it for this?”.

Recap

• Data lineage is the record of where data came from and every step that changed it, in order. Provenance is the start of it: who made the data, how, when and under which licence.
• The pipeline already writes the lineage records: the snapshot table, manifest.json, cleaning.json, quarantine.csv, build_summary.json, split_manifest.json and validation.json.
• The Larkfield export is synthetic, made by generate.py with seed 2026, under CC0 1.0. A checksum proves that a file is the expected file.
• A new export, a changed rule or a changed selection makes a new dataset version. Running the same steps on the same snapshot must not.
• DATASET.md is the dataset card: provenance, transformation order, columns, versions and limitations. Copy every number from a record.

Next, you put all the steps behind one command. You prove that it gives the same files twice, and you make it fail clearly when the export changes.

Next: Deliver a reusable dataset