Validation rules

What you will be able to do

Write down what Priya can rely on in the dataset, as rules that a script checks every time. Then a broken dataset stops with a clear message before it reaches a model.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every query, script, task and check of this lesson, with the instructions. It builds scripts/validate.py in four cells, then breaks the data on purpose to test the rules. Its first cells install pandas, download the practice data and rebuild the files of Modules 3 to 5, so it works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• GROUP BY and HAVING, from Aggregate correctly. NOT IN and NULL, from Select and filter.
• The modelling table and its target, from Features and target timing, and the time split, from Choose a split.
• Which columns leak the answer, from Fit transformations correctly.

What you need on your computer (only for the local route; the notebook and the boxes on this page need none of it)

• The project ~/projects/ticket-data with larkfield.db, made in Set up a practice database: Build the database. If the database is missing or changed, reset the database with python get_data.py --reset.
• The virtual environment .venv with pandas 3.0.6, made in Profile before changing: The same profile in pandas. Activate it before you run a script.
• The scripts and outputs of Modules 3 to 5, run in this order:
  • scripts/extract.py, queries/extract_tickets.sql and queries/extract_outcomes.sql, from Reproducible extraction: Build extract.py. They write data/extract/.
  • scripts/clean.py, from Handle messy values: Finish clean.py. It writes data/clean/ and reports/cleaning.json.
  • scripts/build_features.py, from Features and target timing: build_features.py. It writes data/model/tickets_model.csv.
  • scripts/split.py, from Choose a split: Make and check the split. It writes train.csv, valid.csv and test.csv in data/model/.
• If a script is missing or broken, copy the course’s finished version from C04/pipeline into the same place in your project. Then run the four scripts again, in the order above. This lesson changes build_features.py and test.csv on purpose; the steps say how to repair them.
• The saved queries of this module go in queries/, numbered from 60-.

Tested with Python 3.14.6 and pandas 3.0.6 on macOS, and with SQLite 3.49.1 in the browser. The notebook was tested in Jupyter.

A promise about the data

Priya will train a model on Mei’s files. She asks: “What can I rely on? Is ticket_id unique? Can channel be missing? Can the target be 2?” Mei answers with a list of rules. Each rule says what must be true in every delivered file.

Two terms first:

• The schema of a table is the list of its columns and the type of each column. For example: ticket_id is text, priority is an integer.
• A data contract is an agreement between the person who makes a dataset and the people who use it. It contains the schema and the rules for the values. The rules say which values are allowed, which columns must have a value, and which column is a unique key. They also say what the data looks like as a whole.

A validation rule is one check from the contract. Here are five tickets and four rules. Read the table and find the problems before you read the last column.

ticket_id	channel	priority	escalated_72h	Breaks a rule?
T-1	email	2	0	no
T-2	Email	1	1	allowed values: Email is not in the list
T-3	chat	5	0	range: priority must be 1 to 3; unique key: T-3 is twice
T-3	chat	5	0	the same two rules
T-4	NULL	3	0	required value: channel is missing

The four rules:

1. Required values. channel must have a value in every row.
2. Allowed values and ranges. channel is one of email, chat, phone, web_form. priority is 1 to 3.
3. Unique key. Each ticket_id appears once.
4. Expected distribution. Between 5% and 30% of the tickets are escalated. Here 1 of 5 is escalated: 20%. This rule passes.

The first three rules look at single rows. The fourth rule looks at the whole table. It catches a different kind of mistake: every row looks fine, but the total is wrong. For example, if a bug in the target code marks almost no ticket as escalated, every value is still 0 or 1. Only the rate shows the bug.

None of these problems raises an error in SQL or pandas. A rule makes a quiet mistake loud.

Rules as queries that must return no rows

A useful way to write a rule: write a query that returns the rows that break the rule. The rule passes when the query returns 0 rows. When it fails, the result shows you the bad rows, not only “false”.

The boxes below run each rule on the raw export, the tickets table in larkfield.db. The raw export is not clean, so most rules find problems here. After the pipeline, the same rules must find nothing.

Required columns. The first rule checks the schema itself: are the columns there? SQLite lists a table’s columns with pragma_table_info('tickets'). The query lists each expected column that the table does not have.

Run SQL in your browser

Try it: are the required columns there?

Run the query. The box says “The statement ran and returned no rows”: every expected column exists. Then change 'channel' to 'contact_channel' and run it again. You should see one row, contact_channel. In the last lesson of this module, a new export renames channel in this way.

SQL query (you can change it)WITH expected(name) AS ( VALUES ('ticket_id'), ('customer_id'), ('created_at'), ('channel'), ('team'), ('priority'), ('order_value'), ('word_count') ) SELECT name AS missing_column FROM expected WHERE name NOT IN (SELECT name FROM pragma_table_info('tickets'));

Unique key. A primary key is a column whose value is different in every row, so it names one row. The export does not guarantee one, so you check it.

Predict first: how many ticket IDs appear more than once? You found them in Module 1.

Run SQL in your browser

Try it: is ticket_id unique?

Run the query. You should see 14 rows, from T-10031 to T-11291, each with 2 copies. These are the exact duplicate rows from the export page overlap. clean.py removes them.

SQL query (you can change it)SELECT ticket_id, COUNT(*) AS copies FROM tickets GROUP BY ticket_id HAVING COUNT(*) > 1 ORDER BY ticket_id;

Allowed values. The next query looks correct, but it has a quiet mistake. Predict: how many groups does it return, and is the missing channel among them?

Run SQL in your browser

Your turn: catch every channel that is not allowed

Run the query. You see 4 groups: chat, EMAIL, Email and webform. The 38 rows with no channel are not there. For a NULL channel, channel NOT IN (...) is not true, it is unknown, so WHERE drops the row. Change the query so that it also returns the NULL group. The check compares your result with ours.

SQL query (you can change it)SELECT channel, COUNT(*) AS n FROM tickets WHERE channel NOT IN ('email', 'chat', 'phone', 'web_form') GROUP BY channel;

Show the answer

SELECT channel, COUNT(*) AS n
FROM tickets
WHERE channel IS NULL
   OR channel NOT IN ('email', 'chat', 'phone', 'web_form')
GROUP BY channel
ORDER BY n DESC;

Five groups: NULL 38, Email 20, chat 8, EMAIL 7, webform 5. That is 78 rows. A rule about allowed values must say what it does with a missing value. Here, missing is not allowed.

Ranges. A range rule gives the smallest and the largest allowed value. Grace says that an order above $2,000 is very rare at Larkfield, and a negative order is not possible. Predict: which two problems from Module 4 does this rule find?

Run SQL in your browser

Try it: order values outside 0 to 2,000

Run the query. You should see 36 rows. The first 11 have -1, the “unknown” sentinel. The other 25 are January web-form tickets, in cents. The last of them, T-10147, is also the ticket with the year 2036.

SQL query (you can change it)SELECT ticket_id, created_at, channel, order_value FROM tickets WHERE order_value < 0 OR order_value > 2000 ORDER BY order_value;

Run SQL in your browser

Your turn: a range rule for word_count

The contract says: word_count is 0 to 10,000. Zero is allowed, because a message can be an attachment only. Write a query that returns ticket_id and word_count for every row that breaks this rule. The check compares your result with ours.

SQL query (you can change it)-- Write your query here

Show the answer

SELECT ticket_id, word_count
FROM tickets
WHERE word_count < 0
   OR word_count > 10000;

One row: T-10361 with 48,210 words, the pasted log file. clean.py puts it in quarantine with the reason word_count_above_10000.

On your computer: all the rules in one report

On your computer, you can put the rules in one query that counts the problems of each rule. UNION ALL puts the results of several SELECT statements one under the other. Save this as queries/60-contract-checks.sql:

queries/60-contract-checks.sql

-- Each rule counts the rows that break it. A rule passes with 0.
SELECT 'ticket_id is unique' AS rule,
       (SELECT COUNT(*) FROM (SELECT ticket_id FROM tickets
                              GROUP BY ticket_id HAVING COUNT(*) > 1)) AS problems
UNION ALL
SELECT 'required values are present', COUNT(*)
FROM tickets
WHERE ticket_id IS NULL OR created_at IS NULL OR team IS NULL OR priority IS NULL
UNION ALL
SELECT 'channel has an allowed value', COUNT(*)
FROM tickets
WHERE channel IS NULL OR channel NOT IN ('email', 'chat', 'phone', 'web_form')
UNION ALL
SELECT 'order_value is 0 to 2000', COUNT(*)
FROM tickets
WHERE order_value < 0 OR order_value > 2000
UNION ALL
SELECT 'word_count is 0 to 10000', COUNT(*)
FROM tickets
WHERE word_count < 0 OR word_count > 10000
UNION ALL
SELECT 'created_at is in the first half of 2026', COUNT(*)
FROM tickets
WHERE created_at < '2026-01-01' OR created_at >= '2026-07-01';

python q.py queries/60-contract-checks.sql

rule                                     problems
---------------------------------------  --------
ticket_id is unique                      14
required values are present              0
channel has an allowed value             78
order_value is 0 to 2000                 36
word_count is 0 to 10000                 1
created_at is in the first half of 2026  1

6 rows

The first row counts ticket IDs, the others count rows. One rule passes on the raw export: no ticket misses its ID, time, team or priority. The other five find the problems that Modules 3 to 5 handle. That is the point: the rules describe the data after the pipeline.

The contract in validate.py

The SQL rules check the raw table in the database. Priya receives CSV files, so Mei writes the contract again in Python, for the files in data/model/. The script is scripts/validate.py. It uses pandas only, with no extra library. It has four parts.

Part 1: the contract as data

The rules are at the top of the file, as plain Python values. Anyone can read them without reading the code, and a change to the contract is a change to these lines.

scripts/validate.py (part 1 of 4)

"""Check the delivered dataset against its contract: the rules that Priya can
rely on in data/model/. Prints PASS or FAIL for each rule, writes
reports/validation.json, and exits with code 1 if any rule fails."""

import json
import sys
from pathlib import Path

import pandas as pd

MODEL = Path("data/model")
REPORT = Path("reports/validation.json")
TIME_FORMAT = "%Y-%m-%d %H:%M:%S"
SPLITS = ["train", "valid", "test"]

# ---------------------------------------------------------------- the contract

# Every column, its type, and whether a value is required.
COLUMNS = {
    "ticket_id": ("text", True),
    "customer_id": ("text", False),            # guests have no customer
    "created_at": ("datetime", True),
    "channel": ("text", True),
    "team": ("text", True),
    "priority": ("integer", True),
    "segment": ("text", True),
    "region": ("text", True),
    "order_value": ("number", False),          # no order (account) or unknown
    "word_count": ("integer", True),
    "customer_tenure_days": ("number", False),  # guests and deleted customers
    "prior_tickets_90d": ("integer", True),
    "created_hour": ("integer", True),
    "escalated_72h": ("integer", True),        # the target
}
# Known only after the ticket is created: never in the dataset.
NEVER = ["priority_now", "first_reply_minutes", "closed_at"]
ALLOWED = {
    "channel": {"email", "chat", "phone", "web_form", "unknown"},
    "team": {"delivery", "returns", "payment", "warranty", "account"},
    "segment": {"home", "trade", "business", "guest", "unknown"},
    "region": {"north", "south", "east", "west", "unknown"},
}
RANGES = {  # inclusive; None means no limit
    "priority": (1, 3),
    "order_value": (0, 2000),
    "word_count": (0, 10_000),
    "customer_tenure_days": (0, None),
    "prior_tickets_90d": (0, None),
    "created_hour": (0, 23),
    # From the start of the selection to the snapshot minus 72 hours.
    "created_at": (pd.Timestamp("2026-01-01"), pd.Timestamp("2026-06-28 06:00:00")),
}
ESCALATION_RATE = (0.05, 0.30)

Each value comes from a decision in an earlier module:

Rule	Why this value
order_value and customer_tenure_days may be missing	Account tickets have no order, and the order system’s −1 now is missing (Module 4). Guests and deleted customers have no join date.
channel allows unknown	clean.py turns a missing channel into its own category, unknown.
NEVER	These columns are filled in after the ticket is created. They leak the answer (Module 5).
created_at up to 2026-06-28 06:00	A ticket needs its full 72 hours before the snapshot, or its target is not final.
Escalation rate 5% to 30%	The export has 13.7%. Grace expects the rate to move with policy changes, but not by a factor of 3. A rate outside this range means that the target code changed, not the help desk.

Part 2: small helpers

scripts/validate.py (part 2 of 4)

# ---------------------------------------------------------------- helpers

results = []


def rule(name, problems):
    """Record one rule. An empty list of problems means the rule passes."""
    results.append({"rule": name, "passed": not problems, "problems": problems})
    print(f"{'PASS' if not problems else 'FAIL'}  {name}")
    for problem in problems:
        print(f"      {problem}")


def first(df, mask, column):
    """The first bad row, as an example: its ticket ID and its value."""
    row = df[mask].iloc[0]
    value = row[column]
    shown = repr(value) if isinstance(value, str) else value
    return f"first: {row['ticket_id']}, {shown}"


def n(count, word):
    """'1 row', '3 rows'."""
    return f"{count} {word}" + ("" if count == 1 else "s")


def load(path):
    """Read every value as text. Types are checked by the rules, not assumed."""
    return pd.read_csv(path, dtype=str, keep_default_na=False, na_values=[""])


def convert(text, kind):
    """Convert a text column to its type. A value that cannot be read becomes missing."""
    if kind == "text":
        return text
    if kind == "datetime":
        return pd.to_datetime(text, format=TIME_FORMAT, errors="coerce")
    numbers = pd.to_numeric(text, errors="coerce")
    if kind == "integer":
        numbers = numbers.where(numbers % 1 == 0)  # 2.5 is not an integer
    return numbers

rule() is the heart of the script. Each check makes a list of problems, and rule() prints PASS or FAIL with the problems under it. Every message names the column, the number of rows and the first bad ticket, so you can find it.

load() reads every value as text. Why not let pandas choose the types? Because pandas chooses quietly. If one value in order_value is $10.85, pandas reads the whole column as text, with no warning. convert() then tries each value. A value that is present but cannot be converted is a type problem, and the next part counts it.

Part 3: rules for the modelling table

scripts/validate.py (part 3 of 4)

# ---------------------------------------------------------------- the rules

def check_table(raw):
    """Rules for the modelling table, one row per ticket. Returns the typed table."""
    missing = [c for c in COLUMNS if c not in raw.columns]
    leaky = [c for c in raw.columns if c in NEVER]
    extra = [c for c in raw.columns if c not in COLUMNS and c not in NEVER]
    rule("required columns are present",
         [f"missing column: {c}" for c in missing])
    rule("no column that is known only after creation",
         [f"{c} must not be in the dataset" for c in leaky] + [f"unexpected column: {c}" for c in extra])
    if missing:
        return None

    df = raw.copy()
    problems = []
    for column, (kind, _) in COLUMNS.items():
        df[column] = convert(raw[column], kind)
        bad = raw[column].notna() & df[column].isna()
        if bad.any():
            problems.append(f"{column}: not a valid {kind} in {n(bad.sum(), 'row')} ({first(raw, bad, column)})")
    rule("every value has its column's type", problems)

    problems = []
    for column, (_, required) in COLUMNS.items():
        if required and raw[column].isna().any():
            problems.append(f"{column}: missing in {n(raw[column].isna().sum(), 'row')}")
    rule("no missing values in required columns", problems)

    repeated = df["ticket_id"].duplicated(keep=False)
    rule("ticket_id is unique",
         [f"{n(df.loc[repeated, 'ticket_id'].nunique(), 'ticket ID')} repeated "
          f"(first: {df.loc[repeated, 'ticket_id'].iloc[0]})"] if repeated.any() else [])

    problems = []
    for column, allowed in ALLOWED.items():
        bad = df[column].notna() & ~df[column].isin(allowed)
        if bad.any():
            values = sorted(df.loc[bad, column].unique())
            problems.append(f"{column}: values not allowed in {n(bad.sum(), 'row')}: {values}")
    rule("categories use allowed values only", problems)

    problems = []
    for column, (low, high) in RANGES.items():
        values = df[column]
        bad = pd.Series(False, index=df.index)
        if low is not None:
            bad |= values < low
        if high is not None:
            bad |= values > high
        if bad.any():
            problems.append(f"{column}: outside {low} to {high} in {n(bad.sum(), 'row')} ({first(df, bad, column)})")
    rule("numbers and dates are in their allowed ranges", problems)

    bad = ~df["escalated_72h"].isin([0, 1])
    rule("the target escalated_72h is 0 or 1",
         [f"not 0 or 1 in {n(bad.sum(), 'row')} ({first(df, bad, 'escalated_72h')})"] if bad.any() else [])
    return df

Each block is one rule, and each rule finds the bad rows first, as the SQL rules did. Two details:

• duplicated(keep=False) marks every copy of a repeated ID, not only the second one.
• The allowed-values rule skips missing values (notna()). A missing value is the job of the required-values rule. So each problem is reported once, by the rule that explains it best.

Part 4: rules for the splits, and the result

scripts/validate.py (part 4 of 4)

def rate_problem(name, target):
    """The escalation rate must be in the expected range, or something upstream changed."""
    low, high = ESCALATION_RATE
    rate = pd.to_numeric(target).mean()
    if not low <= rate <= high:
        return [f"{name}: escalation rate {rate:.1%} is outside {low:.0%} to {high:.0%}"]
    return []


def check_splits(table, parts):
    """Rules for train, valid and test."""
    empty = [name for name, part in parts.items() if len(part) == 0]
    rule("every split has rows", [f"{name} has no rows" for name in empty])
    if empty:
        return

    problems = []
    for name, part in parts.items():
        if list(part.columns) != list(table.columns):
            problems.append(f"{name}: the columns are not the same as in tickets_model.csv")
    together = pd.concat(parts.values())
    if len(together) != len(table):
        problems.append(f"the splits have {len(together)} rows together; tickets_model.csv has {len(table)}")
    if set(together["ticket_id"]) != set(table["ticket_id"]):
        problems.append("the splits do not contain the same tickets as tickets_model.csv")
    rule("together, the splits are exactly the modelling table", problems)

    problems = []
    for i, a in enumerate(SPLITS):
        for b in SPLITS[i + 1:]:
            shared = set(parts[a]["ticket_id"]) & set(parts[b]["ticket_id"])
            if shared:
                problems.append(f"{a} and {b} share {n(len(shared), 'ticket ID')} (first: {sorted(shared)[0]})")
    rule("no ticket is in two splits", problems)

    problems = []
    for a, b in zip(SPLITS, SPLITS[1:]):
        last = pd.to_datetime(parts[a]["created_at"], format=TIME_FORMAT).max()
        start = pd.to_datetime(parts[b]["created_at"], format=TIME_FORMAT).min()
        if not last < start:
            problems.append(f"{a} ends at {last}, which is not before the start of {b} at {start}")
    rule("the splits are in time order: train, then valid, then test", problems)

    problems = rate_problem("all rows", table["escalated_72h"])
    for name, part in parts.items():
        problems += rate_problem(name, part["escalated_72h"])
    rule(f"escalation rate between {ESCALATION_RATE[0]:.0%} and {ESCALATION_RATE[1]:.0%}", problems)


def main():
    files = ["tickets_model.csv"] + [f"{name}.csv" for name in SPLITS]
    missing = [f for f in files if not (MODEL / f).exists()]
    if missing:
        print(f"FAIL  missing file(s) in {MODEL}: {', '.join(missing)}")
        print("Run build_features.py and split.py first.")
        sys.exit(1)

    print(f"Checking {MODEL} against the contract")
    table = check_table(load(MODEL / "tickets_model.csv"))
    if table is not None:
        check_splits(table, {name: load(MODEL / f"{name}.csv") for name in SPLITS})

    failed = [r for r in results if not r["passed"]]
    REPORT.parent.mkdir(exist_ok=True)
    REPORT.write_text(json.dumps({"files": files, "passed": not failed, "rules": results}, indent=2) + "\n",
                      encoding="utf-8")
    if failed:
        print(f"{len(failed)} of {len(results)} rules failed. Fix the step that made the data, then run it again.")
        sys.exit(1)
    print(f"All {len(results)} rules passed.")


if __name__ == "__main__":
    main()

The split rules protect the Module 5 decisions. A ticket in two splits is a leak: the model sees a test ticket during training. Splits out of time order mix the future into training. The rate rule runs on every split, because one broken split can hide in a good total.

sys.exit(1) ends the script with exit code 1. Every program returns an exit code to the terminal when it ends: 0 means success, any other number means failure. Other programs read the code, not the text. In the last lesson of this module, run_pipeline.py uses it to stop.

Mei runs it on the files from Module 5:

python scripts/validate.py

Checking data/model against the contract
PASS  required columns are present
PASS  no column that is known only after creation
PASS  every value has its column's type
PASS  no missing values in required columns
PASS  ticket_id is unique
PASS  categories use allowed values only
PASS  numbers and dates are in their allowed ranges
PASS  the target escalated_72h is 0 or 1
PASS  every split has rows
PASS  together, the splits are exactly the modelling table
PASS  no ticket is in two splits
PASS  the splits are in time order: train, then valid, then test
PASS  escalation rate between 5% and 30%
All 13 rules passed.

The script also saves the same results in reports/validation.json. That file is the quality report that Priya receives with the data.

What the distribution rule catches

All 13 rules pass, so do the rules work? A rule that never fails proves nothing. Mei tests the distribution rule with a real bug. In build_features.py, she types minutes instead of hours:

scripts/build_features.py (a bug, for the test only)

WINDOW = pd.Timedelta(minutes=72)

build_features.py runs with no error. Then she runs split.py and validate.py. Two rules fail:

python scripts/validate.py (with the bug)

FAIL  numbers and dates are in their allowed ranges
      created_at: outside 2026-01-01 00:00:00 to 2026-06-28 06:00:00 in 29 rows (first: T-11374, 2026-06-28 07:14:18)
...
FAIL  escalation rate between 5% and 30%
      all rows: escalation rate 0.8% is outside 5% to 30%
      train: escalation rate 0.5% is outside 5% to 30%
      valid: escalation rate 1.8% is outside 5% to 30%
      test: escalation rate 0.8% is outside 5% to 30%
2 of 13 rules failed. Fix the step that made the data, then run it again.

Every row is still valid: each target is 0 or 1. But only 11 of 1,398 tickets are now escalated, because few tickets reach a senior agent within 72 minutes. The bug also moved the cut-off for a final target to 72 minutes before the snapshot. So the 29 tickets that are too recent came back. Without the rules, Priya trains a model on the wrong target, and its scores look normal. Mei changes the line back to hours=72.

Guided practice: run the contract, then break it

1. Make scripts/validate.py with the four parts above, in order, in one file.

2. In the terminal, go to your project folder and check that the environment is active.

3. Run the script:

   python scripts/validate.py

   You should see 13 PASS lines and All 13 rules passed.

4. Read the exit code. It is 0.

   macOS

   Terminal

   echo $?

   Windows

   PowerShell

   $LASTEXITCODE

5. Break one thing on purpose. This command adds the last train row to the end of test.csv:

   python -c "import pandas as pd; train = pd.read_csv('data/model/train.csv'); train.tail(1).to_csv('data/model/test.csv', mode='a', header=False, index=False)"

6. Predict which rules fail. Then run python scripts/validate.py again. You should see:

   FAIL  together, the splits are exactly the modelling table
         the splits have 1370 rows together; tickets_model.csv has 1369
   FAIL  no ticket is in two splits
         train and test share 1 ticket ID (first: T-10929)
   FAIL  the splits are in time order: train, then valid, then test
         valid ends at 2026-05-31 20:57:12, which is not before the start of test at 2026-04-30 23:54:47
   PASS  escalation rate between 5% and 30%
   3 of 13 rules failed. Fix the step that made the data, then run it again.

7. Read the exit code again, as in step 4. It is 1.

8. Repair the files. Run the step that made them: python scripts/split.py.

9. Run python scripts/validate.py again. All 13 rules pass.

One wrong row broke three rules, and each message points to it from a different side. The rate rule still passes: one row does not change a rate much. That is why a contract needs row rules and distribution rules.

Your turn: a rule for the size of the evaluation sets

Priya adds a request: “With fewer than 30 escalated tickets, a score on valid or test changes a lot from one ticket to the next. I want at least 30 escalated tickets in each of them.”

1. Add a constant MIN_ESCALATED = 30 to the contract part.
2. Add a rule at the end of check_splits() that counts the escalated tickets in valid and in test.
3. Predict: does the rule pass? Then run the script.
4. Change the value to 40, and predict again.

Show the answer

scripts/validate.py (in the contract part)

MIN_ESCALATED = 30

scripts/validate.py (at the end of check_splits)

    problems = []
    for name in ["valid", "test"]:
        escalated = int(pd.to_numeric(parts[name]["escalated_72h"]).sum())
        if escalated < MIN_ESCALATED:
            problems.append(f"{name}: {escalated} escalated tickets; at least {MIN_ESCALATED} are needed")
    rule(f"valid and test have at least {MIN_ESCALATED} escalated tickets each", problems)

With 30, the rule passes: valid has 45 escalated tickets, test has 35. The last line is All 14 rules passed. With 40, it fails:

FAIL  valid and test have at least 40 escalated tickets each
      test: 35 escalated tickets; at least 40 are needed
1 of 14 rules failed. Fix the step that made the data, then run it again.

The data is not wrong here: the test set is small. The fix is a decision for Priya and Mei, for example a longer test period. A rule can fail because the data does not meet a need. That is useful too: it shows the problem before anyone trusts a score.

Common mistake

A rule that cannot fail

Symptom: the rule for allowed channels passes on the raw export, where you know that Email and webform exist.

Cause: the allowed values come from the data that the rule checks. For example:

the mistake

import pandas as pd

df = pd.read_csv("data/extract/tickets.csv", dtype=str, keep_default_na=False, na_values=[""])
allowed = set(df["channel"].dropna())  # the mistake: the rule comes from the data
print(sorted(allowed))
print("rows that break the rule:", (~df["channel"].dropna().isin(allowed)).sum())

Output

[' chat', 'EMAIL', 'Email', 'chat', 'email', 'phone', 'web_form', 'webform']
rows that break the rule: 0

Every value in the data is “allowed”, because the list is made from the data. The rule can never fail.

Fix:

1. Write the allowed values by hand, from the agreement with the people who know the data. Grace knows the four channels.
2. Test each new rule once on data that must break it, as in the guided practice. If it does not fail, it does not work.

the fix

contract = {"email", "chat", "phone", "web_form"}  # the fix: the rule comes from the agreement
print("rows that break the rule:", (~df["channel"].dropna().isin(contract)).sum())

Output

rows that break the rule: 40

In AI work: a check that always passes is worse than no check, because people trust it. A new category in production, for example a channel social, then reaches the model with nobody noticing.

Check your understanding

Why does a rule return the rows that break it, and not only true or false?

Show the answer

Because the rows tell you what to fix. “False” says that something is wrong. “36 rows, the first is T-10046 with −1” says what is wrong and where to look. The count also shows the size of the problem. One row is often a typing mistake. 25 rows are often a system problem, such as the cents from the old web form.

Check your understanding

Predict: Tomás adds priority_now back to tickets_model.csv, because 'it works so well'. Which rule fails, and what does the message say?

Show the answer

The rule “no column that is known only after creation” fails with priority_now must not be in the dataset. The column is in NEVER. If the splits do not have the column too, a second rule fails: “together, the splits are exactly the modelling table”. The columns are not the same. The contract turns the Module 5 decision into a check. Nobody has to remember it.

Check your understanding

Train has an escalation rate of 11.7%, valid 19.8%. Omar suggests a rule: every split must be within 3 percentage points of the overall rate of 13.7%. What happens, and is it a good rule?

Show the answer

Valid fails: 19.8% is 6.1 points above 13.7%. But the data is correct. The rate rose after 2026-05-01 because of the new warranty policy, and a time split shows that change on purpose. A distribution rule must describe what is normal, from people who know the data, such as Grace. A rule that fails on correct data teaches people to ignore failures. The 5% to 30% range catches bugs and allows real changes.

See it end to endExplain an online shop's salesCase study

The online-shop case study delivers a monthly table that passes six validation rules.

See it end to endPrepare a leak-free table for city repair requestsCase study

The repair-requests case study checks its delivery against a contract of 13 rules, with a test that proves each rule can fail.

Recap

• A schema lists the columns and their types. A data contract adds rules about the values, and it is an agreement between the maker and the users of a dataset.
• Four kinds of rule: required values, allowed values and ranges, unique keys, and expected distributions.
• Write a rule as a check that returns the rows that break it. It passes with 0 rows.
• In SQL, NOT IN drops NULL rows. Say what a rule does with missing values.
• validate.py reads every value as text, checks 13 rules, writes reports/validation.json and exits with code 1 on failure.
• Test each rule on data that must break it. A rule that cannot fail is worse than no rule.

Next, you record where the data came from and what happened to it on the way, so that Priya can trust it and repeat it.

Next: Track lineage