Clean, split and check for leakage

What you will be able to do

Remove the records that would teach the model something wrong or flatter its score. Then split the rest fairly, and count which kinds of ticket the training data never shows.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• Training examples and the gap in the labels, from Example structure and permissions.
• The jobs of the three sets, from Train, validation and test sets in Machine Learning: From Problem to Reliable Model.
• The exact-copy check and its limit, from Avoid contamination.

What you need on your computer

• Your project from the last lesson: python -m pytest prints 12 passed.

Tested with Python 3.12 on macOS. The Windows (PowerShell) commands are written but not tested.

Before the folder goes to the new employee

Before the folder of solved tickets goes to the new employee, a senior colleague reads it. She takes out sheets with a wrong answer or a customer’s password. She removes photocopies, because ten copies teach one sheet ten times, and any sheet that is almost a test question. Last, she counts: is there an example of every kind of ticket? This lesson turns these steps into one command, prepare, which records every removal and its reason.

The problem: four kinds of bad record

Each kind does its damage silently: nothing crashes, and the training loss goes down.

• A wrong label teaches the wrong answer. The deep learning course made 70 team labels wrong on purpose.
• A secret, such as a password or a card number, can come back word for word in a model’s reply.
• A duplicate counts one ticket twice. A copy in training and another in validation makes the validation score too good.
• An evaluation leak is a training record that is the same as, or nearly the same as, an evaluation case. The model learns the answer to a test question.

The general name for the last problem is data leakage: test information reaches the training data.

The idea: fixed rules, each rejection written down

prepare checks every record against four rules, in this order: known wrong label, secret, duplicate or near copy, evaluation leak. The first rule that a record breaks rejects it, and the record goes to data/prepared/rejected.jsonl with its reason.

Exact copies are easy to find; near copies need a measure of similarity. The project uses word 3-gram overlap. Cut each text into every run of three words. Then divide the runs that both texts share by all the different runs. Identical texts score 1. Here is the one real leak in the data, in lowercase and without punctuation:

Word 3-gram overlap on the one real leak: 5 shared triples out of 8 different ones.

Text description of the diagram

The training ticket T-64315 says “Please delete my account and all my data??”. The holdout case T-64954 says “Just delete my account and all my data already”. The first text has six word triples and the second has seven. Five triples are in both: “delete my account”, “my account and”, “account and all”, “and all my” and “all my data”. One is only in the training ticket (“please delete my”) and two are only in the holdout case (“just delete my” and “my data already”). So there are 8 different triples, and the overlap is 5 ÷ 8 = 0.62. The project’s limit is 0.40, so the training ticket is rejected.

The limit, 0.40, has a cost. Short tickets reach it easily: “Hello, locked out after too many attempts. Thank you” and “Help! Locked out after too many attempts re parasol” score exactly 0.40, so the first one goes. That loses one good record of 4,696. A missed leak costs more: a test that you can no longer trust.

The first secret rule was wrong. It matched “password is” followed by any word, and flagged 12 training tickets. Not one held a password; they said things like “my password is wrong”. The rule now needs a digit or a symbol in that word, and it flags none of them. Test a cleaning rule on real records before you trust it.

Related records: check what a shared ID means

Records that belong together, such as two messages from one customer, must stay on the same side of a split. Otherwise the test asks about the other half of a story that training showed. The usual fix is to group records by a shared ID and split whole groups.

Priya looks for a shared ID. One order number, LK-482913, is in 111 training tickets, and 201 training tickets share an order number with an evaluation case. Large-scale leakage? She reads the tickets: different products, teams and situations. The program that wrote them reused a few numbers. So check what a shared ID means before you group by it. Here each ticket is its own group, and the near-copy check catches a ticket written twice. In real data, group by customer or conversation.

Sample and split

From the 4,602 clean records, prepare takes 200 per team and always keeps the rare records that need a person. Then 10% of each group (team, needs a person or not) goes to validation, so both parts have every group. A fixed seed gives the same files every time. The validation set watches the training run in Module 4; dev and holdout cases stay for the decisions.

Guided practice: prepare the training data

1. Copy this lesson’s files from the snapshot end-of-m02:

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m02
   mkdir -p data/exercise
   cp "$S/adapt/prepare.py" adapt/
   cp "$S"/data/exercise/* data/exercise/
   cp "$S/tests/test_prepare.py" tests/
   cp "$S/README.md" .

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m02"
   New-Item -ItemType Directory -Force data\exercise | Out-Null
   Copy-Item "$S\adapt\prepare.py" adapt\
   Copy-Item "$S\data\exercise\*" data\exercise\
   Copy-Item "$S\tests\test_prepare.py" tests\
   Copy-Item "$S\README.md" .

2. Prepare the data:

   python -m adapt prepare

   4696 source records
     rejected  23  duplicate
                  T-60709: same words as T-60281
                  T-61069: near copy of T-60555 (overlap 0.40)
                  T-61419: near copy of T-60483 (overlap 0.50)
     rejected   1  evaluation leak
                  T-64315: near copy (overlap 0.62) of evaluation case T-64954 (holdout)
     rejected  70  known wrong label
                  T-60124: team payment is wrong on purpose
                  T-60158: team payment is wrong on purpose
                  T-60236: team returns is wrong on purpose
   4602 clean records
   train  900 records  (account 180, delivery 180, payment 180, returns 180, warranty 180; needs a person 78)
   val    100 records  (account 20, delivery 20, payment 20, returns 20, warranty 20; needs a person 9)
   Not covered by any training record: French tickets, and needs-a-person rules H2 to H5
   Wrote data/prepared/train.jsonl, val.jsonl, rejected.jsonl and manifest.json

   The output shows three examples per reason; all 94 are in rejected.jsonl. No secret was found. Two of the 89 records that need a person had a wrong team, so 87 remain: 78 and 9.

3. Run the tests. They take about half a minute, because some of them run prepare again:

   python -m pytest

   The last line says 17 passed. The tests check that training, validation and evaluation share no ticket, and that prepare gives the same training file every time: fingerprint 73f29c94b022.

The training-data pipeline with the real counts. The strip at the bottom is the coverage check: five zeros before any training.

Text description of the diagram

In the top row, 4,696 source tickets pass four checks in order. The known-wrong-label check removes 70, the secret check none, the duplicate check 23 and the evaluation-leak check 1. All 94 go to rejected.jsonl, each with its reason. The second row runs from right to left. 4,602 clean records remain. Taking 200 per team gives 1,000, and every record that needs a person is kept. The split gives 900 training records (78 need a person) and 100 validation records (9 need a person). The strip at the bottom counts what the 900 training records show. Rule H1, safety, has 78. H2 legal, H3 outside the policy, H4 someone else’s order, H5 no readable request and French have zero.

Coverage: count what the data never shows

Coverage checks that every kind of case that the model will meet has training examples. The last output line is this check; the chart shows it per 100 records.

• Training records (900)
• Dev cases (102)

FR = French tickets. Rule H1 (safety) appears about as often in training as in the dev cases. Rules H2 to H5 and French tickets appear in the dev cases and never in training.

Text description of the chart

A column chart with pairs of columns for H1, H2, H3, H4, H5 and French (FR). The values are records per 100. H1: 8.7 in the training records and 9.8 in the dev cases. H2: 0 and 2.0. H3: 0 and 1.0. H4: 0 and 6.9. H5: 0 and 2.9. French: 0 and 9.8. Only H1 has a training column above zero.

Rule H1 is covered at about the dev rate. The other four rules and French have no record, yet they are 23 of the 102 dev cases. The teams have a gap too: the 78 records that need a person are all warranty (60) or delivery (18). A model learns only the patterns that its examples show.

Common mistake

The fingerprint of the training file changed

Symptom: a later training run gives other numbers than the course, and data/prepared/manifest.json shows another training fingerprint.

Cause: prepare --extra rewrote data/prepared/. Even one kept record changes the sample.

Diagnose: if settings.extra in manifest.json names a file, the data is not the course’s.

Fix: run python -m adapt prepare with no options: the fingerprint is 73f29c94b022 again.

Discuss: train now, or write the missing records first?

Priya shows the coverage chart to the team.

• Tomás: “Train now. The run takes minutes on a laptop, and with 78 safety examples the model will learn when a ticket needs a person.”
• Grace: “A legal threat is not a safety worry. Nothing in this data says that it needs a person.”
• Kwame: “Good records for four rules and French take Grace days. The run costs almost nothing.”
• Omar: “If we ship this, what happens to a customer who threatens legal action?”

Before you read on, answer these:

1. Predict: what will an adapter trained on this data answer for the 13 dev cases under rules H2 to H5?
2. Is the run still worth doing? What can it measure?
3. What would you write in the go/no-go memo today, before the run?

The course’s answer. Train, but as a measurement, not as a candidate to ship, and write the prediction down first. The coverage counts are the evidence: 0 of 900 training records show H2 to H5 or French. So the team predicts that the adapter will not send those tickets to a person. targets.toml asks that every ticket that needs a person reaches one, so the team expects this adapter to fail that target. The run is still worth its small cost: it shows what this data can teach, such as the format and the routing. Regressions and operational value checks the prediction.

What would change the answer: an expensive run, such as hours on a rented GPU; then write the missing records first. Or records for H2 to H5 and French, written and checked by Grace. The practice task shows why one record per rule is not enough.

Module practice

Guided: a colleague’s four records

data/exercise/extra_records.jsonl holds four made-up records that a colleague wants to add. Read them, and decide which three must not be used, and why. Then run:

python -m adapt prepare --extra data/exercise/extra_records.jsonl

4700 source records
  rejected  24  duplicate
               X-003: same words as T-60001
               T-60709: same words as T-60281
               T-61069: near copy of T-60555 (overlap 0.40)
  rejected   2  evaluation leak
               X-001: near copy (overlap 0.91) of evaluation case T-64710 (holdout)
               T-64315: near copy (overlap 0.62) of evaluation case T-64954 (holdout)
  rejected  70  known wrong label
               T-60124: team payment is wrong on purpose
               T-60158: team payment is wrong on purpose
               T-60236: team returns is wrong on purpose
  rejected   1  secret
               X-002: password
4603 clean records
train  900 records  (account 180, delivery 180, payment 180, returns 180, warranty 180; needs a person 78)
val    100 records  (account 20, delivery 20, payment 20, returns 20, warranty 20; needs a person 9)
Not covered by any training record: French tickets, and needs-a-person rules H2 to H5
Wrote data/prepared/train.jsonl, val.jsonl, rejected.jsonl and manifest.json

X-001 is a holdout case with one digit changed (overlap 0.91). X-002 has a password. X-003 differs from T-60001 only in punctuation. X-004 is kept. It is not among the 900, but it changed them: 101 of the 180 delivery records differ, and the fingerprint is 4968527efbc5. Run python -m adapt prepare to restore 73f29c94b022.

Your turn: a small synthetic dataset with rejection rules

Write 5 to 8 made-up records in data/exercise/my_records.jsonl, one JSON object per line with id, text, team, needs_human, rule, language and source. Include two for rules H2 to H5 or in French, and two that your rules must reject. Write your rejection rules and expected results first. Then run prepare --extra with your file, compare, and run prepare again.

Show a worked solution

The author’s file has six records, all made up:

ID	Text, shortened	Labels	Expected
Y-001	“…I have written to my lawyer, and I will take you to the small claims court…”	returns, true, H2	kept
Y-002	“The same patio heater costs 40 dollars less at another shop. Will you match that price…”	payment, true, H3	kept
Y-003	“My neighbour is away this week. Can you tell me where her parcel is?…”	delivery, true, H4	kept
Y-004	“Bonjour, mon parasol est arrivé hier, mais la boîte était vide…”	delivery, false, French	kept
Y-005	“Please change the email on my account to sam.lee@example.org…”	account, false	rejected: secret (email address)
Y-006	dev case T-81026 with one digit of the order number changed	payment, true, H2	rejected: evaluation leak

The rejection rules are three. No secret (card number, password, email address) in the text. No near copy of a training record. No near copy of any evaluation case. The output, shortened:

4702 source records
  rejected   2  evaluation leak
               Y-006: near copy (overlap 0.79) of evaluation case T-81026 (dev)
  rejected   1  secret
               Y-005: email address
4606 clean records
train  900 records  (account 180, delivery 180, payment 180, returns 180, warranty 180; needs a person 81)
Not covered by any training record: French tickets, and needs-a-person rule H5

Both rejections were expected. Y-001 to Y-003 reached training, so “needs a person” rose from 78 to 81 and only H5 is still missing. One result was a surprise: Y-004 did not reach training. The sample keeps every record that needs a person. A French ticket that needs none competes with 1,244 other delivery records for 179 places. So “French tickets” is still in the last line. Coverage needs enough records, and a sampling rule that keeps the rare kinds. Three records against 78 for rule H1 is still far too few to teach three rules.

Module check: remove a contamination

Copy a dev or holdout case into your extra file and change one word. Show that prepare --extra rejects it, and how you know that the prepared data holds no evaluation case. Then restore the course’s data.

Show the answer

The solution above did this with Y-006, a copy of dev case T-81026 with one digit changed. prepare printed Y-006: near copy (overlap 0.79) of evaluation case T-81026 (dev) under evaluation leak. The record is in rejected.jsonl and in neither train.jsonl nor val.jsonl. The test test_validation_and_training_share_no_ticket_and_no_evaluation_case checks the same thing on every run. Last, python -m adapt prepare with no options restores the course’s data, with training fingerprint 73f29c94b022.

Reset: python -m adapt prepare writes the course’s prepared data again. If your project is broken, copy the snapshot end-of-m02.

See it end to endShould this team fine-tune?Case study

The case study applies the same near-copy rule (word 3-gram overlap of 0.4 or more) to a public dataset. It removes 45 training queries that differ from a dev or holdout query in one place name, such as “the time zone of france” and “the time zone of china”.

Knowledge check

Check your understanding

Sam suggests raising the near-copy limit from 0.40 to 0.90, so that fewer records are lost. What would change in this data, and is it a good trade?

Show the answer

The one real leak, T-64315 (overlap 0.62 with holdout case T-64954), would pass into training. So would all 22 near duplicates, with overlaps from 0.40 to 0.80. The holdout score would then partly measure memory. The gain is about 23 records out of 4,602, which changes almost nothing for training. Losing a fair test costs far more, so it is a bad trade.

Check your understanding

The order number LK-482913 is in 111 training tickets. Should you put all 111 in the same split? What do you check first?

Show the answer

Check what the shared ID means. Read some of the tickets: they are about different products, teams and situations, because the program that wrote them reused a few order numbers. They are not related, so grouping them would be wrong. Group records only by an ID that really joins them, such as one customer or one conversation.

Check your understanding

The team decides to train although the coverage strip shows five zeros. Which evidence, before or after the run, would make you change that decision?

Show the answer

Before the run: a high cost per run, such as rented GPU hours, would make it better to write the missing records first. After the run, the dev cases under rules H2 to H5 decide. If the adapter sent some of them to a person, the prediction was wrong, and it would need a closer look. If it sent none, the prediction was right, and the next step is better data, not more training. In both cases, the result counts only on cases that the model never saw.

Check your understanding

Practical check: after prepare --extra with a near copy of an evaluation case, which lines of the output show that the copy did not reach training, and how do you restore the course's data?

Show the answer

The evaluation leak block names the record and the case, for example Y-006: near copy (overlap 0.79) of evaluation case T-81026 (dev). The record is in data/prepared/rejected.jsonl, and python -m pytest still passes the test that training, validation and evaluation share no ticket. python -m adapt prepare with no options restores the data: the training fingerprint in manifest.json is 73f29c94b022.

Recap

• prepare applies four rules in order and writes every rejection with its reason: 94 in all.
• Data leakage: test information reaches the training data. Word 3-gram overlap finds near copies (5 of 8 shared = 0.62, above 0.40).
• Test cleaning rules on real records. Check what a shared ID means before you group by it.
• Coverage: 78 safety records, and zero for rules H2 to H5 and French. The team trains to measure, and writes its prediction first.
• Your project matches end-of-m02: 17 passed, training fingerprint 73f29c94b022.
• These counts describe this made-up data. Your own data will give other counts; the checks stay the same.

Next: Full and parameter-efficient tuning