Example structure and permissions

What you will be able to do

Turn tickets into training examples with one shape, and write down where each record comes from and what you may use it for. A model copies every habit of its examples.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The narrow task, a ticket in and {"team", "needs_human"} out as exact JSON, and the strict answer check, from Clarify the failure.
• The targets that the team wrote before any training, from Define success and constraints.
• Grace’s rules H1 to H5, and why dev and holdout cases stay out of tuning, from Avoid contamination in Evaluating and Testing AI Systems.
• The roles of the messages in a chat request (system, user, assistant), from Request anatomy in Building Reliable Applications with LLM APIs.

What you need on your computer

• Your project ~/projects/ticket-adapter at the end of Module 1, with its .venv active: python -m pytest prints 9 passed. If it does not, copy the snapshot end-of-m01, as the snapshots page explains.
• The labs repository in ~/projects/nextia-learning-labs, from the same page.

No model download and no PyTorch yet. Tested with Python 3.12 on macOS. The Windows (PowerShell) commands are written but not tested.

A folder of solved tickets

A new help-desk employee learns from a folder of solved tickets: a customer’s message and the team’s answer on each sheet. She copies what she sees. If half of the sheets name the team “Payment” and half “billing”, she uses both.

Some sheets also came from her final test, and some from another company’s binder that Larkfield may not copy. Nobody wrote down which. Now nobody can say if her test score means anything, or if the folder may be used at all. A model learns in the same way, only more literally. This lesson fixes both problems: one shape for every example, and a record of where each one comes from.

The problem: the model learns the mess too

The narrow task has one correct answer format, and Clarify the failure built a strict check for it. Here are three ways that a person could write the same answer, and what the check says about each. Start python in the project folder and paste the code:

from adapt.answer import format_answer, parse_answer

replies = [
    '{"team": "payment", "needs_human": false}',
    '{"needs_human": "no", "team": "Payment"}',
    "Team: payment. No person needed.",
]
for reply in replies:
    answer, problem = parse_answer(reply)
    print(f"{reply:42s} -> {answer or problem}")
print(format_answer("payment", False))

{"team": "payment", "needs_human": false}  -> {'team': 'payment', 'needs_human': False}
{"needs_human": "no", "team": "Payment"}   -> unknown team
Team: payment. No person needed.           -> not JSON
{"team": "payment", "needs_human": false}

All three mean the same thing to a person, but only the first one passes. If the training examples mix the three, the model learns that all three are acceptable and writes each of them some of the time. The model did exactly what the data showed.

The idea: one shape, written by one function

A training example is one input together with the output that the model must learn to write: its target. For a chat model, the example is a short conversation in the format that the model was made for. The system message gives the instruction, the user message is the ticket, and the assistant message is the answer.

Three conventions make sure that only the ticket and its labels change between examples:

1. The system message is the same short instruction in every record. The base model is tested with it too.
2. One function, format_answer, writes every answer. Its output is the last line above: the same key order, lowercase team names, true or false, and nothing else.
3. The details for people stay beside the messages: the source, how the label was made, the writing style.

One training example. Only the ticket and its answer change from record to record; the provenance fields stay beside the messages.

Text description of the diagram

The diagram shows source ticket T-60001 as a training example. The first box is the system message: the short instruction, the same in every record. The second box is the user message: the ticket, which is the input and differs in every record. The third box, in green, is the assistant message: the answer {"team": "payment", "needs_human": false}, which is the target that the model learns to write. One function writes it, so it has one format. A dashed box at the bottom holds the fields for people, which the model never sees. They are the source (tickets), how the label was made (not read: no safety word) and the style (plain).

Where each record comes from

Provenance answers three questions in writing. Where does this data come from? Who made its labels, and how? What may we use it for? The last answer is the permitted use. In a company, it comes from contracts, licences, privacy law and what customers agreed to. A data protection person decides it, not the engineer who wants to train.

The project keeps provenance in data/sources.toml: per source, its origin, licence, permitted use and SHA-256 fingerprint. A fingerprint is a long number computed from the file’s bytes; one changed byte changes it. The tickets were made up for the courses by a language model; no real customer wrote them. So the licence is simple (CC0, free for any use).

The needs_human labels need the most care, because the course made them. The course author read only the 98 training tickets with a word from the evaluation course’s safety word list, and applied rule H1, safety. Of these, 89 need a person and 9 do not; one of the 9 only says that “the box with the safety net is missing”. The other 4,598 tickets were not read. They are false because they have no safety word: an assumption, not a check.

How the needs_human labels were made. Every label records which path it took, in the column label_how.

Text description of the diagram

All 4,696 training tickets go through one question. Does the text contain a word from the safety word list, such as fire, smoke or safety? For 98 tickets the answer is yes. The course author read each of them with rule H1: 89 need a person and 9 do not. For 4,598 tickets the answer is no. Nobody read them, and they are labelled false. That is an assumption, shown in orange. A dashed box below lists what this data never labels. These are legal threats (H2), requests outside the policy (H3), someone else’s order (H4), no readable request (H5), and French tickets.

The column label_how keeps this difference visible in every record, so a reviewer sees which labels a person checked. The diagram also shows a gap: no training ticket is labelled under rules H2 to H5, and none is in French.

Guided practice: add the sources to your project

1. In ~/projects/ticket-adapter, with .venv active, copy this lesson’s files from the snapshot end-of-m02. The new __main__.py adds sources, record and prepare (for the next lesson).

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m02
   mkdir -p data/source
   cp "$S/adapt/records.py" "$S/adapt/__main__.py" adapt/
   cp "$S"/data/source/* data/source/
   cp "$S/data/sources.toml" "$S/data/SHA256SUMS" data/
   cp "$S/tests/test_records.py" tests/

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m02"
   New-Item -ItemType Directory -Force data\source | Out-Null
   Copy-Item "$S\adapt\records.py", "$S\adapt\__main__.py" adapt\
   Copy-Item "$S\data\source\*" data\source\
   Copy-Item "$S\data\sources.toml", "$S\data\SHA256SUMS" data\
   Copy-Item "$S\tests\test_records.py" tests\

2. Run the tests:

   python -m pytest

   The last line says 12 passed. The three new tests check the chat format, the fingerprints and the 89 labels.

3. Read the provenance of every source:

   python -m adapt sources

   tickets: data/source/tickets.csv
     4,696 ticket messages with a team: the TRAINING split of the deep learning course
     licence CC0 1.0; permitted use: training, validation and teaching
   needs_human: data/source/needs_human.csv
     needs_human for each of the 4,696 tickets
     licence CC0 1.0; permitted use: training, validation and teaching
   known_wrong_labels: data/source/known_wrong_labels.txt
     70 ticket IDs whose team is wrong on purpose (1.5% label noise, put in by the deep learning course)
     licence CC0 1.0; permitted use: removing wrong labels
   evaluation: data/eval/cases.jsonl
     271 labelled cases of the evaluation course: dev 102, holdout 98, contaminated 71
     licence CC0 1.0; permitted use: EVALUATION ONLY: never training. dev for choices, holdout for the final decision.

   Three permitted uses: the 70 wrong labels may only remove records, and the evaluation cases may never be training data.

4. Show one safety ticket as a training example:

   python -m adapt record T-60003

   {
    "id": "T-60003",
    "messages": [
     {
      "role": "system",
      "content": "You sort support tickets for Larkfield, a home and garden shop. Reply with JSON only, in this form: {\"team\": \"delivery\" or \"returns\" or \"payment\" or \"warranty\" or \"account\", \"needs_human\": true or false}"
     },
     {
      "role": "user",
      "content": "Hello, I hope I'm writing to the right place. Last weekend I was cleaning the gutters at the back of the house using the step ladder I bought from you a few months ago. Everything was fine until I climbed to the third rung and the left leg made a loud cracking noise and slid outwards on the patio. I managed to grab the gutter and stay upright, but it was quite a scare and my wife was very upset when she saw me. Afterwards I looked at the ladder and found a hairline crack in the metal near the hinge on that same leg, and the rubber foot is also loose. I haven't used it since and it's standing in the garage now. Could you please advise how I should report this safety problem, and whether the ladder should be inspected or replaced? Thank you for your help."
     },
     {
      "role": "assistant",
      "content": "{\"team\": \"warranty\", \"needs_human\": true}"
     }
    ],
    "team": "warranty",
    "needs_human": true,
    "style": "long",
    "source": "tickets",
    "label_how": "read: a safety concern or an injury (H1)"
   }

   Compare it with python -m adapt record T-60001, the short payment ticket in the diagram. Its label_how is not read: no safety word.

5. Check that the data files are the recorded ones. On macOS and Linux, go into data/ and compare every fingerprint:

   cd data && shasum -a 256 -c SHA256SUMS; cd ..

   Each of the five lines ends with OK. On Windows, compare Get-FileHash data\source\needs_human.csv with the sha256 line in data/sources.toml. On every system, a test does the same check.

Common mistake

A label fixed by hand

Symptom: after a colleague “corrects” one label, python -m pytest prints 3 failed, with assert 90 == 89. shasum -a 256 -c SHA256SUMS prints source/needs_human.csv: FAILED.

Cause: in the author’s test, T-60001 changed to true,H1 but kept not read: no safety word. The file no longer matches its fingerprint.

Diagnose: the failed fingerprint names the file. Compare it with the snapshot’s file to find the changed line.

Fix: if the change is wrong, copy the snapshot’s file again. If it is right, record a new version of the data. Write how the label was made in the how column. Then update the fingerprint in data/sources.toml and data/SHA256SUMS.

When to do this, and what it costs

Do both before every training run, even a small one. A format function and a provenance file cost about an hour. Without them, you cannot explain a bad result or answer the first question of any review: “May we use this data?”

Real data needs more. Real tickets contain names, addresses and order histories, so training on them needs a legal basis and a decision about personal data. A public dataset needs its licence read at the source, because some licences forbid commercial use.

Discuss: can we train on the evaluation cases?

Tomás looks at the dashed box in the label diagram and has an idea.

• Tomás: “The training data has no French ticket and nothing for rules H2 to H5. But the evaluation cases have exactly those. The dev cases alone have 10 French tickets and 13 that need a person under H2 to H5. Grace already labelled them. Let’s add them to the training data.”
• Priya: “Then dev and holdout stop measuring anything. The adapter would be tested on its own homework.”
• Grace: “I wrote those labels to test the assistant. The file says evaluation only.”
• Kwame: “Writing new French and H2 to H5 tickets costs Grace hours. Is the rule worth that?”

Before you read on, answer these:

1. After training on the dev cases, what would the dev score of the adapter tell you?
2. If the 13 H2 to H5 cases and the 10 French cases moved to training, what would be left to measure those slices?
3. What could the team do instead to get examples of those rules?

The course’s answer. No. The first reason is the permitted use in data/sources.toml: the owner of those labels gave them for evaluation only. The counts give the second reason. Dev holds only 13 cases under H2 to H5 and 10 French tickets. If they move to training, no case is left to show whether the adapter learned those rules. A score that cannot fail is not evidence. The next lesson adds a check that rejects any training record that nearly copies an evaluation case.

What would change the answer: new records written for training, with their own entry in sources.toml. The module’s practice task asks you to write a few. Or a larger labelled pool, split into training and test sets before anyone looks at it.

Knowledge check

Check your understanding

A colleague writes 100 training answers by hand, some as {"team": "Payment", ...} and some in the right format. Predict what the adapter will do on the strict answer check, and say how the project prevents this.

Show the answer

The adapter learns that both forms are acceptable, so it writes the capitalised team name some of the time. The strict check rejects those replies as unknown team, so the share of valid answers falls. The project prevents it: one function, format_answer, writes every answer, so the training data shows one format only.

Check your understanding

The label_how of T-60001 is "not read: no safety word". What does that tell you about its needs_human = false? Which kind of training ticket could get a wrong label in this way?

Show the answer

Nobody read the ticket. It is false only because it contains no word from the safety word list: an assumption, not a check. A ticket that needs a person under another rule often has no safety word. Examples are a legal threat (H2) or a question about someone else’s order (H4). So it would be labelled false, although Grace’s policy says true. The label_how column makes this visible, so a reviewer knows which labels to trust.

Check your understanding

Practical check: which sources in your project may be used as training data, and how do you show that data/source/needs_human.csv is the file that the provenance describes?

Show the answer

python -m adapt sources lists the permitted use of each source. tickets and needs_human may be used for training, validation and teaching. known_wrong_labels may only be used to remove records, and the evaluation cases are evaluation only. The fingerprint check proves the file: shasum -a 256 -c SHA256SUMS in data/ prints source/needs_human.csv: OK, and python -m pytest passes test_every_data_file_matches_its_recorded_source (12 passed in all).

Recap

• A training example is an input and its target: here, the same instruction, the ticket, and an answer written by format_answer. A model copies every habit of its examples.
• Provenance says where data comes from, how its labels were made and what it may be used for: data/sources.toml, with a fingerprint per file.
• 98 needs_human labels were read with rule H1; 4,598 were not read. No training record shows rules H2 to H5 or French.
• The evaluation cases stay evaluation only, even when they hold the examples that the training data lacks.
• Your project has sources, record and the source data: 12 passed.

Next: Clean, split and check for leakage