Final assignment

What you will be able to do

Use every module of the course on a system that you build yourself: a mock support assistant made of rules and templates. You give it a versioned, labelled evaluation set and measure it with the right metrics and slices. You check its replies with a rubric and a second rater. Last, you compare it with Larkfield’s baseline in a report that anyone can rebuild.

Before you startWhat to know and to have readyAbout 300 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them.
• Start from your own eval-harness at the end of Module 6, or copy the snapshot end-of-m06 as Use a snapshot explains. Run python -m pytest first: it must show 106 passed, 1 deselected before you change anything. Commit that state, or keep a copy, so that you can compare.
• No account and no key are required. A live judge (Ollama or your own key) is optional in Part 4, always through the cap.
• Some tests pin the course’s own set, for example 271 cases. They fail after Part 1, as expected. Update them, and list in your report each test that you changed and why.
• Tested with Python 3.12 and the versions in requirements.txt, on macOS.

The change: a candidate with no model at all

Omar has a new idea after the month in production: “The model is slow and it reasons for too long. What if a few rules choose the team and needs_human, and the reply comes from a template? It would cost nothing and never go silent.” Grace answers: “Then show me that it never misses a safety ticket, and that its replies are still useful.”

Your candidate is that mock support assistant: plain Python, no model. It is a real candidate, and the harness must judge it like any other. It also gives you something that the recorded runs cannot: a system that you can change and run again in a second.

The production month that Part 1 uses is simulated, and so is every ticket that you write. Say so in your report.

Part 1: extend and version the evaluation set

Uses Select representative cases, Label and version, Avoid contamination and Production feedback.

1. Write at least 10 new dev cases for what the simulated month showed. Write at least 5 price-drop requests and at least 3 more French tickets. Among the price-drop requests, write one in French, one that also mentions a safety problem, and one that asks for nothing the policy forbids. Give each case every field that the other lines have. Label each one with Grace’s policy (data/policy.md): team, needs_human and rule, next step, criteria.must and criteria.must_not, critical, and a label that says who labelled it, how and when.
2. Add them to data/cases.jsonl with save_cases(), as in Add new cases under control. check() must pass. Show that the holdout’s inputs version is still 1be23cc44ea9.
3. Record the new dataset version, and add a section to data/dataset.md: what you added, why, who labelled it, and what is still missing.
4. Ask a second person to label 5 of your cases without seeing your labels. Compare, and record how you resolved each disagreement. If nobody can help, say so, and mark the labels as reviewed by one person only.

Part 2: the mock assistant, as a run

Uses Establish a baseline, Controlled experiments and Harness structure.

1. Write mock_assistant.py: a function that takes a ticket text and returns team, needs_human, reason and reply. Use keyword rules for the team and for rules H1 to H5, and one reply template per next step. Keep it under about 100 lines: the point is the evaluation, not the assistant.
2. Write its outputs as a run, in the format of the saved runs. Write one line per case in outputs/<run ID>.jsonl, with every field of Output (cost 0, latency 0, finish_reason stop). Add one entry in outputs/runs.json, with every field of Run. Build the ID with make_run_id(), with the SHA-256 of your script as the prompt’s fingerprint.
3. Run it twice and show that the two output files are identical: a deterministic system needs one run, not five. Explain why in one sentence.
4. Use the full run ID in every command. If you record a second run of the same system with repeat 1, the short name stops working (No run 'rules'), on purpose.

Part 3: metrics and slices

Uses Prediction metrics, Generated-output metrics and Uncertainty and slices.

1. Score your run on dev with score, --by slice and --by criterion: valid answers, team, needs a person (caught, false alarms, precision and recall), reply criteria.
2. Explain which measure matters most for this candidate, and why. A missed safety ticket and a misrouted delivery question do not cost the same.
3. Find one failure that the overall score hides, by slice or by criterion, and show its cases.
4. Read every failure of a must-not rule and every missed needs-a-person case with show. Mark each as a real failure or a checker error.

Part 4: a human-reviewed judging process

Uses Write a usable rubric, Calibrate reviewers Model-based judging and Model-based judging, continued.

1. Make a blind sheet of 20 replies: 10 from the baseline and 10 from your mock, on the same dev cases. Shuffle them, and remove the system’s name. Keep the key in a separate file.
2. Rate every reply with Grace’s rubric (harness/prompts/rubric.md), with a one-line note each.
3. Get a second rating of the same 20 replies: a second person (best), or a live judge through the cap. Compare with python -m harness agree, and discuss every pair that is 2 or more points apart.
4. Try the recorded judge on your mock’s replies with python -m harness judge <your run ID>. Explain the result: it answers No recording for this request, because the mock judge can only replay requests that were recorded.
5. Write the rule for your process. Which replies must a person always read? For example: every critical case, and every case where the judge gives no verdict, such as T-81040. Which replies may a judge score alone, if any?

Part 5: a reproducible release comparison

Uses Trade-offs and regressions, Harness structure and Regression checks.

1. The baseline has no outputs for your new cases. Run report and compare on dev, and find the difference. report counts a missing baseline output as a failure (n = 103 with one new case). compare uses only the cases that both runs have (n = 102). Change report.py so that it compares only the cases that both runs have. Its first lines must say how many cases it left out, and why. Add a test for it.
2. Fix the tests that your new cases broke (for example, the regression checks look up a baseline output that does not exist). Keep the rules; change only how they find their cases. List each change.
3. Add at least one regression check for your mock: for example, every safety ticket goes to a person. It must pass on your final run.
4. Make the decision on dev with decide, change the mock, and run again. Record each attempt as a new run with its own ID. Stop when you are satisfied or when you have tried three changes.
5. Then decide on the holdout set, once, with your final run, and write the report into report-holdout.md.
6. Rebuild the report in a fresh copy of your project and compare the SHA-256 values. Trace every number of the Measures table to its cases.

What to submit

Your eval-harness folder (without .venv and .env), with the new cases, the mock, its runs, your rating sheets and REPORT.md:

# A mock support assistant, evaluated
## Evaluation set          (Part 1: new cases, labels and who made them, versions, the second labeller, what is missing)
## The mock assistant      (Part 2: what it does, its run IDs, why one run is enough)
## Measures and slices     (Part 3: the measures, the one that matters most, the failure the overall score hides)
## Judging                 (Part 4: the rubric ratings, the second rating and agreement, where a person must read)
## Release comparison      (Part 5: the report change, the tests changed and added, dev attempts, the holdout decision)
## Reproduce and trace     (the SHA-256 in two copies; every number of the Measures table traced to its cases)
## Limits                  (simulated tickets, one or two raters, what you did not test)

Check your work

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a reason. To complete the course, all four required items must be demonstrated. No strength elsewhere makes up for a failed safety or access check. If your report says “ship” while a safety ticket goes to no one, the comparison item is not demonstrated.

• Versioned dataset and labels (required). Demonstrated: at least 10 new dev cases with complete labels and provenance; check() passes; a new dataset version is recorded; the holdout’s inputs version is unchanged; the dataset card says what was added and by whom; a second labelling, or its absence, is recorded. Partially: the cases are there, but labels lack provenance, or the holdout changed, or the version is not recorded.
• Appropriate metrics and slices (required). Demonstrated: decisions, needs-a-person precision and recall, and reply criteria, with a reasoned choice of the most important measure; one failure hidden by the overall score, shown by its cases; every must-not failure read. Partially: only overall numbers, or no hidden failure found.
• Human-reviewed judging process (required). Demonstrated: 20 blind rubric ratings with notes; a second rating and its agreement (exact, within one, Cohen’s kappa); the large disagreements discussed; a written rule for where a person must read, with a reason from the course. Partially: ratings without a second rater or without the rule.
• Reproducible baseline-versus-candidate comparison (required). Demonstrated: the report compares only shared cases and says so; tests cover it and your regression check; dev attempts are separate runs with their own IDs; the holdout is used once; the report is identical in a fresh copy; every number in the Measures table is traced to its cases. Partially: the decision is right, but the report cannot be rebuilt, or a number cannot be traced.

There are no key numbers to compare: your cases and your mock are your own. The course’s numbers for the recorded runs are in Trade-offs and regressions and Harness structure.

Stretch, not required

Record a second candidate with a real model: a free local model with Ollama, with prompt v2, written as a run in the same format. Compare it with the baseline and with your mock. Keep the cap, and say which model and version you used.

Recap

You built what a team needs before it can trust a change. The set learns from production while its holdout stays frozen. The measures look past the average, and a second rater checks the ratings. Anyone can rebuild the release report and trace it to its cases. If you have not done one yet, choose a case study for the certificate: Calibrate a model judge against human ratings, Evaluate summaries against expert ratings or Regression-test a classifier release by slice.