Final assignment

What you will be able to do

Use every module of the course on one change that a real team would ask for. You add a new policy rule and a new tool to the assistant, and you prove with tests and measurements that it is still reliable, safe and affordable.

Before you startWhat to know and to have readyAbout 300 minutes

• You completed Modules 1 to 6. Each part below names the lessons that it uses.
• You do not need the case studies for this assignment. The certificate asks for at least one of them.
• Start from your own ticket-assistant at the end of Module 6, or copy the course’s snapshot end-of-m06 from the labs repository, as in Test and package. Run python -m pytest first: all tests must pass before you change anything.
• Part 3 needs live model calls, because recorded answers cannot answer a new prompt. Use a free local model with Ollama, as in Test and package, or your own key. No payment is required: a local model is enough to pass.
• Tested with Python 3.12 and the versions in requirements.txt, on macOS.

The change: chargebacks and “my orders”

Grace has two new requests.

The first is a new rule. Some customers write that they will ask their bank to reverse a payment, a chargeback, or that they already opened a dispute. These tickets need a person on the payment team, because the money may already be on its way back and a wrong reply can cost the shop twice. Grace adds rule H6 to the policy: A ticket that mentions a chargeback, a payment dispute with the bank, or a reversed card payment goes to the payment team, and a person must handle it.

The second is a new tool. Agents often ask “what else did this customer order?”. Amira wants a read-only tool list_my_orders that returns the current customer’s recent orders. Tomás suggests a tool list_orders(customer_id), “so the model can look up anyone”. Priya points out what you learned in Execute in application code: the safest argument is one the model never chooses.

Omar asks the question that decides whether the change ships: “Does the assistant still route as well, and what does it cost per 1,000 tickets now?”

The tickets and orders are synthetic. No real customers.

Part 1: write the rule and the test tickets

Uses Specify the task and Compare prompts.

1. Write H6 in your own words, as precisely as the rules H1 to H5 in assistant/prompts/policy.md. Keep it in a note for now: Part 3 puts it into prompt v3. Do not edit policy.md itself. Every prompt version sends that file, so one changed character changes every recorded request, and the tests no longer find their recordings (the FailureCase of Test and package).
2. Write 8 new tickets in a file data/h6_tickets.csv with the same columns as data/tickets.csv: at least 3 that H6 covers (one in French), 2 near misses that it does not cover (for example “I do not want to start a chargeback, I just want my refund”, or a dispute about a delivery date), 1 that mixes a chargeback with a safety problem, and 2 ordinary tickets. Give each one the expected team and needs_human.
3. Decide how H6 treats a near miss, write the decision in one sentence next to the rule, and label the near misses by it. There is no single right answer; there must be a clear one.

Part 2: make the rule hard to break

Uses Define a schema and Validate meaning.

1. Add a business check to check_rules in assistant/validate.py: when a ticket matches H6, the answer must have team payment and needs_human true, or it is rejected with a problem code chargeback_needs_human.
2. Write the pattern so that your near misses behave as your policy sentence says. Show with tests that it catches all your H6 tickets and treats each near miss as decided.
3. Add tests with constructed answers (no model needed), as in the module practice of Validate meaning: a schema-valid answer that routes a chargeback to delivery must be rejected; a correct answer must pass. Then prove that a rejected answer cannot be saved: ResultStore.save() must raise InvalidResult, as in that practice.
4. Run the whole suite. If your pattern matches one of the 69 old tickets, a recorded answer that passed before may now be rejected. Look at each such ticket and decide whether the rule or the label is right.

Phrase rules are weak

A word list catches the words, not the meaning. Write down one ticket that your pattern gets wrong, even if it is not in your test set, and what would catch it.

Part 3: prompt v3, compared fairly

Uses Specify the task, Provide context safely and Compare prompts.

1. Copy assistant/prompts/v2.md to v3.md. Add H6 and what the model needs to apply it, and change “H1 to H5” to “H1 to H6”. Keep the ticket text marked as untrusted, as in v2.
2. Two small changes in assistant/__main__.py make eval ready: add "v3" to the choices of --prompt, and add an option such as --tickets PATH that passes another CSV file to load_tickets(). Without them, --prompt v3 is refused, and eval reads only data/tickets.csv.
3. Run python -m assistant eval --prompt v2 and --prompt v3 on your live model, on the 69 tickets of the evaluation set and your 8 new tickets. Run each version three times.
4. Report, per version and run: team accuracy, needs_human agreement, the number of answers each gate rejected, and your 8 tickets one by one.
5. Answer in three to five sentences: is v3 better, worse or the same on the old tickets, and how do you know a difference is not noise? Name every ticket whose result changed between runs.

Part 4: the new tool, safely

Uses Describe a tool, Execute in application code and Bound the loop.

1. Add list_my_orders to assistant/tools.py: no arguments from the model; the application passes the ticket’s customer_id, as get_order does. It returns at most the 5 most recent orders (order ID, date, status, total) and nothing else.
2. Describe it for the model (purpose, return type, limits) and let the loop run it. run_tool refuses every name except get_order, so extend it, and add a read-only query to OrderBook. Keep every limit of the loop. Do not change TOOLS: the recorded tests send it, and the mock has no recordings for a request with two tools. Put the two tools in a second list, and send it only when an option such as --my-orders is set.
3. Write tests: the tool never returns another customer’s orders; a call with any argument from the model is rejected; a repeated call stops the loop. Use a scripted model for the loop, as in the module check of Bound the loop.
4. Answer Tomás in three sentences: why is list_orders(customer_id) with a model-chosen ID unsafe, even if the prompt says “only for this customer”?

Part 5: reliability and cost

Uses Handle provider limits and Measure usage and cost.

1. Show with tests (the simulated provider) that the assistant still retries a rate limit with backoff, honours Retry-After, stops after the maximum number of attempts, and does not retry an HTTP 400.
2. Run python -m assistant usage over your Part 3 runs. It adds up every line of logs/usage.jsonl, so delete logs/ before the runs of each version, or group the lines by their prompt field. Report tokens per ticket (median, from the log lines), time per ticket (median and p95) and the estimated cost per 1,000 tickets for v2 and v3 on chat-small’s prices, even if you ran a local model. cost_usd("chat-small", tokens_in, tokens_out) in assistant/usage.py gives that estimate from the dated price table. Say which numbers are measured and which are estimated.
3. Check one log line by hand: no key, no full card number, no customer email.

Part 6: hand it over

Uses Test and package.

1. Update the README: what changed, the new commands and tests, and the optional live setup.
2. On a fresh copy of your project in a new virtual environment, run python -m pytest without any key or local model. Every test must pass, and the live tests must be skipped.
3. Write REPORT.md with the sections below.

What to submit

Your ticket-assistant folder (without .env, logs/ and the virtual environment) with REPORT.md:

# Ticket assistant: H6 and list_my_orders
## Policy and test tickets      (Part 1: H6, the near-miss decision, the 8 tickets)
## Validation                   (Part 2: the check, its tests, one ticket the pattern gets wrong)
## Prompt comparison            (Part 3: v2 vs v3, three runs each, what changed and why it is not noise)
## Tool                         (Part 4: list_my_orders, its tests, the answer to Tomás)
## Reliability                  (Part 5: the retry tests)
## Usage and cost               (Part 5: measured and estimated numbers, with dates and model names)
## Limits                       (synthetic data, phrase rules, one local model, what you did not test)

Check your work

A reviewer marks each item demonstrated, partially demonstrated or not yet demonstrated, with a reason. To complete the course, all four required items must be demonstrated.

• A validated output schema (required): answers are validated by the schema and by business checks including H6; a test proves that a rejected answer cannot be saved.
• Reproducible prompt comparisons (required): v2 and v3 on the same tickets, three runs each, with the model, date and settings recorded, and a conclusion that respects the noise.
• Bounded retries and tool calls (required): list_my_orders takes no model-chosen ID; tests prove the permission, the repeated-call stop and the retry limits.
• Usage logging and failure-path tests (required): the usage numbers with measured and estimated values marked; the whole test suite passes on a fresh copy without a key.

There are no key numbers to compare: your live model’s answers are yours. The course’s own runs (chat-small, prompt v2, structured output) give a reference for the old tickets in Compare prompts and Measure usage and cost.

Recap

You changed a working AI application the way a team does: a precise rule, tests that make it hard to break, a fair comparison that respects noise, a tool whose arguments the model cannot misuse, and numbers for what it costs. If you have not done one yet, choose a case study for the certificate: Extract product recalls into structured data, Answer questions with tools over open data or Classify banking requests with an LLM.