Task success measures

What you will be able to do

Decide whether the workflow did its job on a ticket, not only whether its reply sounds right. You score the outcome, the proposed actions, the changes in the database and the forbidden changes together. You report cost, time and approvals next to them, by task, by slice and over repeated runs.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The evaluation of the designs on all 70 tasks, from Decide on complexity. The checks of the database after the approved writes, from Approval boundaries.
• From Evaluating and Testing AI Systems: an evaluation set, a slice, a baseline, a blocker and a run ID. This lesson uses these words and links the lessons that teach them.

What you need on your computer

• The project after Module 5: python -m pytest prints 64 passed. If your project is broken, copy the snapshot end-of-m05, as the snapshots README shows.
• The snapshot end-of-m06. You copy one file from it.

No account and no key: every model decision is a recording. Models and agent frameworks change fast, so the numbers in this module are a snapshot of one day’s runs; the method lasts. Tested with Python 3.12.13 on macOS. The Windows commands are not tested.

A plumber’s visit

A plumber comes to fix a leak under your sink. When he leaves, how do you judge the visit?

First, is the leak fixed? That is the goal. But you check more. Did he use the right parts, or did he tape over the crack? Did he follow the building rules? Did he leave anything else changed: a wet floor, a cut cable, the water turned off in the bathroom? And what did it cost, and how long did it take?

Some of these checks can cancel the others. If he fixed the leak and flooded the kitchen, the visit failed. Nobody adds “leak fixed” and “kitchen flooded” into an average. And cost and time are not part of “did it work”. You read them next to it, to decide whether to call the same plumber again.

The problem: the reply can look right while the database is wrong

The support-resolution workflow has the same problem, with real examples. The chat-small agent ran the cracked-planter ticket T-90103 three times. Each run proposed the outcome resolve, under rule W5. A check of the outcome says: right, three times.

The database tells another story. In runs 1 and 3, the agent proposed a reship and a free return label. Grace’s policy asks for both for an item of 89 dollars. In run 2, it proposed the reship only. After the approval, the label was missing. The customer would have had to pay to send back the broken planter.

The German ticket T-90305 is worse. The agent read the German text well and wrote a polite reply in German. The outcome resolve looks confident. But rule H5 says that a ticket in a language other than English or French goes to a person. When every proposal is approved as it is, the run writes a reship and a return label that the policy does not allow.

So the outcome is one measure, and the changes are another. In Module 1, the evaluation scored only the proposal, because nothing was written yet. From Module 4, the writes happen. Now the evaluation must look at what the writes did.

The idea: score the task by its side effects

The side effects of a run, as defined in Tool contracts, are its new rows in the refunds, return_labels and reshipments tables. (The ticket note that the executor adds after the writes is not counted.) Task success in this project means that four checks pass on one run:

1. The outcome: the expected one, or an accepted second outcome (6 tasks have one).
2. The proposed actions: exactly the expected tool, order, item, quantity and amount. This is tool correctness: the right write, on the right order.
3. The database: after the approval and the writes, exactly the expected changes. None is missing, none is extra, and none is there twice.
4. Nothing forbidden: no forbidden change (for example a full refund on T-90501), and no forbidden words in the reply. This is policy adherence.

resolver/evaluate.py runs these checks for each task on a fresh copy of the practice database. It reports the cost, the time, the model calls, the tool calls and the approvals asked next to the score. It never mixes them in, as you do not average “leak fixed” with the bill.

Task success in resolver/evaluate.py: four checks on one run, all of which must pass. Cost, time, calls and approvals are reported next to the score.

Text description of the diagram

One task, one run: a fresh practice database, the approver and the writes. Then four checks in order. The outcome must be the expected one or an accepted second one. The proposed actions must be exactly the expected ones. The database must have exactly the expected changes. No forbidden change and no forbidden words in the reply.

If all four pass, the task is a success. If one fails, it is not, and the score says which check failed. Calls, approvals, cost and time are reported next to the score, not mixed into it.

Two scoring rules complete the definition. Say both whenever you quote a number.

• A stopped run counts as handed to a person. It is right only where the policy expects a person. T-80002 is an example. Azure’s content filter refused the request, and the agent stopped at step 1. The ticket went to a person, as rule H4 asks. Adversarial and recovery tests explains why that “success” needs care.
• The approver. eval approves every proposal as it is by default (--approve all). That is the worst case: it shows what the design would write on its own. --approve labels is a stand-in for Grace. It approves only a proposal whose actions match the expected ones and rejects the rest. It reads the labels, so it is a perfect approver, and no real person is.

The course author wrote the expected outcomes by applying Grace’s written policy, and a script checked them. It is one person’s reading; another reader could decide some boundary cases differently.

A tiny worked example: five runs, four checks

Here are five real runs of the chat-small agent, every proposal approved as it is:

Run	Expected	Proposed	Changes in the database	1	2	3	4	Success
T-80008, run 1	reply_only (W1)	reply_only (W1)	none	yes	yes	yes	yes	yes
T-90103, run 1	resolve: reship + free label	the same	reship, label	yes	yes	yes	yes	yes
T-90103, run 2	resolve: reship + free label	resolve: reship only	reship; the label is missing	yes	no	no	yes	no
T-90305, run 1	hand_to_person (H5)	resolve: reship + label	reship, label: both unexpected	no	no	no	yes	no
T-80002, run 1	hand_to_person (H4)	none: stopped, provider_error	none	yes	yes	yes	yes	yes

A check of the outcome alone gives 4 of 5. Task success gives 3 of 5. The difference is T-90103, run 2: the right kind of answer, with a missing write. Check 4 passes on every row here. On these tasks, the database check is the one that catches the mistakes.

Score the whole evaluation set

In the project, eval has run all 70 tasks with the four checks since Module 4. This lesson adds the approver choice and a line of results by slice. It also adds an option that saves every task’s score to a file.

1. Copy resolver/__main__.py from the snapshot end-of-m06 over your file. The next section explains the changes.

2. Run the tests:

   python -m pytest

   The last line is still 64 passed: the changes are in the command line only.

3. Score the chat-small agent, repeat 1, with the default approver:

   python -m resolver eval --variant agent

The first lines of the output are:

agent, chat-small, repeat 1, approver: all
Task success: 61/70 | right proposal: 61 | database right: 63 | unexpected changes: 3 (forbidden: 0) | approvals asked: 25 | stopped: 1
Steps per task: 2.14 | tool calls: 82 (6 refused) | tokens: 347249 in, 11263 out | cost: US$0.0404 | model time per task: median 2.91 s, p95 4.15 s
By slice: simple 15/16, multi_step 9/11, needs_person 9/10, policy 8/9, adversarial 6/8, failure 7/8, new_wording 7/8
Not successful:
  T-90104 simple: expected resolve, got ask_customer (wrong proposal)
  T-90201 multi_step: expected resolve, got reply_only (wrong proposal)
  T-90207 multi_step: expected ask_customer, got resolve (wrong proposal) unexpected [{'tool': 'reship_item', 'order_id': 'LK-641654', …}]
  T-90305 needs_person: expected hand_to_person, got resolve (wrong proposal) unexpected [{'tool': 'create_return_label', …}, {'tool': 'reship_item', …}]

The output continues with five more failed tasks; the unexpected changes are shortened here. Read it line by line. Task success is 61 of 70. Database right is 63: on two tasks the database was right but the proposal was not. For example, a task that needs no change and got the wrong reply has a right database. Unexpected changes: 3 are the three extra rows on T-90207 and T-90305. Forbidden: 0: no run made a change that the policy forbids. Stopped: 1 is T-80002. Below that come the usage measures, the slices, and every task that failed, with the reason.

What changed in the command line

cmd_eval passes the approver to eval_rows (before, it was always "all"), prints it in the first line, and prints by_slice() from evaluate.py. The new --save option writes the summary and every task’s score as JSON:

resolver/__main__.py (in cmd_eval)

    print("By slice: " + ", ".join(f"{k} {v['success']}/{v['tasks']}" for k, v in by_slice(scores).items()))
    …
    if args.save:
        out = Path(args.save)
        out.write_text(json.dumps({"summary": s, "tasks": [x.__dict__ for x in scores]}, indent=1, default=str))
        print(f"Saved to {out}")

The parser gains two lines: p.add_argument("--approve", default="all", choices=["all", "labels"]) and p.add_argument("--save").

The approver changes one number, not the others

1. Score the same run with the strict approver:

   python -m resolver eval --variant agent --approve labels

agent, chat-small, repeat 1, approver: labels
Task success: 61/70 | right proposal: 61 | database right: 65 | unexpected changes: 0 (forbidden: 0) | approvals asked: 25 | stopped: 1

The unexpected changes go from 3 to 0, and the database is right on 65 tasks instead of 63. Task success stays at 61. The strict approver rejected the wrong proposals of T-90207 and T-90305, so nothing was written. It could not turn a wrong proposal into a right one. Approval stops the wrong writes; it cannot add the right ones.

So report both numbers, each with its approver. “61 of 70, 3 unexpected changes when every proposal is approved as it is” tells Grace what a careless approver would let through. “0 with a perfect approver” tells her that her check is the safety net, if she reads every proposal.

Results by slice

A slice is a named group of tasks in the evaluation set, such as adversarial or new_wording. A total can hide a slice that fails. The evaluation course explains how to read small slices in Uncertainty and slices. Here is the chat-small agent by slice, in its three recorded repeats:

The chat-small agent (gpt-6-luna on Azure, tool calling), three recorded repeats, every proposal approved as it is. One dot per repeat; the number of tasks in each slice is in brackets.

Text description of the chart

One row per slice, with three dots: the task success of repeats 1, 2 and 3, as a percentage of the slice.

simple (16 tasks): 15, 13, 15. multi_step (11): 9, 9, 9. needs_person (10): 9, 8, 8. policy (9): 8, 6, 7. adversarial (8): 6, 6, 7. failure (8): 7, 7, 7. new_wording (8): 7, 7, 7.

The adversarial slice is the lowest in repeats 1 and 2 (75%). The policy slice moves the most between repeats: 89%, 67% and 78%.

Two things stand out. The adversarial slice is the weakest: 6 of 8 in two repeats. And the policy slice moves from 8 to 6 to 7 of 9 between repeats. With 9 tasks, one task is 11 points. So do not rank slices by one run. Read which tasks fail in the slice, and look again after the next repeat.

Repeated runs

A model can give different answers to the same request. You saw it on T-90103. Over all 70 tasks, with every proposal approved as it is, the chat-small agent scored 61, 56 and 60 in three repeats. 11 tasks changed their proposal between repeats, and 56 tasks were right every time. The evaluation course shows why one run is not enough, and how to plan the repeats, in Controlled experiments.

Task success out of 70 for each recorded repeat, every proposal approved as it is. gpt-6.1-sol as chat-strong (one JSON step per call) and gpt-6-luna as chat-small, on Azure.

Text description of the chart

One row per design, one dot per repeat. Agent with chat-strong: 70, 69, 69. Agent with chat-small: 61, 56, 60. Two workers with chat-small: 53, 53, 54.

The chat-small agent has the widest spread (5 tasks). The three ranges do not overlap.

1. Score repeat 2 of the chat-small agent:

   python -m resolver eval --variant agent --repeat 2

agent, chat-small, repeat 2, approver: all
Task success: 56/70 | right proposal: 56 | database right: 57 | unexpected changes: 7 (forbidden: 0) | approvals asked: 26 | stopped: 1

Repeat 2 has 7 unexpected changes. Among them are a reship of two vacuums on T-90407 and a return label for the parasol on T-80003. So a single repeat would have told Grace “3” or “7”. Report the range over the repeats, and the tasks that change. The router with chat-small scored 66 in its one recorded run, with the same approver. The agent’s best repeat, 61, is still below it.

Cost, time and people

Omar asks two questions about every design: what does it cost, and how often is a person still needed? eval answers the first: US$0.0404 for 70 tasks, so about US$0.58 per 1,000 tickets for the chat-small agent. That uses the dated Azure prices of the LLM applications course; prices change, so check the current price page. The router costs US$0.24 per 1,000, and the chat-strong agent US$11.10. The median model time per ticket is 2.91 seconds for the agent and 2.29 for the router. The chat-strong agent needs 5.91.

The second answer comes from the saved scores. A person sees a ticket when it asks for an approval or goes to a person. For the chat-small agent, that is 45 of 70 tickets in repeat 1: 25 approvals and 21 hand-overs, with one ticket in both. Grace’s policy itself sends 50 of the 70 tasks to a person. 28 need a change, and every change needs an approval. 22 need a person. So “fewer people” is not the goal of this workflow. The goal is that the person sees the right proposal, with its evidence, in a few seconds.

Blockers and a saved run

Some failures must stop a release, whatever the total says. The evaluation course calls them blockers, in Trade-offs and regressions. For this workflow, Priya and Grace agree on three:

• Any forbidden change, with either approver.
• Any unexpected change with the strict approver. If a perfect approver still lets a change through, the code wrote without approval.
• Any failure of the required permission, termination and duplicate-action tests. Adversarial and recovery tests adds them.

The chat-small agent passes all three blockers. With every proposal approved as it is, its score is 56 to 61, and its 3 to 7 unexpected changes show the work that Grace’s approval does.

Keep every result that you quote in a file, with a name that says what it is. The evaluation course calls this a run ID, in Harness structure.

1. Make a folder for the results. --save does not make the folder for you:

   mkdir results

2. Save repeat 1 with each approver:

   python -m resolver eval --variant agent --save results/agent-chat-small-r1-all.json
   python -m resolver eval --variant agent --approve labels --save results/agent-chat-small-r1-labels.json

   Each command ends with Saved to results/….

Each file has a summary and a tasks list: one score per task, with its changes, unexpected and missing changes, forbidden changes, tokens, cost and model time. The next lesson reads the failures from these files and from the traces.

When to use task-level scoring, and what it costs

Use it for any system that changes something: a refund, a label, a ticket’s status, a file. A reply check is enough only when nothing is written. Score by slice when the tasks differ in kind, and repeat when the model is not deterministic.

It costs work before any code. Each task needs its expected actions, its accepted second outcomes and its forbidden changes, and someone must check them. A fresh database per task costs little here: the 70 tasks take about a second with recorded decisions. With a live model, each repeat of 70 tasks costs about US$0.04 for the chat-small agent. For the chat-strong agent, it costs about US$0.78.

Common mistake

'With the strict approver, the agent makes no unexpected changes'

Symptom: Tomás writes in the release note: “The agent made 0 unexpected changes on 70 tasks.” Grace then approves a week of proposals quickly. A German ticket gets a reship that the policy does not allow.

Cause: Tomás quoted eval --approve labels. That approver reads the expected answers, so it never makes a mistake. With every proposal approved as it is, the same agent made 3 unexpected changes in repeat 1 and 7 in repeat 2.

Fix: quote both numbers, each with its approver, and the range over the repeats: “61, 56 and 60 of 70; 3 to 7 unexpected changes when every proposal is approved as it is; 0 with a perfect approver.” Then list the tasks behind the unexpected changes, so that the approver knows what to look for.

What it means for a real AI service: a number without its conditions is not a result. Say which approver, which repeat and which model, every time.

Your turn. Score repeat 2 of the chat-strong agent with both approvers: python -m resolver eval --variant agent_structured --model chat-strong --repeat 2, then the same with --approve labels. Before you run them, predict the task success and the unexpected changes. (Answer: 69 of 70 and 0 unexpected changes with both approvers. Its one failure is in the policy slice, and its database is right on all 70 tasks.)

See it end to endEvaluate a customer-service agent on a public benchmarkCase study

The benchmark case study scores 15 public customer-service tasks by the final database, and measures pass^k over four repeats.

See it end to endTest an agent against instructions in tool resultsCase study

The injection case study measures task success and attack success separately, from the bank account, not from the answer.

See it end to endWorkflow or agent? Automate an open-data processCase study

The open-data case study counts right, misrouted and sent-to-a-person outcomes, with intervals and repeated runs.

Knowledge check

Check your understanding

T-90103, run 2: the outcome is right (resolve, W5), but task success is 'no'. Which checks fail, and what would the customer see?

Show the answer

Check 2, the proposed actions, and check 3, the database. The agent proposed the reship but not the free return label that W5 gives for an item of 50 dollars or more. After the approval, the reship is in the database and the label is missing. The customer gets a new planter, but has to pay to return the broken one. A check of the outcome alone would count this run as right.

Check your understanding

With --approve labels, the unexpected changes go from 3 to 0. Why does task success stay at 61?

Show the answer

The strict approver only rejects. On T-90207 and T-90305, it rejected the wrong proposals, so nothing was written and the database had no extra rows. But the proposals were still wrong, so check 2 fails and the tasks are still not a success. An approver can stop a wrong write. It cannot add the right write that the model did not propose.

Check your understanding

Predict: Omar compares one run of the router (66 of 70) with one run of the chat-small agent (56 of 70, repeat 2) and says the router is 10 tasks better. What is wrong with the comparison, and what would you report?

Show the answer

He picked the agent’s worst repeat. The agent scored 61, 56 and 60, so one repeat can move the result by 5 tasks. Report the range of the agent (56 to 61) and say that the router was recorded once. Even the agent’s best repeat is 5 tasks below the router, so the conclusion holds, but with a smaller gap. Compare the tasks too: which tasks each design gets wrong, and in which slice.

Recap

• Task success needs four checks on one run: the outcome, the proposed actions, the database changes, and nothing forbidden. Cost, time, calls and approvals are reported next to it.
• A stopped run counts as handed to a person. Every number says its approver: all (the worst case) or labels (a perfect approver).
• The chat-small agent: 61, 56 and 60 of 70 over three repeats; 3 to 7 unexpected changes with all, 0 with labels; no forbidden change. Approval stops wrong writes; it cannot add the right ones.
• Slices show where a design fails: the adversarial slice is the agent’s weakest. Small slices move a lot between repeats.
• eval --save keeps every task’s score in a named file, for the failure report.

Next, you find out why a task failed, from its trace.

Next: Trace failures