Skip to content
Français

Service levels for agents

  • Lesson 23
  • 35 min
  • Module 7: Alerts, service levels and quality

Where we are

In Module 6, we made the workflow's decisions answerable after the fact: who proposed, who approved, what changed. But nobody should need an audit question to learn that the agent stopped doing its job. In this lesson, we choose service levels for an agent: the indicators that match what the help desk promises, and the objectives we hold them to. We start with a pizza shop's promise.

Plays from YouTube. Nothing loads from YouTube until you press play. Privacy

What you will be able to do

Write down what the help desk promises. Then turn each promise into a number that the agent’s telemetry can measure, and give it a target. Then “is the agent working?” has an answer every day, not only after an incident.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer

  • Your project as the last module left it: the snapshot end-of-m06, where python -m pytest -q ends with 104 passed, 1 skipped, 1 deselected.
  • The virtual environment active, and a terminal in the project folder, observable-workflow.

Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run. The replayed week’s decisions are real recordings of the agents course; its traffic is constructed, and the approval times and the reviews are simulated.

A pizza shop writes on its door: “Your pizza, hot and right, in 30 minutes.” That sentence is a promise to customers. To keep it, the shop counts two things at the door: the pizzas that left late, and the ones that came back wrong.

Two details make the count honest. First, a pizza that arrives on time with the wrong topping is not a success. It fails even if the oven, the driver and the till all worked. Second, the 30 minutes count from when the shop is open. An order placed online at midnight for the next lunch is not 12 hours late.

So a good promise has a number you can count, and the count follows the customer’s view and the shop’s own clock.

The problem: one number says good, another says bad, and both are wrong

Section titled “The problem: one number says good, another says bad, and both are wrong”

Larkfield’s workflow already has numbers. Here is one from the cost report of Module 5, split by the language of the ticket:

language runs resolved cost US$ per run per resolved success
de 6 5 0.0029 0.00048 0.00057 83.3%
en 302 190 0.6480 0.00215 0.00341 62.9%
fr 8 8 0.0158 0.00198 0.00198 100.0%

The German tickets look like the best slice after French: 83.3% “success”. But that column counts tickets closed as resolved. Larkfield’s rule H5 says that a German ticket goes to a person. The agent resolved five of the six instead: it reshipped a compost bin and wrote to the customer in German. By the course’s labels, only 1 of the 6 German runs was right. A high “success” was a failure.

The other number goes the opposite way. The double refund’s ticket, T-90104, waited 15 h 25 min for a person to approve it. That sounds like a broken promise. But the proposal arrived on Wednesday at 17:52, after the help desk closed. Sam approved it at 09:17 the next morning, 47 minutes after the desk opened. Nobody was slow.

Both numbers are real. Neither one measures what the help desk promises.

Start from the people, as the MLOps course does. A service level indicator, or SLI, is a measured share: good events ÷ all events. A service level objective, or SLO, is a target for an SLI over a window, the stretch of work it counts. The SLI is the counter at the pizza shop’s door; the SLO is the sentence on the door.

For one web service, the indicators are often “answered” and “answered fast”. For an agent, four things change.

  1. Success is the outcome, judged by a person. “Resolved” only says the run ended there. A run can end without an error, with the reply sent, and still do the wrong thing. The indicator needs a judgement: a person’s review of a sample of runs. The last lesson of this module shows how to collect it.
  2. A handoff is both a promise and a cost. Some tickets must reach a person, such as a German ticket (rule H5). Each handoff also takes Grace’s team time. A handoff rate that climbs means the agent finishes less. A rate that falls can mean the agent resolves tickets it should hand over, as with the German ones.
  3. The agent’s time includes people. A run that waits for an approval waits for a person. Count that wait on the help desk’s clock, which stops when the desk is closed.
  4. Cost per resolved ticket is a budget, not a promise. Customers do not feel it, but Kwame and Omar do. Treat it like an objective with an owner.

At Larkfield, every window is counted in closed runs, not in hours: 45 runs, about one day of tickets. A count of runs means the same thing in a practice replay and on a real day.

Four rows, each from a promise to an indicator to an objective. One: your ticket gets the right outcome; reviewed runs judged right divided by reviewed runs; at least 85% over a week of reviews, owner Priya. Two: what the agent cannot finish reaches a person who has time; tickets handed to a person divided by closed tickets; at most 45% over the last 45 runs, owner Grace. Three: a person decides on a proposal within an hour of staffed time; approvals decided within 1 staffed hour divided by approvals; at least 95% over a week, owner Grace. Four, in a dashed orange box: a budget, not a promise, model cost per resolved ticket; model cost divided by resolved tickets; at most US$0.002 over the last 45 runs, owner Kwame.
Larkfield's four service levels for the workflow, from the promise to the objective. The targets are assumptions: Larkfield's decisions.
Text description of the diagram

Each row starts from what someone was promised. The first three are promises of the help desk: the right outcome, a person when one is needed, and a fast decision on a proposal. The fourth is Omar’s budget for the model calls. Each promise becomes an indicator that the telemetry can count: reviews, runs by outcome, the approval wait, and the run cost. Each indicator gets a target, a window and one owner.

Here are the first ten runs that closed in the replayed week, early on Monday 12 October. The last column is the course’s own label for each run. A real team does not have it: it is here so that you can see what the indicators miss.

# Run Outcome Cost US$ Approval wait (wall / staffed) Review Right by the labels
1 R-1012-001 resolved 0.001105 – – no
2 R-1012-002 handed to a person 0.000266 – – yes
3 R-1012-003 handed to a person 0.000518 – – yes
4 R-1012-004 resolved 0.000744 – – yes
5 R-1012-006 resolved 0.000540 – – no
6 R-1012-005 resolved 0.000542 123 min / 19 min pass yes
7 R-1012-008 resolved 0.000503 – – yes
8 R-1012-009 handed to a person 0.000579 – – yes
9 R-1012-007 resolved 0.000811 39 min / 39 min – yes
10 R-1012-010 handed to a person 0.000266 – – yes

Compute the four indicators of this window by hand:

  • Handoff rate: 4 handed to a person ÷ 10 closed = 40%, under the 45% target.
  • Cost per resolved ticket: the ten costs add up to US$0.005874. Six runs ended resolved. So the cost per resolved ticket is US$0.005874 ÷ 6 = US$0.00098, under US$0.002.
  • Approval wait: two runs waited for a person. Run 6 was proposed at 06:46, before the desk opened at 08:30. It waited 123 minutes on the wall clock, but only 19 minutes of staffed time. So 2 of 2 were decided within one staffed hour.
  • Right outcome: one run of the ten was reviewed, run 6, and it passed: 1 of 1.

Now look at the last column. Runs 1 and 5 ended resolved, but the labels say they were wrong. Both are tickets from the agents course’s adversarial slice, written to trick the agent. The resolved share, 6 of 10, counts them as successes. The review sample did not pick them, so the right-outcome indicator cannot see them either. One review is not a measurement. That is why the objective for the right outcome counts a week of reviews, about 60, and not one day.

The approval wait needs its own clock. Larkfield’s desk is staffed on weekdays from 08:30 to 17:30, and on weekends from 10:00 to 16:00. A proposal made at night waits until the morning. That is the plan, not a failure.

A time line from Wednesday 17:00 to Thursday 10:00. Green bands mark staffed hours: until 17:30 on Wednesday and from 08:30 on Thursday; between them, a grey band says closed: night. A vertical line marks the proposal at 17:52 and another marks the decision at 09:17. Below, an orange bar from 17:52 to 09:17 says wall clock: 15 h 25 min. Under it, a short green bar from 08:30 to 09:17 says help desk clock: 47 min.
One wait, two clocks: T-90104 in the replayed week (run R-1014-040). Approval times are simulated.
Text description of the diagram

The proposal for T-90104 was made on Wednesday at 17:52, 22 minutes after the help desk closed. Sam approved it on Thursday at 09:17. On the wall clock, the run waited 15 hours and 25 minutes. On the help desk’s clock, which only runs in staffed hours, it waited 47 minutes: from 08:30 to 09:17.

The project now computes both. obs slo adds up, for each approval, the part of the wait that falls in staffed hours (staffed_seconds in obs/alerts.py). The indicator is then: approvals decided within one staffed hour ÷ all approvals.

Grace, Priya, Omar and Kwame wrote one objective for each indicator. Each target is an assumption: Larkfield’s decision, set from what the healthy week does.

Indicator Target and window Owner Why this target
Right outcome, by the sampled reviews at least 85%, over a week of reviews (about 60) Priya By the labels, the healthy week was right on 269 of 316 runs (85.1%). A higher target would fail every week.
Handoff rate at most 45%, over the last 45 closed runs Grace Grace’s team can take about 20 handoffs a day. In the healthy week, no window of 45 runs went above 42.2%.
Approvals decided within 1 staffed hour at least 95%, over a week Grace The desk’s own promise to its agents: nobody waits more than an hour of working time.
Cost per resolved ticket at most US$0.002, over the last 45 closed runs Kwame Omar’s budget from Module 5. The healthy week costs US$0.00091.

The right-outcome SLO also gives an error budget: the number of bad events that the target allows. With 85% of 60 reviews, the budget is 15% × 60 = 9 failed reviews a week. The MLOps course shows how to read how fast a budget is spent, the burn rate, in Service objectives.

Add this module’s files and read the week

Section titled “Add this module’s files and read the week”

This module’s project change adds four commands: obs slo, obs alerts, obs reviews and resolver review. It also adds the alert rules and the counters that the alerts need. First check that the command does not exist yet.

  1. Run python -m obs slo --data data/telemetry/week. On end-of-m06 it fails with invalid choice: 'slo'.

  2. Copy this module’s files from the snapshot end-of-m07:

    Terminal window
    S=../nextia-learning-labs/C31/snapshots/end-of-m07
    cp $S/obs/alerts.py $S/obs/__main__.py obs/
    cp $S/resolver/__main__.py $S/resolver/execute.py $S/resolver/observe.py $S/resolver/telemetry.py resolver/
    cp $S/scenarios/week.py scenarios/
    cp -R $S/grafana/alerts grafana/
    cp $S/grafana/compose.yaml grafana/
    cp $S/tests/test_saved_week.py $S/tests/test_stage_m07.py tests/
    cp $S/README.md .
  3. Run the tests: python -m pytest -q. The last line says 112 passed, 1 skipped, 1 deselected. Your project is now the same as the snapshot end-of-m07.

  4. Read the replayed week’s service levels:

    Terminal window
    python -m obs slo --data data/telemetry/week
    window (closed runs) runs resolved handoff US$/resolved wait p95 h unrecon. reply fail loops flagged
    Mon 12 Oct 00:57 - 12 Oct 21:03 45 57.8% 28.9% 0.00107 2.0 0 0 0 0
    Tue 13 Oct 02:08 - 13 Oct 15:58 45 75.6% 22.2% 0.00078 15.5 0 0 0 0
    Tue 13 Oct 16:32 - 14 Oct 14:15 45 64.4% 24.4% 0.00399 15.8 1 0 0 1
    Wed 14 Oct 15:00 - 15 Oct 14:03 45 68.9% 31.1% 0.00537 15.4 2 5 0 4
    Thu 15 Oct 14:55 - 16 Oct 11:14 45 71.1% 22.2% 0.00361 6.2 0 4 0 2
    Fri 16 Oct 11:21 - 17 Oct 11:15 45 60.0% 37.8% 0.00452 15.9 0 0 4 2
    Sat 17 Oct 11:18 - 19 Oct 08:45 45 51.1% 42.2% 0.00398 16.7 0 0 9 3
    Mon 19 Oct 08:55 - 19 Oct 08:55 1 100.0% 0.0% 0.00058 14.2 0 0 0 0
    the whole week 316 64.2% 29.7% 0.00328 15.4 3 9 13 12
    Approvals: 118; median wait 31.8 min; 19 waited more than 8 h.
    In staffed hours only (the clock stops at night): decided within 1 h: 118 of 118 (100.0%); median 25.6 min, longest 58.1 min.
    Sampled reviews: 60 of 316 runs (19.0%); right outcome 53 (88.3%). By slice: python -m obs reviews --by language
  5. Run the same command on the healthy week: python -m obs slo --data data/telemetry/healthy-week. Its last three lines are:

    Approvals: 112; median wait 32.0 min; 19 waited more than 8 h.
    In staffed hours only (the clock stops at night): decided within 1 h: 112 of 112 (100.0%); median 25.6 min, longest 59.4 min.
    Sampled reviews: 60 of 316 runs (19.0%); right outcome 55 (91.7%). By slice: python -m obs reviews --by language

Read the table one block at a time. Each line is a block of 45 closed runs, in the order they closed. The last block, on Monday 19 October, holds a single run: an approval that closed after the week. A rate over one run means nothing, so skip it. The next lesson comes back to that problem.

The wait p95 h column is the wall clock. Its value is about 15 hours in almost every block, in both weeks. That is not a slow desk: it is the night. Nineteen proposals waited more than 8 hours. Each one arrived after the desk had closed, or in its last ten minutes. Here are the 118 approvals of the week by their wall-clock wait:

Simulated approval times on the replayed week. Blue: waits that also count as under one staffed hour; orange: waits that include a night or a closed weekend morning.
Text description of the chart

Six bars. On the wall clock, 24 approvals waited under 15 minutes, 30 waited 15 to 30 minutes and 34 waited 30 to 60 minutes. Then 11 waited 1 to 8 hours, 8 waited 8 to 15 hours and 11 waited more than 15 hours. On the help desk’s clock, all 118 waits are under one hour: 31 under 15 minutes, 39 from 15 to 30 and 48 from 30 to 60 minutes.

Now read each objective against the output.

  • Right outcome: 53 of 60 sampled reviews passed, 88.3%. The target is 85%, so the objective is met. Seven reviews failed: 78% of the week’s budget of 9. The healthy week did better on its sample, 91.7%.
  • Handoff rate: the blocks of 45 runs reach 42.2% at most, under 45%. But a block cuts the week at fixed places. The alert of the next lesson looks at the last 45 runs after every run. It finds a peak of 53.3% on Saturday evening. The objective is broken on Saturday.
  • Approvals: 118 of 118 were decided within one staffed hour, 100%. Be honest about why: the practice data draws every decision 4 to 60 minutes into staffed time. This objective cannot fail in the course’s data. On a real desk, a sick day or a busy Monday would show here.
  • Cost per resolved ticket: US$0.00328 over the week, and over US$0.002 in every full block from Tuesday evening on. Module 5 found the cause: Omar’s switch to chat-strong for hard tickets.

So the replayed week broke two objectives, the handoff rate and the budget, and met two. The healthy week met all four: its blocks reach at most 33.3% handoffs and US$0.00107 per resolved ticket. Look at the healthy week’s reviews too. German failed there as well, with one failed review of one. Hold that thought for the last lesson of this module.

When to use service levels, and what they cost

Section titled “When to use service levels, and what they cost”

Use an objective for each promise that someone would complain about, and keep them few. Larkfield has four for this workflow. Each one needs an owner who looks at it every week, and who acts when it breaks.

Sam asks: “Why not one number, ‘the agent works: yes or no’?”

Priya answers: “Because the week had two different problems. The budget broke on Wednesday, and the handoffs broke on Saturday. One number would say ‘no’ and nothing else.”

Each indicator has a cost. The right outcome needs people’s time: Priya reviews about 60 runs a week, and the result arrives a day late. The approval wait needs the desk’s hours written in code, and someone must change them when the hours change. Windows counted in runs need enough runs: on a quiet night, 45 runs take many hours. And do not put an objective on the raw numbers that the system happens to have, such as “resolved” or the wall-clock wait. They are easy to count, and that is the only reason they exist.

Do not set an objective for something nobody can change. If Larkfield never staffs nights, an objective on the wall-clock wait would fail every night, and the team would learn to ignore it.

Common mistake

The approval objective fails every night

Symptom: you set “p95 of the approval wait under 1 hour” on the wait p95 h column. It fails in every block of both weeks, including the healthy one.

Cause: the column uses the wall clock. Every proposal made after 17:30 waits until the next morning, so the slowest 5% are always night waits.

Diagnose: compare the healthy week with the replayed week. If an indicator is just as bad on a week with no incident, it measures the plan, not a failure. Then look at the slow runs: python -m obs trace R-1014-040 --data data/telemetry/week shows a proposal at 17:52 and a decision at 09:17.

Recover: count the wait on the help desk’s clock, as obs slo now does on its second-to-last line. If the business promises an answer by the next morning, write that promise as a second objective, with its own clock.

Your turn: the healthy week against the objectives

Section titled “Your turn: the healthy week against the objectives”

Run python -m obs slo --data data/telemetry/healthy-week. For each of the four objectives, write the value, the target and whether it is met. Then answer Grace’s question: “If the healthy week met everything, is the workflow good?” Use the German tickets in your answer. Our answer is in the first knowledge check.

Check your understanding

Your turn: does the healthy week meet the four objectives, and does that make the workflow good?

Show the answer

It meets all four. Right outcome: 55 of 60 sampled reviews passed, 91.7%, against 85%. Handoff rate: the blocks reach 33.3% at most, and the last 45 runs never went above 42.2%, against 45%. Approvals: 112 of 112 were decided within one staffed hour, 100%, against 95%. Cost: US$0.00091 per resolved ticket over the week, and at most US$0.00107 in a block, against US$0.002. But “met” does not mean “good”. The German ticket fails in the healthy week too. Its one review failed, and by the course’s labels, 0 of the 6 German runs were right. The objectives count the whole workflow; a small slice that always fails hides inside a good average. The last lesson of this module looks at slices.

Check your understanding

Predict: why is the 'resolved' share not the task success indicator? Use the German tickets.

Show the answer

“Resolved” only says that the run ended with a reply and no handoff. It does not say whether that was the right end. Larkfield’s rule H5 sends German tickets to a person. The agent resolved 5 of the 6 German tickets of the week, so the German slice shows 83.3% “success” in obs cost --by language. By the course’s labels, only 1 of the 6 was right. A run can end cleanly, with no error and the reply sent, and still be wrong. Success needs a judgement of the outcome, such as a person’s review of a sample of runs.

Check your understanding

T-90104 waited 15 h 25 min for its approval. Did it break the approval objective? Show the numbers.

Show the answer

No. The proposal was made on Wednesday at 17:52, after the desk closed at 17:30. Nobody was there until Thursday at 08:30, and Sam approved it at 09:17. On the help desk’s clock, which counts only staffed hours, it waited 47 minutes, under the one-hour target. The wall-clock wait, 15 h 25 min, measures the night, not the desk. The objective counts approvals decided within one staffed hour, and the replayed week had 118 of 118.

Recap

  • An SLI is a measured share of good events; an SLO is its target over a window. Start from the promise, not from the numbers you happen to have.
  • For an agent, success is the right outcome judged by a person, not “resolved”. A handoff is a promise and a cost. An approval wait is counted on the help desk’s clock. Cost per resolved ticket is a budget with an owner.
  • Larkfield’s four objectives: at least 85% right outcomes in a week of reviews; at most 45% handoffs in the last 45 runs. Also at least 95% of approvals within one staffed hour, and at most US$0.002 per resolved ticket.
  • obs slo reads them from the saved week. The replayed week broke the budget from Tuesday evening and the handoff rate on Saturday; the healthy week met all four.
  • T-90104 waited 15 h 25 min on the wall clock, and 47 minutes on the help desk’s clock.

You can now say, for any day, whether the workflow keeps its promises. But nobody reads a table every hour. When an objective breaks on a Saturday evening, someone must hear about it, with enough to act on. Next, you turn these indicators into alerts that lead to action.

Next: Alerts that lead to action