Learn from corrections
- Lesson 34
- 30 min
- Module 8: Confidence and people in the loop
Where we are
People now review the unsure messages, and they move some messages to another queue. Each move is a new label. In this lesson, we record corrections as labels, and we check them before we train on them, because a quick move is not always a correct one.
Lesson 34 of 37 · Text Classification and Routing: Send Each Message to the Right PlaceLearn from corrections
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Turn the queue moves that agents make into new labels, and check them before the model learns from them. A move made in a few seconds, or a move to a route that the model finds very unlikely, gets a second look first.
NotebookDo this lesson in a notebook
The notebook reads the 300 queue moves and gives the model’s probability for each new route. It applies the two checks and keeps the moves that pass as labels, with checks. It installs pandas, scikit-learn and matplotlib and downloads the data. About 20 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The three zones and the review work, from Decide, review or send to a person.
- Why wrong labels hurt a model, and how a model can point to them, from Find noisy labels.
What you need on your computer
Your larkroute project with the files of the last two lessons (27 tests pass), or the notebook above.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows commands were not tested.
The logbook at the check desk
Section titled “The logbook at the check desk”Back at the post office. The clerk at the check desk sometimes moves a letter to another tray. Then she writes it in a logbook: the letter, the old tray, the new tray. At the end of the month, the manager uses the logbook to teach new sorters. “Look: letters like this one go to tray 4, not tray 2.”
The logbook is a gift. Nobody had to label those letters on purpose. But one busy clerk moves letters after a two-second glance. Some of her moves are wrong. If the new sorters learn from her lines in the logbook, they learn her mistakes too, and they repeat them every day.
So the manager reads some lines again before teaching. Not all of them: only the lines that look hasty or strange.
The problem: a free label can be a wrong label
Section titled “The problem: a free label can be a wrong label”At Larkfield, agents move messages between queues all day. In the launch month, they made 300 queue moves. Each move says “the right route for this message is the new queue”. That is a correction: a label that came from the work itself.
Here is one of them. A customer wrote:
Where is my refund? I returned the rug weeks ago. Also, can I return the throw if I don’t like the colour?
An agent moved it from billing to orders, in 3 seconds. The guidelines say that a refund after a return belongs to returns_refunds. The model gives orders a probability of 0.005 for this message, and returns_refunds 0.823. If this move becomes a training label, the next model learns that a refund question can mean orders.
The moves in this lesson are a practice problem we made on purpose. We planted wrong moves, and we know which ones they are. That answer key stays hidden from the data, as it would in real work. The lesson tells you what it says.
The idea: record every move, check some before training
Section titled “The idea: record every move, check some before training”Two steps, in this order.
- Record each move as a label, with its source. Keep the message, the old route, the new route, the agent and the time spent. The time is free to record, and it turns out to matter. Keep the source too (“agent move”), so that you can count, compare and remove these labels later.
- Check the moves that look risky before they join the training data. The project uses two cheap signals:
- quick: the agent spent under 10 seconds. That is too fast to read most messages.
- unlikely: the model gives the new route a probability under 0.05. The model and the agent disagree strongly.
A flagged move goes to a second person. A move that passes both checks becomes a label at once.
Text description of the diagram
A loop of seven boxes. Step 1: the model routes the message. Step 2: an agent moves it to another queue. Step 3: record the move, with the old queue, the new queue, the agent and the seconds spent. Step 4, highlighted: check the move. Was it quick, under 10 seconds? Is it unlikely for the model, under 0.05? An arrow labelled yes, 87, leads to step 5: a second person checks the move. A dashed arrow labelled no, 213 of 300, goes straight to step 6: the checked moves become labels. Step 7: train, test and release the next version. An arrow leads from step 7 back to step 1.
Step 7 is Module 9’s work. A new label never goes straight into the running model. It goes into the next version, which must pass the same tests as the old one.
A worked example: five moves
Section titled “A worked example: five moves”Five real moves from the practice data, with the model’s probability for the new route. The last column comes from the hidden answer key.
| Message (shortened) | Move | Seconds | Model’s probability for the new route | Flagged? | Answer key |
|---|---|---|---|---|---|
| “Where is my refund? I returned the rug weeks ago…” | billing → orders | 3 | 0.005 | yes: quick and unlikely | wrong |
| “I’m following up about the issue with my gas grill…” | delivery → returns_refunds | 6 | 0.721 | yes: quick | right |
| “Could you please cancel order LK-438326?” | billing → account | 23 | 0.000 | yes: unlikely | wrong |
| “…see my last email. You received the return on Nov 09…” | billing → returns_refunds | 22 | 0.982 | no | right |
| “Could you please update my phone number…” | needs_person → account | 25 | 0.995 | no | right |
Three moves are flagged. Two of them are wrong, and the checks catch both. The second move is right, and a second person checks it for nothing. That is the price of the quick check. The third move took 23 seconds, so the agent did read the message. Only the model’s doubt catches it: a request to cancel an order is not an account question.
Run the check on Larkfield’s moves
Section titled “Run the check on Larkfield’s moves”-
In your project folder, with the virtual environment on, copy the last files of the stage from the snapshot
end-of-m08. They are the command, its test and the README. On macOS and Linux:Terminal window S=../nextia-learning-labs/C36/snapshots/end-of-m08cp $S/larkroute/commands/corrections.py larkroute/commands/cp $S/tests/test_corrections.py tests/cp $S/README.md .On Windows (PowerShell):
Terminal window $S = "..\nextia-learning-labs\C36\snapshots\end-of-m08"Copy-Item $S\larkroute\commands\corrections.py larkroute\commands\Copy-Item $S\tests\test_corrections.py tests\Copy-Item $S\README.md . -
Run the tests. You see
28 passed. Your project now matches the snapshotend-of-m08.Terminal window python -m pytest -q -
Run the check. It trains the model, reads
data/corrections.csvand the launch-month messages, and takes about 3 seconds.Terminal window route corrections
300 queue moves by 12 agents; 76 made in less than 10 seconds. To check before training: 87 (29%).to_routeaccount 47billing 35delivery 68needs_person 17orders 50product 42returns_refunds 41Of the 300 moves, 87 go to a second person. That is 29%, about 3 in 10. The other 213 can become labels now. The table counts the moves by their new route.
To get the list of moves to check, add --write:
route corrections --writeThe output is the same, with one more line: Written: queue/corrections_to_check.csv. The file has one row per flagged move, in the columns of corrections.csv:
message_id,from_route,to_route,agent,secondsM-200042,delivery,returns_refunds,A-07,6M-200047,delivery,product,A-03,5M-200070,returns_refunds,orders,A-02,7route reset deletes the folder queue/, as it deletes models/ and runs/. Keep a copy elsewhere if you need it.
What the check finds
Section titled “What the check finds”The answer key of the practice data says which moves are wrong: 39 of the 300. Here is how they split by the two checks.
Text description of the diagram
A grid of four cells. The columns are the time: under 10 seconds, or 10 seconds or more. The rows are the model’s view: it gives the new route a probability under 0.05, or it finds the new route likely. Quick and unlikely: 26 moves, all 26 wrong. Slow but unlikely: 11 moves, all 11 wrong. Quick but likely: 50 moves, 2 wrong. Neither quick nor unlikely: 213 moves, 0 wrong. The three flagged cells have a blue border. A note says: flagged, 87 moves, 39 wrong and 48 right; not flagged, 213 moves, 0 wrong; all 39 wrong moves are flagged.
Text description of the chart
A column chart with two columns, on an axis from 0% to 50%. Moves made in under 10 seconds: 36.8% wrong, 28 of 76. Moves made in 10 seconds or more: 4.9% wrong, 11 of 224. A wrong move took 7 seconds in the middle (the median); a right move took 24 seconds.
Three things to see:
- Speed matters. Of the 76 quick moves, 36.8% are wrong (28). Of the 224 slower moves, only 4.9% are (11). A wrong move took 7 seconds in the middle (the median); a right one took 24.
- The two checks together find all 39 wrong moves. The check flags 87 moves, so a second person reads 48 right moves for nothing. That is the price of finding every wrong one, here.
- Neither check is enough alone. The quick check misses the 11 slow wrong moves. The model check misses 2 quick wrong moves that it finds likely.
Be careful with “finds all 39”
Section titled “Be careful with “finds all 39””This result is better than you should expect on real data. In the practice data, a wrong move went to a route with no link to the message. For example, a cancel request was moved to account. The model sees that easily: 37 of the 39 wrong moves are “unlikely” for it. In a real inbox, many wrong moves go to a nearby route, like billing instead of returns_refunds for a refund. The model gives those a fair probability, so the model check catches fewer of them. The quick check does not depend on the model, so it still helps.
Also, the model check uses the model that you want to improve. If the model is wrong about a new kind of message, it flags the agents’ right moves, and you may keep its own mistake. That is why a flagged move goes to a person, not to the bin. In Module 9, the launch month brings a whole new topic, and the model’s doubt is the first sign of it.
Use corrections well
Section titled “Use corrections well”Four rules for any team that learns from its agents’ moves:
- Record the move, not only the result. Old route, new route, agent, seconds. Without the seconds, the quick check is impossible.
- Keep the source of every label. A move that passed the checks is still not the same as a label that Chloé wrote with the guidelines open. You can then compare the two, or remove one kind if it causes trouble.
- Do not test the model on corrections only. Moves exist only for messages that someone looked at and disagreed with. They are a strange sample: mostly unsure or wrong decisions. Keep the test set from Meet the inbox for the scores.
- Use the checks to help agents, not to blame them. A high share of quick, wrong moves often means too much work, or a guideline that is unclear. Grace fixes those; the model cannot.
Your turn
Section titled “Your turn”Section 4 of the notebook asks you to keep the moves that are not flagged, as a table labels. It has three columns: message_id, text and route (the agent’s new route). Predict first: how many rows?
Show the answer
labels = (moves.loc[~moves["to_check"], ["message_id", "text", "to_route"]] .rename(columns={"to_route": "route"}).reset_index(drop=True))The check cell prints Check passed: the number of rows in labels is 213. That is 300 moves minus the 87 flagged. By the answer key, none of the 213 is wrong. Section 5 then adds a column source with the value agent move, not flagged.
Common mistake
Every move becomes a label
Symptom: Tomás writes a nightly job: “Each queue move goes into the training data. The agents are the experts, so their moves are the truth.” The refund question above, moved to orders in 3 seconds, is now a training label.
Cause: 39 of the 300 moves are wrong, and 28 of those were made in under 10 seconds. The job treats a 3-second glance like a careful review. The labels from quick moves are wrong 36.8% of the time. That is worse than the old queue of Module 2, where 11.2% of the labels were wrong.
Fix: record the seconds and the agent with each move. Flag quick moves and moves that the model finds very unlikely, as route corrections does, and let a second person check them. Keep the source of each label.
Consequence in AI work: a model that learns from unchecked feedback repeats the fastest agent’s habits. Its mistakes come back as new moves, and the loop makes them stronger with each version.
Knowledge check
Section titled “Knowledge check”Check your understanding
Name the two checks that route corrections uses, and say which wrong moves each one misses in Larkfield's 300 moves.
Show the answer
Quick: the agent spent under 10 seconds. Unlikely: the model gives the new route a probability under 0.05. The quick check misses the 11 wrong moves that took 10 seconds or more. The model check misses 2 quick wrong moves that it finds likely. Together they flag 87 moves and find all 39 wrong ones. The moves are constructed, so a real inbox will be harder.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The check flags 87 moves and finds all 39 wrong ones. Why should you not expect the same on real data?
Show the answer
In the practice data, a wrong move goes to a route with no link to the message. So the model finds 37 of the 39 wrong moves “unlikely”. Real wrong moves often go to a nearby route, like billing instead of returns_refunds, which the model finds fairly likely. The model check also depends on the model that you want to improve: it can flag right moves on a new topic. The quick check does not depend on the model, so it still helps.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why should a flagged move go to a second person instead of being deleted, and why keep the source of every label?
Show the answer
A flag is a doubt, not a verdict: 48 of the 87 flagged moves are right. Some of them teach the model something new, for example a new topic that it does not know yet. Deleting them would throw away exactly those labels. The source lets you count and compare labels from agents’ moves with labels from careful review, and remove one kind later if it causes trouble.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A correction is a queue move by an agent. It is a free label, and it can be wrong.
- Record the message, the old and new routes, the agent and the seconds. Keep the source of every label.
- Check before training: a second person checks a move made in under 10 seconds. The same goes for a move to a route that the model finds very unlikely (under 0.05).
- On Larkfield’s 300 constructed moves,
route correctionsflags 87 (29%). The answer key says 39 moves are wrong. Of the 76 quick moves, 36.8% are wrong; of the slower ones, 4.9%. The check finds all 39. - That last result is easier than real data: the planted wrong moves go to unrelated routes. A real inbox has wrong moves to nearby routes.
- The project is now at the snapshot
end-of-m08:route calibration,route zonesandroute corrections, with 28 tests.
You can now trust the confidence, choose the zones, and grow the labels from the team’s own work. Next, Amira turns the router into a small service that answers all day, with a model version, a review queue and a log.
Next: A routing service