Sort requests with few labels
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we learned from thousands of Larkfield labels. In this case study, a new assistant starts with no labels at all, in English and French. We label 10 requests per intent, then 50, then all, and we find out when more labels stop paying for themselves.
Case study 3 of 4 · Text Classification and Routing: Send Each Message to the Right PlaceSort requests with few labels
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Start a router with no labels and find out how many it needs. You will compare five kinds of method on real requests in English and French. You will draw the learning curve in both languages, and decide when one more batch of labels stops paying for itself.
The notebook runs every step of this page, with the same code and output, plus two tasks with checks. It needs no account and no key: the LLM’s answers were recorded. After its setup cells install and download what it needs, it runs in about 3 minutes on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study uses most of the course. These lessons matter most:
- Why labels cost time, and how to choose them, from When you have few labels.
- TF-IDF with logistic regression, from Naive Bayes, logistic regression and linear SVM.
- Embeddings, the nearest centre and SetFit, from Few labels with embeddings.
- Zero-shot models, from Zero-shot without an LLM, and an LLM with examples in the prompt, from Few-shot examples and label descriptions.
- The cost table and the review threshold, from The cost of each mistake and Decide, review or send to a person.
What you need on your computer (only for the local route; the notebook needs none of it)
- A new folder with a Python 3.12 virtual environment. The notebook’s first cell installs the packages it needs. This case study does not use or change your
larkrouteproject. - About 1.5 GB of free disk for the packages, the two models and the data. To start again, delete the folder
massive-requeststhat the notebook makes.
Tested on macOS (Apple silicon) with Python 3.12, scikit-learn 1.9.1, sentence-transformers 6.2.0, SetFit 1.2.0, GLiClass 0.1.22 and PyTorch 2.15.0 (CPU), in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux were not tested.
The situation
Section titled “The situation”Inès leads the product team of a small company in Montréal. The company makes a calendar, email and list app. Next month, the app gets a text box: people type what they want, in English or in French, and the app does it. “Wake me up at six” sets an alarm. “Ajoute du lait à la liste” adds milk to a list.
Behind the text box, a router must choose one of 13 actions. Karim, the team’s machine learning engineer, has the requests that a test group typed, but no labels. Sophie, a bilingual support agent, can label requests for a few hours each week. Each label is her time.
Inès asks three questions:
- How many labels do we need before the app can act on its own?
- Do we need labels in French too, or are English labels enough?
- When do we stop labelling?
When the router is not sure, the app asks the user: “Did you mean set an alarm or show your alarms?” That costs a few seconds; a wrong action costs more.
What you will build
Section titled “What you will build”You will build the router and the evidence for each of Inès’s answers. You start with 10 labelled requests per intent, then 50, then all. At each step you compare five kinds of method on the same validation requests and costs. Then you choose a threshold below which the app asks the user, and compute what each batch of labels costs and saves. At the end, you test the router once, run it on new requests, and write it up.
The data
Section titled “The data”The requests come from MASSIVE, a public dataset by Amazon. It has the same 16,521 short requests to a voice assistant in 52 languages, each with its intent: what the person wants done. We use the English and French files.
The notebook downloads Amazon’s archive (40 MB), checks its checksum and keeps the English and French files. Karim keeps the app’s 4 scenarios: alarms, calendar, email and lists.
SCENARIOS = ["alarm", "calendar", "email", "lists"]requests = pd.concat([read("en-US", "en"), read("fr-FR", "fr")], ignore_index=True)10,188 requests in 13 intents, 4 scenarioslanguage en frsplittrain 3570 3570validation 613 613test 911 911Each language has 5,094 requests. We keep MASSIVE’s own split. We choose the methods and the threshold on validation; the test requests stay closed until the end.
First checks
Section titled “First checks”One row is one request, such as “wake me up at nine am on friday”, labelled alarm_set. The French row with the same id is its translation: “réveille-moi à neuf heures du matin le vendredi”. The notebook’s checks:
English and French rows with the same id and the same intent: 100% (each French request is a translation)en: missing texts 0 | repeated texts in train 11 | test texts also in train 7 | median words 7fr: missing texts 0 | repeated texts in train 51 | test texts also in train 29 | median words 8train requests per intent (each language): from 54 (email_addcontact) to 810 (calendar_set)The requests are short: 7 words in English, 8 in French. A few test texts are also in training, so they are easy. calendar_set has 15 times more requests than email_addcontact, the rare intent. We watch its recall.
Karim reads 100 random requests in each language. Most are clear. Some are not:
- “nice” is labelled
calendar_set, and “then tap ok” isalarm_query. These rows carry no intent at all. - “through outlook” (
email_sendemail) became “à travers les perspectives” in French: the translator translated the program’s name. - One French validation row, “créer un événement de calendrier vide pour samedi prochain”, is labelled
calendar_remove. Its English row is “delete day after event”. The translation says the opposite.
We keep these wrong labels, because a real inbox has them too.
Frame the problem
Section titled “Frame the problem”The 13 intents
Section titled “The 13 intents”| Scenario | Intents (training requests per language) |
|---|---|
| Alarm | alarm_set (182), alarm_query (130), alarm_remove (78) |
| Calendar | calendar_set (810), calendar_query (566), calendar_remove (312) |
email_query (418), email_sendemail (354), email_querycontact (127), email_addcontact (54) |
|
| Lists | lists_query (198), lists_createoradd (177), lists_remove (164) |
Each intent also has a plain description, such as “add an event or a reminder to the calendar”, for the zero-shot models and the LLM. They are in data/intents.csv.
What a mistake costs
Section titled “What a mistake costs”These costs are assumptions that Inès and Karim agreed on; your app needs its own.
| Outcome | Cost (US$) | Why |
|---|---|---|
| The app asks the user to choose | 0.10 | A few seconds of the user’s time, and a little annoyance. |
| A wrong action | 0.50 | The user sees the wrong result and types again. |
A wrong action when the user wanted alarm_set or calendar_set |
3.00 | Some users do not notice, and miss the alarm or the meeting. |
The app asks the user to confirm before it deletes anything. So a wrong “remove” costs no more than any other wrong action.
If the app asked the user every time, it would cost US$100 per 1,000 requests: any router must cost less. The notebook scores every method with the course’s routing.py and these costs.
The router
Section titled “The router”Text description of the diagram
A request, in English or in French, goes to the model. The model gives a confidence for each of the 13 intents. If the highest confidence is at least the threshold, 0.69 here, the app acts. Example: “ajoute un rendez-vous chez le dentiste jeudi à 15 h” gets calendar_set with 0.94, so the app adds the appointment. If it is below the threshold, the app asks. Example: “take milk off the shopping list” gets lists_remove with only 0.63, so the app asks “Did you mean …?” and shows the likely actions.
The router has three parts: a model, a number of labels, and a threshold. This case study chooses all three.
The baseline: TF-IDF with 10 labels per intent
Section titled “The baseline: TF-IDF with 10 labels per intent”Sophie labels 10 requests per intent: 130 in each language, about an hour of work in each. The notebook draws them at random, with a fixed seed. The French labels are the translations of the English ones.
Karim trains the course project’s TF-IDF pipeline, one model per language, on its 130 labels.
method labels_per_intent accuracy accuracy_en accuracy_fr total_per_10000 everything to the user 0 0.00 0.000 0.000 100.01 TF-IDF + LR 10 0.69 0.728 0.653 426.2TF-IDF gets 69% of the validation requests right, but its mistakes cost US$426 per 1,000: four times more than asking every time. With 10 labels per intent, it has seen too few words. Its 380 mistakes cost US$522.50, and calendar_set alone makes US$357: it is the most common intent, and a missed reminder costs US$3.
The alternatives
Section titled “The alternatives”Embeddings: nearest centre and logistic regression
Section titled “Embeddings: nearest centre and logistic regression”The course’s multilingual embedding model turns each request into 384 numbers. Two methods learn from the 130 labels on top of them, as in Few labels with embeddings:
- the nearest centre: the average of each intent’s embeddings; a new request goes to the most similar centre;
- logistic regression on the 384 numbers.
SetFit
Section titled “SetFit”SetFit fine-tunes the embedding model itself on pairs of the 130 labelled requests, then trains logistic regression on top. It takes about 30 seconds per language on a laptop CPU.
Zero-shot: no labels at all
Section titled “Zero-shot: no labels at all”A zero-shot model needs no labels, only the 13 descriptions. GLiClass scores each description against the request, as in Zero-shot without an LLM. A hosted LLM (Azure chat-small) reads the 13 intents and their descriptions in its prompt. It answers in JSON with an intent and a confidence.
Few-shot LLM
Section titled “Few-shot LLM”The same LLM, with the 8 most similar of Sophie’s 130 labelled requests in the prompt, as in Few-shot examples and label descriptions. We recorded its answers once, so you replay them with no key.
Ten, fifty or all labels
Section titled “Ten, fifty or all labels”Karim trains each method with 10, 50 and all labels per intent (“all” is 3,570 per language). The table shows validation accuracy, rare intent recall and cost per 1,000 requests. There is no threshold yet: every method acts on every request.
| Labels per intent | Method | Accuracy | English | French | Rare intent recall | US$ per 1,000 |
|---|---|---|---|---|---|---|
| 0 | LLM zero-shot (recorded) | 0.927 | 0.931 | 0.922 | 1.0 | 87.1 |
| 0 | GLiClass zero-shot | 0.396 | 0.445 | 0.347 | 0.4 | 781.0 |
| 10 | LLM few-shot (recorded) | 0.932 | 0.933 | 0.931 | 1.0 | 95.1 |
| 10 | SetFit | 0.840 | 0.861 | 0.819 | 0.9 | 263.5 |
| 10 | TF-IDF + LR | 0.690 | 0.728 | 0.653 | 0.9 | 426.2 |
| 10 | Embeddings + LR | 0.749 | 0.772 | 0.726 | 0.8 | 437.6 |
| 10 | Embeddings + nearest centre | 0.746 | 0.775 | 0.718 | 0.8 | 440.9 |
| 50 | TF-IDF + LR | 0.847 | 0.850 | 0.843 | 1.0 | 188.8 |
| 50 | Embeddings + LR | 0.852 | 0.873 | 0.830 | 0.8 | 200.7 |
| 50 | Embeddings + nearest centre | 0.816 | 0.843 | 0.790 | 0.7 | 250.8 |
| all | TF-IDF + LR | 0.911 | 0.915 | 0.907 | 0.9 | 75.0 |
| all | Embeddings + LR | 0.907 | 0.923 | 0.891 | 0.9 | 97.5 |
| all | Embeddings + nearest centre | 0.836 | 0.861 | 0.811 | 0.7 | 204.3 |
| – | Ask the user every time | – | – | – | – | 100.0 |
The validation set has only 5 email_addcontact requests per language, so the rare intent’s recall moves in steps of 0.1. SetFit varies a little between CPU runs: in the notebook’s run, it got 0.841.
What this tells us.
- The hosted LLM needs no labels here. With the 13 descriptions only, it gets 0.927, more than TF-IDF with all 3,570 labels per language (0.911). On Larkfield’s inbox, it got only 0.765 with the route names only. Here it has plain descriptions of everyday assistant tasks. Also, MASSIVE has been public since 2022, so the LLM may have seen it. Your own users’ requests are new to it.
- The 130 examples add little to the LLM: 0.932.
- With 10 labels, every trained model costs more than asking every time. SetFit is the best, at 0.840. It is worth about 40 labels per intent: logistic regression needs 50 to reach 0.852. On Larkfield’s inbox, SetFit gave nothing (0.630 against 0.632). These requests have more varied words.
- GLiClass fails (0.396). With 13 close intents, it cannot tell “set” from “ask” from “delete”.
- With all labels, TF-IDF is the cheapest trained model: US$75.0 per 1,000.
On Larkfield’s inbox, TF-IDF reached 0.962 with 6,000 labels. Here it reaches 0.911 with 3,570 per language. Real requests are short, varied and sometimes wrongly labelled.
The learning curve in two languages
Section titled “The learning curve in two languages”Think of a child who learns to sort laundry. The first ten socks teach a lot. The hundredth sock teaches almost nothing. A learning curve shows accuracy against the number of labels. It rises fast and then flattens. Where it flattens, more labels buy little.
One draw of 10 labels can be lucky or unlucky, as Few labels with embeddings showed. So the notebook draws 5 sets for each size, from 5 to 100 labels per intent, and plots the mean. “All” is one set.
- Embeddings + logistic regression
- TF-IDF + logistic regression
- Embeddings + nearest centre
Text description of the chart
Three lines over six sizes: 5, 10, 20, 50, 100 and all labels per intent. Embeddings with logistic regression: 69.0%, 75.9%, 83.6%, 87.2%, 90.0% and 92.3%. TF-IDF with logistic regression starts lower and climbs longer: 55.0%, 66.3%, 75.0%, 82.3%, 87.7% and 91.5%. The nearest centre (dashed) starts close to logistic regression, 67.3% and 74.8%, then flattens: 79.0%, 82.9%, 85.1% and 86.1%.
- Embeddings + logistic regression
- TF-IDF + logistic regression
- Embeddings + nearest centre
Text description of the chart
The same three lines for the French requests. Embeddings with logistic regression: 62.3%, 71.4%, 77.8%, 83.8%, 85.9% and 89.1%. TF-IDF with logistic regression: 53.6%, 63.0%, 73.3%, 81.4%, 86.2% and 90.7%, so it catches up at 100 labels per intent and passes the embeddings with all labels. The nearest centre: 61.2%, 69.2%, 73.5%, 78.6%, 79.9% and 81.1%.
What this tells us.
- Embeddings win with few labels. With 10 labels per intent: 0.759 in English, against 0.663 for TF-IDF.
- TF-IDF catches up later than on Larkfield, where it passed the embeddings at 64 labels per route. In English, the embeddings still lead with all labels. In French, TF-IDF catches up at 100 per intent and passes with all labels (0.907 against 0.891).
- French is a few points lower for every method: the model knows English better, and some translations are wrong.
- The embeddings’ curve flattens; TF-IDF’s does not yet. In English, embeddings + LR gains 11.3 points from 10 to 50 labels and 2.8 from 50 to 100. From 100 to all (about 275 per intent), it gains 2.3. TF-IDF still gains 9.2 points from 50 to all.
Do we need French labels?
Section titled “Do we need French labels?”Sophie’s French labels cost as much as her English ones. Karim tries something cheaper: train on English labels only, and route the French requests.
This works only if the model puts a French request close to its English meaning. A multilingual embedding model learned from many translated sentences, so it does. TF-IDF only counts words, and English and French share almost none.
Text description of the diagram
Two inputs: an English request with its label, “wake me up at nine am on friday”, alarm_set, and a French request, “réveille-moi à neuf heures du matin le vendredi”. With the course’s embedding model, their cosine similarity is 0.92, so a model trained on the English label sends the French request to alarm_set. For comparison, “wake me up at nine am on friday” and “what alarms do i have” have a cosine similarity of only 0.35. With TF-IDF, the two requests share no word, so the English label does not help.
This is your first task in the notebook. The French validation accuracy:
| Labels per intent | Method | Trained on French labels | Trained on English labels only |
|---|---|---|---|
| 10 | TF-IDF + LR | 0.653 | 0.418 |
| 10 | Embeddings + LR | 0.726 | 0.737 |
| 50 | TF-IDF + LR | 0.843 | 0.413 |
| 50 | Embeddings + LR | 0.830 | 0.822 |
| all | TF-IDF + LR | 0.907 | 0.506 |
| all | Embeddings + LR | 0.891 | 0.869 |
With embeddings, English labels are almost as good as French ones: 0.737 against 0.726 with 10 labels per intent, and 0.869 against 0.891 with all. TF-IDF trained on English falls to 0.41–0.51. So: label only in English and use embeddings, or label both languages and use TF-IDF.
The French here is translated from English. Real typed French is less close, so test with your own French requests first.
A fair comparison under one measure
Section titled “A fair comparison under one measure”So far, every method acted on every request. But the app can ask when it is not sure. For each candidate, the notebook tries every threshold from 0.00 to 0.99 on the validation requests and keeps the cheapest, as in Decide, review or send to a person. The cost now includes the questions.
| Method | Labels per intent | Cheapest threshold | Asks the user | US$ per 1,000 (with the questions) |
|---|---|---|---|---|
| TF-IDF + LR | all | 0.69 | 15.0% | 38.3 |
| Embeddings + LR | all | 0.83 | 25.4% | 46.2 |
| LLM few-shot (recorded) | 10 | 0.98 | 34.1% | 47.7 |
| LLM zero-shot (recorded) | 0 | 0.92 | 17.4% | 50.8 |
| TF-IDF + LR | 50 | 0.81 | 53.4% | 62.8 |
| Embeddings + LR | 50 | 0.81 | 54.4% | 71.1 |
| Embeddings + LR | 10 | 0.66 | 71.5% | 75.5 |
| SetFit | 10 | 0.89 | 69.4% | 77.6 |
What this tells us.
- A threshold makes every candidate cheaper than asking every time. Even the models with 10 labels go below US$100. But they ask about 70% of the time: more annoying than helpful.
- The hosted LLM is a good router from the first day. With no labels, it asks 17.4% of the time and costs US$50.8 per 1,000. Its confidence is useful only above 0.92: it writes 0.95 or more for most answers.
- TF-IDF with all labels is the cheapest: US$38.3 per 1,000, asking 15.0% of the time. It runs on the app’s server in half a millisecond. The LLM takes about 1.3 seconds and sends every request to a hosted service.
Ask the user when unsure
Section titled “Ask the user when unsure”The threshold works like a new shop assistant. When she is sure, she helps at once. When she is not, she asks a short question. Too many questions tire customers; no questions bring costly mistakes. The chart shows this for TF-IDF with all labels.
- Cost per 1,000 requests (US$)
- Share of requests the app asks about (%)
Text description of the chart
Two lines against the threshold, from 0.30 to 0.99. The share of requests the app asks about rises from 1.0% at 0.30 to 74.1% at 0.99. The cost per 1,000 requests falls from US$68.3 at 0.30 to its lowest point, US$38.3 at 0.69, where the app asks about 15.0% of the requests. Above that, the questions cost more than the mistakes they prevent: US$40.5 at 0.80, US$47.0 at 0.90 and US$76.5 at 0.99.
Your turn (the second task in the notebook): the product team accepts that the app asks at most 1 time in 10. Under that limit, the best threshold is 0.59: the app asks 9.3% of the time, for US$44.0 per 1,000 on validation. The limit costs US$5.7 per 1,000.
When to stop labelling
Section titled “When to stop labelling”Labels cost money once. A better router saves money every month. So we stop labelling when the next batch takes too long to pay for itself.
Two more assumptions. Sophie labels about 120 of these short requests an hour, at US$28 an hour: about US$0.23 per label. The app gets 20,000 requests a month in each language.
For each number of labels, the table takes the cheapest model that runs on the app’s server, with its threshold. It computes the extra label cost in both languages, the router’s cost per month, and the months of savings that pay for the extra labels.
| Labels per intent | Best local model | Labels per language | Label cost, both languages (US$) | US$ per 1,000 | Router cost per month (US$) | Extra labels (US$) | Saving per month (US$) | Months to pay back |
|---|---|---|---|---|---|---|---|---|
| 10 | Embeddings + LR | 130 | 61 | 75.5 | 3,020 | – | – | – |
| 50 | TF-IDF + LR | 650 | 303 | 62.8 | 2,512 | 242 | 508 | 0.5 |
| all | TF-IDF + LR | 3,570 | 1,666 | 38.3 | 1,532 | 1,363 | 980 | 1.4 |
For comparison: asking every time costs US$4,000 a month, and the few-shot LLM with its threshold US$1,908 a month.
What this tells us.
- Every step paid back fast: from 10 to 50 labels per intent in half a month, from 50 to all in 1.4 months.
- We have not reached the point to stop. TF-IDF’s curve was still rising. With real users, Karim keeps labelling while a batch pays back within a few months. He stops when it takes longer than the app’s planning horizon.
- The LLM changes the start, not the end. Without labels, it is far cheaper than asking every time. With all labels, local TF-IDF costs less and keeps the requests on the app’s server.
The recommendation
Section titled “The recommendation”Inès’s three answers, from the validation requests:
- How many labels before the app acts alone? With a local model, 50 per intent still means asking about half the time; all 3,570 per language, 15% of the time. With the hosted LLM, none.
- French labels? Only if you use TF-IDF. With multilingual embeddings, English labels reach 0.869 on French against 0.891 with French labels. TF-IDF with both languages is cheaper, so Karim labels both.
- When to stop? Not yet: the last batch paid back in 1.4 months.
The plan:
- At launch: the zero-shot LLM, ask below 0.92, if the privacy notice and the provider’s contract allow it. If not, the app asks every time while Sophie labels.
- The router, once Sophie has labelled both languages: TF-IDF with all labels, one model per language, ask below 0.69. On validation it asks 15.0% of the time and costs US$38.3 per 1,000, about US$1,532 a month. It replaces the LLM: no request leaves the app, and an answer takes half a millisecond.
The test, once
Section titled “The test, once”Now Karim opens the test requests: 911 per language, never used before. He scores every candidate once, with and without its threshold. “Right without asking” counts the requests the app handles correctly alone; “right when it acts” leaves out the requests it asks about.
| Router | Right without asking | Right when it acts | English | French | Asks the user | US$ per 1,000 |
|---|---|---|---|---|---|---|
| TF-IDF, all labels, ask below 0.69 | 0.814 | 0.955 | 0.823 | 0.806 | 14.8% | 50.2 |
| Embeddings + LR, all labels, ask below 0.83 | 0.692 | 0.971 | 0.729 | 0.654 | 28.7% | 50.4 |
| LLM few-shot, ask below 0.98 | 0.623 | 0.983 | 0.641 | 0.606 | 36.6% | 55.8 |
| LLM zero-shot, ask below 0.92 | 0.784 | 0.955 | 0.789 | 0.779 | 17.9% | 77.3 |
| TF-IDF, all labels, never asks | 0.900 | 0.900 | 0.905 | 0.896 | 0% | 93.9 |
| Ask the user every time | 0 | – | – | – | 100% | 100.0 |
| LLM zero-shot, never asks | 0.919 | 0.919 | 0.929 | 0.910 | 0.4% | 103.1 |
| LLM few-shot, never asks | 0.929 | 0.929 | 0.932 | 0.926 | 0.1% | 109.3 |
What this tells us.
- The chosen router holds: US$50.2 per 1,000, asking 14.8% of the time, and right 95.5% of the time when it acts. It is the cheapest row on test too.
- Test costs more than validation for almost every row (US$50.2 against US$38.3 for the chosen router). Thresholds chosen on 1,226 requests carry noise, as Decide, review or send to a person showed.
- The LLM moved the most. Without a threshold, the zero-shot LLM costs US$103.1 on test against US$87.1 on validation, and US$77.3 with its threshold of 0.92. Its written confidence is a weaker guide than TF-IDF’s probabilities.
- The most accurate router is not the cheapest. The few-shot LLM gets 0.929 right, but costs US$109.3 per 1,000: twice the chosen router.
An end-to-end run on new requests
Section titled “An end-to-end run on new requests”Karim writes six new requests, three in English and three in French. These are constructed examples, not MASSIVE rows. The chosen router acts or asks.
| Language | Request | Intent | Confidence | Decision |
|---|---|---|---|---|
| en | wake me at 6 tomorrow, I have a flight | alarm_set |
0.57 | ask the user |
| en | did Sam answer my message about the budget | email_sendemail |
0.44 | ask the user |
| en | take milk off the shopping list | lists_remove |
0.63 | ask the user |
| fr | ajoute un rendez-vous chez le dentiste jeudi à 15 h | calendar_set |
0.94 | do: calendar_set |
| fr | est-ce que j’ai une alarme pour demain matin | alarm_query |
0.72 | do: alarm_query |
| fr | envoie un courriel à Anna pour dire que je serai en retard | email_sendemail |
0.98 | do: email_sendemail |
The three French requests get the right intent with enough confidence, so the app acts. Two English requests get the right intent but stay below 0.69, so the app asks. “did Sam answer my message” is wrong: it asks about email (email_query), but the model says email_sendemail at 0.44. The threshold catches it. Our English requests have digits and punctuation (“6”, a comma), which MASSIVE’s English lacks. That may explain the low confidences.
Common mistake
Balanced labels, unbalanced requests
Symptom: with 10 labels per intent, the models send many calendar_set requests to calendar_query. With embeddings + LR, 87 of the 1,226 validation requests make exactly this mistake. Each one costs US$3: a reminder that was never set.
Cause: Sophie labelled 10 requests for every intent, so the model saw calendar_set as often as email_addcontact. In the real requests, calendar_set is 15 times more common. Requests such as “remind me about the party tomorrow” sit between “set” and “ask”, and the model has no reason to prefer the common intent.
Fix: after the first balanced round, label requests at random, so that common intents get more labels. Or use active learning, which picks the requests the model is least sure about, as in When you have few labels. Watch the costly intents in the confusion table, not only the accuracy.
Consequence in AI work: “10 per class” is a fair start, but the model inherits its balance. It can be wrong most often on the intents that users send most.
Insights and limits
Section titled “Insights and limits”- The best method depends on the number of labels. With none, the hosted LLM; with 10, SetFit; with all, TF-IDF. Draw the curve on your own data first.
- Labels paid back in weeks here, because 40,000 requests a month multiply every saving. With 1,000 requests a month, the same labels would take years to pay back, and the LLM would stay.
- These scores are this data’s scores. Crowd workers wrote the requests, the French is translated, some labels are wrong, and the LLM may have seen MASSIVE. The content filter refused 10 of the 3,048 zero-shot requests, all French and harmless, such as “retire l’alarme de neuf heures du matin de mardi”. We count a refusal as “ask the user”.
- The costs are assumptions. With other costs, the threshold and the stopping point move. Change
COSTS,LABEL_COSTandMONTHLYand run again.
Responsible use
Section titled “Responsible use”- Ask before you act on a guess, and confirm before any action that deletes or sends something. A wrong “send email” cannot be undone.
- Requests are personal data: names, appointments, email contents. Label them only with the users’ agreement, keep the labelled set protected, and remove names before you share examples.
- A hosted LLM sees every request. Use it only if your privacy notice and your provider contract allow it.
- Give credit. MASSIVE is CC BY 4.0: name Amazon and SLURP’s authors, link the licence, and say what you changed, as the notebook’s
dataset.mddoes. - Do not claim more than you measured. Report the French results as results on translated requests, and the scores as scores on this data.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can start a text classifier with almost no labels and plan the labelling with numbers. Make it yours with one change: another MASSIVE scenario (such as play or iot), another language pair, other costs, or active learning. Make it easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
learning_curve.png: the curve in both languages, from the notebook’s plot.labelling_plan.md: the costs and the volume you assumed, the table of section “When to stop labelling”, and your decision.- The data source and licence, with the attribution text from the box above. A link and the checksums are enough.
A README template:
# How many labels does an English and French assistant need?
## ProblemAn app routes typed requests to 13 actions (alarms, calendar, email, lists), in Englishand French. A wrong action costs 0.50 dollars (3 for a missed alarm or event); askingthe user costs 0.10. Labels cost 0.23 dollars each. How many labels, which method?
## Data and licenceMASSIVE 1.1 (Amazon, CC BY 4.0; built from SLURP, CC BY 4.0), en-US and fr-FR,scenarios alarm, calendar, email, lists: 3,570 / 613 / 911 requests per language.
## Approach10, 50 and all labels per intent; TF-IDF, embeddings (nearest centre, LR), SetFit,GLiClass zero-shot, LLM zero-shot and few-shot (recorded). Learning curve with 5 draws.English labels tested on French. Threshold for "ask the user" chosen on validation.
## Results (test set, used once)Router: TF-IDF + logistic regression, all labels, one model per language, ask the userbelow 0.69. Asks 14.8%, right 95.5% when it acts, 50.2 dollars per 1,000 requests(asking every time: 100). The zero-shot LLM (no labels) cost 77.3 with its threshold.English labels alone reach 0.869 on French with embeddings (0.891 with French labels).
## LimitsCrowd-written requests, translated French, some wrong labels, assumed costs.How to describe it in an interview, in two sentences: “I measured how many labels a bilingual assistant needs, and found that a hosted LLM needed none while a local TF-IDF model needed all 3,570 per language to be cheaper. I chose the threshold below which the app asks the user, and showed that each batch of labels paid for itself within two months.”
Go further:
- Replace the random draws after the first 10 per intent with active learning, as in When you have few labels. Does it reach the same accuracy with fewer labels?
- Train SetFit with 50 labels per intent. Predict first: does it still beat logistic regression by as much?
- Let the LLM label 1,000 training requests and check a sample as Sophie would. Then train TF-IDF on them, as in Distillation: let the LLM teach a cheap model.
- Use the few-shot LLM only for the requests that the cheap model is unsure about, as in Cascades.
Knowledge check
Section titled “Knowledge check”Check your understanding
With 10 labels per intent, TF-IDF gets 69% of the validation requests right, but it costs US$426 per 1,000, against US$100 for asking the user every time. How can a model that is right most of the time cost more than asking every time?
Show the answer
Accuracy counts every mistake the same; the cost does not. TF-IDF makes 380 mistakes on the 1,226 validation requests. Each costs at least US$0.50, and US$3 for alarm_set or calendar_set: calendar_set alone makes US$357 of the US$522.50. That is US$426 per 1,000. Asking every time costs US$0.10 per request, US$100 per 1,000. A model is worth using only when its mistakes cost less than the questions it saves.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Inès wants to save money and label only English requests. Using the numbers of this case study, say when that works, which method it needs, and what you would check before you rely on it.
Show the answer
It works with a multilingual embedding model. Trained on English labels, embeddings + LR reach 0.737 on French with 10 labels per intent (0.726 with French labels). With all labels, they reach 0.869 (0.891). It does not work with TF-IDF, which falls to 0.41–0.51, because English and French share almost no words. Before you rely on it, check it on French requests from your own users: MASSIVE’s French is translated, so these numbers are optimistic. Also compare the cost: here TF-IDF with labels in both languages was the cheapest router.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The labelling table says that going from 50 labels per intent to all labels pays for itself in a number of months. Name two assumptions that change this number, and say in which direction each one moves it.
Show the answer
The number of requests a month: the saving grows with the volume, so more requests mean a faster payback. With 1,000 requests a month instead of 40,000, the 1.4 months become more than four years. The price of a label: a higher price (longer requests, a more expensive labeller) means a slower payback. The costs of a mistake and of a question also matter: if a wrong action costs more, the better router saves more and pays back faster.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- On this data, a hosted LLM with a plain description of each intent got 0.927 right with no labels. The trained models needed all 3,570 labels per language to come close.
- With few labels, embeddings win; SetFit with 10 labels per intent matched logistic regression with 50. TF-IDF catches up later than on Larkfield, and was still climbing with all labels.
- A learning curve (with several draws) shows where labels stop helping. With a multilingual model, English labels reached 0.869 on French against 0.891 with French labels; TF-IDF cannot do this.
- A threshold made every candidate cheaper than asking every time. The chosen router, TF-IDF with all labels and “ask below 0.69”, cost US$50.2 per 1,000 on test and asked 14.8% of the time.
- Stop labelling when the next batch takes too long to pay back. Here, the last batch paid back in 1.4 months: keep going.
Next, the last case study gives each research paper several tags at once, from many possible labels.