Skip to content
Français

Find what the bot cannot handle

  • Case study 2
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, we sent unsure messages to a person. In this case study, we use real open data: an intent dataset with requests that no intent covers. We detect them, send them to a person, and count the wrong automatic answers this prevents.

Case study 2 of 4 · Text Classification and Routing: Send Each Message to the Right PlaceFind what the bot cannot handle

What you will be able to do

Teach a virtual assistant to say “I cannot help with that, here is a person” at the right moments. You will compare three ways to detect requests that the assistant was not built for, and measure what each way costs. Then you choose the threshold from the price of a wrong answer and the price of a hand-off.

Case studyPortfolio piece

Licence
CC BY 3.0
Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Out-of-scope detection
  • Cost-based threshold
  • Calibration check
  • Sentence embeddings
  • Nearest-example rule
  • Replayed LLM answers

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, plus two tasks with checks. Its setup cells install the packages and download the data (about 2.5 MB) from the dataset’s own repository. The embedding model (about 470 MB) downloads once. It needs no account and no key. It runs in about 2 minutes on a recent laptop CPU; free Colab is slower.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it locally, open it in Jupyter with Python 3.12. Its setup cell works in a new folder clinc-oos; to start again, delete that folder. This case study does not change your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, sentence-transformers 6.2.0 and PyTorch 2.15.0 (CPU), in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux were not tested.

At the post office, now and then a letter arrives for a street that does not exist here. A good sorter does not push it into the nearest tray. She puts it on the desk of the person who handles odd letters.

Larkfield plans a chat assistant on its website for simple requests, such as “where is my order”. Grace, the support lead, has one worry. “Customers will ask it things it was never built for. If it answers those anyway, it gives a wrong answer with a confident face. I would rather it says I cannot help with that, here is a person.”

Tomás has an answer ready: “An LLM knows when a question is strange. Let it decide.” Priya, the data scientist, wants to measure it first. The assistant has no real chat log yet, so she uses a public dataset made for this question.

A request that none of the assistant’s intents covers is out of scope. An intent is one thing the assistant knows how to do, such as “check the balance”. The question of this case study is: when should the bot hand a request to a person?

You will build an out-of-scope detector for an assistant with 150 intents. You compare three ways:

  1. a confidence threshold on a classifier: hand off when the model is not sure enough;
  2. a distance to the nearest example: hand off when no training request is similar enough;
  3. an explicit out-of-scope class: the classifier learns “none of the above” from examples.

You measure two numbers together. In-scope accuracy is the share of in-scope requests answered right. Out-of-scope recall is the share of out-of-scope requests handed to a person. You choose the threshold from two prices, a wrong answer and a hand-off, and replay a hosted LLM’s recorded answers for Tomás.

The data is CLINC150. It has 23,700 short requests to an imagined virtual assistant. Crowd workers wrote them for 150 intents in 10 domains, such as banking, travel and small talk. They also wrote 1,200 out-of-scope requests that fit none of the 150 intents.

The file has six lists of [text, intent] pairs. Priya reads them into three tables and looks at their size:

import json
raw = json.load(open("data/data_full.json", encoding="utf-8"))
DOMAIN_OF = {i: d for d, intents in json.load(open("data/domains.json")).items() for i in intents}
def part(*keys):
"""One part of CLINC150 as a table. The out-of-scope requests have the intent "oos"."""
rows = [(text, intent) for k in keys for text, intent in raw[k]]
df = pd.DataFrame(rows, columns=["text", "intent"])
df["domain"] = df.intent.map(DOMAIN_OF).fillna("out of scope")
return df
train, validation, test = part("train", "oos_train"), part("val", "oos_val"), part("test", "oos_test")
# CLINC150 has no request IDs: we number the rows in the order of the file.
for prefix, df in [("TR", train), ("VA", validation), ("TE", test)]:
df.insert(0, "message_id", [f"C-{prefix}-{i:05d}" for i in range(1, len(df) + 1)])
for name, df in [("train", train), ("validation", validation), ("test", test)]:
oos = (df.intent == "oos").sum()
print(f"{name:10}: {len(df):6,} requests, {df.intent.nunique() - 1} intents, out of scope {oos:5,} ({oos / len(df):.1%})")
print("domains:", sorted(set(DOMAIN_OF.values())))

The output:

train : 15,100 requests, 150 intents, out of scope 100 (0.7%)
validation: 3,100 requests, 150 intents, out of scope 100 (3.2%)
test : 5,500 requests, 150 intents, out of scope 1,000 (18.2%)
domains: ['auto_and_commute', 'banking', 'credit_cards', 'home', 'kitchen_and_dining', 'meta', 'small_talk', 'travel', 'utility', 'work']

What this tells us. One row is one request with its intent, or oos for out of scope. The share of out-of-scope requests differs in each part: 0.7% in train, 3.2% in validation, 18.2% in test. The authors chose few examples to learn from and many to test on.

words = train.text.str.split().str.len()
print(f"words per request: median {words.median():.0f}, 90% have at most {words.quantile(0.9):.0f}, longest {words.max()}")
print("missing or empty texts:", int((train.text.str.strip() == "").sum() + train.text.isna().sum()))
print("repeated requests in train:", int(train.text.duplicated().sum()),
"| test requests also in train:", int(test.text.isin(train.text).sum()))
print("in-scope requests per intent: train", train[train.intent != "oos"].intent.value_counts().unique(),
"| validation", validation[validation.intent != "oos"].intent.value_counts().unique(),
"| test", test[test.intent != "oos"].intent.value_counts().unique())
print("\nThree out-of-scope requests from the training part:")
for t in train[train.intent == "oos"].text.head(3):
print(" -", t)

The output:

words per request: median 8, 90% have at most 13, longest 28
missing or empty texts: 0
repeated requests in train: 0 | test requests also in train: 2
in-scope requests per intent: train [100] | validation [20] | test [30]
Three out-of-scope requests from the training part:
- how much is an overdraft fee for bank
- why are exponents preformed before multiplication in the order of operations
- what size wipers does this car take

What this tells us. The requests are short (median 8 words). There are no gaps or repeats, only 2 test requests also appear in train, and every intent has the same number of requests.

The three out-of-scope requests show why the task is hard. “How much is an overdraft fee for bank” sounds like banking, but none of the 15 banking intents answers fee questions. An out-of-scope request is often close to something the bot can do.

For each request, the bot does one of two things. It answers with an intent, or it hands off to a person. Each outcome can be right or wrong:

The request is … The bot answers right The bot answers wrong The bot hands off
in scope (one of the 150 intents) US$0 US$4: a wrong answer US$1.50: a person does what the bot could do
out of scope (not possible) US$4: a wrong answer US$0: the right place

These prices are assumptions that follow Larkfield’s cost table: a person’s review costs US$1.50 (one minute of an agent), and a message in the wrong queue US$4. Here a wrong answer means the customer reads nonsense, asks again, and often reaches a person anyway. An out-of-scope request that reaches a person costs nothing extra.

The notebook calls the hand-off needs_person, like Larkfield’s route, so the course’s helper routing.py computes every cost. Its helper cell defines decide(), score() (one row of the comparison table) and sweep() (every threshold).

The cost per 1,000 requests depends on the mix, and Larkfield’s real mix is unknown until the assistant runs. So Priya fixes one assumption for every cost on this page: 1 request in 5 is out of scope. With it, “everything to a person” costs US$1,200 per 1,000 requests: 800 in-scope requests at US$1.50 each. That is the “no bot” row, the number every method must beat.

Here is a practice problem with 6 requests. Below the threshold, the bot hands off.

Request True intent The model says Confidence At 0.50 Cost At 0.75 Cost
what’s my checking balance balance balance 0.91 answer 0 answer 0
book a table for two at 8 restaurant_reservation restaurant_reservation 0.62 answer 0 hand off 1.50
how do i fix my sink out of scope schedule_maintenance 0.34 hand off 0 hand off 0
transfer 50 dollars to sam transfer transfer 0.48 hand off 1.50 hand off 1.50
what is the tallest mountain out of scope distance 0.71 answer 4 hand off 0
set an alarm for 6 alarm reminder 0.55 answer 4 hand off 1.50

At 0.50, the total is US$9.50 for 6 requests, with two wrong answers. At 0.75, the total is US$4.50. The two wrong answers became hand-offs, but one right answer did too: the table booking (0.62).

When is the trade worth it? Say the model is right with a chance p. Answering costs (1 − p) × 4 on average, and handing off an in-scope request costs 1.50. The two are equal at p = 1 − 1.50 ÷ 4 = 0.625. That is the break-even point, but only if the confidence were a true chance.

A diagram. A new request, 'can you recommend a good plumber near me', goes to three rules. Rule 1, confidence threshold: the classifier picks one of 150 intents; is its top probability under the threshold? Rule 2, nearest example: find the most similar training request; is the similarity under the threshold? Rule 3, an out-of-scope class: the classifier also learned oos from 100 examples; does it predict oos? Each rule leads to 'Yes: hand off, I cannot help with that, here is a person' or 'No: answer with the intent'.
The three ways that this case study compares. Each one ends in the same two outcomes.
Text description of the diagram

A flow diagram. On the left, a new request: “can you recommend a good plumber near me”. Arrows lead to three boxes. Box 1, confidence threshold: the classifier picks one of 150 intents, and the rule asks whether its top probability is under the threshold. Box 2, nearest example: the rule finds the most similar training request and asks whether the similarity is under the threshold. Box 3, an out-of-scope class: the classifier also learned the label oos from 100 examples, and the rule asks whether it predicts oos. From each box, one arrow leads to “Yes: hand off. I cannot help with that. Here is a person.” and one to “No: answer with the intent”.

The third way needs examples of out-of-scope requests: CLINC150 has 100 in its training part. Every method gets its own threshold, chosen on validation at the lowest cost.

The first model is the course’s TF-IDF with logistic regression, as in Naive Bayes, logistic regression and linear SVM. Priya trains it on the 15,000 in-scope training requests only, so it has no way to say “none of them”.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import FeatureUnion, make_pipeline
def tfidf_lr():
"""The course project's TF-IDF + logistic regression: words and word pairs, plus letter groups."""
words = TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True, strip_accents="unicode")
letters = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 5), min_df=3, sublinear_tf=True)
return make_pipeline(FeatureUnion([("words", words), ("letters", letters)]), LogisticRegression(C=10, max_iter=3000))
in_scope = train[train.intent != "oos"]
t0 = time.perf_counter()
tfidf = tfidf_lr().fit(in_scope.text, in_scope.intent) # learns the 150 intents only
print(f"fitted on {len(in_scope):,} in-scope requests in {time.perf_counter() - t0:.0f} s")
p = tfidf.predict_proba(validation.text)
tfidf_val = (tfidf.classes_[p.argmax(axis=1)], p.max(axis=1)) # (intent, confidence) for each request
tfidf_ms = ms_per_message(tfidf.predict_proba, list(validation.text))
rows_val = [score("everything to a person", validation, [HAND_OFF] * len(validation)),
score("TF-IDF + LR, always answers", validation, tfidf_val[0], cpu_cost(tfidf_ms), tfidf_ms)]
table(rows_val)

The output:

fitted on 15,000 in-scope requests in 9 s
method in_scope_accuracy oos_recall macro_f1 hand_off_share wrong_answers mistakes_per_1000 run_per_1000 total_per_1000 ms_per_message
0 TF-IDF + LR, always answers 0.924 0.0 0.903 0.0 0.261 1043.2 0.0 1043.2 0.56
1 everything to a person 0.000 1.0 0.000 1.0 0.000 1200.0 0.0 1200.0 NaN

What this tells us. On validation, the model answers 0.924 of the in-scope requests right. But its out-of-scope recall is 0, so about 1 answer in 4 is wrong (wrong_answers 0.261). The cost is US$1,043.2 per 1,000, only a little cheaper than no bot (US$1,200).

In the table, a hand-off counts as not right for in_scope_accuracy. macro_f1 treats the hand-off as a 151st label. hand_off_share and wrong_answers are shares of all requests, at the assumed mix.

On Larkfield’s synthetic inbox, the same model reached 0.962. Here it is 0.924 on in-scope requests alone, with 150 labels instead of 7 and wording from many people.

Two questions decide whether a threshold can work. Is the confidence true? And do out-of-scope requests get a lower confidence?

Is the confidence true? taught the reliability table. Priya checks the in-scope validation requests with the course helper rt.reliability():

# Is the confidence true? The reliability table of the in-scope validation requests (Module 8).
ins = (validation.intent != "oos").to_numpy()
rel, ece = rt.reliability(tfidf_val[1][ins], tfidf_val[0][ins] == validation.intent[ins])
print("expected calibration error:", ece)
print("median confidence: in scope", round(float(np.median(tfidf_val[1][ins])), 3),
"| out of scope", round(float(np.median(tfidf_val[1][~ins])), 3))
rel

The output:

expected calibration error: 0.0967
median confidence: in scope 0.944 | out of scope 0.11
bin messages mean_confidence accuracy
0 0.0-0.1 29 0.073 0.310
1 0.1-0.2 73 0.152 0.452
2 0.2-0.3 77 0.256 0.506
3 0.3-0.4 85 0.349 0.671
4 0.4-0.5 104 0.456 0.702
5 0.5-0.6 128 0.550 0.805
6 0.6-0.7 153 0.652 0.895
7 0.7-0.8 201 0.750 0.935
8 0.8-0.9 343 0.858 0.974
9 0.9-1.0 1807 0.972 0.996

What this tells us. The expected calibration error is 0.097: worse than on Larkfield’s inbox (0.050). The model is too shy. Where it says 0.45, it is right 70% of the time; where it says 0.15, 45% of the time. A model that spreads its belief over 150 intents gives each a small share. So the break-even point of 0.625 does not apply, and the threshold must come from the costs.

The good news: in-scope requests get a median confidence of 0.944, and out-of-scope requests 0.11.

  • In scope (3,000 requests)
  • Out of scope (100 requests)
The confidence of TF-IDF + logistic regression on the 3,100 validation requests, as a share of each group (measured). Out-of-scope requests pile up on the left, in-scope requests on the right; the overlap in the middle is where every threshold makes mistakes.
Text description of the chart

A line chart with the confidence from 0 to 1 across, in bins of 0.05, and the share of each group up. The out-of-scope line starts high: 19% of the out-of-scope requests sit between 0 and 0.05, 24% between 0.05 and 0.10, and 16% between 0.10 and 0.15. Then it falls, and stays between 0% and 4% from 0.30 on. The in-scope line stays under 3.5% per bin up to 0.80, rises to 7% at 0.85 and 13% at 0.90, and jumps to 48% in the last bin, 0.95 to 1.00. About 79% of the out-of-scope requests are under 0.30. About 60% of the in-scope requests are at 0.90 or more.

What this tells us. Most out-of-scope requests sit under 0.3, and most in-scope requests above 0.9. A threshold in the middle separates most of them. The overlap is the price: some in-scope requests sit low, and a few out-of-scope requests sit high.

Priya tries every threshold from 0.00 to 0.99 on the validation requests and keeps the cheapest, as in Decide, review or send to a person, with one threshold instead of two.

thresholds = np.round(np.arange(0.0, 1.0, 0.01), 2)
curve_tfidf = sweep("TF-IDF + LR", validation, *tfidf_val, thresholds)
best = curve_tfidf.loc[curve_tfidf.total_per_1000.idxmin()]
T_TFIDF = float(best.threshold)
print(f"lowest cost on validation: hand off below {T_TFIDF:.2f}; in-scope accuracy {best.in_scope_accuracy:.3f}, "
f"out-of-scope recall {best.oos_recall:.3f}, {best.total_per_1000:.1f} dollars per 1,000")
shown = curve_tfidf.threshold.isin([0.0, 0.1, 0.2, T_TFIDF, 0.4, 0.5, 0.6, 0.7, 0.9])
curve_tfidf.loc[shown, ["threshold", "in_scope_accuracy", "oos_recall", "hand_off_share", "wrong_answers", "total_per_1000"]]

The output:

lowest cost on validation: hand off below 0.55; in-scope accuracy 0.838, out-of-scope recall 0.900, 312.1 dollars per 1,000
threshold in_scope_accuracy oos_recall hand_off_share wrong_answers total_per_1000
0 0.00 0.924 0.00 0.000 0.261 1043.2
10 0.10 0.921 0.43 0.094 0.169 689.5
20 0.20 0.910 0.67 0.161 0.111 484.0
40 0.40 0.878 0.84 0.238 0.059 342.4
50 0.50 0.854 0.86 0.270 0.047 334.9
55 0.55 0.838 0.90 0.295 0.035 312.1
60 0.60 0.819 0.91 0.314 0.030 319.5
70 0.70 0.774 0.95 0.363 0.018 331.6
90 0.90 0.600 0.99 0.516 0.004 493.7
  • TF-IDF + LR (the baseline)
  • Embeddings + LR with an oos class (best)
The cost per 1,000 requests at each threshold, on the validation requests, with 1 request in 5 out of scope (measured; the costs and the share are assumptions). The dots mark the cheapest threshold of each method.
Text description of the chart

A line chart with the threshold from 0 to 1 across and the cost per 1,000 requests up, from 0 to 1,200. The TF-IDF line starts at US$1,043.2 at 0.00, falls quickly to US$484.0 at 0.20 and US$342.4 at 0.40, reaches its lowest point, US$312.1, at 0.55, and rises slowly to US$493.7 at 0.90 and US$958.3 at 0.99. The line of embeddings + LR with an oos class starts lower, at US$551.2, because the class already hands off half of the out-of-scope requests. It reaches its lowest point, US$212.8, at 0.62, and rises to US$1,089.6 at 0.99.

What this tells us. The cost falls fast from US$1,043.2 at 0.00 to its lowest at 0.55: US$312.1 per 1,000. There, the bot hands 90% of the out-of-scope requests to a person and answers 0.838 of the in-scope ones right, against 0.924 before. 0.55 is below the break-even point of 0.625, because the model is shy.

From 0.40 to 0.70, the cost stays between US$312.1 and US$346.9. The exact threshold matters less than having one.

The second model uses the course’s older embedding model (from Sentence embeddings for classification), frozen. Each request becomes 384 numbers, and a logistic regression learns the 150 intents on top.

from sentence_transformers import SentenceTransformer
torch.set_num_threads(4)
# The course's older embedding model, frozen. Each request becomes 384 numbers of length 1.
embedder = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
revision="e8f8c211226b894fcb81acc59f3b34ba3efd5f42", device="cpu")
t0 = time.perf_counter()
E = {name: embedder.encode(list(df.text), batch_size=128, normalize_embeddings=True)
for name, df in [("train", train), ("validation", validation), ("test", test)]}
print(f"{sum(len(v) for v in E.values()):,} requests embedded in {time.perf_counter() - t0:.0f} s")
ins_train = (train.intent != "oos").to_numpy()
frozen = LogisticRegression(C=10, max_iter=3000).fit(E["train"][ins_train], train.intent[ins_train])
p = frozen.predict_proba(E["validation"])
frozen_val = (frozen.classes_[p.argmax(axis=1)], p.max(axis=1))
frozen_ms = ms_per_message(lambda t: frozen.predict_proba(embedder.encode(t, normalize_embeddings=True)),
list(validation.text), n=100)
curve_frozen = sweep("embeddings + LR", validation, *frozen_val, thresholds)
T_FROZEN = float(curve_frozen.loc[curve_frozen.total_per_1000.idxmin(), "threshold"])
print("embeddings + LR: lowest cost on validation when it hands off below", T_FROZEN)

The output:

23,700 requests embedded in 46 s
embeddings + LR: lowest cost on validation when it hands off below 0.6

Embedding all 23,700 requests takes under a minute, once.

The second way needs no classifier. Think of a sorter with a box of example letters. If even the most similar example is far away, the letter is probably for no tray here.

The cosine similarity of two requests is 1 when they mean the same thing and near 0 when they are unrelated. The rule: find the in-scope training request with the highest similarity. If the similarity is at the threshold or more, answer with its intent. If not, hand off.

# Distance to the nearest example: how similar is the request to the closest in-scope training request?
bank, bank_intent = E["train"][ins_train], train.intent[ins_train].to_numpy()
def nearest(X, size=1000):
"""For each row of X: the intent of the most similar in-scope training request, and the cosine similarity."""
intents, sims = [], []
for i in range(0, len(X), size):
S = X[i:i + size] @ bank.T # the vectors have length 1, so this is the cosine similarity
intents.append(bank_intent[S.argmax(axis=1)])
sims.append(S.max(axis=1))
return np.concatenate(intents), np.concatenate(sims)
nearest_val = nearest(E["validation"])
curve_nearest = sweep("nearest example", validation, *nearest_val, thresholds)
T_NEAREST = float(curve_nearest.loc[curve_nearest.total_per_1000.idxmin(), "threshold"])
ins = (validation.intent != "oos").to_numpy()
print("median similarity to the nearest example: in scope", round(float(np.median(nearest_val[1][ins])), 3),
"| out of scope", round(float(np.median(nearest_val[1][~ins])), 3))
print("nearest example: lowest cost on validation when it hands off below", T_NEAREST)

The output:

median similarity to the nearest example: in scope 0.876 | out of scope 0.529
nearest example: lowest cost on validation when it hands off below 0.76

What this tells us. In-scope requests have a median similarity of 0.876 to their nearest example; out-of-scope requests 0.529. But the cheapest threshold is high (0.76), and this rule hands off many in-scope requests too.

The third way gives the model the 100 out-of-scope training requests as a 151st intent, oos. When it predicts oos, the bot hands off. Priya tries it twice: alone, and with a threshold on top.

# An explicit class: the 100 out-of-scope training requests become a 151st intent, "oos".
oos_tfidf = tfidf_lr().fit(train.text, train.intent)
p = oos_tfidf.predict_proba(validation.text)
oos_tfidf_val = (oos_tfidf.classes_[p.argmax(axis=1)], p.max(axis=1))
oos_frozen = LogisticRegression(C=10, max_iter=3000).fit(E["train"], train.intent)
p = oos_frozen.predict_proba(E["validation"])
oos_frozen_val = (oos_frozen.classes_[p.argmax(axis=1)], p.max(axis=1))
curve_oos_tfidf = sweep("TF-IDF + LR, oos class", validation, *oos_tfidf_val, thresholds)
curve_oos_frozen = sweep("embeddings + LR, oos class", validation, *oos_frozen_val, thresholds)
T_OOS_TFIDF = float(curve_oos_tfidf.loc[curve_oos_tfidf.total_per_1000.idxmin(), "threshold"])
T_OOS_FROZEN = float(curve_oos_frozen.loc[curve_oos_frozen.total_per_1000.idxmin(), "threshold"])
rows_val += [score("TF-IDF + LR, oos class, no threshold", validation, decide(*oos_tfidf_val, 0.0),
cpu_cost(tfidf_ms), tfidf_ms),
score("embeddings + LR, oos class, no threshold", validation, decide(*oos_frozen_val, 0.0),
cpu_cost(frozen_ms), frozen_ms)]
print("thresholds with the oos class: TF-IDF", T_OOS_TFIDF, "| embeddings", T_OOS_FROZEN)

The output:

thresholds with the oos class: TF-IDF 0.49 | embeddings 0.62

Common mistake

An out-of-scope class without a threshold

Symptom: Tomás says: “We have examples of out-of-scope requests. Train them as a class, and we are done.” With no threshold, embeddings + LR with the oos class finds only 50% of the out-of-scope validation requests. TF-IDF with the class finds 41%. The cost is US$551.2 per 1,000.

Cause: out of scope is not one topic. It is the whole world minus 150 intents, and 100 examples cannot show all of that.

Fix: keep the class, and add a threshold on its confidence. The class catches the out-of-scope requests that look like its examples. The threshold catches the ones that look like nothing the model knows.

Consequence in AI work: a demo with known strange questions passes. Then real users ask about something new, and the bot answers with full confidence.

The table puts every method at its own threshold, cheapest first, by the cost per 1,000 requests.

# Every method at the threshold chosen on validation (the lowest cost), one row each.
CHOSEN = {"TF-IDF + LR": (tfidf_val, T_TFIDF, tfidf_ms), "embeddings + LR": (frozen_val, T_FROZEN, frozen_ms),
"nearest example": (nearest_val, T_NEAREST, frozen_ms),
"TF-IDF + LR, oos class": (oos_tfidf_val, T_OOS_TFIDF, tfidf_ms),
"embeddings + LR, oos class": (oos_frozen_val, T_OOS_FROZEN, frozen_ms)}
rows_val += [score(f"{m}, hand off below {t:.2f}", validation, decide(*v, t), cpu_cost(ms), ms)
for m, (v, t, ms) in CHOSEN.items()]
table(rows_val)

The output:

method in_scope_accuracy oos_recall macro_f1 hand_off_share wrong_answers mistakes_per_1000 run_per_1000 total_per_1000 ms_per_message
0 embeddings + LR, oos class, hand off below 0.62 0.884 0.93 0.925 0.271 0.021 212.8 0.0 212.8 11.92
1 embeddings + LR, hand off below 0.60 0.891 0.90 0.928 0.259 0.028 230.8 0.0 230.8 11.92
2 TF-IDF + LR, oos class, hand off below 0.49 0.856 0.91 0.899 0.277 0.038 295.9 0.0 295.9 0.56
3 TF-IDF + LR, hand off below 0.55 0.838 0.90 0.890 0.295 0.035 312.1 0.0 312.1 0.56
4 nearest example, hand off below 0.76 0.738 0.97 0.798 0.367 0.043 432.1 0.0 432.1 11.92
5 embeddings + LR, oos class, no threshold 0.951 0.50 0.942 0.103 0.137 551.2 0.0 551.2 11.92
6 TF-IDF + LR, oos class, no threshold 0.923 0.41 0.912 0.084 0.178 714.8 0.0 714.8 0.56
7 TF-IDF + LR, always answers 0.924 0.00 0.903 0.000 0.261 1043.2 0.0 1043.2 0.56
8 everything to a person 0.000 1.00 0.000 1.000 0.000 1200.0 0.0 1200.0 NaN

What this tells us.

  • Every threshold helps a lot. The worst method with a threshold (the nearest example, US$432.1) is cheaper than the best method without one (US$551.2).
  • The best is embeddings + LR with the oos class, handing off below 0.62: US$212.8 per 1,000. It combines the class and the threshold.
  • Embeddings beat TF-IDF here. On Larkfield’s inbox, TF-IDF won (0.962 against 0.920). On these short, varied requests, meaning matters more than exact words.
  • The nearest example is the weakest. It finds the most out-of-scope requests (0.97) but answers only 0.738 of the in-scope ones right. One neighbour is a noisy vote; the classifier uses all 100 examples of each intent.

A picture shows all the thresholds. Each gives one point: out-of-scope recall across, in-scope accuracy up. Up and to the right is better.

  • TF-IDF + LR
  • TF-IDF + LR, oos class (dashed)
  • Embeddings + LR (dashed)
  • Embeddings + LR, oos class
  • Nearest example
The trade-off on the 3,100 validation requests (measured): one point per threshold, from 0.00 at the left end until the in-scope accuracy falls under 50%. The dot is the threshold chosen by cost for the best method.
Text description of the chart

A line chart with out-of-scope recall from 0 to 1 (0% to 100%) across and in-scope accuracy from 50% to 100% up. Each of the five lines starts at its left end (threshold 0) and falls to the right as the threshold rises. TF-IDF + LR starts at 0% recall and 92.4% accuracy, embeddings + LR at 0% and 95.1%, the nearest example at 0% and 88.0%. The two lines with an oos class start further right: TF-IDF at 41% recall and 92.3%, embeddings at 50% and 95.1%. At 90% recall, the in-scope accuracy is 90.9% for embeddings + LR with the oos class (the highest), 89.1% for embeddings + LR, 85.9% for TF-IDF with the class, 83.8% for TF-IDF without it, and 80.7% for the nearest example (the lowest). A dot marks the chosen threshold, 0.62, at 93% recall and 88.4% accuracy.

What this tells us. Every curve falls to the right: finding more out-of-scope requests always costs some right answers. At 90% out-of-scope recall, the in-scope accuracy goes from 0.909 (embeddings with the oos class) down to 0.807 (the nearest example).

Only 100 out-of-scope validation requests make these curves, so one request is one point of recall. A difference of 2 or 3 points is noise.

Priya gave the course’s hosted model, chat-small, the 150 intents by domain and one way out: out_of_scope. It answered in JSON with an intent and a confidence, as in Zero-shot classification with an LLM. Its answers were recorded for a sample, so you need no key. The samples have 3 requests per intent plus 100 out of scope (validation) and 6 per intent plus 200 (test): 18.2% out of scope in both.

# The prompt of the recorded answers: the 150 intents by domain, and a way out.
by_domain = "\n".join(f"- {d}: " + ", ".join(sorted(i for i in INTENTS if DOMAIN_OF[i] == d))
for d in sorted(set(DOMAIN_OF.values())))
SYSTEM = ("You are the intent classifier of a virtual assistant. Choose exactly one intent from this list "
f"(grouped by domain):\n{by_domain}\n"
"If the request fits none of these intents, choose out_of_scope. "
"Answer in JSON with the intent and your confidence from 0 to 1.")
print(SYSTEM[:400], "...")
print(f"\n{len(SYSTEM.split())} words in the instructions")

The output:

You are the intent classifier of a virtual assistant. Choose exactly one intent from this list (grouped by domain):
- auto_and_commute: current_location, directions, distance, gas, gas_type, jump_start, last_maintenance, mpg, oil_change_how, oil_change_when, schedule_maintenance, tire_change, tire_pressure, traffic, uber
- banking: account_blocked, balance, bill_balance, bill_due, freeze_account, ...
212 words in the instructions

The notebook scores the LLM next to the other methods on the same sample. Each method keeps the threshold that it chose on the full validation part:

def recorded(mode):
"""The recorded answers of the hosted LLM (chat-small), one line per request."""
rows = [json.loads(x) for x in open(f"recordings/llm-{mode}.jsonl", encoding="utf-8")]
return pd.DataFrame(rows).set_index("message_id")
zero = recorded("zero-shot")
filtered = zero.get("filtered", pd.Series(False, index=zero.index)).fillna(False).astype(bool)
answered = zero[~filtered] # the provider's content filter refused a few requests: no answer, a hand-off
PRICE_IN, PRICE_OUT = 0.10, 0.50 # chat-small, dollars per 1 million tokens (the course's price table)
llm_run = (answered.tokens_in.mean() * PRICE_IN + answered.tokens_out.mean() * PRICE_OUT) / 1e6 * 1000
llm_ms = answered.latency_s.median() * 1000
print(f"zero-shot: {len(zero)} requests, {len(answered)} answers, all valid: {answered.valid.all()}, "
f"refused by the content filter: {filtered.sum()} | tokens in {answered.tokens_in.mean():.0f}, "
f"out {answered.tokens_out.mean():.0f} | {llm_run:.3f} dollars per 1,000 | median {llm_ms:.0f} ms")
def llm_answers(part):
a = zero.loc[part.message_id]
return a.intent.replace("out_of_scope", "oos").to_numpy(), a.confidence.to_numpy()
llm_val = llm_answers(validation_sample)
curve_llm = sweep("LLM zero-shot", validation_sample, *llm_val, thresholds)
T_LLM = float(curve_llm.loc[curve_llm.total_per_1000.idxmin(), "threshold"])
vs = validation_sample.index
rows_sample = [score(f"{m}, hand off below {t:.2f}", validation_sample,
decide(v[0][vs], v[1][vs], t), cpu_cost(ms), ms) for m, (v, t, ms) in CHOSEN.items()]
rows_sample += [score("LLM zero-shot, its own out_of_scope only", validation_sample, decide(*llm_val, 0.0), llm_run, llm_ms),
score(f"LLM zero-shot, hand off below {T_LLM:.2f}", validation_sample, decide(*llm_val, T_LLM), llm_run, llm_ms)]
table(rows_sample)

The output:

zero-shot: 1650 requests, 1646 answers, all valid: True, refused by the content filter: 4 | tokens in 1132, out 18 | 0.122 dollars per 1,000 | median 1236 ms
method in_scope_accuracy oos_recall macro_f1 hand_off_share wrong_answers mistakes_per_1000 run_per_1000 total_per_1000 ms_per_message
0 embeddings + LR, oos class, hand off below 0.62 0.891 0.93 0.919 0.268 0.019 200.0 0.000 200.0 11.92
1 embeddings + LR, hand off below 0.60 0.898 0.90 0.921 0.256 0.025 216.0 0.000 216.0 11.92
2 LLM zero-shot, hand off below 0.59 0.913 0.94 0.919 0.231 0.039 218.7 0.122 218.8 1236.00
3 LLM zero-shot, its own out_of_scope only 0.916 0.94 0.919 0.225 0.042 224.9 0.122 225.0 1236.00
4 TF-IDF + LR, oos class, hand off below 0.49 0.873 0.91 0.895 0.266 0.036 268.4 0.000 268.4 0.56
5 TF-IDF + LR, hand off below 0.55 0.853 0.90 0.884 0.287 0.031 282.7 0.000 282.7 0.56
6 nearest example, hand off below 0.76 0.749 0.97 0.799 0.366 0.034 396.4 0.000 396.4 11.92

What this tells us. Tomás is partly right. With only its own out_of_scope answer, the LLM finds 0.94 of the out-of-scope requests and answers 0.916 of the in-scope ones right, with no training. No local method is as good on both. A threshold on its written confidence (0.59) changes little. But the best local method is cheaper on this sample: US$200.0 against US$218.8. It hands off the in-scope requests where the LLM gives a wrong intent, at US$1.50 each.

The LLM also has the costs from Cost per 1,000 messages. It takes about 1,236 ms per request, against about 12 ms for the local embeddings. Every request goes outside, and the bill is US$0.12 per 1,000.

The threshold came from two assumptions: the share of out-of-scope requests and the price of a wrong answer.

Predict first: if more requests are out of scope, does the best threshold go up or down?

# Change one thing: the share of out-of-scope requests. Choose the threshold again for each share.
for share in [0.05, 0.20, 0.40]:
c = pd.DataFrame([{"threshold": t, **score("", validation, decide(*oos_frozen_val, t), share=share)}
for t in thresholds])
b = c.loc[c.total_per_1000.idxmin()]
print(f"{share:.0%} out of scope: hand off below {b.threshold:.2f}, out-of-scope recall {b.oos_recall:.3f}, "
f"in-scope accuracy {b.in_scope_accuracy:.3f}, {b.total_per_1000:.1f} dollars per 1,000")

The output:

5% out of scope: hand off below 0.52, out-of-scope recall 0.910, in-scope accuracy 0.908, 185.5 dollars per 1,000
20% out of scope: hand off below 0.62, out-of-scope recall 0.930, in-scope accuracy 0.884, 212.8 dollars per 1,000
40% out of scope: hand off below 0.69, out-of-scope recall 0.950, in-scope accuracy 0.859, 216.4 dollars per 1,000

What this tells us. The threshold goes up with the share: 0.52 at 5%, 0.62 at 20%, 0.69 at 40%. With more out-of-scope requests, every answer is a bigger risk. So the first weeks of the real assistant must count the share, from a sample of the hand-offs.

In a bank, a wrong answer can move money. The notebook’s task sets its price to US$10 and asks for the new cheapest threshold of the best method.

Show the answer
# Change one thing: a wrong answer costs 10 dollars, not 4 (a bank, where a wrong answer moves money).
COSTS_4 = dict(COSTS)
COSTS.update(wrong_queue=10.0, needs_person_sent_to_team=10.0)
c = sweep("", validation, *oos_frozen_val, thresholds)
threshold_10 = float(c.loc[c.total_per_1000.idxmin(), "threshold"])
COSTS.update(COSTS_4)
print("a wrong answer costs 10 dollars: hand off below", threshold_10)
print(c.loc[c.threshold == threshold_10, ["in_scope_accuracy", "oos_recall", "hand_off_share", "total_per_1000"]].to_string(index=False))
a wrong answer costs 10 dollars: hand off below 0.69
in_scope_accuracy oos_recall hand_off_share total_per_1000
0.859 0.95 0.298 312.3

The check cell prints Check passed: threshold_10 is 0.69. The bot becomes more careful: it hands off below 0.69 instead of 0.62. It finds 95% of the out-of-scope requests, and hands off 29.8% of all requests.

The test part has 5,500 requests that no step has touched, 1,000 of them out of scope. Priya scores the chosen thresholds once.

# The test requests, once. Every method and threshold was chosen on validation; nothing changes after this cell.
p = tfidf.predict_proba(test.text); tfidf_test = (tfidf.classes_[p.argmax(axis=1)], p.max(axis=1))
p = frozen.predict_proba(E["test"]); frozen_test = (frozen.classes_[p.argmax(axis=1)], p.max(axis=1))
p = oos_tfidf.predict_proba(test.text); oos_tfidf_test = (oos_tfidf.classes_[p.argmax(axis=1)], p.max(axis=1))
p = oos_frozen.predict_proba(E["test"]); oos_frozen_test = (oos_frozen.classes_[p.argmax(axis=1)], p.max(axis=1))
nearest_test = nearest(E["test"])
TESTS = {"TF-IDF + LR": tfidf_test, "embeddings + LR": frozen_test, "nearest example": nearest_test,
"TF-IDF + LR, oos class": oos_tfidf_test, "embeddings + LR, oos class": oos_frozen_test}
rows_test = [score("everything to a person", test, [HAND_OFF] * len(test)),
score("TF-IDF + LR, always answers", test, tfidf_test[0], cpu_cost(tfidf_ms), tfidf_ms)]
rows_test += [score(f"{m}, hand off below {t:.2f}", test, decide(*TESTS[m], t), cpu_cost(ms), ms)
for m, (v, t, ms) in CHOSEN.items()]
table(rows_test)

The output:

method in_scope_accuracy oos_recall macro_f1 hand_off_share wrong_answers mistakes_per_1000 run_per_1000 total_per_1000 ms_per_message
0 embeddings + LR, oos class, hand off below 0.62 0.866 0.892 0.906 0.276 0.031 271.5 0.0 271.5 11.92
1 TF-IDF + LR, oos class, hand off below 0.49 0.859 0.907 0.897 0.273 0.039 295.7 0.0 295.7 0.56
2 embeddings + LR, hand off below 0.60 0.874 0.852 0.906 0.261 0.040 296.4 0.0 296.4 11.92
3 TF-IDF + LR, hand off below 0.55 0.842 0.913 0.889 0.293 0.034 300.1 0.0 300.1 0.56
4 nearest example, hand off below 0.76 0.740 0.959 0.797 0.358 0.049 446.9 0.0 446.9 11.92
5 TF-IDF + LR, always answers 0.928 0.000 0.841 0.000 0.258 1031.1 0.0 1031.1 0.56
6 everything to a person 0.000 1.000 0.002 1.000 0.000 1200.0 0.0 1200.0 NaN

What this tells us. The best and the worst methods are the same as on validation; the three in the middle are within US$5 of each other. Embeddings + LR with the oos class, handing off below 0.62, costs US$271.5 per 1,000 requests on test (validation: US$212.8). It answers 0.866 of the in-scope requests right and hands 0.892 of the out-of-scope requests to a person. Without detection, the same kind of bot costs US$1,031.1; no bot costs US$1,200.

What does it prevent? Count the wrong automatic answers per 1,000 requests: 258 for the bot that always answers, 31 for the chosen one. The price is 276 hand-offs per 1,000, of which 178 are out-of-scope requests that belong with a person anyway.

The LLM was recorded for a sample of 1,100 test requests. On that same sample:

# The LLM was recorded for a sample of the test requests: compare every method on that same sample.
ts = test_sample.index
llm_test = llm_answers(test_sample)
rows_test_sample = [score(f"{m}, hand off below {t:.2f}", test_sample, decide(TESTS[m][0][ts], TESTS[m][1][ts], t),
cpu_cost(ms), ms) for m, (v, t, ms) in CHOSEN.items()]
rows_test_sample += [score("LLM zero-shot, its own out_of_scope only", test_sample, decide(*llm_test, 0.0), llm_run, llm_ms),
score(f"LLM zero-shot, hand off below {T_LLM:.2f}", test_sample, decide(*llm_test, T_LLM), llm_run, llm_ms)]
table(rows_test_sample)

The output:

method in_scope_accuracy oos_recall macro_f1 hand_off_share wrong_answers mistakes_per_1000 run_per_1000 total_per_1000 ms_per_message
0 embeddings + LR, oos class, hand off below 0.62 0.874 0.915 0.908 0.272 0.029 247.6 0.000 247.6 11.92
1 LLM zero-shot, hand off below 0.59 0.903 0.960 0.913 0.229 0.048 248.0 0.122 248.1 1236.00
2 TF-IDF + LR, oos class, hand off below 0.49 0.864 0.945 0.900 0.279 0.030 253.3 0.000 253.3 0.56
3 LLM zero-shot, its own out_of_scope only 0.907 0.935 0.914 0.222 0.053 264.0 0.122 264.1 1236.00
4 TF-IDF + LR, hand off below 0.55 0.847 0.940 0.889 0.295 0.028 272.0 0.000 272.0 0.56
5 embeddings + LR, hand off below 0.60 0.886 0.865 0.910 0.252 0.039 276.4 0.000 276.4 11.92
6 nearest example, hand off below 0.76 0.744 0.960 0.791 0.356 0.048 438.7 0.000 438.7 11.92

What this tells us. On this sample, the LLM with its threshold and the best local method cost the same: US$248.1 and US$247.6 per 1,000. The LLM finds more out-of-scope requests (0.960 against 0.915) and answers more in-scope requests right (0.903 against 0.874). But it gives more wrong answers: 4.8% of requests against 2.9%.

Four of the 1,650 recorded requests got no answer: the provider’s content filter refused harmless requests such as “tell me what the point of life is”. The notebook counts them as hand-offs. Your system needs a path for refusals.

  • TF-IDF + LR (baseline)
  • Embeddings + LR, oos class (chosen)
  • Nearest example
The trade-off on the 5,500 test requests (measured), for three of the five methods. The dots are the thresholds chosen on validation; no choice was made on these requests.
Text description of the chart

A line chart like the one above, on the test requests, with three lines. TF-IDF + LR starts at 0% recall and 92.8% in-scope accuracy, the nearest example at 0% and 88.1%, and embeddings + LR with an oos class at 37.9% and 94.6%. All three fall to the right, and the nearest example stays lowest. Dots mark the thresholds chosen on validation: TF-IDF at 0.55 (91.3% recall, 84.2% accuracy), the nearest example at 0.76 (95.9%, 74.0%), and embeddings with the oos class at 0.62 (89.2%, 86.6%).

The test curves are smoother: 1,000 out-of-scope requests instead of 100. The chosen dots sit near the bend of each curve. The chosen method finds a few fewer out-of-scope requests than on validation (0.892 against 0.93). At 90% recall, the two methods with an oos class are almost equal: 0.862 and 0.864 in-scope accuracy. Embeddings with the class still cost the least at their chosen point, US$271.5 against US$295.7, because they give fewer wrong answers (3.1% of requests against 3.9%). Report the order of the methods, not the exact gap.

Priya reads the mistakes of the chosen method, not only their count.

# Where does the chosen method still go wrong on the test requests?
chosen_intent, chosen_conf = oos_frozen_test
d = decide(chosen_intent, chosen_conf, T_OOS_FROZEN)
oos = (test.intent == "oos").to_numpy()
missed = test[oos & (d != HAND_OFF)].assign(answered_as=d[oos & (d != HAND_OFF)],
confidence=chosen_conf[oos & (d != HAND_OFF)].round(3))
print(f"out-of-scope requests answered by the bot: {len(missed)} of {oos.sum()}")
handed = test[~oos & (d == HAND_OFF)]
print(f"in-scope requests handed to a person: {len(handed)} of {(~oos).sum()}; most often:",
handed.intent.value_counts().head(5).to_dict())
missed.sort_values("confidence", ascending=False)[["text", "answered_as", "confidence"]].head(8)

The output:

out-of-scope requests answered by the bot: 108 of 1000
in-scope requests handed to a person: 550 of 4500; most often: {'distance': 13, 'reminder_update': 13, 'order': 13, 'change_user_name': 12, 'restaurant_suggestion': 12}
text answered_as confidence
5223 read my friend's text message text 0.998
5138 look up the conversion rate for the euro to dollar exchange exchange_rate 0.993
5362 can you order some ointment for me order 0.987
5365 please read the text message i just received text 0.983
5017 give me the weather forecast for today weather 0.978
4605 why can you have the square root of a negative number calculator 0.973
4927 can you look at what the oldest age is someone can enlist in the army how_old_are_you 0.961
4501 how many prime numbers are there between 0 and 100 calculator 0.961

What this tells us. 108 of the 1,000 out-of-scope test requests still get an answer. Some are close to an intent: “Read my friend’s text message” is near the text intent, which sends a message. Some look like labelling choices: “give me the weather forecast for today” is labelled out of scope, and the bot answers weather at 0.978. Real data has borderline labels, as Larkfield’s inbox had in Labelling guidelines.

On the other side, 550 of the 4,500 in-scope requests (12%) go to a person. No intent stands out: distance, reminder_update and order each lose 13 of their 30 test requests.

The notebook’s second task finds the domain that answers the most out-of-scope requests: utility (27 of 108), with broad intents such as weather, calculator and text. Collect out-of-scope examples near the utility intents first.

The last step runs the chosen method on eight requests that we wrote.

# New input: requests that we wrote for this case study (not from CLINC150).
new = pd.DataFrame({"text": [
"how much money is in my savings account",
"please move 200 dollars from checking to savings",
"what's the weather going to be like in boston tomorrow",
"can you recommend a good plumber near me",
"my card was stolen, someone is using it right now",
"what is the meaning of life",
"how many calories are in a banana",
"write a poem about my cat"]})
X = embedder.encode(list(new.text), normalize_embeddings=True)
p = oos_frozen.predict_proba(X)
new["intent"], new["confidence"] = oos_frozen.classes_[p.argmax(axis=1)], p.max(axis=1).round(3)
new["decision"] = decide(new.intent, new.confidence, T_OOS_FROZEN)
new["decision"] = np.where(new.decision == HAND_OFF, "hand off to a person", "answer: " + new.decision)
new

The output:

text intent confidence decision
0 how much money is in my savings account balance 0.907 answer: balance
1 please move 200 dollars from checking to savings transfer 0.969 answer: transfer
2 what's the weather going to be like in boston tomorrow weather 0.947 answer: weather
3 can you recommend a good plumber near me directions 0.259 hand off to a person
4 my card was stolen, someone is using it right now report_lost_card 0.818 answer: report_lost_card
5 what is the meaning of life meaning_of_life 0.986 answer: meaning_of_life
6 how many calories are in a banana calories 0.982 answer: calories
7 write a poem about my cat text 0.130 hand off to a person

What this tells us. The bot answers six requests and hands off two: the plumber (0.259) and the poem (0.130). “What is the meaning of life” has a small-talk intent, so the bot answers at 0.986. “My card was stolen, someone is using it right now” goes to report_lost_card at 0.818. But it is urgent, and sending urgent requests to a person at once is a business rule (see “Responsible use” below).

Priya writes to Grace in her words:

  • Method: embeddings (the course’s older model, frozen) with logistic regression. It learns the 150 intents plus an out-of-scope class from examples.
  • Threshold: hand off when the class is oos or the confidence is below 0.62.
  • What goes to people: 27.6% of all requests on the test part, at 1 request in 5 out of scope. About 12% (550 of 4,500) of the in-scope requests are handed off too.
  • What it costs: US$271.5 per 1,000 requests on test (US$212.8 on validation). A bot that always answers costs US$1,031.1, and a person on everything US$1,200. The computer time is under US$0.001 per 1,000.
  • What it prevents: wrong automatic answers fall from 258 to 31 per 1,000 requests.

And for Tomás: on the test sample, the hosted LLM found as many out-of-scope requests as any method (0.960), with no training. It is the right second step for the requests that the local model hands off, as in Cascades.

For Larkfield: the method transfers, the numbers do not. Collect out-of-scope examples from the first weeks of hand-offs, then choose the threshold again.

  • A bot that always answers is barely better than no bot (US$1,031.1 against US$1,200). Out-of-scope detection is most of the value.
  • Combine two signals: the class and the threshold.
  • The confidence was too shy (ECE 0.097), so the formula 1 − 1.50 ÷ 4 would have given the wrong threshold.
  • The threshold depends on the mix. 5%, 20% and 40% out of scope give three thresholds. The mix is a business fact that you must measure.
  • Noise. Only 100 out-of-scope validation requests chose every threshold. On test, the three middle methods came within US$5 of each other.
  • Crowd-written data from 2019. The requests are short, clean and in English. Real chats are messier, and real out-of-scope requests change with the news.
  • One intent per request. A real message may mix an in-scope and an out-of-scope request. This dataset cannot test that.
  • A hand-off must reach a person. “I cannot help with that” with no person behind it is a dead end. Measure how long hand-offs wait, as Grace measures her queue.
  • Some out-of-scope requests are urgent. CLINC150’s out-of-scope test requests include “how long does it take to call an ambulance”. Add a rule for safety words before the model, as Larkfield’s needs_person route does, and test it.
  • Say what the bot cannot do. Tell the customer plainly, and keep the request so that the person does not ask again.
  • The data leaves your computer only with the LLM. The LLM option sends each request to an outside service: check your privacy rules first. The recordings hold only CLINC150’s public requests.
  • Credit the data. CC BY 3.0 asks for credit wherever you use CLINC150: use the attribution text in the box above.

This case study shows that you can make a bot safer, not only more accurate. Make it yours with one change. Try other prices, another share of out-of-scope requests, or a fine-tuned encoder from Fine-tune a small encoder on CPU. Then make it public, and easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • The trade-off picture on the test requests, with the chosen point.
  • decision.json: the method, the model and its revision, the threshold, the costs, the assumed share, and the data’s commit and SHA-256.
  • The data source and licence, with the attribution text. CC BY 3.0 lets you share the data with credit, but a link and a checksum are enough.

A README template:

# When should a chatbot hand off? Out-of-scope detection on CLINC150
## Problem
A virtual assistant with 150 intents gets requests it was not built for. A wrong
answer costs about $4, an unneeded hand-off $1.50. Assume 1 request in 5 is out of scope.
## Data and licence
CLINC150 (Larson et al., 2019), 23,700 requests, 150 intents + out of scope,
CC BY 3.0, https://github.com/clinc/oos-eval (commit 828f809). Rows numbered; no other changes.
## Approach
Three ways: a confidence threshold, the nearest example (cosine similarity of
sentence embeddings), an explicit out-of-scope class. Each threshold chosen on
validation by expected cost. A hosted LLM replayed on a sample.
## Results (test, used once)
Embeddings + LR with an oos class, hand off below 0.62: $271.5 per 1,000
(always answer: $1,031.1; everyone to a person: $1,200). In-scope accuracy 0.866,
out-of-scope recall 0.892. Wrong answers per 1,000: 258 -> 31.
## Limits
Crowd-written English requests from 2019; 100 out-of-scope validation examples;
the threshold depends on the assumed share and prices.

How to describe it in an interview, in two sentences: “I built an out-of-scope detector for a 150-intent assistant and chose its hand-off threshold from the price of a wrong answer against a hand-off; on a held-out test it cut wrong automatic answers from 258 to 31 per 1,000 requests. I compared a confidence threshold, a nearest-example rule and an out-of-scope class, and showed that the best threshold moves with the share of out-of-scope traffic.”

Go further:

  • Calibrate the classifier (temperature scaling, from Ask for a decision, not for text) and check whether the cheapest threshold moves toward 0.625.
  • Build the cascade: the local model decides when it is sure, the replayed LLM looks at the hand-offs, a person gets the rest.
  • Write five out-of-scope requests about safety, and test whether your safety rule catches them before the model.

Check your understanding

If the confidence were a true chance, the break-even point would be 1 − 1.50 ÷ 4 = 0.625. Yet the cheapest threshold for TF-IDF on validation is lower. Explain why, and say how you would check it.

Show the answer

A model that spreads its probability over 150 intents is shy: its confidence is lower than its real chance of being right. Where TF-IDF says 0.45, it is right about 70% of the time; its expected calibration error is about 0.10. The break-even formula assumes a true confidence, so it gives a threshold that is too high. The cost search scores every threshold with the real outcomes. To check, read the reliability table, or calibrate the model and see whether the cheapest threshold moves toward 0.625.

Check your understanding

Tomás trains an explicit out-of-scope class on 100 examples and says the problem is solved. What do you show him, and what do you add?

Show the answer

Show him its out-of-scope recall without a threshold. Embeddings + LR with the class finds only about half of the out-of-scope validation requests. Out of scope is everything outside 150 intents, and 100 examples cannot cover it. Add a threshold on the confidence: it catches the requests that look like nothing the model knows. Together they were the cheapest method on validation and on test.

Check your understanding

Predict: Larkfield's assistant goes live, and a sample of real chats shows that 40% of the requests are out of scope, not 20%. Does the threshold go up or down, and why?

Show the answer

It goes up. With more out-of-scope requests, a wrong answer is more likely at every confidence. The bot should answer only when it is surer, and hand off more. On validation, the best method’s threshold moved from 0.52 at 5% to 0.62 at 20% and 0.69 at 40%. The share is a business fact: measure it from real traffic, then choose the threshold again.

Recap

  • A bot that always answers gets every out-of-scope request wrong. Here, it cost US$1,031.1 per 1,000 requests on test, against US$1,200 for no bot.
  • Three ways to hand off: a confidence threshold, the distance to the nearest example, and an explicit out-of-scope class. Each threshold came from the costs on validation. A wrong answer costs US$4, a hand-off US$1.50, and 1 request in 5 is out of scope.
  • Measure in-scope accuracy and out-of-scope recall together, and draw the trade-off. Finding more out-of-scope requests always costs some right answers.
  • The best was embeddings + LR with the oos class and a threshold of 0.62. On test, it cost US$271.5 per 1,000, and wrong answers fell from 258 to 31 per 1,000.
  • The threshold moves with the share of out-of-scope requests and with the price of a wrong answer. The replayed LLM detected out-of-scope requests well and fits as a second step.

Next, another real dataset: requests in English and French, with only a few labels per intent.

Next: Sort requests with few labels