Judgments from a language model
- Lesson 7
- 35 min
- Module 2: Measure relevance before you change anything
Where we are
Our judgments come from people, and people are slow and expensive. A language model can write labels too. In this lesson, we compare a model's recorded labels with people's labels, and we decide where the model is good enough.
Lesson 7 of 30 · Search and Ranking: Find the Best MatchJudgments from a language model
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Decide where labels from a language model can replace or add to labels from people. You check a model’s recorded labels against the WANDS judges and find where it agrees. Then you use its labels to fill the holes in the test set, and you say so.
NotebookDo this lesson in a notebook
The notebook reads the model’s recorded labels and builds the table against the WANDS judges. You compute how often each of its grades is right, then fill the unjudged results of one search. It needs no model key: the answers are recorded. About 30 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Judgments, guidelines and Cohen’s kappa, from Relevance judgments.
- NDCG@10 and the rule “an unjudged product counts as irrelevant”, from Ranking measures.
What you need on your computer (only for the guided practice)
- The project
larkfield-searchwith this module’s files, from the guided practice of Relevance judgments. No model key: the model’s answers are recorded in the practice files.
A new shop assistant
Section titled “A new shop assistant”A furniture shop hires a new assistant. She is fast and cheap, and she never gets tired. On her first day, the manager does not let her serve customers alone. He gives her 450 customer requests that experienced staff have already answered, and he compares her answers with theirs.
Where she agrees with the experienced staff, he lets her work alone. Where she often disagrees, someone checks her work. The manager does not ask “Is she good?” He asks “Is she good enough for this job?”
A language model that writes relevance judgments is that new assistant. This lesson checks her work.
The holes in the judgments
Section titled “The holes in the judgments”In Ranking measures, you met the rule: an unjudged product counts as irrelevant. The WANDS judges graded products that an older search showed them. A new search finds other products, and many of them were never graded.
This matters most for the searches that the course builds later. Search by meaning finds products that share no word with the query. Many of them have no judgment at all. On Larkfield’s English and French test queries:
| Search | Judged in the top 10 |
|---|---|
| BM25 with text cleaning, spelling and synonyms (Synonyms and query rewriting) | 71% |
| Search by meaning (Search by meaning) | 63% |
| Hybrid of the two (Hybrid search) | 72% |
A third of the results of search by meaning have no grade. Each one counts as 0. If some are good products, the search looks worse than it is. Tomás says: “We are punishing the new search for finding products that nobody looked at.”
You could ask people to judge the missing products. That is the most reliable way. But each judgment needs a person to read the query, the title, the description and the attributes. For every new search and every new test query, the holes come back.
A model as a judge
Section titled “A model as a judge”The idea: give a language model the same job as a person. A language model reads text and writes text. You give it the query, the product’s title, class and description, and the judging guidelines. You ask for one grade: 2, 1 or 0. A model used like this is often called an LLM judge (“LLM” means large language model).
For this course, the team asked chat-small, a small, low-cost chat model on Azure, for 750 labels. The answers were recorded once. The project and the notebook replay them, so you need no key and every run gives the same answers. The 750 labels cost about US$0.028 in all, about 4 cents per 1,000 labels.
The labels are in two groups:
- 450 pairs that the WANDS judges already graded: 150 exact, 150 partial and 150 irrelevant. These test the model.
- 300 unjudged pairs from the top 10 of search by meaning and the hybrid search, on the English and French test queries. These fill holes.
Never use the second group before you have checked the model on the first.
Text description of the diagram
A row of boxes. “Take pairs that people already judged, 150 per grade” leads to “The model grades the same pairs (recorded answers)”, then to “Compare with people: table, agreement, kappa”, then to a decision: “Good enough for this job?”. The “yes” arrow leads to a box with a thick border: “Label unjudged results; report it beside the people-only number”. The “no” arrow leads to a box with a red border: “Ask people, or fix the guidelines and test again”.
Check the model against people
Section titled “Check the model against people”Here are the model’s grades for the 450 pairs, against the WANDS grades. Each row is a WANDS grade. Each column is the model’s grade.
| WANDS judges ↓ / model → | Exact | Partial | Irrelevant | Pairs |
|---|---|---|---|---|
| Exact | 95 | 50 | 5 | 150 |
| Partial | 7 | 85 | 58 | 150 |
| Irrelevant | 0 | 21 | 129 | 150 |
| Model total | 102 | 156 | 192 | 450 |
The diagonal (95, 85, 129) is where they agree: 309 of 450 pairs, so agreement 68.7%.
Now the kappa, as in Relevance judgments. The WANDS grades are one third each. So two random judges with these shares would agree on one third of the pairs, whatever the model’s shares are. Chance agreement is 0.333.
kappa = (0.687 − 0.333) ÷ (1 − 0.333) = 0.354 ÷ 0.667 = 0.53
That is moderate agreement. In the first lesson of this module, one careful person and the WANDS judges had about the same agreement (kappa 0.55 on 17 pairs). The weighted kappa, which counts a one-step disagreement as half, is 0.63.
A strict judge
Section titled “A strict judge”Look at where the disagreements are. Below the diagonal (to its left), the model gives a higher grade than the people: 7 + 0 + 21 = 28 pairs. Above the diagonal (to its right), it gives a lower grade: 50 + 5 + 58 = 113 pairs. This model is a strict judge. When it disagrees, it usually says “worse than the people said”.
The pairs where it disagrees most show why. Here are the five pairs that WANDS called exact and the model called irrelevant:
| Query | Product | WANDS | Model |
|---|---|---|---|
| leather chairs | aoting 26.5’’ wide polyester wingback chair | exact | irrelevant |
| leather chairs | murrieta upholstered dining chair | exact | irrelevant |
| papasan chair frame only | kylah 26’’ wide papasan chair | exact | irrelevant |
| marble | wiley frame 2 nesting tables | exact | irrelevant |
| benjiamino faux leather power lift chair | hortencia genuine leather conference chair | exact | irrelevant |
The model reads every word of the query literally. A polyester chair is not leather. A whole papasan chair is not “frame only”. Some of these WANDS grades look too high, and the model may be right. But the guidelines say that a missing detail makes a product partial, not irrelevant. On these pairs, both judges break the guidelines in opposite directions.
Which of its grades can you trust?
Section titled “Which of its grades can you trust?”Agreement is one number for the whole table. For a decision, ask a sharper question: when the model gives a grade, how often do the people agree?
Text description of the chart
A column chart with three columns. When the model says exact, the WANDS judges agree on 93% of the pairs (95 of 102). When it says partial: 54% (85 of 156). When it says irrelevant: 67% (129 of 192).
The model’s “exact” is reliable: 95 of 102. Its “partial” and “irrelevant” are not: the line between them is where it and the people disagree most. This is the same line where the careful person in the first lesson disagreed with WANDS.
One warning about these shares. The test pairs were chosen 150 per grade. Real search results are mostly partial: 62% of all WANDS judgments. On a natural mix, these shares would change.
Fill the holes, and say so
Section titled “Fill the holes, and say so”Now use the 300 unjudged pairs. The model called 22 of them exact, 75 partial and 203 irrelevant. Fill these labels into the judgments, and measure the three searches again on the same 180 English and French test queries.
Text description of the chart
Six rows of points. BM25 with text cleaning, spelling and synonyms: 0.57 with people’s judgments only, 0.573 with the model’s labels added. Search by meaning: 0.494, then 0.514. Hybrid: 0.585, then 0.602.
The method of each search did not change. Only the judgments did. The search with the most holes, search by meaning, gains the most (+0.020). The lexical search, with few holes, hardly moves (+0.003). The model mostly called the missing products irrelevant, so the change is small. That is a useful result too: the holes did not hide many good products.
Report both numbers, and say which judgments each one uses. For example: “hybrid 0.585 with people’s judgments, 0.602 with the model’s labels for unjudged results”.
Where the model is good enough, and where not
Section titled “Where the model is good enough, and where not”The model is good enough for some jobs and not for others.
Good enough:
- Filling holes, reported beside the people-only number. Its labels move a number by a little and show whether the holes hide good products.
- Its “exact” label. When it says exact, people agree 93% of the time. A list of model-exact products is a good first draft.
- Screening. It can grade thousands of new pairs cheaply. People then check a sample, starting with the pairs between partial and irrelevant.
Not good enough:
- Replacing people for the final test set. Kappa 0.53 is moderate, not substantial. The change from filling holes (+0.017 for the hybrid search) is as large as some differences this course compares. A change of judge alone could create a “winner”.
- The line between partial and irrelevant. Its partial is right only 54% of the time.
- French. All 450 checked pairs are English queries. Of the 200 unjudged French pairs, the model called 162 irrelevant. Nobody checked those labels against people, so we do not know if they are right.
- Results written by a similar model. In Synonyms and query rewriting, a language model rewrites queries. A judge from the same family may prefer its own style. We did not measure this, so treat it as a risk.
The evaluation course goes deeper into model judges: how to write the instructions, and how to check a judge against people before you trust it. See Model-based judging and Calibrate a model judge against human ratings in Evaluating and Testing AI Systems.
Guided practice: check the model judge
Section titled “Guided practice: check the model judge”-
In
larkfield-search, with the virtual environment active, compare the recorded model labels with the WANDS judges:Terminal window search judge llmRecorded model labels and the WANDS judges on 450 results (A = WANDS, B = model): agreement 69%, Cohen's kappa 0.53, weighted kappa 0.63judge B exact partial irrelevantjudge Aexact 95 50 5partial 7 85 58irrelevant 0 21 129 -
Find the cells below the diagonal (the model is higher) and above it (the model is lower). Add each group. You get 28 and 113.
-
Compare this line with your own
search judge agreeline from Relevance judgments. Who agrees more with WANDS: you or the model?
The table is the same as the one above. The agreement is printed rounded, 69%. The command reads the labels from the practice files (recorded/llm_judgments.jsonl), so it needs no key and no network.
Your turn: read the worst disagreements
Section titled “Your turn: read the worst disagreements”Change one thing: look at the pairs, not the counts. Predict: when WANDS says exact and the model says irrelevant, who is right more often? Start Python in larkfield-search with python, then type:
import pandas as pdfrom larkfield_search import pathsd = paths.data_dir()j = pd.read_json(d / "recorded" / "llm_judgments.jsonl", lines=True)c = pd.read_csv(d / "catalogue.csv.gz", usecols=["product_id", "title"])far = j[(j.human_grade == 2) & (j.llm_grade == 0)].merge(c, on="product_id")print(far[["query", "title"]].to_string(index=False))Type exit() to leave Python. Then change the filter to (j.human_grade == 1) & (j.llm_grade == 0) and read the 58 partial pairs that the model called irrelevant.
The output, and what to look for
query title leather chairs aoting 26.5 '' wide polyester wingback chair leather chairs murrieta upholstered dining chair papasan chair frame only kylah 26 '' wide papasan chair marble wiley frame 2 nesting tablesbenjiamino faux leather power lift chair hortencia genuine leather conference chairThese are the five pairs from the table above. Under Larkfield’s guidelines, most of them are partial: the right kind of thing with one detail wrong. Neither judge followed rule 2. In the 58 partial pairs, look for the same pattern: a missing detail that the model treats as “wrong thing”. A sentence in the model’s instructions that repeats rule 2 is the first thing to test.
Common mistake
A better number from a different judge
Symptom: Tomás reports: “Search by meaning went from 0.494 to 0.514 this week.” Omar is pleased. Amira asks what changed in the search. Nothing did.
Cause: Tomás filled the unjudged products with the model’s labels. The judgments changed, not the search. The two numbers are not comparable.
Fix: keep the judgments fixed when you compare two searches, and name them in every result: “people only” or “people + model labels”. When you change the judgments, measure every search again on the new judgments, and report the old and the new number side by side.
Consequence in search work: a test set is a measuring instrument. If you change the instrument and the thing you measure at the same time, you cannot tell which one moved the number.
Knowledge check
Section titled “Knowledge check”Check your understanding
The model agrees with the WANDS judges on 68.7% of 450 pairs, with kappa 0.53. Omar asks: “Can we stop paying people to judge?” What do you answer?
Show the answer
Not for the final test set. Kappa 0.53 is moderate agreement. The model is also strict: when it disagrees, it gives a lower grade on 113 pairs and a higher one on only 28. Its “exact” label is reliable (people agree 93% of the time). Its partial and irrelevant labels are not, and nobody checked it on French queries. Use it to fill holes and to screen new pairs, and keep people for the test set and for a sample of the model’s labels.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With unjudged products counted as irrelevant, search by meaning scores 0.494; with the model's labels, 0.514. BM25 with text cleaning moves only from 0.570 to 0.573. Why does one move more than the other?
Show the answer
Search by meaning has more unjudged products in its top 10: 63% are judged, against 71% for the lexical search. It finds products that share no word with the query, and the WANDS judges never saw many of them. More holes means more products that the model’s labels can change from 0 to a gain.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A model judge says exact on 102 pairs, and the people agree on 95 of them. It says partial on 156 pairs, and the people agree on 85. Compute both shares. Which label would you trust without a check?
Show the answer
Exact: 95 ÷ 102 = 0.93. Partial: 85 ÷ 156 = 0.54. You can trust its “exact” label much more than its “partial” label. Even so, check a sample: these pairs were chosen 150 per grade, and on real results the shares would change.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- New searches find products that nobody judged. In the top 10 of search by meaning, only 63% of the results have a judgment.
- An LLM judge grades query–product pairs from the guidelines. The course’s 750 labels from chat-small are recorded and cost about US$0.028.
- Check it on pairs that people judged: agreement 68.7%, kappa 0.53, weighted 0.63 on 450 pairs. It is strict: lower than the people on 113 pairs, higher on 28.
- Its “exact” is right 93% of the time; its “partial” (54%) and “irrelevant” (67%) are weaker. It was not checked on French.
- Filling holes moves hybrid from 0.585 to 0.602 and search by meaning from 0.494 to 0.514. Report both numbers, and never compare numbers made with different judgments.
You now have judgments, measures, a test set you understand and a way to fill its holes. In the next module, you start to change Larkfield’s search, beginning with how a search finds words fast: the inverted index.
Next: The inverted index