Skip to content
Français

Judgments from a language model

  • Lesson 7
  • 35 min
  • Module 2: Measure relevance before you change anything

Where we are

Our judgments come from people, and people are slow and expensive. A language model can write labels too. In this lesson, we compare a model's recorded labels with people's labels, and we decide where the model is good enough.

What you will be able to do

Decide where labels from a language model can replace or add to labels from people. You check a model’s recorded labels against the WANDS judges and find where it agrees. Then you use its labels to fill the holes in the test set, and you say so.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook reads the model’s recorded labels and builds the table against the WANDS judges. You compute how often each of its grades is right, then fill the unjudged results of one search. It needs no model key: the answers are recorded. About 30 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the guided practice)

  • The project larkfield-search with this module’s files, from the guided practice of Relevance judgments. No model key: the model’s answers are recorded in the practice files.

A furniture shop hires a new assistant. She is fast and cheap, and she never gets tired. On her first day, the manager does not let her serve customers alone. He gives her 450 customer requests that experienced staff have already answered, and he compares her answers with theirs.

Where she agrees with the experienced staff, he lets her work alone. Where she often disagrees, someone checks her work. The manager does not ask “Is she good?” He asks “Is she good enough for this job?”

A language model that writes relevance judgments is that new assistant. This lesson checks her work.

In Ranking measures, you met the rule: an unjudged product counts as irrelevant. The WANDS judges graded products that an older search showed them. A new search finds other products, and many of them were never graded.

This matters most for the searches that the course builds later. Search by meaning finds products that share no word with the query. Many of them have no judgment at all. On Larkfield’s English and French test queries:

Search Judged in the top 10
BM25 with text cleaning, spelling and synonyms (Synonyms and query rewriting) 71%
Search by meaning (Search by meaning) 63%
Hybrid of the two (Hybrid search) 72%

A third of the results of search by meaning have no grade. Each one counts as 0. If some are good products, the search looks worse than it is. Tomás says: “We are punishing the new search for finding products that nobody looked at.”

You could ask people to judge the missing products. That is the most reliable way. But each judgment needs a person to read the query, the title, the description and the attributes. For every new search and every new test query, the holes come back.

The idea: give a language model the same job as a person. A language model reads text and writes text. You give it the query, the product’s title, class and description, and the judging guidelines. You ask for one grade: 2, 1 or 0. A model used like this is often called an LLM judge (“LLM” means large language model).

For this course, the team asked chat-small, a small, low-cost chat model on Azure, for 750 labels. The answers were recorded once. The project and the notebook replay them, so you need no key and every run gives the same answers. The 750 labels cost about US$0.028 in all, about 4 cents per 1,000 labels.

The labels are in two groups:

  • 450 pairs that the WANDS judges already graded: 150 exact, 150 partial and 150 irrelevant. These test the model.
  • 300 unjudged pairs from the top 10 of search by meaning and the hybrid search, on the English and French test queries. These fill holes.

Never use the second group before you have checked the model on the first.

How to check a model judge. Take pairs that people already judged, 150 per grade. The model grades the same pairs (recorded answers). Compare with people: table, agreement, kappa. Good enough for this job? Yes: label unjudged results and report it beside the people-only number. No: ask people, or fix the guidelines and test again.
Check the model on pairs that people have judged before you let it label new pairs.
Text description of the diagram

A row of boxes. “Take pairs that people already judged, 150 per grade” leads to “The model grades the same pairs (recorded answers)”, then to “Compare with people: table, agreement, kappa”, then to a decision: “Good enough for this job?”. The “yes” arrow leads to a box with a thick border: “Label unjudged results; report it beside the people-only number”. The “no” arrow leads to a box with a red border: “Ask people, or fix the guidelines and test again”.

Here are the model’s grades for the 450 pairs, against the WANDS grades. Each row is a WANDS grade. Each column is the model’s grade.

WANDS judges ↓ / model → Exact Partial Irrelevant Pairs
Exact 95 50 5 150
Partial 7 85 58 150
Irrelevant 0 21 129 150
Model total 102 156 192 450

The diagonal (95, 85, 129) is where they agree: 309 of 450 pairs, so agreement 68.7%.

Now the kappa, as in Relevance judgments. The WANDS grades are one third each. So two random judges with these shares would agree on one third of the pairs, whatever the model’s shares are. Chance agreement is 0.333.

kappa = (0.687 − 0.333) ÷ (1 − 0.333) = 0.354 ÷ 0.667 = 0.53

That is moderate agreement. In the first lesson of this module, one careful person and the WANDS judges had about the same agreement (kappa 0.55 on 17 pairs). The weighted kappa, which counts a one-step disagreement as half, is 0.63.

Look at where the disagreements are. Below the diagonal (to its left), the model gives a higher grade than the people: 7 + 0 + 21 = 28 pairs. Above the diagonal (to its right), it gives a lower grade: 50 + 5 + 58 = 113 pairs. This model is a strict judge. When it disagrees, it usually says “worse than the people said”.

The pairs where it disagrees most show why. Here are the five pairs that WANDS called exact and the model called irrelevant:

Query Product WANDS Model
leather chairs aoting 26.5’’ wide polyester wingback chair exact irrelevant
leather chairs murrieta upholstered dining chair exact irrelevant
papasan chair frame only kylah 26’’ wide papasan chair exact irrelevant
marble wiley frame 2 nesting tables exact irrelevant
benjiamino faux leather power lift chair hortencia genuine leather conference chair exact irrelevant

The model reads every word of the query literally. A polyester chair is not leather. A whole papasan chair is not “frame only”. Some of these WANDS grades look too high, and the model may be right. But the guidelines say that a missing detail makes a product partial, not irrelevant. On these pairs, both judges break the guidelines in opposite directions.

Agreement is one number for the whole table. For a decision, ask a sharper question: when the model gives a grade, how often do the people agree?

When the model says exact, the WANDS judges agree 93% of the time. Its partial and irrelevant grades are much weaker.
Text description of the chart

A column chart with three columns. When the model says exact, the WANDS judges agree on 93% of the pairs (95 of 102). When it says partial: 54% (85 of 156). When it says irrelevant: 67% (129 of 192).

The model’s “exact” is reliable: 95 of 102. Its “partial” and “irrelevant” are not: the line between them is where it and the people disagree most. This is the same line where the careful person in the first lesson disagreed with WANDS.

One warning about these shares. The test pairs were chosen 150 per grade. Real search results are mostly partial: 62% of all WANDS judgments. On a natural mix, these shares would change.

Now use the 300 unjudged pairs. The model called 22 of them exact, 75 partial and 203 irrelevant. Fill these labels into the judgments, and measure the three searches again on the same 180 English and French test queries.

NDCG@10 of three searches on 180 English and French test queries, with unjudged products as irrelevant (grey) and with the model's labels filled in. The searches with more holes gain more.
Text description of the chart

Six rows of points. BM25 with text cleaning, spelling and synonyms: 0.57 with people’s judgments only, 0.573 with the model’s labels added. Search by meaning: 0.494, then 0.514. Hybrid: 0.585, then 0.602.

The method of each search did not change. Only the judgments did. The search with the most holes, search by meaning, gains the most (+0.020). The lexical search, with few holes, hardly moves (+0.003). The model mostly called the missing products irrelevant, so the change is small. That is a useful result too: the holes did not hide many good products.

Report both numbers, and say which judgments each one uses. For example: “hybrid 0.585 with people’s judgments, 0.602 with the model’s labels for unjudged results”.

Where the model is good enough, and where not

Section titled “Where the model is good enough, and where not”

The model is good enough for some jobs and not for others.

Good enough:

  • Filling holes, reported beside the people-only number. Its labels move a number by a little and show whether the holes hide good products.
  • Its “exact” label. When it says exact, people agree 93% of the time. A list of model-exact products is a good first draft.
  • Screening. It can grade thousands of new pairs cheaply. People then check a sample, starting with the pairs between partial and irrelevant.

Not good enough:

  • Replacing people for the final test set. Kappa 0.53 is moderate, not substantial. The change from filling holes (+0.017 for the hybrid search) is as large as some differences this course compares. A change of judge alone could create a “winner”.
  • The line between partial and irrelevant. Its partial is right only 54% of the time.
  • French. All 450 checked pairs are English queries. Of the 200 unjudged French pairs, the model called 162 irrelevant. Nobody checked those labels against people, so we do not know if they are right.
  • Results written by a similar model. In Synonyms and query rewriting, a language model rewrites queries. A judge from the same family may prefer its own style. We did not measure this, so treat it as a risk.

The evaluation course goes deeper into model judges: how to write the instructions, and how to check a judge against people before you trust it. See Model-based judging and Calibrate a model judge against human ratings in Evaluating and Testing AI Systems.

  1. In larkfield-search, with the virtual environment active, compare the recorded model labels with the WANDS judges:

    Terminal window
    search judge llm
    Recorded model labels and the WANDS judges on 450 results (A = WANDS, B = model): agreement 69%, Cohen's kappa 0.53, weighted kappa 0.63
    judge B exact partial irrelevant
    judge A
    exact 95 50 5
    partial 7 85 58
    irrelevant 0 21 129
  2. Find the cells below the diagonal (the model is higher) and above it (the model is lower). Add each group. You get 28 and 113.

  3. Compare this line with your own search judge agree line from Relevance judgments. Who agrees more with WANDS: you or the model?

The table is the same as the one above. The agreement is printed rounded, 69%. The command reads the labels from the practice files (recorded/llm_judgments.jsonl), so it needs no key and no network.

Change one thing: look at the pairs, not the counts. Predict: when WANDS says exact and the model says irrelevant, who is right more often? Start Python in larkfield-search with python, then type:

import pandas as pd
from larkfield_search import paths
d = paths.data_dir()
j = pd.read_json(d / "recorded" / "llm_judgments.jsonl", lines=True)
c = pd.read_csv(d / "catalogue.csv.gz", usecols=["product_id", "title"])
far = j[(j.human_grade == 2) & (j.llm_grade == 0)].merge(c, on="product_id")
print(far[["query", "title"]].to_string(index=False))

Type exit() to leave Python. Then change the filter to (j.human_grade == 1) & (j.llm_grade == 0) and read the 58 partial pairs that the model called irrelevant.

The output, and what to look for
query title
leather chairs aoting 26.5 '' wide polyester wingback chair
leather chairs murrieta upholstered dining chair
papasan chair frame only kylah 26 '' wide papasan chair
marble wiley frame 2 nesting tables
benjiamino faux leather power lift chair hortencia genuine leather conference chair

These are the five pairs from the table above. Under Larkfield’s guidelines, most of them are partial: the right kind of thing with one detail wrong. Neither judge followed rule 2. In the 58 partial pairs, look for the same pattern: a missing detail that the model treats as “wrong thing”. A sentence in the model’s instructions that repeats rule 2 is the first thing to test.

Common mistake

A better number from a different judge

Symptom: Tomás reports: “Search by meaning went from 0.494 to 0.514 this week.” Omar is pleased. Amira asks what changed in the search. Nothing did.

Cause: Tomás filled the unjudged products with the model’s labels. The judgments changed, not the search. The two numbers are not comparable.

Fix: keep the judgments fixed when you compare two searches, and name them in every result: “people only” or “people + model labels”. When you change the judgments, measure every search again on the new judgments, and report the old and the new number side by side.

Consequence in search work: a test set is a measuring instrument. If you change the instrument and the thing you measure at the same time, you cannot tell which one moved the number.

Check your understanding

The model agrees with the WANDS judges on 68.7% of 450 pairs, with kappa 0.53. Omar asks: “Can we stop paying people to judge?” What do you answer?

Show the answer

Not for the final test set. Kappa 0.53 is moderate agreement. The model is also strict: when it disagrees, it gives a lower grade on 113 pairs and a higher one on only 28. Its “exact” label is reliable (people agree 93% of the time). Its partial and irrelevant labels are not, and nobody checked it on French queries. Use it to fill holes and to screen new pairs, and keep people for the test set and for a sample of the model’s labels.

Check your understanding

With unjudged products counted as irrelevant, search by meaning scores 0.494; with the model's labels, 0.514. BM25 with text cleaning moves only from 0.570 to 0.573. Why does one move more than the other?

Show the answer

Search by meaning has more unjudged products in its top 10: 63% are judged, against 71% for the lexical search. It finds products that share no word with the query, and the WANDS judges never saw many of them. More holes means more products that the model’s labels can change from 0 to a gain.

Check your understanding

A model judge says exact on 102 pairs, and the people agree on 95 of them. It says partial on 156 pairs, and the people agree on 85. Compute both shares. Which label would you trust without a check?

Show the answer

Exact: 95 ÷ 102 = 0.93. Partial: 85 ÷ 156 = 0.54. You can trust its “exact” label much more than its “partial” label. Even so, check a sample: these pairs were chosen 150 per grade, and on real results the shares would change.

Recap

  • New searches find products that nobody judged. In the top 10 of search by meaning, only 63% of the results have a judgment.
  • An LLM judge grades query–product pairs from the guidelines. The course’s 750 labels from chat-small are recorded and cost about US$0.028.
  • Check it on pairs that people judged: agreement 68.7%, kappa 0.53, weighted 0.63 on 450 pairs. It is strict: lower than the people on 113 pairs, higher on 28.
  • Its “exact” is right 93% of the time; its “partial” (54%) and “irrelevant” (67%) are weaker. It was not checked on French.
  • Filling holes moves hybrid from 0.585 to 0.602 and search by meaning from 0.494 to 0.514. Report both numbers, and never compare numbers made with different judgments.

You now have judgments, measures, a test set you understand and a way to fill its holes. In the next module, you start to change Larkfield’s search, beginning with how a search finds words fast: the inverted index.

Next: The inverted index