Relevance judgments
- Lesson 4
- 40 min
- Module 2: Measure relevance before you change anything
Where we are
We found queries where Larkfield's search fails. Before we change anything, we need a way to say which result is good. In this lesson, we use graded relevance judgments, judging guidelines and agreement between judges.
Lesson 4 of 30 · Search and Ranking: Find the Best MatchRelevance judgments
1. The search problem
2. Measure relevance before you change anything
3. Lexical search: the baseline to beat
4. Understand the query
5. Dense and hybrid retrieval
6. Re-rank the top results
7. The parts of search that people see
8. Run search in production
What you will be able to do
Decide which search results are good with written grades, not with a feeling. With a grade for each product and query, you can measure every later change the same way. Another person can also check your work.
NotebookDo this lesson in a notebook
The notebook reads the judgments of one query, compares two judges in a table and asks you to compute Cohen’s kappa by hand. Its setup cells download the practice data, so it needs no local setup. About 35 minutes.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- The Larkfield catalogue, the queries and the synthetic query log, from Meet the catalogue and the query log.
- The kinds of search (known-item, specific, broad), from Kinds of search.
What you need on your computer (only for the guided practice; the notebook needs none of it)
- The project folder
larkfield-searchwith its.venvenvironment, and the practice files in../nextia-learning-labsbeside it, from Meet the catalogue and the query log.search checkmust end withSetup check passed.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) commands are given but were not run by the course team.
A customer and a shop assistant
Section titled “A customer and a shop assistant”A customer walks into a furniture shop and says: “I want a wooden chair for outside.” The shop assistant brings five things. The first is a wooden patio chair with cushions. The second is a metal garden chair. The third is a wooden rocking chair for a porch. The fourth is a cushion for a dining chair. The fifth is a child’s desk with a chair.
You watch, and without thinking much you sort them. The patio chair and the rocking chair are what she asked for. The metal chair is close: a chair for outside, but not wooden. The cushion is only an accessory, and the desk is the wrong thing. You give each item a quiet grade: right, close or wrong.
That quiet grade is the start of this whole module. A search engine is the shop assistant. To know if it is good, someone must grade each product that it brings. Is the product right, close or wrong for the words that the customer typed?
“It looks better to me”
Section titled ““It looks better to me””In Meet the catalogue and the query log, you met two searches. Larkfield’s search today (current) shows a product only when every word of the query is in it. BM25 (bm25) lets any word match and ranks the products by a score. Tomás, the AI engineer, tries both on the shop’s most popular queries. He types “bee”, “rack glass” and “dumbbells”, looks at the first page, and says: “BM25 is no better. I would keep what we have.”
Priya, the data scientist, measures the same two searches in another way, which you learn in this module. She scores both on the 243 test queries, using written grades. The result splits by band. The head is the 40 most searched texts of the log, the torso the next 120, and the tail all the others.
Text description of the chart
A column chart with three columns. Head (the 40 most searched texts, 16 test queries): −0.015. Torso (the next 120 texts, 41 test queries): +0.190. Tail (every other text, 184 test queries): +0.265. The axis goes from −0.05 to 0.3. The head column is just below zero; the other two are high above it.
Tomás was right about what he saw. On the popular queries, BM25 is a little worse (−0.015). But most of Larkfield’s different queries are in the tail, and there BM25 is much better (+0.265). On all test queries together, BM25 is better by +0.234. The number on the axis is NDCG@10, a score from 0 to 1 that the next lesson explains. Here, only the direction matters.
Tomás looked at a few queries and trusted his eyes. His eyes were honest, but they saw only a small, special part of the traffic, and nobody could check them. Priya’s number is different. Anyone can run it again on the same queries and get the same answer.
Text description of the diagram
The diagram starts from one box, “A change to the search”, and splits into two paths. The upper path goes to “Look at a few popular queries”, then to “It looks better to me”, and ends in a box with a red border: “No number: nobody can check it”. The lower path goes to “Run every judged test query”, then to “Score each list with the judgments”, and ends in a box with a thick border: “One number with an interval. BM25 minus today: +0.234 (+0.197 to +0.270)”.
To score a list, you need the written grades first. They have a name.
Relevance judgments
Section titled “Relevance judgments”A relevance judgment is a grade that says how well one product answers one query. “Relevant” means “useful for what the person asked”. A judgment is always about a pair: the same product can be exact for “wooden chair outdoor” and irrelevant for “dining table”.
Larkfield uses three grades, the grades of the shop assistant:
| Grade | Name | Meaning |
|---|---|---|
| 2 | Exact | The product is what the person asked for, with every detail in the query. |
| 1 | Partial | The right kind of thing, but one detail is different or missing, or a close substitute. |
| 0 | Irrelevant | Not what the person asked for: another kind of product, or an accessory or part of it. |
These are graded judgments: they have more than two levels. A yes/no grade would put the metal garden chair with the patio chair, or with the dining cushion. Neither is fair. The middle grade keeps the difference between “close” and “wrong”, and the ranking measures of the next lesson use it.
A tiny example: “wooden chair outdoor”
Section titled “A tiny example: “wooden chair outdoor””Here are six real results for the query “wooden chair outdoor” (W-367), with the grades of the WANDS judges:
| Product | Product class | Grade | Why |
|---|---|---|---|
| patio chair with cushions (acacia wood) | Patio Lounge Chairs | 2 exact | A wooden chair for outside |
| abdool wooden rocking chair | Patio Rockers & Gliders | 2 exact | A wooden chair for a patio |
| cooney wooden porch rocking chair | Patio Rockers & Gliders | 2 exact | A wooden chair for a porch |
| keating 28’’ wide outdoor chair with cushions (aluminium) | Patio Lounge Chairs | 1 partial | Outdoor chair, but metal, not wood |
| biergh wooden 31’’ writing desk and chair set | Kids Desks | 1 partial | A wooden chair, but indoors, and part of a desk set |
| memory foam honeycomb … dining chair cushion | Furniture Cushions | not judged | — |
Look at the desk set. The WANDS judges called it partial. Many people would call it irrelevant: it is a child’s desk for a bedroom. That is not a mistake in the table. It shows that judging needs rules, and that two careful people can still disagree. The rest of this lesson is about both.
Look also at the last row. Nobody judged the cushion for this query. This happens often, and the course counts an unjudged product as irrelevant. Judgments from a language model comes back to this choice.
Who writes the judgments
Section titled “Who writes the judgments”Larkfield’s judgments are real. They come from WANDS, the Wayfair ANnotation DataSet (MIT licence). People hired by the furniture shop Wayfair graded products for 480 real customer queries. The course renamed the products as Larkfield’s, and kept the queries and the grades.
Text description of the chart
A column chart with three columns: exact 36,109, partial 180,879, irrelevant 74,242. The partial column is about five times the height of the exact column.
The course has 291,230 judgments in all, for 624 public queries. A WANDS query has a median of 199 judged products. The 130 French queries were machine-translated from English queries, and they share the English queries’ judgments. Some other queries are practice problems we made on purpose, such as misspellings and product codes. Every query has a split: train, validation or test.
Who writes judgments in a real company? Often a small team of trained people, or the search team itself, or a paid service. Each judgment takes time: the person must read the query, the title, the description and the attributes. That is why every judgment matters, and why judges need guidelines.
Judging guidelines
Section titled “Judging guidelines”Judging guidelines are the written rules that every judge follows. Without them, each person brings their own idea of “close”. With them, two people grade the same pair the same way more often, and a new judge can learn the job.
The project has Larkfield’s guidelines in docs/judging-guidelines.md. They define the three grades as in the table above, and they add rules for hard cases:
- Accessories are irrelevant. “sofa” → a sofa cover or a sofa table is 0, even though the word matches.
- A missing detail makes it partial. “grey outdoor cushion” → a blue outdoor cushion is 1.
- A different size is partial, unless the size is the whole point of the query.
- Brand or name queries: “kisner coffee table” → the Kisner coffee table is 2; another coffee table is 1; a Kisner sofa is 0.
- If you cannot tell, read the description and the attributes. If you still cannot tell, choose the lower grade and write a note.
- Language does not matter: a French query and an English product can be an exact match.
Rule 1 matters most for search. A search that matches words loves accessories: “sofa cover” contains “sofa”. The guidelines say clearly that the cover does not answer the query.
Omar, the product manager, asks: “Why not let the customers judge, with their clicks?” Clicks are useful, and Learn from clicks uses them. But a customer can only click what the search showed, and people click the first result often whether it is good or not. A judgment answers a cleaner question: is this product right for this query?
Agreement between judges
Section titled “Agreement between judges”Guidelines help, but people still disagree, as the desk set showed. Before you trust a set of judgments, you measure how often two judges agree.
A tiny example: two judges, ten pairs
Section titled “A tiny example: two judges, ten pairs”Two judges, A and B, grade the same ten query–product pairs:
| Pair | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Judge A | 2 | 2 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 1 |
| Judge B | 2 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 |
| Same? | yes | no | yes | yes | no | yes | yes | yes | no | yes |
They give the same grade on 7 of 10 pairs. That is the raw agreement: 0.70.
But two judges agree sometimes by luck. Imagine that both grade at random, and each uses each grade as often as here. Judge A gives exact to 2 of 10 pairs, partial to 5 and irrelevant to 3. Judge B gives exact to 1, partial to 6 and irrelevant to 3. The chance that both say “exact” on a pair is 0.2 × 0.1 = 0.02. Both “partial”: 0.5 × 0.6 = 0.30. Both “irrelevant”: 0.3 × 0.3 = 0.09. So random judges would agree on 0.02 + 0.30 + 0.09 = 0.41 of the pairs.
Cohen’s kappa (κ, said “kappa”) keeps only the agreement above that chance level:
kappa = (observed − chance) ÷ (1 − chance) = (0.70 − 0.41) ÷ (1 − 0.41) = 0.29 ÷ 0.59 = 0.49
A kappa of 1 means perfect agreement. A kappa of 0 means no better than chance. A common rough guide calls 0.41 to 0.60 “moderate” agreement and 0.61 to 0.80 “substantial”. The name comes from Jacob Cohen, who published it in 1960.
There is one more version. Look at pair 2: one judge said exact, the other partial. Pair 5: partial against irrelevant. Both disagreements are one step apart. An “exact” against “irrelevant” would be much worse. The weighted kappa counts a one-step disagreement as half a disagreement, so it is usually higher. The project prints both.
Why agreement matters
Section titled “Why agreement matters”Why measure agreement at all? Because it sets a limit. Suppose that two careful judges agree on only 70% of the pairs. Then the grades of one judge contain some noise, and no search can remove it. It also tells you if the guidelines work. A low kappa often means that a rule is missing, like rule 1 about accessories.
Guided practice: judge 30 results
Section titled “Guided practice: judge 30 results”You now grade 30 results yourself and compare your grades with the WANDS judges. First, you add this module’s files to your project.
-
In
larkfield-search, with the virtual environment active, copy the files of this module from the snapshotend-of-m02. They are the judging code, the comparison command of a later lesson, the guidelines, their tests and the project’s README. If you downloaded the labs as a ZIP file, writenextia-learning-labs-maininstead ofnextia-learning-labs, here and in later modules.Terminal window S=../nextia-learning-labs/C38/snapshots/end-of-m02mkdir -p docsfor f in larkfield_search/judge.py larkfield_search/commands/judge.py larkfield_search/commands/compare.py docs/judging-guidelines.md tests/test_judge.py README.md; docp "$S/$f" "$f"doneTerminal window $S = "..\nextia-learning-labs\C38\snapshots\end-of-m02"New-Item -ItemType Directory -Force docs | Out-Nullforeach ($f in 'larkfield_search\judge.py', 'larkfield_search\commands\judge.py', 'larkfield_search\commands\compare.py', 'docs\judging-guidelines.md', 'tests\test_judge.py', 'README.md') { Copy-Item "$S\$f" $f } -
Run the tests. They use a tiny fake data folder, so they need no index:
Terminal window python -m pytest -q................ [100%]16 passed in 0.86sThe start of the module had 12 tests; the four new ones test the judging code. Your time will differ.
-
Write a sample of results to judge:
Terminal window search judge sampleWrote 30 results of 3 queries to outputs/judge_sample.csv.Fill the column your_grade with 2 (exact), 1 (partial) or 0 (irrelevant), following docs/judging-guidelines.md. Then run: search judge agreeThe sample is BM25’s top 10 for three validation queries: “bedroom wall decor floral, multicolored with some teal (prints)”, “fleur de lis living candle wall sconce bronze” and “wooden chair outdoor”.
-
Open
docs/judging-guidelines.mdand read it once. -
Open
outputs/judge_sample.csvin a spreadsheet program or in VS Code. Each row has the query, the product’s title, class, a part of its description and its attributes. -
Fill the column
your_gradewith 2, 1 or 0 for each row. Do not look at the WANDS grades first. Save the file as CSV. -
Compare your grades with the WANDS judges:
Terminal window search judge agreeHere is the output for one set of grades, written while we prepared this lesson:
You and the WANDS judges on 17 results (A = WANDS, B = you): agreement 71%, Cohen's kappa 0.55, weighted kappa 0.65judge B exact partial irrelevantjudge Aexact 4 0 0partial 0 4 3irrelevant 0 2 4Your numbers will differ: they depend on your grades.
Read the output in three parts. First, only 17 of the 30 results count. The other 13 have no WANDS judgment, so there is nothing to compare them with. Second, the summary line: these grades agree with WANDS on 71% of the 17 pairs, with a kappa of 0.55, which is moderate. Third, the table. Its rows are the WANDS grades, and its columns are yours. The diagonal (4, 4, 4) is where the two agree.
The table shows where the disagreement is. Every pair that WANDS called exact was also exact here. All five disagreements are between partial and irrelevant. Three products that WANDS called partial were irrelevant here. Two are the child’s desk sets for “wooden chair outdoor”. One is a fleur-de-lis cake stand for a candle sconce. Rule 1 of the guidelines pushed this judge to “irrelevant”. The WANDS judges gave those a higher grade. Neither side is wrong. The disagreement says that the line between partial and irrelevant needs a clearer rule.
Your turn: another sample
Section titled “Your turn: another sample”Change one thing: the sample. Run search judge sample --seed 2. It writes 30 results of three other validation queries, starting with “sunflower”. Before you grade them, predict: will your agreement with WANDS be higher or lower than on the first sample? Grade the 30 results, run search judge agree, and find the cell of the table with the most disagreements.
What to look for
A single word like “sunflower” can mean many kinds of product: wall art, a rug, a garden statue, seeds. The guidelines say that the product must be “what the person asked for”. With a broad query, judges disagree more about what that is, so agreement is often lower than on a long, precise query. If most of your disagreements are again between partial and irrelevant, write the rule that would settle them. That rule is your first improvement to the guidelines.
Common mistake
Words instead of numbers in the grade column
Symptom: Tomás types exact in the first row instead of 2, saves the file and runs search judge agree. It stops with a long error that ends:
ValueError: invalid literal for int() with base 10: 'exact'If he leaves every grade empty, it says instead:
No judged rows with a WANDS judgment yet: fill your_grade first.Cause: the command reads the column as whole numbers. It skips empty cells, but a word is not a number, and Python cannot turn it into one.
Fix: use only 2, 1 or 0 in your_grade, and leave a cell empty if you skip a row. Then run search judge agree again.
Consequence in search work: grades are data. A typing mistake in one cell can stop a measurement. Worse, it can change the measurement without an error, for example a 3 where the scale ends at 2. Check the values of a judgment file before you use it: only the allowed grades, and the number of empty cells that you expect.
Knowledge check
Section titled “Knowledge check”Check your understanding
Tomás looks at the first page of results for the shop's five most common queries and says that the new search is no better. Why is this not a measure? What would you do instead?
Show the answer
It is a feeling about a few queries. Nobody can check it or repeat it, and the five most common queries are a special part of the traffic. On Larkfield’s test queries, BM25 is a little worse than today’s search on the head (−0.015) but much better on the tail (+0.265). Instead: grade the results of many test queries with written guidelines, score both searches on the same queries with those grades, and compare the scores.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
For the query “grey outdoor cushion”, grade these three products with Larkfield's guidelines and say why: a grey outdoor seat cushion; a blue outdoor seat cushion; a cover for an outdoor sofa.
Show the answer
- Grey outdoor seat cushion: 2 (exact). It is what the person asked for, with every detail: grey, outdoor, cushion.
- Blue outdoor seat cushion: 1 (partial). The right kind of thing with one detail different, the colour (rule 2).
- Cover for an outdoor sofa: 0 (irrelevant). Another kind of product. It is an accessory for outdoor furniture, not a cushion (rule 1).
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two judges grade 10 pairs and agree on 8. If they graded at random with the same shares of each grade, they would agree on 0.50 of the pairs. Compute Cohen's kappa. Is the agreement better than moderate?
Show the answer
kappa = (0.80 − 0.50) ÷ (1 − 0.50) = 0.30 ÷ 0.50 = 0.60. That is at the top of the “moderate” band (0.41 to 0.60) of the common rough guide. It is not yet “substantial”. The raw agreement of 80% looks better than it is, because half of the pairs would agree by chance.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- “It looks better to me” is not a measure. Tomás saw the popular queries, where BM25 is a little worse (−0.015); on all test queries it is better by +0.234.
- A relevance judgment grades one product for one query: exact (2), partial (1) or irrelevant (0). Larkfield’s 291,230 judgments come from the WANDS judges; an unjudged product counts as irrelevant.
- Judging guidelines are the written rules for every judge. The most important rule for search: accessories are irrelevant.
- Judges disagree. Raw agreement counts the same grades; Cohen’s kappa removes the agreement that chance would give: (observed − chance) ÷ (1 − chance). The weighted kappa counts a one-step disagreement as half.
search judge samplewrites 30 results to grade;search judge agreecompares your grades with the WANDS judges.
You can now grade results and check the grades. But a ranked list has ten grades, and you need one number for the whole list. Next, you turn the grades of a list into precision, recall, MRR and NDCG.
Next: Ranking measures