Skip to content
NextiaNextia Learning

Evaluate summaries against expert ratings

  • Case study 2
  • 120 min
  • Optional · 1 case study needed for the certificate
Case study 2 of 3 · Evaluating and Testing AI SystemsEvaluate summaries against expert ratings
Course overview

1. Define what good means

  1. Map goals to evidence
  2. Choose evaluation layers
  3. Establish a baseline

2. Build an evaluation dataset

  1. Select representative cases
  2. Label and version
  3. Avoid contamination

3. Choose component metrics

  1. Prediction metrics
  2. Retrieval metrics
  3. Generated-output metrics

4. Use human and model judges

  1. Write a usable rubric
  2. Calibrate reviewers
  3. Model-based judging
  4. Model-based judging, continued

5. Compare systems responsibly

  1. Controlled experiments
  2. Uncertainty and slices
  3. Trade-offs and regressions

6. Automate evaluation and feedback

  1. Harness structure
  2. Regression checks
  3. Production feedback

7. Case studies

  1. Calibrate a model judge against human ratings
  2. Evaluate summaries against expert ratings
  3. Regression-test a classifier release by slice

What you will be able to do

Find out which automatic measures of summary quality agree with experts, and on which quality. You compare cheap measures and two model judges with the ratings of three summarisation experts, with intervals. Then you decide which checks a team can automate, and which still need a person.

Case studyPortfolio piece

Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Rank correlation
  • Bootstrap by article
  • Reference-based and reference-free measures
  • Model judge with a rubric per quality
  • Dev and holdout splits
  • Reading failures
  • What to automate
Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is an analysis, so it comes as a notebook. The notebook downloads the ratings and the articles from their sources, computes every free measure itself, and replays the model judges from recordings. You need no account and no key. It runs in a few minutes, most of it downloads. A live judge call, to a local model or with your own key, is optional at the end.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses these lessons most:

You do not need the course project. This case study does not change it.

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it in VS Code, make a new virtual environment. Install pandas, scipy, matplotlib, pyarrow, rouge-score 0.1.2 and sentence-transformers. The last one also installs PyTorch (about 1 GB). The setup cell makes a folder summary-eval; to start again, delete that folder.

We tested it in Jupyter on macOS, from a fresh start. The versions: Python 3.12, scipy 1.18.1, rouge-score 0.1.2 and sentence-transformers 6.1.0. We did not run it in Colab or Kaggle.

Models change fast. The judge scores, prices and times on this page are a snapshot. The method is what lasts.

The situation

Section titled “The situation”

Kestrel Briefings is a small company. Every morning it sends its clients a digest of short summaries of the day’s news. A summarisation model writes about 400 summaries a night. Before 7 a.m., two editors read every one of them, and fix or remove the bad ones. It is slow, and the editors are tired.

Daniel, the machine learning engineer, wants to automate the check. “Every paper reports ROUGE. We compute it for each summary and send the editors only the low scores.” Ines, the editor-in-chief, is not convinced. “My editors look at four things. Does it read as one text? Is every fact right? Are the sentences clean? Did it keep what matters? Which of your numbers agrees with my editors, and on which of the four?”

Each error has a cost. A wrong fact in a client’s digest costs trust, and sometimes a correction. A clumsy sentence costs little. A summary that misses the point wastes the client’s time. So Ines cares most about facts.

Kestrel has no expert ratings of its own yet, and collecting them takes weeks. But a public dataset has what they need to learn the method first: news summaries rated by experts on these same four qualities.

What you will build

Section titled “What you will build”

You will build a scorecard. It compares each automatic measure with the experts, quality by quality, on summaries that no choice was tuned on. Then you read the summaries behind the numbers, and run the chosen checks on a new article. Last, you write a recommendation: which checks to automate, and which need an editor.

One summary of one news article. To the left, three experts rate it from 1 to 5 on four qualities: coherence, consistency, fluency, relevance. To the right, the automatic measures: free ones on your computer (length; ROUGE, which needs a reference; copied word pairs, which needs the article; embedding similarity) and a paid one, a model judge with a rubric, which needs the article. Below: for each quality, the 640 holdout summaries are ordered by the experts' mean and by the measure, and the two orders are compared with a rank correlation and an interval.
What is compared with what. Each measure gives every summary a number; the experts give it four ratings. For each quality: do the two orders agree?
Text description of the diagram

One summary of one news article goes to two sides. Three experts rate it from 1 to 5 on four qualities: coherence, consistency, fluency and relevance. The automatic measures give it one number each. The free ones run on your computer: length; ROUGE, which needs a reference summary; copied word pairs, which needs the article; and embedding similarity. The paid one is a model judge with a rubric, which needs the article. For each quality, the 640 holdout summaries are ordered by the experts’ mean and by the measure. A rank correlation, with an interval, says how well the two orders agree.

The skills come from the whole course:

The data

Section titled “The data”

The data is SummEval, from a 2021 study of how to evaluate summaries. Its authors took 100 news articles from the test set of the CNN/DailyMail dataset. They added the summaries of 16 summarisation systems from 2017 to 2020: 1,600 summaries. Then they had every summary rated.

One row is one summary of one article. The experts rated four qualities from 1 (worst) to 5 (best), defined in the paper:

Quality What the experts rated
Coherence The summary as a whole: well organised, building from sentence to sentence
Consistency The facts: every statement is supported by the article, nothing invented
Fluency Each sentence on its own: no fragments, no broken grammar
Relevance The choice of content: the important points, without repetition or side details

Most systems wrote in lower case, with spaces around punctuation. That is their raw output, and the experts saw it too.

Who rated

Section titled “Who rated”

The paper says who made the ratings, and it matters here. The experts were three people “who have written papers on summarization either for academic conferences (2) or as part of a senior thesis (1)”. Each of them rated every summary, in two rounds. In the second round, each expert checked again any score that was more than 2 points from the others. While doing so, they could see the others’ first scores. That made their agreement higher.

The paper also collected crowd ratings: five workers from Amazon Mechanical Turk per summary, with the same instructions.

No rating in this dataset comes from the course. The model judge scores later on this page are model outputs, recorded by the course. They are never human ratings.

First checks

Section titled “First checks”

The notebook loads the ratings, joins them to the articles by ID, and counts the summaries by their experts’ mean, rounded:

     coherence  consistency  fluency  relevance
1.0         38           35       10         11
2.0        326           82       39        120
3.0        422           37       82        362
4.0        579           59      178        837
5.0        235         1387     1291        270

What this tells us. All 100 articles were found, with 16 summaries each. Coherence and relevance spread over the scale. Consistency and fluency do not: 1,387 of 1,600 summaries round to 5 on consistency, and 1,291 on fluency. Factual errors are rare here, so any result on consistency rests on a small number of faulty summaries. Kestrel’s editors know this pattern: most summaries are fine, and the job is to find the few that are not.

Frame the problem

Section titled “Frame the problem”

The unit is one summary. The target is the mean of the three experts’ ratings, for each quality separately. A measure does not need the experts’ scale. It needs to put the summaries in the same order: better summaries higher.

The split is by article, because the 16 summaries of one article share its topic and its difficulty. Ten dev articles (160 summaries) are where the judge’s prompt was written and tried. Forty holdout articles (640 summaries) are where every measure is compared, once. The other 50 articles were not sent to a judge, to keep the recording within the case study’s budget of US$5.

What each measure needs is part of the framing. ROUGE needs a reference summary written by a person, and Kestrel has none for tomorrow’s news. Copied word pairs and the judge need only the article. Length needs nothing. A measure that needs a reference cannot run on new summaries without more human work.

The new idea: compare orders, not scores

Section titled “The new idea: compare orders, not scores”

Two wine tasters

Section titled “Two wine tasters”

Two wine tasters score the same five wines. One scores from 1 to 5, the other from 0 to 100. Their numbers never match, but they may agree completely: the same wine is first for both, and the same wine is last. To compare them, you compare their orders.

The problem: the scales differ

Section titled “The problem: the scales differ”

One summary has ROUGE-2 0.308 and an expert relevance of 4.67. Is that agreement? The question has no answer. ROUGE lives between 0 and 1, the experts between 1 and 5, a judge on whole numbers. And a measure can rise with quality without following a straight line. An ordinary correlation of the raw numbers mixes these effects.

The idea and its name

Section titled “The idea and its name”

Spearman’s rank correlation replaces each number by its rank in its own list. Then it measures how well the two lists of ranks agree. It is +1 when the orders are the same, 0 when they are unrelated, and −1 when one order is the reverse of the other. Ties share a rank. Its symbol is ρ (rho).

A tiny worked example

Section titled “A tiny worked example”

Here are five real summaries of one article, about an auction of a dress from the film Gone With the Wind. On the left: the experts’ mean relevance. On the right: ROUGE-2, which “The measures” explains below.

Two columns of five summaries, each in its own order. Experts (relevance, mean of 3): 1. E 4.67, 2. A 4.00, 3. C 3.67, 4. D 3.33, 5. B 3.00. ROUGE-2: 1. E 0.308, 2. C 0.305, 3. B 0.265, 4. A 0.257, 5. D 0.179. Lines join each summary in the two columns; only E's line is straight. Rank differences, experts minus ROUGE-2: A 2 − 4 = −2, B 5 − 3 = +2, C 3 − 2 = +1, D 4 − 5 = −1, E 0. The squares add up to 10. Spearman = 1 − 6 × 10 / (5 × (25 − 1)) = 1 − 60 / 120 = 0.5.
Spearman's rank correlation by hand, on five real summaries of one article. Only E keeps its place; each crossing line is a disagreement. ρ = 0.5.
Text description of the diagram

Summary E is first in both orders. A is second for the experts and fourth for ROUGE-2. B is last for the experts and third for ROUGE-2. C is third and second; D is fourth and fifth. The rank differences are −2, +2, +1, −1 and 0. Their squares add up to 10. With no ties, Spearman equals 1 − 6 × (the sum of the squared differences) / (n × (n² − 1)) = 1 − 60 / 120 = 0.5.

So ROUGE-2 and the experts agree partly on these five. One article is far too few to judge a measure. The real comparison uses 640 summaries, and an interval.

The interval: resample articles

Section titled “The interval: resample articles”

A correlation on 640 summaries is still an estimate. The bootstrap from Uncertainty and slices gives its interval, with one change. The 640 summaries are not 640 independent cases: they are 16 summaries of each of 40 articles. So each resample draws 40 articles with replacement and keeps all 16 summaries of each. There are 2,000 resamples (seed 2026). The interval is the middle 95% of their correlations.

Explore what the decision needs: how far do people agree?

Section titled “Explore what the decision needs: how far do people agree?”

Before any machine, look at the people. A measure cannot agree with the experts much more than the experts agree with each other. The notebook compares each expert with the mean of the other two, on the holdout, and averages the three results. That is the ceiling. It also compares the crowd’s mean with the experts’ mean.

How far people agree with the experts' mean, per quality, on the 640 holdout summaries. Blue: one expert against the other two (the ceiling). Orange: the mean of five crowd workers. 95% intervals, articles resampled.
Text description of the chart

Spearman with the experts’ mean, with 95% intervals. Coherence: one expert +0.73 (+0.69 to +0.76), the crowd +0.04 (−0.04 to +0.12). Consistency: one expert +0.75 (+0.69 to +0.81), the crowd −0.03 (−0.10 to +0.05). Fluency: one expert +0.59 (+0.53 to +0.66), the crowd +0.04 (−0.02 to +0.11). Relevance: one expert +0.55 (+0.49 to +0.61), the crowd +0.12 (+0.03 to +0.20).

What this tells us. The experts agree well with each other: from +0.55 on relevance to +0.75 on consistency. Relevance and fluency are the hardest to agree on. The crowd’s mean hardly agrees with the experts at all, although it averages five people. The SummEval paper reports the same. For Kestrel, this is the first lesson: who rates matters. “A person rated it” is not evidence by itself; trained raters with shared definitions are. The ceiling is a little optimistic, because the experts saw each other’s scores in round two.

The baseline: length

Section titled “The baseline: length”

The cheapest measure is the length of the summary in words. It knows nothing about quality. A measure that does not beat it adds nothing.

On the holdout, length gives +0.08 (−0.01 to +0.15) on coherence, +0.08 (0.00 to +0.15) on consistency, 0.00 (−0.09 to +0.09) on fluency, and +0.29 (+0.19 to +0.38) on relevance.

What this tells us. On three qualities, length is near zero, as it should be. On relevance it is clearly above zero: longer summaries keep more of the article, and the experts rewarded that. This is the bar for every measure of relevance.

The measures

Section titled “The measures”

Each measure gets a reason to try it, and its cost.

ROUGE is the measure that most summarisation papers report. It counts what the summary shares with a reference summary written by a person: single words (ROUGE-1), word pairs (ROUGE-2), or the longest shared sequence of words (ROUGE-L). Each is an F1 score between 0 and 1. The reference here is the article’s original highlights. The notebook uses Google Research’s rouge-score 0.1.2 with word stems, so “sold” matches “sells”. The reason to try it: it is free, fast and standard. The risk: Label and version showed that word overlap can prefer a wrong reply.

Copied word pairs is the share of the summary’s word pairs that also appear in the article. It needs no reference. A summary that copies the article cannot easily invent a fact, so it may track consistency. It is free.

Embedding similarity turns each text into a vector with a small embedding model, all-MiniLM-L6-v2 (Apache-2.0). Then it takes the cosine similarity with the reference or with the article. It can reward the same meaning in other words, which ROUGE cannot. It embeds all 1,800 texts in about 10 seconds on a laptop CPU. The model reads at most 256 word pieces, so it sees only the start of an article.

A model judge reads the article and the summary and rates the four qualities, as in Model-based judging. Its prompt gives each quality its own definition, written from the paper’s definitions, and anchors for 1, 3 and 5. It also asks for the first unsupported statement, if there is one. One call rates all four qualities. The course recorded two judges: gpt-6-luna as chat-small and gpt-6.1-sol as chat-strong, on Azure. The prompt was written once and tried on the dev articles. It did not change before the holdout run.

The judge needs the article in its prompt, and that had a real cost. Azure’s content filter refused every judge request for 3 of the 50 judged articles, on both deployments. These articles report two crimes and an injury. chat-small also refused one summary of a fourth article. The judge cannot score these summaries at all.

The fair comparison

Section titled “The fair comparison”

Now every measure takes the same test: the holdout summaries, one metric (Spearman with the experts’ mean), and one interval method. The chart shows the main measures; the table after it has all of them.

Each measure against the experts' mean, per quality, on the holdout summaries (fewer for the judges: the content filter refused some articles). Grey: the length baseline. Blue: free measures. Orange: model judges. The dashed line is the ceiling: one expert against the other two. 95% intervals, articles resampled.
Text description of the chart

Spearman with the experts’ mean, with 95% intervals. Coherence: length +0.08 (−0.01 to +0.15), ROUGE-2 +0.18 (+0.08 to +0.28), copied word pairs +0.22 (+0.13 to +0.30), embedding similarity with the article +0.13 (+0.04 to +0.21), judge chat-small +0.57 (+0.51 to +0.63), judge chat-strong +0.65 (+0.61 to +0.71); the experts’ ceiling +0.73. Consistency: length +0.08 (0.00 to +0.15), ROUGE-2 +0.17 (+0.09 to +0.24), copied word pairs +0.36 (+0.27 to +0.44), embedding +0.17 (+0.07 to +0.26), chat-small +0.60 (+0.49 to +0.70), chat-strong +0.63 (+0.53 to +0.72); ceiling +0.75. Fluency: length 0.00 (−0.09 to +0.09), ROUGE-2 +0.04 (−0.04 to +0.13), copied word pairs +0.23 (+0.16 to +0.31), embedding +0.12 (+0.05 to +0.19), chat-small +0.59 (+0.52 to +0.65), chat-strong +0.62 (+0.56 to +0.69); ceiling +0.59. Relevance: length +0.29 (+0.19 to +0.38), ROUGE-2 +0.33 (+0.21 to +0.44), copied word pairs +0.17 (+0.05 to +0.27), embedding +0.21 (+0.10 to +0.31), chat-small +0.49 (+0.42 to +0.55), chat-strong +0.54 (+0.46 to +0.60); ceiling +0.55.

Measure (holdout) Coherence Consistency Fluency Relevance
Length (baseline) +0.08 +0.08 0.00 +0.29
ROUGE-1 +0.18 +0.16 +0.07 +0.36
ROUGE-2 +0.18 +0.17 +0.04 +0.33
ROUGE-L +0.21 +0.16 +0.09 +0.35
Copied word pairs +0.22 +0.36 +0.23 +0.17
Embedding vs reference +0.13 +0.12 +0.09 +0.27
Embedding vs article +0.13 +0.17 +0.12 +0.21
Crowd (mean of 5 people) +0.04 −0.03 +0.04 +0.12
Judge, chat-small +0.57 +0.60 +0.59 +0.49
Judge, chat-strong +0.65 +0.63 +0.62 +0.54
One expert vs the other two +0.73 +0.75 +0.59 +0.55

What this tells us. Read the table by column: each quality has its own answer.

  • ROUGE agrees weakly with the experts on every quality. Its best result is on relevance, +0.36 for ROUGE-1, and length alone reaches +0.29 there. On the same summaries, ROUGE-2 minus length on relevance is +0.04 (−0.10 to +0.18): the interval includes zero. Daniel’s plan would send the editors the summaries with fewest shared words, not the worst ones.
  • For facts, the best free signal is copied word pairs (+0.36). ROUGE sees less (+0.17): sharing words with the reference says little about invented facts.
  • The model judges agree far better than every free measure, on every quality. chat-strong reaches +0.65 on coherence, +0.63 on consistency, +0.62 on fluency and +0.54 on relevance, close to the experts’ own ceiling (on fluency, above it). Against the best free measure of each quality, its gain on consistency is +0.27 (+0.19 to +0.34).
  • The strong judge is a little better, at a much higher price. chat-strong minus chat-small on the same summaries: +0.08 (+0.03 to +0.13) on coherence and +0.03 (0.00 to +0.07) on consistency. On fluency and relevance the intervals include zero. The strong judge costs about 15 times as much: US$3.19 against US$0.22 per 1,000 summaries.
  • Consistency has the widest intervals, because so few summaries have a factual error. A correlation there rests on a few dozen summaries.

Where each measure fails

Section titled “Where each measure fails”

A correlation summarises hundreds of cases. Before Kestrel trusts one, read the cases behind it, as Generated-output metrics asks. Each example below is a real summary from the holdout, with its real scores.

ROUGE rewards the reference’s words, not a good summary. An article reports that two football players went to watch their club’s basketball team. System M10 wrote:

neymar helped brazil beat chile 1-0 at the emirates stadium on sunday . real madrid remain top of their euro league division . luis enrique 's barcelona had won their el clasico to move four points clear at the top of la liga .

It has one of the highest ROUGE-2 scores of the holdout (0.56). The reason: the reference highlights contain the same football facts. The experts gave it relevance 1.00 and coherence 1.67: it lists loose facts and misses the story. ROUGE is only as good as its reference, and news highlights are not always good summaries.

The reverse happens too. System M22 summarised an article about a football club that wants to sign a 14-year-old player:

Manuel Pellegrini is keen to sign Anderlecht youngster Evangelos Patoulidis . The 14-year-old playmaker is regarded as one of the best talents to emerge from the Belgian club 's youth set-up . Arsenal and Barcelona are also interested in the youngster .

All three experts gave it 5 on every quality. Its ROUGE-2 is 0.03, near the bottom of the holdout: it says the same thing as the reference in other words.

When the judge and the experts disagree, the judge is not always the one who is wrong. System M20 summarised an article about a remake of the 1977 miniseries Roots:

A&E Networks will simulcast the original "Roots" in 2016. The original "Roots" premiered in 1977 and ran for four seasons. The miniseries followed Kunta Kinte, a free black man in Virginia, as he was sold into slavery.

The experts gave it consistency 4.67. chat-strong gave it 1 and copied the first sentence as unsupported. The article says that the networks will show a remake, not the original. It also describes Kunta Kinte as an African who was captured and sold, not as “a free black man in Virginia”. The judge is right, and the experts missed both errors. The experts are the best reference here, but they are not perfect.

The judge also applies its rubric more literally than the experts. System M22 ended a correct summary with a stray web phrase, “Click here for more Bayern Munich news”. The judge marked it unsupported and gave consistency 3; the experts gave 5. Both readings are defensible. Kestrel’s rubric would need to say whether such a phrase is a factual error or a formatting one.

And the judge can be too kind. System M11 wrote that Usain Bolt “is the iaaf/btc general secretary”. The experts gave consistency 1.67. chat-strong noticed the sentence and copied it as unsupported, but still gave 3, not the 1 that its own rubric asks for when statements contradict the article.

Over the holdout, chat-strong scored consistency at least 2 points below the experts’ mean on 62 summaries, and at least 2 points above on 4. It errs on the side of caution, which suits a screen that sends doubtful summaries to an editor.

Systems, not summaries

Section titled “Systems, not summaries”

A team that chooses a system compares the systems’ means, not single summaries. Averages remove much of the noise. Over the 16 systems’ holdout means, ROUGE-1 agrees with the experts’ relevance at +0.70, against +0.36 for single summaries. The strong judge reaches +0.95 on relevance and +0.82 on consistency.

What this tells us. Two different jobs need two different reports. Ranking systems is easier, but it rests on only 16 points. One system that moves changes the result a lot, so an interval here would be too wide to use. Checking one summary, Kestrel’s nightly job, is harder. A measure can rank systems well and still be useless for the summary in front of an editor. Always say which level you report.

The final choice: what to automate

Section titled “The final choice: what to automate”

Daniel and Ines turn the scorecard into a plan. The rule: automate a check only where a measure clearly agrees with the experts. Keep an editor wherever the measure is far from the ceiling, or cannot run.

Quality What runs on every summary When an editor reads it
Consistency (facts) The model judge, plus copied word pairs as a free warning A judge score below 5 or an unsupported statement; every refused article; a weekly sample of the rest
Coherence The model judge Scores of 3 or less
Fluency The model judge Scores of 3 or less; rare with modern models, so check the rate monthly
Relevance The model judge, as a hint only A daily sample: even the experts agree least here, and what matters depends on the client
ROUGE Not per summary Only to compare two systems on a fixed set with references

What this tells us. No measure replaces the editors. The judge becomes a screen: it decides which summaries an editor reads first, and it is cautious on facts. The editors stop reading 400 summaries and start reading the doubtful ones, plus a sample. That sample also gives Kestrel its own expert ratings. With them, it can check that the judge agrees with its editors, on its summaries, before it trusts these numbers.

A new article, end to end

Section titled “A new article, end to end”

Kestrel’s real question is about tomorrow’s summaries, which have no expert rating and no reference. The new input is a Wikinews article about a Sydney council’s plan to charge for parking at its beaches.

chat-small wrote a three-sentence summary of it (recorded). The course then constructed two faulty versions, to see what each check catches: one with a single changed number, and one that keeps three true side details and drops the main point. The notebook runs every free check that needs no reference, and shows both judges’ recorded verdicts:

Summary Length Copied word pairs Embedding vs article chat-small: consistency / relevance chat-strong: consistency / relevance
Real, by chat-small 63 0.38 0.82 5 / 5 5 / 5
A$6 million became A$16 million (constructed) 63 0.35 0.82 3 / 5 3 / 5
Side details only (constructed) 40 0.78 0.47 5 / 1 5 / 1

What this tells us. Both judges caught both faults. They lowered consistency for the changed number and copied that sentence as unsupported; they gave relevance 1 to the summary that misses the point. The free measures did not help. One changed number barely moves copied word pairs or the embedding. The off-point summary copies more of the article than the real one, so copied word pairs would rate it safer. ROUGE cannot run at all, because there is no reference. Three summaries are an illustration, not evidence: the scorecard is the evidence.

Insights and limits

Section titled “Insights and limits”
  • Which measure works depends on the quality. There is no single “summary quality” number to automate.
  • ROUGE is a weak check of a single summary, and on relevance it barely beats length. It needs references, which new summaries do not have.
  • A rubric judge agrees with experts nearly as well as the experts agree with each other, on these summaries. It is cautious on facts, and its disagreements need reading: some are the experts’ errors.
  • Old systems. The 16 systems date from 2017 to 2020, and their errors (garbled phrases, repeated words) are easier to see than a modern model’s. A judge may find today’s subtle errors harder.
  • The model may know this data. SummEval and CNN/DailyMail have been public for years. The judges may have seen them in training, so their agreement here may be a little better than on Kestrel’s own summaries.
  • One recording per summary, one prompt. The judge’s scores would move a little on a second run, as Controlled experiments showed. The prompt rates four qualities in one call; separate calls might do better, at four times the cost.
  • Small numbers on consistency. Few summaries have factual errors, so its intervals are wide.
  • The experts’ second round makes the ceiling a little optimistic.

Responsible use

Section titled “Responsible use”

Name the raters, and never blur the line. The ratings here are real human ratings by three experts, made as the SummEval paper describes. The judge scores are model outputs. A report that calls the judge’s scores “human-level” or mixes them with human ratings misleads the reader. Say who rated what, every time.

The judge cannot score everything. Azure’s content filter refused whole articles about crimes and an injury. These are exactly the stories where a wrong fact can hurt someone named in them. A refusal must route the summary to an editor, never to “no problem found”.

People named in the news. News summaries name real people. A summary that gets a fact wrong about a person can harm them, and a client may act on it. Kestrel needs a correction process, whatever the screen says.

What leaves the company. A hosted judge receives the full article. Check the provider’s terms before sending licensed news text or client material, as Licensing and data terms in the language models course explains.

Respect the data’s terms. Credit SummEval and the systems’ papers. Do not publish the CNN/DailyMail articles; link to their source, as the notebook does.

Make it a portfolio piece

Section titled “Make it a portfolio piece”

This case study shows an employer that you can test whether an automatic measure deserves trust. It gives the evidence and its limits. Make it yours with one change. For example, rate each quality in its own judge call, try a local model with Ollama, or use another rated dataset. Then make it public, and easy to read in five minutes.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • prompts/ with every judge prompt version you measured, never edited after you measured it.
  • The data sources and licences, with the attribution from the box above. Never commit the articles or an API key.

A README template:

# Which automatic checks agree with experts on summary quality?


## Problem
A news-digest team wants to stop reading every machine summary. Which automatic
measures can stand in for an editor, and on which quality?


## Data and licence
SummEval (Fabbri et al., 2021, MIT): 1,600 summaries of 100 CNN/DailyMail
articles by 16 systems; 3 expert ratings per summary on coherence, consistency,
fluency, relevance. Articles downloaded from their source, not redistributed.


## Approach
Split by article: 10 dev (judge prompt tried), 40 holdout (640 summaries,
compared once). Spearman with the experts' mean per quality; 95% bootstrap
intervals resampling articles. Measures: length, ROUGE-1/2/L, copied word pairs,
embedding similarity, two LLM judges with a rubric per quality.


## Results (holdout)
<your table: measure x quality, with intervals; the experts' ceiling>


## Limits
Old systems (2017-2020); one judge run; the model may have seen the data;
few factual errors, so wide intervals on consistency.

How to describe it in an interview, in three sentences: “I tested which automatic checks agree with expert ratings of news summaries, quality by quality, with intervals that respect the data’s structure. ROUGE barely beat summary length, while a rubric-based LLM judge came close to the experts’ own agreement. So I recommended the judge as a screen that sends doubtful summaries to an editor, never as a replacement.”

Go further:

  • Rate each quality in its own judge call, and compare on the same holdout.
  • Run a local judge with Ollama on the dev articles, and place it on the scorecard.
  • Repeat on another dataset with human ratings, and see whether the ranking of measures holds.

Knowledge check

Section titled “Knowledge check”

Check your understanding

Daniel says: "ROUGE-1 has a correlation of +0.36 with the experts on relevance, so it is a fine relevance check." What do you answer, with the case study's numbers?

Show the answer

Compare it with the baseline, not with zero. The summary’s length alone reaches +0.29 on relevance, and the paired interval of ROUGE-2 minus length includes zero. So ROUGE adds little that counting words does not. It also needs a reference, which tomorrow’s summaries do not have. The judge with a rubric reached +0.54 on the same summaries. And for checking one summary, even that is far from perfect: read the cases where it disagrees.

Check your understanding

Why does the interval resample whole articles instead of single summaries?

Show the answer

The 640 summaries are 16 summaries of each of 40 articles. Summaries of the same article share its topic and difficulty, so they are not independent. If you resample single summaries, the bootstrap acts as if there were 640 independent cases. Then the interval comes out too narrow. Resampling articles matches how the data was collected: 40 articles were drawn. The notebook’s failure case shows both intervals side by side.

Check your understanding

Ines asks: 'The judge agrees with the experts almost as well as they agree with each other. Can we stop reading summaries?' What do you recommend?

Show the answer

No, but the editors can read far fewer. The judge agrees well on average. But on single summaries it can be too kind, and it refused whole articles about crimes and an injury. It was also measured on old systems and on public data that it may have seen. Use it as a screen. Editors read every summary with a doubtful fact score or an unsupported statement, every refused article, and a sample of the rest. Rate that sample with Kestrel’s own editors, and measure the judge against them before you trust it more.

Recap

  • Compare a measure with people by order, per quality: Spearman’s rank correlation, with an interval that resamples the unit that was sampled (here, articles).
  • Measure the ceiling first: here, one expert against the other two. The crowd’s ratings hardly agreed with the experts: who rates matters.
  • On 640 holdout summaries, ROUGE agreed weakly on every quality, and on relevance it barely beat length. Copied word pairs was the best free signal for facts.
  • Two model judges with a rubric per quality came close to the experts’ agreement. Read their disagreements: some were the experts’ errors, some were the judge’s.
  • The content filter refused whole articles: those summaries need a person.
  • Automate a screen, not a verdict, and check it against your own experts before you trust it.

Next: Regression-test a classifier release by slice