Compare baselines

What you will be able to do

Find out whether training the adapter was worth it. You test it against every cheaper option and against the system in use, on the same held-out tickets, with the same strict check.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• The strict answer check and the hosted assistant’s score, from Clarify the failure.
• The targets in targets.toml, written before training, from Define success and constraints.
• The two training runs, 2 and 6 epochs, from Monitor and recover.
• Why every system must answer the same cases, from Uncertainty and slices in Evaluating and Testing AI Systems.

What you need on your computer

• ~/projects/ticket-adapter at the end of Module 4, with .venv active: python -m pytest shows 27 passed, 1 deselected. If it does not, copy the snapshot end-of-m04, as the snapshots README shows.
• The labs repository in ~/projects/nextia-learning-labs, from Clarify the failure.
• Only for the optional run: the model weights, from python -m adapt download in Module 4.

Tested with Python 3.12.13 on macOS (Apple silicon). The Windows commands are written but not tested.

A fair test in the garden

A seller visits Larkfield with a new plant food. “Tomatoes with our food grew three times more!” he says. Then you ask: three times more than what? He compared them with plants that got no water at all.

A gardener tests it in another way. She plants the same seeds in the same bed, on the same day. Some plants get the new food, some get ordinary compost, and some get only water. She measures them all with the same ruler. And she compares them with the best tomatoes that she already grows. The new food is worth its price only if it beats the cheap options. It must also be as good as what she has now.

The problem: better than what?

Tomás shows the team a slide. “The base model got 0 of 102 decisions right. With the adapter, it gets 74. Fine-tuning works!”

Every number on the slide is true, and still it says little. The base model with the short instruction never answers in JSON, as you saw in Clarify the failure. Beating it is like beating plants with no water. Does something cheaper also get 74? Is 74 enough to replace the hosted assistant, which gets 90? The slide does not say.

The idea: a ladder of baselines

A baseline is a system that you compare a new one with. One baseline is not enough. Use a ladder of them, cheapest first, and let each rung answer one question. The adapter is worth its cost only if it beats every rung below it. To replace the hosted assistant, it must also be as good as the top rung.

The ladder of baselines. Every rung answers the same 102 dev tickets, and the same strict check scores it.

Text description of the diagram

The first three rungs use the adapter’s small model with no training. The second adds Grace’s rules and three example tickets to the prompt. In the third, the model does not write freely: the code chooses the most likely of the 10 valid answers (5 teams, true or false). The fourth rung uses no language model at all.

A strong baseline is the best cheap option that you can build. A weak baseline makes any new system look good. That is the mistake on Tomás’s slide.

A tiny example: four tickets, four systems

Four real dev tickets, and the real replies of four systems. A decision is right when the team and needs_human are both right.

Ticket	Right answer	Hosted	Word counts	Better prompt	Adapter
T-65799 “listed at 64.00 … charged 71.50”	payment, false	✓	✓	✗ warranty	✓
T-81012 “Pouvez-vous m’envoyer une facture …” (French)	payment, false	✓	✗ returns	✗ warranty, true	✓
T-64709 “a frayed section … it isn’t safe to use”	delivery, true	✓	✓	✗ warranty, false	✗ false
T-81026 “filing a complaint with the consumer protection office”	payment, true	✓	✗ false	✗ warranty, false	✗ false
Decisions right		4 of 4	2 of 4	0 of 4	2 of 4

The word-count model and the adapter both get 2 of 4, but they fail on different tickets. The word-count model knows no French, and its safety word list catches the unsafe hammock. The adapter routes the French invoice, and it says false to both tickets that need a person. Equal scores can hide different mistakes.

Compare every system in your project

Module 5 adds adapt/predict.py, which runs the local systems, and the saved replies of every system in runs/predictions/. The course’s authors made the word-count model outside your project, with scikit-learn. It uses TF-IDF and logistic regression, as in Machine Learning: From Problem to Reliable Model. Your project has only its saved replies.

1. In ~/projects/ticket-adapter, with .venv active, copy this lesson’s files from the snapshot end-of-m05:

   macOS and Linux

   S=../nextia-learning-labs/C12/snapshots/end-of-m05
   for f in adapt/predict.py adapt/__main__.py tests/test_compare.py README.md; do cp "$S/$f" "$f"; done
   cp $S/runs/predictions/* runs/predictions/

   Windows

   $S = "..\nextia-learning-labs\C12\snapshots\end-of-m05"
   foreach ($f in 'adapt\predict.py', 'adapt\__main__.py', 'tests\test_compare.py', 'README.md') { Copy-Item "$S\$f" $f }
   Copy-Item "$S\runs\predictions\*" runs\predictions\

2. Run the tests: python -m pytest. Expect 30 passed, 2 deselected. One new test checks that every saved system answered all 271 cases.

3. Predict before you run: which system has the most decisions right after the hosted assistant?

4. Compare every system on the dev cases:

   python -m adapt compare

102 cases (dev)
system          answers     team  caught  false  decisions  s/answer
chat-small      101/102   97/100  23/23       9     90/102     2.977
words           102/102   86/100   6/23       1     74/102     0.000
base-short        0/102    0/100   0/23       0      0/102     0.399
base-policy     101/102   22/100   8/23      25     12/102     0.406
base-choice     102/102   22/100   8/23      25     12/102     0.465
adapter         102/102   81/100   0/23       0     68/102     0.215
adapter-long    102/102   89/100   0/23       0     74/102     0.235
caught = needs a person, sent to one; false = sent to a person without a rule; s/answer = median

Decisions right on the 102 dev tickets. The adapter after 6 epochs ties with the word-count model at 74. The hosted assistant gets 90.

Text description of the chart

One row per system, cheapest first: 0, 12, 12, 74 (word-count model), 68 (adapter, 2 epochs), 74 (adapter, 6 epochs) and 90 (hosted assistant).

Read the table one rung at a time:

• The better prompt fixes the format, not the task. It gives 101 valid answers, but it says “warranty” for 100 of the 102 tickets. Its 22 right teams are the warranty tickets.
• Constraining the output changes nothing here. base-choice can only give a valid answer, and it makes the same 12 right decisions. So the small model’s problem is not the format: it does not know the task.
• Training taught the task. The adapter gives a valid answer every time and gets 89 teams right. Its short prompt also makes it faster than the better prompt (0.24 s against 0.41 s).
• A model with no language model ties with it. The word-count model also gets 74, and it answers in a fraction of a millisecond.
• No system beats the hosted assistant. It gets 90, and it catches all 23 tickets that need a person. The adapter catches none.

Times are medians on the course’s computer with 2 threads, and your times will differ. The hosted time includes the network.

Your turn: choose the adapter on dev

You have two adapters. Use only the dev cases and the validation loss, as Priya did. Run python -m adapt compare --systems adapter adapter-long --by slice. Which adapter do you keep, and which numbers decide it? What did the 4 extra epochs not change?

Show an example answer

102 cases (dev)
system          answers     team  caught  false  decisions  s/answer
adapter         102/102   81/100   0/23       0     68/102     0.215
    boundary      decisions 11/14
    difficult     decisions 26/30
    french        decisions 6/10
    needs_person  decisions 0/22
    normal        decisions 11/12
    unanswerable  decisions 14/14
adapter-long    102/102   89/100   0/23       0     74/102     0.235
    boundary      decisions 13/14
    difficult     decisions 29/30
    french        decisions 7/10
    needs_person  decisions 0/22
    normal        decisions 12/12
    unanswerable  decisions 13/14
caught = needs a person, sent to one; false = sent to a person without a rule; s/answer = median

Keep the 6-epoch adapter. Its validation loss is lower (0.0193 against 0.0584), it gets 8 more teams right and 6 more decisions, and it is as fast. The extra epochs did not change the safety result: both catch 0 of 23. (The needs_person slice has 22 cases: the 23rd ticket that needs a person is French.) A lower loss made the adapter better at what it already did. It did not teach a rule that the data does not show.

Read the holdout once

You chose the 6-epoch adapter on dev and on the validation loss. Now read the holdout cases, once. They are the final exam: nobody chose anything with them.

python -m adapt compare --split holdout

Note: the holdout cases are for the final decision only. Read them once, at the end.
98 cases (holdout)
system          answers     team  caught  false  decisions  s/answer
chat-small       98/98    97/97   24/24       4     94/98      2.813
words            98/98    86/97    8/24       0     74/98      0.000
base-short        0/98     0/97    0/24       0      0/98      0.396
base-policy      98/98    26/97    8/24      30     11/98      0.406
base-choice      98/98    28/97    9/24      31     13/98      0.460
adapter          98/98    74/97    0/24       0     57/98      0.211
adapter-long     98/98    87/97    0/24       0     65/98      0.237
caught = needs a person, sent to one; false = sent to a person without a rule; s/answer = median

The adapter falls from 74 to 65 decisions, and the word-count model now beats it by 9. The adapter still sends no ticket to a person: 0 of 24. The next lesson finds where the 9 decisions went.

Discuss: is 74 good news?

Tomás puts the dev table on the screen. He is pleased.

• Tomás: “Valid JSON every time, 89 teams right, a quarter of a second, on our own computers.”
• Priya: “The word-count model also gets 74 on dev. On holdout it gets 74, and we get 65. It trained in 0.17 seconds.”
• Kwame: “So why do we need a language model at all?”
• Grace, looking at another column: “Caught: 0 of 23. Who reads the ticket about the unsafe hammock?”

Before you read on, answer these:

1. Which rungs would you put on Tomás’s slide, and why?
2. The adapter and the word-count model tie on dev. Are they equally good?
3. What result would make you prefer the adapter to the hosted assistant?

The course’s answer. Put every rung on the slide, with the hosted assistant at the top. The strong baselines show what training added: the task (0 decisions, then 12 with the best prompt, then 74). They also show its limit: a word-count model does as well on dev, and better on holdout.

The tie on dev is not equal quality. The two systems fail on different tickets: the word-count model on French, the adapter on every ticket that needs a person. Grace’s question matters most. The go/no-go memo said that a model that misses a safety ticket is not acceptable at any price. The adapter misses all of them.

What would change the answer: an adapter that catches every ticket that needs a person and gets at least 88% of the decisions right. It must do so on dev, and then once on holdout. The next lesson asks why it misses them, and what it would take.

When to compare against baselines, and what it costs

Build the ladder before you train. The cheap rungs need little work: a prompt, a few lines of code, a classic model. Sometimes one of them solves the problem, and you never train. Always include the system that you have now.

The cost is mostly honesty: a strong baseline can make your new system look ordinary. And on 102 cases, a difference of a few tickets can be luck, as Uncertainty and slices showed.

Common mistake

A system that answered only some of the cases

Symptom: you run the adapter again on 10 tickets to check it, and compare. The new system gets 10/102 valid answers and 8/102 decisions: it looks far worse than adapter-long.

Cause: compare scores every system on all 102 dev cases. A case with no saved reply counts as no answer, so that ticket reaches no person. Your 10-ticket run is scored on 102 cases.

Fix: compare systems only when they answered the same cases. Run the whole split (predict adapter --name my-adapter, without --limit), or delete the partial file. The test test_every_saved_system_answered_all_271_cases protects only the systems in systems.json.

Guided practice: run the adapter yourself (optional)

This step needs the model weights and about a minute. The saved replies are enough for the rest of the module.

1. Run the 6-epoch adapter on the first 10 dev tickets. Here adapter names the kind of system, the short instruction with an adapter. Without --adapter, predict loads the 6-epoch one, runs/lora-r8-long/adapter. Always give a new --name, or you write over saved replies.

   python -m adapt predict adapter --name my-adapter --limit 10

2. Compare your run with the saved one: python -m adapt compare --systems my-adapter adapter-long.

3. Predict: why does my-adapter get so few decisions? Then delete runs/predictions/my-adapter.jsonl.

Show the answer

The course’s run printed this. Your times will differ:

Wrote runs/predictions/my-adapter.jsonl: 10 cases; peak memory 1186.2 MB

102 cases (dev)
system          answers     team  caught  false  decisions  s/answer
my-adapter       10/102   10/100   0/23       0      8/102     0.226
adapter-long    102/102   89/100   0/23       0     74/102     0.235
caught = needs a person, sent to one; false = sent to a person without a rule; s/answer = median

my-adapter answered only 10 of the 102 cases, and compare counts the other 92 as no answer. On its 10 tickets, it got all 10 teams right and 8 decisions. This is the failure case above. When the course ran all 102 dev cases (predict adapter --name my-adapter, 24 seconds), all 102 replies were identical to the saved adapter-long ones: greedy decoding gives the same reply for the same input.

See it end to endShould this team fine-tune?Case study

The case study scores six systems on the same 200 dev messages with one function: the hosted model, word counts trained on the same data as each adapter, the small model with a fixed choice, and two adapters. The best adapter gets 183 decisions; the prompt-only system gets 198.

Knowledge check

Check your understanding

What is a strong baseline, and why does Tomás's slide (0 → 74 decisions) need one?

Show the answer

A strong baseline is the best cheap option that you can build for the same task, measured on the same cases with the same check. Tomás compares the adapter with the base model and a short instruction, which never answers in JSON. Any working system beats that. The strong baselines show that a better prompt gets 12, a word-count model gets 74 and the hosted assistant gets 90. So the adapter teaches the small model the task, but it does not beat a cheap classifier or the system in use.

Check your understanding

Predict: Sam adds a new rung, the hosted assistant with a shorter prompt, and saves its replies for the 102 dev cases only. Can you compare it with the others on holdout? What do you do?

Show the answer

No. compare --split holdout counts each missing reply as no answer, so the new system would look terrible. Run it on the holdout cases too, so that every system answers the same cases. And choose the prompt on dev first: read the holdout once, after the choice.

Check your understanding

The adapter and the word-count model both get 74 of 102 dev decisions. Which evidence would make you prefer one of them?

Show the answer

Look at which cases each one gets wrong, not only the total. The word-count model fails on French (0 of 10) and catches 6 of 23 tickets that need a person. The adapter gets 7 of 10 French decisions and catches none. Then look at the holdout, read once: the word-count model gets 74 of 98 and the adapter 65. Cost and speed matter last, because both are fast and cheap. Neither meets the targets, so neither replaces the hosted assistant.

Recap

• A baseline is a system that you compare a new one with. Use a ladder of them, cheapest first, with the system in use at the top. A strong baseline is the best cheap option.
• Every system answers the same cases, and one strict check scores them: python -m adapt compare.
• The better prompt fixes the format but says “warranty” for 100 of 102 tickets. Constraining the output gives the same 12 decisions.
• Training taught the task: 74 of 102 dev decisions. A word-count model also gets 74, and it beats the adapter on holdout, 74 against 65.
• No system matches the hosted assistant, and the adapter sends no ticket to a person.
• Your project: 30 passed, 2 deselected.

Next, you look for what the adapter lost and why, and you weigh time, memory and cost before you decide.

Next: Regressions and operational value