Choose evaluation layers

What you will be able to do

Choose where to check each criterion: in a plain function, in one part of the assistant, on a whole ticket, or in production. Then set up the course project, eval-harness, and prove that it works.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• What a measurable criterion is, from Map goals to evidence.
• How to open a terminal and move between folders, from Files and terminal in Python for Practical AI Engineering.
• How to make and activate a virtual environment, from Isolate dependencies.

What you need on your computer

• Python 3.12 or newer, from Install and verify Python.
• The folder ~/projects. If it is missing, run mkdir ~/projects first.
• An editor such as VS Code, and an internet connection to download the project and its two packages (about 40 MB with the virtual environment).
• Git is optional: you can download a ZIP file instead.

You need no account, no API key and no payment. Nothing in this lesson calls a model.

Tested on macOS (Apple silicon) with Python 3.12.13, pydantic 2.14.0 and pytest 9.1.1. The Windows and Linux commands follow the Python course; they were not run for this lesson.

Checking a car before it reaches the road

A car factory does not test a car only by driving it. Machines check every bolt as it goes in: thousands of checks a day, each one in a second. Each brake system is tested on a bench before it goes into a car. Some finished cars are driven on a test track, which takes an hour and a driver. And after the cars are sold, the factory reads the reports from garages and owners.

Each level catches problems that the others cannot. A bolt check cannot tell whether the car is pleasant to drive. A test drive cannot tell which of 2,000 bolts is loose. And no test before the sale shows how people drive in the snow. So the factory uses all four levels: many cheap checks at the bottom, and a few slow ones at the top.

The problem: one kind of check is not enough

Priya, the data scientist, owns Larkfield’s evaluation. Her first idea was simple. For every change to the project, run all 271 tickets through the assistant, then have a reader score every reply. That is the test drive: it checks what the customer really gets.

But it is slow and it costs money. A new run of the assistant on 271 tickets took about 5 minutes and cost US$0.09 on chat-small. A model that scores each reply adds about US$0.004 per reply, and a person’s time is worth much more. Priya changes the code ten times a day. She cannot pay for a test drive each time.

Worse, a test drive can miss a simple bug. Suppose a code change makes the project load the holdout cases with the development cases. The holdout cases are the ones kept back for the final release decision, as Module 2 explains. Every score still looks normal, a little higher or a little lower. Nothing in the replies shows the bug, but one line of test code catches it in a millisecond.

The idea: layers of evaluation

An evaluation has layers, like the levels of the car factory. Each layer checks a bigger piece of the system, with fewer and more expensive checks. The usual picture is a pyramid.

The evaluation pyramid for Larkfield's assistant, from plain functions at the bottom to production at the top. The costs are the course's real costs.

Text description of the diagram

Four layers, from the bottom up.

Layer 1, unit tests of plain functions. Code with one right answer. Example: load_cases() returns 271 cases, and it never loads the holdout set unless you ask; the decision counts on five made-up cases are right. These checks take under a second, cost nothing, and run on every change.

Layer 2, component evaluations. One part of the assistant on many cases. Example: the team and needs_human answers of a saved run, on the 102 development cases (next lesson). Scoring saved outputs takes seconds. A new run of the assistant on 271 tickets takes about 5 minutes.

Layer 3, end-to-end tasks. A whole ticket to a whole reply: does the customer get what they need? A reader with a rubric, or a model used as a judge, decides (Module 4). A model judge costs about US$0.004 per reply; a person’s time costs more.

Layer 4, monitoring. Real tickets after the release: agents’ edits, thumbs up or down, new topics. Module 6 uses a simulated log. It takes weeks, and it comes last.

An arrow on the left points down: more checks, faster and cheaper. An arrow on the right points up: closer to what the customer really gets.

Layer 1: unit tests of plain functions. A unit test checks one small piece of code against a known right answer. It works for code that is deterministic: the same input always gives the same output. Loading the cases, counting caught tickets and building a version string are all deterministic. No model is involved, so a unit test is exact, fast and free.

Layer 2: component evaluations. A component is one part of the AI system with a clear job, such as the team decision, the search or the reply. A component evaluation measures that part on many labelled cases. The result is not “pass” or “fail” but a score, such as “the team is right on 97 of 100”. A model is involved, so each answer can be wrong, and only many cases show how often.

Layer 3: end-to-end tasks. An end-to-end check gives the whole system a whole task, from the ticket to the reply. It judges the result as the user would. Did the customer get the right next step, in a friendly tone, with nothing invented? This needs a reader, or a model that acts as a reader. It is the layer closest to what customers see, and the most expensive one.

Layer 4: monitoring in production. Production is the real service, with real customers. Monitoring means watching it after the release: how often agents change a draft, what customers say, which new topics appear. Only this layer sees tickets that no one thought of. But it sees problems after customers have met them.

Why a pyramid? The bottom is wide: many fast, cheap checks. The top is narrow: a few slow, expensive ones. Turned upside down, every change is slow and costs money, and small bugs hide inside averages.

A tiny example: one function, one test

Here is a real layer-1 test from the project. The function load_cases() reads the evaluation set. It must never return the holdout cases unless you ask for them:

harness/dataset.py (part)

def load_cases(path: Path = CASES, split: str | None = None) -> list[Case]:
    """All cases, or the cases of one split. The holdout set is never loaded by accident: ask for it."""
    cases = [Case.model_validate_json(line) for line in path.read_text(encoding="utf-8").splitlines() if line]
    if split is None:
        return [c for c in cases if c.split != "holdout"]
    if split == "all":
        return cases

tests/test_first_run.py (part)

def test_holdout_is_not_loaded_by_default():
    assert all(c.split != "holdout" for c in load_cases())

The test has one right answer: no case of the holdout split is in the default list. It needs no model, no labels and no judgement. If someone deletes the if split is None line, the test fails at once, and the message names the test.

Now compare a layer-2 check. Next lesson, you score the assistant’s saved answers. The result is a line such as needs a person: caught 23/23, false alarms 9. That is not pass or fail. It is a measure of how well one component does on 102 cases. Whether 9 false alarms is good enough is a decision, made with the criteria of the last lesson.

Which layer catches which failure

The rule for choosing a layer is short: check each failure at the lowest layer that can catch it. A lower layer is faster and cheaper, and it points to the cause more exactly. Move up only for failures that a lower layer cannot see.

Four ways the assistant or the harness can fail, each with the lowest layer that catches it.

Text description of the diagram

Four failures, each with the lowest layer that catches it.

First: a code change makes load_cases() also load the holdout set. A unit test catches it: test_holdout_is_not_loaded_by_default fails.

Second: a new prompt sends a safety ticket to no one. A component evaluation catches it: the count of caught tickets that need a person falls below 23 of 23.

Third: a reply promises something that the policy does not give. An end-to-end check catches it: a reader with the rubric marks the reply.

Fourth: customers start to ask about a topic that is not in the evaluation set. Only monitoring catches it: a sample of real tickets shows the new topic.

Look at the second row. A reader could also find a safety ticket sent to no one, at layer 3. But needs_human is plain data that code can compare with the label, so the check belongs at layer 2. The third row is different. Whether a sentence promises too much needs someone who understands the sentence. Code finds some cases with phrase rules (Module 3), and a reader decides the rest.

Larkfield: the layers in this course

The course builds the layers in order, in the project eval-harness. The project never runs the assistant. Every output of the assistant was recorded once from the real model and saved with the course. So layer 2 costs nothing for you: the harness scores saved outputs.

Layer	What the harness checks	Where in the course
1. Unit tests	Loading the cases, counting, run IDs, the dataset version, the scorers on tiny made-up examples	Every module, in tests/; regression tests in Module 6
2. Component evaluations	The team and needs_human decisions; the reply rules; search results	This module’s next lesson; Module 3
3. End-to-end tasks	Each whole reply, scored with a rubric by you and by a model judge	Module 4
4. Monitoring	A simulated production log: feedback, agents’ edits, new topics	Module 6

Set up the project

The project comes from the course’s labs repository, as snapshots: copies of the project at the start and at the end of each module. You start from start.

1. Open a terminal and move to your projects folder:

   cd ~/projects

2. If you have Git, clone the labs repository. If you cloned it for an earlier course, run git -C nextia-learning-labs pull instead.

   git clone https://github.com/Nextia-Intelligence/nextia-learning-labs.git

   Without Git, download the ZIP file of the repository. Unzip it in ~/projects. Its folder is nextia-learning-labs-main: use that name in the next steps.

3. Copy the start snapshot into a new folder, eval-harness:

   macOS

   cp -R nextia-learning-labs/C11/snapshots/start eval-harness

   Windows

   Copy-Item -Recurse nextia-learning-labs\C11\snapshots\start eval-harness

   Linux

   cp -R nextia-learning-labs/C11/snapshots/start eval-harness

4. Move into the project, and open it in your editor:

   cd eval-harness
   code .

   If code is not found, open the folder from the editor’s File menu.

You work in your copy, eval-harness. The labs folder stays as it is, so that you can copy a snapshot again at any time.

Next, the virtual environment. The project needs two packages at exact versions: pydantic checks that every case has the right fields, and pytest runs the tests.

1. Make and activate the environment:

   macOS

   python3 -m venv .venv
   source .venv/bin/activate

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1

   Linux

   python3 -m venv .venv
   source .venv/bin/activate

   The prompt now starts with (.venv). From here, type python on every system.

2. Install the packages from requirements.txt:

   python -m pip install -r requirements.txt

   The last line of the output:

   Successfully installed annotated-types-0.8.0 iniconfig-2.3.1 packaging-26.3 pluggy-1.6.0 pydantic-2.14.0 pydantic-core-2.50.0 pygments-2.21.0 pytest-9.1.1 typing-extensions-4.16.0 typing-inspection-0.4.4

   You asked for two packages, and pip also installed the packages that they need. A notice about a new pip version is safe to ignore.

The known-good first run

Two checks prove that the setup works: the tests, then the evaluation set.

1. Run the tests:

   python -m pytest

   The output (your folder path differs):

   ============================= test session starts ==============================
   platform darwin -- Python 3.12.13, pytest-9.1.1, pluggy-1.6.0
   rootdir: /Users/priya/projects/eval-harness
   configfile: pytest.ini
   testpaths: tests
   collected 3 items
   
   tests/test_first_run.py ...                                              [100%]
   
   ============================== 3 passed in 0.32s ===============================

   Each dot is one test that passed. These are layer-1 tests: they took a third of a second.

2. Count the cases:

   python -m harness cases

   The output:

   271 cases, dataset version d4c96eaa10c0
     dev            102   needs a person:  23   French: 10
     holdout         98   needs a person:  24   French: 10
     contaminated    71   needs a person:  10   French:  1

Your output must be the same, character for character. d4c96eaa10c0 is the dataset version: a fingerprint of the file data/cases.jsonl. If one character of the file changes, the version changes. So a score that names this version names the exact cases behind it. The three groups are called splits, and Module 2 explains them. dev is for everyday work, and holdout is kept back for the release decision. contaminated holds the tickets that the prompt was written on.

The setup check. Three passed tests and the known-good first line mean that you are ready.

Text description of the diagram

First run python -m pytest. If you do not see “3 passed”, check that the prompt starts with (.venv) and that you are in the eval-harness folder. Install the requirements again, and run the tests again. When you see “3 passed”, run python -m harness cases. If the first line is “271 cases, dataset version d4c96eaa10c0”, the setup works. If it is not, reset: copy the start snapshot again into a new folder, make a new .venv, and start the checks again.

What is in the project

Path	What it does
data/cases.jsonl	The evaluation set: 271 cases, one per line, each a ticket with its labels
data/policy.md	Grace’s support policy, the same as in the LLM applications course
data/dataset.md	The dataset card: where the cases come from, and what they cannot tell you
harness/dataset.py	Loads the cases, checks their fields, and computes the dataset version
harness/__main__.py	The command line: python -m harness cases and show
tests/	The three layer-1 tests that you ran

Look at one case

Every criterion of the last lesson needs evidence from a case. Look at a safety ticket:

python -m harness show T-64740

The output:

T-64740 | split dev | slice needs_person | source c06-valid
Text: Hi there. I logged in without trouble this morning and the refund from my last order went through, so that part is fine. I'm writing about my folding parasol, order LK-774120. The pole joint cracked while I was opening it on the patio and the canopy dropped toward my daughter's head. Could you tell me how to report this as a safety problem and what happens next? Thank you.
Expected: team warranty | needs_human true (H1) | next step none
Reply must: person_will_contact, stop_using | must not: claims_action, asks_secret
Labelled by: course author, 2026-10-09; Grace's policy (teams, H1-H5) applied by reading; team: the deep learning course's label (two models agreed), read again by the course author; next_step: warranty_claim -> none on review (on a safety ticket the next step is to stop using it and wait for a person; the warranty route is that person's decision)

Read it line by line. The ticket starts with two topics that do not matter: a sign-in and a refund. The real request is about a parasol that fell toward a child. Rule H1 says that a safety concern always needs a person. So the labels say: team warranty, needs_human true, and a reply that tells the customer to stop using it and that a person will contact them.

The last line says who labelled the case. The course author wrote the labels by applying Grace’s written policy, and scripts checked them. It is one person’s reading. Module 2 shows this work, and in Module 4 you are the second rater.

Each line of the case feeds a different layer. needs_human true is plain data, so a layer-2 evaluation compares it with the assistant’s answer. “Reply must: stop_using” is partly layer 2, with phrase rules in code. Whether the reply is calm, clear and kind is layer 3: it needs a reader.

When to use each layer, and what it costs

• Unit tests: for all plain code, on every change. They cost minutes to write and nothing to run, but they cannot tell you whether the model is good.
• Component evaluations: for every decision that has a label, on every new prompt or model. On saved outputs they are free; a new run costs money and minutes. They cannot tell you whether a whole reply helps.
• End-to-end tasks: on a sample of whole replies, before a release. A reader’s time is the main cost, and a model judge needs checking first (Module 4).
• Monitoring: after every release. It finds what no test set contains, such as new topics, but only after customers have met the problem.

Guided practice: place five checks

For each check, choose the lowest layer that can do it. Write your answers, then open ours.

1. python -m harness cases prints 271 cases.
2. On the French development cases, the reply is in French.
3. The reply to T-64740 is calm and clear, and the customer would know what to do.
4. The share of drafts that agents rewrite rises in the second week after a release.
5. On the 23 development cases that need a person, needs_human is true.

Show the answers

1. Layer 1. Counting lines is plain code with one right answer. A test in tests/test_first_run.py checks it.
2. Layer 2. The language of each saved reply can be checked by code against the label, on 10 cases. A reader is needed only where the code cannot decide.
3. Layer 3. “Calm, clear, and the customer would know what to do” needs a reader with a rubric.
4. Layer 4. Only production has agents and real drafts. No test before the release shows this.
5. Layer 2. needs_human is a field: code compares it with the label on each case.

Your turn: place your criteria

Open success-criteria.md from the last lesson. Next to each criterion, write the lowest layer that can check it. Then find one criterion that needs two layers: for example, code finds most failures, and a reader decides the rest.

Show an example answer

• Team right on 95 of every 100 cases: layer 2.
• Every ticket under rules H1 to H5 goes to a person: layer 2. The function that counts them is tested at layer 1, on a tiny made-up example.
• The reply contains the next step: layer 2 with phrase rules, and layer 3 for the replies that the rules cannot decide.
• No reply says that a refund was made: layer 2 with phrase rules, and layer 3 for the rest. This criterion needs two layers.
• p95 time and cost per 1,000 tickets: layer 2, from the time and cost saved with each output. After the release, layer 4 measures them on real traffic.
• Every ticket gets a valid answer: layer 2.

Common mistake

No module named harness

Symptom: you run python -m harness cases and see:

/Users/priya/projects/eval-harness/.venv/bin/python: No module named harness

Cause: the terminal is not in the project folder. python -m harness looks for the harness package in the current folder, and here it ran from ~/projects. A different message means a different cause:

ModuleNotFoundError: No module named 'pydantic'

This one means that the environment is not active: the computer’s own Python has no pydantic.

Fix:

1. Move into the project: cd ~/projects/eval-harness.
2. Check that the prompt starts with (.venv). If it does not, activate the environment again.
3. Run the command again.

On Windows, if PowerShell says that running scripts is disabled, follow the failure case “Running scripts is disabled” in Isolate dependencies. If the project still fails, reset it.

Reset the project

If the project stops working and you cannot find why, start again from the snapshot. Your old folder stays, with a new name, so you lose nothing.

1. Deactivate the environment, and move to your projects folder:

   deactivate
   cd ~/projects

2. Rename your project:

   macOS

   mv eval-harness eval-harness-old

   Windows

   Rename-Item eval-harness eval-harness-old

   Linux

   mv eval-harness eval-harness-old

3. Copy the snapshot again, as in step 3 of Set up the project.

4. Make a new environment and install the packages.

5. Run the two checks of the known-good first run.

Later modules use the same steps with their own snapshot. For example, Module 2 can start from end-of-m01.

Knowledge check

Check your understanding

In your own words: why is the evaluation pyramid wide at the bottom and narrow at the top?

Show the answer

The bottom layers are fast and cheap, so you can have many checks and run them on every change. Unit tests of plain functions take milliseconds and cost nothing. The top layers are slow and expensive: a reader for each reply, or weeks of production. So you have fewer of them, and you use them only for what a lower layer cannot check. A lower layer also points to the cause more exactly.

Check your understanding

Predict: someone changes load_cases() so that it also returns the holdout cases by default. Which command shows it first, and what do you see?

Show the answer

python -m pytest shows it first: test_holdout_is_not_loaded_by_default fails, and the summary says 1 failed, 2 passed. This is a layer-1 test, so it catches the bug in under a second, before any score is computed. python -m harness cases would still print 271 cases, because it asks for all the splits on purpose.

Check your understanding

Grace wants to know whether the assistant's replies to safety tickets are kind enough. Which layer checks this, and why not a lower one?

Show the answer

Layer 3, end-to-end tasks: a reader scores each whole reply with a rubric. Code can check that a reply says “stop using” and that a person will contact the customer (layer 2). But “kind enough” depends on how the whole reply reads. That needs someone who understands the sentences, and a rubric so that two readers score in the same way.

Recap

• An evaluation has four layers. Unit tests check plain, deterministic functions. Component evaluations check one part on many labelled cases. End-to-end tasks are judged as the user sees them. Monitoring watches production.
• The evaluation pyramid has many fast, cheap checks at the bottom and few slow, expensive ones at the top. Check each failure at the lowest layer that can catch it.
• The project eval-harness comes from the labs snapshot start. The known-good first run: python -m pytest gives 3 passed, and python -m harness cases prints 271 cases, dataset version d4c96eaa10c0.
• python -m harness show CASE shows one case and its labels: the evidence that the criteria need.
• If a check fails, look at the folder and the (.venv) prompt first. If it still fails, rename your project and copy the snapshot again.

Next, you add the current assistant’s saved answers to the project, score them, and make them the baseline that every new version must beat.

Next: Establish a baseline