Skip to content
NextiaNextia Learning

Evaluating and Testing AI Systems

Find out whether a new version of an AI system is really better, and whether it broke anything that matters. You build an evaluation harness for Larkfield's support assistant, with a versioned set of tickets, scores you can trace, human and model judges you have checked, and a release decision you can defend.

Start course See the curriculum

Read every lesson as a guest, or sign in to enroll and track your progress.

What you'll learn

By the end of this course, you will be able to:

  • Design representative evaluation datasets.
  • Choose metrics for models, retrieval and generated outputs.
  • Calibrate human and model-based judgments.
  • Build repeatable regression checks and release criteria.

About this course

A new version of an AI system often feels better. Two nice replies are easy to find, and a demo goes well. But “it feels better” is not evidence. The new version may be better on average and still fail the one ticket that matters most.

This course teaches you to answer two questions with evidence: is the new version better, and did it break anything that matters? You start from what the support team needs and turn it into criteria you can check. Then you build a versioned set of test cases, choose measures for decisions, search results and written replies, and check your human and model judges before you trust them.

You work on Larkfield’s support assistant from Building Reliable Applications with LLM APIs. You do not build the assistant again. You build an evaluation harness around its saved outputs: it loads the cases, scores every output, compares a baseline with a candidate, and writes a report in which every number traces back to a case.

You do not need an account or a paid key. Every output and judgment comes from a real model and is saved with the course. Diagrams and explorers carry the main ideas, so the text stays short.

Before you start

You completed Essential Mathematics and Statistics for AI and Building Reliable Applications with LLM APIs, or you have the same skills. You do not need an API key or a paid account. Every system output and every model judgment is saved, so the whole course runs on your computer; a live model judge with a free local model or your own key is optional.

Courses to take first: Essential Mathematics and Statistics for AI, Building Reliable Applications with LLM APIs.

Who this is for

People who have built an application on a model API, as in Building Reliable Applications with LLM APIs, and now must decide whether a change makes it better. Developers, data scientists, testers and professionals moving into AI engineering.

Lead or use AI, but do not build it?

You can study the ideas of this course without writing code. Read modules 1–5 (about 10–13 hours), answer their knowledge checks, and get a free concepts certificate. You can come back for the practice later.

The paths for people who lead or use AI take this course this way. See the paths

How we made this course

Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses

Inside the course

Module 2ConceptsBuild an evaluation dataset

3 lessons

Build the evaluation set. Choose normal, rare, difficult, French and unanswerable tickets, label them with criteria and record who labelled them, and keep the development and holdout sets apart.

  1. Select representative cases45 min
  2. Label and version50 min
  3. Avoid contamination45 min

Module 3ConceptsChoose component metrics

3 lessons

Choose measures that match the failures you care about. Score decisions with precision and recall, search results with ranking measures, and replies with checks for content, language and refusal, then find what the overall score hides.

  1. Prediction metrics50 min
  2. Retrieval metrics45 min
  3. Generated-output metrics50 min

Module 6Automate evaluation and feedback

3 lessons

Make evaluation repeatable. Structure the harness with stable run IDs, turn the release rules into regression tests, and plan how feedback from production adds new cases without touching the holdout set.

  1. Harness structure45 min
  2. Regression checks50 min
  3. Production feedback45 min

Final module · choose at least oneCase studies

3 case studies

Three end-to-end case studies on open data. Calibrate a model judge against human ratings, compare summary measures with expert ratings, and regression-test a classifier release by slice. Complete at least one for the certificate; each one is a portfolio piece.

Complete at least one case study for the certificate. The others are optional. What each one covers

  1. Calibrate a model judge against human ratings120 min
  2. Evaluate summaries against expert ratings120 min
  3. Regression-test a classifier release by slice120 min

After the case studiesFinal assignment

Use every module in one practice project. You collect evidence, and check it against the course outcomes.

  1. Final assignment300 min

Case studies

Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.

  1. Calibrate a model judge against human ratings2 hr

    Measure how well a model judge agrees with several human raters on an open dataset, find its order bias, and decide where people must still review.

  2. Evaluate summaries against expert ratings2 hr

    Compare automatic measures of summary quality with expert ratings on an open dataset, and find which measure agrees with the experts on which quality.

  3. Regression-test a classifier release by slice2 hr

    Compare two versions of a classifier on an open labelled set overall and by slice, with intervals, and make a release decision with blockers.

Reviews

Loading the reviews…

    Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.

    Share this course