Extract product recalls into structured data

What you will be able to do

Turn real product-recall notices into structured records that a safety team can search, and measure how often each field is right. You build the checks that stop a bad record. You compare plain code with three models on 40 notices checked by hand, and find the errors that pass every check. Recalls are safety information, so a person checks every record before anyone uses it.

Case studyPortfolio piece

Dataset

CPSC recall notices, home and garden, April to June 2026 (40 notices, and 1 new)

Licence

US government work; CPSC allows free copying of recall notices

Time

About 3 hours

Runs in

VS Code, Python 3.12+

Skills used

• Information extraction
• Pydantic schema
• Structured output
• Business checks
• Hand-checked reference
• Field-level accuracy
• Cost per 1,000 notices
• Mock provider

Download the starter project (.zip)Finished project (.zip)

Unzip the starter, open the folder in VS Code, and follow the steps below. They start with a new virtual environment.

The starter is a small Python project, recall-extractor. It reuses parts of the course project: the adapter, the mock provider and the two gates. It has the 41 notices, the hand-checked reference, and real model answers. You write the business checks and the scoring. The finished project is the result, for when you are stuck or want to compare. The code is under the MIT licence, so you can publish your version.

Before you startWhat to know and to have readyAbout 180 minutes

What you need to know

This case study uses Modules 2 and 3 most:

• A prompt as a specification, from Specify the task. Untrusted text between tags, from Provide context safely.
• A fair comparison on a fixed set, from Compare prompts.
• Pydantic models and strict structured output, from Define a schema: Structured output.
• Gate 1 and its problem codes, from Handle incomplete results.
• Gate 2, and “valid JSON can still be wrong”, from Validate meaning.
• The mock provider and the adapter, from A first run without an account and The application boundary. Cost per 1,000 requests, from Measure usage and cost.

The page teaches one new idea: information extraction, and how to measure it field by field.

What you need on your computer

• Python 3.12 or newer and VS Code, as in the course. About 60 MB of free disk space for the packages.
• No account and no key. The mock replays the recorded answers. A live run with a local model is optional, at the end.

Tested with Python 3.12.13 and 3.14.6 on macOS (Apple silicon). Versions: openai 3.26.1, pydantic 2.14.0, httpx 0.28.1, pytest 9.1.1. The Windows commands follow the course’s commands, but they are not tested.

The situation

LarkfieldTermLarkfieldAn online shop for the home. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more AI in everyday productsAI Career Foundations: Roles, Skills and Roadmaps sells home and garden products, and many of them are also sold in the United States under the same brand. When a product is dangerous, the US Consumer Product Safety Commission (CPSC) publishes a recall notice. It says what the product is, why it is dangerous, what buyers should do, and how many units were sold.

GraceTermGraceThe lead of Larkfield’s help desk. She writes the support policy and decides what a good answer to a customer is.’s help desk gets questions such as “Is my heater one of the recalled ones?” Today, Grace reads every new notice by hand: CPSC published 471 from January to early October 2026, about 12 a week.

Grace wants a safety list with one row per notice. Each row has the product, the hazard, the units, the remedy, the dates of sale, the shops and the contact. Her team can then filter it (“heaters, fire hazard, sold since 2024”) and match it with Larkfield’s products. AmiraTermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. builds the tool that fills the list. OmarTermOmarLarkfield’s product manager. He asks the business questions: what a change costs, and what it gives the shop. asks what it costs. PriyaTermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses. measures how often it is right. TomásTermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. has ideas for shortcuts.

The decision stays with Grace: which recalls need action this week. The tool prepares the records. It does not decide, and a person checks every record before it goes on the list.

Larkfield and the team are invented. The notices are real.

The objective

Turn each notice into a record of fixed fields, so that a person can check it in a minute instead of reading the whole notice. The measures were fixed before any model ran:

• Fields right, against a reference that a person checked by hand: 10 fields on each of 40 notices, so 400 values.
• Notices with every field right: a record with one wrong field still needs a fix.
• Gate 1 failures (no usable record) and gate 2 flags (a value that does not agree with the notice), counted apart.
• Cost per 1,000 notices and the time per notice.

Grace’s bar: if more than about 1 field in 50 is wrong, checking the drafts takes longer than reading the notices. So the target is 98% of fields right, with every failure and flag visible to the person.

The idea: extraction

Think of a clerk who copies paper forms into a spreadsheet. Some boxes are easy: the date is printed, and the clerk only changes its format. Some need care: “about 5,600, and 516 more in Canada” must become one number. Some need judgement: is a canopy bed “furniture”? A good clerk follows written rules, and leaves a box empty when the form says nothing.

Information extraction is that job, done by a program. The input is text written for people. The output is a record: the same fields for every text, with values in fixed forms. A model helps here because it reads text as a person does. The schema from Define a schema fixes the shape of the record. The picture shows one real notice, line by line, and the fields that come from each line:

One notice becomes one record. Each field needs a different kind of work: copy, change the format, add up, split, or choose from a list.

Each arrow names the kind of work, because each kind fails in its own way. A copy can come from the wrong place. A sum can add the wrong numbers. A choice from a list can be a guess that still looks right. The schema checks only that a value has the right shape. It cannot see whether the value is true.

The data

The notices come from the CPSC Recalls API. It gives every notice as JSON, with the same sections as the notice page on cpsc.gov.

Source and licence

Recall notices are works of the US government. CPSC’s privacy and security notice says: “You may freely copy and distribute recall notices, including photographs of recalled items, without permission.” CPSC asks for credit, and the information “may not be used in a way that states or implies CPSC endorsement.” This case study credits CPSC, keeps no photos, and says that its records are not official.

The sample was fixed once, on 2026-10-08:

1. Download every recall from 2026-01-01 to 2026-10-07: 471 notices.
2. Keep the recall dates from April to June 2026: 160 notices.
3. Keep the 49 whose product a home-and-garden shop could sell. Leave out toys, clothing, medicines, vehicles and sports gear.
4. Draw 40 of them with a fixed random seed. Keep one later notice, 26773 (September 2026), apart for the end-to-end run.

The 41 notices are in data/notices.json, with a SHA-256 checksum that the project checks when it loads them. Two other sources were compared. Health Canada’s open recall data has short labels only, without units, shops or dates. openFDA does not cover home products. The project’s DATASET.md has the details.

First checks

Reading the 40 notices found four problems in the source. They are real, and the project keeps them:

Notice	What is wrong
26554	The title is about Arctic Zone coolers. Every other field is about children’s pajamas. The API joined a title to the wrong notice.
26444	The title says that Thermos recalls 8.2 million units. The Units line says 5.8 million and 2.3 million: 8.1 million.
26406	The brand is BAYOTAK in one line and BATOYAK in another.
26423	The email runs into the next word: LR023784@outlook.comor.

So the reference follows written rules. For 26554, the record follows the body, because the body has the hazard and the contact. For 26444, the record follows the Units line.

The sample is also easier than real life. The notices have labelled sections, and many are near copies: 9 pool or spa drain covers, 5 dressers, 4 pressure washers. Keep that in mind when you read the results.

The reference: 40 records checked by hand

A score needs a correct answer to compare with. The case-study author was the checker: the author read each notice and wrote its record by hand, with the rules that the model gets (recalls/prompts/extract.md). That happened before any model ran, and the project’s history shows it. After the runs, the author read again the 3 fields where a hosted modelTermHosted modelA model that runs on a provider’s computers. Your code sends it requests over the internet, and you pay for each call.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One disagreed. No value changed.

So the reference is one careful reading of written rules, not the truth. One person checked each notice once. A second checker would make it stronger.

Set up the project

The steps use VS Code and its terminal. Every command runs in the project folder.

1. Download the starter project from the box at the top of this page. Unzip it into your projects folder. You get the folder recall-extractor.

2. In VS Code, choose File > Open Folder, and open recall-extractor. Then open a terminal: Terminal > New Terminal.

3. Make and activate a virtual environmentTermVirtual environmentA folder with its own Python and its own packages, for one project. Packages that you install there do not change other projects.Learn more Isolate dependenciesPython for Practical AI Engineering, and install the packages:

   macOS / Linux

   python3 -m venv .venv
   source .venv/bin/activate
   python -m pip install -r requirements.txt

   Windows

   python -m venv .venv
   .venv\Scripts\Activate.ps1
   python -m pip install -r requirements.txt

4. Run the tests:

   python -m pytest

   The last line says 12 failed, 20 passed, and the time. The 12 failures are your work: the business checks (Part 3) and the scoring (Part 4). When they pass, the project is complete.

5. Look at one notice, as the model sees it:

   python -m recalls show 26382

   The output starts with:

   Recall number: 26382
   Title: Samson International Recalls Bellevue and Oaklynn Canopy Bed Frames Due to Impact and Injury Hazards
   Name of product: Universal Broadmoore brand Bellevue and Oaklynn model canopy bed frames
   Hazard: The canopy beams of the bed frames can collapse primarily when the bed is moved, posing impact and injury hazards to consumers.
   Remedy: Repair
   Recall date: April 2, 2026
   Units: About 5,600 (In addition, about 516 units were sold in Canada)

The project’s files, by job:

File	Its job
recalls/notices.py	Loads the notices, checks the SHA-256, and lays each record out as text
recalls/schema.py	The record, as a Pydantic model and as a strict JSON schema
recalls/prompts/extract.md	The rules for each field: the same text with and without the schema
recalls/providers.py, recalls/config.py	The course’s adapter: the mock (default) or any OpenAI-compatible server
recalls/checks.py	Gate 1, parse(), and gate 2, check_rules()
recalls/score.py	The comparison with data/reference.json
recalls/baseline.py	The baseline without a model
recordings/*.jsonl	The answers that the mock replays

Part 1: the schema and the rules

Open recalls/schema.py. The record has 13 fields. These lines do most of the work:

recalls/schema.py (part)

Hazard = Literal["fire", "burn", "electric_shock", "fall", "tip_over", "entrapment", "drowning", "impact",
                 "laceration", "amputation", "chemical", "poisoning", "choking", "carbon_monoxide", "other"]

class RecallRecord(BaseModel):
    model_config = ConfigDict(extra="forbid")
    ...
    hazards: list[Hazard] = Field(min_length=1, max_length=4,
                                  description="Every hazard type that the notice names. Use other if none fits.")
    hazard_quote: str = Field(min_length=1, max_length=600,
                              description="The sentence of the notice that states the hazard, copied word for word.")
    units: int | None = Field(description="Units sold in the United States, as a whole number. ...")
    sold_from: str | None = Field(pattern=r"^\d{4}-\d{2}$",
                                  description="The first month of sale, YYYY-MM. Null if the notice does not say.")

• Enums with “other”. Short lists make filtering possible: “every fire hazard” becomes one query. other gives the model an honest answer when nothing fits.
• Null instead of a guess. In strict mode every field is required, so “unknown” must be a value that the model may write: null.
• A quote as evidence. hazard_quote shows where the hazard came from. Gate 2 checks that the quote is really in the notice.
• A fixed form for every value. Dates are YYYY-MM-DD, months YYYY-MM, units a whole number, so code can sort and compare them.

The rules for each field are in recalls/prompts/extract.md. For example: “Do not add units sold in Canada, Mexico or in an earlier recall.” The same rules guided the hand-checked reference. So each model is judged against the rules that it was given, not against a secret answer key.

recalls/request.py builds the request in two ways. With the schema, it adds response_format with strict: true, as in Define a schema. With --plain, it sends the same messages without it. Only the schema changes, so the comparison is fair.

Part 2: one notice end to end

1. Extract notice 26382 with chat-smallTermchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs and the schema (the defaults):

   python -m recalls extract 26382

   The output (gpt-6-luna on Azure, replayed by the mock):

   {
     "recall_number": "26382",
     "product_name": "Universal Broadmoore brand Bellevue and Oaklynn model canopy bed frames",
     "category": "furniture",
     "hazards": [
       "impact"
     ],
     "hazard_quote": "The canopy beams of the bed frames can collapse primarily when the bed is moved, posing impact and injury hazards to consumers.",
     "units": 5600,
     "remedies": [
       "repair"
     ],
     "recall_date": "2026-04-02",
     "sold_from": "2024-10",
     "sold_to": "2026-02",
     "retailers": [
       "Costco.com"
     ],
     "contact_phone": "800-357-0701",
     "contact_email": "samson@rsvpcomm.com"
   }
   valid | chat-small | schema | 1791 tokens in, 274 out (127 reasoning) | 2.26 s | US$0.00032

2. Compare the record with the picture above. The model left out the 516 units sold in Canada, and it did not add “injury” as a hazard type. Both follow the rules.

3. Note the status: valid. In the starter, gate 2 is still empty, so every record that passes the schema is valid. You write gate 2 next.

Part 3: gate 2, checked against the notice

Gate 1, parse(), is already in recalls/checks.py. In Validate meaning, gate 2 used Larkfield’s policy and orders. Here, there is one source of truth: the notice itself. Every value must be visible in the notice, and the dates must make sense.

Problem code	The check	It catches
wrong_notice	The recall number is the notice’s number	an answer about another notice
date_in_future	The recall date is not after today	an invented year
date_not_in_notice	The recall date, written as “April 2, 2026”, is in the notice	a wrong day
dates_out_of_order	sold_from is not after sold_to	swapped dates
sold_after_recall	sold_to is not after the recall month	an invented end of sale
units_not_positive	Units, when given, are more than 0	a negative or zero count
units_not_in_notice	The number is written in the notice (5600, 5,600 or “1.7 million”)	an invented or added-up number
quote_not_in_notice	The hazard quote is in the notice, word for word	a quote that the model wrote itself
hazard_not_in_notice	Each hazard type has its word in the Hazard line or the title (“drown” for drowning)	a hazard that the notice does not name
retailer_not_in_notice	Each shop is named under Sold at	an invented shop
phone_not_in_notice, email_not_in_notice	The phone and the email are in the consumer contact	an invented contact

The checks compare text after small changes: lower case, one space between words, straight quotes. checks.py already has these helpers (_flat, _long_date, _month, _units_forms) and the word lists in HAZARD_WORDS. “Today” is fixed at 2026-10-08, the day of the download, so the results do not change with the calendar.

1. In recalls/checks.py, replace the function check_rules with this one:

   recalls/checks.py

   def check_rules(record: RecallRecord, notice: Notice, today: date) -> list[Problem]:
       """Gate 2: valid JSON can still be wrong. Check it against the notice and the calendar."""
       problems = []
       text = _flat(notice.text)
       hazard_text = _flat(section(notice.text, "Hazard") + " " + section(notice.text, "Title"))
       sold_at = _flat(section(notice.text, "Sold at"))
       contact = _flat(section(notice.text, "Consumer contact"))
   
       if record.recall_number != notice.number:
           problems.append(Problem("wrong_notice", f"Recall number {record.recall_number} is not {notice.number}."))
       recalled = date.fromisoformat(record.recall_date)
       if recalled > today:
           problems.append(Problem("date_in_future", f"The recall date {recalled} is after today ({today})."))
       if _flat(_long_date(recalled)) not in text:
           problems.append(Problem("date_not_in_notice", f"The notice does not give {_long_date(recalled)}."))
       if record.sold_from and record.sold_to and _month(record.sold_from) > _month(record.sold_to):
           problems.append(Problem("dates_out_of_order", f"Sold from {record.sold_from} is after {record.sold_to}."))
       if record.sold_to and _month(record.sold_to) > (recalled.year, recalled.month):
           problems.append(Problem("sold_after_recall", f"Sold to {record.sold_to} is after the recall date."))
       if record.units is not None:
           if record.units <= 0:
               problems.append(Problem("units_not_positive", f"Units must be a positive number, not {record.units}."))
           elif not any(_flat(form) in text for form in _units_forms(record.units)):
               problems.append(Problem("units_not_in_notice",
                                       f"{record.units:,} units is not written in the notice: a person checks the sum."))
       if _flat(record.hazard_quote) not in text:
           problems.append(Problem("quote_not_in_notice", "The hazard quote is not in the notice word for word."))
       for hazard in record.hazards:
           if not re.search(HAZARD_WORDS[hazard], hazard_text, re.IGNORECASE):
               problems.append(Problem("hazard_not_in_notice", f"The notice's hazard text does not mention {hazard}."))
       for name in record.retailers:
           if _flat(name) not in sold_at:
               problems.append(Problem("retailer_not_in_notice", f"{name!r} is not named under Sold at."))
       if record.contact_phone and _flat(record.contact_phone) not in contact:
           problems.append(Problem("phone_not_in_notice", f"{record.contact_phone} is not in the consumer contact."))
       if record.contact_email and _flat(record.contact_email) not in contact:
           problems.append(Problem("email_not_in_notice", f"{record.contact_email} is not in the consumer contact."))
       return problems

2. Run the tests of gate 2:

   python -m pytest tests/test_checks.py

   The last line says 5 passed. One test gives notice 26406 a record with six wrong values and expects six problem codes. Another gives notice 26382 the US and Canadian units added up (6,116) and expects units_not_in_notice.

3. Extract the Thermos notice:

   python -m recalls extract 26444

   The record has "units": 8100000, and the last lines are:

     units_not_in_notice: 8,100,000 units is not written in the notice: a person checks the sum.
   flagged | chat-small | schema | 1944 tokens in, 355 out (162 reasoning) | 3.63 s | US$0.00037

The Thermos record is right: 5.8 million and 2.3 million make 8.1 million. Gate 2 flags it anyway, because the sum is not written anywhere. That is the purpose of a flag. It does not say “wrong”. It says “a person must look at this value first”. A check that a value appears in the text cannot verify arithmetic, so it hands the arithmetic to a person.

Part 4: score field by field

A record can be partly right: nine right fields and one wrong one. So the score compares each of 10 fields with the reference. Two free-text fields are not scored: the product name, which a person reads anyway, and the hazard quote, which gate 2 checks.

Some values need a fair form before the comparison. “Walmart” and “Walmart.com” are the same shop for Grace’s team. Two lists with the same items in another order are equal. A phone number is its digits.

1. In recalls/score.py, replace the three functions retailer_key, normalise and compare with these:

   recalls/score.py

   def retailer_key(name: str) -> str:
       """'www.Walmart.com' and 'Walmart' are the same shop for the safety team: 'walmart'."""
       name = name.lower().strip()
       name = re.sub(r"^www\.", "", name)
       name = re.sub(r"\.com$", "", name)
       return re.sub(r"[^a-z0-9]", "", name)
   
   
   def normalise(field: str, value):
       """Put a value in the form we compare: sets for lists, digits for phones, lower case for emails."""
       if value is None:
           return None
       if field in ("hazards", "remedies"):
           return frozenset(value)
       if field == "retailers":
           return frozenset(retailer_key(v) for v in value)
       if field == "contact_phone":
           return re.sub(r"\D", "", value)
       if field == "contact_email":
           return value.lower()
       return value
   
   
   def compare(record: RecallRecord | None, reference: dict) -> dict[str, bool]:
       """True for each field that matches the reference."""
       if record is None:
           return {f: False for f in FIELDS}
       data = record.model_dump()
       return {f: normalise(f, data[f]) == normalise(f, reference[f]) for f in FIELDS}

2. Add import re below import json at the top of the file.

3. Run all the tests:

   python -m pytest

   The last line says 32 passed.

A notice without a record scores 0 on every field. A flagged record is still scored, and the flags are counted apart: a flag is not a wrong answer.

Part 5: a baseline without a model

Before any model, ask what plain code can do. The notices have labelled lines, so recalls/baseline.py reads them with patterns: the date with strptime, the first number of the Units line, the options of the Remedy line, the first phone and email, and keywords for the hazard and the category.

python -m recalls eval --baseline

The output:

baseline (no model) | 40 notices
gate 1 (schema): 40 passed
gate 2 (business checks): 0 flagged
  category       33/40
  hazards        36/40
  units          39/40
  remedies       39/40
  recall_date    40/40
  sold_from      40/40
  sold_to        40/40
  retailers      31/40
  contact_phone  40/40
  contact_email  39/40
fields right: 377/400 (94.2%) | notices with every field right: 21/40

94.2% costs nothing, and the dates and the phone are perfect: use plain code for fields that are already structured. The baseline fails where the work is reading: the category, the hazards and the shops.

The baseline’s patterns were written while reading these same 40 notices, so 94.2% flatters it. Run python -m recalls errors --baseline to see its 23 wrong fields. Every one of them passed both gates. Two of them:

26423 contact_email got "LR023784@outlook.comor" | reference "LR023784@outlook.com" | passed both gates
26444 units         got 5800000 | reference 8100000 | passed both gates

5.8 million is written in the notice, so gate 2 cannot object, but it is only the first part of the sum. The broken email is in the notice too, typing error included. Gate 2 checks that a value comes from the notice. It cannot check that it is the whole value, or the right one.

Part 6: the models, one fair comparison

Three models answered all 40 notices, each with the schema and with the plain prompt. They are gpt-6-luna (chat-small) and gpt-6.1-sol (chat-strong) on Azure, and Gemma 3 4BTermGemma 3 4BA small open model from Google, with 4 billion parameters. It can run on a laptop with Ollama, without an account or a cost per call.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One through OllamaTermOllamaA free program that downloads open models and runs them on your own computer. Your code calls it at a local address, as it calls a hosted API.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One 0.40.1 on a Mac mini. The requests are identical except for the model and the schema. The mock replays every answer.

python -m recalls eval
python -m recalls eval --model chat-strong
python -m recalls eval --model gemma3:4b

The output of the first command:

chat-small | schema | 40 notices
gate 1 (schema): 40 passed
gate 2 (business checks): 1 flagged {'units_not_in_notice': 1}
  category       40/40
  hazards        39/40
  units          40/40
  remedies       40/40
  recall_date    40/40
  sold_from      40/40
  sold_to        40/40
  retailers      38/40
  contact_phone  40/40
  contact_email  40/40
fields right: 397/400 (99.2%) | notices with every field right: 37/40
median tokens in / out: 1796 / 270 | latency median 2.40 s, p95 3.46 s
cost per 1,000 notices: US$0.33 (prices in usage.py)

Run the other commands, and each again with --plain. Together:

Approach	Gate 1 passed	Gate 2 flagged	Fields right (of 400)	Every field right (of 40)	Tokens in / out (median)	Seconds (median / p95)	US$ per 1,000 notices
Plain code (baseline)	40	0	377 (94.2%)	21	none	none	0
Gemma 3 4B, plain prompt	0	0	0	0	1,301 / 250	9.04 / 11.44	0 (local)
Gemma 3 4B, schema	9	6	77 (19.2%)	1	1,303 / 251	8.87 / 12.13	0 (local)
chat-small, plain prompt	40	1	397 (99.2%)	37	1,214 / 285	2.71 / 3.90	0.28
chat-small, schema	40	1	397 (99.2%)	37	1,796 / 270	2.40 / 3.46	0.33
chat-strong, plain prompt	40	1	398 (99.5%)	38	1,214 / 184	2.72 / 4.14	4.45
chat-strong, schema	40	1	398 (99.5%)	38	1,796 / 151	2.76 / 4.35	5.34

Fields that match the hand-checked reference: 40 notices × 10 fields, models with the schema (the baseline needs none). A notice without a record scores 0 on every field.

Text description of the chart

A column chart with four columns. Code (the plain-code baseline): 377 of 400. Gemma (Gemma 3 4B with the schema): 77. gpt-6-luna (chat-small): 397. gpt-6.1-sol (chat-strongTermchat-strongThe name that Larkfield’s code gives to a larger hosted model: gpt-6.1-sol on Azure. It is often more accurate than chat-small, and it costs about 14 to 18 times more per call.Learn more Request anatomyBuilding Reliable Applications with LLM APIs): 398. The two hosted models are near the top. The small local model is far below the plain code.

What the table says, most important first:

• The hosted models reach the target. chat-small has 397 of 400 fields right (99.2%), chat-strong 398. One field on 40 notices is not evidence of a difference: see How sure can 24 tickets be? in the language models course.
• chat-strong costs about 16 times as much. At about 600 notices a year, that is about US$3.30 against US$0.20 a year. Both are small next to the time of the person who checks, but there is no reason to pay 16 times more.
• For the hosted models, the schema did not change one field. Every plain answer was clean JSON. The schema adds 582 input tokens per notice: about US$0.05 more per 1,000 notices with chat-small.
• For the small local model, the schema decided everything. Without it, all 40 answers were wrapped in a MarkdownTermMarkdownA plain-text way to show formatting, for example **bold** or # for a heading. Models often add it, so a program that expects plain JSON must remove it or reject the answer. code fence (three backticks and the word json), so json.loads failed. With it, every answer was JSON, but 30 failed the schema anyway, and one request failed on the server (HTTP 500, “token repeat limit reached”).
• Plain code beats the small model by far, and loses to the hosted models by 20 fields.

What the schema stopped, and what it let through

Look at Gemma’s answer to the Thermos notice, with the schema:

python -m recalls extract 26444 --model gemma3:4b

The output:

The answer starts with:
{
  "recall_number": "26444",
  "product_name": "Thermos Stainless King 3000 and 3020 Food Jars and Thermos Sportsman 3010 Food & Beverage Bottles",
  "category": "home_appliance",
  schema: sold_from: String should match pattern '^\d{4}-\d{2}$'; sold_to: String should match pattern '^\d{4}-\d{2}$'
rejected | gemma3:4b | schema | 1457 tokens in, 317 out (0 reasoning) | 12.13 s | US$0.00000

The recording shows the rest. Gemma wrote "sold_from": "2008-03-01" and "sold_to": "2024-12-31": days, where the schema asks for months. Ollama sent the schema to the model, but it did not enforce the pattern, as the course saw in Define a schema. Gate 1 in our own code did enforce it, and stopped the record.

That format error hid factual errors. The end of sale is December 2024, a month that the notice never names. The units are 8,200,000, from the title. The hazards add fall and tip_over, which the notice never names. With months in the right form, gate 2 would flag the two hazards. It would pass the units, because the title really says “8.2 Million”. It would also pass the category home_appliance for a food jar.

Now list every wrong field that passed the schema:

python -m recalls errors
python -m recalls errors --model chat-strong
python -m recalls errors --model gemma3:4b

For the hosted models, the lists are short. Every one of these records passed both gates:

Model	Notice	Field	The record	The reference
chat-small	26485	retailers	“Costco retail stores”, “Costco.com”	“Costco”, “Costco.com”
chat-small, chat-strong	26537	retailers	“HSN.com”, “HSN televised shows”, “HSN digital shopping platforms”	“HSN.com”, “HSN”
chat-small, chat-strong	26582	hazards	fire, burn, other	burn, other

These three are close calls of the rules. Notice 26582 is a smoke and carbon monoxide detector that “can fail to alert consumers of a fire, posing a risk of serious injury or death from smoke inhalation or burns.” The rule lists the hazards that the notice “names as a risk”: smoke inhalation and burns. All four hosted runs added fire, and a careful reader could agree with them. When two careful readers disagree, the rule needs one more sentence.

Gemma’s 9 records have 13 wrong fields, and they are a different kind of error:

Notice	Field	The record	The reference	Gates
26512	sold_to	2025-12	2025-10	passed both
26423	hazards	electric_shock, other	electric_shock	passed both
26377, 26445, 26504, 26532	sold_to	2026-12	2026-02 to 2026-05	flagged: sold_after_recall
26394, 26537	hazards	fire added to burn (26394), chemical added to fire (26537)	burn; fire	flagged: hazard_not_in_notice
26445, 26504, 26532, 26537	hazards, category, retailers	chemical added (26445), other added (26504, 26532), home_appliance for a heater (26532), “HSN televised shows” (26537)	the rules’ values	in a flagged record, but this field not flagged

The notice for 26512 says “through October 2025”, and the record says December. The value is invented, in the right form and range, and it passed both gates. Four more inventions of “December” were caught only because they fell after the recall date. Gemma’s chemical for the kerosene notice was not flagged, because the firm’s name, Alliance Chemical, is in the title. Word checks are a safety net, not a judge, as in Validate meaning.

Common mistake

It passed the schema, so it is right

Symptom: Tomás looks at the run of Gemma with the schema. “Every answer that came back is JSON with the right types. Let’s put the 9 that passed straight on the safety list.” Of those 9 records, 6 have a flag. Two of the other 3 have a wrong field. One adds a hazard type that the notice does not name, and one ends the sale two months late.

Cause: the schema checks the shape of each value: a month is YYYY-MM, a hazard is one of 15 words. It cannot know whether the notice says October or December. Strict structured outputTermStructured outputA request that sends the schema of the answer to the provider, so the model can write only an answer of that shape. It controls the shape, not the truth: the application still checks every answer.Learn more Define a schemaBuilding Reliable Applications with LLM APIs even makes a confident value in the right shape more likely, whether or not it is true.

Fix: keep both gates, and treat a record that passes both as a draft, not a fact. A person reads every record next to its notice, flagged values first. Measure on a reference set, so that you know how often the drafts are wrong. In Validate meaning the same mistake was an invented order ID; here it is an invented month.

The final approach

The team chooses chat-small with the schema, both gates, and a person who approves every record:

Every path ends with a person. The gates decide only what the person looks at first.

• chat-small, because chat-strong was not measurably better on 40 notices, and costs 16 times more.
• The schema, although it did not help chat-small here. For about US$0.05 per 1,000 notices, it guarantees the shape if the team later changes the provider, the model or the prompt.
• Both gates. Gate 1 stopped 30 of Gemma’s 40 answers with the schema, and all 40 without it. Gate 2 flagged invented months, and a sum that a person must check.
• A person, because records that passed both gates still had wrong fields, with every model.

End to end: a new notice

Notice 26773 is from September 2026. It was not in the sample, and it has no reference record, so this run is like next week’s real work.

python -m recalls extract 26773

The output:

{
  "recall_number": "26773",
  "product_name": "Charbroil Bistro Pro Electric Grills",
  "category": "outdoor_living",
  "hazards": [
    "electric_shock"
  ],
  "hazard_quote": "The grounding wire for the grill's heating element can disconnect and lose fault protection, posing an electric shock hazard.",
  "units": 10605,
  "remedies": [
    "repair",
    "other"
  ],
  "recall_date": "2026-09-17",
  "sold_from": "2025-10",
  "sold_to": "2026-06",
  "retailers": [
    "Ace Hardware",
    "Lowe's",
    "Walmart",
    "Wayfair",
    "QVC",
    "Amazon.com",
    "Charbroil.com",
    "Ace Hardware.com",
    "Lowes.com",
    "Walmart.com",
    "Wayfair.com",
    "QVC.com"
  ],
  "contact_phone": "866-671-7988",
  "contact_email": "recall@charbroil.com"
}
valid | chat-small | schema | 2066 tokens in, 324 out (139 reasoning) | 2.79 s | US$0.00037

Grace checks the record next to the notice. The units leave out the 342 sold in Canada, as the rule says, and other is the US$25 gift card that Char-Broil offers after the repair. Then a buyer searches Larkfield’s catalogue for “Bistro Pro”. If Larkfield sells it, the team reads Health Canada’s notice before anyone answers a customer.

Insights and limits

• Measure before you choose a model. Plain code reached 94.2% for free; the hosted models added 20 fields, where the work is reading and choosing.
• The schema guarantees shape, not truth. It changed nothing for the hosted models, and the local model’s records still had errors inside.
• “Is the value in the text?” is a cheap, partial check. It caught invented months, and missed a half sum, a broken email and a wrong category.
• The reference is one reader’s work, and the sample is easy. Labelled sections, near copies, one source, one quarter. Other notices need their own reference.
• Models and APIs change fast. These numbers are a snapshot. The method lasts: written rules, a hand-checked reference, two gates, field scores, and a person at the end.

Responsible use

Recalls are safety information

A wrong extraction can hide a hazard. A record can say “burn” when the notice says “fire”. It can give 5,800,000 units instead of 8,100,000, or the wrong end of sale. Then a person can decide that a product is safe when it is not.

• A person checks every record next to its notice before anyone uses it, flagged values first.
• Never present extracted data as official. Every record links to its notice on cpsc.gov, and the list says that it is a working copy. CPSC does not allow uses that imply its endorsement.
• Keep the notice as the source. When a customer asks, the help desk answers from the notice, not from the record.
• Missing is not safe. A notice that the tool could not read (no record, or a provider failure) goes to a person. It never becomes “no recall found”.
• Measure again after any change of model, provider, prompt or schema, on the same reference, before the team relies on the new version.

Your turn

1. A check for the months. Gemma’s invented “2025-12” for notice 26512 passed both gates. Add a check, sold_month_not_in_notice: the month and year of sold_from and sold_to (for example “October 2025”) must be in the Sold at line. Run python -m recalls eval --model gemma3:4b before and after. How many more of Gemma’s records are flagged? Does any hosted record get a false flag? Think about notices that give only a year.
2. Sharpen one rule. Write one more sentence for the hazards rule that settles notice 26582, and decide what the reference should say. Then explain why the recordings cannot measure the new rule. Hint: the key that the mock uses to find a recording includes the prompt.
3. Optional, live and free: a local model. Install Ollama, run ollama pull gemma3:4b, copy .env.example to .env, and set RECALLS_PROVIDER=openai_compatible and RECALLS_BASE_URL=http://localhost:11434/v1. Then run python -m recalls extract 26382 --model gemma3:4b. Your answer can differ from the recording. One notice took about 9 seconds on the Mac mini where we recorded, so a live eval of 40 notices takes several minutes. Set RECALLS_PROVIDER=mock again when you are done.

Show an example answer for task 1

recalls/checks.py (in check_rules, before return problems)

    for label, value in (("sold_from", record.sold_from), ("sold_to", record.sold_to)):
        if value:
            year, month = _month(value)
            words = _flat(f"{date(year, month, 1):%B} {year}")
            if words not in sold_at and _flat(f"{date(year, month, 1):%B} of {year}") not in sold_at:
                problems.append(Problem("sold_month_not_in_notice", f"{label} {value} is not in the Sold at line."))

The second form (“January of 2026”) is in notice 26406. On the recordings, Gemma’s flagged records go from 6 to 7: 26512 is now flagged. The hosted runs and the baseline get no new flag. A notice that gives only a year would get a false flag, and a person then checks it, which is safe. This check was written after looking at the errors, so test it on new notices before you trust it.

Make it a portfolio piece

This case study can count toward your certificate. To publish it:

1. Make a Git repository from your project folder and push it to GitHub. .gitignore keeps out the virtual environment and .env.
2. Keep LICENSE (MIT, for the code) and DATASET.md (the source, the licence and the sample).
3. Make sure that python -m pytest passes from a fresh clone.
4. Write the README for a stranger. A README template:

README.md (outline)

# recall-extractor

## What it is
Extraction of 13 fields from US CPSC recall notices into checked records,
for a shop's safety team. Drafts only: a person approves every record.

## Results (40 hand-checked notices, 400 fields, recorded 2026-10-08)
- Plain code 94.2%, gpt-6-luna 99.2% (US$0.33 per 1,000 notices),
  gpt-6.1-sol 99.5% (US$5.34), Gemma 3 4B 19.2% (local).
- The errors that passed every check, and why.

## Design
- Pydantic schema with enums and null; strict structured output.
- Gate 2: every value must be visible in the notice; dates in order.
- Field-level scoring against a reference written before any model ran.

## Limits and responsible use
(easy sample, one reader, not official data, a person checks)

How to describe it in an interview, in three sentences: “I built an extractor that turns US product-recall notices into checked records, with a strict schema and checks against the source text. On 40 notices that I checked by hand, plain code got 94% of fields right, and a small hosted model 99% at 33 cents per 1,000 notices. I also showed which wrong values still passed every check, so a person approves each record.”

Go further:

• Canadian units. Add units_canada for Larkfield, with its own rule and reference values.
• A second source. Extract the same fields from Health Canada notices (Open Government Licence – Canada), with their own reference.
• Match the catalogue. Match product_name to a small made-up catalogue, and measure the false matches.

Knowledge check

Check your understanding

chat-small with the schema and with the plain prompt both got 397 of 400 fields right. Tomás says: 'Then drop the schema: it costs 582 tokens per notice.' What do you answer?

Show the answer

On this sample and this model, the schema did not change any field, and it costs about US$0.05 more per 1,000 notices. But it is not there for accuracy. It guarantees the shape of every answer, whatever the model, the provider or the prompt version. The local model showed the risk: without the schema, 40 of 40 answers were wrapped in a Markdown fence, and none could be parsed. The cost is small, a later change of model is likely, and gate 1 in our own code checks the shape anyway. So keep the schema, and keep measuring.

Check your understanding

Gate 2 flagged chat-small's record for the Thermos notice (units_not_in_notice), but the record is right. Is this check broken? Explain what a flag means here.

Show the answer

It is not broken. The check asks whether the number is written in the notice. 8,100,000 is the right sum of 5.8 million and 2.3 million. But it is not written anywhere, so the check cannot confirm it. A flag means “a person must look at this value first”, not “this value is wrong”. For this reason, flags are counted apart from the field score. The opposite case is worse: the baseline’s 5,800,000 is written in the notice, so it passed, although it is only part of the units.

Check your understanding

A colleague suggests: 'The hosted model is 99.2% right, so publish the records on our website as the official recall list, and save Grace the time.' Give two reasons from this case study not to.

Show the answer

First, 99.2% still means about 1 wrong field in 130, and some wrong values passed both gates. For safety information, a wrong value can hide a hazard, and a customer could decide that a product is safe. Second, the records are not official. CPSC does not allow uses that imply its endorsement, and the notice on cpsc.gov is the source. Also, 99.2% was measured on 40 easy notices with labelled sections and one reader’s reference. It promises nothing about next month’s notices. A person checks every record, and the list links each record to its notice.

Recap

• Information extraction turns text into a record of fixed fields. Each field needs its own kind of work: copy, change the format, add up, split or choose.
• Written rules make the task, the reference and the score agree. Write them before any model runs.
• A schema guarantees the shape. Gate 2 checks values against the source text. Neither proves that a value is true.
• Score field by field against a hand-checked reference, and count failures and flags apart.
• Start with a baseline: plain code reached 94.2% here, and it was perfect on the structured fields.
• Choose the cheapest model that reaches the target. chat-strong was not measurably better and cost 16 times more.
• For safety information, a person checks every record, and a record is never presented as official.

The next case study answers questions over open data with read-only tools, and checks that every number in an answer comes from a tool result.

Next: Answer questions with tools over open data