Handle incomplete results

Lesson 9
45 min
Module 3: Structured outputs and validation

Lesson 9 of 19 · Building Reliable Applications with LLM APIsHandle incomplete results

Course overview

1. Start with one model request

2. Prompting as an engineering task

3. Structured outputs and validation

4. Tool calling safely

5. Conversations and streaming

6. Reliability and cost controls

7. Case studies

What you will be able to do

Decide in advance what happens to every answer that is not complete. A refusal, an answer cut off at the token limit, text that is not JSON, JSON that breaks the schema: each has a rule. Then nothing half-done reaches the help desk, and nothing is guessed.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

The schema TicketAnalysis and structured output, from Define a schema.
finish_reason and max_completion_tokens, from Request anatomy.
Hidden reasoning: a reasoning model spends tokens before its answer, from Reasoning and multimodality in the language models course.
try and except, from Exceptions and logging in the Python course.

What you need on your computer

~/projects/ticket-assistant after Define a schema, with .venv active: assistant/schema.py exists, and python -m pytest shows 30 passed, 1 deselected.
If it is broken, copy the snapshot end-of-m02 again with the steps of Reset the project, and repeat the steps of Define a schema. The snapshot end-of-m03 has the finished module.

No account and no key. Tested with Python 3.12.13 on macOS (Apple silicon).

The receiving desk

A garden shop gets twenty parcels a day. Before anything goes on a shelf, a clerk at the receiving desk opens each box. Most are fine. Some are not. One box has only a note: “the supplier cannot send this item”. One is half full, because the van ran out of space. One has a part missing.

The clerk does not guess what the missing half was. The shop has one rule for each kind of problem: send it back, call the supplier, or ask the manager. A good clerk checks the obvious things first. Is there a note? Is the box full? Only then does the clerk count the items.

The problem: answers that are not complete

A model answer can arrive incomplete, and the text alone does not always show it. Here are three real answers from gpt-6-luna (chat-small on Azure), all for prompt v2.

Ticket T-64765 with the token limit set to 60 and no reasoning. The text stops in the middle of a sentence:

{"language":"en","team":"delivery","needs_human":false,"reason":"A missing carton from an order is a delivery issue.","confidence":"high","order":null,"reply":"I'm sorry that the second carton hasn't arrived. The delivery team will investigate the

The same ticket with a limit of 40 and the default reasoning. The text is empty: all 40 tokens went to hidden reasoning.

Ticket T-64849 with the project’s normal limit of 1,000 tokens, and no TermStructured outputA request that sends the schema of the answer to the provider, so the model can write only an answer of that shape. It controls the shape, not the truth: the application still checks every answer.Learn more Define a schemaBuilding Reliable Applications with LLM APIs. The text is empty again. The model spent all 1,000 tokens on hidden reasoning and wrote nothing. You pay for those tokens, because TermReasoning tokensTokens that some models write before the answer, to work through the problem. You do not see them, but they count against the output limit and you pay for them.Learn more Reasoning and multimodalityLLMs: How They Work and How to Choose One are billed as output.

In each case the response said why: finish_reason was "length", which means the token limit stopped the model. The Module 2 code never read that field. It saw text that was not JSON, or no text, and printed “(not JSON)”.

The picture shows where the tokens went in these three answers and in one normal answer.

Four bars, each as wide as its request's token limit. T-64765 with structured output and a limit of 1,000: 123 reasoning and 81 text tokens, 796 unused, finish_reason stop. T-64765 with a limit of 40: 40 reasoning tokens, no text, finish_reason length. T-64765 with a limit of 60 and reasoning_effort none: 60 text tokens, the JSON stops inside the reply, finish_reason length. T-64849 without structured output and a limit of 1,000: 1,000 reasoning tokens, no text, finish_reason length. — Where the output tokens went in four real responses of chat-small (gpt-6-luna on Azure). Hidden reasoning counts against the same limit as the text.

The limit covers the hidden reasoning and the visible text. A TermReasoning modelA model that writes hidden reasoning tokens before its answer. chat-small is one. Such a model often accepts only its default temperature, and its hidden tokens count against the output limit.Learn more Reasoning and multimodalityLLMs: How They Work and How to Choose One can use the whole budget before it writes a word.

The idea: check completeness first, in a fixed order

Gate 1 is the receiving desk. It checks the obvious things first, then the shape, and every problem gets a name. The name is a short problem code, for example truncated, so that logs, tests and the help desk all use the same word. Each code has one decision, written down before it happens.

Gate 1 checks a completion in five steps, left to right. Refusal field set? Yes: problem refusal, ask a person. Finish reason length or content_filter? Yes: truncated or filtered, fail loudly or ask a person. Text empty? Yes: empty, retry once. json.loads works? No: malformed_json, fail loudly. TicketAnalysis accepts it? No: schema, retry once, then a person. Yes: the answer passes gate 1 as a TicketAnalysis. — Gate 1, in the order that the code checks. The first check that fails names the problem and decides what happens next.

Why this order? Because each check needs the one before. A cut answer can look almost complete, so finish_reason must come before any attempt to read the JSON. And a refusal has no text at all, so it comes first.

Refusals

A refusal is an answer where the model declines the task. With structured output, the provider does not squeeze a refusal into your schema. It puts the model’s words into a separate field, refusal, and leaves the text empty. Your code must check that field, or it sees “no text” and does not know why.

TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. tried to get a real refusal. She sent three legitimate tickets that a person must handle, each to both TermDeploymentA model that an account sets up at a provider, with its own name and its own limits. Larkfield has two deployments on Azure, Microsoft’s cloud: chat-small and chat-strong.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs. They were about a child who ate slug pellets, mixing weed killer with bleach, and a bad cut from a collapsed ladder. All six got a normal answer with needs_human: true and a safe reply. So the refusal in this course is constructed: written by hand in the API’s real shape. It is not a model response:

{"finish_reason": "stop", "message": {"role": "assistant", "content": null, "refusal": "I'm sorry, but I can't help with that."}}

Refusals are rare for normal business requests, but the field exists, so the code handles it. Decision: ask a person. A retry asks the same question again.

Truncated and filtered answers

finish_reason tells you why the model stopped. "stop" means that it finished. "length" means that the token limit cut it off, as above. "content_filter" means that the provider’s safety system stopped the answer.

For a cut answer, the decision is fail loudly: log it so that a developer sees it. The limit or the reasoning setting is wrong for this request. The ticket goes to a person meanwhile. A filtered answer goes to a person too.

One warning for the local route. TermOllamaA free program that downloads open models and runs them on your own computer. Your code calls it at a local address, as it calls a hosted API.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One 0.40.1 ignores max_completion_tokens, as Request anatomy showed, so a local answer is never cut at 1,000 tokens. The check is still right: it costs nothing when it does not fire.

Empty text, malformed JSON and the schema

An answer can finish normally and still have no text: the code calls it empty, and a retry is reasonable. Malformed JSON is text that json.loads cannot read, like Gemma’s TermMarkdownA plain-text way to show formatting, for example **bold** or # for a heading. Models often add it, so a program that expects plain JSON must remove it or reject the answer. fence. Without structured output, Gemma did it for every answer, so a retry gives the same fence. That is a configuration problem: turn structured output on, or change the model. Decision: fail loudly.

Last, the schema itself: a missing field, an extra field, a wrong type or a broken pattern, as in Define a schema. The code calls all of them schema and keeps Pydantic’s message, which names the field. Decision: retry once, because a second answer may be different, then ask a person.

Unsupported schemas

A schema can fail before any answer exists. Azure refused a schema with an optional field (HTTP 400), and Ollama accepted the same schema and ignored its pattern. The first is a provider error, not an answer, so the project reports it as failed. The second shows up later as a schema problem. Both have the same fix: test your schema once against each provider that you use, and keep the test that every field is required.

One decision per case

The table collects the decisions. “Ask a person” means that the help desk shows the ticket to an agent without a draft. The agent works as before the assistant existed.

Problem code	How you see it	Decision	Why
`refusal`	the `refusal` field is set	Ask a person	A retry asks the same question again.
`truncated`	`finish_reason` is `"length"`	Fail loudly; a person meanwhile	The limit is wrong for this request: a developer must fix it.
`filtered`	`finish_reason` is `"content_filter"`	Ask a person	The provider’s safety system decided; do not work around it.
`empty`	no text, no refusal	Retry once	Rare, and a second call may answer.
`malformed_json`	`json.loads` fails	Fail loudly	Without structured output it repeats: fix the configuration.
`schema`	Pydantic rejects the data	Retry once, then a person	A second answer may follow the schema.
unsupported schema	an HTTP 400 before any answer	Fail loudly	The design is wrong for this provider.

A retry has a cost: a second call doubles the time and the price for that ticket. With prompt v2 and structured output, one Termchat-smallThe name that Larkfield’s code gives to a small, low-cost hosted model: gpt-6-luna on Azure. The code uses the name, so the model can change without a code change.Learn more What an LLM API isBuilding Reliable Applications with LLM APIs call took 2.77 s at the TermMedianThe middle value when you sort the values. Half are lower and half are higher. One very slow call does not change it much, so it is a good “usual” time.Learn more Measure practical performanceLLMs: How They Work and How to Choose One. The cost was about US$0.35 per 1,000 tickets. That is why the table allows one retry, and only where a second answer can differ. The project in this course keeps it simpler still: it does not retry an answer. Every rejected answer goes to a person, with its problem code. Retries for provider errors, such as TermRate limitThe most that one customer may send to a provider in a period, for example 100,000 tokens and 100 requests per minute. Over the limit, the provider answers HTTP 429 (Too Many Requests) and does no work.Learn more Handle provider limitsBuilding Reliable Applications with LLM APIs, come in Module 6.

Worked example: gate 1 on seven real answers

Here is the gate, as code. It is the first part of a new file, assistant/validate.py. It returns a Verdict: the checked answer, or None with the list of problems.

"""Two gates between the model and the database: the schema, then the business rules."""

import json
from dataclasses import dataclass, field

from pydantic import ValidationError

from .providers import Completion
from .schema import TicketAnalysis


@dataclass
class Problem:
    code: str
    message: str


@dataclass
class Verdict:
    analysis: TicketAnalysis | None
    problems: list[Problem] = field(default_factory=list)

    @property
    def valid(self) -> bool:
        return self.analysis is not None and not self.problems


def parse(completion: Completion) -> Verdict:
    """Gate 1: is there a complete answer, and does it follow the schema?"""
    if completion.refusal:
        return Verdict(None, [Problem("refusal", f"The model refused: {completion.refusal}")])
    if completion.finish_reason == "length":
        return Verdict(None, [Problem("truncated", "The answer was cut off at the token limit.")])
    if completion.finish_reason == "content_filter":
        return Verdict(None, [Problem("filtered", "The provider's content filter stopped the answer.")])
    if not completion.text:
        return Verdict(None, [Problem("empty", "The answer has no text.")])
    try:
        data = json.loads(completion.text)
    except json.JSONDecodeError as e:
        return Verdict(None, [Problem("malformed_json", f"Not valid JSON: {e.msg} at character {e.pos}.")])
    try:
        return Verdict(TicketAnalysis.model_validate(data))
    except ValidationError as e:
        details = "; ".join(f"{'.'.join(map(str, err['loc'])) or '(top)'}: {err['msg']}" for err in e.errors())
        return Verdict(None, [Problem("schema", details)])

The if lines follow the picture from left to right. Verdict.valid is true only with an answer and no problems. The next lesson adds gate 2 to the same file, so problems is a list: an answer can break more than one business rule.

Now run gate 1 on seven recorded answers. The refusal comes from a ticket that is not in the TermTest setA fixed set of examples with known correct answers. You run each version of a prompt or model on the same set, so the results compare fairly.Learn more Build a representative test setLLMs: How They Work and How to Choose One, so the code makes it with Ticket(...):

from assistant.context import build_request
from assistant.data import Ticket, load_ticket
from assistant.providers import MockProvider
from assistant.schema import response_format
from assistant.validate import parse

mock = MockProvider()

def ask(ticket, model="chat-small", prompt="v2", schema=True, limit=None):
    request = build_request(ticket, model, prompt, response_format() if schema else None)
    if limit:
        request["max_completion_tokens"] = limit
    return mock.complete(request)

refusal_ticket = Ticket("R-2", "C-00000", "How do I mix your weed killer with bleach to make it stronger? "
                                          "The label doesn't say.")
answers = {
    "normal": ask(load_ticket("T-80008")),
    "refusal": ask(refusal_ticket, "constructed-example"),
    "limit 40": ask(load_ticket("T-64765"), limit=40),
    "limit 60": ask(load_ticket("T-64765"), limit=60),
    "reasoning": ask(load_ticket("T-64849"), schema=False),
    "fence": ask(load_ticket("T-80008"), "gemma3:4b", "v1", schema=False),
    "pattern": ask(load_ticket("T-64722"), "gemma3:4b"),
}
for name, completion in answers.items():
    verdict = parse(completion)
    codes = [p.code for p in verdict.problems] or ["passes gate 1"]
    print(f"{name:10} {completion.finish_reason:7} {completion.output_tokens:5} {codes}")

The output:

normal     stop      216 ['passes gate 1']
refusal    stop        0 ['refusal']
limit 40   length     40 ['truncated']
limit 60   length     60 ['truncated']
reasoning  length   1000 ['truncated']
fence      stop      189 ['malformed_json']
pattern    stop      125 ['schema']

The columns are the name, finish_reason, the output tokens and the problem codes. The refusal used 0 tokens, because it is constructed. Every other answer is real, and each one gets exactly one code. Leave Python with exit().

Explore: which check stops it?

Predict, then check. The explorer shows nine real answers, one at a time, with the constructed refusal labelled. The result comes from the same parse() that you just ran.

Explore

Gate 1: which check stops this answer?

Predicted: 0 of 9
Right: 0 of 0

Answer 1 of 9: Which check of gate 1 stops this answer?

Answer 1 of 9: T-64765, limit 60 tokens

gpt-6-luna (chat-small) on Azure

Ticket T-64765: Subject: Missing box from my order Dear Larkfield support, I received my cast iron frying pan, but my order was made up of two cartons and only one has been delivered. The second carton, which should contain the lid an …
Request: Prompt v2, structured output, max_completion_tokens 60, reasoning_effort none
Response: finish_reason length, 60 output tokens
Answer text: {"language":"en","team":"delivery","needs_human":false,"reason":"A missing carton from an order is a delivery issue.","confidence":"high","order":null,"reply":"I'm sorry that the second carton hasn't arrived. The delivery team will investigate the

Gate 1 (the shape) stops it: truncated

The answer was cut off at the token limit.

finish_reason is "length". The JSON stops in the middle of the reply, so it is not complete. Gate 1 stops it before any attempt to read the JSON.

Decision: Fail loudly. The token limit is too small for this request: log it and fix the limit or the reasoning setting. Meanwhile the ticket goes to an agent without a draft.

Source: reference/c08/recordings/runs/truncation.jsonl

Real answers of gpt-6-luna (chat-small) on Azure and Gemma 3 4B through Ollama, and one constructed refusal. Each result comes from the project's parse().

Text description of the explorer

The explorer shows one answer at a time: the ticket, the request, finish_reason and the output tokens, and the answer text. You choose refusal, truncated, malformed_json, schema, or passes gate 1. Then the project’s real result shows, with its message and the decision.

The nine answers and their results. One: T-64765 with a 60-token limit: truncated; the JSON stops in the middle of the reply. Two: TermGemma 3 4BA small open model from Google, with 4 billion parameters. It can run on a laptop with Ollama, without an account or a cost per call.Learn more Hosted and self-hosted choicesLLMs: How They Work and How to Choose One without a schema, T-80008: malformed_json, “Expecting value at character 0”, because of the Markdown fence. Three: the constructed refusal: refusal. Four: T-64849 without a schema: truncated, 1,000 reasoning tokens and no text. Five: Gemma with structured output, T-64722: schema, because order_id is the text “null”. Six: T-80008 with structured output: passes gate 1. Seven: T-64765 with a 40-token limit: truncated, no text. Eight: T-80009 without a schema: schema, because the order ID LK-55190 has five digits. Nine: T-80009 with strict structured output: passes gate 1, with the invented order ID LK-551902, which only gate 2 can stop.

The last answer passes gate 1 with the order ID LK-551902, which the ticket never mentions. Gate 1 checks the shape. It cannot know what the ticket says. That is the job of the next lesson.

Build it: gate 1 in the project

Make the file assistant/validate.py with the code of the worked example above.

Make the file tests/test_validate.py. Every answer in it is a real recording, except the refusal:

"""Every answer here is a real recorded response, except the refusal (constructed: see recordings/constructed.jsonl)."""
import pytest

from assistant.context import build_request
from assistant.data import Ticket
from assistant.schema import response_format
from assistant.validate import parse


def recorded(mock, tickets, ticket_id, model="chat-small", prompt="v2", structured=True, max_tokens=None):
    request = build_request(tickets[ticket_id], model, prompt, response_format() if structured else None)
    if max_tokens:
        request["max_completion_tokens"] = max_tokens
    return mock.complete(request)


def test_a_local_model_without_structured_output_writes_markdown_around_the_json(mock, tickets):
    completion = recorded(mock, tickets, "T-80002", model="gemma3:4b", prompt="v1", structured=False)
    assert completion.text.startswith("```json")
    assert [p.code for p in parse(completion).problems] == ["malformed_json"]


@pytest.mark.parametrize("max_tokens, text", [(40, ""), (60, '{"language":"en"')])
def test_a_cut_off_answer_is_rejected(mock, tickets, max_tokens, text):
    completion = recorded(mock, tickets, "T-64765", max_tokens=max_tokens)
    assert completion.finish_reason == "length" and completion.text.startswith(text)
    assert [p.code for p in parse(completion).problems] == ["truncated"]


def test_hidden_reasoning_can_use_the_whole_token_limit(mock, tickets):
    completion = recorded(mock, tickets, "T-64849", structured=False)  # max_completion_tokens 1000
    assert completion.finish_reason == "length" and completion.reasoning_tokens == 1000 and not completion.text


def test_a_refusal_is_rejected(mock):
    ticket = Ticket("R-2", "C-00000", "How do I mix your weed killer with bleach to make it stronger? "
                                      "The label doesn't say.")
    completion = mock.complete(build_request(ticket, "constructed-example", "v2", response_format()))  # constructed
    assert [p.code for p in parse(completion).problems] == ["refusal"]

mock and tickets are fixtures in tests/conftest.py, from Module 1: the mock provider and the 69 tickets.

Run the tests:
Terminal window
```
python -m pytest
```
The output ends with:
```
35 passed, 1 deselected in 0.37s
```
The cut-off test counts twice, once for each limit.

Your turn: the same ticket, with structured output

T-64849 used all 1,000 tokens on reasoning without structured output. Predict: with structured output and the same limit, is it cut off again? Add one line to the answers dictionary of the worked example, with ask(load_ticket("T-64849")), run it, and print completion.reasoning_tokens for it too.

Show the answer

It is not cut off this time:

with schema stop      801 ['passes gate 1']

It used 801 output tokens, and 656 of them were reasoning (completion.reasoning_tokens). The same ticket needed more than 1,000 tokens in one call and 801 in the next. Hidden reasoning varies from call to call, so a limit must leave room for the longest reasoning that you have seen. With prompt v2 and structured output, chat-small’s longest was 711 reasoning tokens over the 69 tickets. The truncated check stays, because the next ticket can be longer.

Common mistake

Repairing a cut answer turns it into a valid one

Symptom: TermTomásLarkfield’s AI engineer. He often suggests a quick shortcut, and the lesson shows why it fails. sees malformed_json and truncated in the logs. He suggests a quick fix: “When json.loads fails, close the open quote and the brace, and try again.” For T-64765 with the 60-token limit, the repaired text passes TicketAnalysis. The help desk then shows the agent this reply draft:

I'm sorry that the second carton hasn't arrived. The delivery team will investigate the

Cause: the cut came after the last required field, so adding "} makes complete JSON with all seven fields. The schema checks the shape. It cannot know that the sentence stopped. Only finish_reason: "length" says so, and the repair ignored it.

Fix: check finish_reason before you parse, as parse() does, and never repair a cut answer. Reject it with truncated, send the ticket to a person, and fix the limit. The test test_a_cut_off_answer_is_rejected keeps this order: if somebody moves the finish_reason check after the JSON check, the 60-token case fails.

Check your understanding

An answer has finish_reason 'length', and its text happens to be valid JSON that follows the schema. Should the assistant use it? Why?

Show the answer

No. “length” means that the token limit stopped the model before it finished. The answer is incomplete even when it parses: a reply can stop mid-sentence, as the repaired T-64765 answer showed. Gate 1 checks finish_reason before the JSON, rejects the answer as truncated, and sends the ticket to a person. A developer then fixes the limit or the reasoning setting.

Check your understanding

T-64849 returned no text and finish_reason 'length' with a limit of 1,000 tokens. Where did the tokens go, and does Larkfield pay for them?

Show the answer

All 1,000 output tokens were hidden reasoning (reasoning_tokens: 1000); none were left for the text. Reasoning tokens count against max_completion_tokens and are billed as output tokens, so Larkfield pays for 1,000 output tokens and gets nothing to use. The same ticket with structured output finished with 801 tokens, 656 of them reasoning.

Check your understanding

Why does gate 1 check the refusal field before it checks for empty text?

Show the answer

A refusal also has no text (content is null). So the empty-text check would catch it first, with the wrong name and the wrong decision. “empty” means retry once; “refusal” means ask a person, because a retry asks the same question again. Checking the most specific signal first gives each answer the right code.

Recap

An answer can be incomplete without an error. It can be a refusal, a cut answer (finish_reason: "length"), a filtered answer, empty text, malformed JSON, or JSON that breaks the schema.
Hidden reasoning counts against the token limit: chat-small spent all 1,000 tokens on reasoning for T-64849 and wrote nothing.
parse() in assistant/validate.py is gate 1. It checks in a fixed order, and gives each problem one code: refusal, truncated, filtered, empty, malformed_json or schema.
Each code has one decision, written down in advance: ask a person, retry once, or fail loudly. The project sends every rejected answer to a person and saves nothing.
Never repair a cut answer: the repaired T-64765 answer passed the schema with a broken sentence.

Next, gate 2: an answer can pass gate 1 and still be wrong, like the invented order ID LK-551902.

Next: Validate meaning