The GenAI conventions for agent runs
- Lesson 5
- 40 min
- Module 2: OpenTelemetry for agents
Where we are
In the last lesson, we made one run into one trace, and our workflow recorded its first span. But if every team names its attributes in its own way, no tool can read them all. In this lesson, we use the OpenTelemetry conventions for generative AI: the names for an agent run, a model call and a tool call, and for tokens, models and errors. We start with the label on a food package.
Lesson 5 of 28 · Observability for AI Agents: Traces, Costs and Audit TrailsThe GenAI conventions for agent runs
1. Why agents need their own observability
2. OpenTelemetry for agents
3. Instrument the workflow
4. Find agent failures in Grafana
5. Cost
6. Audit trails
7. Alerts, service levels and quality
8. Tools: Langfuse, others and the run inspector
What you will be able to do
Give every span of an agent run the names that tools already understand. They name the operation, the model, the tokens, the tool, the prompt version and the error. Pin the version of those names, because they are still changing.
Before you startWhat to know and to have ready
What you need to know
- Spans, traces, the root span and the run clock, from Spans, traces and context.
- What a model call and a tool call are in the workflow, from the agents course, AI Agents and Workflow Orchestration.
What you need on your computer
- Your project as the last lesson left it: the files for spans copied, 79 tests pass.
Tested on macOS (Apple silicon) with Python 3.12.13. The Windows and Linux commands are written, not run.
The label on a food package
Section titled “The label on a food package”Pick up two boxes of cereal from two companies. Each box has a nutrition label. The labels use the same words, “Energy”, “Fat”, “Sugars”, with the same units, per the same serving. So you can compare the two boxes in a few seconds, without knowing either company.
The companies did not choose those words. A rule set them, and the rule sometimes changes: a new line is added, a name is changed. When the rule changes, a careful company says which version of the rule its label follows.
Telemetry needs the same thing. A span’s attributes are its label. If every team writes its own words, no tool can compare two runs.
The problem: a name that no tool knows
Section titled “The problem: a name that no tool knows”Here is a small experiment from this module’s practice. Later in this lesson, you add a span for the workflow’s reply step. Suppose you write the operation’s key as gen_ai.operation, a name that looks right:
attrs = {"gen_ai.operation": c.OP_TOOL, c.TOOL_NAME: "send_reply", ...}The run works, and obs trace shows the span. But it shows nothing about it:
0.0 s | | 0.00 s execute_tool send_reply 0.0 s | | 0.00 s invoke_agent support-resolver reply_only rule W1obs reads the attribute gen_ai.operation.name to decide what a span is. Without it, the span is only a name. It has no “sent” or “failed”, and a later count of tool calls would miss it. The course’s Grafana queries and other tools work the same way. The course team also sent the replayed week to Langfuse, a tool built for traces of AI applications (Module 8 uses it). Langfuse typed the spans from the GenAI operation names. It found 718 model calls, 750 tool calls and 316 agent runs. One wrong key, and a span falls out of every view.
So the names must be agreed, and every span must use them exactly.
The idea: semantic conventions
Section titled “The idea: semantic conventions”OpenTelemetry publishes semantic conventions: agreed names for spans, attributes and metrics, so that any tool can read any application’s telemetry. The GenAI semantic conventions are the part for generative AI: model calls, agents and tools. Their attribute names start with gen_ai..
They name three kinds of spans that an agent run needs. The operation name, gen_ai.operation.name, says which kind a span is:
| Operation | Span name | What it is | In our workflow |
|---|---|---|---|
invoke_agent |
invoke_agent {agent name} |
one run of an agent | invoke_agent support-resolver |
invoke_workflow |
invoke_workflow {workflow name} |
a run where code, or several agents, decide the steps | invoke_workflow support-resolution (fixed, router, two workers) |
chat |
chat {model} |
one call to a model | chat chat-strong |
execute_tool |
execute_tool {tool name} |
one tool call | execute_tool get_order |
Our project also uses approve for a person’s decision. No convention has that operation: it is ours, and the lessons of Module 3 say so where it appears.
The attributes then describe each span. They hold the model asked for and the model that answered, and the tokens. They hold the reason the model stopped, the tool’s name, the prompt’s name and version, and the error.
A worked example: three spans of the double refund
Section titled “A worked example: three spans of the double refund”Read the attributes of real spans from the saved week, all from run R-1014-040:
Text description of the diagram
Each card is one span and some of its attributes. The agent run carries the run’s totals and IDs. The model call carries the model asked for, the model that answered, the tokens, why it stopped, and the prompt’s name and version. The tool call carries the tool’s name and type, and Larkfield’s own read-or-write mark and status. The failed refund call carries error.type = timeout and the span status ERROR.
Now follow the values:
- The model asked for and the model that answered. The model call has
gen_ai.request.model = chat-strong: the name the workflow used. It also hasgen_ai.response.model = gpt-6.1-sol-2026-09-29: the exact model version that the provider says answered. A name likechat-strongcan point to a new model tomorrow. The response model tells you which one it was. - Tokens. The first model call used 2,160 tokens in and 47 out. The second call used 2,426 in and 218 out. The root span has the sums: 2,160 + 2,426 = 4,586 in, and 47 + 218 = 265 out. The token attributes are
gen_ai.usage.input_tokensandgen_ai.usage.output_tokens, on every model call. - Why the model stopped.
gen_ai.response.finish_reasons = stop: the model finished its answer. Other values, such aslengthortool_calls, say that it ran out of room or asked for a tool. - The prompt, without its text.
gen_ai.prompt.name = agent_structurednames the prompt file.gen_ai.prompt.version = 39329efais the first 8 characters of a SHA-256 hash of that file and the policy file. A hash is a fingerprint: the same text always gives the same value, and one changed word gives another. So you can see that two runs used different prompts without storing a word of the prompt. - Errors. The refund call that timed out has
error.type = timeout, a name from OpenTelemetry’s main conventions, and the span status ERROR. Both are needed: the status says that the step failed, anderror.typesays how.
You can make the prompt’s version yourself. In the project folder:
python -c "from resolver.telemetry import prompt_version; print(prompt_version('agent_structured'))"39329efaIt is the same value as on the span. Change one word in resolver/prompts/agent_structured.md, run the command again, and the value changes. Undo your change afterwards.
What the conventions keep out
Section titled “What the conventions keep out”The conventions also define attributes for the text itself: gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions. They are opt-in: an instrumentation must not record them unless you ask. The conventions warn that they are “likely to contain sensitive information”. Our workflow never records them: OBS_CAPTURE_CONTENT is off. The last lesson of this module removes them in the Collector too, in case a library adds them.
Our own names
Section titled “Our own names”Some things that Larkfield needs have no convention. Examples are the ticket, the customer’s ID, the release, a read or a write, and a refund’s idempotency key. The project gives them its own prefix, larkfield.. A tool that does not know them shows them as plain attributes, and they never clash with a future gen_ai. name. The rule for every name in the project: it carries a count, a status, an ID, or the name of a model or a tool. It never carries text.
The conventions are still in development
Section titled “The conventions are still in development”The GenAI conventions have the status development. That status means that a name can still change or be removed in a later version. We saw this happen while the course was built. The OpenTelemetry site moved the GenAI pages to a new repository, open-telemetry/semantic-conventions-genai. That repository has no release yet, and its README says “Schema URL: TODO”. Meanwhile, the Python package opentelemetry-semantic-conventions 0.66b1 still has the older names, each marked as deprecated and moved.
The move changed real names, not only their place:
Text description of the diagram
On the left, the names of release 1.44.0 of the main conventions. On the right, the names at the pinned commit of the new GenAI repository. The duration of a model call has a new name. The single token histogram became two counters, one for input and one for output tokens. Agent runs and tool calls got their own duration metrics. The project writes all these names once, in resolver/conventions.py.
So the project does three things, and you can do the same in your own work:
- Pin the version. This course uses the GenAI repository at commit
9ffc7e00, status development. The pin is written in one place. - Write the names once. Every name is a constant in
resolver/conventions.py. The rest of the code uses only these constants, never a string like"gen_ai.usage.input_tokens". The code does not even import the Python package’s constants, because they are the older names. - Expect the names to move. When you change the pin, you change one file and run the tests. A dashboard or an alert that uses a name must change at the same time.
Open the pin:
grep -n "PINNED" -A7 resolver/conventions.pySelect-String -Path resolver\conventions.py -Pattern "PINNED" -Context 0,7grep -n "PINNED" -A7 resolver/conventions.pyThe block that starts at line 14 names the repository, the full commit 9ffc7e00c639be7712e544e3e54331e885dccf84, the status development, and the core conventions 1.44.0. The line under it sets SCHEMA_URL to the core version’s schema, because the GenAI repository has none yet.
Your turn: a span for the reply
Section titled “Your turn: a span for the reply”The workflow sends the reply to the customer in resolver/live.py, in the function _send. It is a write: the customer gets a message. But no span records it yet. Add one, with the correct attributes, and see it in obs trace.
Here is the function as it is now:
def _send(state, world) -> str: """Send the reply to the customer (a mock help desk). 'sent' or 'failed'.""" try: helpdesk.send_reply(world, state.run_id, state.proposal.reply) return "sent" except ServiceError: return "failed"-
Decide the span’s name and attributes from the table and the worked example. It is a tool call, its tool is
send_reply, and it is a write. Use only constants fromconventions.py, whichlive.pyalready imports asc. -
Wrap the call in a span. The project’s helper
telemetry.get().span(name, attributes)starts a span on the run clock and makes it the current span inside thewithblock. -
When the send works, set
larkfield.reply.statustosent. When it fails, set it tofailed. Also seterror.typeand the span status ERROR. -
Run a ticket and read its trace:
Terminal window python -m resolver run T-80008python -m obs trace T-80008 -
Run it again with a simulated failure of the help desk, and read the trace again:
Terminal window python -m resolver run T-80008 --fault send_reply:unavailablepython -m obs trace T-80008 -
Check that nothing private left the application, and that the tests still pass:
Terminal window python -m obs check-redaction work/telemetrypython -m pytest -q
Try it yourself before you read on.
One answer, and the course’s output
from opentelemetry.trace import Status, StatusCode # with the other imports at the top
def _send(state, world) -> str: """Send the reply to the customer (a mock help desk). 'sent' or 'failed'.""" attrs = {c.OPERATION: c.OP_TOOL, c.TOOL_NAME: "send_reply", c.TOOL_TYPE: "function", c.TOOL_KIND: "write", c.TICKET_ID: state.task_id} with telemetry.get().span("execute_tool send_reply", attrs) as span: try: helpdesk.send_reply(world, state.run_id, state.proposal.reply) span.set_attribute(c.REPLY_STATUS, "sent") return "sent" except ServiceError as e: span.set_attribute(c.REPLY_STATUS, "failed") span.set_attribute(c.ERROR_TYPE, e.code) span.set_status(Status(StatusCode.ERROR, e.code)) return "failed"The first run, step 4:
Trace 1 2b8bc035a939318923c1f9eaf2da8277 … 0.0 s 0.0 s | | 0.00 s execute_tool send_reply sent 0.0 s | | 0.00 s invoke_agent support-resolver reply_only rule W1The run with the failure, step 5. The workflow prints Reply failed. Ticket: handed to person., and the trace says:
Trace 1 10c8900ca208d9854d8280000471cc25 … 0.0 s 0.0 s | | 0.00 s execute_tool send_reply ERROR failed 0.0 s | | 0.00 s invoke_agent support-resolver reply_only rule W1
Outcome: handed_to_personcheck-redaction checked 4 spans and found no problem, and the tests still pass: 79 passed, 1 deselected.
Look at the order of the two lines. Your span prints above its parent, but it is indented, so it is a child. Both spans start and end at the same moment on the run clock: no model span has moved the clock yet. So obs cannot order them by time. The indentation, which comes from the parent span ID, is the truth. In Module 3, the model spans move the clock, and the order looks natural again.
Why not record the reply’s text, to see what the customer got? Because the text is personal data, and the span is read by many people and tools. The status and the ticket ID answer the operator’s question: did the customer get a reply? The help desk keeps the text, for the people allowed to read it.
Keep your change or undo it, as you like. Module 3 replaces _send with the snapshot’s version, which records the same span through a shared helper.
Common mistake
The span is there, but obs shows no status
Symptom: obs trace shows execute_tool send_reply with nothing after the name, even when the reply was sent.
Cause: a key is not the convention’s exact name. For example, "gen_ai.operation" instead of gen_ai.operation.name. obs decides that a span is a tool call from gen_ai.operation.name = execute_tool, so it does not read the tool’s status. The same span would also be missing from a Grafana query for tool calls.
Diagnose: open work/telemetry/traces.jsonl and find the span’s attributes. Compare each key with resolver/conventions.py.
Recover: use the constants (c.OPERATION, c.TOOL_NAME), never a typed string. A wrong constant name fails at once with an AttributeError; a wrong string fails silently.
When to follow the conventions, and what it costs
Section titled “When to follow the conventions, and what it costs”Follow them for every span that a shared tool will read: agent runs, model calls, tool calls, errors. Then Grafana, Langfuse and other tools show your runs without a mapping of your own. Add your own prefixed names for what the conventions do not cover.
The cost is the change. In development status, names move, as they did while this course was built. Pinning, one file of names and tests make a change a small, planned task instead of a broken dashboard. Do not copy names from a blog post or an old library version without checking the pinned version.
Knowledge check
Section titled “Knowledge check”Check your understanding
Omar asks: 'The conventions are only in development. Why not wait until they are stable, and use our own names until then?' What do you answer?
Show the answer
Our own names would work in our code, but no tool would understand them. Grafana’s queries, Langfuse’s types and other tools read the GenAI names. Langfuse typed the 2,027 spans of the replayed week from the GenAI operation names. Waiting means a mapping for every tool now, and a full rename later. Following the development names, pinned at one commit and written once in conventions.py, means one small change when they move. We saw that change happen during the course: new metric names and two token counters.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
A colleague adds a span for the refund tool with these attributes: tool = 'refund', tokens = 0, ticket_text = the customer's message. Name three problems, and write the attributes you would use.
Show the answer
First, tool is not the convention’s key: use gen_ai.operation.name = execute_tool and gen_ai.tool.name = request_refund, with the span name execute_tool request_refund. Second, tokens belong on model calls, not on tool calls: a tool span has no token attributes. Third, the ticket’s text is personal data and must never be on a span. Add gen_ai.tool.type = function, Larkfield’s larkfield.tool.kind = write, the order ID and the amount. On a failure, add error.type and the status ERROR.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Semantic conventions are agreed names, so that every tool reads every application’s telemetry. The GenAI conventions name agent runs (
invoke_agent,invoke_workflow), model calls (chat) and tool calls (execute_tool). - A model call records the model asked for and the model that answered, and the tokens in and out. It also records the finish reason and the prompt’s name and version hash, never its text.
- Errors:
error.typeand the span status ERROR, both. - Larkfield’s own names carry the prefix
larkfield., and only counts, statuses, IDs and names. - The GenAI conventions are in development: the course pins commit
9ffc7e00, writes every name once inresolver/conventions.py, and expects them to move. - Your workflow can now record its reply as
execute_tool send_reply, sent or failed.
You can now name every step of a run so that tools understand it. But our spans still go to files on your computer. Where should they go in production, and who removes the data that must not leave? Next, you put the OpenTelemetry Collector between the application and the tools.
Next: The Collector