Observability for AI Agents: Traces, Costs and Audit Trails
See what your AI agents do, what they cost and who approved each change. Instrument a live support workflow with OpenTelemetry, find agent failures in traces, split cost by model and step, and keep an audit trail that answers "why did the agent do this?".
Read every lesson as a guest, or sign in to enroll and track your progress.
Enroll to save the lessons that you complete. It is free.
You are enrolled. From now on, the lessons that you complete are saved.
To get your certificate, mark every knowledge check as matching. These lessons have checks to finish:
Is this the right course for you?What this course covers and does not cover, and 5 courses to take for other needs.
This course teaches how to observe AI agents in production: OpenTelemetry for multi-step runs, approvals and handoffs, the failure patterns that only agents have, cost per run and per resolved ticket, and an audit trail that is complete and tamper-evident. It does not teach how to build the agent, general service telemetry, evaluation datasets or the platform controls themselves.
Explain what to record for an agent (traces, metrics, events, audit records) and what must not be recorded, and why one tool is not enough.
Instrument a multi-step, tool-using agent with OpenTelemetry and its GenAI conventions, including approvals that pause a run and work split across agents.
Find agent-specific failures in traces (loops, hidden tool errors, duplicate writes, success without action, followed injected instructions) and prove the cause with evidence.
Attribute cost to runs, steps, customers and models, and set budgets and alerts that lead to a decision.
Keep an audit trail that is complete, tamper-evident and access-controlled, and answer an audit question from it.
Run the same telemetry into Grafana and Langfuse, choose tools for a team with stated criteria, and judge what a purpose-built view adds.
About this course
Larkfield’s support workflow is live. Every day it reads tickets, looks up orders and policy, proposes refunds, returns and reshipments, waits for an agent’s approval and then writes. Last week one customer was refunded twice. The dashboard stayed green, and nobody could say which step, which model decision or which person caused it.
In this course, you follow Amira, the developer who owns the workflow, as she makes it observable. You record each run as one trace with OpenTelemetry, from the ticket through the model calls, the tool calls and the approval to the write. You send the same data to Grafana and to Langfuse, and you keep a separate audit trail that nobody can change without detection.
Then you use what you recorded. You find loops, hidden tool errors, duplicate writes and injected instructions in the traces of a replayed week. You find out why the cost per resolved ticket rose after a model switch. And you answer Grace’s question: why did we refund that ticket twice?
Everything runs on your computer with no account and no key. Every model decision is a real recording from the agents course, replayed. The practice incidents are cases we caused on purpose, and each lesson says which evidence is replayed and which is simulated. Prompt text, personal data and secrets stay out of the telemetry by default, and the Collector removes them again before anything leaves your computer.
Before you start
You completed AI Agents and Workflow Orchestration, or you have the same skills. You can read and change Python, use a terminal and read JSON. The MLOps, agents governance and MCP courses help but are not required; lessons that use their ideas explain them briefly and link to them. You need no account, no key and no money. Grafana runs in Docker if you want it, and every lesson also works from saved telemetry. Langfuse is optional, in one module, with a free account.
AI engineers, backend developers and platform or operations engineers who run AI agents in production, and the technical leads who must say what an agent did, why, and what it cost.
Lead or use AI, but do not build it?
You can study the ideas of this course without writing code. Read modules 1, 5 and 6 (about 6–7 hours), answer their knowledge checks, and get a free concepts certificate. You can come back for the practice later.
The paths for people who lead or use AI take this course this way. See the paths
How we made this course
Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses
Inside the course
Module 1ConceptsWhy agents need their own observability
3 lessons
Why agents need their own observability. Meet a refund paid twice that nobody can explain, decide what to record and what to keep out, and run the instrumented workflow on your computer.
OpenTelemetry for agents. Learn spans, traces and context, the GenAI conventions for agent runs, model calls and tool calls, and the Collector that redacts, samples and routes outside the application.
Instrument the workflow. Record model calls, tool calls with their idempotency keys, approvals that pause a run overnight, handoffs between workers and across MCP, and join the app's traces with the model gateway's records.
Find agent failures in Grafana. Read the waterfall of a run, search runs by structure with TraceQL, find the five agent failure patterns, and move from a trace to the metrics and events around it.
Cost. Compute cost per run and per step from a dated price table, split it by model, slice and customer without exploding the metrics, and set budget alerts that lead to the runs behind a spike.
Audit trails. Record proposals, approvals, refusals, writes and data access with a fixed schema, keep them in an append-only store with a hash chain, and answer the audit question about the double refund.
Alerts, service levels and quality. Choose service levels for an agent, write alerts that fire on their scenario and stay quiet on a healthy week, and add sampled human review with scores on traces.
Module 8Tools: Langfuse, others and the run inspector
3 lessons
Tools. Send the same redacted traces to Langfuse, compare tools for a team on stated criteria, and tour the run inspector to see what a view built for agent runs adds.
Three end-to-end case studies on open data. Find failures in public agent runs, measure the cost of a coding agent, and design the records that would have answered a public case. Complete at least one for the certificate; each one is a portfolio piece.
Complete at least one case study for the certificate. The others are optional. What each one covers
Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.
Convert an open set of recorded customer-service agent runs to OpenTelemetry, load them into Grafana, and find and count failure patterns with evidence.
Measure cost per task, per step and per outcome on open coding-agent trajectories, find the runs that loop or over-spend, and propose a budget and a step limit.
Take a public, documented case of an AI assistant's answer that a company had to defend, list the questions it raised, and design the audit trail and traces that would have answered them.
Reviews
Loading the reviews…
Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.