Keep an AI service working for its users after it goes live. You make Larkfield's ticket service operable one module at a time, with owners, objectives, versions, telemetry, monitoring, safe releases and runbooks, and you catch a bad release before customers tell you.
Read every lesson as a guest, or sign in to enroll and track your progress.
Enroll to save the lessons that you complete. It is free.
By the end of this course, you will be able to:
Larkfield’s ticket service is live. Customers send tickets, an AI assistant chooses the team, decides if a person must handle the ticket, and drafts a reply. The dashboard says the service is up 99.9% of the time. But last week, customers waited two days for an answer to safety tickets, and nobody noticed. Grace asks: “How would we know that the assistant is failing our customers, before they tell us?”
In this course, you answer Grace’s question. First you list what the service depends on and who owns each part, and you turn “working” into objectives that you can measure. Then you give every answer a release ID, so that you can find the code, data, prompt and model versions that made it.
Next you add telemetry: traces, metrics and logs that follow one ticket through the whole service, with its tokens, its cost and its quality, and without private text. You watch for changes in the inputs, in the labels that arrive later, and in the documents that the assistant cites.
Finally you release a new prompt with a gate and a canary, catch its failure, roll back, and prove that the service recovered. You handle a provider outage with a runbook and write the review afterwards. Every number in the course is labelled: measured on the course team’s computer, replayed from real recorded answers, simulated, or a stated assumption.
You completed Evaluating and Testing AI Systems; Deploying AI Models for Real Users; CI/CD, Testing and Safe Releases; Databases and Storage for AI Applications; and Scaling APIs and AI Workloads, or you have the same skills. You do not need an account or money. The AI provider is simulated on your computer, and its answers are real recorded answers. To run the service, you need Python and Docker Desktop (for PostgreSQL, Valkey and Azurite). Grafana runs in one more container and is optional. Without Docker, every analysis in the course still works from saved telemetry.
Courses to take first: Evaluating and Testing AI Systems, Deploying AI Models for Real Users, CI/CD, Testing and Safe Releases, Databases and Storage for AI Applications, Scaling APIs and AI Workloads.
People who have evaluated an AI assistant, deployed a model behind an API, and built a release pipeline, a database and a queue, as in Evaluating and Testing AI Systems; Deploying AI Models for Real Users; CI/CD, Testing and Safe Releases; Databases and Storage for AI Applications; and Scaling APIs and AI Workloads. Now the service is live and they must know when it fails its users. Developers, AI engineers and data scientists who run AI in production.
You can study the ideas of this course without writing code. Read modules 1, 4 and 6 (about 5–7 hours), answer their knowledge checks, and get a free concepts certificate. You can come back for the practice later.
The paths for people who lead or use AI take this course this way. See the paths
Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses
Know what you operate and what "working" means. Run the live service, list every part it depends on, turn user journeys into objectives you can measure, and give every part and every alert an owner.
Make every answer traceable. Write a release manifest that ties code, data, prompts, model and index versions together, compare two releases setting by setting, and keep records of experiments and decisions.
See inside the service. Add traces, metrics and logs with OpenTelemetry, follow one ticket from the API to the model and back, track quality and cost per version, and keep private text out of the telemetry.
Notice when behaviour changes. Test input drift with a stated method, plan for labels that arrive late, and watch the failures that are special to language models and retrieval.
Release changes safely. Combine evaluation results with cost and latency limits in a release gate, roll out with a canary and roll back with proof, and choose the fix that matches the cause.
Handle bad days. Write runbooks, keep the service useful during a provider outage, verify the recovery, and learn from incidents without blame.
Two end-to-end case studies on public material. Monitor a model on a public dataset that changes over time, and review a published provider outage in the course's format. Complete at least one for the certificate; each one is a portfolio piece.
Complete at least one case study for the certificate. The others are optional. What each one covers
Use every module in one practice project. You collect evidence, and check it against the course outcomes.
Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.
Monitor a model on public data that changes2 hr
Train a simple model on early months of a public, time-stamped dataset, then build its drift and delayed-label monitor month by month, and say what the monitor can and cannot prove.
Post-incident review of a public outage2 hr 30 min
Rebuild the timeline, detection, mitigation and preventive actions of a published provider incident report in the course's runbook and review format.
Loading the reviews…
Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.
