From baseline to deployment

What you will be able to do

Know the steps that every serious AI project goes through, in order, and the decision at the end of each step. Interviewers often ask about these steps. You also need them to plan a portfolio project.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

• Larkfield’s frame for the ticket project and its success measures, from Problem framing: Worked example: framing the ticket project.

What you need

• Your notes.md file in the ai-career folder. You made it in AI in everyday products: Set up your course notes. If it is missing, make it again as in that section. This lesson adds the heading ## Next step.

The lifecycle, with its decisions

The AI project lifecycle. Each diamond is a decision that can send the project back, or stop it.

Text description of the diagram

1. Frame: the user, the decision, the data and the cost of errors.
2. Decision: is AI needed? If no, solve it without AI: a form, a rule or a process. If yes, or not sure, continue.
3. Baseline: the simplest method, measured.
4. Experiment: try models, and compare them with the baseline.
5. Decision: better than the baseline, for the errors that matter? If no, go back to Frame. If yes, continue.
6. Integrate: into the real work, with a person checking.
7. Monitor: measure every week.
8. Decision: still good enough? If yes, keep monitoring. If no, go back to Experiment.

Step 0: the solution without AI

In the last lesson, the team decided to try a topic menu on the contact form first. After four weeks, Priya measures it: about 60 of 100 tickets now arrive with the correct topic. About 25 have the topic “Other”, and about 15 have a wrong topic.

The menu helps, so Larkfield keeps it. But 40 of 100 tickets still need sorting, and urgent tickets are still not marked. So the team continues with AI for those tickets.

Step 1: the baseline

A baseline is the simplest reasonable method, measured in the same way as everything that comes later. It answers the question: “Better than what?” Without a baseline, a result such as “85% correct” means nothing.

Tomás writes keyword rules in one afternoon. For example, “refund” or “return” goes to returns, “parcel” or “tracking” goes to delivery, and “charged twice” or “fraud” means urgent. If you take the API course, you build a classifier like this.

To measure it, Priya needs a test set: tickets with correct answers that no method has seen before. She takes 1,000 recent tickets with their final team. Grace checks 200 of them by hand, and corrects 14 wrong labels.

Step 2: experiments

Now the team tries more complex methods and compares each one with the baseline, on the same test set:

Method	Correct team (of 100)	Urgent marked urgent (of 100)	Cost per month
Baseline: keyword rules	74	81	Almost nothing
ML model trained on 90,000 past tickets	91	89	Low
A language model with written instructions	90	93	Medium
ML model for the team, plus the keyword rules for urgent	91	97	Low

The last row is the winner, and it is not the most advanced method. The ML model sorts teams well. The simple keyword rules catch urgent tickets better than any model, because Grace gave Tomás the exact words that customers use when it is urgent.

This is common in real projects. Advanced methods do not win automatically. They must beat the baseline on the errors that matter.

Step 3: evaluate the details

The total is not enough. Priya breaks the result down by team:

Team	Correct (of 100)	Tickets in the test set
Returns	95	310
Delivery	94	290
Payment	92	180
Account	90	150
Warranty	72	70

Warranty is much worse. Grace reads the 20 wrong warranty tickets, and she sees the reason at once: customers write “broken” both for “arrived broken” (delivery) and “stopped working” (warranty). It is the same confusion as in the document assistant. The team adds 300 more warranty examples, labelled by Grace, and warranty rises to 86.

Decision: the method beats the baseline, and urgent tickets meet the target of 95. Omar decides: go to integration, but warranty tickets always get a person’s check.

Step 4: integrate, with a person checking

Integration means putting the AI into the real work. Larkfield does it in two stages:

1. Suggest only (4 weeks). The co-ordinator sees the suggested team and priority, and accepts or changes each one. The system records every change. The co-ordinator accepts 89 of 100 suggestions. Sorting time falls from 2 hours a day to 35 minutes.
2. Automatic when sure. The model gives a confidence: how sure it is. Tickets with high confidence go directly to the team. The co-ordinator checks only the others, about 30 of 100. Sorting time falls to 20 minutes.

Starting with suggestions is safer. Mistakes are caught by a person, and the team learns where the model is weak before it acts alone.

Step 5: monitor

A model learns from the past. When the world changes, the model can become worse without any error message. Monitoring means measuring the system regularly in real use, so that you see this early.

Every week, the co-ordinator checks a random sample of 50 automatically sorted tickets. Here are four weeks in spring:

Week	Correct team (of 50)
14	46
15	45
16	41
17	39

The numbers fall. Priya reads the wrong tickets: most of them are about parasols and garden chairs. In week 15, Larkfield started to sell garden furniture. The model never saw these words. The team adds labelled garden tickets, trains the model again, checks it on the test set, and the weekly sample is back at 46 of 50.

Who did what

Look back at One project, many roles. The same roles appear again:

Step	Main roles
Frame	Product manager, support lead (SME)
Baseline	AI engineer, with the SME’s keywords
Experiments	Data scientist, ML engineer
Evaluation	Data scientist, SME
Integration	Software engineer, AI engineer, product manager
Monitoring	Co-ordinator, data scientist, ML engineer

Guided practice: what happens next?

For each situation, write in notes.md under ## Next step which step the team is in, and what it should do next.

1. A new language model is released. Tomás says: “It is much better, let us use it.”
2. In suggest-only mode, the co-ordinator changes 40 of 100 suggestions.
3. The weekly sample is stable at 46 of 50 for three months.
4. Larkfield plans to sell in a new country, where customers write in a second language.

Show the answers

1. Experiment. Test the new model on the same test set, and compare it with the current method on the errors that matter, especially urgent tickets. Use it only if it wins there.
2. Integration shows a problem. 40 changes is far worse than the test set suggested. Find out why: is the test set different from today’s tickets? Do not move to automatic mode. Go back to experiments.
3. Monitoring is working. Keep checking. Nothing needs to change.
4. Back to framing. The data has no tickets in the new language. The team must decide: collect and label examples, use a method that handles both languages, or have people sort those tickets at first.

Independent variation: your weekly sample falls from 46 to 38 of 50 in one week, not slowly. Write two possible causes that are different from the garden furniture case, and how you would check each one.

Common mistake

No baseline

Symptom: the team reports “our language model sorts 90% of tickets correctly”, and everybody is happy. Months later, someone tries simple keyword rules and gets 88%, at almost no cost.

Cause: there was no baseline, so 90% looked good on its own. The team paid for complexity that added very little.

Fix: measure the simplest reasonable method first, on the same test set. Report every result next to the baseline: “90%, compared with 88% for keyword rules”. Then the decision is honest.

Check your understanding

Why must the test set contain tickets that no method has seen before?

Show the answer

Because the real system will get new tickets. If a model is tested on tickets it learned from, it can score well by remembering them, not by understanding the pattern. The score would be too high, and the team would only find out after release. A test set of unseen tickets shows how the system works on new input.

Check your understanding

Name the reason that this project might not have needed AI, and why the team still continued.

Show the answer

The topic menu on the contact form sorted about 60 of 100 tickets correctly without AI. If it had sorted almost all of them, the project could have stopped there. The team continued because 40 of 100 tickets still needed sorting, and urgent tickets were still not marked.

Recap

• The lifecycle: frame, decide if AI is needed, baseline, experiment, evaluate, integrate, monitor. Each step ends with a decision.
• A baseline is the simplest method, measured. Every result is compared with it.
• Test on unseen data, and look at the errors that matter, not only the total.
• Integrate in stages, with a person checking first.
• Models get worse when the world changes. Monitoring finds it.

Next, you look at the work around the model: stakeholders, requirements, documentation, privacy and how to talk about uncertainty.

Next: Team and business context