Run an AI team day to day

What you will be able to do

Lead AI work week by week when nobody knows the result in advance. Set goals that the team can meet honestly, and keep a short rhythm of meetings. Share the load of the live service, and see early when the team is stuck.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• Experiments, time boxes and the three decisions (continue, change, stop), from Why AI plans are different.
• The experiment review, the error-analysis session and the decision record, from Team rituals and AI-assisted work.

What you need

• A document for your notes. This lesson has no template: in “Your turn” you plan one week for Larkfield’s team or your own.

The meeting times, the agenda and the situations in this lesson are made up for the course. They are the course’s recommendations, with the reason for each, not measured results.

A garden in spring

A gardener tries three new kinds of tomato. She does not know which will grow in her soil. So she does not promise “twenty tomatoes on the first of July”. She promises something that she controls: “I try three kinds in small pots for four weeks. Then I keep the best one and pull the others.”

Every Monday she walks the rows and writes what grew. When one kind dies, she writes why, without blame: too much shade. And she says no to the neighbour who wants a tour every afternoon.

The problem: a manager’s week that does not fit

Omar used to lead a web team. Its goals were delivery goals: “the new returns page goes live on Friday.” That works when the team knows what “done” looks like.

Picture Omar with the AI team (a constructed situation). He sets Priya a goal in the same style: “Raise the judge score to 4.5 by the end of the sprint.” Priya tunes the prompt for three weeks. The score goes up a little, then down a little. Nobody decides to stop, because the goal has no stop point. Meanwhile other teams ask for three demos, and Tomás answers every alert at night because nobody else knows the service.

The goal promised a result that nobody controls. The week had no place to decide “stop”. And the load fell on whoever could not say no.

A score target has one more danger. In the evaluation course, a judge without the policy preferred prompt v3 in 126 of 200 pairs, against 48 for v2. Yet v3 was the worse prompt (recorded, Evaluating and Testing AI Systems). A team that chases a score can win the number and lose the customer.

The idea: goals for learning, dates for decisions

For uncertain work, set a learning goal. It names the question that the team will answer and the evidence that answers it. It has a time box with a stop point. The team controls all three. It does not control the quality that the answer turns out to have.

A delivery goal for quality (avoid)	A learning goal (use)
“Judge score 4.5 by the end of the sprint.”	“In two weeks, find out if a policy reminder in the prompt stops invented steps on the 20 latest error examples. Stop if fewer than half improve.” (constructed)
“The assistant sends replies alone from 1 March.”	“On 1 March, decide with the gate’s evidence if the assistant moves one level up the autonomy ladder.”

The second row shows that dates do not disappear. A date belongs to a decision, not to a level of quality. And for work whose result is known, such as a review screen with an agreed design, a normal delivery date is fine.

A weekly rhythm that fits

Put the rituals of the last lesson in a week, with planning and focus time around them.

A week for an AI team (a suggestion for a team of Larkfield's size). The order matters more than the times: plan, focus, review, learn from failures.

Text description of the diagram

The week starts on Monday with 20 minutes of planning. For each experiment, the team writes its question, its time box and its stop point. From Tuesday to Thursday, the mornings are for focus, with no meetings and no demos. One-to-ones happen in the afternoons. On Thursday, the experiment review decides for each experiment: continue, change or stop. Every second Friday, the error-analysis session reads failures for 60 minutes, and the one demo slot follows it. The next Monday plans the work from what the team learned.

If an experiment has no stop point on Monday, it does not start. The Thursday review adds one rule to the experiment review: every experiment whose time box ended gets a decision that day.

One-to-ones that help

A one-to-one is a short, private meeting between the leader and one person. It is not a status update: it is for what a group meeting hides. The person owns the agenda; the leader listens and acts.

A one-to-one agenda of 30 minutes, every two weeks (a suggestion). It ends with one action that the leader owns.

Text description of the diagram

The agenda has four parts and an action. First, blockers, for five minutes: what is waiting, and for whom? The leader removes the blocker. Second, what the person learned, for ten minutes, from the last experiment, including what did not work. Third, load, for five minutes: on-call, demos, requests and hours. Fourth, growth, for ten minutes: one skill and the next step. Last, the leader writes down one action, with a date.

Here is a tiny example: Nadia’s one-to-one with Priya (constructed). Blockers: Grace’s agents have no time to label the new tickets. Learned: the policy reminder stopped no invented steps, but a reply that copies the policy text looks better. Load: two demo requests this week, and three late evenings. Growth: Priya wants to run the error session alone. Nadia’s action: ask Grace for four hours a week of agent time by Friday, and send both demo requests to the demo slot.

A failed experiment, handled without blame

Many experiments fail; they exist to find closed doors early. A negative result that is well run is progress: it answers a question with evidence, so nobody spends months on it again.

Larkfield has a good one. In the fine-tuning course, the team trained its own small model for routing. It sent 0 of 23 development tickets that needed a person to a person (recorded, Fine-Tuning and Adapting AI Models). The course decided: do not ship it. No customer saw it, and the cheap route was closed with evidence.

When an experiment fails, the leader asks three questions: “What did we learn? What would make it worth trying again? What do we stop doing now?” Never ask “who failed?” A team that is blamed for negative results stops reporting them, and then it repeats them. Record each failed experiment in one short decision record, as in the decision record for prompt v3:

What to record	The routing model
The question	Can our own small model route tickets as well as the hosted model, for less money?
What was tried, on which evidence	A fine-tuned small model, on the development and holdout tickets of the evaluation set
The result	Often the right team; no ticket sent to a person, on either set
The decision	Stop. Keep the hosted model
What would reopen it	A new method that sends every ticket that needs a person to a person, on a fresh holdout set

The course MLOps and LLMOps: Operating AI in Production uses the same rule after an incident. In Post-incident learning, it looks for the decision that failed, not the person.

Protect the team’s focus

AI work attracts two kinds of interruption. The first is the demo request: “Can you show the assistant to our director this afternoon?” A demo of a few hand-picked tickets also misleads, as Run a fair pilot showed. So requests go to the leader, not the engineer, and demos happen only in the fixed slot, with recorded examples and failures.

The second is “just one more model”. A new model appears, and someone wants to try it today. Treat it as a new experiment: it needs a question, a time box and a stop point, and it waits for Monday’s planning. Ask one question first: “If it wins, what would we decide differently?” If the answer is “nothing”, it can wait.

Operations and on-call once the feature is live

After launch, someone must act when the service fails. Being on call means being the first person called during a set period, usually a week. In the MLOps course, Larkfield wrote down an owner and a backup for each part of the service. Amira is called first for three of its seven objectives (Ownership and support). That is sensible for one alert, but heavy week after week.

Share the load with three rules. First, a rotation: at least three people take turns, one week each, with a named backup. Second, an on-call week counts as work. Assume about four hours of alerts and checks in a quiet week (an assumption; measure your own). Plan less experiment work for that person that week, and give time back after a night call. Third, the rotation needs written steps for each alert, so a new person can act. The MLOps course calls them runbooks. An alert that fires again and again goes into Monday’s planning as work to fix.

Signs that a team is stuck

A stuck team still looks busy. Watch for these signs:

Sign	What you see	What the leader does
No stop decisions	Every time box ends with “two more weeks”	Hold the decision on the planned day: continue, change or stop. No experiment starts without a stop point
Endless tuning	The same prompt changes for weeks; the score moves up and down	Ask for the range, as in Review work you cannot write yourself. If the change is inside the noise, stop or try a different idea
Nobody reads the errors	Meetings show averages and colours only	Restart the error-analysis session: 20 failures, chosen by a rule, with the domain expert
One person carries the service	The same name on every alert	Start the rotation, with runbooks and a backup

Each sign is a missing decision, so the leader acts. A team rarely stops its own work: stopping feels like failing.

Discuss: a date for the director?

Nadia’s director asks: “When can the assistant send replies without a person checking them?”

• Omar: “Give a date. Say three months. A deadline makes the team work harder.”
• Priya: “We cannot promise a quality. Give no date at all.”
• Kwame: “Give the date of the next decision, and say what evidence it needs.”
• Tomás: “And who carries the alerts meanwhile? I was on call four weeks in a row.” (constructed)

Before you read on, answer these:

1. What can Nadia honestly promise with a date?
2. What could go wrong with Omar’s deadline?
3. What does Tomás’s point change in the plan?

The course’s answer. Promise a date for a decision, not for a quality. Kwame is right: “In three months we decide, with the gate’s evidence, if the assistant moves one level up. If it fails, we say what we try next, or we stop.” Omar’s deadline invites tuning to the score. The recorded judge preferred the worse prompt 126 times to 48. Priya is half right: no quality can be promised, but a decision date is honest. Tomás’s point is a load problem, and without a fix the plan is fiction. Start a rotation of at least three people with a backup, and plan less experiment work for each on-call week.

What would change the answer: if the evidence already passed the gate on a fresh holdout set, a launch date would be reasonable. And for work with a known result, such as a new review screen, a delivery date is the right promise.

When to use this, and what it costs

Use it from the first experiment, and add the on-call rules on the day the service goes live. Do not put learning goals on work whose result is known: it needs a normal delivery date. Do not turn one-to-ones into status meetings. What it costs: for the leader, 30 minutes per person every two weeks, so about two hours a week for a team of eight. For each person, about one hour a week of planning and review.

Common mistake

Blame after a failed experiment

Symptom: after an experiment failed, the next three reports show only good results, and nobody mentions what did not work.

Cause: the last failure was discussed as “whose fault?”. People learned to hide negative results.

Fix: in the next review, thank the person for the negative result and ask what the team learned. Record it in a decision record, with the condition that would reopen it.

Common mistake

One person carries every alert

Symptom: Tomás’s experiments slip every week, and he answers alerts at night.

Cause: there is no rotation, and on-call time is not counted as work. Only one person knows the service, so every alert goes to him.

Fix: a rotation of at least three people, one week each, with a backup and runbooks. Each on-call week carries less planned work.

See it end to endRecover a stalled AI projectCase study

The third case study shows a stuck team from the inside. It has six months of “nearly ready”, a test set and a measure that changed, and no stop decision. You diagnose it and write a recovery plan.

Your turn

Plan one week for Larkfield’s team, or for your own. Write four things:

1. One experiment as a learning goal: its question, time box and stop point.
2. The meeting rhythm for the week, with days and lengths.
3. Who is on call, and who is the backup.
4. The one-to-one agenda for one person, with the action you expect to own.

Then change one condition: the director asks for a live demo for the board on Wednesday morning. What do you change, and what do you keep?

Show a worked solution (Larkfield’s team)

1. Learning goal. “In two weeks, find out if a reply that copies the policy text stops invented steps on the 20 latest error examples. Stop if fewer than 10 improve, or if any safety ticket loses its answer.” (The numbers are made up for this example.)

2. Rhythm. Monday planning, 20 minutes. Focus mornings from Tuesday to Thursday. Experiment review on Thursday, 30 minutes. The error-analysis session is next Friday, 60 minutes, followed by the demo slot.

3. On call. Mei this week, Amira as backup. Tomás is off the rotation for two weeks after four weeks in a row. Mei’s experiment work is planned at three days, not five.

4. One-to-one with Tomás. Blockers: none. Learned: the routing result, and what would reopen it. Load: the four on-call weeks, and the night calls. Growth: he wants to write the runbook for the “no answer” alert. Nadia’s action: the rotation of three names, published by Wednesday.

The board demo. Keep the focus mornings and the decision dates. Nadia, not an engineer, takes the request. She offers a recorded demo from the evidence pack: ten tickets chosen by a written rule, failures included, with the gate’s result. If the board must see it on Wednesday, she moves the demo slot, not the review.

Knowledge check

Check your understanding

Omar writes the goal 'Raise the judge score to 4.5 by the end of the month.' Rewrite it as a learning goal, and give two reasons why the first version is a problem.

Show the answer

A learning goal: “In two weeks, find out if change X improves the replies on the agreed evaluation set, without losing any safety ticket. Stop if it does not.” It names a question, the evidence, a time box and a stop point. The first version promises a quality that nobody controls, so the team either tunes without end or misses the goal. And a score target invites tuning to the score. The recorded judge without the policy preferred the worse prompt in 126 of 200 pairs.

Check your understanding

An experiment shows that a cheaper model never sends a ticket to a person. What does the leader say to the team, and what does the team record?

Show the answer

The leader thanks the team and asks what it learned, what would make the idea worth trying again, and what it stops now. The leader does not ask who failed: blame makes people hide negative results. The team writes a short decision record. It names the question, what was tried on which tickets, the result, the decision to stop, and the condition that would reopen it. An example is a new method that sends every ticket that needs a person to one, on a fresh holdout set.

Check your understanding

Name two signs that an AI team is stuck, and for each one say what the leader does.

Show the answer

Any two of these. No stop decisions: every time box ends with “two more weeks”. The leader holds the decision on the planned day and starts no experiment without a stop point. Endless tuning: the score moves up and down for weeks. The leader asks for the range and stops changes that are inside the noise. Nobody reads the errors: the leader restarts the error-analysis session with failures chosen by a rule. One person carries the service: the leader starts a rotation with runbooks and a backup.

Recap

• For uncertain work, set learning goals: a question, the evidence, a time box and a stop point. Put dates on decisions, not on quality.
• A weekly rhythm: short planning on Monday, focus mornings and the experiment review on Thursday. Every second Friday brings the error session and the one demo slot.
• A one-to-one covers blockers, what the person learned, load and growth, and ends with one action for the leader.
• A negative result that is well run is progress. Ask what the team learned, never who failed, and record it.
• Once the feature is live, share on-call in a rotation of at least three people, count it as work, and write runbooks.
• A stuck team makes no stop decisions, tunes without end, reads no errors, or leaves one person with the service. Each sign is a decision for the leader.

Next, you help your specialists grow, and you keep them.

Next: Grow and keep AI specialists