Team rituals and AI-assisted work

What you will be able to do

Turn good reviews into team habits. Run a short experiment review every week, and read failures together. Write down decisions that are hard to undo, and agree how the team uses coding assistants.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• The ten-line experiment-review checklist, from Review work you cannot write yourself.
• The role map: who decides, who does, who is consulted, from Team roles by stage.

What you need

• The templates templates/decision-record.md and templates/ai-assisted-development-policy.md.
• For the module practice at the end: evidence/experiment-report.md and the checklist.

The match on video

After a match, a football team watches the video on Monday. The coach does not only read the score. The team watches every goal that it let in, and sorts them: a lost ball in midfield, a corner, a slow defender. One kind keeps coming back, so the team practises it that week. And when the coach changes the formation, she writes the reason in the team’s book. A player who joins in March can read why.

The score alone would not tell the team what to practise. And without the book, the team argues about the same formation every month.

The problem: a team that forgets

Picture Larkfield three months after the gate stopped prompt v3 (a constructed situation). Someone proposes it again: “It’s the shorter prompt, so it’s cheaper.” Priya is on holiday, and nobody in the room remembers the holdout result. The judge’s 126-to-48 preference is still in an old slide. The team spends two meetings on the same argument.

Two habits were missing. Nobody had written the decision down with its reasons. And nobody read failures as a team, so the knowledge that “warm replies invent steps” lived in one person’s head.

The idea: three team rituals

A ritual here is a short meeting or document that the team repeats on a fixed rhythm. Then reviews happen without anyone remembering to ask for them. Three rituals cover most of an AI team’s needs.

Three rituals in one rhythm. The times are a suggestion for a team of Larkfield's size; the order is what matters.

Text description of the diagram

The experiment review happens every week, for 30 minutes. The author shows the evidence, and a reviewer uses the checklist. The error-analysis session happens every two weeks, for 60 minutes: the domain expert and the team read 20 failures and sort them into kinds. The team writes a decision record when a choice is hard to undo. It is one page: the context, the options, the decision and the trigger to review it. When a trigger fires or a new result arrives, the cycle starts again with an experiment review. A new kind of failure from the error-analysis session becomes a new test case.

The experiment review is the checklist of the last lesson, on a fixed day. The author sends the evidence one day before. A reviewer who did not build the change reads it with the checklist. The meeting ends with one line: accept, ask for more, or reject.

The error-analysis session is the team’s video. The domain expert and the builders read real failures together, sort them into kinds, and count each kind. Choose the failures by a written rule, for example “the first 20 drafts that agents rewrote this week”. Then nobody picks the worst or the best on purpose.

The decision record is the coach’s book: one page per decision that is hard to undo or that someone will question later. The template has five parts: context, options, decision, consequences and review.

A tiny example: one error-analysis session

Grace, Priya and Tomás read the five examples of the evidence pack. The pack chose them by written rules. Here is how they sort them:

Ticket	What went wrong	Kind	What it costs
T-64740	Prompt v3 gave no answer on a safety ticket	No answer	High: nobody is told
T-65350	Prompt v3 invented a “saved-card section”	Invented step	A customer looks for a page that does not exist
T-65479	The judge without the policy preferred an invented step	Judge misled	Hidden: the scores look better than the replies
T-65022	Prompt v2 sent a parasol warranty ticket to a person	False alarm (debatable)	An agent’s time
T-65450	Prompt v2 gave no answer on a bulky return	No answer	A customer waits

Count the kinds: no answer 2, invented step 1, judge misled 1, false alarm 1. Five tickets are too few for shares, but the session already gives two actions. First, every ticket with no answer must go to a person. The course MLOps and LLMOps: Operating AI in Production later changed the service to do exactly that. Second, the judge must always see the policy. Each new kind also becomes a test case in the evaluation set.

A decision record for prompt v3

Nadia asks Priya to write the record that the team was missing. Here is a short version, filled from the template:

DR-001: Do not release prompt v3. Status: accepted. Decision owner: Nadia. Context: on the 98 holdout tickets, v3 got both decisions right on 83 against 94 for v2; the range is entirely below zero. v3 gave no answer on two safety tickets (T-64740, T-65663). Model cost US$0.48 against US$0.34 per 1,000 tickets (recorded, evaluation course). Options: release v3; keep v2; try v3 on a small share of tickets first. Decision: keep prompt v2. Consequences: replies stay less warm; the warmth idea needs a new version and a new test. Review: when a new version answers every ticket and sends every ticket that needs a person to one. Test it on a fresh holdout set, in at least five runs.

The next time someone says “it’s cheaper”, the team reads one page instead of holding two meetings.

AI-assisted work: who owns what the assistant writes

Most of Larkfield’s developers now use coding assistants: tools that write or change code from a request in plain words. A leader does not need to use one. But the team needs a written policy for them, and the leader makes sure that it exists.

Think of a new junior colleague who drafts a letter for you. The draft may be good. But you read it before you sign, because your name is on it. The colleague’s speed does not change who is responsible.

The course AI-Assisted Software Development recorded what happens without that care. An assistant got the same feature twice. The first request was vague: “Add a way to filter the report by category.” The second was a bounded task with written acceptance criteria: the checks that the change must pass to count as done.

The same feature, asked for in two ways. The vague request passed 4 of 8 acceptance checks, although its own tests passed. The bounded task passed 8 of 8 (recorded).

Text description of the chart

Two columns on an axis from 0 to 8. The vague request: 4 of 8 acceptance checks passed. Its change touched 4 files and added 77 lines, and its own tests passed (29 passed). The bounded task with acceptance criteria: 8 of 8 passed, with 5 files and 145 lines added.

The vague request gave the smaller change, 77 lines, and its own tests passed. It still failed half of the checks. The size of a change and “the tests pass” say little; the acceptance criteria say whether it is done. The same course also asked an assistant to write tests for an existing function that had a bug. It wrote 11 tests. 0 of them caught the bug, and 2 of them checked that the bug was there. Those tests failed after the fix. A test that copies the code can only agree with the code. The course explains both results in Problem and acceptance criteria and Choose meaningful checks.

So the policy says what the assistant may do, what a person must check, and who owns the result:

The team's policy for AI-assisted development, from the template. The assistant can speed up the work; a person checks it, and the author owns it.

Text description of the diagram

The work starts from a task with written acceptance criteria. The assistant may write code, tests and notes, explain code, and draft the text of the pull request, the request to merge a change. It may not see secrets, customer data or personal data. It may not add a new dependency without a person’s check. It may not delete, deploy or change shared systems unless a person approves each step. Before the merge, a person checks five things. Every acceptance criterion is met. One test fails before the change and passes after it. The tests test the requirement, not a copy of the code. There are no unrelated changes. Every new dependency is checked. After the merge, the person who opened the pull request owns the change, as if they had written every line.

Discuss: ban the assistants, or let them write the tests too?

Nadia asks the team to agree a policy before the limited launch.

• Kwame: “Ban coding assistants on the help-desk code. We cannot check what they do.”
• Amira: “Let the assistant write the tests as well. It’s much faster, and the tests always pass.”
• Omar: “No new rules, please. Speed is the reason we pay for these tools.”
• Sam: “If the assistant writes a bug, whose fault is it?”

Before you read on, answer these:

1. What does the recorded evidence say about Kwame’s ban?
2. What is wrong with “the tests always pass”?
3. Answer Sam’s question in one sentence.

The course’s answer. Do not ban the assistants. The recorded bounded task passed all 8 checks: the problem was the vague request and the missing review, not the tool. Allow them inside the policy above. Amira’s tests are the risk. A test written from the code agrees with the code, bugs included: none of the 11 caught the bug. So a person writes or checks one test that fails before the change. Omar gets his speed, with five checks that take minutes. Sam’s answer: the person who opens the pull request owns the change. “The assistant wrote it” is never a reason in a review or an incident.

What would change the answer: code that touches customer data, payments or secrets needs stricter rules. Section 6 of the template then names the approved tools and accounts, and what data each tool may see.

When to use these rituals, and what they cost

Use them from the pilot onward, when there is evidence to review and failures to read. A team of two can do all three in one hour a week. Do not turn the experiment review into a status meeting, or the error session into a search for someone to blame. The aim is kinds of failure, not people. What they cost: about two hours of meetings per person every two weeks, and half an hour per decision record. The forgotten decision about prompt v3 cost two meetings by itself.

Common mistake

The error session reads only scores

Symptom: the team looks at a chart of judge scores every two weeks, and it never finds a new kind of failure.

Cause: nobody reads the failures themselves. A score says how often something goes wrong, not what goes wrong.

Fix: read 20 real failures, chosen by a written rule, with the domain expert in the room. Sort and count the kinds before you look at any score.

Common mistake

A policy nobody can check

Symptom: the policy says “use assistants responsibly”. A pull request with a new, unknown dependency is merged without comment.

Cause: the policy names a value, not checks. A reviewer cannot see if “responsibly” was followed.

Fix: write each rule as a check that a reviewer can see in the pull request. For example: every acceptance criterion is met, and one test fails before the change.

Module practice: review an experiment report

The worked example for this practice is Nadia’s review of prompt v3, in Review work you cannot write yourself. Now you review a report yourself.

Tomás, the AI engineer, trained Larkfield’s own small model for routing. It chooses the team for each ticket and decides if a person must handle it. He recommends that routing switches to it next month. His report is evidence/experiment-report.md. Its text is constructed for the course, and its numbers are recorded in Fine-Tuning and Adapting AI Models.

Guided practice: the first three lines

1. Copy the checklist into your document.
2. Read the report once, from start to end. Do not fill anything yet.
3. Fill line 1, the decision. Expected: “switch routing from the hosted model to our own model next month”. The report states it.
4. Fill line 2, the baseline. Look for the hosted model’s result on the same tickets. Expected: missing. The report gives the hosted model’s cost, but not its results.
5. Fill line 3, the evaluation set. Expected: “102 development tickets”. No holdout tickets are mentioned.

Check yourself: you have one line found and two lines missing. A missing line is a question for Tomás.

Independent task

Fill lines 4 to 10 alone. Then write three questions to Tomás and a decision: accept, ask for more, or reject. Name the missing evidence that matters most, and say why it matters to the decision.

Show a worked solution

Line	Found
1. The decision	Switch routing to the small model. Stated.
2. Baseline	Missing. The hosted model on the same tickets: team 97 of 100, needs a person sent to one 23 of 23. A simple word-count model: team 86 of 100.
3. Evaluation set	Development tickets only. Holdout missing.
4. The measure that matters	Format and team are shown. “Needs a person” is missing. The training loss is not a user’s result.
5. Groups	Missing: safety and French tickets are not shown.
6. Range and runs	Missing.
7. Error examples	Missing.
8. Cost	Model fees only: US$8.53 a month saved. People’s time and the computer are not counted.
9. Not tested	Not stated.
10. Repeatable	Not stated.

The missing evidence that matters most is line 4: tickets that need a person. The fine-tuning course recorded it:

Tomás's model sends 0 of 23 development tickets that need a person to a person (holdout: 0 of 24). The hosted model sends all of them (recorded, fine-tuning course).

Text description of the chart

Three columns, on an axis from 0 to 23. The hosted model in use today: 23 of 23. A simple word-count model: 6 of 23. Tomás’s model: 0 of 23. On the holdout tickets: hosted model 24 of 24, word counts 8 of 24, Tomás’s model 0 of 24.

The small model never sends a ticket to a person, safety tickets included. “No false alarms at all” in the report is the same fact, seen from the other side. A model that never says “yes” never says it wrongly.

Three questions: 1. What does the hosted model score on the same tickets, including “needs a person”? 2. What are the results on the holdout tickets, and by group (safety, French)? 3. Show ten tickets that needed a person, with the model’s answers.

Decision: reject the switch. The saving is US$8.53 a month; the risk is every safety ticket. The fine-tuning course reached the same decision: do not ship this model. Other defensible answers exist, for example “ask for more” with the three questions, if the decision stays “no switch” until the answers arrive.

Knowledge check

Check your understanding

Why does an error-analysis session read failures chosen by a written rule, and not the failures that someone remembers?

Show the answer

A person who picks failures by hand picks the memorable ones: the worst, or the ones that prove their point. Then the counts of each kind say more about the person than about the product. A written rule, such as “the first 20 drafts that agents rewrote this week”, gives a fair sample, and the next session can repeat it. The evidence pack chose its five examples by rules for the same reason.

Check your understanding

Amira says: 'The assistant wrote eleven tests and they all pass, so the change is safe.' What do you answer, using the recorded result?

Show the answer

Tests that pass prove only that the code does what the tests expect. When an assistant writes tests from the code, they copy what the code does, bugs included. In the AI-assisted development course, none of 11 generated tests caught the bug, and 2 of them checked that the bug was there. Ask for one test, written or checked by a person from the requirement, that fails before the change and passes after it. And check every acceptance criterion.

Check your understanding

Which decisions deserve a decision record? Give one Larkfield example of each kind, and one decision that does not need a record.

Show the answer

A record is for a decision that is hard to undo, or one that someone will question later. Hard to undo: switching the help desk to a new model provider, with a contract. Questioned later: not releasing prompt v3, which someone will propose again because it looks cheaper. No record needed: the wording of one reply template, which anyone can change back in minutes.

Check your understanding

Completion check. In Tomás's experiment report, name one missing piece of evidence and explain why it matters to the decision to switch routing.

Show the answer

The result for tickets that need a person (line 4) is missing. It matters most because the decision is about routing, and a ticket that needs a person, such as a safety ticket, must reach one. The fine-tuning course recorded 0 of 23 on development and 0 of 24 on holdout: the model never sends a ticket to a person. The saving is US$8.53 a month, so the missing line hides a risk far larger than the gain. Other good answers: the missing baseline (the hosted model: 23 of 23, and the right team for 97 of 100), or the missing holdout set.

Recap

This lesson

• Three rituals keep reviews going. A weekly experiment review uses the checklist. An error-analysis session reads and sorts failures every two weeks. A decision record keeps each decision that is hard to undo.
• An AI-assisted development policy says what assistants may do, what a person checks, and that the author owns the change. Recorded: a vague request passed 4 of 8 checks, a bounded one 8 of 8; none of 11 generated tests caught the bug.

Module 5

• A role map puts one person who decides in each stage, keeps the builder and the judge apart, and gives adoption and operations an owner.
• A role profile starts from the work; a work sample with a rubric tests it; weekly work is grown or hired, a project is borrowed.
• You review AI work without code: ten lines of evidence, then accept, ask for more or reject. Tomás’s report hides the result that matters: 0 of 23 tickets that needed a person reached one.

Next, you run the team week by week: goals for uncertain work, one-to-ones, failed experiments and the on-call load.

Next: Run an AI team day to day