Grow and keep AI specialists

What you will be able to do

Keep the specialists who make AI work trustworthy, and help them grow. Reward judgement, not tools; count good work that does not ship; and make sure that no knowledge sits with one person only.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• Role profiles that start from the work, from Hire and grow skills.
• The decision record and the error-analysis session, from Team rituals and AI-assisted work.
• One-to-ones and the on-call rotation, from Run an AI team day to day.
• Helpful, not required: the AI job families, from The main job families in AI Career Foundations: Roles, Skills and Roadmaps.

What you need

• A document for your notes. In “Your turn” you write a one-page plan for one specialist.

Priya’s situation, the career tracks and the conversations in this lesson are made up for the course. They are the course’s recommendations, with a reason for each.

The senior nurse

On a hospital ward, the senior nurse is not senior because she knows the most machines. She sees trouble early in a patient, explains why to the doctor, and teaches the new nurses. Her judgement is the skill.

Two things keep a ward strong. Each shift hands over in writing, so no patient depends on one nurse’s memory. And a good nurse can grow without leaving the patients. If her only way up is an office job, the ward loses its best nurse. If she does not want the office, it may lose her completely.

The problem: the person nobody can lose

Picture Larkfield one year later (a constructed situation). Only Priya knows how the evaluation set was built and why the holdout exists. Only she can run the judge with the policy. Her best work this year stopped prompt v3: a launch that did not happen. Her yearly review says “no feature shipped”. Her only way up is “manager”. Every week someone asks her to test a new model. Then she tells Nadia that she has an offer from another company.

Three problems meet here. The career path rewards the wrong thing. The review cannot see good work that does not ship. And the team’s most important knowledge sits with one person.

The idea: two tracks, one meaning of “senior”

A career track is a ladder of levels with what each level means. AI teams need two tracks of equal standing. On the specialist track, a person grows depth in one area, such as evaluation or data. On the lead track, a person grows breadth: work across areas, and other people. Both tracks reach the same levels and the same pay range, so nobody must stop doing the work they are best at to grow.

Two career tracks of equal standing (a recommendation). Each level is described by judgement, evidence and mentoring, not by the tools that a person knows.

Text description of the diagram

The ladder starts with a junior, who does a well-defined task, with review. A mid-level person runs an experiment alone, from the question to the evidence. A senior person shows judgement about which question matters and when to stop. They bring evidence that others can check, and they mentor one or two people. After senior, the ladder splits. On the specialist track, a staff specialist sets the standard in one area, such as evaluation. On the lead track, a team or product lead works across areas and grows people.

In AI work, senior means three things. Judgement: the person knows which question matters and when to stop. Evidence: their decisions rest on results that others can check. Mentoring: the people around them get better. It does not mean the number of tools that a person knows. Tools change every year; judgement does not.

A tiny example

Two ways to describe Priya’s year in a promotion case:

Described by tools	Described by judgement, evidence and mentoring
“Used three judge models and two evaluation libraries.”	“Insisted on a holdout set that showed prompt v3 was worse. Wrote the rule that picks error examples. Taught two senior agents to label.”

The first column could describe anyone. The second describes the work that protected Larkfield’s customers.

Recognise good work that does not ship

Much of the best AI work ends in “no”, and a review that counts only launches misses it. Name such work in the team meeting, in the yearly review and in the promotion case. The question to ask is: “Which decision did your evidence change, and what did it save?”

Good work that did not ship	Why it counts
A well-run no-go. The routing model sent 0 of 23 tickets that needed a person to a person (recorded, Fine-Tuning and Adapting AI Models). The team said “do not ship”.	It closed an unsafe route before any customer saw it.
A found leak. Mei sees that some holdout tickets also appear in the prompt’s examples (constructed).	Every score on that holdout was too good. The evaluation course calls this contamination.
An honest holdout. On development tickets alone, v3’s difference in “both decisions right” was −0.059, with a range that included zero. The holdout showed −0.112, with the whole range below zero (recorded).	Without the holdout, the team could not show that v3 was worse.

Protect learning time

AI tools and methods change fast, so a specialist who stops learning falls behind quickly. Give each person protected learning time, for example half a day every two weeks (a suggestion). Tie it to the team’s work. For example, take a course for a skill on the person’s plan, or try a method that the team may need. Put it in the calendar like a meeting. Under pressure, learning time is the first thing to go, so the leader defends it.

Spread the knowledge

Knowledge on one person is a risk, even if that person stays: they get ill, take holidays and change roles. The lesson Team rituals and AI-assisted work showed the cost. Priya was on holiday, and the team spent two meetings on an old argument.

Three ways to move knowledge off one person. Use all three; pairing first for the work that matters most.

Text description of the diagram

On the left, a red box shows knowledge on one person. Only Priya knows how the evaluation set was built, why the holdout exists and how to run the judge. If nothing changes and Priya is away or gone, the team argues again or ships without the check. Three arrows show how to spread it. Pairing: Sam builds the next evaluation set with Priya. Decision records: the reasons for the holdout and for stopping prompt v3 are written down. Rotation: a different person leads each error-analysis session.

Pairing means two people do one piece of real work together, the expert and a learner. Written decision records keep the reasons when people forget them. Rotation gives a duty, such as leading the error session or being on call, to a different person each time. Start with pairing on the work that the team can least afford to lose.

Burnout and hype pressure

Burnout is long-term exhaustion from work: the person is tired, cynical and less effective. AI teams face two special causes. The first is a deadline for quality, such as “the score must reach 4.5 by March”. The person cannot control the result, so the pressure never ends. The second is constant model churn: a new model every week, and someone who wants it tested today.

The leader’s tools come from the last lesson. Put dates on decisions, not on quality. Treat every new model as a planned experiment with a question and a stop point. Ask one person each month to read the new releases for an hour and suggest at most one trial. And ask about load in every one-to-one. Late nights, cancelled holidays and silence in reviews are signs to act on, not to admire.

Early signs that someone may leave

People rarely leave without signs. Watch for someone who stops proposing ideas or goes quiet in reviews. Watch too for learning time that is always cancelled, and questions about growth that get no answer.

Have the conversation early, in private, and ask directly: “What would make this a place where you want to work in two years?” Listen first. Agree one concrete change with a date, and follow it up in the next one-to-one. A counter-offer of money in the last week rarely fixes a problem that was never about money.

Work with people outside the team

Specialists who only meet each other lose touch with the problem. Give each one a regular contact outside the team. Grace’s agents in the error session keep the work honest about real tickets. Legal and privacy colleagues, met early, prevent late surprises: see Governance in practice. Support colleagues bring the failures that users report.

Discuss: how do we keep Priya?

Priya has an offer from another company. Nadia asks the team.

• Omar: “Match the salary. That is what people care about.”
• Kwame: “Promote her to manager of the data team. That is the next step.”
• Grace: “She is the one who reads tickets with my team. If she leaves, nobody else can.”
• Sam: “I want to grow too, but nobody reviews my evaluation work. Only Priya knows it.”

Before you read on, answer these:

1. What does Priya most likely need? How will Nadia find out?
2. What is the risk of Kwame’s offer?
3. What should Nadia do about Grace’s and Sam’s points, even if Priya stays?

The course’s answer. Ask Priya first, in a one-to-one; do not guess. The situation points to recognition, a path and less churn. Her best work was counted as nothing, her only way up leaves the work she is best at, and the model requests never stop. So: correct the review, offer the specialist track at the same level as a lead, protect her learning time, and route new models through planning. Check the pay too. If it is below the market, fix it, but money alone does not fix a review that cannot see her work. Kwame’s offer may take her away from evaluation, so offer it only if she wants to lead people. Grace’s and Sam’s points stand even if Priya stays. Pair Sam with Priya on the next evaluation set, write the decision records, and rotate the error session.

What would change the answer: if Priya says that she wants to lead people, the lead track is right. If her pay is far below what others offer for the same work, pay comes first.

When to use this, and what it costs

Use it as soon as the team has specialists who could grow. Do not copy a tool list into a career ladder, and do not promise a level that the company cannot give. What it costs: a few days to write the two tracks with the people involved. Learning time of half a day every two weeks is about one working day a month per person. Pairing slows the expert for a few weeks. Losing the only person who knows the evaluation work costs more: the next release may ship without the check.

Common mistake

The only way up is management

Symptom: the best evaluation specialist becomes a manager, is unhappy, and the evaluation work slips.

Cause: the company has one track. Growth means leaving the specialist work.

Fix: write a specialist track with the same levels and pay range as the lead track. Describe each level by judgement, evidence and mentoring.

Common mistake

The expert who never takes a holiday

Symptom: Priya cancels her holiday, because “nobody else can run the gate”.

Cause: the knowledge sits with one person, and the team treats that as loyalty, not as risk.

Fix: pair a second person on the next gate, and write the steps and the reasons down. Let the second person run one gate before the holiday.

Your turn

Write a one-page plan for one specialist on your team, or for Priya:

1. Their track and their next level, in terms of judgement, evidence and mentoring.
2. One piece of good work that did not ship, and how you will recognise it.
3. Their learning time, and what it is for.
4. One way to spread their knowledge, and with whom.
5. One regular contact outside the team.

Then change one condition: the person tells you that they want to lead people. Which parts of your plan change?

Show a worked solution (Priya)

1. Track. Specialist track; next level: staff evaluation specialist. Judgement: she decides which evidence each release needs. Evidence: every gate has a written report. Mentoring: two people can run a gate without her within six months.

2. Recognition. The holdout that stopped prompt v3, named in the yearly review as “protected customers from a worse reply on two safety tickets”. It is also named at the next team meeting.

3. Learning time. Half a day every two weeks, for methods to check a judge against people’s scores.

4. Spread. Sam pairs with her on the next evaluation set. Priya writes the decision record for the holdout. The error session rotates between Priya, Sam and a senior agent.

5. Outside contact. Grace’s team, every two weeks in the error session.

If she wants to lead people, parts 1 and 3 change: the lead track, and learning time for running a team. Parts 2, 4 and 5 stay. Spreading her knowledge matters even more, because she will have less time for evaluation work.

Knowledge check

Check your understanding

Two people want promotion to senior. One knows five evaluation tools. The other chose the holdout that stopped a bad release and taught two colleagues to label. Who shows seniority, and why?

Show the answer

The second person. In AI work, senior means judgement (which question matters and when to stop), evidence that others can check, and mentoring. Choosing the holdout was judgement backed by evidence, and teaching two colleagues is mentoring. Knowing five tools says little, because tools change every year and a person can know them without good judgement.

Check your understanding

A team member's experiment ends in a well-run no-go. How do you make sure that this counts as good work?

Show the answer

Name it where work is counted: in the team meeting, in the yearly review and in the promotion case. Describe it by the decision that the evidence changed and what it saved. For example: “The routing model sent 0 of 23 tickets that needed a person to a person. The no-go kept an unsafe change away from customers.” A review that counts only launches teaches people to hide negative results.

Check your understanding

Only one person knows how the release gate works. Name three ways to spread that knowledge, and say which one you start with.

Show the answer

Pairing: a second person runs the next gate with the expert. Written decision records: the steps and the reasons are written down. Rotation: a different person leads the gate or the error session each time. Start with pairing on the gate itself. It is the work that the team can least afford to lose, and real work teaches more than a document.

Recap

• Offer two career tracks of equal standing: specialist and lead. Senior means judgement, evidence and mentoring, not tools.
• Recognise good work that does not ship: a well-run no-go, a found leak, an honest holdout. Ask which decision the evidence changed.
• Protect learning time in the calendar, tied to the team’s work.
• Spread knowledge with pairing, decision records and rotation, so the work does not depend on one person.
• Act early on burnout (deadlines for quality, constant model churn) and on signs that someone may leave: ask directly, agree one change with a date.
• Give each specialist a regular contact outside the team.

Next, Module 6 turns to risk: who could be harmed, how badly, and who owns each risk.

Next: Risk at leadership level