Hire and grow skills

What you will be able to do

Fill a missing role well. Describe the work first, and test candidates on a piece of that work. Then decide whether to hire someone new, grow someone you have, or borrow the skill.

Before you startWhat to know and to have readyAbout 30 minutes

What you need to know

• Larkfield’s role map and its empty ML engineer row, from Team roles by stage.
• What the evidence pack measured and what it did not (one person labelled the tickets), from Read the evidence pack.

What you need

• The template templates/role-profile.md, copied into your own document.

The job advertisement, the profile and the work sample in this lesson are made up for the course.

Two ways to hire a cook

A small restaurant needs a cook. The first advertisement says: “Five years with a named brand of oven. Experience with a named knife set and a named ordering app.” The second says: “Cook our lunch menu for 40 people a day. Keep the food cost of each plate under a third of its price. Teach our junior cook in three months.”

The first advertisement describes the kitchen. The second describes the work, and how the owner will know it is done well. Then the second restaurant does one more thing. Each candidate cooks three dishes from the real menu, in the real kitchen, for two hours. Two people taste each plate and score it with the same short list. The cook who knows the oven brand may not taste the sauce before it leaves the kitchen. The trial shows that in two hours. An interview may never show it.

The problem: a profile built from tools

Larkfield’s role map has gaps, and Omar wants to fill one quickly. He writes a first draft of a job advertisement (constructed):

Senior AI engineer. Must have: five or more years with large language models. Experience with a named agent framework, a named vector database and one named cloud. Fine-tuning of open models. Container orchestration.

Every line is a tool or a technique. Not one line says what the person must produce at Larkfield, or how anyone will know it is good.

Look at what that costs. In the fine-tuning course, Larkfield trained a small open model for routing. Its training error fell steadily, and its answers had a perfect format. But it sent 0 of 23 development tickets that needed a person to a person (recorded, Fine-Tuning and Adapting AI Models). The valuable skill was not the training. It was the evaluation that found the zero, and the judgement to say “do not ship”. Omar’s advertisement asks for the first skill and never mentions the second.

Tools also change faster than the work. The work “check that every safety ticket reaches a person” stays the same for years. The best tool to do it may change twice in that time.

The idea: a role profile from the work

A role profile describes a role by its work, in five parts, and in this order:

A role profile starts from the work and ends with the tools. A tool list starts at the end and skips the parts that tell you what a good hire looks like.

Text description of the diagram

Box 1, the work: the outcomes that the person must produce in the first six months. For each one, the team writes how it will judge it. Box 2: the decisions that the person makes alone, and the ones they bring to others. Box 3: the skills that the work needs, each marked “must have” or “can learn here”. Box 4: a work sample, a real task of two to three hours, scored by two people with one rubric. Box 5: tools, only the few that truly matter. A tool list starts at box 5 and skips the first four.

Two words need a definition. A work sample is a short, real task from the job, which a candidate does instead of a quiz. A rubric is a short written list of what a good answer contains, so that two people score the same answer the same way. The template templates/role-profile.md has a section for each part.

A tiny example

Here is the first part of a profile that Priya drafts for the work that Larkfield really lacks. The evidence pack says that one person labelled every test ticket, and that the judge model agrees with that person only fairly. So the missing work is a second, careful reading of the tickets.

Outcome in the first six months	How we know it is done well
A second person labels the 200 development and holdout tickets with Grace’s policy	The agreement with the first labels is counted, and every disagreement is listed with a reason
Every new prompt or model goes through the release gate before launch	No release without a gate report
A session every two weeks where Grace’s team reads 20 failures	The failures are sorted into kinds, and new kinds become test cases

Then come the skills. “Applies a written policy the same way every time” is a must-have, tested by the work sample. Someone can learn “Counts agreement between two labellers” at Larkfield, from Calibrate reviewers in Evaluating and Testing AI Systems. The only tool is a spreadsheet.

The work sample: 10 tickets, Grace’s policy, and the assistant’s draft for each. The candidate marks each draft “send”, “edit” or “wrong”, and writes one paragraph about the worst one. The rubric lists what a good answer finds. For example, one draft invents a “saved-card section” that the policy does not have.

Why two scorers and one rubric

Scoring a candidate’s answers is labelling, like labelling tickets, and two careful readers often disagree. The evaluation course measured it. A strong judge model and the course author both scored the same 28 replies from 1 to 5.

Two careful scorers of the same 28 replies gave the same score 46.4% of the time, and a score within one point 82.1% of the time (recorded in the evaluation course).

Text description of the chart

Two columns on an axis from 0 to 100%. The judge model and the course author gave exactly the same score to 46.4% of the 28 replies. They gave scores within one point of each other to 82.1%. So about one reply in five got scores two or more points apart.

Less than half of the scores matched exactly (recorded, Evaluating and Testing AI Systems). Interviewers are no different. Without a rubric, one interviewer rewards confidence and the other rewards caution. So each interviewer scores the work sample alone, with the rubric. Then they compare. A difference of two points is a reason to talk, and often a reason to improve the rubric.

Hire, grow or borrow

A family that needs a ladder once a year borrows one from a neighbour. A painter buys one, because she uses it every day. A missing skill works the same way. There are three ways to fill it. Hire a new person, grow someone you have, or borrow the skill from a contractor, a partner or another team.

Six questions that decide it

Question	Why it matters	What the answer points to
1. How long does the need last?	A hire is for years; a contract ends.	For years: hire or grow. A project or a peak: borrow.
2. How core is the skill to the product?	A core skill decides your quality or your users’ trust.	Core: keep it inside. A commodity that many firms do the same way: borrow.
3. How urgent is it?	Hiring and growing both take months.	Needed now: borrow first, even if you grow or hire later.
4. Does someone inside have an adjacent skill, and the time to learn?	An adjacent skill is half the learning. Time is the other half.	Both: grow. Neither: hire.
5. What is lost if the person leaves?	Knowledge that sits with one outsider leaves with the contract.	High: keep it inside, or write knowledge transfer into the contract.
6. What does each option cost over a year?	Compare a whole year, with the same rate for everyone’s time.	The cheapest option that passes questions 1 to 5.

The first four questions give a flow. Questions 5 and 6 then check the answer.

Four questions choose between hiring, growing and borrowing. The common mix: borrow now, grow inside, with the knowledge transfer written into the contract.

Text description of the diagram

The first question asks how long the need lasts. For a project or a peak, borrow, with a handover written into the contract. For years, the second question asks if the skill is core to the product’s quality or trust. If it is a commodity, borrow. If it is core, the third question asks if someone inside has an adjacent skill and the time to learn. If not, hire, with a role profile and a work sample. If yes, the fourth question asks if the skill is needed before they can learn it. If not, grow them. If yes, borrow now and grow inside at the same time, with knowledge transfer written into the contract.

A worked example: on-call for the live assistant

Larkfield’s limited launch is six weeks away (constructed). After launch, someone must act when the service fails. The evidence pack’s “last week” shows why: 10 tickets that needed a person reached nobody, while the dashboard looked green (recorded answers, constructed situation). Nadia takes the need through the questions.

Question	Larkfield’s answer
1. How long?	For as long as the assistant runs: years.
2. How core?	Core: a silent failure hurts customers directly.
3. How urgent?	Needed on launch day, in six weeks.
4. Adjacent skill and time?	Yes. Amira runs the service and Mei the database, as in the MLOps course. Nobody has run an incident yet.
5. Lost if a person leaves?	High if one outsider holds it.
6. Cost over a year?	See below. All rates and hours are assumptions.

The costs use the course’s labour rate of US$60 an hour (base) and US$100 (high), both assumptions. A full-time hire is about 1,800 hours a year, so about US$108,000. Borrowing an experienced operator half-time for three months is about 240 hours at the high rate: US$24,000. Growing Amira, Mei and Tomás takes about 60 hours of learning and pairing each: 180 hours, US$10,800. At Larkfield’s size, on-call is not a full-time job (an assumption: about four hours in a quiet week). So the hire is the dearest answer to a small need.

The decision: borrow now, grow inside. The operator runs on-call for three months with Amira, Mei and Tomás, for about US$34,800 in all. The contract names the transfer: the runbooks are written together, and by the second month an inside person leads each incident. What would change it: a launch a year away means grow only. With nobody inside who runs the service, hire.

The main risk of each option

Option	Main risk	How to reduce it
Hire	Slow, and expensive if the need ends	Hire only for work that lasts years. Borrow while you hire
Grow	Slow, and the person may leave once trained	Real work with pairing, a second learner, and a career path, as in Grow and keep AI specialists
Borrow	The knowledge leaves with the contractor	Write the transfer into the contract: pairing, written steps, a handover. Name an inside owner from day one

Recover a stalled AI project shows the borrow risk (constructed). A contract data engineer built the data pipeline, and her contract ended. When the feed changed, the team used a copy of old data for months.

Growing has a plan, or it does not happen. The template’s last section names the person, the skill, how they learn it, and by which stage.

Discuss: one new person, or none?

Nadia has the budget for one new person this year. The team has views.

• Omar: “Hire the senior AI engineer from my advertisement. The market is moving fast, and we need someone who knows the tools.”
• Priya: “What we lack is a second labeller and time to read failures. Nobody reads errors with Grace’s team today.”
• Grace: “Two of my senior agents know the policy better than any new hire. Give them the time, and they can label.”
• Kwame: “Hire an ML engineer now. Then we can run our own model one day and stop paying the provider.”

Before you read on, answer these:

1. Which work is really missing from Larkfield’s role map?
2. For each piece of missing work, would you hire, grow or borrow?
3. What does the evidence say about Kwame’s idea?

The course’s answer. The missing work is evaluation: a second labeller, and someone who reads failures with Grace’s team every two weeks. It is weekly work, and two senior agents can learn it with Priya before the limited launch. So grow them, and pay for their hours from the budget. Do not hire from Omar’s advertisement: it describes tools, not Larkfield’s work. Kwame’s idea fails on the evidence. The fine-tuning course estimated the hosted model at US$8.53 a month at 822 tickets a day. Hosting a fine-tuned model has a list price of US$1.70 an hour, about US$1,241 a month (price). Prices change, so check the current page. And the trained model sent 0 of 23 tickets that needed a person to a person. There is no weekly ML-engineering work today.

What would change the answer: a decision, with evidence, to run Larkfield’s own model. Then ML engineering becomes weekly work that nobody on the team can learn in time. That is a hire, with a profile from the work and a work sample. Or the evaluation work grows beyond what two agents can do part-time: then the evaluation lead becomes a hire.

When to use a role profile and a work sample, and what they cost

Use them for every role on an AI team, also when you grow someone inside. The outcomes in part 1 tell them what success looks like. Do not ask for a long unpaid project. Two to three hours is the limit in the template, because a long unpaid task is unfair to people who have less free time. What they cost: about two hours to write the profile and the rubric, and two interviewers’ time to score each sample. A wrong hire costs much more.

Common mistake

A work sample that nobody can score

Symptom: two interviewers score the same candidate’s work sample 2 and 5 out of 5. Each is sure that they are right.

Cause: there is no rubric, so each interviewer scores what they personally value.

Fix: write the rubric before the first candidate. List three to five things that a good answer contains, and one automatic “no”. For example: a safety ticket with no answer, marked “send”. Score alone, then compare. If a difference of two points remains, improve the rubric.

Common mistake

A course without real work

Symptom: Omar completes an evaluation course, but at the next gate he still asks only “what is the average score?”.

Cause: the learning plan said “take a course” and nothing else. Nobody used the new skill on real work.

Fix: pair the course with a real task. Omar reviews the next two experiment reports with Priya next to him, using the checklist of the next lesson.

Your turn

Copy templates/role-profile.md into your document. Write parts 1, 3 and 4 for one role that your own team, or Larkfield, lacks. Then change one condition: the work is needed only for the next four months. Which part of your answer changes?

Show a worked solution (Larkfield’s two senior agents)

Role title: Evaluation labeller (part-time, grown inside). Team: support. Reports to: Grace.

Outcome the person must produce	How we will know it is done well
A second label for each of the 200 development and holdout tickets	Agreement with the first labels counted; every disagreement listed with the policy line
20 failures read and sorted every two weeks, with Priya	A short list of failure kinds, with new test cases for new kinds

Skill	Must have / Can learn here	How we test it
Applies Grace’s policy the same way every time	Must have	The work sample: 10 tickets and drafts
Writes a clear reason for each label	Must have	The paragraph about the worst draft
Counts agreement between two labellers	Can learn here	Pairing with Priya, and Calibrate reviewers

Work sample: 10 tickets, the policy and the assistant’s drafts; mark each “send”, “edit” or “wrong”; one paragraph about the worst. Rubric: finds the invented step, finds the draft with no answer on a safety ticket, gives a policy line for each “wrong”. A safety ticket marked “send” is an automatic “no”.

If the work lasts only four months, the answer to the first question of the flow is “no”. Borrow: for example, a contractor labels, with a handover to Grace’s team. Parts 1 and 4 stay the same. The work and the test do not change; only the way you fill the role changes.

Knowledge check

Check your understanding

Why is a work sample a better test than a quiz about tools? Answer in two or three sentences, with a Larkfield example.

Show the answer

A work sample tests the work that the person will do, with real-looking material, so it shows how they think and what they notice. A quiz about tools tests what they remember about a tool that may change next year. At Larkfield, one draft invents a “saved-card section”. A candidate who marks it “wrong” and explains why from the policy shows the skill that the role needs. Knowing a tool’s name does not show it.

Check your understanding

Two interviewers scored the same work sample 2 and 5. What do you do, and why is this not surprising?

Show the answer

Scoring is labelling, and careful labellers often disagree. In the evaluation course, a judge model and the course author gave the same score to only 46.4% of 28 replies. First check that each interviewer scored alone, with a written rubric. Then compare the two scores line by line, find the point where they differ, and improve the rubric before the next candidate. If there is no rubric, write one now and score the sample again.

Check your understanding

For each case, choose hire, grow or borrow, and give the reason: (a) move five years of old tickets into a new database, a three-month project; (b) read failures with Grace's team every two weeks, from now on; (c) run and update Larkfield's own model every week, if Larkfield decides to run one.

Show the answer

(a) Borrow: the work is a project, not weekly work, so a contractor or another team does it, with a handover. (b) Grow: the work is weekly, and Grace’s senior agents can learn it with Priya before the next stage. They already know the policy, and that is the hardest part. (c) Hire: the work is weekly, and nobody on the team can learn it in time. Write the profile from the work and use a work sample.

Check your understanding

Larkfield needs a privacy review of the assistant's logs before the full launch: about three weeks of work, once a year. Nobody inside knows privacy law. Hire, grow or borrow? Name the main risk of your choice and how you reduce it.

Show the answer

Borrow. The need is a short project once a year, not weekly work for years, and nobody inside has an adjacent skill. A specialist from outside, such as a privacy lawyer, fits. The main risk of borrowing is that the knowledge leaves with the contractor. Reduce it in the contract: the specialist writes the review and a short checklist that Larkfield’s team can use next year. Name an inside owner, for example Kwame, who works through the review with the specialist.

Recap

• A role profile starts from the work: outcomes in six months and how to judge them. Then come decisions, skills, a work sample, and tools last.
• A tool list hires for the technique, not the judgement. The fine-tuning course’s trained model had a perfect format and sent 0 of 23 tickets that needed a person to a person.
• A work sample is a real task of two to three hours, scored by two people alone with one rubric, then compared. Careful scorers agree less than you think: 46.4% the same score on 28 replies.
• Hire, grow or borrow: six questions decide it. How long is the need, how core and how urgent? Can someone inside learn it? What is lost if the person leaves, and what is the cost over a year? A common mix is borrow now, grow inside, with knowledge transfer written into the contract.
• Each option has a main risk. A hire is slow, and dear if the need ends. A person you grow may leave once trained. A contractor takes the knowledge away.
• Larkfield grows two senior agents into labellers. It does not hire an ML engineer, because it has no weekly ML-engineering work.

Next, you review an experiment report that you could not have written yourself, with a ten-line checklist.

Next: Review work you cannot write yourself