The data scientist

Many people who apply for AI Engineer roles are data scientists. They have trained models and designed experiments for years, and many have used large language models (LLMs) at work, often to classify text. When they read “help define our evaluation strategy”, they think, correctly, “I know how to do that.”

This profile is strong where many engineers are weak: deciding what “good” means, and measuring it honestly. The usual gap is on the other side. A data scientist often hands the model to an engineer, who runs it. So the data scientist has rarely owned a service or been on call. Mariana’s interview shows both sides.

MC

The candidate

Mariana Costa

Data scientist, 5 years, moving into AI engineering

• 5 years as a data scientist
• Python, SQL, statistics
• MSc Statistics
• Toronto

• Data Scientist, Basketry (online grocery, about 900 people), 2021 to now
• Built churn and weekly demand models (gradient boosting) for the marketing and supply teams
• Designed and analysed 40+ A/B tests; wrote the team’s guide to sample sizes and stopping rules
• Led an LLM-based classification of customer complaints into 14 categories: labelled set of 1,200 complaints, 3 prompts × 2 models compared, macro-F1 0.84 (previous model: 0.71)
• Built and deployed it to production as a nightly batch job that routes about 6,000 complaints a day
• Python (pandas, scikit-learn), SQL, Jupyter, dbt, Git, some Docker

What Nadia sees before the call

Nadia reads the CV next to the job description. The complaint project has things that she rarely sees on these CVs. It has a labelled set with a size, a comparison of options, and a baseline. If these are real, Mariana can help with the team’s most urgent gap, the missing evaluation framework.

“Built and deployed” makes her careful. It is a strong claim from someone whose other tools are notebooks and dbt. Nadia does not assume that it is false. But the role owns features “end to end”, so she wants to know exactly who did what.

Her short list for the call:

• Are the labels good, and is the test honest?
• What did she own after the model was ready?
• What does she do when something fails in production?
• Can she transfer this to claim summaries, which are free text, not 14 categories?

The posting lists fine-tuning, but the team has never fine-tuned. Nadia does not test it.

Terms on this page

• TF-IDF and logistic regression: a simple, older text classifier that counts which words appear, and then learns a weight for each word.
• Macro-F1: the F1 score for each category, then averaged, so that a small category counts as much as a large one. See Classification metrics.
• Cohen’s kappa: how often two people agree on a label, after you remove the agreement that you expect by chance.
• dbt and Airflow: tools that data teams use to transform data in a database (dbt) and to run jobs on a schedule (Airflow).
• Structured output: the model must answer in a fixed format, such as JSON that matches a schema, so that code can read it safely.
• Golden set (eval set): a fixed set of examples with correct answers from experts, used to measure each new version.
• Hallucination (unsupported statement): something that the model writes and the source does not say.
• LLM-as-judge: a language model that grades another model’s output. It has its own errors, so you check it against people first.
• Retrieval: finding the relevant parts of a long document and giving only those to the model. The context window is the most text that the model can read at once.
• Queue, and retry with exponential backoff and jitter: a queue holds jobs until a worker is free. Each retry waits longer than the one before, for example 2, 4, then 8 seconds. A small random extra stops failed calls from all retrying together.
• Shadow mode: the new system runs on real data, but no user sees its output.

The interview

Minutes 0 to 5: the opening

Nadia(interviewer):

Hi Mariana, thanks for making the time. I’m Nadia, staff engineer on Claims Intelligence. I built most of our intake extraction. So, first I’d like to hear about something you built, and I’ll ask a lot of follow-up questions. That’s normal. Then a problem we have. Then your questions. And like the invite said, no AI assistants in this round. Paper notes are fine. If I look down, I’m only writing mine.

Mariana(candidate):

Okay, good. I should say, I’m a data scientist, not an engineer by title. So if I use too much statistics language, please stop me.

Minutes 5 to 25: the project, then deeper

Nadia(interviewer):

Sure. And you can stop me when I use too much engineering language. Pick a project you’re proud of and walk me through it.

Mariana(candidate):

Okay. The complaints classifier. So, Basketry gets around 6,000 complaints a day, from the app, email and chat. Before, keyword rules sent them to the teams, and about 30% went to the wrong team. We know this from re-assignments. When an agent passes a complaint to another team, it’s logged. And each wrong route, the customer waited about one day more.

So before any prompt, I built a labelled set. 1,200 complaints, 14 categories, like late delivery, substitution, food safety, billing. I kept 400 for developing the prompts and 800 for the final test.

Nadia(interviewer):

Mm-hm. And what did you compare?

Mariana(candidate):

Three prompts on two models, a large one and a small, cheaper one. And our old model, TF-IDF with logistic regression, that was macro-F1 0.71. The best setup got 0.84. The small model was almost the same like the large one, the difference was in the noise, and it cost about one sixth. So we chose the small one.

Nadia(interviewer):

And in production?

Mariana(candidate):

After six weeks live, re-assignments went from about 30% to about 12%.

What the interviewer is checking

Nadia hears many good signs in a few minutes. There is a baseline and a test set that was kept apart. There is a business measure, and a reason based on cost. She starts with the base of everything: the labels.

Nadia(interviewer):

Let’s start with the labels. Who labelled the 1,200, and how did you know the labels were right?

Mariana(candidate):

Me and Inês, she is a team lead in customer support. First we both labelled the same 300, separately. The agreement was… not so good. Cohen’s kappa of 0.62.

Nadia(interviewer):

Where did you disagree?

Mariana(candidate):

Mostly in two pairs. The big one was “missing item” versus “substitution”. For example, someone orders oat milk, gets almond milk, and writes “my oat milk is missing”. For us that’s a substitution, and it goes to a different team. So we added a rule to the guide: if something else came in its place, it’s substitution. It doesn’t matter what words the customer uses. We rewrote the guide, labelled 150 more, and got 0.81.

Nadia(interviewer):

And the other pair?

Mariana(candidate):

“Refund not received” and “billing error”. That one we never fully fixed. Some complaints are really both.

Why this works

Mariana gives a specific example, the rule that came from it, and the reason. Only the person who did the work knows this. She also says what she did not solve.

Nadia(interviewer):

Were the 14 categories balanced?

Mariana(candidate):

No, not at all. Late delivery was about 31%. Food safety, so allergens, spoiled food, something inside the food, was about 2%. Around 24 complaints in the whole set. And that’s the one that matters most, because the quality team must answer within 24 hours. So I reported recall per category, and I read every food-safety miss.

Nadia(interviewer):

That’s maybe 16 in your test split. How sure can you be about recall on 16?

Mariana(candidate):

Not very. We got 14 of 16, so 88%, but the 95% interval is roughly 64 to 97%. I put that in the report. My manager was not very happy.

Nadia(interviewer):

So what did you do?

Mariana(candidate):

We searched the archive with keywords and pulled about 60 more complaints that were probably food safety. We labelled them and measured recall on those. But I knew it makes us look better than we are, because a keyword search finds the easy ones. And precision I couldn’t measure at all, because that sample isn’t random.

Why this works

A percentage from 16 examples is a range, not one number. A keyword search finds mostly easy food-safety complaints. So recall on them looks better than it is, and they cannot show how often the model wrongly says “food safety”. Mariana knows both limits. Quillfield needs this skill, because its claims experts can label only a few packets a week. See The precision–recall trade-off.

How the interviewer goes deeper

Three levels down, Nadia still has not found the edge on evaluation. More questions here will not teach her much. So she turns to the part of the CV that she is less sure about.

Nadia(interviewer):

Your CV says you built and deployed it. Tell me about the production side.

Mariana(candidate):

Right. Honestly, “deployed” is too much. I wrote the classification code, the prompt and the parsing, as a Python module. Jonas, our ML engineer, made it a nightly Airflow job and set up the table it writes to. I reviewed his pull request, but for the logic, not the infrastructure. I should fix that line.

Where this goes wrong

“Deployed” on her CV was not true. Mariana corrects it at once, so it costs her little. If she defended it, Nadia would start to doubt the evaluation story too. See Build real skills, and learn to explain them.

Nadia(interviewer):

Thanks for being clear. So, at 2 a.m., what happens when the model returns something that isn’t one of your 14 categories? Or the call fails?

Mariana(candidate):

The first one I know. We used strict structured output, so the category is always one of the 14. The 0.3% failures were refusals, and responses that were… how do you say, cut off at the length limit. Those go back to the old keyword rules.

Nadia(interviewer):

And the call failing? Have you ever been paged for it?

Mariana(candidate):

Sorry, paged? Ah, you mean the alert, when you are on call. No, never. I believe Jonas added retries. But how many, or what happens if the whole night fails, I don’t know.

What the interviewer is checking

To be paged is to get an alert as the engineer on call. Mariana asks what the word means, and that is completely fine. Your English does not need to be perfect; the interviewer listens for your thinking. Here is the first sign of the edge: she knows how the output can be wrong, but not how the system can fail. Nadia keeps it for later.

Nadia(interviewer):

Last one on this. Six months later, how do you know it still works?

Mariana(candidate):

Every month, Inês and I label a random 100 from the last month. And I watch the mix of categories per day. In March, substitution jumped from about 9% to… 15%, I think, in one week. It could be real, there was a dairy shortage. But I ran that week again on the old model version, and it was the model. The provider had updated the model behind the name we called. Since then we pin the exact version, and we test before we move.

Why this works

This is real monitoring. There is a regular labelled sample, and a cheap signal that something changed. Then a check shows whether the world changed or the model changed.

Minutes 25 to 48: the Quillfield problem

Nadia(interviewer):

Here’s a real problem on our side. A claim packet comes in: 20 to 60 pages, scanned forms, the adjuster’s notes, emails, photos, a contractor’s estimate. Adjusters tell us they spend about 40 minutes reading a packet before they can act. We want to give them a one-page draft summary: the key facts, the open questions, and what’s missing. How would you approach it?

Mariana(candidate):

Hmm. Before models, I want to know how we will score it. So, who reads it, and when? And what is the worst mistake? Leaving a fact out, or saying something that isn’t in the packet? Because these need different measures.

Nadia(interviewer):

The adjuster, before the first contact with the customer. The worst mistakes are a wrong fact, like the date of loss, or leaving out prior damage or an exclusion that the packet mentions. And the summary is never the only basis for a decision. The adjuster confirms.

Mariana(candidate):

Okay. Then I wouldn’t score the summary as one piece of text. I’d break it into facts, with three error types. A key fact is missing, a fact is wrong, or a statement that no page supports.

Nadia(interviewer):

Mm-hm. And the “what’s missing” part?

Mariana(candidate):

That one I’d score like a classifier. Did it list what’s really missing, and only that? It’s easier to check than free text. And every sentence should point to its page. Then “is this supported” is a question about one page, not 60. But the correct answers must come from your claims experts. How much of their time is there, really?

Nadia(interviewer):

Honestly? About four hours a week between the two of them, in total. And most of that goes to the weekly intake sample.

What the interviewer is checking

Mariana designs the golden set before any prompt. She also asks about the experts’ time, which the posting does not mention.

Mariana(candidate):

Four hours. Okay, so every labelled packet is very valuable. Nobody should write ideal summaries, they’re slow and hard to compare. I’d ask for a checklist per packet. The 10 or 15 facts an adjuster must see, plus what’s missing. Maybe 15 minutes a packet, when they’re used to it. If I get two of those hours, that’s around 8 packets a week. And the 32 extraction fields you already label are free facts, so I’d start with those packets.

Nadia(interviewer):

Which packets would you pick?

Mariana(candidate):

Not random. Stratified. Home and car, injury and property-only, short and long, clean and bad scans, several insurers. I’d take more injury claims and bad scans, where I expect errors, and weight back for any overall number. About one packet in five goes to both experts, so we know how much they agree. Oh, and injury packets have health information. So I’d check who is allowed to see them for labelling, and where the set is stored.

Why this works

Mariana changes her plan to fit the few expert hours, and she reuses labels that already exist. She picks extra injury claims and bad scans, because errors are likely there. Then she corrects the overall score, so the extra hard packets do not make it look worse than it is. She also raises health information before anyone asks.

Nadia(interviewer):

Eight packets a week is tiny. You’ll change the prompt more often than that. How do you check the rest?

Mariana(candidate):

An LLM judge, for some of it. “Is this sentence supported by the page it cites” is a narrow question. I think a judge can do it well. But I’d measure it against the experts’ labels first, like I measured Inês and me. And per claim type, because maybe it’s fine on car claims and bad on injury.

Nadia(interviewer):

And for missing facts?

Mariana(candidate):

There I trust a judge less. To know what’s missing, you need to know what matters to an adjuster. That stays with the experts’ checklist.

The next few minutes, in short

They discuss whether to give the model the whole packet or to retrieve parts of it. Mariana says that she has not built retrieval. She proposes to test both on the golden set. She also wants to check if missed facts appear more in the middle of long packets. Nadia agrees that this is how the team would decide. Then she changes direction.

Nadia(interviewer):

I’m moving to a different area now. Say the golden set looks good and you’re live. Summaries run at intake, on about 9,000 packets a day. At the morning peak, about 3% of model calls time out. What happens, and what do you do?

Mariana(candidate):

Retry them. Although… if they time out because the provider is busy at the peak, retrying immediately adds more load at the worst moment. So wait a little first? I’m honestly not sure what the standard pattern is.

Nadia(interviewer):

And the packets where the retries also fail?

Mariana(candidate):

Then the adjuster opens the claim and there’s no summary. I’d want them to see “summary not available”, not half a summary. Then they read the packet like today. After that… I never designed this part. At Basketry, Jonas would have.

Nadia(interviewer):

Can a retry cause a problem of its own?

Mariana(candidate):

You mean it costs more?

Nadia(interviewer):

Sorry, I wasn’t clear. I mean, can the retry change what ends up stored?

Mariana(candidate):

Ah. Hmm. Maybe the first call did finish, only late, and now there are two summaries for one packet? Give me a second.

Pause.

No, I don’t know what you would do about that. I’d look at how intake extraction handles it, because it already runs on the same packets.

The edge

This is the edge. Mariana handles it well. She does not invent a pattern, and she points to the system that already solves the problem. But “Jonas would have” is a risk, because here the AI engineer owns the feature in production. Nadia writes one word: “ownership”.

A stronger answer

Mariana knew more than she used. She had depended on batch jobs for years. A stronger answer starts from that experience and says plainly where it stops:

“I haven’t owned this, so let me reason from the batch job I know. At Basketry, the results were written by complaint ID. If the job ran again, it replaced rows, it didn’t add new ones. I’d want the same here. One summary job per packet, keyed by packet ID and prompt version, so running it twice gives one result, not two. I think engineers call that idempotent: doing it twice is the same as doing it once. And a timeout on my side doesn’t mean the provider stopped. We may pay for both calls, and the late one may still write a result. That’s why the write must be keyed.

At intake, nobody is waiting. The adjuster opens the claim hours later. So I wouldn’t fight the peak. I’d put the jobs in a queue, limit how many calls run at once, and retry a failed call after a wait that grows each time. With a bit of randomness, so they don’t all come back together. After three or four tries, it goes to a failed list, and the adjuster sees ‘summary not available’. Someone gets an alert if that list grows past an agreed level. Then I’d measure the failure rate per hour and how long packets wait. I’m sure I’m missing details, so I’d ask how intake extraction does it.”

This answer does not pretend to come from an expert. It uses her own work as evidence.

Nadia(interviewer):

Okay, let’s stop there on that. One more: how would you release it to adjusters?

Mariana(candidate):

Oh, this I like. Shadow mode first. Generate summaries for real packets, show them to nobody, and score a sample against the experts. Then one pilot insurer.

Nadia(interviewer):

And how do you know it saves the 40 minutes?

Mariana(candidate):

I’d randomise by adjuster, not by claim. If one adjuster gets summaries on some claims and not on others, they start to read differently on all of them. Then the difference means nothing. I’d measure reading time, and also how often adjusters correct a fact, with a button in the interface. And a stop rule. If corrections of wrong facts go above an agreed rate, we pause.

Back on her strength

Choosing the adjuster as the unit of the experiment avoids a real trap. It is also a release plan, so it partly answers the earlier question. For the engineering side, see Progressive release.

Minutes 48 to 57: Mariana's questions

Mariana(candidate):

The weekly sample of 200 packets for extraction. How is it chosen? And did the accuracy move over the 14 months?

Nadia(interviewer):

Random by day. Field-level accuracy is around 94%, and fairly flat. But it drops on handwritten forms, and a random sample has few of those. Honestly, after what you said earlier, I think we should stratify it.

Mariana(candidate):

Who decides when the summary is good enough to release? Is there a number agreed with product?

Nadia(interviewer):

No. Nobody has decided that. We don’t have a shared evaluation framework. Each feature measured itself in its own way. That’s a big part of why we’re hiring.

Why this works

These questions make Nadia say two true things that the posting does not say. The sample has a blind spot, and the evaluation framework does not exist. See An interview is a conversation in both directions.

Mariana(candidate):

If something I built breaks at night, who gets the call? Would that be me? Because I’ve never been on call.

Nadia(interviewer):

Yes, after some time. One week in six, mostly the extraction pipeline and the policy assistant pilot. New people shadow someone for two rotations first. Most weeks are quiet. Some aren’t.

Mariana(candidate):

Okay, thanks. And how is the team culture?

Nadia(interviewer):

Friendly, quite direct. We argue about designs in documents more than in meetings.

Where this goes wrong

The last question is too general to get a real answer. A specific question works better: “Tell me about the last time the team disagreed on a design. How did it end?”

Minutes 57 to 60: the close

Nadia(interviewer):

Thanks, Mariana. I enjoyed that, especially the labelling story. Hannah will contact you this week about next steps. The next stage would be system design with Owen.

Mariana(candidate):

Thank you. And I will fix the “deployed” line before anyone else reads it.

After the call: Nadia’s notes

Candidate: Mariana Costa. Technical deep dive, 60 minutes.

Strong evidence

• Evaluation strategy. The best I have heard for this role. Fact-level error types, stratified sampling, expert agreement, an LLM judge checked against experts. Planned around four expert hours a week.
• Experiment design. Held-out test set, honest ranges on small classes. Randomise by adjuster: a trap we could have fallen into.
• Honesty. Corrected her own CV (“deployed” was the ML engineer’s work) at the first question.

Some evidence

• Python. Wrote the classifier as a module, pins model versions. No tests mentioned, no service of her own.
• Retrieval over long documents. Not built. Turned the question into a sound experiment.

No evidence

• APIs, FastAPI, Azure, CI/CD. Not discussed; “some Docker” on the CV.

Concern

• Ownership of production. Reached her edge on timeouts, retries and duplicate runs. Honest, did not bluff. But she has never owned a running service, and this role does, including on-call.

Recommendation: move forward to system design. Strong on the hardest part of this role (evaluation), weak on ownership of production. Owen: please spend most of the time on a pipeline for 9,000 packets a day, what fails, and what the adjuster sees then. If she can reason through it with some help, I would hire her at the intermediate level with a mentoring plan.

What happened next

Before the system design interview, Mariana asked Jonas to show her the nightly job at Basketry. He showed her its retries, a lock against two runs at once, and an alert nobody had ever received.

With Owen, she designed a queue of jobs keyed by packet ID and explained why a re-run must not create a second summary. He had to help her with capacity at the morning peak. His note said “learns fast; not senior yet”. Sofia’s values conversation went very well.

In the debrief, everyone agreed that Mariana would raise the quality of evaluation for the whole team. She would also need about six months of support to own a service alone. The team had one vacancy. It went to another finalist, whose claims expertise the team could not hire anywhere else. Marcus then asked his director for a second role at the intermediate level, at the lower end of the band. It would focus on the evaluation framework and metrics, working with the new hire on the summary. Mariana would pair with Nadia on the pipeline and shadow on-call for two rotations.

At the time of writing, the budget for that role is not decided. Hannah told Mariana this directly and gave her a date. Mariana keeps interviewing elsewhere, which is the right thing to do.

This ending is common. A candidate can be the best in the process at one part of the job and still not get the one role. The decision now depends on a budget, not on Mariana. That is part of luck in hiring. But her interview changed something: the team now wants a role that did not exist before.

What to take from this

• Depth in one area can move you forward. Mariana did not know retries, but her evaluation answers were specific enough that Nadia recommended her.
• Correct your own CV before the interviewer does. Because she said it first, it cost her almost nothing.
• At the edge, start from your own experience. She had depended on batch jobs for years. The stronger answer uses that, and then says where it stops.
• Ask questions that reveal the real work. Her questions showed her skill and taught her what the job is.

If your profile is like Mariana’s

Your evaluation skills are rare in AI engineering. The gap is ownership, and you can close much of it yourself.

1. Ship one of your models as a small service. Put it behind an API, write tests, put it in a container, and add a pipeline that runs the tests. Our API, Docker and CI/CD courses take you through each step.
2. Make a batch job fail on purpose. Call a model for 1,000 items and make 3% of the calls time out. Then add retries with backoff, results keyed by item ID so a re-run is safe, a list of failed items, and a daily count. In the README, write what happens in each failure.
3. Ask the engineer who deploys your work to show you how it runs: what fails, how they know, and who gets the alert.
4. Put your evaluation into the release process. A golden set that blocks a release when quality drops joins your strength to production. See AI evaluation gate.
5. Practise your evaluation story out loud, with the numbers, the decisions and the limits. See Explain decisions.
6. Read your CV verb by verb. For each “built”, “deployed” or “led”, ask what exactly you did, and what others did.