The backend engineer

Many AI Engineer candidates come from backend engineering. They ran services in production for years. Then their company asked them for a feature with an LLM (a large language model, which reads and writes text), and they liked the work.

This profile is strong where many AI candidates are weak: production. The usual gap is the other half of the job: how do you know that the output is correct? An LLM can write text that is fluent and wrong at the same time.

Daniel is this candidate. Marcus Chen, the hiring manager, also joins his call. We watch how Nadia finds the edge of what Daniel knows. We also watch Daniel when a senior person says something confident and wrong.

DM

The candidate

Daniel Mensah

Backend engineer, 7 years, moving into AI

• 7 years in backend
• Python and Go
• On call for 6 years
• Toronto

• Senior Software Engineer, Ledgerline (payments, 600 people), 2019 to now
• Built and ran an LLM-powered summariser for long support tickets, used by 120 support agents; average handling time down about 20%
• Designed rate limiting, retries, caching and a cost dashboard for all LLM calls; 85% agent satisfaction
• Owned the payout reconciliation services in Go; on call since 2020; led more than 30 incident reviews
• Side project: question answering (RAG) over Toronto’s open transit data, on GitHub
• Python, Go, FastAPI, PostgreSQL, Kafka, Kubernetes, AWS

What Nadia sees before the call

Nadia reads the CV next to the job description. The evidence for production engineering is strong. Daniel has about one year with LLMs in production, not the “2+ years” in the posting. But one year with real users can teach more than two years of demos.

One line gets a mark in her notes: “85% agent satisfaction”. Satisfaction is not correctness. Evaluation is the most urgent part of the role, so she plans to go deep here.

She wants to find out:

• Did Daniel design the reliability work himself, and does he know why each part works?
• How did he know that the summaries were correct, not only liked?
• How does he think about a claim summary at Quillfield’s volumes?

Terms on this page

• RAG (retrieval-augmented generation): the system finds the relevant parts of your documents and gives only those parts to the model.
• Token: a small piece of text, often part of a word. Providers charge per token.
• Context window: the largest amount of text, in tokens, that a model can read in one request.
• p95 latency: 95% of requests finish faster than this time.
• Eval set (or golden set): a fixed set of examples with answers that people checked. Every new version runs against it.
• LLM-as-judge: a second model that grades the output of the first model.
• Rate limit (and 429): the provider allows only so many requests or tokens per minute. Above that, it returns the error 429, “too many requests”.
• Exponential backoff: after a failed call, wait before the retry, and double the wait each time.
• OCR (optical character recognition): software that turns a scanned page into text, with some mistakes.
• Prompt caching: the provider stores a repeated start of a prompt, so later calls with the same start cost less.

The interview

Minutes 0 to 5: the opening

Nadia(interviewer):

Hi Daniel, thanks for making the time. I’m Nadia, I’m a staff engineer on Claims Intelligence. I built most of our intake extraction. And you know Marcus from last week.

Marcus(interviewer):

Hi Daniel. I’m only listening today. Just talk to Nadia.

Nadia(interviewer):

So, the plan. First I’d like to hear about something you built, and I’ll ask a lot of follow-up questions. That’s normal. Then a problem we have. Then your questions. Oh, and like the invite said, no AI assistants in this one. Paper notes are fine.

Daniel(candidate):

Okay, sounds good. Just so you know, I’ll probably ask about on-call at the end.

Minutes 5 to 25: the project, then deeper

Nadia(interviewer):

Pick a project you’re proud of and walk me through it.

Daniel(candidate):

Sure. So, at Ledgerline, merchant support handles things like failed payouts, disputes, account checks. The hard tickets get long, like 50, 100 messages. The merchant, a few agents, internal notes, bots. When a new agent picked up a ticket, they spent around eight minutes just catching up. So last year I built a summariser. It puts a short summary at the top of the ticket: what the merchant wants, what we did, what’s still open.

Nadia(interviewer):

Mm-hm. And how is it built?

Daniel(candidate):

A Python service, FastAPI, a queue and some workers, and it calls a hosted LLM. Honestly, most of my work was around the call, not the prompt. Rate limiting, retries, caching, a cost dashboard.

Nadia(interviewer):

How many people use it?

Daniel(candidate):

About 120 agents. I think around 4,000 summaries a day, it goes up and down. And the result: support ops gave it to three of their six teams for six weeks. Handling time went down about 20% on those teams. On the other teams, about 2%, so almost nothing.

What the interviewer is checking

Nadia listens for the decisions in the story. Each decision leads to her next question. Daniel’s result has a comparison group, so it is more believable than a before-and-after number. Nadia could still ask how support ops chose the three teams.

Nadia(interviewer):

Okay, thanks. Let’s start with timing. Is the summary generated when the agent opens the ticket?

Daniel(candidate):

In the first version, yes. And it was slow. The p95 was… about six seconds, I think. Agents hated it.

Nadia(interviewer):

So what did you change?

Daniel(candidate):

Now a new message puts an event on a queue. We wait two minutes, because messages come in groups, and then we generate. When the agent opens the ticket, they get the stored summary. If it’s missing, we make one on demand. Now about 90% of opens show a summary in under a second.

Nadia(interviewer):

And when the provider is slow, or it rate-limits you?

Daniel(candidate):

Hmm. Okay, a few pieces. There’s a token bucket, sized to our tokens-per-minute limit. It has two lanes. If an agent is waiting, that request goes first. Background work gets what’s left.

Nadia(interviewer):

Mm-hm. And retries?

Daniel(candidate):

Exponential backoff with jitter. Three retries at most, I think, yes, three. On a 429 we respect the Retry-After header. Oh, and there’s a circuit breaker. If too many calls fail, we stop calling for thirty seconds, and the page shows the last summary with “may be out of date”.

Nadia(interviewer):

Why the circuit breaker?

Daniel(candidate):

Because of an incident, actually. In the beginning we called the model inside the page load. The provider got slow, and the whole ticket page hung. My fault. So now the ticket page never waits for the summary.

Why this works

Daniel gives details, and he talks about a failure that he caused and fixed. A circuit breaker stops calls to a failing service for a short time, so the failure does not spread. Jitter adds a small random delay to each retry, so many clients do not all retry at the same moment.

Nadia(interviewer):

Long tickets, a hundred messages. Do you send the whole thread every time?

Daniel(candidate):

No, that was for cost. We send the previous summary plus the new messages, and the model updates it. On long threads it’s much cheaper.

Nadia(interviewer):

What happens if the previous summary had a mistake in it?

Daniel(candidate):

Yeah. Then it carries forward. We had one case: an agent reported a refund of 1,200 dollars, and it was really 120. It was wrong for, I think, six updates. So now we rebuild from the full thread every tenth update, and when a ticket is reopened.

Nadia(interviewer):

Why ten?

Daniel(candidate):

Honestly? It felt reasonable. It cost about 15% more. I didn’t measure the error rate before and after.

How the interviewer goes deeper

Each question comes from the last answer. Daniel saved money, and he paid for it with some correctness. But he did not measure the correctness side. That is how Nadia gets to evaluation.

Nadia(interviewer):

Let’s stay there. How did you know the summaries were good?

Daniel(candidate):

Every summary has a thumbs up and a thumbs down. We got around 85% thumbs up. And handling time went down 20%, so… they were useful.

Where this goes wrong

Thumbs and handling time show that agents like and use the summaries. They do not show that the summaries are correct. A fast, wrong summary also saves time, until it causes a wrong refund.

Nadia(interviewer):

How many summaries got a vote at all?

Daniel(candidate):

Hm. At the start, maybe one in five? Later, less. People vote in the first weeks, then they stop.

Nadia(interviewer):

And the thumbs down, what was in them?

Daniel(candidate):

I read maybe thirty. Mostly missing context, something the agent cared about wasn’t there. A couple of wrong amounts. And one I remember well. It said the merchant got the refund, but the agent had only promised a refund.

Nadia(interviewer):

That last one. Promised versus done. How often does that happen in the summaries with a thumbs up, or with no vote?

Daniel(candidate):

I… give me a second.

No. I don’t know. We never read the thumbs-up ones. And if an agent trusts the summary, they never see it’s wrong, so they never vote it down. The thumbs can’t tell me that. That’s… yeah, that’s a real gap.

The edge

This is the edge of what Daniel knows. He asks for a moment, says “I don’t know”, and explains why his metric cannot answer. He does not defend the 85%.

Nadia(interviewer):

That’s fine. Say you had two weeks to find out. What would you do?

Daniel(candidate):

Okay, let me think out loud. A sample, maybe 200 summaries, short and long threads. A senior agent reads the full thread and checks the summary line by line. Is each statement in the thread, or is it wrong? And then, separately, what’s missing.

Nadia(interviewer):

Okay. And then?

Daniel(candidate):

Then I have counts. Wrong facts, missing facts. And “promised versus done” as its own group, because that one costs money. I’d keep the 200 fixed and run every prompt or model change on them. Right now we just look at about twenty tickets by hand. Is that more or less how people do it?

Nadia(interviewer):

It’s close to what we do for intake. One more level. Two senior agents read the same thread. One says the summary misses something important, the other says it’s fine. What do you do?

Daniel(candidate):

Hmm. Give them both the same fifty, and see how often they agree? If they disagree a lot, the problem is what “important” means, not the model. There’s a statistic for this, for agreement… I never used it.

Nadia(interviewer):

Cohen’s kappa, for two people. But you have the important part. If the people don’t agree, you can’t measure the model. Okay, let’s move on to a problem of ours.

A stronger answer

Daniel got to a good answer, but only after many questions. With some preparation, he can give it to the first question:

“Honestly, only in part. The thumbs show that agents liked it, and handling time shows that they used it. Neither tells me how often a summary is wrong. In the thumbs-down ones I found missing context, wrong amounts, and a promised refund described as done. That last kind worries me most, because an agent who trusts the summary never sees it. If I built it again, I’d start with 200 threads labelled by two senior agents, and count wrong, missing and unsupported statements on every change.”

This answer does not claim any work that he did not do.

Minutes 25 to 48: the Quillfield problem

Nadia(interviewer):

Here’s a real problem on our side. A claim packet comes in: 20 to 60 pages, scanned forms, the adjuster’s notes, emails, photos, a contractor’s estimate. Adjusters tell us they spend about 40 minutes reading a packet before they can act. We want to give them a one-page draft summary: the key facts, the open questions, and what’s missing. How would you approach it?

Daniel(candidate):

Okay. Before I design anything, some questions. Who reads it, and when? Right after intake, or when the adjuster picks up the claim?

Nadia(interviewer):

Adjusters, usually within a day of intake.

Daniel(candidate):

And “key fact”. Is there a list somewhere, or is it in the adjusters’ heads? And what does a bad mistake look like?

Nadia(interviewer):

The claims experts have an informal checklist per claim type. A bad mistake is a summary that leaves out a policy exclusion or a deadline. Or it says something as a fact, and the packet doesn’t say it. The summary never decides anything; the adjuster confirms.

Daniel(candidate):

Okay. Last one: does intake already give you text per page?

Nadia(interviewer):

Yes. Intake runs OCR on every page and keeps the text with page numbers. Each of the 32 fields has its page too.

Daniel(candidate):

Oh, good. Then I wouldn’t start from the raw packet. So, a rough sketch. A job that runs right after intake extraction. The input is the 32 fields with their pages, plus the page text. The output is structured, not free text. Facts, each with its page. Open questions. A missing list.

Nadia(interviewer):

Mm-hm.

Daniel(candidate):

The missing list, I’d mostly do in code. Compare the checklist for this claim type with the 32 fields. The model can add to it, but the base is normal code, easy to test. And before we store anything, we check every citation. Does the page exist, and does the quoted text appear on it? If not, drop the fact, or flag it.

Nadia(interviewer):

“Does the quoted text appear on it.” Exact match, on OCR text?

Daniel(candidate):

No, no, that would fail all the time. Fuzzy, I think. Normalise spaces and case, then a similarity score over a part of the page, with a threshold we tune. But honestly, I don’t know OCR well. I’d want to look at a hundred of your worst scans first.

Why this works

Daniel asks who reads the summary and what a mistake costs before he designs. He reuses what the team already has, and he moves the missing list into normal code. Checking each citation is a backend habit: check data before you trust it. When Nadia tests the detail, he says where his knowledge stops.

Nadia(interviewer):

Why at intake, and not when the adjuster opens the claim?

Daniel(candidate):

Same lesson as the tickets. Nobody wants to wait thirty seconds, and usually you have hours. 9,000 a day is… about six a minute, on average. But I guess you get spikes?

Nadia(interviewer):

After a big storm, yes. A hail event last year gave us about four times the normal volume, for ten days. And what does it cost, roughly?

Daniel(candidate):

For the spikes, a queue with priorities. The workers are sized to the rate limit in your contract, not to the peak. In a storm the summaries come later, but nothing breaks. When new documents arrive, you regenerate, keyed on a hash of the packet and the prompt version, so… sorry, I lost the second part of your question.

Nadia(interviewer):

Cost. Roughly.

Daniel(candidate):

Right, cost. Let me think out loud. Forty pages, maybe 500 to 700 tokens a page after OCR. So let’s say 25,000 tokens in, and one or two thousand out. Times 9,000… about 225 million input tokens a day. I don’t know your price per token, but I’d start from that number.

Nadia(interviewer):

Okay.

Daniel(candidate):

Oh, and nobody waits for the overnight work or the storm backlog. So I’d check the provider’s batch API. It’s often about half the price, but results can take up to a day. And I’d confirm that batch keeps Canadian data in Canada, because some batch offers run only in global regions.

Marcus(interviewer):

Can I add something? I’ve been listening, and honestly I think this is getting over-engineered. With million-token context windows, retrieval is basically dead. We’ll just put the whole packet and the full policy in the prompt.

Daniel(candidate):

Hm. For the packet, I mostly agree. 25,000 tokens fits easily, that’s what I assumed. Where I see it differently is “retrieval is dead”, and the policy part. A few reasons… let me see if I can get them in order.

Cost, first. If the policy with its endorsements is another hundred pages, all 9,000 calls a day pay for it.

Marcus(interviewer):

Caching makes the policy cheap, though?

Daniel(candidate):

For the base wording, yes. That part repeats, so prompt caching helps. But the endorsements are different for every policyholder, and so is the version that applied on the date of loss. That part doesn’t repeat, so you pay for it every time. And that’s actually my next point.

Marcus(interviewer):

Go on.

Daniel(candidate):

Last week you told me the policy assistant sometimes misses an endorsement. I don’t think that’s a size problem. The problem is knowing which version and which endorsements applied on the date of loss. That’s a database lookup, not a prompt.

Then quality. Models don’t use every part of a very long input equally well. It’s much better than before, but it depends on the model and the task, so I’d test it on our documents. And privacy. We shouldn’t send more personal information to a model than the task needs.

So I’m not saying “always retrieve” either. For the policy, I’d guess a version lookup, then retrieval. But it’s a guess. We could find out in a week. Take 50 packets the claims experts know well, run them both ways, and compare missing facts, wrong facts, cost and time.

Disagreeing well

Marcus is the hiring manager, and he believes what he says; it is not a test. Daniel first agrees with the part that is right. He says “I see it differently”, not “you’re wrong”. When Marcus pushes back, Daniel answers with a fact about this team’s documents. He ends with an experiment, so the data decides, not the person with the higher title. An endorsement is a change to a standard policy, for example extra cover for a basement. See our advice on disagreeing with reasons and on claims that a tool is “dead”.

Nadia(interviewer):

I’d support that. We never measured it. We had the same discussion about intake, and we decided by opinion. Marcus, I can set it up.

Marcus(interviewer):

Okay. Measure it.

Nadia(interviewer):

Let’s go back to quality, because that’s where we stopped on your ticket project. How would you know these summaries are good enough to ship?

Daniel(candidate):

This time I’d do it before, not after. A labelled set, where the claims experts write down the key facts. Then we score missing facts, wrong facts and unsupported statements. How much time do the experts have?

Nadia(interviewer):

About four hours a week, between the two of them. Most of it goes to the weekly intake sample.

Daniel(candidate):

Wait, four hours in total? Okay. Then their time is the scarcest thing in the whole design. I’d use them only where nobody else can help. The missing list and the citation check are code. For “is this statement supported by the page it cites”, a second model as a judge. And the experts label a smaller set, and we use that to check the judge.

Nadia(interviewer):

How do you know the judge is good enough?

Daniel(candidate):

Compare it with the experts on the same statements. Maybe a hundred to start. And look where they disagree. Mostly where the judge says “supported” and the expert says “no”, because those are the dangerous ones.

Nadia(interviewer):

If only 5 of the 100 are unsupported, what do you learn about that direction?

Daniel(candidate):

Not much. Five cases, the judge catches four, and… that tells me almost nothing. Hm. So I need more bad statements in the sample than in real life. Pick the ones that are probably unsupported, like where the citation check almost failed. Or make them: take good summaries, change an amount or a date, and see if the judge catches it. For the exact numbers, I’d ask someone who knows statistics better than me.

Why this works

Daniel uses the lesson from the first half, and he works with a real limit. A judge model is useful only after you compare it with people. A random sample has too few errors for that test, so he adds more errors, some of them on purpose.

Nadia then asks about health information in injury claims. Daniel answers from his payments work:

• Keep full prompts out of general logs.
• Keep Canadian data in Canada, and test it.
• Include health details only when the adjuster needs them.

He says that he does not know the privacy laws well enough to say more. These are PIPEDA, the federal law for private companies, and provincial laws such as Quebec’s.

For the release, Daniel proposes a silent run first, where adjusters do not see the output. Then one pilot insurer, with feedback reasons (“wrong fact”, “missing fact”) instead of a thumbs up.

Minutes 48 to 57: Daniel's questions

Daniel(candidate):

So, on-call, like I said at the start. What wakes people up at night for an LLM feature here? Is there a rotation?

Nadia(interviewer):

Yes. One week in six, after you shadow two rotations. It’s not in the posting, and it should be. Mostly it’s the provider: latency spikes, rate limits during storms. Sometimes a model version change breaks the extraction of one field.

Daniel(candidate):

How do you notice that one? It doesn’t throw any errors, right?

Nadia(interviewer):

Slowly, right now. The weekly sample of 200 finds it. That’s one reason this role exists.

Why this works

The on-call question finds a fact that the posting leaves out. Daniel’s follow-up connects on-call to quality failures that make no error. Nadia’s answer shows that there is no shared evaluation framework yet. See an interview is a conversation in both directions.

Daniel(candidate):

And labelling. Is there a plan to get more expert time, or are the four hours fixed?

Nadia(interviewer):

Honestly, plan around four hours for the first six months. Marcus is asking for more, but he can’t promise it.

Minutes 57 to 60: the close

Nadia(interviewer):

Okay, we’re out of time. Next is usually system design with Owen Fraser, our principal engineer. Hannah will contact you this week.

Daniel(candidate):

Thanks. The evaluation part was uncomfortable, but it was useful.

Nadia(interviewer):

Good. That’s the part I’d prepare for next. Thanks, Daniel.

After the call: Nadia’s notes

Nadia writes her notes against the job description:

Production engineering and Python services: strong evidence. He designed and ran it, and explained a failure that he caused and fixed.

LLMs in production: some evidence. One feature, one year, 120 real users. Deep, but short.

RAG and retrieval: some evidence. Side project only, but his reasoning about long context was sound.

Evaluation strategy: concern. His project measured satisfaction, not correctness. At the edge he reasoned well: a labelled set, error categories, a judge checked against people. He could not yet lead our evaluation work alone.

Privacy: strong evidence for the engineering controls. No knowledge of the law, and he said so.

Communication: strong evidence. He disagreed with Marcus politely, with reasons.

Recommendation: yes, move to system design. Owen, please focus part of it on evaluation. Hannah, please tell him.

What happened next

Hannah told Daniel the focus. He spent a week of evenings adding an eval set of 80 questions to his transit side project. In the system design interview with Owen, his evaluation plan was one of the strongest parts of his answer. His values conversation with Sofia Alvarez also went well.

There was one vacancy. In the debrief, the team chose another finalist, who had years of claims experience that nobody on the team had. Marcus called Daniel himself. He shared Nadia’s and Owen’s feedback, and he asked if he could contact Daniel when the next senior role opens. Daniel did nothing wrong. Sometimes another person fits one team’s gap better, and that is part of luck in hiring.

Separately, Marcus asked an engineer to run the long-context experiment. For packets, long context was fine. For policies, the result was mixed: the version lookup fixed most of the missed endorsements, and retrieval helped less than Daniel guessed.

What to take from this

• Production experience is AI engineering experience. Retries, rate limits, queues and cost were half of Daniel’s interview.
• “Liked” and “used” are not “correct”. For every LLM project, know how often it is wrong and in which way, or say how you would find out.
• The edge matters most. Daniel said “I don’t know” and then reasoned in the open. That earned more trust than a smooth answer.
• Disagree with reasons from the problem in front of you, and propose a test. See our advice on disagreeing.
• Your questions are evidence too. Daniel’s questions found two facts that the posting does not say.

If your profile is like Daniel’s

Your production experience is a head start (see Your experience is a head start). Close the evaluation gap before the interview, not during it.

1. Add an eval set to a project that you already have. Ask someone who knows the work to label 100 to 200 real examples, count each kind of error, and rerun the set on every change. Learn the basics of agreement between labellers and of checking a judge model against people. The ideas in train, validation and test sets apply here too.
2. Learn to talk about kinds of error. “Wrong”, “missing” and “unsupported” are not the same, and their cost is different. Classification metrics and the precision and recall trade-off give you the words.
3. Practise your project story out loud, with numbers and one failure that you caused. Our lesson Explain decisions shows how.
4. Stay focused. The posting lists three agent frameworks, and nobody asked Daniel about any of them. Depth in evaluation helps you more. See keep learning, but stay focused.