The domain expert

Many people who move into AI engineering come from the work that the software is for. They were nurses, accountants or claims adjusters. Every day they saw the same slow task. Then one evening they started to learn Python to fix it.

They often apply with a doubt: “The others have degrees and years of code. What do I have?” The answer is: a lot. They know which mistakes are dangerous. They know what users really do, and which rules apply. Our lesson Your experience is a head start explains why.

But domain knowledge does not replace engineering, and a serious interview checks both. Let’s follow Leila through the technical deep dive. Watch where her experience leads the conversation, and where her knowledge stops.

LH

The candidate

Leila Haddad

Claims team lead, 11 years in property and casualty insurance, learned to build

• 11 years in P&C claims
• Adjuster 7 years, team lead 4 years
• Python, 2 years (evenings)
• Mississauga, Ontario

• Team Lead, Property Claims, Northgate Mutual (2022 to now). Lead 9 adjusters who handle home and small commercial property claims in Ontario.
• Built an AI-powered email triage tool for the claims team: classifies incoming claim emails into 7 categories and drafts acknowledgement letters. Used daily by 9 adjusters, about 140 emails a day.
• Checked the tool by hand on 300 real emails. Found and fixed a confusion between subrogation emails and new claims.
• Property Claims Adjuster, Northgate Mutual (2015 to 2022). Water, fire, theft and wind losses; large-loss support.
• Online learning path in Python, APIs and machine learning, with certificates (2024 to 2026).

What Nadia sees before the call

Nadia reads the CV next to the job description. Leila does not meet the main requirements. She has no 4 years of software work, no RAG, no agent frameworks and no fine-tuning. If Nadia read the CV as a checklist, this call would not happen.

But the team’s next problem is claim summaries, and Leila has read thousands of claim packets. She also built a real tool that people use every day. Marcus already checked her claims knowledge, so Nadia’s question is different. Can Leila build? And can she grow into the engineering part of the role? Nadia writes down three things to find out:

• How much of the email tool did Leila build herself? What happens when it fails?
• “Fixed a confusion” is a good line. How did she know that the fix worked?
• On the summary problem, does her domain knowledge turn into engineering decisions? And where does her technical knowledge stop?

Terms on this page

• P&C insurance (property and casualty): insurance for things, such as homes and cars, and for damage that you cause to other people.
• Insured (as a noun): the person or company that the policy covers.
• Exclusion: a part of the policy that says what it does not cover.
• Endorsement: a change to a standard policy, for example to add a coverage that the base wording excludes.
• Subrogation: after an insurer pays its insured, it tries to recover the money from the person who caused the loss, or from that person’s insurer.
• Proof of loss: a signed statement from the insured about what was lost and how much it costs. The policy or the law sets when it is due.
• Reserve: the money that an insurer sets aside for the expected cost of a claim.
• LLM (large language model): a model that reads and writes text, like the models behind chat assistants.
• RAG (retrieval-augmented generation): the system first finds the relevant parts of the documents, then gives only those parts to the model.
• Chunk: one of those parts. Embedding: a list of numbers that represents the meaning of a text, so that the system can find texts with a similar meaning.
• Eval set (or golden set): real examples with answers that experts checked. You run the system on it to measure quality.
• PII (personally identifiable information): information that identifies a person. Health information is one of the most sensitive kinds.

The interview

Minutes 0 to 5: the opening

Nadia(interviewer):

Hi Leila, thanks for making the time. I’m Nadia, I’m a staff engineer on Claims Intelligence. I built most of our intake extraction.

Leila(candidate):

Hi Nadia, nice to meet you.

Nadia(interviewer):

So, the plan. First you tell me about something you built, and I’ll ask a lot of questions about it. That’s normal. Then I give you a problem we have, and at the end, your questions. Oh, and as the invite said, no AI assistants in this round. Paper notes are fine. Okay?

Leila(candidate):

Yes, okay. I’ll be honest, I’m a bit nervous. It’s my first interview for an engineering job. In claims, I’m usually the one who asks the questions.

Nadia(interviewer):

Then you know how it works. I’m not trying to trick you. I just want to understand what you did. Let’s start.

Minutes 5 to 25: the project, then deeper

Nadia(interviewer):

So, pick a project you’re proud of and walk me through it.

Leila(candidate):

Okay. So, at Northgate my team has a shared claims inbox. About 140 emails a day. New losses, updates, invoices, letters from other insurers, from lawyers. Someone had to read every one, file it to the right claim and send an acknowledgement. That was about two hours of an adjuster’s time, every day.

So I built a Python script. Every ten minutes it reads the new emails through the Microsoft Graph API. It sends the text to the LLM that our IT approved, and it asks for one of seven categories, and the claim number if there is one. Then it drafts the acknowledgement letter from a template for that category.

Nadia(interviewer):

And the letter goes out automatically?

Leila(candidate):

No, no. Nothing goes out by itself. The adjuster sends the draft or changes it. IT approved it, and the team has used it every day for… eight months now, yes.

Nadia(interviewer):

Nice. On your CV you wrote that you “fixed a confusion between subrogation emails and new claims”. What happened there?

Leila(candidate):

Right. So, subrogation goes both ways. Other insurers send us demands when they think our insured caused their loss. The letter says something like “our insured had water damage on March 3rd, caused by a leak from your insured’s unit above”.

Nadia(interviewer):

And to the model, that looks like a new claim.

Leila(candidate):

Exactly. It reads just like a new claim. When I checked 300 emails by hand, 22 were subrogation, and the tool called 13 of them “new claim”. That’s a bad mistake. Someone opens a duplicate claim. And the demand often has a date for a response, and it sits in the wrong queue.

Nadia(interviewer):

How did you fix it?

Leila(candidate):

Two things. First, I explained the difference in the prompt, with three examples. Names removed, of course. Second, I added a rule in the code, before the model. If the sender’s domain is on a list of insurers and recovery firms, and the text says “your insured”, it goes straight to subrogation.

Nadia(interviewer):

A simple rule.

Leila(candidate):

Yes, it’s boring. But I know exactly when it fires.

Why this works

This is the strongest part of her project, and it comes from her domain knowledge. She knows that only the other side writes “your insured”. She also knows why this mistake costs so much. Someone who only looked at overall accuracy might never notice that this one category mattered most.

Nadia(interviewer):

And after the fix, how did you know it worked?

Leila(candidate):

I ran it again on the 300. And the subrogation ones were all correct, 22 out of 22.

Nadia(interviewer):

The same 300 that you took the examples from?

Leila(candidate):

Yes… oh. Okay, I see. I took my three examples from those emails, so of course it got them right.

Nadia(interviewer):

Mm-hm. Did you test it on other emails later?

Leila(candidate):

Yes, a month later, because my manager asked. I checked 100 new emails. Nine were subrogation, I think, and it got eight. The one it missed came from a lawyer, not an insurer, so my domain list didn’t catch it. And I never fixed that properly.

How Nadia goes deeper

Nadia’s question tests a basic idea of measurement. When you change a prompt based on some examples, a score on those same examples looks better than the truth. You need a separate test set that you did not look at while you made changes (see Train, validation and test sets). Leila did not use the term. But she saw the problem at once, and she told the truth about it.

Nadia(interviewer):

Thanks, that’s honest. Now, what happens when the model call fails?

Leila(candidate):

You mean when it gives the wrong category?

Nadia(interviewer):

No, sorry, I wasn’t clear. I mean when the call itself fails. A timeout, an error from the API.

Leila(candidate):

Ah, okay. Then it tries three times, with a wait in between. After that, the email stays “not classified”, and the team handles it the old way. And if more than 20 fail in one day, I get an email.

Nadia(interviewer):

Okay. And if the script crashes in the middle of a batch, can the same email get processed twice?

Leila(candidate):

Twice… I think that’s fine? I mark each email in Outlook after I save the draft. Hmm. Wait. Actually, no. If it crashes after the draft and before the mark, the next run makes a second draft. That happened once. Someone deleted the extra one. I never really fixed it.

Where this goes wrong

The gap is a crash between two steps. In a tool for 9 people, someone deletes the extra draft. At 9,000 packets a day, the same gap can create duplicates every day. Nadia does not need the word “idempotent” (safe to run twice). She wants to see whether Leila finds the gap, and she does.

Nadia(interviewer):

Who reviews your code? And do you have tests?

Leila(candidate):

Nobody, really. IT read it once before they approved it, mostly for security. Tests… I added about fifteen pytest tests three months ago, mostly for the subrogation rule and the templates. Before that, I tested by running it and looking at the result.

Nadia(interviewer):

Okay.

Leila(candidate):

And I should say, I’ve never worked in a codebase with other people.

Nadia(interviewer):

That’s useful to know, thanks. Okay, let’s move to our problem.

Minutes 25 to 48: the Quillfield problem

Nadia(interviewer):

So, this is a real problem on our side. A claim packet comes in. It’s 20 to 60 pages: scanned forms, the adjuster’s notes, emails, photos, a contractor’s estimate. Adjusters tell us they spend about 40 minutes reading a packet before they can act.

Leila(candidate):

Yes, forty minutes sounds right.

Nadia(interviewer):

We want to give them a one-page draft summary. The key facts, the open questions, and what’s missing. How would you approach it?

Leila(candidate):

Okay. Before I design anything… whose forty minutes? The adjuster who just got the file, or someone else? And is it home and auto together? Because a key fact in a water loss is not a key fact in a car accident with an injury.

Nadia(interviewer):

Home claims first, with one pilot insurer. The way I see it: the assigned adjuster opens the claim, reads the summary top to bottom, and it saves most of the 40 minutes.

Leila(candidate):

Hmm. Can I disagree a little with that?

Nadia(interviewer):

Sure, go on.

Leila(candidate):

Adjusters won’t read it top to bottom. And it won’t replace the packet, not for anything that affects money. We’re trained to go to the source, and we’re audited on it. If I put a payment on a file, my supervisor asks “where does it say that?” And “the summary said so” is not an answer.

Nadia(interviewer):

Okay. So how would they use it?

Leila(candidate):

Like a map. A good adjuster jumps to the things that could stop the claim from being covered, and to what’s missing. Then they open those pages. So it’s not 40 minutes to zero. It’s more like… 40 minutes of searching becomes maybe 15 minutes of checking.

Nadia(interviewer):

Mm-hm.

Leila(candidate):

And honestly, I think the bigger use is the second reader. Files move between people all the time. Someone goes on vacation, a claim moves to the large-loss team, a supervisor reviews the reserve. That person reads the file with no background at all. That’s where a summary saves the most.

Nadia(interviewer):

Hm. Okay, that’s a fair correction. Our claims experts said something close to that about file transfers, and I didn’t connect it to the design. If it’s a map, then the layout and the links to pages matter more than nice sentences. Let me write that down.

Disagreeing well

Nadia assumed that adjusters would read the summary top to bottom and save most of the 40 minutes. Leila corrects her with what she has seen, not with an opinion. Then she says what it means for the design. She also adds a new idea, the second reader. Nadia accepts it, and the design gets better. See Interviewers are not always right.

Nadia(interviewer):

So, with that in mind, what goes on the page?

Leila(candidate):

Not a free-text story. I’d do three blocks. First, the facts, and each fact with its page. You already extract 32 fields, right? So I’d start from those.

Nadia(interviewer):

Right.

Leila(candidate):

Second, what’s missing. A complete water loss file has the plumber’s report on the cause, photos, the contents list, the proof of loss. That’s almost a checklist for each claim type. It’s not even AI.

Nadia(interviewer):

And the third?

Leila(candidate):

The third is the one that matters most for me: what contradicts. For example, the first notice says the leak started yesterday. But the contractor’s estimate mentions mould and a rotted subfloor. That’s not a one-day leak. Most home policies cover sudden and accidental water damage, and exclude a leak that continues over time. So that one detail can change coverage.

Nadia(interviewer):

Would the summary say that? Like, “probably excluded”?

Leila(candidate):

No, never. Coverage is the adjuster’s decision. It should say “Possible gradual leak, see pages 31 and 44. Check the exclusion.” A question, not an answer.

And to be honest, I’m more scared of what’s left out than of what’s wrong. A wrong date, the adjuster will see it. But if it leaves out an exclusion, or an earlier claim on the same basement, or a deadline like the proof of loss… nobody looks for something they don’t know is there.

Why this is the strongest answer of the day

Leila turns domain knowledge into design decisions:

• Each fact has its page.
• What is missing is a checklist, because a list is easier to check than free text.
• Contradictions come first, because they point to coverage questions.
• The summary never states a coverage conclusion.

And she names the worst error: something important left out.

Nadia(interviewer):

How would you measure “left out”?

Leila(candidate):

Hmm. Let me think. I’d ask claims people to take real packets and write down, for each one, the things that must be on the page. Not a perfect summary, only the must-haves.

Nadia(interviewer):

Mm-hm.

Leila(candidate):

Then for each generated summary we count. Which must-haves are there, which are missing, and which statements no page supports. Missing counts worst. How many packets… I don’t know. Maybe a hundred? Spread over the claim types.

Nadia(interviewer):

That’s close to what we’d do. Okay, now the technical part. For the exclusion question, the model needs the policy too. That’s 40 pages, plus endorsements. In our policy assistant we use RAG for that. How would you give the model the right clauses?

Leila(candidate):

I’ve only done RAG in a tutorial, so… But I’d split the policy by clause, not every 500 words. Because an exclusion cut in half is dangerous. “We do not cover water damage” is in one piece, and “unless caused by a sudden and accidental escape of water” is in the next one. If the model only gets the first piece, it says the opposite of the policy.

Nadia(interviewer):

Good. So here’s our real problem. In the pilot, the assistant sometimes cites the base policy and misses the endorsement that changed the clause. How would you fix that?

Leila(candidate):

Oh, I know this one from the claims side. Sewer backup is the typical case. The base wording excludes it, and the insured buys it back with an endorsement, often with its own limit. If the system only reads the base policy, it says “excluded”, and the adjuster denies a claim that’s covered.

Nadia(interviewer):

Right. And technically?

Leila(candidate):

Technically… I’m guessing now. Could you attach each endorsement to the clause it changes? Like a label on the piece?

Nadia(interviewer):

Maybe. But how would the search find the endorsement? What makes a piece “match” the question?

Leila(candidate):

I think the text is turned into numbers, embeddings, and it finds pieces with close numbers? After that, honestly, I don’t know how it ranks them, or why it would prefer the base clause. That’s where my knowledge stops. Before I guess more, I’d look at what it actually retrieves for a few of the failing questions.

Nadia(interviewer):

That’s the right first step. Let’s stop there. Last topic: some of these packets are injury claims.

Leila(candidate):

Yes, I wanted to ask about that. Medical reports, sometimes mental health history. In my team, only the adjuster on the file and their supervisor see those. If more people can see the summary than the file, a diagnosis on page one is a privacy problem.

Nadia(interviewer):

So what would you do?

Leila(candidate):

The summary follows the same access rules as the documents. And it says “medical records received, pages 12 to 19”, without the details.

A stronger answer at the edge

Leila did well at the edge of her knowledge. She said what she knew, said where it stopped, and suggested a first step. But she did not need more retrieval knowledge to go further. She could reason from what an adjuster does:

“Hmm, what do I do as an adjuster? I pull the declarations page first, because it lists every endorsement on the policy for that term. Can the system do the same? Look up the declarations for the date of loss, and always put those endorsements next to the base clauses. Then search is only needed for the base wording. I’d test it on the pilot questions where the assistant missed an endorsement.”

This answer replaces a search problem with a lookup that always gives the same result. An experienced engineer would take it seriously.

Nadia then asked about volume. At 9,000 packets a day, should the summary be made at intake, or when an adjuster opens the claim? Leila reasoned from the work: documents keep arriving, so she would make a new summary when new documents arrive. She could not estimate the cost, and she said so.

Then Nadia gives the last part of the call to Leila.

Minutes 48 to 57: Leila's questions

Leila(candidate):

I have a few. First: who decides what a correct summary is? The two claims experts, the pilot insurer’s adjusters, or engineering? Because I think those three won’t agree.

Nadia(interviewer):

Honest answer? Right now, nobody, not formally. We don’t have a shared evaluation framework yet. That’s the “help define our evaluation strategy” line in the posting, and it’s urgent. And our two experts have about four hours a week between them. In total. Most of it goes to the weekly intake sample.

Leila(candidate):

Four hours, for both of them? Okay. Then their time is the most expensive thing in the project, and I’d protect it. Second: who would review my code? I’ve never had a review.

Nadia(interviewer):

Every change gets a review from another engineer. Mine too. Pairing for the first months is a question for Marcus, so ask him directly.

Leila(candidate):

Okay, I will. And when an adjuster at the pilot thinks the summary is wrong, where does that go today?

Nadia(interviewer):

Hmm. Honestly, today it’s an email to the product manager. There’s no button in the screen yet.

Leila(candidate):

Okay. And, um, the last one. Do you think I’m technical enough for this?

Nadia(interviewer):

I can’t give you a decision here. What I can say is this: I’ve learned where your strengths are and where your gaps are, and I’ll write both down honestly.

The good questions

Leila’s first questions are about the work, and each answer tells her something real:

• Evaluation is not built yet.
• Expert time is very small.
• User feedback has no channel yet.

This is the interview as a conversation in both directions.

The weaker question

“Do you think I’m technical enough?” asks the interviewer for comfort. Nadia cannot answer it, so Leila learns nothing from it. A better question gets useful information about the role: “If I joined, what would you want me to do on my own after six months?”

Minutes 57 to 60: the close

Nadia(interviewer):

Thanks, Leila, I enjoyed this. Next is a system design interview with Owen, and a conversation with Sofia, who runs claims operations. Hannah will contact you this week.

Leila(candidate):

Thank you. And thanks for the question about the same 300 emails. I’m going to fix that on Monday.

After the call: Nadia’s notes

Nadia writes her notes the same afternoon, against the job description.

Leila Haddad, technical deep dive

• Domain and problem framing: strong evidence. Best framing of the summary problem so far: facts with pages, a checklist for what is missing, contradictions, “never state coverage”. She corrected my assumption about how adjusters use summaries, and she was right.
• Evaluation: some evidence. Measured her tool by hand and found the expensive error herself. Tuned and scored on the same set, and did not notice until I asked. Right instinct: omissions first.
• Python and production: some evidence. A script in daily use for 8 months, with retries and alerts. No code review ever; tests are recent; a known duplicate bug that she did not fix.
• RAG and retrieval: no evidence beyond the basics. Knows RAG from a tutorial; split by clause for a good reason. Did not know how ranking works and said so clearly.
• Scale and operations: no evidence. Reasoned well from the work. Could not estimate cost.
• Privacy: strong evidence. Raised health information herself, with the right rule: the summary follows the access rules of the documents.

Recommendation: yes, with a plan. She does not meet “4+ years” or “2+ years with LLMs in production”. But she brings what is hardest for us to hire, and she is honest at her edge. Pair her with a senior engineer for three months, and put her on the summary evaluation from week one. We can teach her gaps. We cannot teach her judgment about claims.

What happened next

Owen’s system design interview was mixed. Leila designed a sensible flow, but she struggled with queues, batching and cost at 9,000 packets a day. Sofia’s values conversation was a strong yes. Sofia is a former adjuster, and she wrote: “She thinks about the insured on the other end of every file. And she will tell engineers no.”

In the debrief, Owen was worried. He thought that other engineers would have to do her infrastructure work for a year. Nadia answered that the team already had strong infrastructure engineers. But it had nobody who knew claims well enough to decide, with the claims experts, what the summary must contain. There was one vacancy and several strong finalists, and the team chose Leila. Marcus made an offer in the lower part of the band: CAD 145,000 plus bonus and equity. The plan: three months of pairing with a senior engineer, on-call only after shadowing two rotations, and a review at six months.

She asked for CAD 160,000, because she would own the must-have lists and the summary requirements with the claims experts. She also asked for the pairing plan in her offer letter. Marcus answered with CAD 152,000 and the plan in writing, and she accepted.

This is one realistic ending, not a promise. Another company could have said no because of the years on the posting. Luck is part of it.

What to take from this

• Domain knowledge wins when it becomes design. Leila did not only say “I know claims”. She said what must be on the page, what must never be on it, and why. Each point changed the system.
• You may correct the interviewer about your own field. Use what you have seen, not what you believe, and say what it means for the design.
• Your project will be checked like real engineering. Prepare to explain how you measured it, what happens when it fails, and who reviews the code.
• Ask about the work. Leila’s questions showed her three real problems: no evaluation framework yet, very little expert time, and no channel for feedback.
• Knowing where you stop is not a failure. Leila said “That’s where my knowledge stops” and suggested a first step. Because she was clear about this, Nadia could trust everything else she said.

If your profile is like Leila’s

1. Build one tool for your real work, and measure it. Check it on real cases, and count the errors by type, not only in total. Find the error that costs the most in your field.
2. Keep a separate test set. Do not take prompt examples from the cases that you score on. Run the same checked cases after every change to the prompt or the model. See Classification metrics.
3. Close the engineering gaps that interviews check first. Write tests (Focused tests), handle failures (Exceptions and logging), and get your code reviewed (Read and review diffs). If nobody at work can review it, publish a copy without private data and ask a developer in a community.
4. Learn one level deeper than your project. If you called an LLM API, learn how retrieval finds and ranks chunks, and how to measure it. Then you can say clearly where you stop.
5. Turn your domain stories into design decisions. For each story, say the mistake, why it is expensive, and what the system should do about it. Your experience is a head start and Domain knowledge in practice help you write them down. Explain decisions helps you say them aloud.