The self-taught builder

Many people who apply for AI engineering roles have never had a software job. They work in a warehouse, a shop or a hospital, and they learn in the evenings. Some of them build something real. They often apply for roles that ask for “4+ years”.

Yusuf is one of them. He did not apply through the job board. Two months before this interview, he gave a 20-minute talk at a local applied AI meetup. The title was “What broke when I built a bylaw search for my neighbourhood”. Joel Okafor, an engineer on Quillfield’s Claims Intelligence team, was in the room. He read the README on the train home. A week later, he offered to refer Yusuf. Yusuf shared what he built, and the right person saw it. That is luck that you make larger yourself.

YD

The candidate

Yusuf Demir

Warehouse shift lead, self-taught, builder of Bylaw Helper

• 26
• No professional software job
• 4 years as a warehouse shift lead
• Evening study, no degree
• Referred after a meetup talk

• Bylaw Helper (2025 to now, open source): question answering over my city’s bylaws and council minutes, about 3,800 public documents. Hybrid search (keywords and embeddings); every answer cites the clause.
• Evaluation set of 150 questions, written with two volunteers from my neighbourhood association. Recall@5 0.88; 80% of answers fully correct.
• Found and fixed a chunking problem with tables in PDFs (recall@5 on table questions from 0.39 to 0.83).
• Deployed for the neighbourhood association: about 60 users, 400 questions a month, about CAD 11 a month to run.
• Shift lead, regional distribution centre (2022 to now): lead a night team of 12; wrote Python scripts to clean the daily pick-error report.

What Nadia sees before the call

Nadia reads the CV next to the job description. The first requirement is 4+ years of professional software engineering. Yusuf has none, and his CV does not pretend otherwise.

The rest of the CV has specific numbers: 150 questions, 0.39 to 0.83, CAD 11 a month. A specific number is either real or easy to break with one question. Joel’s referral note says only, “Ask him about the tables.”

She writes down three things to find out:

• Did he build it himself, and does he know why it works? She will ask about details that only the builder knows.
• How deep does it go? Retrieval, evaluation and failure cases are the hardest parts of this role.
• What has he never had to do? Code review, on-call, scale, personal data. This is what the “4+ years” line is really about.

Terms on this page

• RAG (retrieval-augmented generation): the system first searches for relevant text, then gives that text to a language model to write the answer.
• Embedding: a list of numbers that represents the meaning of a text. Texts with similar meanings get similar numbers.
• Chunk: one of the small pieces that you split documents into before you search them.
• Eval set: a fixed set of questions with known good answers. You use it to measure the system the same way every time.
• Recall@5: the share of questions where at least one correct source appears in the top 5 search results. It measures the search, not the answer.
• BM25: a standard formula that scores how well a text matches the keywords of a question.
• Cosine similarity: a score for how close two embeddings are, so how similar two texts are in meaning.
• Reciprocal rank fusion: a method that combines two ranked lists using only the positions, not the scores. This matters because a keyword score and a meaning score use different scales.
• Cron job: a task that a computer runs automatically on a schedule, for example every night.
• VM (virtual machine): a rented computer in the cloud.
• OCR (optical character recognition): software that turns a picture of text, such as a scanned form, into text. It makes mistakes, especially with handwriting and numbers.
• LLM-as-judge: a language model that grades another model’s output in place of a person. It is fast and cheap, but it has its own errors.
• PII (personally identifiable information): information that identifies a person, such as a name, an address or a policy number.

The interview

Minutes 0 to 5: the opening

Nadia(interviewer):

Hi Yusuf, I’m Nadia. I’m a staff AI engineer on Claims Intelligence. I built most of our intake extraction. So, the plan: first you tell me about something you built, and I ask a lot of questions. That’s normal. Then a problem we have, then your questions. And like in the invite, no AI assistants in this round. Paper and pen are fine for notes. OK?

Yusuf(candidate):

OK, yes. I should say, it’s my first interview like this. If I talk too long, please stop me.

Nadia(interviewer):

I will, don’t worry.

Minutes 5 to 25: the project, then deeper

Nadia(interviewer):

Joel says your talk was good. Tell me about Bylaw Helper. What is it, who uses it, how does it work?

Yusuf(candidate):

So, in our neighbourhood association, people ask the same questions again and again. Can I build a fence this high? What did the council decide about the bike lane? The answers are public, but they are in about 3,800 documents. Around 1,100 are bylaws and amendments, the rest is council minutes. Mostly PDFs. Nobody can search them properly.

So I built a question-answering system. You ask in normal English, it finds the clauses, and a language model writes a short answer with the clause, like “Bylaw 2019-114, section 4.2(b)”. It’s RAG, basically. The search is hybrid. Postgres full-text search for keywords, and embeddings in pgvector, and I combine the two. About 60 people use it. And I have 150 test questions, two volunteers helped me write them, so I can measure when I change something.

Nadia(interviewer):

OK. Let’s stay on the chunks. How did you split the documents?

Yusuf(candidate):

First I did what every tutorial does. 512 tokens, overlap 64. It worked badly.

Nadia(interviewer):

Badly how?

Yusuf(candidate):

A bylaw clause is short. The median is, I think, around 90 tokens. So one chunk had five or six clauses, and the citation pointed to the wrong one. And chunks cut across sections.

So now I split on the structure of the document. One chunk is one subsection, maximum about 350 tokens, and it never goes over the… how do you say, the border of the section. The boundary. And every chunk gets a header, like “Fences Bylaw 2019-114, Part 4 Height, section 4.2”.

Nadia(interviewer):

Why the header?

Yusuf(candidate):

Because the clause alone says only “the maximum height is 2.0 metres”. The word “fence” is only in the title. Without the header, for a fence question, neither search finds it.

What the interviewer is checking

There is no “best chunk size”. Nadia wants to know if he chose his chunks from his own data. “The word fence is only in the title” is something you know only if you looked at the documents yourself.

Nadia(interviewer):

And you combine two rankings. How?

Yusuf(candidate):

Reciprocal rank fusion. Each chunk gets one over sixty plus its rank, from each list, and you add them. First I tried a weighted sum of the scores. But ts_rank and cosine similarity are not on the same scale, and ts_rank is not even real BM25. Every time I changed something, the best weight moved. With ranks, I don’t have this problem.

Nadia(interviewer):

Did it help? Do you have numbers?

Yusuf(candidate):

Yes. On the 150 questions, recall@5 with keywords only was around 0.63. Embeddings only, 0.71, I think. And hybrid 0.84. The table is in the README. Embeddings help when people say “granny flat” and the bylaw says “accessory dwelling unit”.

Nadia(interviewer):

Who decided which clause is correct for each question?

Yusuf(candidate):

In the answer? The model chooses from the top chunks, and then…

Nadia(interviewer):

No, sorry, I mean in your test set. Who wrote down the correct clause?

Yusuf(candidate):

Ah, OK. The volunteers and me. And here I made a mistake. At first, I saved chunk IDs as the correct answer. Then I changed the chunking, every ID changed, and my labels were useless. I had to do all of them again.

Nadia(interviewer):

Ouch. And now?

Yusuf(candidate):

Now the labels are clause references, like “2019-114 s. 4.2(b)”. The script checks if a retrieved chunk contains that clause. So I can change the chunking as much as I want.

Nadia(interviewer):

What did the volunteers disagree on?

Yusuf(candidate):

Oh, a lot. Hélène goes to almost every council meeting. Dev runs the newsletter. They labelled the same first 40 answers, and they agreed on… 31, I think. The big discussion was this: the answer is right, but it cites a clause that was amended later. Dev said it’s correct, the number is right. Hélène said it’s wrong, because if you show that clause to the city, they tell you it’s old.

Nadia(interviewer):

Who won?

Yusuf(candidate):

Hélène. Then we wrote a one-page guide. Three labels, and one yes-or-no question: “Does the cited clause support the answer?” After that, they agreed on 36 of 40.

What the interviewer is checking

This question checks that he built it himself. Only the person who sat with the volunteers knows what they argued about, and who won.

Nadia(interviewer):

That argument is close to a problem we have with policy endorsements. OK. Joel told me to ask about the tables.

Yusuf(candidate):

Ha, yes. The fees. “How much is a pool permit?” The fees are in tables at the end of the bylaw. The PDF text extractor read them line by line, so the columns got mixed. And my chunker cut tables in half, so the chunk with the numbers had no header row. The model gave the commercial fee for a house. Very confident.

Nadia(interviewer):

How did you find it?

Yusuf(candidate):

18 of my 150 questions are about tables. Recall@5 on those was 0.39. I remember it because it was so bad. Now I use a PDF tool that finds tables. I turn each table into Markdown, and a table is always its own chunk. If it’s too long, I split by rows and repeat the header row in every piece. Table questions went to 0.83. Overall, from 0.84 to 0.88.

Nadia(interviewer):

0.84 to 0.88 on 150 questions is six questions. How do you know that’s not noise?

Yusuf(candidate):

Hmm. Honestly, I didn’t do a statistic test. I looked at which questions changed. Eight went from miss to hit, two from hit to miss. All eight were table questions. So I believe the table fix is real, because I can see why each one changed. But 0.88 overall, I wouldn’t say it’s very exact.

Why this works

This is a strong answer, even without statistics. On 150 questions, 4 points can move by chance. But Yusuf compared the same questions before and after, and he can explain every change. The formal tool for this kind of before-and-after comparison is McNemar’s test. On so few changed questions, McNemar’s test would not be significant (p ≈ 0.1). Here, Yusuf’s question-by-question reason is the stronger evidence.

Nadia(interviewer):

Let’s move to how it runs. New council minutes come out every two weeks. Has the update ever broken?

Yusuf(candidate):

Yes. A cron job runs every night: scrape, extract, chunk, embed, insert. One small VM, the database is on it too. In March, the city changed its website. The scraper still got a 200, but the links went to a new viewer page. So for one week it indexed about 40 HTML pages that said only “Loading”.

Nadia(interviewer):

Who noticed?

Yusuf(candidate):

Hélène, not me. She asked me why the answers are so strange. Now the job checks that each file is a real PDF with text, and it sends me an email if a run adds nothing.

Nadia(interviewer):

Now imagine it’s not 60 neighbours but 4,000 adjusters. And it’s not you who gets the email at 2 a.m. It’s the person on call. How do they know what’s wrong, and what do they do?

The video freezes for a few seconds.

Yusuf(candidate):

Sorry, you froze. Can you repeat the last part? After the 4,000 adjusters.

Nadia(interviewer):

Sure. The person on call gets the alert at 2 a.m., not you. How do they know what’s wrong, and what do they do?

Yusuf(candidate):

OK. They need more than one email. A dashboard, maybe. Documents in, failures, how long answers take. And a page that says “if you see this, check that”.

Honestly, here I’m guessing. I was never in an on-call rotation. I don’t know what a real alert looks like, or how a team decides when to wake someone up. On my night shift, I know exactly what to do when something breaks. But that’s a warehouse.

The edge, the first time

Here Nadia reaches the edge of what Yusuf knows. He gives a guess, and he says it is a guess. He also names the experience that he does not have.

Nadia(interviewer):

That’s fair. Has anyone else ever reviewed your code?

Yusuf(candidate):

Once. Someone from the meetup added Docker in a pull request. He saw my API key in a config file in the repo folder. Not committed, but close. I have tests for the chunker and the eval script, because those broke the most. For the scraper, none.

Minutes 25 to 48: the Quillfield problem

Nadia(interviewer):

I’m moving on now. Here’s a real problem on our side. A claim packet comes in: 20 to 60 pages, scanned forms, the adjuster’s notes, emails, photos, a contractor’s estimate. Adjusters tell us they spend about 40 minutes reading a packet before they can act. We want to give them a one-page draft summary: the key facts, the open questions, and what’s missing. How would you approach it?

Yusuf(candidate):

OK, my first picture. Pages go through OCR, then a model reads the whole packet and writes the summary, and every line links back to a page. Like my clause citations. Then…

Nadia(interviewer):

Before you design, who do you think reads this?

Yusuf(candidate):

Right. The adjuster, I guess? Before they read the packet? And then the important thing is what a mistake costs them. If it says the date of loss is the 12th and it’s the 21st, does that change what gets paid?

Nadia(interviewer):

Yes, adjusters, before they open the packet, to decide what to do first. Key facts are different for each claim type. And a wrong date of loss can decide if the policy was in force at all. The adjuster always confirms and decides. The summary never decides anything.

Yusuf(candidate):

Then my picture gets more strict. Every line points to a page. “Date of loss: 12 March, page 3.” One click, and the adjuster sees the page.

Nadia(interviewer):

Mm-hm. And “what’s missing”?

Yusuf(candidate):

You already have the 32 fields from intake extraction, right? Then “what’s missing” is partly a checklist. A water claim needs a cause, an estimate, photos. If the field is empty, it’s missing. For that part, the model doesn’t need to write anything.

Why this works

Yusuf brings two habits from his project without help. Every statement gets a citation. And he reuses what already exists. A “what’s missing” list built on the existing extraction is also much easier to test than free text.

Nadia(interviewer):

Would you retrieve chunks from the packet, like in Bylaw Helper? Or give the model the whole packet?

Yusuf(candidate):

For one packet… I’d start with the whole packet. Forty pages is maybe 20 to 40 thousand tokens. Current models take that easily. But I’d test the longest packets on purpose.

Nadia(interviewer):

And the citations?

Yusuf(candidate):

I’d put a number on every page in the input. The model gives a page number and a short quote with every fact. Then I check that the quote is really on that page. With scans, the check needs a fuzzy match, because of OCR errors. And if OCR reads a 7 as a 1, the model quotes it perfectly and it’s still wrong. Same lesson as my tables.

Nadia(interviewer):

How would you know the summaries are good, before any adjuster sees one?

Yusuf(candidate):

Same as the bylaws, but with your claims experts, not Hélène and Dev. Take 30 packets, different claim types. The experts note the key facts with pages. For each summary I count missing facts, wrong facts, and statements with no support in the packet. Those are different mistakes, and you fix them in different ways.

Oh, and first, both experts do the same ten packets. To see if they even agree what a key fact is.

Nadia(interviewer):

Our two claims experts have about four hours a week between them, in total. Most of it goes to the weekly intake sample.

Yusuf(candidate):

Four hours for both? Hmm. So maybe one hour for me, in a good week. Then they can’t write anything for me. I’d show them a summary next to the packet. They mark each line right, wrong or unsupported, and add what’s missing. Checking is faster than writing.

Nadia(interviewer):

Which packets would you give them?

Yusuf(candidate):

The hard ones. Long packets, injury claims. And a model as a judge, maybe, for quick checks between changes. I tried it on my bylaw answers. It agreed with the volunteers on 51 of 60, something like that. But it was too kind on “partly correct”. So the judge is for quick checks, and the experts decide.

Nadia(interviewer):

Is there anything about the data itself that worries you? Think about what’s in an injury claim.

Yusuf(candidate):

Oh. Right. Medical information. Names, addresses, health problems. I didn’t think about that at all, and I should. All my data is public.

So the summary must be visible only to people who can already see the claim. I can’t log full prompts and outputs like I do now. And the eval set of 30 real packets is full of personal information too, so it can’t be on someone’s laptop. Is the law here PIPEDA?

Nadia(interviewer):

For private companies, at the federal level, yes. Some provinces have their own laws. You don’t need to know the law by heart. You need to notice the data. That’s the habit.

Where this goes wrong

Yusuf missed personal information until Nadia pointed at it. This is common after projects with public data. Here it matters, because many packets contain personal information (PII: information that identifies a person) and health information, which is even more sensitive. He recovered well. But in a hiring decision, one missed risk like this can count more than three good ideas.

A stronger answer

He could raise it himself, right after Nadia told him who reads the summary. He already knew enough:

“One thing is very different from my project: my data was all public. Yours is people’s claims, and injury claims have health information. So before the design, I want to know the rules. Where can the data go? Who can see a summary? What can we log? And the eval set is real claims, so it needs the same protection.”

This answer needs no legal knowledge. It needs one question: “How is this data different from the data that I know?”

Nadia(interviewer):

Last part. We get about 9,000 packets a day, and say we write each summary when the packet arrives. Now it’s Monday, 9 a.m. The weekend backlog comes in: 4,000 packets in an hour. The provider starts to return rate-limit errors. What happens to those packets?

Yusuf(candidate):

They go into a queue. Workers retry, and they wait between tries, longer each time.

Nadia(interviewer):

OK. How many workers? And what does the adjuster see while it waits?

Yusuf(candidate):

That I don’t know. How many workers, how to stop the retries from making it worse, what the adjuster sees… I’ve never run a queue under real load. If one of my 400 questions a month fails, the person asks again. But your extraction has the same 9,000 packets, no? I’d want to see how it handles this first.

The edge, the second time

This is the main edge, and it is the same one as before: running a system at scale, as a team. Yusuf reasons correctly to a queue, with retries that wait longer each time. Then he stops. He points to the existing extraction, which already solves this problem.

Minutes 48 to 57: Yusuf's questions

Yusuf(candidate):

First, the direct question. The posting says 4+ years. I have zero. Is that a hard line?

Nadia(interviewer):

I’ll be honest, because you were. For me, it’s not a filter. The number stands for things. You’ve worked in a codebase that other people own. You’ve been reviewed, you’ve been on call, you’ve broken production and fixed it. This role owns a feature alone quite soon. Today you showed real depth on retrieval and evaluation. You also showed that those other things are new to you. I’ll write both down.

Yusuf(candidate):

OK. That’s clear. Thank you. Then, if I join at some point, who do I learn from, day to day? How does review work?

Nadia(interviewer):

Every change gets a reviewer from the team. New people pair with someone for the first weeks. There’s also an on-call rotation, one week in six. New people shadow two rotations before they take one alone.

Yusuf(candidate):

And the posting says “help define our evaluation strategy”. So there isn’t one yet? How do you know today that extraction works?

Nadia(interviewer):

For extraction, the experts label 200 packets a week, and we measure each field. For everything else, it’s ad hoc. And you just saw why: the experts’ time. That’s one reason we’re hiring.

Why this works

All three questions are about the work. The first one is brave, and it gets an honest answer. The question about learning is the right one for someone without team experience. Its answer also shows something that the posting does not say: there is on-call. The last question connects a vague line in the posting with the experts’ time. See an interview is a conversation in both directions.

In the last three minutes, Nadia explains the next steps. She writes her notes the same day, and Hannah will contact Yusuf within a week. Yusuf says that he will think more about the 2 a.m. question.

After the call: Nadia’s notes

Candidate: Yusuf Demir. Role: AI Engineer, Claims Intelligence (intermediate to senior).

Strong evidence. Retrieval: chunks designed from his own data; hybrid search measured against each method alone. Evaluation: clause-level labels, agreement between labellers measured and improved, a paired check of changes. Better than some senior candidates. Honest and calm at the edge.

Some evidence. Deployment: one VM, cron, Docker, some tests; small. Transfer to our problem: a page for every fact, reuse of the 32 fields.

No evidence. Team practice (one code review ever). Production at scale: queues, monitoring, on-call.

Concern. Did not notice personal and health information until prompted. Recovered well.

Authorship. No doubt. Nobody invents the re-done labels, the amended-clause argument or the March scraper failure.

Recommendation. Not at the level of this posting. Not because of the years line, but because of the missing team and production experience. Strong yes for the junior AI engineer role planned for next quarter. Marcus, please keep him in the process; no standard rejection.

What happened next

Marcus agreed. Two days later, Hannah called Yusuf. He would not move forward for this role. But the team wanted him for a junior role that was not posted yet. She also sent Nadia’s feedback in three points: personal data, production operations, team practice.

Yusuf was disappointed, and he said so. Then he added a section to his README: “What I would do differently with private data”.

The junior role will open a month later than planned, because of the budget cycle. Hannah has already invited Yusuf to a system design interview with Owen for that role in the new year. Nothing is promised. He is also applying to junior roles at other companies.

What “4+ years” meant here

But a level is more than a number. It sets the pay band. It sets how much help the team plans to give, and what the person must do alone in the first months. Yusuf could not show the team and production part. So for this role, the requirement stayed, and for a good reason. Applying was still right. It gave him an honest assessment and a place in the next process.

What to take from this

• A real project can carry a serious interview. Every answer came from something that Yusuf did and measured. See show work that is close to real.
• Details show that you built it. Only the builder knows what the volunteers argued about. You cannot prepare those answers. You can only have them.
• Your edge is what your project could not teach you. For someone who builds alone, that is usually team practice, scale and private data. When you reach it, say it plainly.
• Ask what is different about the new data. Yusuf brought his methods. He also brought his habits from public data.

If your profile is like Yusuf’s

1. Build one project with real users and real failure cases. Small is fine. Use messy data, and add an eval set, a deployment and a README with decisions and limits. Use Portfolio anatomy as the checklist.
2. Measure search and answers separately. Label against stable references, such as clauses or pages, not chunk IDs. Then your labels still work after the next change. Ask a second person to label the same 30 items, and count how often you agree.
3. Keep a log of what broke and how you found out. The details are your best answers, and you forget them fast.
4. Work on your gaps on purpose. Add fake personal data to a copy of your project. Decide what you log and who can see what. Put the project in a container, with tests and a pipeline: the Docker and CI/CD courses help. Then get your code reviewed. A few accepted pull requests to an open-source project teach you review, and how to read code that you did not write.
5. Practise saying it aloud, then in public. Use Explain decisions: the problem, the options, the choice, the numbers, the limits. Then give a short talk at a meetup. It is also networking.
6. Apply anyway, and read the level honestly. Apply for roles a little above you, and for junior roles too. When you hear “not at this level”, ask what the level needs, and write it down. See Prepare for applications.