The data scientist

Think of a cook who ran a small restaurant kitchen for five years. She planned the menu, bought the food, cooked, and heard every complaint. Now she applies to a large hotel kitchen. She has never cooked for 2,000 people at once. But she knows what it means when a dish comes back.

Many data scientists who apply for ML engineer jobs are like this. In a small company, they did everything around one model, from the data to the angry call when something went wrong. They had no platform team, one Docker container, and a lot of responsibility. Follow Mateo through the ML deep dive. Watch how his ownership of one model helps him on a new problem, and where scale and online testing take him to his edge.

MR

The candidate

Mateo Rojas

Data scientist, 5 years in consumer lending in Bogotá, in Canada for one year

• 5 years as a data scientist
• Credit risk, end to end
• Permanent resident
• Toronto, Ontario

• Data Analyst, Brookvale Home (furniture retailer), Mississauga (2025 to now). Weekly sales and stock reports in SQL and Power BI. A small Python forecast for store stock.
• Senior Data Scientist, Lunaria Crédito, Bogotá (2022 to 2025); Data Scientist (2020 to 2022). Small personal loans through a mobile app.
• Built and deployed the credit risk model (LightGBM) that replaced a rules-based scorecard. About 15% more approvals at the same default rate.
• Served it as a Flask API in Docker for real-time loan decisions, plus nightly batch scoring for credit limit reviews.
• Owned the full model lifecycle: training, deployment, monitoring, retraining.
• Set up drift monitoring (PSI on all model features, weekly report and alerts) after a change in the customer mix in 2023.
• Python, SQL, LightGBM, scikit-learn, Docker, Airflow, AWS.

Mateo has worked in Canada for one year, as a data analyst. Many people who move take a smaller job first. His real experience is the five years in Bogotá.

Canadian experience is not required. Since 1 January 2026, a public job posting in Ontario must not require Canadian work experience, and Fennick’s posting says so. For Ravi and Ioana, Mateo’s five years in Colombia are five years of experience. If you moved recently, do not hide or apologise for your work from your home country. Describe it the same way you would describe a Canadian job.

What Ioana sees before the call

Mateo has never worked on delivery or on real-time systems at Fennick’s size. But he put a model in production and watched it after launch. The PSI line says that something went wrong and he built a fix. That is often the most interesting story in a CV.

She writes down what she wants to find out:

• “15% more approvals at the same default rate.” How do you know the default rate of people the old system rejected?
• “Owned the full model lifecycle.” What exactly did he own? Who was on call?
• The drift story. How did he notice, and how long did it take?
• Can he move from credit decisions to a model that answers 1.5 million times a day? Does he think about the customer and the couriers?

Terms on this page

• Default: the borrower does not pay back. At Lunaria, a loan “defaulted” if a payment was more than 60 days late in the first six months.
• PSI (population stability index): a number that says how much the distribution of one feature changed between two periods. A common rule of thumb: under 0.1 is stable, 0.1 to 0.25 is a moderate change, over 0.25 is a large change.
• p99 latency: 99% of requests are faster than this time. Fennick’s budget for the ETA service is about 50 ms at p99.
• Percentile (or quantile) prediction: instead of the average time, the model predicts a time that, for example, 90% of orders will beat. This is the “90th percentile” or “0.9 quantile”.
• Training/serving skew: a feature is computed one way for training and another way in production, so the model sees different numbers live than it learned from.
• Switchback test: an online test that switches the whole of one zone between the old and the new model in time blocks, instead of splitting customers.

The interview

Minutes 0 to 5: the opening

Ioana(interviewer):

Hi Mateo. Can you hear me okay?

Mateo(candidate):

Yes, yes, I hear you well. Hi, nice to meet you.

Ioana(interviewer):

Nice to meet you. I’m Ioana, staff ML engineer in Delivery Intelligence. I built the ETA service, the one that shows delivery time in the app. So, the plan: first you tell me about a model you put in production, and I will ask many questions. Then a real problem from our side. At the end, your questions. One thing: no AI assistants in this round, please. Paper is fine.

Mateo(candidate):

Okay. And, sorry, before we start: my English is okay but not perfect. If I say something not clear, please ask me.

Ioana(interviewer):

Same for me, English is my second language too. We will manage. Let’s start.

Saying it at the start

Mateo says it once and moves on. Your English does not need to be perfect; a good interviewer listens for your thinking. See Your English does not need to be perfect.

Minutes 5 to 25: the credit model, then deeper

Ioana(interviewer):

So, tell me about the model.

Mateo(candidate):

Okay. So, Lunaria gives small loans in an app, like 500,000 to 3 million pesos, for people who often don’t have a bank history. When I joined, the decision was a scorecard, a logistic regression a consultant made years before, plus many manual rules. The problem was we rejected a lot of good people. So I built a LightGBM model on the application data and the credit bureau data. It went live in, I think, March 2022. The API decides in the app while the person waits, and at night we score all customers again for credit limit changes.

Ioana(interviewer):

Mm-hm. What exactly does the model predict?

Mateo(candidate):

If the person defaults. For us default was more than 60 days late, in the first six months of the loan.

Ioana(interviewer):

Why six months? Why not the whole loan?

Mateo(candidate):

Because of time. Some loans are 18 months, so I wait 18 months to know the answer. And most defaults happen early. I checked: about, hmm, 80% were in the first six months. So it was a compromise.

Ioana(interviewer):

Okay. And how did you split the data?

Mateo(candidate):

By time. Train on the old loans, test on the newer ones. And the last six months I could not use at all, because those loans don’t have a label yet. This was painful. The newest data is the most useful, and I couldn’t touch it.

What the interviewer is checking

Two basic questions: what is the label, and how was the data split? Mateo shows that time matters twice. Future data must not leak into training, and a label needs time to “mature”. See Train, validation and test sets.

Ioana(interviewer):

Your CV says 15% more approvals at the same default rate. How do you know the default rate of people the old scorecard rejected? They never got a loan.

Mateo(candidate):

Yes. This is the big problem in credit, the… how do you say, reject inference. We don’t know. So for three months, we approved a random 2% of the people the scorecard rejected, with a small amount. The risk team was not happy. But then we had labels for some of them.

Ioana(interviewer):

And the 15% came from that?

Mateo(candidate):

Hmm. Partly. The 15% was the offline estimate, plus the first months. Honestly, after one year, the real number was less. More like 10, maybe 12%. I should change it in the CV.

Ioana(interviewer):

Ten percent is still good. Okay, the drift story. What happened in 2023?

Mateo(candidate):

Ah, this one. The marketing team made a partnership with a chain of phone stores. You buy a phone, you can get a loan in the store. Very fast, a lot of new customers. But they were very young, and many had no bureau history at all. Before, people without history were about 10% of applicants. After, in a few weeks, about 35%.

Ioana(interviewer):

And the model?

Mateo(candidate):

The model approved many of them. It had seen few people like this in training. And nobody looked. After almost three months, the collections team asked why so many people don’t pay the first payment. For that channel, it was about double the normal.

Ioana(interviewer):

Three months. How did you find the cause?

Mateo(candidate):

I compared the feature distributions, before and after. That’s when I started with PSI. The bureau score, with missing values as their own bin, had PSI almost 0.4. Very big. So I built a weekly job: PSI for every feature against the training data, and an alert if something is over 0.25. And a dashboard of first payment default by channel, because that label comes in one month, not six.

Why this is the best part of his project

Mateo did more than add a monitoring tool. He found a label that arrives faster than the real one: a missed first payment shows in one month, a default in six. A fast signal like this warns you early.

Ioana(interviewer):

And you fixed the model?

Mateo(candidate):

First, a rule: the phone store channel without bureau history gets a smaller amount. That was the same week. Then we retrained with the new customers when we had labels, maybe five months later. In 2023 we have changed also the retraining: every three months, not once a year.

Ioana(interviewer):

Okay. “Owned the full lifecycle.” Who was on call for the API?

Mateo(candidate):

Hmm. Honestly, the alerts for the API went to the backend team first. If the API was down or slow, they handled it, and they called me if it was the model. It happened maybe two or three times. So the model was mine. The servers, not really.

Ioana(interviewer):

And how fast was the API?

Mateo(candidate):

I don’t remember exactly. Under 300 milliseconds, I think, most of the time. Nobody worried, because the person is filling a form. Two hundred milliseconds more is nothing for them.

The word "owned"

“Owned the full model lifecycle” was bigger than the truth. Mateo did not own the service in production. He corrected it himself, so it did not hurt him much, but it cost a little trust. Write what you owned in plain words: “Owned the model, its monitoring and retraining; the backend team ran the API.”

Minutes 25 to 48: the Fennick problem

Ioana(interviewer):

Okay, now a real problem from our side. Customers see a delivery time in the app before they pay. At dinner time and when the weather is bad, our estimate is too optimistic. On normal days about 9% of orders arrive more than ten minutes late compared to the estimate; on those days it’s about 18%. People are angry, support tickets go up. How would you approach this?

Mateo(candidate):

Okay. Let me write this down. Nine percent normal, eighteen at dinner and bad weather. He writes. First question: what does the number in the app mean today? Is it the average time? The most likely?

Ioana(interviewer):

Today the model is trained to predict the expected time. So, more or less, the average.

Mateo(candidate):

Then I think part of the problem is the question, not only the model. If you show the average, many people will be late by design. At dinner the times are more… spread? More variance. So the average is far from the bad cases.

Ioana(interviewer):

So what would you show?

Mateo(candidate):

A higher percentile. Maybe the 80th or the 90th. This is like credit, actually. Being late is worse than being early, like approving a bad loan is worse than rejecting a good one. The costs are not the same, so the prediction should not be in the middle.

Ioana(interviewer):

Okay. 80 or 90, who decides?

Mateo(candidate):

Not me alone. If the number is too high, people don’t order. So I would ask product: if we show five minutes more, how many orders do we lose? And I would ask operations what late costs them, support tickets, refunds. Then we choose together. Maybe 85.

Why this works

Mateo moves the problem from “make the model better” to “what number should the customer see”. That is the most important step. His credit experience helps: unequal costs mean that the best prediction is not in the middle. And he does not choose the percentile alone.

Ioana(interviewer):

We have one data point. In March, operations added five minutes when it rained, in two cities. Orders went down about one or two percent. How would you train the model for the 85th percentile?

Mateo(candidate):

LightGBM has a quantile objective. You give alpha 0.85, and it learns that point with the pinball loss. I used it once, for a cash forecast, not for credit. Another way is to keep the average model and add a buffer, but the buffer is the same for all orders, and the problem is the dinner ones.

Ioana(interviewer):

Good. And how do you know offline that it works?

Mateo(candidate):

Test on the most recent weeks, by time, like before. Then I check: did about 85% of orders arrive before the shown time? But not only in total. Separately for dinner, for rain, for snow, per city. If it’s 85% in total but 70% at dinner, the problem is still there.

Ioana(interviewer):

Mm-hm. That’s what we call coverage by condition. Okay. What features are missing, do you think?

Mateo(candidate):

Weather, for sure. And how busy the store is right now, and how many couriers are free near the store.

Ioana(interviewer):

The second two we have. Weather we don’t have yet. How would you add it?

Mateo(candidate):

Hmm. Careful here. For training, it’s easy to take the historical weather, what really happened that evening. But at checkout, we only know the weather now, or the forecast. So in training I must use the weather known at the moment of the order. In credit we had the same with the bureau: data as of the application day, not today’s, or the model learns from the future.

Ioana(interviewer):

Yes. We had a related bug last year. A feature came from a 15-minute batch table in training and real-time counts live. Six weeks before we noticed. My question for you: would your PSI have caught it?

Mateo(candidate):

Let me think. Pause. If I compare the training distribution with the live distribution of that feature… the real-time count moves more, the batch one is smoother. So maybe the shape is different, and PSI goes up a bit. But maybe not over the alert. It’s not a strong check for this. Better is to log the features that the service really used, for each order, and later compute the training features for the same orders. Then compare order by order. If they are different, it’s a bug, not drift.

How Ioana goes deeper

Ioana turns her own bug into a question about Mateo’s tool. It tests whether he understands PSI or only uses it. His answer is right. PSI compares two groups of numbers. A skew bug is a difference for the same order, so you find it by comparing the same orders, row by row.

Ioana(interviewer):

Okay. Now the service. The app calls the ETA model on every checkout page view. About 1.5 million calls a day, and the budget is 50 milliseconds at p99. How does that change your design?

Mateo(candidate):

Hmm. 1.5 million a day is… about 17 per second, on average? But not at night, so in the evening it’s more. Maybe five times more at dinner?

Ioana(interviewer):

A bit less, but close.

Mateo(candidate):

Okay. The model itself is not the problem, I think. LightGBM with 120 features is fast, around one millisecond. The time goes to getting the features. The real-time ones must be ready in memory, not computed in the request. But honestly, I never worked with a budget like 50 milliseconds. I would need to learn how your feature store works.

A stronger answer about latency

Mateo’s reasoning is right, but general. A candidate with real-time experience would split the 50 ms into parts: “Network, a few milliseconds. Online feature lookup, maybe 10 to 20 at p99. Model, one or two. And a fallback: if a lookup is too slow, use the last known value or a simple default ETA, and never block the checkout page.” Mateo said honestly that he had not done this. That was right, because if you guess the numbers, the next question shows it.

Ioana(interviewer):

Good. Now, you have a new model, and offline it looks better. How do you test it with real customers?

Mateo(candidate):

An A/B test. Half the customers see the new estimate, half the old. Random by customer. Then compare the late rate, the support tickets, and how many people order.

Ioana(interviewer):

Okay. One problem. The couriers are shared. The customers in group A and group B, their orders go to the same couriers in the same zone. What does that do to your test?

Mateo(candidate):

Sorry, can you say it again? I lost the part after “shared”.

Ioana(interviewer):

Sure. Customers in both groups use the same couriers. A courier who takes an order from group A cannot take one from group B. So the two groups are not independent. What does that do to the test?

Mateo(candidate):

Ah, okay. Hmm. Long pause. So, the new model shows longer times, especially at dinner. Some people in group B see a longer time and don’t order. Then there are fewer orders… and the couriers have more time… for everybody? Also for group A?

Ioana(interviewer):

Yes. Continue.

Mateo(candidate):

Then group A becomes faster because of group B. So the difference between A and B is wrong. Smaller, or… I’m not sure which direction. I understand the problem. But I don’t know the method to fix it. In credit, one customer’s loan doesn’t change another customer’s loan.

Ioana(interviewer):

One common answer is a switchback test. You don’t split customers. You split time and place: a zone uses the new model for one hour, then the old one, and it switches many times over some weeks.

Mateo(candidate):

Okay, I see. So in one zone, in one hour, everybody has the same model, and the couriers are not mixed. Then the unit is the zone-hour, not the order. So I have fewer units, and I need a longer test. And… the switch moment is a problem? An order from 6:55 is still on the road at 7:10, with the old estimate. So maybe I ignore the first part of each block?

Ioana(interviewer):

Yes, exactly. That’s the carryover problem. We usually drop a short window after each switch.

At the edge

This is Mateo’s edge, and it is the most useful minute of the call. He asks for the question again instead of answering one he did not hear. He does not know switchback tests, and he says so. After Ioana’s two sentences, he finds the next problems himself: the unit of analysis, fewer units, and the carryover at each switch.

Where the first answer went wrong

“Random by customer” is the standard A/B test, and it is right for many products. In a marketplace, what one group sees changes what happens to the other. Before you propose an online test, ask: do the two groups share something, like couriers, stock or a budget?

In the last minutes on the problem, Ioana asked about retraining. Mateo smiled: at Fennick, the true delivery time is known an hour later, not six months later. So for him the hard part was not weekly retraining. It was checking each new model before it goes live, and knowing who understands the retraining job.

Minutes 48 to 57: Mateo's questions

Mateo(candidate):

I have three questions. First: when the late rate goes up at dinner, who sees it first, your team or operations?

Ioana(interviewer):

Honestly? Operations. Laura’s team sees support tickets the same evening. We see the error in the next morning’s report. We have no monitoring on the features, only on the error after delivery.

Mateo(candidate):

So the same as my phone store story. The business sees it before the model team.

Mateo also asked who owns the weekly retraining, and whether the role has on-call. Ioana answered honestly: a cron job that one engineer really understands, and on-call one week in five, which the posting does not say.

Good questions

Each answer tells Mateo something real: the business sees problems first, retraining depends on one person, and there is on-call that the posting does not mention. His first question is his own: it comes from his phone store story. Many candidates are afraid to ask about on-call. It is a normal question. See An interview goes in both directions.

Minutes 57 to 60: the close

Ioana(interviewer):

Thanks, Mateo. Next is the system design round with Ben, and a conversation with Laura from operations. Chantal will write to you this week.

Mateo(candidate):

Thank you, this was interesting. I learned something today, the switchback.

After the call: Ioana’s notes

Ioana writes her notes the same afternoon, against the job description.

Mateo Rojas, ML deep dive

• Owning a model in production: strong evidence. Label choice with a reason (six months), split by time with label maturity, a real baseline (the old scorecard), a reject-inference test. Monitoring built after a real incident, with a fast proxy label.
• Problem framing and business sense: strong evidence. Moved straight to “what number should the customer see”. Proposed a high percentile and named who decides (product, operations).
• Evaluation: strong evidence. Coverage by condition, not only in total. Point-in-time weather without a hint. Correct answer on PSI vs skew: compare the same orders.
• Real-time serving at our scale: some evidence. Reasoned correctly that features, not the model, use the time budget. Never worked under a 50 ms budget, and said so.
• Online experiments with interference: no evidence before today. Did not know switchbacks. After two sentences, found the unit of analysis and the carryover problem himself.
• Concern (small): “owned the full lifecycle” was bigger than the truth; he corrected it himself. The 15% on the CV is really 10–12%; he corrected that too.

Recommendation: yes. Strong ML judgment and real ownership. The gaps are real-time systems and marketplace experiments; both can be learned on the team, with Ben’s round as a check on the first.

What happened next

Ben’s system design round was solid. Mateo designed a clear pipeline, but he was slow on the real-time feature path and needed help with caching. Laura’s values conversation was a clear yes. She wrote: “He asked what a late order costs us before he talked about models.”

There was one vacancy and two strong finalists. The team chose Hana Kowalczyk, who had built ETA models before. Ravi called Mateo himself: “You were our second choice, and it was close. I would like to stay in touch.” Mateo asked for feedback. Ravi gave two points: online feature serving, and experiment design for marketplaces.

In January, Fennick approved a second ML engineer position on the team, for the demand forecast. Ravi called Mateo first, before the job was posted. Mateo had one more short call with Ravi and Ben, and then an offer near the middle of the band. He accepted.

This is one realistic ending, not a promise. Many companies never call their second choice back. Luck is part of it.

What to take from this

• Your experience from another country counts. Mateo described his Colombian work exactly as he would describe a Canadian job, with numbers and decisions. In Ontario, a posting cannot require Canadian experience.
• Ownership shows in the story after launch. Mateo’s strongest minutes were about what went wrong in production and what he built after.
• Bring your old domain into the new problem. Unequal costs in credit became a high percentile for ETAs. Point-in-time bureau data became point-in-time weather.
• Ask for the question again, then reason. “Sorry, can you say it again?” cost Mateo five seconds. At the edge, he did not know the method, but he saw why the simple test fails and continued after a short explanation.
• Correct your own numbers. “It was more like 10, maybe 12%” made the rest of his numbers believable.

If your profile is like Mateo’s

1. Prepare your production story. What you built, what went wrong after launch, how you found it, and what you changed. Use the numbers you really remember. Explain decisions helps you say it aloud.
2. Check the words “owned” and “end to end” on your CV. Replace them with what you did and what other teams did.
3. Learn percentile predictions and how to check them. Train a LightGBM model with the quantile objective on a public dataset, then check coverage by group, not only in total. The Bike-share demand case study is a good dataset for it, and Slice analysis shows how to look at groups.
4. Learn the real-time side. Read how an online feature store works. Build a small service with a strict latency budget, and measure p99, not the average. The API course and Docker course help with the service.
5. Learn experiments for marketplaces. Read about interference, switchback tests and carryover. Explain why a test by customer can be wrong when customers share couriers or stock.
6. Practise out loud, in English, about your own projects. You do not need to sound like a native speaker. You need to be clear.