The researcher

Imagine a very good cook who has spent five years on one dish. She knows every ingredient, she has tested every change, and she has written about it. Now a busy restaurant asks her to cook for 300 people every night, with a cheap oven and twelve minutes per plate. Her skill is real. But the question is now different.

Many people finish a PhD in machine learning and apply for Machine Learning Engineer roles. They have trained more models than most engineers. They read data carefully, and they know how a test can lie. But often nobody else has used their models, there is no pager, and no customer waits 50 milliseconds for an answer. Most of their work used one kind of data, often images or text, on large GPUs.

Anika Sharma is this candidate. Watch how her research habits help her on Fennick’s problem, and watch the moment she sees that she reached for what she already knew. Also watch Ioana make a small mistake, and how Anika corrects her.

AS

The candidate

Anika Sharma

PhD in computer vision, moving from research to industry

• PhD, computer vision, 2026
• 2 papers
• PyTorch and GPUs
• No production role
• Toronto

• PhD, Computer Science (computer vision), 2026. Thesis: detecting animals in camera-trap images from new locations
• 2 papers on how image models fail at new locations (one at a main conference, one at a workshop)
• Deployed a species classifier used by an ecology research group to process 2.3 million images
• Research Intern, Halden Labs (Montreal), 4 months in 2024: self-supervised pre-training for video
• Expert in deep learning: PyTorch, CUDA, multi-GPU training, experiment tracking
• Teaching assistant, Machine Learning, 3 terms

What Ioana sees before the call

Ioana Petrescu is a Staff ML Engineer at Fennick. She built the ETA service. She reads Anika’s CV next to the job description.

The modelling evidence is strong: two papers, a good internship, three terms of teaching. The posting says “4+ years in production, or 2+ years with a PhD”. Anika has the PhD, but no production years at all. And Fennick’s work is mostly tables, not images.

Two words get a mark: “Deployed” and “Expert”. Ioana wants to know what “deployed” means here. She writes down three things to find out:

• What happened to the species classifier after it left Anika’s laptop? Who used it, and who checked that it still worked?
• Does she test models in a way that matches how they are used?
• On the ETA problem, does she choose a model for the data and the limits, or for what she knows?

Terms on this page

• Camera trap: a camera in a forest that takes photos when an animal moves in front of it. Ecologists use the photos to count animals.
• ETA (estimated time of arrival): here, the time from the order to the customer’s door that the app shows before the customer pays.
• p99 latency: 99% of requests finish faster than this time.
• Gradient boosting (for example LightGBM): many small decision trees, each one corrects the errors of the ones before. It is often the best choice for table data.
• Quantile (or percentile): the 90th percentile of delivery times is the time that 90% of orders are faster than.
• Pinball loss (or quantile loss): a training loss that makes a model predict a chosen quantile instead of the average. It counts errors on one side more than on the other.
• Coverage: how often the real value falls under the predicted quantile (or inside a predicted range). For a 90th percentile, it should be about 90%.
• Training/serving skew: a feature is computed one way for training and another way in production.

The interview

Minutes 0 to 5: the opening

Ioana(interviewer):

Hi Anika. I’m Ioana, staff ML engineer, I work on the delivery time model. Thanks for joining. It’s late afternoon here, so if I’m slow, it’s the coffee that is missing.

Anika(candidate):

Laughs. No problem. Hi.

Ioana(interviewer):

So the format. You tell me about a model you built, and I ask many questions, going deeper. Then a problem we have at Fennick. Then you ask me things. And please, no AI tools in this round, just us. Paper is okay.

Anika(candidate):

Okay. I should say, I haven’t worked in production. So the “model in production” part, I’ll take the closest thing I have.

Why this works

Anika says her gap at the start, in one sentence, and then moves on. She does not apologise for it. Ioana now knows what to expect, and the conversation can stay on what Anika did. See no experience is not no evidence.

Minutes 5 to 25: the project, then deeper

Anika(candidate):

So, my PhD was about camera traps. Ecologists put cameras in forests, and the cameras take millions of photos. Most photos are empty, the wind moves a branch. Someone has to look at all of them and say “empty”, “deer”, “fox”, and so on. It takes months.

There are models for this. But when you put a camera in a new place, they get much worse. That was my research question: why, and how to fix it. And one ecology group used my model on their real data. That’s the “deployed” in my CV.

Ioana(interviewer):

Okay. What does “used” mean, in practice?

Anika(candidate):

Honestly, a script. They copy a season of photos to the lab GPU server, they run my script, it writes a CSV with a label for each photo. Then a student checks the uncertain ones. The first season was about 2.3 million images. So, “deployed” is a big word for it, I know.

Ioana(interviewer):

Smiles. It’s okay, real people used it. What was the label? What did the model predict?

Anika(candidate):

One label per image. Empty, or one of 14 species. Actually 15 with “other animal”. Around 70% of images were empty.

Ioana(interviewer):

So how did you measure it? Accuracy would be high just from the empty ones.

Anika(candidate):

Yes, exactly, accuracy was useless. A model that says “empty” always gets 70%. I used recall per species, and the average over species. The ecologists cared most about the rare ones. Lynx, for example. If you miss a lynx, that’s a real loss for them. If you call an empty image “deer”, a student fixes it in two seconds.

Ioana(interviewer):

And how did you split the data?

Anika(candidate):

This is actually my first paper. Most people split images randomly. But the cameras take three photos in a burst, one second apart. So with a random split, the same animal, the same tree, is in training and in test. The model just remembers the background.

With a random split, we got about 95% average recall. When I split by camera, so test cameras are never seen in training, it went down to about 81%, I think. Yes, 81. And by camera is the real use, because the ecologists always add new cameras.

Why this works

This is Anika’s strongest moment, and it is the most important skill in the role. She asked how the model will be used, and she made the test match it. A new camera in a forest is like a new week at Fennick: the model will always meet data it has not seen. Her split by camera is the same idea as a time-based split. See Train, validation and test sets.

Ioana(interviewer):

Nice. And a baseline?

Anika(candidate):

A public animal detector, plus a simple classifier on top. That was what the ecologists used before. My model was better on new cameras, around eight points on average recall. On the rare species, more.

Ioana(interviewer):

Okay. Then the ecologists use it for a season. How did you know it still worked?

Anika(candidate):

Hmm. The honest answer is, I didn’t. The first winter, it snowed. At night the camera uses infrared, and falling snow in the flash looks like… white shapes. The model called a lot of empty images “deer”. The ecologist told me the deer count was double the last winter. That’s how I found out.

Ioana(interviewer):

So the user found it, not you.

Anika(candidate):

Yes. I labelled about 800 winter images, fine-tuned, and it got better. And now, every season, a student checks a random 200 images before they use the counts. But that check, I added it after.

Ioana(interviewer):

And when you leave the university, who runs it?

Anika(candidate):

Pause. A README. Nobody, really. I’m thinking about this now, actually. I should talk to them.

Where this goes wrong

The model had no monitoring and has no owner. In research this is normal: you publish, and you move to the next question. In an ML engineer role it is the main part of the job. Anika could have counted the predictions per species each week, and compared them with the last season. A sudden change in the counts would have warned her before the ecologist did. See Plan production checks.

Minutes 25 to 48: the Fennick problem

Ioana(interviewer):

Okay, now a real problem from our side. Customers see a delivery time in the app before they pay. At dinner time and when the weather is bad, our estimate is too optimistic. On normal days about 9% of orders arrive more than ten minutes late compared to the estimate; on those days it’s about 18%. People are angry, support tickets go up. How would you approach this?

Anika(candidate):

Interesting. So the time depends a lot on the route, on traffic. My first idea: use the courier’s route. GPS points over time, as a sequence. A transformer over the sequence could learn traffic patterns, how long each street takes at dinner. And you can add the weather as a token, or as an extra input.

Ioana(interviewer):

Okay. Two things. When the customer sees the estimate, they did not pay yet. Do we know the courier at that moment?

Anika(candidate):

Ah. No. The order doesn’t exist yet. So there’s no courier, and no route.

Ioana(interviewer):

Right. And second, the app calls us on every checkout page view, about 1.5 million times a day. We have about 50 milliseconds, at p99.

Anika(candidate):

50 milliseconds for the model?

Ioana(interviewer):

For everything. Reading the features, the model, the answer.

Anika(candidate):

Okay. Can I take a minute to think?

Ioana(interviewer):

Sure, take your time.

Silence, about 40 seconds. Anika writes on paper.

Anika(candidate):

Okay. I think I reached for what I know. Five years of images and sequences, so I saw a sequence. But what you actually have at checkout is a table. The store, the time, how many orders are open at the store, how many couriers are near, the distance, the weather. For table data, gradient boosting is usually better than a neural network, and it’s very fast. One prediction with a few hundred small trees takes around a millisecond on a CPU, often less. So I’d start there. Probably that’s what you use?

Ioana(interviewer):

Yes, LightGBM, about 120 features.

Why this is a good moment

Anika’s first idea did not fit the problem. Two facts from Ioana showed it: no route exists before the order, and the time limit is very small. Anika did not defend her idea. She asked for time, thought about the real data, and said clearly what had happened: “I reached for what I know.” Interviewers trust a person who can see their own habits.

Anika(candidate):

Then I don’t think the problem is the model family. If you have a good LightGBM and it’s still too optimistic at dinner… what is it trained to predict? The mean? With squared error?

Ioana(interviewer):

Yes, squared error. One number, the expected time.

Anika(candidate):

Then that’s the issue, I think. In the rain, delivery times get a long tail. Some orders take much longer. The mean moves a little, but the tail moves a lot. And the customer doesn’t feel the mean. They feel “it came 15 minutes after what you told me.” So I’d predict a high quantile instead. Train with pinball loss, at 0.9 maybe.

Ioana(interviewer):

Mm. Quantile loss. That’s basically MAE with a weight on it, no? You multiply the error by 0.9.

Anika(candidate):

Hmm, almost, but… not exactly, I think. MAE is the special case. It’s the 0.5 quantile: both sides get the same weight, so it’s MAE, just divided by two. For 0.9, the weight depends on the side. If the order comes later than the prediction, the error counts 0.9. If it comes earlier, it counts only 0.1. So late errors are nine times more expensive, and that pushes the prediction up.

Ioana(interviewer):

She writes on her paper for a moment. Late by ten minutes, cost nine. Early by ten, cost one. Yes. Smiles. You’re right, I said it badly. I was thinking of one weight for every row. I explain this to new people every year, and today I explained it wrong.

Anika(candidate):

Laughs. It’s close. The weighted-row idea is also useful. You could give rainy orders more weight. But that’s a different thing.

Disagreeing well

Ioana said something that sounded right and was a little wrong. Anika did not say “no, you’re wrong”. She said “almost”, then gave the exact rule with numbers that Ioana could check on paper. Then she gave Ioana credit for the part that was useful. Ioana checked, agreed, and laughed at herself. A good interviewer is happy to be corrected. See Interviewers are not always right.

Check the rule yourself

Pinball loss for the quantile τ (say 0.9): if the real time is later than the prediction, the loss is τ × the error. If it is earlier, the loss is (1 − τ) × the error. At τ = 0.5 both sides get 0.5, which is half of the absolute error, so it has the same best answer as MAE: the median. At τ = 0.9, the best answer is the time that 90% of orders are faster than. In LightGBM, this is objective="quantile" with alpha=0.9.

Ioana(interviewer):

Okay. So you have a model for the 90th percentile. How do you test it?

Anika(candidate):

Hmm. By time first, of course, train on older weeks, test on newer. Like my cameras. And maybe several windows, because rain is not every week. And then… coverage. For the 90th, about 90% should arrive before it.

Ioana(interviewer):

Overall?

Anika(candidate):

No, by slice. Dinner, rain, each city, maybe the busiest stores. In my paper, the average looked fine and four cameras were terrible. I’d expect the same here.

Ioana(interviewer):

Good. That’s very close to what we would do. Now, why 0.9?

Anika(candidate):

Hmm. It felt like a standard choice. Like a confidence level. Honestly, I picked it like a statistician.

Ioana(interviewer):

If we show the 90th percentile to everybody, the number in the app gets bigger. What happens?

Anika(candidate):

Fewer late orders. And… people see 45 minutes instead of 35, and some don’t order. Okay, so it’s a trade-off. Too low, people are angry. Too high, people leave. I don’t know how to measure the second part. Do you have data on how customers react to the number they see?

Ioana(interviewer):

Some. And operations has strong opinions. They want fewer support tickets, product wants more orders.

Anika(candidate):

Then 0.9 is not my decision alone, I think. Maybe 0.8 at quiet times is fine, or maybe you show a range. I’d need to talk with them. I haven’t done this kind of decision before. In research, I choose the metric, and that’s it.

Where this goes wrong

This is Anika’s edge. She treated the quantile as a statistics choice, not a business decision. The number in the app is a promise to a customer, and it has a cost on both sides. Anika saw it once Ioana asked, and she was honest that she has never made this kind of decision. But in this role, she would need to bring it up herself, early.

A stronger answer at the edge

Anika could have reasoned from the two costs, even without the data:

“0.9 is a guess. What I’d really want is two curves from past data: for each quantile, how many orders would be late, and how much longer the shown time gets. Then operations and product choose the point together. If we have any past changes to the shown time, like a buffer, I’d look at what happened to orders then. And I’d test the final choice online, because offline we can’t see who leaves the checkout.”

This answer turns “I don’t know” into a plan, and it puts the decision with the right people. See Thresholds and costs.

Ioana then asked about training/serving skew: the training data comes from the data warehouse, and the live features come from the feature store. Anika saw the risk, but said that in research the same code makes the training and the test data, so she had never had to think about it. Ioana told her about last year’s bug in one line: a feature came from a 15-minute batch table in training and from real-time counts live, and it took six weeks to notice. Anika suggested saving the features that the service really used and training on those, which the team is starting to do now.

Ioana then asked briefly about online testing. Anika suggested an A/B test by customer. When Ioana said that couriers are shared between orders, Anika saw at once that the two groups would affect each other. She had read about randomising by group in medical studies, and suggested splitting by zone. She did not know switchback tests, which also change the group over time, and said so.

Minutes 48 to 57: Anika's questions

Anika(candidate):

I have a few. First: how do you get the label? The real delivery time. Who records it?

Ioana(interviewer):

The courier taps “delivered” in their app. And it’s noisy. Sometimes they tap in the lobby of a building, before they go up. We think a minute or two in many cases. Big buildings, more.

Anika(candidate):

So the label is a bit early in tall buildings. And downtown Toronto is a lot of tall buildings. Interesting.

Anika also asked what she would do in the first three months (weather features, the move to logged features, and on-call one week in five after shadowing) and whether there is time for research. Ioana was honest: maybe 10% of the time in a calm week. The forecasting team, she said, has more freedom for research.

The good questions

The question about the label is a researcher’s question, and a very good one. It finds a real problem with Fennick’s data that most candidates never ask about. The question about research time tells Anika something important for her own decision. This is the interview as a conversation in both directions.

Minutes 57 to 60: the close

Ioana(interviewer):

Thank you, Anika. And thank you for the correction, I mean it. Next is the system design with Ben. Chantal will write to you this week.

Anika(candidate):

Thank you. I learned a lot about delivery today. And I’m going to email the ecology group tonight.

After the call: Ioana’s notes

Ioana writes her notes the same afternoon, against the job description.

Anika Sharma, ML deep dive

• Modelling and ML fundamentals: strong evidence. Very clear on loss functions; corrected me on pinball loss, and she was right. Moved from a transformer to gradient boosting for good reasons.
• Evaluation: strong evidence. Split by camera in her research, time split and coverage by condition on our problem without help. Very strong answer on slices.
• Problem framing: some evidence. Found the real issue (mean vs tail) fast. Treated the quantile as a statistics choice until I asked about the cost.
• Production, serving, latency: no evidence. Never ran a service. Her model has no monitoring and no owner. First idea ignored the 50 ms limit and the data available at checkout.
• Skew and feature pipelines: no evidence. Understood the risk when asked; never had to deal with it.
• Experiments: some evidence. Saw the courier interference at once; did not know switchbacks.
• Communication and honesty: strong evidence. Named her own habit (“I reached for what I know”). Corrected me kindly.

Recommendation: no for this role. This role owns a production model with on-call after a short shadowing period, and the gap in production is too big for the one vacancy. But her evaluation thinking is excellent. Ravi should tell her about the forecasting team’s scientist role next quarter. She asked about research time, and that team fits her better.

What happened next

Ben’s system design notes said the same: a careful design of the model and the tests, and almost nothing about serving, failures or rollout. The team decided not to move forward for this role.

Ravi called Anika himself. He gave her Ioana’s and Ben’s feedback, and told her about the forecasting team’s scientist role, which would open next quarter. He offered to introduce her to the forecasting manager. Anika thanked him and said she was interested.

But three weeks later, Anika got an offer as an applied scientist at a company that builds crop monitoring from satellite images. The role used her image work directly, and it had a clear path to production with an engineer at her side. She accepted. She wrote to Ravi to say thank you, and to say that the question about the label was the one she now asks in every new project.

This is one realistic ending, not a promise. A PhD with no production work can get an ML engineer offer at a team that has more engineers around the model. Luck is part of it.

What to take from this

• Say your gap once, then show your work. Anika said “I haven’t worked in production” in the first minute, without apology. Then she spent the time on what she had done.
• Choose the model for the data and the limits, not for your habits. A transformer is a fine tool, but there was no route at checkout and only 50 ms. When you see that you reached for what you know, say it, and think again.
• Your research habits are valuable. Split the data the way the model is used, look at slices, and question the label. These came from Anika’s PhD, and they were the best part of her interview.
• A predicted number is a promise. The quantile, the threshold and the metric are business decisions too. Ask who pays for each kind of error.
• You may correct the interviewer. Do it with the exact rule and a small example they can check, and give credit for what was right.

If your profile is like Anika’s

1. Put one model behind a real service. Train a small model on table data, serve it with an API, and measure the p99 latency yourself. See Serve a house price model and Package artifacts.
2. Work with table data and gradient boosting. Many ML engineer jobs are mostly tables. Practise until you can explain when boosting beats a neural network, and why. See Ensembles and When a simple model is better.
3. Train for a quantile and check coverage by slice. The bike-share demand case study and the house price case study both end with this exercise. See also Slice analysis.
4. Add monitoring to your research model. Count predictions per class each week and compare them with the last period. Write who owns it when you leave. Then you have a real production story. See Plan production checks.
5. Practise the cost question. For each project, write down who pays for each kind of error, and how much. See Thresholds and costs.
6. Decide what you want. If you want time for research, ask about it, as Anika did. Applied scientist and forecasting roles may fit better than an on-call engineering role. Both paths are good.