The software engineer

Think of a good restaurant kitchen. One person plans the menu and the recipes. Another person makes sure that 300 plates leave the kitchen hot, on time, every night. Both jobs are hard. But if the recipe is wrong, the fastest kitchen in the city still serves a bad meal.

Many people who apply for Machine Learning Engineer roles are the second person. They are backend engineers. A data scientist built a model, and they put it into production: the nightly training jobs, the fast service, the cache, the A/B test. They liked the work, and now they want the whole job.

This profile is strong where many ML candidates are weak. The usual gap is the other half: what should the model predict, how do you measure it, and how do you know that the test is fair? Tunde Bakare is this candidate. Watch where his engineering makes the conversation better, and where Ioana takes him to the edge of his ML judgement.

TB

The candidate

Tunde Bakare

Backend engineer, 6 years, moving into machine learning

• 6 years in backend
• 4 years Lagos, 2 years Toronto
• Python and Go
• On call for 5 years
• Toronto

• Senior Software Engineer, Marlow & Pike (home goods retail, Toronto), 2024 to now
• Designed and deployed an ML recommendation system (“You may also like”), increasing revenue by 12%
• Built the serving layer: FastAPI and Redis, about 2 million requests a day, p99 under 40 ms
• Built the nightly training pipeline (Airflow, BigQuery) and the A/B test assignment and logging
• Software Engineer, Orinpay (payments, Lagos), 2020 to 2024. Payment routing and transaction APIs in Go and Python; on call from 2021
• Python, Go, FastAPI, Redis, Kafka, PostgreSQL, Kubernetes, Airflow
• Online certificate in machine learning (2023)

What Ioana sees before the call

Ioana Petrescu is a Staff ML Engineer at Fennick. She built the ETA service, and she found last year’s training/serving skew bug. She reads Tunde’s CV next to the job description.

The engineering evidence is strong. Two million requests a day with a p99 under 40 ms is close to Fennick’s own load, and he has been on call for years. Ravi, the hiring manager, wrote: “Very clear on systems. Not sure about the modelling.”

One line gets a mark: “Designed and deployed an ML recommendation system, increasing revenue by 12%”. Who designed the model? And 12% of what? She writes down three things to find out:

• Which decisions about the model did Tunde make himself, and which did he only implement?
• Does he understand why the model was trained and tested the way it was?
• On the ETA problem, does he go further than the service? Can he reason about what the model should predict?

Terms on this page

• ETA (estimated time of arrival): here, the time from the order to the customer’s door that the app shows before the customer pays.
• p99 latency: 99% of requests finish faster than this time. The slowest 1% take longer.
• Latency budget: the most time that one request may take, shared between all the steps inside it.
• Feature store: a system that computes model inputs (features) and serves them. It has an offline part for training and an online part for live requests.
• Training/serving skew: a feature is computed one way for training and another way in production. The model learns one meaning and gets another.
• Co-occurrence: how often two products are bought together.
• Re-ranker: a second model that puts a short list of candidates in a better order.
• Time-based split: you train on older data and test on newer data, the way the model will really be used.
• Quantile (or percentile): the 90th percentile of delivery times is the time that 90% of orders are faster than.

The interview

Minutes 0 to 5: the opening

Ioana(interviewer):

Hi Tunde. Can you hear me okay? I’m Ioana, I’m a staff ML engineer in Delivery Intelligence. I built the ETA service, so be careful what you say about it.

Tunde(candidate):

Laughs. Okay, noted. Hi, yes, I hear you well.

Ioana(interviewer):

So, the plan. First you tell me about a model you put in production, and I ask many questions. That is normal, it’s not a bad sign. Then a real problem from our side. At the end, your questions. One rule: no AI assistants for this round. If you want to write on paper, that’s fine.

Minutes 5 to 25: the project, then deeper

Ioana(interviewer):

So, tell me about the recommendation project.

Tunde(candidate):

Okay. So, at Marlow & Pike, on every product page we had a box, “You may also like”. Before, it was just the bestsellers from the same category. Very boring, same eight products for everybody. Our data scientist, Adaeze, built a better model. And my job was to make it work in production.

So how it works: every night, a job counts which products are bought together in the last 90 days. For each product we keep the top 200 or so. Then when a user opens a page, the service takes those 200 and a small model, a logistic regression, puts them in order. We show the top eight. I built the nightly jobs, the service, the cache in Redis, and the A/B test plumbing.

Ioana(interviewer):

Okay. What did the model predict?

Tunde(candidate):

It returns a list of eight product IDs, with a score. And if the model is not available, we return the old bestseller list, so the page never breaks.

Ioana(interviewer):

Sorry, I mean inside. The logistic regression. In training, what was the label? What is a “one” and what is a “zero”?

Tunde(candidate):

Ah, okay, the label. A click, I think. If the user clicked the recommended product, it’s one. Shown and not clicked, zero. Yes, click in the same session.

Ioana(interviewer):

Why click? The business wants sales, no?

Tunde(candidate):

Adaeze decided that. I think because clicks are many and purchases are few. With purchases you have, I don’t know, maybe one for every hundred clicks. Too little data for the model. But I’m not sure that was the only reason.

What the interviewer is checking

Tunde first heard the question as a question about the service. That happens, and Ioana simply asks again, more precisely. She wants to know if Tunde understands the choices inside the model. His reason for clicks is a good guess, and he says that it is a guess.

Ioana(interviewer):

Okay. And how did Adaeze test the model before it went live? Offline.

Tunde(candidate):

She used AUC. And another one, something about the top eight… precision at eight, I think. She took the logs, 80% for training, 20% for testing. Random, I think.

Ioana(interviewer):

Is there a problem with a random split here? Take a moment.

Tunde(candidate):

Hmm. The same user can be in training and in test? But… I’m not sure that is bad. The same user also comes back in real life.

Ioana(interviewer):

Think about how you use it. The model trains Monday night. When does it make predictions?

Tunde(candidate):

Tuesday. Oh, okay. So in real life it always predicts the future. But with a random split, the test has days that are inside the training period. In November, before Black Friday, the products change every day. So the random test looks better than real life. I see. Adaeze should test on the days after training. I never asked her about it, to be honest. I saw it as her part.

Where this goes wrong

“I saw it as her part” is honest, and it is the main concern for this role. At Fennick, the ML engineer owns these decisions. Tunde reached the right idea quickly once Ioana gave him the Monday and Tuesday example. But a random split on data that changes over time is one of the most common mistakes in ML, and the role needs someone who sees it alone. See Choose a split.

Ioana(interviewer):

Your CV says 12% more revenue. Tell me about that number.

Tunde(candidate):

Yes… so. The A/B test was 50/50 by user ID, four weeks. In the new group, add-to-cart from the product page went up about 12%. Revenue, the marketing team calculated something, but I don’t know how. So “revenue” is not correct. I should change that line, honestly.

Ioana(interviewer):

Okay, thanks. Then what happened after launch? Did something break?

Tunde(candidate):

Yes. This one I remember well. About five months after launch, Adaeze added two new features. She retrained, the training metrics were fine, we deployed. But the click rate went down, maybe 10%, from the first day. Nobody saw it for nine days. She saw it on her weekly dashboard, I think.

The problem: in the training code the features were in one order, and in my serving code, after her change, two of them were in a different order. Price ratio and same-category, swapped. The model still ran. No errors. Just wrong.

Ioana(interviewer):

Smiles. I know this kind of bug. What did you change after?

Tunde(candidate):

So, we put the feature code in one module, and training and serving both import it. No copy.

Ioana(interviewer):

Mm-hm.

Tunde(candidate):

And the model file now saves the feature names, so the service checks them when it starts. If they don’t match, it doesn’t start. Oh, and a test in CI, we compare a thousand logged requests, both paths.

Why this works

This is the strongest part of Tunde’s project. He did not stop at the fix. He made the same bug hard to deploy again: one shared code path, a check when the model loads, and a test that compares the two paths. A bug of this kind cost Fennick six weeks last year.

Ioana(interviewer):

Good. Last thing on this. Before Adaeze’s weekly dashboard, who would know if the model got worse?

Tunde(candidate):

Hmm. My alerts were for the service: latency, errors, how often we use the fallback list. If the service is slow, I know in two minutes. If the model is bad but fast… nobody, until Adaeze looks. That’s a gap, yes.

Minutes 25 to 48: the Fennick problem

Ioana(interviewer):

Okay, now a real problem from our side. Customers see a delivery time in the app before they pay. At dinner time and when the weather is bad, our estimate is too optimistic. On normal days about 9% of orders arrive more than ten minutes late compared to the estimate; on those days it’s about 18%. People are angry, support tickets go up. How would you approach this?

Tunde(candidate):

Okay. Can I ask first, what does the app show? One number?

Ioana(interviewer):

One number. “35 minutes”, like this.

Tunde(candidate):

My first question is maybe strange. Is it a model problem, or a data problem? Because dinner time is also when the systems are busy. If the feature store is slow at dinner, maybe the service times out and… sorry, I lost my point. Yes. It times out and uses default values. Then the model is fine, but it gets old or empty inputs exactly when it matters.

Ioana(interviewer):

Hm. Interesting. How would you check that?

Tunde(candidate):

Log it. For every prediction: which features came from the fallback, and how old each real-time feature was. Then look at it by hour. If at 6 p.m. the “orders at the store” value is 20 minutes old, that’s the problem.

Ioana(interviewer):

We did not check freshness by hour, actually. I will write that down. But we had something close last year. The feature “orders waiting at the store”. For training it came from a batch table, updated every 15 minutes. In production, it came from real-time counts. Offline everything looked good, online it was worse. Six weeks before anybody saw it.

Tunde(candidate):

Ah. So the same kind of bug as mine, but quieter. At dinner the real-time count is higher than the 15-minute value, because the queue grows fast. So the model sees a number it never saw in training.

The fix is the same idea: one definition of the feature. Ideally you log the feature values at prediction time, the exact values the service used, and you train on those logs. Then training sees what production sees. It takes some weeks to collect enough logs, but after that the problem is gone.

Why Ioana likes this

Tunde connects his own bug to Fennick’s bug in one step. Then he proposes the strongest fix: train on the feature values that the service really used, logged at prediction time. Many teams call this “log and wait”, because you need to wait until you have enough logs. Ioana did not have to lead him here.

Ioana(interviewer):

Good. Now, we also want to add weather. Today we don’t have it. How do you add it?

Tunde(candidate):

Hmm, not while the customer waits, no. I would have a job that gets the weather every ten minutes for each zone and writes it to the feature store. The service reads it like any other feature. If the value is older than, let’s say, 30 minutes, we still use it but we log a flag.

And for training, keep the history of what the job wrote. Not the real weather later. Because at 5 p.m. you only have the forecast, and the rain at 6 p.m. can be different. If we train on the real rain, it’s leakage, the model learns from the future.

Ioana(interviewer):

Yes. And the latency? The app calls us on every checkout page view, about 1.5 million times a day. Our budget is 50 milliseconds at p99.

Tunde(candidate):

How is it spent today? Do you know?

Ioana(interviewer):

Feature reads are most of it. Maybe 20, 25 ms at p99. The model itself is very fast, a few milliseconds.

Tunde(candidate):

Then weather is one more read. If it’s in the same store and you read all features in one call, it costs almost nothing. If it’s a separate call, do it in parallel, not one after the other. And every read gets a timeout, maybe 30 ms, because we must stay under 50. After that, use a default and log it.

Ioana(interviewer):

Mm-hm.

Tunde(candidate):

Also 1.5 million a day is only around 17 per second on average. Dinner is maybe five or ten times more?

Ioana(interviewer):

Less, maybe three, four times.

Tunde(candidate):

Okay, so the load is fine. The p99 is the hard part, not the volume.

Check the arithmetic

1.5 million calls a day divided by 86,400 seconds is about 17 calls per second. Tunde’s quick estimate is right, and it leads to the right point: the hard part is the slowest 1% of requests, not the volume.

Ioana(interviewer):

Okay. Very good on the service. Now let’s look at the model itself. Let’s say the inputs are perfect. Fresh, no skew, weather is there. Do you think the late orders go away?

Tunde(candidate):

Hmm. Better, I think. But not gone, because… Pause. Sorry, I’m thinking. Even a perfect model makes errors, because some things you can’t know. A courier has a flat tire. But I don’t know if that is 9% or 18%.

Ioana(interviewer):

Okay, let me ask it differently. The model is trained to be right on average. In the rain, delivery times spread more. Some orders take 30 minutes, some take 60. If the model is right on average, what does the customer see?

Tunde(candidate):

A number in the middle. And a lot of people get their order later than that number. More in the rain, because the spread is bigger. Okay. So we add a buffer? Plus five minutes at dinner time, plus ten in the rain?

Ioana(interviewer):

Operations did that in March, actually. Five minutes more when it rained, in two cities. The late rate went down a little. But on quiet streets, close to the store, the estimate became too long. Orders went down about one or two percent.

Tunde(candidate):

Right, because the buffer is the same for everybody. It should depend on the order. But I don’t know how a model does that. Predict the average and also… the spread?

Ioana(interviewer):

That is one way. Another way: you don’t ask the model “what is the average time”. You ask “what time will 90% of orders be faster than?”

Tunde(candidate):

Like p99 latency, but for delivery. The model predicts a percentile directly? I didn’t know you can train for that. How does it learn it?

Ioana(interviewer):

There is a loss for it, quantile loss. If the order is later than the prediction, it costs a lot; if it is earlier, it costs a little. So the model learns to predict higher. LightGBM has it built in.

Tunde(candidate):

Okay. This is new for me. I would need to read about it.

How Ioana goes deeper

This is Tunde’s edge. He does not know quantile regression, and he says so. Ioana rephrases twice, the second time with a simple picture: the time that 90% of orders are faster than. Tunde connects it to p99 latency, which he knows well. Using what you know to understand something new is what an interviewer hopes to see at the edge.

Ioana(interviewer):

That’s fine. Let’s use it. Say we have a model that predicts the 90th percentile. How do you check if it’s good? Offline, before we ship.

Tunde(candidate):

Mean absolute error? Hmm, no, wait. If it predicts high on purpose, the error is always big. That’s not fair. Hmm… for p99 latency, I check that 99% of requests are really under it. So here: count how many orders arrived before the prediction. It should be about 90%.

Ioana(interviewer):

And if it’s 90% overall?

Tunde(candidate):

Then check only dinner. Only rain. Only each city. Because the average can be 90% and rain can be maybe 80%. That’s our problem exactly.

A stronger answer at the edge

Tunde got to the right check, coverage by condition, but Ioana had to lead him to the idea of a percentile. A candidate with more ML experience would start from the customer and the target:

“Before the model: what do we promise the customer? Late is worse than early, so maybe we show a high percentile, not the average. But too high and people don’t order. So I’d track two numbers: the late rate, and how often people leave the checkout.”

You do not need to know every method. You need to ask what the number on the screen promises. See Regression metrics.

Ioana then asked how Tunde would test the new model online. He proposed his familiar A/B test, split by customer ID. Ioana pointed out that couriers are shared: a customer in group A and a customer in group B may wait for the same courier, so the two groups are not independent. Tunde saw the problem quickly. He suggested a split by city, but agreed that 14 cities are too few to compare. Ioana told him that the team splits by zone and time window, and they moved on.

Minutes 48 to 57: Tunde's questions

Tunde(candidate):

I have a few. First: the retraining. Who owns it? How does it run?

Ioana(interviewer):

Laughs a little. Honest answer? It’s a cron job, weekly. One engineer really understands it. It’s on our list.

Tunde(candidate):

Okay. That’s the kind of thing I like to fix, to be honest. Second: on-call. The posting doesn’t say.

Ioana(interviewer):

One week in five, for the ETA service. It’s not a lot of pages, maybe two or three in a bad week. Mostly feature store problems.

Tunde(candidate):

The video freezes for a moment. …sorry, you froze. Did you say two or three a week?

Ioana(interviewer):

In a bad week, yes. Most weeks less.

Tunde(candidate):

Okay, good. Last one. Is there a platform team, or does Delivery Intelligence run its own infrastructure?

Ioana(interviewer):

There is an ML Platform team. They own the feature store and the serving platform. We build on top.

The good questions

Tunde asks about things that a posting usually hides: who owns retraining, on-call, and where the platform ends. Each answer tells him something real about the job. This is the interview as a conversation in both directions.

Minutes 57 to 60: the close

Ioana(interviewer):

Thank you, Tunde. I enjoyed the part about the feature order bug, really. Next is the system design with Ben. Chantal will contact you this week.

Tunde(candidate):

Thank you. And I’m going to read about quantile loss tonight.

After the call: Ioana’s notes

Ioana writes her notes the same afternoon, against the job description.

Tunde Bakare, ML deep dive

• Low-latency serving: strong evidence. Recommendation service at about 2M requests a day, p99 under 40 ms. On our problem: timeouts, parallel reads, quick load estimate, weather outside the request path.
• Feature pipelines and training/serving skew: strong evidence. Found his own skew bug and made it hard to deploy again. Proposed training on logged features without help. Better than most candidates on this.
• Problem framing (what to predict): concern. Did not question the target in his project (“Adaeze decided”). On our problem, needed two hints to reach a percentile target.
• Evaluation: some evidence. Did not see the problem with a random split until I gave an example. Then reasoned well: coverage, checked by condition.
• Experiments: some evidence. Built A/B plumbing. Did not think of shared couriers, but understood quickly.
• Communication and honesty: strong evidence. Corrected his own CV line about revenue without being pushed.

Recommendation: no for this role, for now. This role owns the ETA model, and today Tunde would need someone else to make the model decisions. But he is one of the best engineers I’ve talked to this year. I recommend that Ravi talks to ML Platform. Their open role is close to his strengths.

What happened next

Ben’s system design notes said the same thing: the clearest design of the week on serving and failures, and weak on how to evaluate the model. In the debrief, the team agreed. The one vacancy needed someone to own the model, and another finalist was stronger there.

Ravi called Tunde himself. He shared the feedback honestly: very strong engineering, not enough ML depth for this team now. Then he asked if he could pass Tunde’s profile to the ML Platform team, which had an open senior role. Tunde said yes.

ML Platform reused his Fennick interviews and added one platform design round. Two months later, Tunde joined ML Platform. His first project: replace the weekly retraining cron job with a real pipeline.

This is one realistic ending, not a promise. Not every company has a platform team, or passes your profile on. Luck is part of it.

What to take from this

• Strong engineering is real ML work. Skew, freshness, latency and fallbacks broke Fennick’s model last year. Tunde’s skill here is rare, and Ioana noticed it.
• Know the decisions inside your project, even if you did not make them. “The data scientist decided” is honest, but you should still know why the label, the split and the metric were chosen. Ask them before your interview.
• Fix your CV numbers before the interviewer does. Tunde changed “revenue” to “add-to-cart” by himself. That made the rest of what he said more trustworthy.
• Use what you know to reach what you don’t. Tunde got from p99 latency to a percentile target, and from there to coverage. That is good behaviour at the edge.
• Ask what the number on the screen promises. The model and the metric follow from that answer.

If your profile is like Tunde’s

1. Interview your data scientist. For each model that you served, write down the label, the split, the offline metric, the baseline, and why. If you don’t know, ask the person who chose. If you can’t ask, decide what you would choose and why. See Translate a need and Define a baseline.
2. Learn the time-based split until you can explain it with your own example. Use a dataset with dates, and compare a random split with a split by date. See Choose a split and Features and target timing.
3. Train one model for a percentile. Take a regression dataset, train one model for the median and one for the 90th percentile, and check coverage by group. The bike-share demand case study is a good place to start.
4. Add model monitoring to your service skills. You already watch latency and errors. Add a check on the predictions and on the inputs. See Plan production checks.
5. Prepare your engineering stories as ML stories. For the skew bug, say what the model learned, what it got in production, and how you stopped it from happening again. Explain decisions helps you say this out loud.