The platform engineer

Think of the sound engineer at a concert. She knows every cable, every speaker and every microphone. If something fails during the show, she fixes it in seconds. Without her, there is no concert. But she does not choose the songs, and she does not decide how they are played.

Many MLOps engineers are in this position. They came from DevOps. They build the pipelines, the registries, the deployment templates and the dashboards that let data scientists put models into production. Often they run more models than anyone else in the company. But someone else chooses what each model predicts, how it is measured, and when it is good enough.

They apply for ML engineer jobs because the title looks like the next step, and often because they want to make those decisions themselves. Follow Wei through the ML deep dive. Watch where his platform experience is excellent, and what happens when Ioana asks him to think like the owner of a model.

WZ

The candidate

Wei Zhang

MLOps engineer, 3 years, after 7 years in DevOps

• 10 years in infrastructure
• 3 years in MLOps
• CKA certified
• Toronto, Ontario

• MLOps Engineer, Harbourline Insurance, Toronto (2023 to now). Built end-to-end MLOps platform serving 40+ models on Kubernetes (EKS): Kubeflow Pipelines, MLflow, KServe, Feast.
• Reduced model deployment time from about 6 weeks to 2 days with standard CI/CD templates (GitHub Actions, Argo CD).
• Introduced a feature store (Feast) for 12 models, with point-in-time correct training data.
• Model monitoring with Prometheus, Grafana and Evidently. Infrastructure as code with Terraform.
• Senior DevOps Engineer, Tallis Freight, Toronto (2016 to 2023). Moved 60 services to Kubernetes; CI/CD; on-call lead for two years.
• MSc Computer Science (2015). Certified Kubernetes Administrator; AWS Solutions Architect Professional.
• Python, Go, Bash, Terraform, Docker, Kubernetes, Kubeflow, MLflow, KServe, Feast, Airflow, Spark, Prometheus, Grafana, Evidently, PyTorch, scikit-learn, XGBoost.

What Ioana sees before the call

Ioana likes this CV, and she is careful with it. Fennick’s platform is exactly this kind of stack: Kubernetes, MLflow, an in-house feature store. Wei could fix the retraining job that only one engineer understands. But the job is to own the ETA model, and the CV says almost nothing about model decisions. The skills list ends with PyTorch, scikit-learn and XGBoost, and she cannot tell from the CV if Wei used them or deployed them.

She writes down what she wants to find out:

• “40+ models.” How many are really used? What did he build, and what did the data scientists build?
• Pick one model. Can he explain the label, the metric and why? Who decided?
• Monitoring. When the dashboard showed drift, what happened next?
• On the Fennick problem: does he go to what the model should predict, or only to how it runs?

Terms on this page

• MLOps: the work of running ML models in production: pipelines, deployment, monitoring and retraining.
• Kubeflow Pipelines: a tool that runs ML workflows (for example: get data, train, evaluate, register) as steps on Kubernetes.
• MLflow model registry: a place that stores model versions and marks which one is in production. Current MLflow versions use aliases such as “champion” for this.
• Champion and challenger: the model in production (champion) and a new model that must beat it before it replaces it (challenger).
• Feature store: a system that computes features once and serves them both for training and for live predictions. A point-in-time join gives each training example only the feature values that existed at that moment.
• Canary release: a new version gets a small part of the traffic first, so problems hit few users. Shadow mode: the new model runs on live traffic, but nobody sees its output.
• Quantile prediction: the model predicts a value that, for example, 90% of outcomes will be below. For a delivery time, it means “90% of orders arrive before this”.
• Training/serving skew: a feature is computed one way for training and another way in production.

The interview

Minutes 0 to 5: the opening

Ioana(interviewer):

Hi Wei. Sorry, I’m one minute late, the meeting before took longer. So, I’m Ioana, staff ML engineer. I built the ETA service, the delivery time in the app. Today: you tell me about a model you worked on in production, I ask many questions. Then a problem from our side, then your questions. And please, no AI assistants in this round. Notes on paper are okay.

Wei(candidate):

Sounds good. Just so you know, my side is more the platform. But I’m happy to go deep.

Ioana(interviewer):

That’s fine. We’ll see where it goes.

Minutes 5 to 25: the platform, then one model

Wei(candidate):

So, when I joined Harbourline, every model went to production differently. A data scientist finished a notebook, then they opened a ticket, and someone in IT built a server for it. It took like six weeks, and nobody could say which code made which model. So I built a standard path. Training runs as a Kubeflow pipeline. Every run logs to MLflow. If the new model passes the checks, it’s registered, and Argo CD deploys it to KServe. Features come from Feast. Now a data scientist can go from a merged pull request to production in about two days.

Ioana(interviewer):

Nice. Your CV says 40 plus models. How many are really used every day?

Wei(candidate):

Hmm. Good question. Around 40 are deployed. Really used every day… maybe 15? Some are old versions that teams forgot to turn off. I keep asking them.

Ioana(interviewer):

Okay. Let’s take one of the fifteen, the one you know best.

Wei(candidate):

The auto claims fraud model. When a customer files a car claim, it gives a score, and high scores go to the investigation team.

Ioana(interviewer):

What’s the target?

Wei(candidate):

It’s on KServe, in our production cluster, two replicas with autoscaling.

Ioana(interviewer):

Ah, sorry, I mean… the label. What does the model predict? What is a “yes” in the training data?

Wei(candidate):

Oh, sorry, I heard deployment target. He laughs. The label… a claim that was fraud. I think it’s the claims that the investigation team confirmed as fraud.

Ioana(interviewer):

Only the ones they investigated?

Wei(candidate):

I think so, yes.

Ioana(interviewer):

So a fraud that nobody investigated, it’s a “no” in the data.

Wei(candidate):

Hmm. Yes, if it works like I think. I never thought about it. I don’t know how the data scientists handled that. Maybe they did something, I don’t know.

How Ioana goes deeper

Ioana asks one question about the label, and it opens a real problem. If only investigated claims can be “fraud”, the model learns what investigators looked at before, not all fraud. People who own a model think about this every day. Wei sees the problem when it is pointed out, but he has never had to think about it.

Ioana(interviewer):

Okay. How does a new version of this model get to production?

Wei(candidate):

This part I know well. The pipeline trains on a data snapshot, then evaluates on a holdout set. There’s a gate: the new model’s AUC must not be lower than the champion’s minus 0.01. If it passes, it gets the “challenger” alias in MLflow. Then a data scientist approves, and it becomes champion. Argo CD rolls it out.

Ioana(interviewer):

Who chose AUC?

Wei(candidate):

The data scientist on that model. I built the gate so each team can set its own metric and threshold. She gave me AUC and 0.01.

Ioana(interviewer):

If you owned this model, would you keep AUC?

Wei(candidate):

Pause. I don’t know. AUC is the standard one, no? Honestly, I never asked her why. I should have.

Ioana(interviewer):

Maybe a hint. How many claims can the investigation team look at in a day?

Wei(candidate):

Not many. Maybe 30, 40? So… what matters is the top of the list? If the top 40 are good, the rest doesn’t matter so much. I’m not sure what the metric for that is called.

"The data scientists decide"

In a platform role, this is a normal way to work. In an ML engineer interview, it is the answer that hurts most. The job description asks for someone who owns the model. Wei reached a good idea with a hint, but he had never asked why the metric was chosen. If you work next to models, ask the owners “why?” for every choice. Then you can explain it in an interview, even if you did not decide.

Ioana(interviewer):

Okay. Now something different. Imagine the fraud model gives a strange score in production, and the investigators complain. Can you tell me exactly which data and code made that model?

Wei(candidate):

Yes, completely. Every prediction is logged with the model version. The version points to the MLflow run. The run has the Git commit, the image… and the data snapshot path.

Ioana(interviewer):

And the data is still there?

Wei(candidate):

I think two years. I’d have to check. But yes, I can rebuild the exact model. Last time someone asked, it took me maybe two hours.

Ioana(interviewer):

Good. That’s better than what we have.

Where Wei is strong

This is a clear answer, and it comes from real work. Wei knows the chain from one prediction back to the code and the data, and how long it takes to rebuild. Many ML candidates cannot answer this. For an auditor, or for an incident at 2 a.m., this chain is very valuable. Ioana says so.

Ioana(interviewer):

And monitoring. Your CV says Evidently. What happened when it showed drift?

Wei(candidate):

The reports go to the data science team every week. Honestly? Often nothing. Many reports were red all the time, so people stopped reading them.

Ioana(interviewer):

Why were they always red?

Wei(candidate):

We used the default drift check on every feature. But many features change every week anyway, paydays, holidays. So almost everything was red.

Ioana(interviewer):

Who should decide which change matters?

Wei(candidate):

Someone who knows which change matters for the model. Pause. That’s the owner. Which was not me.

An honest answer about his own system

Wei could have said “we monitored drift with Evidently”. He said that nobody read the reports, and he explained why. That honesty makes his strong answers more believable. And his last sentence shows that he already sees the gap between running a model and owning it.

Minutes 25 to 48: the Fennick problem

Ioana(interviewer):

Okay, now a real problem from our side. Customers see a delivery time in the app before they pay. At dinner time and when the weather is bad, our estimate is too optimistic. On normal days about 9% of orders arrive more than ten minutes late compared to the estimate; on those days it’s about 18%. People are angry, support tickets go up. How would you approach this?

Wei(candidate):

Okay. First I want to see it. Do you have a dashboard of late orders by city and by hour, with alerts?

Ioana(interviewer):

We have the error after delivery, in a report the next morning. No alerts.

Wei(candidate):

Then that’s the first thing. Late rate by city and hour, in real time, with an alert when it goes over, say, 12%. And the model is retrained weekly, right? I would retrain daily. Make it a pipeline with a champion and challenger gate, so it’s safe to do every day.

Ioana(interviewer):

Okay. Would retraining every day fix the dinner problem?

Wei(candidate):

It would use fresher data…

Ioana(interviewer):

But dinner happens every day.

Wei(candidate):

Right. Pause. So the weekly data already has seven dinners in it. Hmm. Then no, probably not. Unless something changed this week, like a new store. I think I went to the part I know.

Going to the tools

Wei’s first answer is about dashboards and retraining frequency. Both are useful, and both are about how the model runs. Neither changes what the model predicts. When you hear a model problem, first ask what the model should do, then how to run it. Wei noticed it himself, which is good.

Ioana(interviewer):

Let’s try something different. Imagine there are no data scientists. You own this model. The product manager comes to you and asks: what number should we show the customer? What do you say?

Wei(candidate):

The most accurate one?

Ioana(interviewer):

Okay. What is “accurate” when one order is ten minutes late, and another is ten minutes early?

Wei(candidate):

For the error, they are the same. For the customer… no. Late is worse. Early, nobody complains.

Ioana(interviewer):

So what do you show?

Wei(candidate):

Maybe add a buffer? At dinner time, add some minutes.

Ioana(interviewer):

How many minutes? And in the rain?

Wei(candidate):

Hmm. I would take last month’s dinner orders, and choose a buffer so that most of them would be on time. Like 90%. And a different one for rain. And snow. And maybe per city… Pause. But then the buffers become a second model on top of the model. That’s not nice to maintain.

Ioana(interviewer):

Yes. That’s the right worry. What you described is the 90th percentile. You can train the model to predict that point directly, with weather inside the model: quantile regression. Have you used it?

Wei(candidate):

I’ve seen the word. Maybe in a config file. I couldn’t explain how it works.

A stronger answer, from where Wei was

Wei did not need to know quantile regression to give a strong answer. He needed to start from the customer, not from the pipeline:

“If I own it, I first ask what the number means for the customer. Late is worse than early, so I would not show the average. I’d show a time that most orders beat, say 85 or 90%, and ask product what longer times cost in orders. Then I’d check it separately for dinner and for rain. I know there are ways to train for that directly; I’d have to read about them. Then the retraining and the dashboards, which I can build fast.”

This answer is honest about the gap and still shows the owner’s way of thinking.

Ioana then moved to production questions, and here Wei was quick again.

Minutes 38 to 48: serving, skew and testing

Ioana(interviewer):

The app calls the ETA service on every checkout page view. About 1.5 million calls a day. Our latency budget is 50 milliseconds at p95. Sorry, p99. At p99.

Wei(candidate):

Okay. Is that 50 measured at the service, or at the app?

Ioana(interviewer):

At the service.

Wei(candidate):

Good, then the network to the phone is not in it. The model is fast; the risk is the feature lookups. I’d put a timeout on each lookup, and a fallback. How long, I’d first look at how fast the lookups are today, at p99, and stay above that. If a real-time feature doesn’t come back, use the last known value, or a default. And alert on how often the fallback is used, because a silent fallback is a quality problem.

Why this works

Wei asks where the 50 ms is measured before he designs anything. That one question shows years of production work. His design is also right for the user: the checkout page must never wait for a slow feature. And he knows that a fallback that nobody counts hides problems.

Ioana(interviewer):

Okay. Last year we had a skew bug. A feature came from a 15-minute batch table in training and real-time counts live. Six weeks before we noticed.

Wei(candidate):

Six weeks, that’s long. So two different sources for one feature. A feature store helps if both paths really use one definition. But if the batch job and the stream compute “orders waiting” differently, the store doesn’t stop that. So I’d also log the values the service really used and compare them with the training values for the same orders.

Ioana(interviewer):

Okay. Last question on this. You have a new model. How do you test it with real customers?

Wei(candidate):

First shadow mode. The new model runs on all traffic, we log its prediction, nobody sees it. After delivery we compare both models with the real times. Then a canary, 5% of traffic, and watch errors and latency.

Ioana(interviewer):

Shadow is a good first step. The canary tells you the service is safe. Does it tell you if customers are happier?

Wei(candidate):

The video freezes for a moment. …sorry, you froze. Can you say the last part again?

Ioana(interviewer):

Does the canary tell you if customers are happier? Fewer late orders, the same number of orders?

Wei(candidate):

Not really. 5% for a few days, it’s for safety. For the effect you need an experiment. At Harbourline, I did the traffic split with feature flags, and the data scientists did the analysis. I’ve never designed one myself.

Ioana(interviewer):

Okay. Here it is also harder, because the couriers are shared between customers. But let’s leave that.

Shadow mode

Shadow mode is a strong first step for the ETA model, and Wei proposed it on his own. The true delivery time arrives within an hour, so a few days in shadow give a fair comparison on live data, with no risk to customers. What shadow mode cannot show is how customers react to a different number. That needs an experiment, and Wei says honestly that he has never designed one.

Wei asked how the retraining runs today. Ioana told him honestly: a cron job that one engineer really understands. He also asked about on-call. It is one week in five, and he said that was fine after years of it.

Minutes 48 to 57: Wei's questions

Wei(candidate):

About the retraining. I could make that a proper pipeline in a month. With the gate, the registry, all of it.

Ioana(interviewer):

I believe you. Honestly, in this role that’s maybe 20%. Most of the work is decisions about the model, like my questions today.

Wei(candidate):

Pause. Okay. Thank you, that’s useful. And honestly, the decisions are the part I want to learn. It’s why I applied. I just don’t have it yet.

Hearing a hard answer

Ioana told Wei, kindly and directly, that his strongest skills are a small part of this job. Wei did not argue or oversell. He agreed, and he said what he wants. This is not a weak moment. It is the moment when both sides learned the most. See An interview goes in both directions.

Minutes 57 to 60: the close

Ioana(interviewer):

Thank you, Wei. Chantal will tell you the next steps this week.

Wei(candidate):

Thank you. And the label question about the fraud model, I’m going to ask about it on Monday.

After the call: Ioana’s notes

Wei Zhang, ML deep dive

• Deployment, reproducibility, reliability: strong evidence. Full chain from one prediction to code, image and data snapshot; rebuilt a model in about two hours. Asked where the 50 ms is measured; timeouts and a counted fallback.
• Training/serving skew and feature stores: strong evidence. Knows a feature store alone does not stop two different computations; log served features and compare. Proposed shadow mode on his own.
• Monitoring: some evidence. Built it. Was honest that nobody acted on it, and saw why: thresholds need an owner.
• Model ownership (label, metric, “good enough”): no evidence. Did not know how the fraud labels were made or why AUC was chosen. “The data scientists decide.”
• Problem framing on the ETA problem: concern. Went to dashboards and retraining first. With help, reached a buffer and then the idea of a percentile. Did not know quantile regression.
• Online experiments: no evidence. Built the traffic split, never designed a test.
• CV: “40+ models” is about 40 deployed, about 15 in daily use. He said so at once.

Recommendation: no for this role. Great platform skills, and very honest. This role needs model ownership, and that is the part he has not done yet. He would be a strong hire for the ML Platform team, if he wants that.

What happened next

Ben’s system design round showed the same pattern. Wei was one of the strongest candidates on serving, rollout and failure handling, and he needed help when Ben asked what the model should optimise. Laura’s values conversation was positive; she liked his calm stories about on-call incidents.

In the debrief, the decision was quick: no for this role. Ravi called Wei himself and gave the feedback in two sentences: “You have great platform skills. This role needs model ownership, and we did not see enough of it yet.”

Ravi had mentioned the ML Platform team in their first call, and he mentioned it again: the team was hiring, and Ioana had recommended Wei. Wei thanked him, and then he was honest: “I’ve done platform work for ten years. I could do that job well. But I applied because I want to own models. If I take another platform job now, I will not learn it. What should I study?” So Wei said no to the ML Platform role.

Ravi asked Ioana, and she sent a short list through Chantal the next week:

• Build one model from start to finish where you choose the label, the metric and the baseline, and write down why. Use a public dataset with time in it.
• Learn regression metrics, quantile predictions and how to check them by group.
• Learn the basics of experiment design, and why a test by customer fails when customers share something.
• At work, ask to own the decisions for one model’s monitoring: which features matter, which thresholds, what happens when an alert fires.

Wei did the last point first. He went to the data scientist who owned the fraud model and asked about the labels. She was happy to explain. They found that the model had not been checked on claims outside the investigators’ usual patterns, and Wei offered to help with the analysis. He did not apply to Fennick again that year. He said he would apply for ML engineer roles when he could answer Ioana’s questions about his own model.

This is a good ending, even without an offer. Wei left with clear feedback, a clear plan, and no false story about his skills. Not every interview is a step toward that company. Some are a step toward the next one. See Keep learning, but stay focused.

What to take from this

• Running a model is not the same as owning it. Wei knew everything about how the fraud model ran, and almost nothing about why it was built that way. ML engineer interviews look for the second part.
• Ask “why” about the models you work next to. If your job puts you close to model decisions, ask the owners why they chose each label, metric and threshold. Then you can explain it, even if you did not decide.
• Hear the question before you answer it. “Target” meant something different to Wei. A short “do you mean the label or the deployment?” would have saved a few seconds. It happens to everyone.
• When a model problem comes, start with what the model should do. Dashboards and retraining are useful, but they come after.
• Honesty is useful in both directions. Wei’s “maybe 15 are really used” and “nobody read the reports” made his strong answers believable. Ioana’s honesty about the 20% helped him decide what he really wants.

If your profile is like Wei’s

1. Rewrite your CV in two parts: what you built and what the models do. “Built the deployment and monitoring platform for about 40 models (15 in daily use)” is stronger than “40+ models”, because you can defend it.
2. Learn one model from your own company deeply. Ask its owner for an hour: the label, the split, the baseline, the metric, the threshold, and what went wrong last year. Write it down and practise telling it.
3. Build one model where you decide. Choose a public dataset with time in it, a time-based split and a simple baseline. Translate a need, Define a baseline and Regression metrics cover the decisions. The Bike-share demand case study is a good start, and you can add a quantile model to it.
4. Learn how to check a model by group. Slice analysis and Thresholds and calibration show what an owner looks at after the overall score.
5. Turn your platform skills into model-owner answers. You know monitoring tools. Now practise saying what you would monitor for one model, at what threshold, and what you would do when it fires. Plan production checks helps.
6. Decide what you want before you apply. If you want an ML Platform role, your CV is already strong. If you want to own models, apply when you can explain one model from label to monitoring.