The modeller

Imagine a very good mechanic. You bring your car in and say, “It makes a strange noise, and I want to sell it next month.” Before you finish, the mechanic has opened the engine. The work is excellent. But you only needed to know one thing: is it safe to sell, or not?

Many strong machine learning people interview for product data science roles. They see “machine learning” in the posting, and they bring their best tool. The posting for Parlo does list machine learning. But most of the work is experiments and product analysis, and a product manager waits for a decision. The usual gap for this profile is not skill. It is the first question: what will someone decide with this?

Artem Kovalenko is from Ukraine. He has lived in Canada for two years. English is his second language, and he speaks in short, direct sentences. His technical skills are the strongest in the process. In this interview you will see a weak start on the case, an honest recovery, and one moment where he is clearly right and says so kindly.

AK

The candidate

Artem Kovalenko

ML data scientist, 3 years in ride-hailing, Kaggle Master

• 3 years ML data scientist
• Kaggle Competitions Master
• Python, LightGBM, PyTorch, SQL
• MSc Applied Mathematics
• Toronto

• Machine learning projects (contract), Toronto, 2024–now: demand forecasting for a warehouse company; a document classifier for a law office
• Data Scientist (ML), Kolo Rides, Kyiv (ride-hailing start-up, about 150 people), 2021–2024
• Built a driver churn model (LightGBM, 140 features, AUC 0.81) used to target retention bonuses; reduced driver churn by 15%
• Built the first ETA model for rides; error down 22% against the rule-based estimate
• Kaggle Competitions Master: 1 gold, 4 silver medals
• Python (pandas, LightGBM, PyTorch, scikit-learn), SQL, Airflow, Docker, MLflow

What Thandiwe sees before the call

Thandiwe Mokoena reads the CV next to the job description. The modelling is clearly real. Kaggle medals and an ETA model with a baseline are hard to fake. The SQL screen notes say “fast and correct; explained what he asked the AI assistant and checked its window function by hand”.

The line that she wants to understand is “reduced driver churn by 15%”. A churn model predicts who will leave. It does not reduce churn by itself. Something else did, the bonuses, and somebody measured it. How did they measure it?

Her short list for the call:

• Does he start from the decision, or from the model?
• Does he know how to show that an action worked, not only that a prediction was good?
• Can he design a fair experiment, and explain it to a product manager?
• Kenji, the product manager, joins the last 15 minutes as an observer. How does Artem talk to him?

Terms on this page

• Churn: when a user or a driver stops using the product. A churn model predicts who is likely to stop.
• SHAP values: a method that shows how much each feature pushed one prediction up or down. It explains the model, not the world.
• AUC: a score from 0.5 (random) to 1 (perfect) for how well a model ranks people from most to least likely.
• Holdout group: a random group that does not get the action, so you can compare. A long-run holdout stays without the feature for months.
• Day-30 retention: of the people who joined on a given day, the share who still use the app about 30 days later.
• p-value: if there were no real effect, how often you would see a difference this big, or bigger, by chance alone.
• Confidence interval: a range of values for the true effect that fit the data well. See Estimate uncertainty.
• Peeking: looking at a test result many times and stopping when it looks good. It makes “significant” results appear more often than they should.
• Novelty effect: people use something new more at first because it is new. The effect fades.

The interview

Minutes 0 to 5: the opening

Thandiwe(interviewer):

Hi Artem. Thanks for joining. I’m Thandiwe, senior data scientist, mostly onboarding. Quick plan. First, one analysis or project of yours, and I’ll go deep on it. Then a real problem from Parlo. Then your questions. Kenji, our product manager, will join for the last fifteen minutes, just to listen. And this round is only us talking, no AI tools.

Artem(candidate):

Okay. Understood. No AI. Fine.

Minutes 5 to 20: an analysis that changed a decision

Thandiwe(interviewer):

So, tell me about an analysis or model that changed a decision.

Artem(candidate):

Driver churn at Kolo. Drivers leave, and to get a new driver costs a lot, onboarding, documents, bonus for first rides. The ops team wanted to give retention bonuses, but only to drivers who will leave. So I built a model. LightGBM, around 140 features. Rides per week, trend of rides, cancellations, rating, hours online, the area of the city. The target was: the driver does zero rides in next 14 days.

AUC was 0.81 on the test month. Ops took the top 10% risk every week and gave them a bonus.

Thandiwe(interviewer):

Okay. How did you split train and test?

Artem(candidate):

By time. Train on months before, test on the next month. Not random. Random split leaks the future, the same driver is in both.

Why this works

A quick, correct answer about a common trap. With data over time, the test set must come after the training set, and the same person should not appear in both. Thandiwe hears that the modelling is real. See Train, validation and test sets.

Thandiwe(interviewer):

Good. How did you choose who got the bonus?

Artem(candidate):

I calibrated the probabilities first, with isotonic regression. Then the threshold, I looked at precision at different—

Thandiwe(interviewer):

Sorry, I mean it differently. Who decided? You, or the ops team?

Artem(candidate):

Ah. The ops team. They had budget for about 700 bonuses a week. So top 10%, it was the budget.

Thandiwe(interviewer):

Okay. Your CV says churn went down 15%. How did you measure that?

Artem(candidate):

Ops measured. Churn in the month after the bonuses started, against the month before.

Thandiwe(interviewer):

Which months?

Artem(candidate):

October, and… November was the bonus month. Hmm.

Short pause.

Yes, I know what you will say. In Kyiv, November is cold, more people take a taxi, drivers earn more. Drivers stay anyway. So it’s not a good comparison. Honestly, the 15% is from the ops report. It is not my number.

Where this goes wrong

A model’s AUC says it ranks drivers well. It does not say that the bonus changed anything. Artem put the ops team’s before-and-after number on his CV as his own result. He sees the problem as soon as he says the months aloud. But a CV line that he cannot defend costs him trust. A safer line: “Model used to target retention bonuses; a small holdout suggested a reduction of about 5 points, with a wide interval.”

Thandiwe(interviewer):

Did anyone compare it in a fairer way?

Artem(candidate):

Yes, later. I asked for a holdout. For one month, 10% of the high-risk drivers did not get the bonus. Random. About 300 drivers without, and about 2,700 with.

Churn was 27% without bonus, 22% with. But the interval was wide. From about zero to ten points, I think. And then the program stopped, the budget was cut. So we never knew well.

Thandiwe(interviewer):

What would you do differently?

Artem(candidate):

Holdout from day one. And bigger. And I would measure money, not churn only. A bonus costs. If the driver stays one more month, is it worth it? We never calculated this.

Why this works

This is a good recovery. Artem knows how to measure an action fairly: a random holdout. He also knows its limits. With 300 drivers in the holdout, a 5-point difference has an interval from about 0 to 10 points. And he adds the business question himself: was the bonus worth its cost?

Minutes 20 to 45: the Parlo case

Thandiwe(interviewer):

Okay, now our real situation. Five weeks ago we launched “Talk with Pip”, AI conversation practice, to 20% of all users. Sorry, of new users. Weekly active users in that group went up about 6%. But day-30 retention for new users went down from 31 to 29%. Our product manager wants to give it to everyone before January. If you were our data scientist, what would you do?

Artem(candidate):

I would build a churn model for new users. Predict who leaves before day 30. Features: Pip conversations, lessons, time of day, language, level. Then with SHAP values I can see if Pip use pushes churn up or down, and for which users.

Thandiwe(interviewer):

Okay. And what does Kenji do with that model?

Artem(candidate):

Pause.

He sees… which users leave because of Pip?

Thandiwe(interviewer):

His decision is: give Pip to everyone in January, yes or no. How does the model help him decide that?

Artem(candidate):

Hmm. It doesn’t, directly. Users who talk with Pip a lot are different from users who don’t. The model sees correlation. It doesn’t tell what happens if we give Pip to everybody.

Okay. Let me start again. I need to understand the data first. How were the 20% chosen?

Where this goes wrong

This is the weak moment. Artem answers with his favourite tool before he asks what decision it serves. A churn model with SHAP values explains predictions. It does not show what Pip causes, because heavy Pip users chose to be heavy users. Thandiwe asks one question, and he sees it himself. “Let me start again” is a good way to recover. Do not defend a wrong first answer.

Thandiwe(interviewer):

Random, by percentage rollout in our experimentation tool. But only on iOS. Android wasn’t ready.

Artem(candidate):

Random inside iOS. Okay, good. Then day-30 retention, five weeks after launch. Only users who joined in the first five days can have day 30. How many is that?

Thandiwe(interviewer):

About 2,400 in the Pip group.

Artem(candidate):

Small. And all of them joined in one week, end of August.

Why this works

Now Artem is fast, and his questions are the right ones. He sees, without help, that a day-30 number after five weeks covers only the first five days of sign-ups. He does not ask yet what the 31% is compared with. That question comes later, and it matters a lot.

Thandiwe(interviewer):

Late August. Back-to-school, for many of our learners. Okay. What about cost? Each Pip conversation costs us about four cents in model calls. Heavy users do twenty or more a week.

Artem(candidate):

Twenty times four cents, 80 cents a week. A month… about three dollars fifty for a heavy user. Plus is 14.99. So for a paying user it’s okay. For free users, it is cost with no money back. I would want to know: does Pip move people from free to Plus?

Thandiwe(interviewer):

Good. So what does “good” mean here? More weekly active, better retention, more Plus, or that people learn more?

Artem(candidate):

This is… a product decision, I think. Kenji decides. I measure what he says.

Thandiwe(interviewer):

And if he asks you what you think?

Artem(candidate):

Then… retention, maybe. Because it is the long one. Weekly active can go up because Pip is new, and then it goes down. I am not sure about learning. I don’t know how you measure it.

The edge

This is the edge, and it is not technical. In this role, the data scientist works alone with a product manager. He or she must help decide what “good” means, not only measure it. Artem’s instinct is to hand that back. When Thandiwe pushes, he gives a reasonable answer about novelty, but the learning question stays open. Thandiwe writes “decision focus” in her notes.

A stronger answer

A stronger answer takes a position and says why:

“I’d propose day-30 retention as the main metric, because January users who leave in February don’t help us. Weekly active users would be a guardrail, not the goal, because a new feature often looks good in the first weeks. I’d also want two more things on the results page: free-to-Plus conversion, because Pip costs money on free users, and lessons completed, to see if Pip replaces lessons. Learning is the real promise of the product. I don’t know how Parlo measures it. Is there anything I could use?”

It is still Kenji’s decision. But a data scientist who proposes a metric, with reasons, makes that decision easier.

Minutes 45 to 50: Kenji joins

Thandiwe(interviewer):

Oh, Kenji’s here. Kenji, you’re on mute, I think.

Kenji(interviewer):

—sorry. Hi, Artem. I’m just listening, please continue.

Thandiwe(interviewer):

Artem, Kenji agrees to a proper experiment. Tell us how you would design it.

Artem(candidate):

Okay. Randomise new users, by user ID, at sign-up. Both platforms, if Android is ready. 50/50. Main metric day-30 retention. Guardrails: lessons completed, free-to-Plus, and cost per user. Decide all of this before we start, and don’t look every day to stop early.

Size: if we want to see one point of retention, around 30% baseline, I think we need around 33,000 users per group. I can check it with a calculator, but it is this size.

Thandiwe(interviewer):

We get about 5,500 new users a day, both platforms.

Artem(candidate):

So 2,750 per group per day. Twelve days to get the people. Then 30 days to see day-30 for the last ones. Around six weeks. If only iOS, more than double the time to get people. Around two months.

Why this works

This is correct and quick. For a difference of 1 point at about 30% retention, with the usual 5% significance level and 80% power, you need about 33,000 users per group. At 2,750 a group per day, that takes about 12 days, and then you wait 30 days. Artem also says that he would check the number, which is honest.

Kenji(interviewer):

Sorry, can I say something? Six weeks is fine, but honestly I think we already know. My dashboard shows p = 0.04 for the retention drop. So there’s a 96% chance the feature hurts retention. A little. And I think plus 6% weekly active is worth a little retention.

Artem(candidate):

Can I disagree a little, on the p-value?

Kenji(interviewer):

Sure, go ahead.

Artem(candidate):

The p-value is not the chance that Pip hurts. It says: if Pip had no effect at all, how often we see a difference this big just by chance. Here, about 4% of the time. That’s a different question.

Kenji(interviewer):

Hm. Isn’t that the same thing, said the other way round?

Artem(candidate):

No, and many people think so. It’s like… if a person is healthy, the test is positive only 4% of the time. This does not mean a person with a positive test is 96% sick. For “the chance Pip hurts”, you need a Bayesian analysis, with a prior. Different calculation.

And here there is a bigger problem. Your dashboard compares which groups?

Kenji(interviewer):

The Pip group against all the other new users, same weeks. Like the dashboard.

Artem(candidate):

Then the p-value only checks for chance. It doesn’t check the groups. Pip is only iOS. The other group has Android inside. Android users stay less, no? Then this control is too high for iOS—no, sorry. Too low. Android pulls it down. Thandiwe, do you have iOS without Pip, same days?

Thandiwe(interviewer):

About 33%.

Artem(candidate):

So the real gap is four points, not two. And if we look every morning and believe the first day it says “significant”, we find “significant” more often than 5%. So: not 96%. But also not “a little”.

Disagreeing well

Kenji is senior, confident and the person who will work with this hire. Artem still corrects him, and he does it well. He asks first (“can I disagree a little?”). He explains with an everyday example, not a formula. Then he moves to the bigger problem, and he finds it by asking Kenji one question: the comparison itself is not fair. He loses his sentence for a moment (“too high—no, too low”), and corrects it. And he does not use the correction to win the argument. The fair comparison is worse for Pip, not better, and he says so. A p-value of 0.04 does not mean 96%, but the drop is also not “a little”. See You can disagree.

Kenji(interviewer):

Four points. Hm. Pause. Okay. And I’ve said 96% in two meetings already. So what number should I use?

Artem(candidate):

iOS against iOS. Minus 4 points. With 2,400 people, plus or minus about two points, I think. So between minus 2 and minus 6. But these are users from one week in August. For January, I don’t know yet.

Minutes 50 to 60: Kenji's question, Artem's questions, and the close

Kenji(interviewer):

Okay, last thing from me. I have a meeting with my VP on Friday. What do I say?

Artem(candidate):

That the dashboard comparison is biased by platform mix. Like-for-like it is minus 4 points, interval from 2 to 6. But one cohort week, possible novelty effect, and peeking. We need a randomised test on both platforms, powered for one point of retention. Primary metric, guardrails, pre-registered. Six weeks.

Kenji(interviewer):

Okay… I’m not sure I can say “pre-registered” to my VP.

Artem(candidate):

Yes. Sorry. Simple: with Pip, from 100 new iOS users, about 29 stay one month, not 33. We see only one week of users. In six weeks we know for sure, before January.

A stronger answer

Artem’s first answer is correct, but it is for a data scientist, not a VP. His second is much better. A full version for Kenji’s Friday meeting has three parts. It says what we know and how sure we are. Then it gives a plan with a date. Then it says what we do if we cannot wait:

“People use the app more with Pip in the first weeks. But in a fair comparison, iPhone users with iPhone users, fewer people stay: about 29 in 100 after a month, instead of 33. We only have one week of users so far, from late August, so we are not sure it is the same in January. Next week we start a fair test on both platforms, and we’ll have an answer at the end of November. Until then, I would not give Pip to everyone.”

See Explain decisions.

Thandiwe(interviewer):

Thanks, Kenji. Artem, your questions?

Artem(candidate):

Yes. The posting says machine learning. How much of the work is models, really?

Thandiwe(interviewer):

Honestly? In this team, maybe 10%. Mostly experiments and analysis. We have one forecasting model, and that’s it.

Artem(candidate):

Okay. Thank you for the honest answer. Then I also want to ask: who builds the recommendations? Which lesson comes next, this kind of thing.

Thandiwe(interviewer):

That’s a different team, recommendations. They do ML all day. They’re hiring too, I believe.

Why this works

This is a risky question for this interview, and a very good one for Artem. He learns that the role is mostly not what he does best. He also learns that another team in the same company is. An interview is a conversation in both directions, and this question changes what happens next. See An interview is a conversation in both directions.

Thandiwe(interviewer):

Okay, we’re at time. Thanks, Artem. Mira will contact you this week.

Artem(candidate):

Thank you. And Kenji, sorry about the 96%.

Kenji(interviewer):

No, no. Better now than in front of the VP.

After the call: Thandiwe’s notes

Candidate: Artem Kovalenko. Product case, 60 minutes. Kenji observed the last 15.

Strong evidence

• Statistics. Correct experiment design and sample size, quickly. Correct on the p-value, and explained it well to Kenji.
• Data sense. Saw alone that day-30 covers only the first five days. Found the platform mix in Kenji’s comparison by asking which groups it compared (iOS against iOS: 4 points, not 2). Late, but he found it.
• Python and modelling. Strongest in the process (time-based split, calibration). Not much needed for this role.

Some evidence

• Communication with a PM. Disagreed with Kenji very well. His first summary for the VP was too technical; he fixed it when asked.

No evidence

• Learning outcomes. Did not know how to approach “do people learn more”; did not try.

Concern

• Decision focus. Started the case with a churn model before asking what decision it serves. On “what is good”, wanted Kenji to decide alone. In this role he would be alone with a PM most weeks. Also the CV line “reduced churn by 15%” is a before-and-after number from ops, which he admitted.

Recommendation: no for this role. Simone, please send him to the recommendations team. He is the best ML candidate I have seen this year, and he asked about them himself.

Kenji added: “He corrected me and he was right. He did it in a way I could accept. I’d work with him.”

What happened next

The product case was the deciding stage. Artem did not go on to the take-home for this role. The one vacancy went to another candidate, Chloe Martin.

Simone forwarded his CV and Thandiwe’s notes to the manager of the recommendations team, which needs an ML data scientist. Mira called Artem and explained both things honestly: a no for this role, and an introduction to another team. The introduction is not an offer. Artem got an interview with the recommendations team. At the time of writing, he has not had it yet.

Artem also changed one line on his CV. It now says what the model did and what the holdout showed, with its wide interval.

This ending is common for strong specialists. The role was not the right one. But because Artem asked an honest question about the work, Thandiwe could point him to a role that fits. That is partly luck, and partly a good question at the right time.

What to take from this

• Ask what the decision is before you choose a method. A model answers “who”, but Kenji needed “what happens if we do this”. See Translate a need.
• A good prediction is not a good result. AUC 0.81 says the model ranks drivers well. Only a fair comparison, like a random holdout, says the bonus worked.
• You can disagree with a senior person. Ask first, explain simply, and say the whole truth, also the part that does not help your point. Kenji respected him more after it.
• Say it twice: once for experts, once for everyone. The VP sentence matters as much as the correct design.
• Ask what the work really is. Artem’s question about machine learning showed him the right team.

If your profile is like Artem’s

Your technical depth is real. For product data science roles, the gap is decision focus and talking with product managers. You can practise both.

1. Start every case with three questions. What is the decision? Who makes it, and when? What would change their mind? Write them on your paper before the interview starts. See Problem framing.
2. For each model on your CV, write what action it led to and how that action was measured. If nobody measured it fairly, say so, and say what you would do. A wide interval from a small holdout is still good evidence of honest work.
3. Practise the “p-value in one sentence” and the “confidence interval in one sentence”. Say them to a friend who does not work with data. Then add one everyday example. See Compare alternatives.
4. Take a position on metrics. For three apps you use, choose a main metric and one or two guardrails, and write why in three lines.
5. Read the posting by its team, not its title. “Data Scientist, Growth & Learning” is mostly experiments. “Data Scientist, Recommendations” or “Machine Learning” is mostly models. Apply to both if you want, but prepare differently. See Prepare for applications.
6. Practise the Friday answer. Take any analysis and explain the recommendation in four sentences, with a date and a plan if the answer is “we don’t know yet”. See Explain decisions.