The analyst

Think of a good mechanic who has fixed hundreds of cars with the garage’s diagnostic computer. They know which questions to ask the owner, and they can read every number on the screen. But ask them why the computer flags a fault at exactly that value, and they may need a moment. They know how to use the tool very well. They are less sure how it works inside.

Many people who apply for Data Scientist roles are product analysts like this. They write SQL all day, build the dashboards that product managers open every morning, and run A/B tests on the company’s platform. They often know the business better than the data scientists. What they want now is the title, and a voice in the decision.

The usual gap is statistics depth. The platform calculates the interval, so the analyst rarely thinks about what it means. Chloe’s interview shows both sides: very strong product questions, and an honest edge.

CM

The candidate

Chloe Martin

Product analyst, 4 years, moving into data science

• 4 years as a product analyst
• SQL, dbt, Looker
• BCom, statistics minor
• Montréal

• Senior Product Analyst, Maplecart (online home goods, about 600 people), 2022 to now
• Ran and analysed about 30 A/B tests on the company’s experimentation platform (checkout, promotions, search)
• Defined the checkout metrics used by 3 product teams; built the main growth dashboard (Looker)
• Showed that a first-order coupon pop-up raised add-to-cart but lowered margin per visitor; the pop-up was removed
• Led the experimentation practice for the growth team
• SQL (advanced), dbt, Looker, Python (pandas, basic statsmodels), Excel

What Thandiwe sees before the call

Thandiwe sees a lot of what the role needs every day: SQL, metric definitions, and four years with product managers. At Parlo, one data scientist works alone with one PM, so that matters.

Two lines make her curious. Does “Led the experimentation practice” mean that Chloe set the rules for tests, or that she ran the most tests? And “about 30 A/B tests” tells her about volume, not depth. Does Chloe know what the platform does for her?

Her short list:

• In her best project, how did she know the result was real?
• In the Parlo case, does she ask how the 20% was chosen before she trusts the numbers?
• Does she know what a confidence interval says, and what it does not say?
• Power and stopping rules: has she ever decided the length of a test herself?

Terms on this page

• Day-30 retention: the share of new users who still use the app about a month after they sign up. At Parlo, a user counts as retained if they open the app at least once in the week that ends on day 30 (days 24 to 30).
• Weekly active users (WAU): people who open the app at least once in a week.
• Percentage rollout: a switch that turns a feature on for a share of users. Often nobody chose a main metric or a test length first.
• Confidence interval (95%): a range around a result. If you repeat the same study many times, about 95% of the ranges built this way contain the true value. See Estimate uncertainty.
• Power: the chance that a test finds an effect, if the effect is really there and has a given size. 80% is the usual target.
• Peeking (optional stopping): looking at the results many times and stopping when they look significant.
• Novelty effect: people use something new because it is new. The effect becomes smaller after some weeks.
• Guardrail metric: a number that must not get worse, even if the main metric gets better.

The interview

Minutes 0 to 5: the opening

Thandiwe(interviewer):

Hi Chloe! Can you hear me okay?

Chloe(candidate):

Yes, yes. Hi. Sorry, one second, my headphones… okay. Now it’s good.

Thandiwe(interviewer):

No problem. I’m Thandiwe, senior data scientist here. So, the plan. First you tell me about an analysis that changed a decision, and I’ll ask a lot of “why”. Then a real problem we have right now. Then your questions. This round is just us talking, no AI tools. And I like to draw tables, so I’ll send you a whiteboard link.

Minutes 5 to 20: an analysis that changed a decision

Chloe(candidate):

Okay. So at Maplecart we had a pop-up for new visitors. “10% off your first order.” Marketing loved it, because add-to-cart went up. And they wanted to make it bigger, 15%.

What I did is, I looked at the test they ran the year before. They only measured add-to-cart. So I asked, okay, but how many of those carts become orders? And at what margin? I rebuilt it with orders and margin, from the same test data. Add-to-cart was up, I think around 4%. Orders were almost flat. And margin per visitor went down, because people who would buy anyway now got 10% off. So we removed the pop-up. We didn’t make it bigger.

Why this works

This is a strong start. Chloe found the problem in the metric, not in the method. She changed the question from “more carts?” to “more money?”. Product teams need exactly this.

Thandiwe(interviewer):

Nice. And the drop in margin per visitor. How did you know it was real, not noise?

Chloe(candidate):

The platform gave a confidence interval. It was all below zero. I think from minus 3% to minus 0.5%, something like this.

Thandiwe(interviewer):

And in your words, what does that interval tell you?

Chloe(candidate):

That there is 95% chance the real effect is between minus 3 and minus 0.5.

Thandiwe(interviewer):

Mm-hm. Is that exactly right?

Chloe(candidate):

Pause. I think… it’s how I explain it to the product managers. I remember in university it was something about repeating the experiment. But I don’t remember the exact way to say it. Sorry.

What a confidence interval says

Thandiwe does not want a textbook sentence. She wants to know whether Chloe knows that her usual explanation is a shortcut. Chloe honestly does not remember, and Thandiwe moves on.

Think of a fishing net. A 95% interval is a method that catches the true value about 95 times in 100 throws. After one throw, the true value is inside your net or it is not, and you do not know which. “We are fairly sure the effect is between minus 3 and minus 0.5” is fine in a meeting. In an interview, show that you know the difference. See Estimate uncertainty.

Thandiwe(interviewer):

That’s fine. Your CV says you led the experimentation practice for growth. What did that mean in practice?

Chloe(candidate):

Ah. Okay. “Led” is maybe a big word. I was the analyst for most of the growth tests. I wrote a template: the hypothesis, the main metric, and what we decide if it wins or loses. People used it. But the platform and the statistics, a data science team in another office owned that.

Where this goes wrong

“Led the experimentation practice” sounded like Chloe set the statistical rules. She did not. She corrects it at once, so it costs her almost nothing. If you write “led”, be ready to say exactly what you led. See Build real skills, and learn to explain them.

Thandiwe(interviewer):

A template that people used is real work. And the pop-up test, what would you do differently?

Chloe(candidate):

Decide the metric before the test. Margin per visitor, from the start. Not find it one year later.

Thandiwe(interviewer):

Okay. I’m moving to our case now. We’ll come back to statistics.

Minutes 20 to 38: the Parlo case, first part

Thandiwe(interviewer):

Okay, now our real situation. Five weeks ago we launched “Talk with Pip”, AI conversation practice, to 20% of new users. Weekly active users in that group went up about 6%. But day-30 retention for new users went down from 31 to 29%. Our product manager wants to give it to everyone before January. If you were our data scientist, what would you do?

Chloe(candidate):

Okay. Before I answer, can I ask some questions? I don’t know yet where the numbers come from. First, the 20%. How were they chosen?

Thandiwe(interviewer):

Random, by user ID, in our experimentation tool. But only on iOS. Android wasn’t ready. So it’s 20% of new iOS users.

Chloe(candidate):

Okay, iOS only. That’s important. And the 31%, what is it? The other iOS users? Or Parlo before the launch?

Thandiwe(interviewer):

It’s from Kenji’s dashboard. All other new users, same weeks, both platforms.

Chloe(candidate):

Hmm. Then it’s not fair. Do iOS users retain the same as Android users?

Thandiwe(interviewer):

No. iOS users retain better. Every month.

Chloe(candidate):

So I want iOS against iOS. The new iOS users without Pip, same weeks. Do you have that?

Thandiwe(interviewer):

I do. About 33%.

Chloe(candidate):

33. Oh. So the gap is not 2 points. It’s 4.

Why this works

Four short questions, and the problem has changed. Kenji’s dashboard compares an all-iOS group with a group that is mostly Android, which retains less. Like for like, the gap is about 4 points, not 2. Thandiwe gave the 33% only because Chloe asked.

The idea behind Simpson's paradox

When two groups have a different mix, and the mix itself changes the result, a total can mislead you. Here the platform mix hid half of the drop. In the extreme case, a result can go one way inside every group and the other way in the total. That is Simpson’s paradox. The cure is what Chloe did: compare like with like.

Chloe(candidate):

And five weeks ago is… end of August? So it’s back to school. A lot of people who sign up end of August are students, or parents, or people who say “this year I learn Spanish”. For the comparison it doesn’t matter, both groups have the same season. But Kenji wants to launch for January. The January users are not the August users.

Thandiwe(interviewer):

Yes, the last week of August. Okay. What else?

Chloe(candidate):

The day-30 retention. If we launched five weeks ago, which users even have a day-30 number? Only people who signed up in the first… five days? Something like that.

Thandiwe(interviewer):

Laughs. Yes. About the first five days.

Chloe(candidate):

And how is “active” defined? If I open Pip and talk, am I active?

Thandiwe(interviewer):

Any session counts. So yes.

Chloe(candidate):

So the 6% is partly… the feature counts itself. People open Pip because it’s new. Do you have the 6% by week? I would expect it high in week one and lower after.

Thandiwe(interviewer):

Honestly, I haven’t looked at it by week. That’s a good thing to check.

Chloe(candidate):

And lessons completed? Did it change?

Thandiwe(interviewer):

It went down in the Pip group. A few percent, I don’t remember the exact number.

Chloe(candidate):

Ah. So maybe people talk to Pip instead of doing lessons. And if lessons bring people back… that could be the retention story. But it’s a guess.

Why this works

Here Chloe is at her best. Each question comes from the last answer. She also sees that a random split removes the season from the comparison, but says nothing about January users. And she calls the lessons story “a guess”. She knows what she knows.

Thandiwe(interviewer):

So how would you look at the data first? Concretely.

Chloe(candidate):

I would split by country. Also by people who tried Pip and people who never opened it.

Thandiwe(interviewer):

Careful with that second one. Why could it mislead you?

Chloe(candidate):

Hmm. Because people choose to try it. People who try new things are maybe more motivated anyway. So they retain better, and it’s not because of Pip. Okay, so the main comparison stays the whole Pip group against the whole iOS group without Pip. The other split only helps to understand. And by country… in some countries people learn on the bus. You can’t talk to an app out loud on the bus.

Thandiwe(interviewer):

That is exactly what our user research found. In some countries many people can’t use the microphone in public.

Two kinds of split

A split by something fixed before the test, like country, is safe: the random split puts about the same share of each country in both groups. A split by what people did after they got the feature, like “used Pip”, is not safe, because people chose themselves. Chloe needs one hint. See Slice analysis.

Minutes 38 to 50: real or noise, and the edge

Thandiwe(interviewer):

Okay. So, the retention drop. Is it real? I’ll put the numbers on the board.

Day-30 retention, first 5 days of the rollout	Users	Retained
Pip (20% of new iOS users)	about 2,400	29%
All other new users, both platforms (Kenji’s dashboard)	about 25,000	31%
New iOS users without Pip	about 9,600	33%

Dashboard: −2 points, 95% interval about −3.9 to −0.1, p about 0.04. iOS against iOS: −4 points, interval about −6 to −2.

Chloe(candidate):

Okay. The dashboard one, zero is almost in the interval. But that comparison is wrong anyway. The iOS one… minus 6 to minus 2. Zero is far away. So for these users, I think the drop is real. Even if it’s only five days of users.

Thandiwe(interviewer):

Good. One more fact. Kenji opens his dashboard every morning. The retention number became significant last Tuesday, and he was very happy to have a clear answer. What do you think about that?

Chloe(candidate):

Laughs a little. Okay, I know this is a problem. We have done it at Maplecart, actually. I stopped a test after four days because it was significant.

Thandiwe(interviewer):

Tell me why it’s a problem. Not the rule, the reason.

Chloe(candidate):

Can I have a minute to think?

Thandiwe(interviewer):

Of course. Take your time.

Chloe(candidate):

About twenty seconds of silence. Okay. So, 5% means: if there is no effect, one time in twenty, the test says “significant” by chance. That’s for one look. But if I look every day, every day is another chance. Like rolling a dice every morning. Roll it once, maybe no six. Roll it twenty times, you get a six at some point. And then I stop, exactly on the lucky day.

Thandiwe(interviewer):

Mm-hm. And so?

Chloe(candidate):

So the real false positive rate is not 5%. It’s more. I don’t know how much. Maybe double?

Thandiwe(interviewer):

With a look every day for a few weeks, often 20 or 25%. About five times what you think you have.

Chloe(candidate):

Wow. Okay. And the days are not independent, it’s the same users adding up, so it’s not exactly a dice. But the idea is the same. Hmm. So my Maplecart test… I don’t know if it was real. Later the number on the dashboard was flat. I thought it was a novelty effect. Maybe it was just never real.

Thandiwe(interviewer):

And here? Does it change what you said about the 4 points?

Chloe(candidate):

Wait, let me think. Kenji’s 0.04, with every-morning looks, I don’t trust it. But for iOS against iOS, the interval is far from zero, so p is very small. Even if peeking makes it a few times bigger, it’s still small. So no. I still believe the 4 points. For these users.

The edge

This is Chloe’s edge, and Thandiwe went there on purpose. Chloe knew the rule “don’t stop early”, but not the reason. She asks for a minute and builds the reason from the definition of 5%. Her dice comparison is not perfect, and she sees that herself. Then she applies the idea twice: to her old test, and to the Parlo numbers. Peeking makes a borderline result like p = 0.04 weak. It does not remove a very strong one: for iOS against iOS, p is about 0.0001.

Why this works

Asking for a minute and thinking in silence is normal. Most interviewers wait. A short silence is better than a fast answer that you are not sure about.

Thandiwe(interviewer):

Thank you, that’s honest. So, is Kenji’s idea finished?

Chloe(candidate):

No, I don’t think so. Pip as it is now probably hurts, for these users. But maybe because it takes the place of lessons. I would try a version where Pip comes after a lesson, not instead of it. And test that properly.

Thandiwe(interviewer):

Okay. How many users would we need to see a 2-point drop? Without the calculator.

Chloe(candidate):

Usually the platform does it for me. Hmm. Smaller effect needs more users. And it’s squared, I think? Half the effect, four times the users.

Thandiwe(interviewer):

Yes. A rough rule: per group, 16 times p times one minus p, divided by the effect squared. That’s for 5% significance and 80% power.

Chloe(candidate):

Okay, so… p, I take about 0.3, it’s between the groups. One minus p is 0.7, so 0.21. Times 16 is… 3.4, about. Divided by 0.02 squared, 0.0004. So around 8,400 per group? And we have 2,400 with Pip. So we found 4 points because 4 is big. If the real drop was 2, we find it maybe… half of the time?

Thandiwe(interviewer):

About half, yes. Power around 50%.

A stronger answer

Chloe got there with help. A stronger answer brings the reasoning without the calculator, and links it to the decision:

“Rough rule: per group, 16 times p times one minus p, over the effect squared. With retention around 30% and a 2-point effect, that’s about 8,400 per group. We saw the 4 points only because 4 is a large effect. If Kenji cares about 1 point, it’s four times more, about 33,000 per group. So before we test, I’d ask Kenji: what is the smallest drop that would change your decision? That number sets the test length, not the calendar.”

The smallest effect you care about decides the size, and the size decides the length. See Compare alternatives.

Thandiwe(interviewer):

So. What would you design?

Chloe(candidate):

A real test, with the new version. New users, random, 50/50. iOS now, Android when it’s ready. Main metric: day-30 retention. Guardrails: lessons completed, and cost. And I would want something about learning, but… do you measure learning?

Thandiwe(interviewer):

There’s a short placement test. Very few people take it.

Chloe(candidate):

Okay, so that’s a gap. I note it for later. And we write before the start: how long, which metric, what result means “launch”. And we look at the end, not every morning.

Thandiwe(interviewer):

How long, with about 2,400 new iOS users a day?

Chloe(candidate):

50/50 is 1,200 per group per day. For 8,400 per group, about a week of sign-ups. Then 30 days to wait. So… five, six weeks? Results end of November. For 1 point it’s almost a month of sign-ups, too late for January.

What the interviewer is checking

The arithmetic is right. One refinement: two full weeks of sign-ups is safer than one, because weekday and weekend users differ. Then about 1.5 points is the smallest drop the test can see, and the result still comes before January.

Thandiwe(interviewer):

Last part. It’s Friday. Kenji asks you, “Can I launch to everyone?” What do you tell him?

Chloe(candidate):

I would say: “Not now. People use Pip, that’s good. But for the first iOS users, retention is down about 4 points, and I think it’s real. Your dashboard showed 2 because it mixes in Android. If we launch to everyone in January, that’s a lot of lost subscribers. My guess is that Pip replaces lessons. Let’s test a version where Pip comes after a lesson, from Monday. You get an answer at the end of November, still before January.” And I would ask him what drop he can accept. Maybe for him 1 point is okay, if more people buy Plus.

Why this works

This is the answer Thandiwe was hoping for: short, with a date, and the risk in business words. It explains the dashboard without blaming anyone, gives Kenji a new idea, not only a “no”, and asks what drop he can accept.

Where this goes wrong

Two things are missing. The first is cost. Each conversation costs Parlo about CAD 0.04, so a heavy user with 20 a week costs about CAD 3.50 a month. If that user is on the free tier, Parlo gets nothing back. The second is the 20% who have Pip now: if it probably hurts their retention, should they keep it until the new test? A data scientist who owns this area raises both.

Minutes 50 to 60: Chloe's questions and the close

Chloe(candidate):

My question. You said one data scientist per area. When the PM and I disagree, like with Kenji now, who decides?

Thandiwe(interviewer):

The PM decides on the product. You decide what the data says, and how sure we are. If it’s serious, Simone joins. It’s not always comfortable.

Chloe(candidate):

And how much of the week is dashboards and ad-hoc questions? At Maplecart it was maybe 60%.

Thandiwe(interviewer):

Here about a third. Some weeks more. And honestly, end of quarter is busy.

Why this works

Both questions show the real job, and the posting says nothing about either one. See An interview is a conversation in both directions.

Chloe(candidate):

Last one. If I join, is there someone I can learn statistics from? Today I saw my gap. I think you saw it too.

Thandiwe(interviewer):

Smiles. I saw it. Yes, there is. Okay, we’re at time. Thank you, Chloe. I really enjoyed your questions at the start of the case. Mira will write to you this week.

Chloe(candidate):

Thank you. And now I go and read about peeking.

After the call: Thandiwe’s notes

Candidate: Chloe Martin. Product case, 60 minutes.

Strong evidence

• Product sense and metric definitions. Best clarifying questions I have had for this case. By asking, she found the unfair dashboard comparison (like for like, the gap is 4 points, not 2), the season, the five days of day-30 data, WAU counting the feature, and fewer lessons.
• Working with a PM. Her Friday answer was short, gave a date and a new version to test, and asked Kenji what drop he can accept.
• Honesty. Corrected “led the experimentation practice” herself. Said “I don’t know” when she did not.

Some evidence

• Experiment design. Good structure (random, 50/50, main metric, guardrails, decide before looking). Needed help with sample size.
• SQL. Not tested today; the SQL screen was strong.

No evidence

• Causal inference without a random split; machine learning. Not discussed.

Concern

• Statistics depth. Reached her edge on the meaning of a confidence interval, power and peeking. She worked out the peeking problem slowly and correctly. She has stopped a test early in the past.

Recommendation: move forward. She will be very useful from week one with a PM. The statistics gap is real but she knows it, and she learns in front of you. If we hire her, I would like to mentor her on statistics for the first six months.

What happened next

In the take-home, Chloe wrote the main metric and the stopping rule at the top of her notebook before she looked at any result. When Simone asked why, she said, “Because of the last interview.” She declared that an AI assistant wrote one chart function, and showed how she checked it.

Kenji’s values conversation went very well. He wrote: “She asked me what drop I could accept. Nobody asks me that.”

In the debrief, the team compared Chloe with Gabriel Santos, who was stronger on everything technical but clearly a senior. Simone asked to open a Senior role on the Monetisation team for him. The one Data Scientist vacancy went to Chloe, at the middle of the band. Her offer included a plan: one hour every two weeks with Thandiwe on statistics, for the first six months. Chloe accepted.

This ending is not luck alone. Chloe did not win on statistics, where she was weaker. She won on the part of the job that happens every day, and she was honest about the part that she still needs to learn.

What to take from this

• Ask before you analyse. Chloe’s first questions found most of what was wrong with the comparison. That needs curiosity, not a formula.
• Know what your tools do for you. Be ready to explain intervals and sample sizes without the platform.
• At the edge, ask for a minute and reason from the definition. Chloe built the peeking problem from “5% means one in twenty”.
• Connect your past mistake to the new idea. “So my Maplecart test… I don’t know if it was real” showed that she learns, not that she failed.
• Give the PM a date and a plan, not only “it depends”. Then ask what effect would change their decision.

If your profile is like Chloe’s

Your product and SQL skills are a real advantage. The gap is the statistics under the platform, and you can close much of it in a few weeks.

1. Learn the three ideas behind every test result: standard error, confidence interval and power. Work through Samples and populations, Estimate uncertainty and Compare alternatives.
2. Simulate peeking yourself. In Python, generate two groups with no real difference. Look at the p-value every “day” for 20 days, and count how often it goes below 0.05 at least once. Do it 1,000 times. Seeing 20 to 25% yourself teaches more than reading it.
3. Calculate one sample size by hand. Take a real test you ran. Use the rule “16 × p × (1 − p) ÷ effect²” per group, and compare it with your platform’s number.
4. Re-read two of your old tests. Write when you looked, when you stopped, and what the interval was. If you stopped one early, say so in interviews.
5. Check every verb on your CV. For “led”, “owned” and “built”, write one sentence about what exactly you did.
6. Practise the Friday answer out loud: three sentences to a PM, with a recommendation, the risk and a date. See Explain decisions.