The economist

Imagine you ask a structural engineer, “Can I put a heavy bookshelf on this wall?” A good one says, “Yes, if you screw it into the wooden studs. Not into the plaster.” A less experienced one starts to explain load calculations, wall types and safety factors. Everything they say is correct, and you still do not know what to do with your bookshelf.

Many people with a PhD in economics apply for Data Scientist roles. They know causal inference better than almost anyone in industry: difference-in-differences, instrumental variables, synthetic control. They have spent years asking “is this effect real, or is it something else?”. Product teams need that rigour.

The gap is usually not knowledge. It is speed, product sense, and a short answer for a person who must decide this week. In research, a careful answer in six months is good. In a product team, a careful answer on Friday is often better than a perfect one in March. Rahul’s interview shows both sides.

RM

The candidate

Rahul Mehta

Economist, PhD and postdoc, moving into industry

• PhD Economics (labour)
• 2 years as a postdoc
• Stata, R, Python
• Toronto

• Postdoctoral Fellow, Centre for Labour and Migration Research, Northgate University, 2024 to now
• PhD Economics, Northgate University, 2024. Thesis: language training and the employment of newcomers
• Paper: “Free language classes and newcomer employment: evidence from a provincial expansion” (difference-in-differences, event study, synthetic control), under review
• Paper: returns to apprenticeship training, using distance to a training centre as an instrument
• Taught econometrics to master’s students (3 terms)
• Stata, R, Python (pandas, statsmodels), SQL (course projects), LaTeX

What Thandiwe sees before the call

Thandiwe reads the CV with interest. The causal methods are exactly what the posting names and does not explain, and the thesis topic, language training, is close to Parlo’s world.

She also sees what is missing: no product, no stakeholder, and no deadline shorter than a journal’s. At Parlo, a data scientist works alone with a PM. So the question is not “does Rahul know causal inference?” It is “can he use it fast enough, and explain it simply enough, for a PM to act on?”

Her short list:

• In his paper, how did he know the effect was real? (She expects this to go well.)
• In the Parlo case, does he start from the decision, or from the method?
• Can he give a recommendation with uncertainty, in a few sentences?
• Does he know what “good” means for a subscription app: learning, engagement, revenue, cost?

Terms on this page

• Difference-in-differences (DiD): compare how much a treated group changed with how much a similar untreated group changed over the same time. The second change removes what would have happened anyway, like the season. It works only if both groups would have moved in parallel without the treatment (“parallel trends”).
• Event study: a DiD that shows the difference for each period before and after the change. The periods before should be near zero. If they are not, the groups were already moving apart.
• Synthetic control: when only one unit is treated (one province, one country), build a “copy” of it from a weighted mix of untreated units that matched it well in the past. Then compare the real unit with its copy after the change.
• Instrumental variable: something that changes the treatment, but affects the outcome only through the treatment. It is used when you cannot run an experiment.
• Clustered standard errors: when people in the same group (a province) are alike, you have less independent information than the number of people suggests. You calculate the uncertainty by group.
• Day-30 retention, WAU, power, peeking, novelty effect: see the terms on The analyst.

The interview

Minutes 0 to 5: the opening

Thandiwe(interviewer):

Hi Rahul. I’m Thandiwe, senior data scientist. Thanks for joining. So, first you tell me about some work that changed a decision. I’ll ask many follow-ups. Then a real problem from Parlo, then your questions. No AI tools in this one, it’s only a conversation. Okay?

Rahul(candidate):

Okay. I should say, in academia “changed a decision” is… not always clear. But I’ll try.

Thandiwe(interviewer):

That’s fine, tell me how it was.

Minutes 5 to 20: the paper, then deeper

Rahul(candidate):

So, my main paper. In 2017, two provinces made their language classes for newcomers free. Before, there was a fee. The question was: did it help people find jobs? I used a national survey of newcomers, and I compared the two provinces with the other provinces, before and after. So difference-in-differences. The main result: two years after arrival, employment was about 3 percentage points higher. The interval was wide, roughly 0.5 to 5.5.

Thandiwe(interviewer):

And the decision?

Rahul(candidate):

One of the provinces did a review of the program in 2025. They kept it, and the review cited my paper. But it cited six other studies too. So I can’t say my paper changed it. Honestly, I don’t know.

Why this works

Rahul does not claim more than he can prove, even about his own impact. That is exactly how he thinks about data, and it is a good sign for this role.

Thandiwe(interviewer):

Fair. How did you know the 3 points were real? That the two provinces were not just different?

Rahul(candidate):

Three things. First, an event study, so the difference for each year before 2017. They were all close to zero, so the provinces moved in parallel before. Second, a placebo: I did the same analysis on people who arrived already speaking English or French. They don’t need the classes. And for them, no effect. Third…

Thandiwe(interviewer):

Go on.

Rahul(candidate):

Third, the standard errors. Only two provinces were treated. So the usual clustered standard errors are not reliable with so few clusters. I used a wild cluster bootstrap. Even that is not perfect with only two treated, so I also did synthetic control as a check, with the other provinces as donors. It gave a similar number, a bit smaller.

What the interviewer is checking

The answer is deep and correct. With only two treated provinces, normal methods make the uncertainty look smaller than it is, and even the better methods struggle. Rahul knows both, and he checks with a second method. Few people outside economics know this. Thandiwe could go deeper, but she has seen enough rigour. What she needs to learn now is different.

Thandiwe(interviewer):

Very clear. If you did it again, with what you know now?

Rahul(candidate):

Hmm. I think I would write the short version first. The policy people at the province, they told me they read only the first page. I spent two years on the method and one afternoon on the first page.

Why this works

This is a valuable answer, and Rahul gives it himself. He already knows that his gap is the short version. Thandiwe will test exactly that in the case.

Minutes 20 to 32: the Parlo case, and a very long first answer

Thandiwe(interviewer):

Okay, now our real situation. Five weeks ago we launched “Talk with Pip”, AI conversation practice, to 20% of new users. Weekly active users in that group went up about 6%. But day-30 retention for new users went down from 31 to 29%. Our product manager wants to give it to everyone before January. If you were our data scientist, what would you do?

Rahul(candidate):

Okay. So my first worry is that the 20% is not comparable with the rest. A launch to a group is not random, usually. So I would want to build a credible counterfactual. What I would do is a synthetic control. You take the treated group, and you build a weighted combination of untreated groups, for example cohorts from different countries or different weeks, that matches the treated group on retention in the pre-period. Then you compare after the launch. And I would want at least eight or ten pre-periods to fit the weights, and then placebo tests, where you pretend each donor was treated and see if the effect for the real group is unusual compared to the placebo distribution. And then for inference…

Thandiwe(interviewer):

Mm-hm.

Rahul(candidate):

…and for inference, I would use the ranking of the post-to-pre error ratio, because with synthetic control the usual standard errors don’t apply. Then, as robustness, a difference-in-differences with the untreated cohorts, with an event study to check pre-trends. And maybe, if there is something that pushes people to use Pip but doesn’t affect retention directly, like a notification that was sent at random, that could be an instrument for the effect of actually using Pip…

The next three minutes, in short

Rahul goes on for about three more minutes. He explains how to choose the donor pool, how many pre-periods are enough, and a second robustness check. Everything he says is correct for the problem that he imagines. Then his video freezes.

Rahul(candidate):

…and that would give us a much more credible…

Thandiwe(interviewer):

Sorry Rahul, you froze for a few seconds. I lost you after “donor pool”. But actually, can I stop you there for a moment?

Rahul(candidate):

Yes, of course. Sorry, I talked a lot.

Thandiwe(interviewer):

It’s okay. You assumed the 20% was not random. Why?

Rahul(candidate):

In my experience, a launch to “20%” is usually a choice. Some region, or the most active users.

Thandiwe(interviewer):

Here it was random, by user ID, in our experimentation tool. But only on iOS, because Android wasn’t ready. So 20% of new iOS users.

Rahul(candidate):

Ah. Pause. Okay. Then I was solving the wrong problem. If it’s random within iOS, we have an experiment. We don’t need a synthetic anything for that comparison. I should have asked first.

Where this goes wrong

Rahul’s first answer took about four minutes, and it was built on an assumption he did not check. One question at the start, “How was the 20% chosen?”, would have shown him the split was random. Then most of his plan was not needed. His worry was partly right, as he finds a minute later: Kenji’s dashboard does compare unlike groups. But the fix was one question, not a study. In a case interview, ask one or two questions first. Then give a short first answer, and offer more: “I can go deeper on any part.”

Why this works

Rahul’s next sentence matters. He does not defend his plan. He says, “I was solving the wrong problem”, and he changes direction at once. Many people find this hard after a long answer. See Build real skills, and learn to explain them.

Minutes 32 to 50: the rigour, and Friday

Thandiwe(interviewer):

Okay. So, with a random split on iOS. What worries you now?

Rahul(candidate):

The 31%. Who is in that group?

Thandiwe(interviewer):

All other new users, same weeks. Both platforms. It’s from Kenji’s dashboard.

Rahul(candidate):

Then the platform is mixed in. People who chose an iPhone are not a random sample of your users. Different countries, maybe higher income, maybe more often a quiet room to speak in. I would expect them to stay longer anyway, with or without Pip. The random split is inside iOS, so the clean comparison is iOS with Pip against iOS without Pip.

Thandiwe(interviewer):

You’re right, iOS users stay longer, every month. iOS without Pip, same days, is about 33%.

Rahul(candidate):

So the drop is about 4 points, not 2. The platform was hiding half of it. And one more thing. Even the 4 points answers only “what does Pip do for new iOS users in late August”. Not “what does Pip do for everybody in January”. For Android users the effect could be different. Even the sign could be different.

Why this works

This is the most important point of the whole case, and few candidates make it so clearly. Rahul makes it twice. First, the dashboard compares a group of iOS users with a mix that is mostly Android, so the platform hides part of the drop. When the mix of two groups differs, a total can mislead you; in the extreme case it reverses (Simpson’s paradox). Second, a random split makes the comparison fair only inside the group that was split. It does not make the result true for other people or another season. Economists call this external validity.

Thandiwe(interviewer):

What else?

Rahul(candidate):

Sorry, can I ask, what is “day-30 retention” exactly? Active on day 30?

Thandiwe(interviewer):

At least one session in the week before day 30. So between day 24 and day 30.

Rahul(candidate):

Then today, only people who signed up in the first few days can have it. How many?

Thandiwe(interviewer):

About the first five days. About 2,400 in the Pip group. 9,600 iOS without Pip. Kenji’s dashboard group is about 25,000. His page says minus 2 points, p about 0.04.

Rahul(candidate):

Let me do the iOS one roughly. Retention around 30%, so the standard error for 2,400 is about… 0.9 points. For 9,600, about half that. Together, about 1 point. So minus 4, plus or minus 2. Roughly minus 6 to minus 2. That’s far from zero. The dashboard one is at the border, but that one is the wrong comparison anyway.

Thandiwe(interviewer):

And people looked at the dashboard more than once.

Rahul(candidate):

Smiles. Every morning, I guess?

Thandiwe(interviewer):

Laughs. Every morning.

Rahul(candidate):

Then his 0.04 doesn’t mean much. With daily looks, the chance of a false “significant” at some point is several times 5%. If you want to look every day, you need a sequential test, with a stricter threshold for each look. But the iOS comparison is about four standard errors from zero. Peeking cannot explain that. So I believe there is a real drop, for these users. What I don’t believe is that it tells us about Android, or January.

What the interviewer is checking

Rahul’s mental arithmetic is right. The square root of 0.3 × 0.7 ÷ 2,400 is about 0.009, so 0.9 points, and the combined standard error is about 1 point. Peeking makes a borderline p-value like 0.04 weak. It does not make a result about four standard errors from zero go away. Rahul separates the two clearly.

Thandiwe(interviewer):

One more fact: lessons completed went down in the Pip group.

Rahul(candidate):

Interesting. Then lessons may be the mechanism. Pip replaces lessons, and lessons bring people back. But one warning: don’t compare retention “for users with the same number of lessons”. Lessons happen after the treatment. If you control for them, you remove part of the effect you want to measure, and you can create a false one.

Why this works

This is a subtle and correct warning. A variable that the feature itself changes, like lessons completed, must not be used to “make the groups equal”. It mixes up the comparison that the random split made fair. Many experienced analysts make this mistake. See Compare alternatives.

Thandiwe(interviewer):

Good. Okay. Now. It’s Friday. Kenji comes to you and asks, “Can I launch to everyone?” What would you tell Kenji on Friday?

Rahul(candidate):

I would say that the dashboard estimate of minus two percentage points is confounded by platform, and its p-value is not valid because of the repeated looks. The within-iOS comparison gives about minus four, with an interval of roughly minus six to minus two, which is robust to the peeking. However, that estimate is local to iOS users who signed up in late August, so its external validity for Android users and for the January cohort is unknown. And the WAU result has the issue that the metric includes the treatment itself…

Thandiwe(interviewer):

Let me stop you. Kenji has two minutes between two meetings. He is not a statistician. What does he hear?

Rahul(candidate):

Long pause. He hears… that I don’t know.

Thandiwe(interviewer):

Maybe. Try again. Shorter.

Rahul(candidate):

“The evidence suggests a negative effect on retention for iOS users, but we cannot generalise it to other users yet.”

Thandiwe(interviewer):

And he says, “Okay. So can I launch or not?”

Rahul(candidate):

Hmm. I would say that’s his decision, not mine. I can give him the evidence.

Thandiwe(interviewer):

It is his decision. But he is asking for your recommendation. Here, the data scientist gives one.

Rahul(candidate):

Pause. Okay. “I recommend… not to launch to everyone now. Because retention went down for iOS users, probably because of Pip, and we don’t know about Android. We can run a proper test and know more in… some weeks.” I’m not sure about the weeks. I would have to calculate.

Thandiwe(interviewer):

That’s closer. Thank you.

The edge

This is Rahul’s edge. It is not statistics. Everything he said on Friday was correct. The edge is turning correct statements into a recommendation that a busy person can act on. His first instinct was to describe the evidence. His second was to say “it’s his decision”. Both are normal in research, where the researcher reports and the reader decides. In a product team, the data scientist must also say “I recommend…”.

A stronger answer

Here is what Kenji needs, in about 30 seconds:

“My recommendation: don’t launch to everyone yet. People like Pip, usage is up. But for new iOS users, retention is down about 4 points, and I believe that is real. Your dashboard showed 2 because it compares iOS with Android. We know nothing yet about Android users or January users. If we launch to all January users and the drop is real for them too, we lose many subscribers. So let’s find out why. Lessons went down, so that is my first guess. Then we test a changed version from Monday, on half of new iOS users, with the rules decided in advance. You’ll have an answer at the end of November, before January.”

Every word of Rahul’s careful version is inside this answer. The difference is the order: recommendation first, the reason in business words, then a date. You can keep the technical details for the follow-up questions. Kenji will ask them if he wants them. See Explain decisions.

Thandiwe(interviewer):

One more. If you had to pick one main metric for the new test, which one?

Rahul(candidate):

Every metric here is a proxy. What we really want is learning, and you don’t measure it well.

Thandiwe(interviewer):

True. Pick one anyway.

Rahul(candidate):

Day-30 retention. With lessons, and… conversion to the paid plan, as secondary. And I would start to give the placement test to a random sample, so in six months you can measure learning.

Thandiwe(interviewer):

And cost? Each conversation costs us about four cents. Heavy users have 20 or more a week.

Rahul(candidate):

I didn’t think about cost at all. Twenty times four cents is 80 cents a week. About 3.50 dollars a month, for one heavy user. If they are on the free plan, that’s real money. So cost per user should be a guardrail. Sorry, this is a gap for me. In research, nobody pays per conversation.

Where this goes wrong

Rahul never asked about cost, revenue or the paid plan until Thandiwe raised them. A product data scientist thinks about these without a prompt, because the PM will. His arithmetic was fast and correct once he had the question. The gap is in the first question he asks, not in the skill.

Thandiwe(interviewer):

And if Kenji launches on Android anyway, without a random split. Is there anything useful you could still do?

Rahul(candidate):

Smiles. Now we are in my world. Yes. A difference-in-differences: Android new users before and after the launch, compared with iOS new users who don’t have Pip over the same weeks. The iOS group removes the season, if both platforms move in parallel. I would check that on the weeks before, like in my paper. It’s weaker than an experiment. But much better than “retention went down after we launched”, because that’s mostly the season.

Back on his strength

This is exactly where his methods help a product team: a feature launched without an experiment, and a question that will come anyway. He chooses a simple design, names its assumption, and says how to check it. Synthetic control would fit a different situation, for example a launch in one whole country, with other countries as the donors.

Minutes 50 to 60: Rahul's questions and the close

Rahul(candidate):

Can I ask, how much time is research, and how much is answering questions for the PM?

Thandiwe(interviewer):

Honestly, little research in your sense. About a third is dashboards and ad-hoc questions. Most of the rest is experiments and analyses like this one, with a deadline of days or weeks.

Rahul(candidate):

And is it possible to publish? Even sometimes?

Thandiwe(interviewer):

Not really. Maybe a blog post once a year.

Rahul(candidate):

Okay. Thank you for being direct. That helps me.

Why this works

Good questions are not only for impressing the interviewer. They also check whether the job fits you. Rahul learns something important for his own decision. See An interview is a conversation in both directions.

Thandiwe(interviewer):

Okay, we’re at time. Thank you, Rahul. The point about iOS users being different, that was the best I’ve heard on this case. Mira will contact you this week.

Rahul(candidate):

Thank you. And sorry again for the long first answer.

After the call: Thandiwe’s notes

Candidate: Rahul Mehta. Product case, 60 minutes.

Strong evidence

• Causal inference. Excellent. Correct on few clusters, placebo tests and pre-trends. Best point of the case from anyone: iOS users are a selected group. So the dashboard (iOS against both platforms) hides half the drop, and even the fair 4-point result may not transfer to Android or January. Did the standard error in his head, correctly. Warned correctly against controlling for lessons completed.
• Honesty. Did not claim impact for his paper. Changed direction at once when his assumption was wrong.

Some evidence

• Experiment statistics. Knows peeking, power and sequential tests well. Did not design the new test in detail (length, split); we ran out of time.

No evidence

• SQL and dashboards in daily work. Course projects only; screen was fine.

Concern

• Communication with a PM. First case answer was four minutes on a method, based on an assumption he did not check. On “what would you tell Kenji on Friday?”, needed three tries and a push to make a recommendation.
• Product sense. Did not raise cost, revenue or conversion without a prompt.

Recommendation: no for this role. In our team he would often be alone with a PM, and today the gap in communication would show every week. His rigour is real and rare. I would be happy to see him again in one or two years, or for a more research-focused role. Please give him detailed feedback; he will use it.

What happened next

Simone agreed with Thandiwe’s recommendation. The one vacancy went to another candidate. Mira called Rahul and offered a feedback call with Thandiwe, and he said yes.

In the call, Thandiwe was specific. She told him his iOS point was the best of the whole process. She told him about the four-minute first answer and the three tries on Friday. She suggested one exercise: for every analysis, write the first three sentences for a person who will read nothing else, before writing anything else.

Three months later, Rahul joined the economic research team of a large bank, where his work is to estimate the effect of policy changes on customers. Two weeks after he started, he wrote to Thandiwe. He said that his first presentation to a director started with a recommendation, and that the director asked him for more analyses like it. He thanked her for the honest feedback.

A “no” does not always mean “not good enough”. Sometimes it means “not this role, not yet”. Rahul used the feedback, and he found a role that needed exactly his rigour. See Luck is part of it.

What to take from this

• Ask before you design. Rahul’s four-minute plan rested on one assumption: that the 20% was not random. One question would have saved him most of it.
• Give a short first answer, then offer more. “I’d do X, mainly because of Y. I can go deeper on any part.” The interviewer decides where to go.
• A data scientist recommends. Describing the evidence is half the job. “I recommend…, because…, and we will know more by…” is the other half.
• Rigour is a strength when it changes the decision. Rahul’s iOS point mattered because it changed what Kenji should do. The details of his method did not.
• Think about cost and revenue without a prompt. In a product company, every feature has a price, and the PM will ask.

If your profile is like Rahul’s

Your causal-inference skills are rare and valuable. The gap is speed and the short answer, and both improve with practice.

1. Practise the “Friday answer” every week. Take a paper you know well. Write the recommendation for a busy manager in three sentences: what to do, why, and when you will know more. Say it out loud in under 30 seconds. See Explain decisions.
2. Practise product cases with a timer. Spend the first two minutes only on questions. Then give a one-minute first answer. Ask a friend to stop you at 90 seconds.
3. Learn the product side of a subscription app: free and paid plans, conversion, retention, cost per user. For each case, ask, “How does this feature make or cost money?”
4. Learn online experiments as well as you know quasi-experiments: sample size and power, peeking, novelty effects, and guardrail metrics. Review Compare alternatives with a product test in mind.
5. Strengthen your SQL with retention and cohort queries. See Time and ranking.
6. Rewrite your CV for industry. Put the decision or the question first, then the method. “Measured whether free language classes increased employment (difference-in-differences)” reads better than the method alone. See Portfolio anatomy.