A minimum wage without a test
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In Module 5, we estimated effects without a test on simulated data. In this case study, we use real public statistics: restaurant jobs in New Jersey and Pennsylvania around a minimum-wage rise. We estimate the effect with difference-in-differences and test our assumptions.
Case study 3 of 3 · Experiments and Causal Impact: Did Our Change Work?A minimum wage without a test
1. Why "it went up" is not an answer
2. Design an A/B test
3. Analyse the test
4. Traps in real tests
5. When you cannot randomise: natural experiments
6. Adjust for confounders in observational data
7. Uplift: who should get the treatment?
8. Experiments for AI features and in production
What you will be able to do
Estimate the effect of a real policy that nobody randomised, and say honestly how much the data can tell. You will build a comparison from public statistics, run seven methods, and test each one on fake dates where the true effect is zero. Then you choose the method you trust, give a recommendation, and say what a fair study would need.
The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install the packages and get the data file (0.5 MB), so it needs no local setup. It runs in under a minute after the install.
Before you startWhat to know and to have ready
What you need to know
This case study uses Modules 5 and 6 most:
- Difference-in-differences, parallel trends and errors clustered by area, from Difference-in-differences.
- A weighted mix of other regions and its placebo test, from Synthetic control.
- Placebo tests of your own answer, and why a pass is not proof, from Test your own answer.
- The plan written first, from Write the plan first.
What you need on your computer
Nothing: the notebook installs its own packages, so it can also run in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, SciPy 1.15.3, statsmodels 0.15.0 and matplotlib 3.11.2, as in the project larkfield-experiments. This case study does not change the project. The setup cell makes a new folder minimum-wage. To start again, delete that folder.
Tested on macOS with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield ships its parcels from one warehouse. A delivery partner packs and carries them, and most of the partner’s packers earn close to the minimum wage. The province has announced a rise in the minimum wage.
Hana owns the delivery budget. She asks TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses.: “Will the partner cut jobs? Should we plan for fewer packers and slower deliveries?” Hana has read two articles. One says that a minimum-wage rise destroys jobs. The other says that it does not.
Priya cannot test this. Nobody can give a minimum wage to random warehouses. So she looks for a natural experiment: a change that hit one place and not a similar place next to it. The most famous one is New Jersey in 1992. She practises on it with official public data, to learn what such data can and cannot say.
Hana and Priya are invented. The states, the dates and the job counts are real.
What you will build
Section titled “What you will build”You will estimate the effect of New Jersey’s 1992 minimum-wage rise on jobs in fast-food-style restaurants, with Pennsylvania as the comparison. You write the plan, run seven methods, test each on fake dates, choose one with a single measure, and run it on new input. You end with a recommendation and what is still unknown.
A natural experiment
Section titled “A natural experiment”On 1 April 1992, New Jersey raised its minimum wage from 4.25 to 5.05 dollars an hour: 19% more. Pennsylvania, next door, stayed at 4.25. Before that, both states followed the federal minimum wage, so both had the same rises in 1990 and 1991.
This is like the two towns in Difference-in-differences. But nobody chose New Jersey at random. Anything else that hit New Jersey and not Pennsylvania at the same time will look like an effect. In 1994, David Card and Alan Krueger published a famous study of this rise. We come back to it near the end.
The data
Section titled “The data”The data comes from the Quarterly Census of Employment and Wages of the US Bureau of Labor Statistics (BLS). Almost every employer reports its number of jobs to the state every quarter, for unemployment insurance. BLS publishes the totals by industry and county.
Why not Card and Krueger’s data? Their survey data is on David Card’s web page, but it states no licence. So we may cite it and link to it, but we may not share it. The BLS data is in the public domain.
From 760 MB to 0.5 MB
Section titled “From 760 MB to 0.5 MB”The official files are large: about 190 MB per year, for every county and industry in the country. Learners should not download 760 MB. So we made a small extract with one short script, extract_qcew.py. It keeps New Jersey and Pennsylvania, private employers and four industries. It changes no values. The notebook gets the 0.5 MB extract and checks its checksum. An optional cell rebuilds the extract from the official files with the same script.
One row is one area (a state or a county), one industry and one quarter. It gives the jobs in each of the quarter’s three months. We use their average and call it jobs.
| Industry code | Industry | Use |
|---|---|---|
| 722211 | limited-service restaurants: you order at a counter and pay first | the outcome: the closest to fast food |
| 7221 | full-service restaurants | new input, at the end |
| 722 | all food services and drinking places | new input, at the end |
| 10 | all private jobs | the whole economy, as a check |
The industry codes are NAICS codes, the North American industry classification. In 1992, the US used an older system. In 2004, BLS rebuilt the old years with the new codes, from employers’ reports of that time. That is why we can separate limited-service restaurants from the others.
Hidden counties
Section titled “Hidden counties”BLS does not publish a number that could reveal one employer’s data. For a small county with one or two restaurant firms, BLS leaves the row out. This is called suppression.
counties in the file: {'NJ': 21, 'PA': 47} | with all 16 quarters: {'NJ': 21, 'PA': 46}share of the state's restaurant jobs in them before the rise: NJ 98.5%, PA 95.8%What this tells us. All 21 New Jersey counties are there. Only 47 of Pennsylvania’s 67 counties are in the file, and one of them has a quarter with 0 jobs. We keep the 46 counties with all 16 quarters. They hold 95.8% of Pennsylvania’s restaurant jobs, so the hidden counties are small. The state totals include the hidden counties. As a check, the main 2 × 2 comparison on the state totals gives −1.1%, against −0.2% on our counties. The hidden counties do not change the story.
Two more limits. The data counts jobs, not hours: if a restaurant cuts hours but keeps its workers, we cannot see it. And a job counts if the person was paid in the pay period that includes the 12th of the month.
The plan before the estimates
Section titled “The plan before the estimates”Priya writes the plan before she estimates anything. Hana agrees.
- The question: did the rise change the number of jobs in limited-service restaurants in New Jersey?
- The comparison: Pennsylvania, by county. 21 New Jersey counties and 46 Pennsylvania counties.
- The outcome: jobs per county and quarter, on the log scale, so that differences read as percentages.
- The methods: the naive before-and-after; a 2 × 2 on whole years; four regressions; a synthetic control.
- One measure for all methods: each method’s error on fake dates, where the true effect is zero (explained below).
- The decision: what Hana should plan for, and how sure we can be.
What we cannot check. In the course, the simulator knew every true effect. Here nobody knows it. So we check the methods on dates and places where we know that the effect is zero.
Look before you estimate
Section titled “Look before you estimate”First, the two states’ restaurant jobs. Each state is set to 100 in the year before the rise (1991 Q2 to 1992 Q1). So we can compare a small state with a large one.
- New Jersey
- Pennsylvania
- New Jersey's rise
Text description of the chart
A line chart with 16 quarters from 1990 Q1 to 1993 Q4 and a dashed vertical line at 1 April 1992. New Jersey: 111.9, 108.7, 113.3 and 101.0 in 1990; 96.9, 99.2, 107.5 and 99.1 in 1991; 94.2, 102.3, 107.3 and 100.5 in 1992; 99.6, 109.8, 113.3 and 106.4 in 1993. Pennsylvania: 93.8, 96.6, 98.8 and 98.0 in 1990; 96.6, 102.0, 100.2 and 98.7 in 1991; 99.0, 102.9, 104.5 and 105.3 in 1992; 98.0, 104.8, 104.2 and 101.6 in 1993.
Two things stand out. First, seasons: New Jersey has its beaches, and its restaurant jobs peak every summer. Second, New Jersey fell, and Pennsylvania rose, before the rise. In 1990 and 1991, the US was in a recession, and New Jersey’s economy fell faster than Pennsylvania’s.
Compare each quarter with the same quarter a year before
Section titled “Compare each quarter with the same quarter a year before”The seasons hide the trends. A simple way past them: compare each quarter with the same quarter a year before. Summer against summer, winter against winter. This is year-over-year growth. The chart shows New Jersey’s growth minus Pennsylvania’s. If the two states moved alike, the line would stay near zero.
- Restaurant jobs
- All private jobs
Text description of the chart
A line chart from 1991 Q1 to 1993 Q4 with a dashed vertical line at 1 April 1992. Restaurant jobs, in points: −17.3, −14.6, −6.7 and −2.6 in 1991; −5.4 in 1992 Q1; then +2.3, −4.4 and −5.1 in 1992; +6.6, +5.2, +5.7 and +9.4 in 1993. All private jobs: −3.4, −3.5, −2.9, −2.1, −2.0, then −1.0, −1.0, −0.4, −0.1, +0.7, +0.6 and +0.7.
What this tells us. The two states did not move alike before the rise. In early 1991, New Jersey’s restaurant jobs grew 17 points less than Pennsylvania’s. The gap shrank during 1991, long before any rise. This is the assumption of parallel trends, failing in plain sight. All private jobs show the same direction, so part of it is New Jersey’s whole economy. But the restaurant gap is about five times larger. Something else moves restaurant jobs too, and we do not know what.
After the rise, the restaurant line goes up. But it was already going up. Any method must separate the rise from this recovery.
The baseline: before and after, and the 2 × 2
Section titled “The baseline: before and after, and the 2 × 2”The plan’s first two methods use whole years: the year after the rise (1992 Q2 to 1993 Q1) against the year before (1991 Q2 to 1992 Q1). Whole years contain every season once, so the seasons cancel.
naive, New Jersey before and after: +2.5%2 x 2: New Jersey +2.5%, Pennsylvania +2.7%, difference -0.2%| Year before | Year after | Change | |
|---|---|---|---|
| New Jersey | 100 | 102.5 | +2.5% |
| Pennsylvania | 100 | 102.7 | +2.7% |
| Difference-in-differences | 2.5 − 2.7 ≈ −0.2% |
What this tells us. The naive reading is “the minimum wage created jobs”: +2.5%. But Pennsylvania’s jobs rose 2.7% without any rise. The difference is almost zero, and the 2 × 2 has no TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI.
Difference-in-differences as a regression
Section titled “Difference-in-differences as a regression”The regression uses every county and every quarter, as in Difference-in-differences. First, we take out each county’s own seasons: from each quarter, we subtract that county’s average for the same season. The model then has one effect per quarter (for what hit both states) and the effect of interest, nj:post. The errors are clustered by county.
def did_reg(p, start, end=16, trend=False): q = p[p["t"] < end].copy() q["post"] = (q["t"] >= start).astype(int) q["y"] = q["y"] - q.groupby(["area_fips", "season"])["y"].transform("mean") f = "y ~ nj:post + C(t)" + (" + nj:t" if trend else "") m = smf.ols(f, q).fit(cov_type="cluster", cov_kwds={"groups": q["area_fips"].astype("category").cat.codes}) ...regression DiD, all PA (67 counties): -3.3% (95% -6.2% to -0.3%, p = 0.031)The 67 counties are all the counties in the regression: 21 in New Jersey and 46 in Pennsylvania.
What this tells us. The regression says that New Jersey lost 3.3% of its restaurant jobs, and the interval does not contain zero. Hana would read: “the rise cost jobs, p = 0.031”. Before anyone says that, Priya tests the method.
A placebo test: a rise that never happened
Section titled “A placebo test: a rise that never happened”Weigh the empty basket first
Section titled “Weigh the empty basket first”At a market, you doubt a seller’s scale. You put an empty basket on it. If the scale shows 2 kg for nothing, you do not trust its 5 kg for your apples. You trust it when the empty basket weighs about zero.
A placebo date is the empty basket. We pretend that the rise happened a year early, in 1991, when nothing changed in New Jersey alone. We keep only the quarters before the real rise, so that the real rise cannot leak in. We run the method, unchanged. The true effect is zero, so any estimate is the method’s error.
A tiny example with every value visible. Suppose that a method gives −6% at the first fake date and −4% at the second. Its placebo error is (6 + 4) ÷ 2 = 5 points. If the same method then says −3% at the real date, its answer is smaller than its own typical error on nothing. We cannot tell −3% from zero.
You met the placebo test in Test your own answer. There, a pass did not prove the answer. Here, we use the size of the failure as our one measure.
The regression on fake dates
Section titled “The regression on fake dates”placebo, a fake rise in 1991 Q1: -7.2% (95% -9.8% to -4.5%)placebo, a fake rise in 1991 Q2: -7.0% (95% -9.9% to -4.0%)What this tells us. The method finds a “loss” of 7% in 1991, when New Jersey’s minimum wage did not change. Its interval is far from zero. So the empty basket weighs 7 kg. The real-date estimate (−3.3%) is smaller than the method’s error on nothing.
One more fact about 1991 Q2. The federal minimum wage rose on 1 April 1991, in both states, by the same amount. Restaurant pay was lower in Pennsylvania (138 dollars a week against 188 in New Jersey, in 1992 Q1). So that rise probably mattered more in Pennsylvania. That would make New Jersey look better, not worse. It cannot explain a fake loss of 7%.
Common mistake
An interval clustered by county, with only one treated state
What you see: a narrow interval and p = 0.031 at the real date. Placebo dates give “significant” effects too.
Why: the 21 New Jersey counties share one state economy. When New Jersey has a bad year, all of them have it together. Clustering by county treats them as 21 independent pieces of evidence. For the question “New Jersey against Pennsylvania”, there is really one treated unit and one comparison.
Fix: do not trust the interval alone. Test the method on placebo dates and placebo places, and compare the real estimate with those errors. More treated states, or many pairs of neighbouring counties, would give honest intervals.
Three other comparisons
Section titled “Three other comparisons”Maybe all of Pennsylvania is the wrong comparison. The plan has three other regressions:
- Eastern Pennsylvania only: 7 counties around Philadelphia and the Lehigh Valley, next to New Jersey (Bucks, Chester, Delaware, Montgomery, Philadelphia, Lehigh and Northampton). Neighbours may share more of New Jersey’s economy.
- The share of private jobs: restaurant jobs divided by all private jobs in the same county. This takes out each county’s whole economy, such as the recession.
- New Jersey’s own trend: a straight line through New Jersey’s gap before the rise, extended after it.
regression DiD, eastern PA -2.5% (95% -5.7% to +0.8%)regression DiD, share of private jobs -2.7% (95% -5.8% to +0.6%)regression DiD, NJ's own trend +2.0% (95% -5.4% to +10.1%)The chart shows each method’s real estimate and its estimate at the fake date 1991 Q2, both with their 95% intervals.
Text description of the chart
Eight rows on an axis from −22% to +12%, with a no-effect line at 0. All PA: real −3.3% (−6.2 to −0.3); fake −7.0% (−9.9 to −4.0). Eastern PA: real −2.5% (−5.7 to +0.8); fake −5.1% (−8.0 to −2.1). Share of private jobs: real −2.7% (−5.8 to +0.6); fake −5.5% (−8.5 to −2.4). NJ’s own trend: real +2.0% (−5.4 to +10.1); fake −7.0% (−12.8 to −0.8).
What this tells us. The comparisons move the real estimate between −3.3% and +2.0%. Every one of them “finds” a loss of 5% to 7% at the fake date. Eastern Pennsylvania and the share of private jobs help a little. New Jersey’s own trend flips the sign, but its fake estimates are the worst: a straight line cannot follow a fall that slows down.
Synthetic control
Section titled “Synthetic control”A synthetic control builds a twin for New Jersey from a weighted mix of Pennsylvania counties, as in Synthetic control. The weights make the mix follow New Jersey before the rise. After the rise, the mix stands for “New Jersey without the rise”.
The plan uses only counties with at least 300 restaurant jobs. Small counties jump when one restaurant opens or closes. We saw this when we first tried all 46 counties. The method gave Union County a weight of 0.42. Union County had 134 restaurant jobs in 1993 Q1 and 246 in 1993 Q2, probably because of one new employer. With that donor, the “effect” was −11.6%.
synthetic New Jersey from 39 donor counties: Monroe County 0.52, Northampton County 0.31, Butler County 0.17fit before: RMSPE 0.0265; effect after: -5.1%What this tells us. Three counties make the twin. Monroe County is in the Pocono Mountains, a holiday area with its own seasons, like the Jersey Shore. The twin follows New Jersey within about 2.7% a quarter before the rise (RMSPE, the typical gap, from Module 5). After the rise, New Jersey is 5.1% below its twin on average.
Is the gap unusual? Every county plays New Jersey
Section titled “Is the gap unusual? Every county plays New Jersey”As in Module 5, we give every donor county the same treatment. Each one gets its own twin from the other donors, and we measure its gap after the “rise”. The chart shows the 32 donors whose twin fitted about as well as New Jersey’s (a fit error at most twice New Jersey’s).
- New Jersey
- A Pennsylvania county as a fake New Jersey
Text description of the chart
A line chart from 1990 to 1993 with a dashed vertical line at 1 April 1992. New Jersey’s gap stays between −3.6 and +5.5 before the rise. After it: −1.2, +3.6 and −3.5 in 1992; −11.8, −7.0, −7.3 and −9.6 in 1993. The 32 grey lines spread between about −31 and +28.
New Jersey's post/pre gap ratio 2.7: rank 26 of 40 (p = 0.65)placebo effects: -18.6% to +32.1%; 18 of 39 at least as large as New Jersey'sWhat this tells us. The ratio compares each unit’s gap after the rise with its gap before. New Jersey ranks 26th of 40: 25 fake New Jerseys moved more, relative to their fit. Its placebo p-value is 0.65. A county’s restaurant jobs often move by 5% in a year for reasons that we do not know. New Jersey’s −5.1% is ordinary.
A fair comparison: one measure for seven methods
Section titled “A fair comparison: one measure for seven methods”Now every method faces the same test: its estimate at the real date, and its error at the two fake dates.
| Method | Real date | Fake 1991 Q1 | Fake 1991 Q2 | Placebo error (points) |
|---|---|---|---|---|
| New Jersey before and after | +2.5% | −7.4% | −4.7% | 6.0 |
| 2 × 2, whole years, all Pennsylvania | −0.2% | −9.8% | −7.1% | 8.4 |
| Regression, all Pennsylvania | −3.3% | −7.2% | −7.0% | 7.1 |
| Regression, eastern Pennsylvania | −2.5% | −4.9% | −5.1% | 5.0 |
| Regression, share of private jobs | −2.7% | −5.9% | −5.5% | 5.7 |
| Regression, New Jersey’s own trend | +2.0% | −14.8% | −7.0% | 10.9 |
| Synthetic control | −5.1% | −18.1% | +0.7% | 9.4 |
Text description of the chart
Seven rows on an axis from 0 to 12 points. New Jersey before and after 6.0. 2 × 2, whole years, all PA 8.4. Regression DiD, all PA 7.1. Regression DiD, eastern PA 5.0. Regression DiD, share of private jobs 5.7. Regression DiD, NJ’s own trend 10.9. Synthetic control 9.4.
What this tells us. By our one measure, the regression with eastern Pennsylvania is the most trustworthy method here. Its error on fake dates is 5.0 points. Its real estimate is −2.5% (−5.7% to +0.8%). That is half its own placebo error. Even the best method cannot tell its answer from its error on nothing.
The synthetic control did well at one fake date (+0.7%) and badly at the other (−18.1%). With only four quarters before 1991 Q1, the twin had too little to learn from. In Module 5, it was the best method. Here, with short and noisy data, it is not. A method’s rank depends on the data.
Compare with Card and Krueger
Section titled “Compare with Card and Krueger”Card and Krueger (1994) surveyed 410 fast-food restaurants in New Jersey and eastern Pennsylvania, in February–March and again in November–December 1992. They found that employment rose in New Jersey relative to Pennsylvania: +2.76 full-time-equivalent workers per restaurant (standard error 1.36). It was a surprise, because the usual theory predicts fewer jobs.
The study started a long debate. Neumark and Wascher (2000) used payroll records from a sample of restaurants and found the opposite. Card and Krueger replied (2000) with BLS data from the same programme as ours, under its earlier name. They found that New Jersey’s fast-food employment grew at “similar or slightly faster” rates than eastern Pennsylvania’s.
We cannot use their data, but we can use their window: 1992 Q1 to 1992 Q4, New Jersey against eastern Pennsylvania.
1992 Q1 to 1992 Q4: New Jersey +6.7%, eastern Pennsylvania +3.2%, difference +3.4%What this tells us. With their window, our public data also shows New Jersey growing a little faster: +3.4%. With whole years and a regression, it shows −2.5%. A placebo error of 5 points covers both. So the honest summary agrees with the reply of 2000: no clear loss of jobs, and no precise answer either.
Sources (cited and linked only):
- Card, D., and Krueger, A. B. (1994). “Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania.” American Economic Review 84(4), 772–793. Data: David Card’s data page (no licence stated).
- Neumark, D., and Wascher, W. (2000). “… Comment.” American Economic Review 90(5), 1362–1396. doi:10.1257/aer.90.5.1362.
- Card, D., and Krueger, A. B. (2000). “… Reply.” American Economic Review 90(5), 1397–1420. doi:10.1257/aer.90.5.1397. Working paper: NBER w6386.
The recommendation
Section titled “The recommendation”Priya writes to Hana in her words.
- This data cannot measure a small effect. The best method says −2.5%, with an error of 5 points on dates when nothing happened. A loss of about 2,200 restaurant jobs and a gain of about 300 are both inside its interval.
- It does show that a large loss is unlikely. A loss of 15% would be three times the best method’s placebo error. No planned method came close: the most negative answer was −5.1%.
- So do not plan for a large cut. Plan the partner’s capacity for about the same staff. Watch the partner’s hours, not only its head count: the data counts jobs, and a minimum wage can change hours.
- Do not accept a single number from either article. Ask the writers for their placebo tests.
What it is worth
Section titled “What it is worth”The cost of the rise to an employer is certain. The job effect is not. A worker on 25 hours a week (our assumption) costs 0.80 × 25 × 52 = 1,040 dollars a year more. That is the number that Hana can plan with today.
The study’s value is in jobs. New Jersey had about 38,300 limited-service restaurant jobs in the year before the rise. A placebo error of 5.0 points is about 1,900 jobs. Any effect smaller than that is inside the noise of this data.
The chosen method on new input
Section titled “The chosen method on new input”A method is useful only if it works on data that it was not chosen on. Priya runs the regression with eastern Pennsylvania, unchanged, on two other industries, with the same placebo test.
full-service restaurants: +1.3% (95% -1.2% to +3.7%); placebo -2.0%, -1.7%all food services and drinking places: -0.8% (95% -2.2% to +0.6%); placebo -3.1%, -2.8%What this tells us. Both estimates are near zero, and the errors on fake dates (2 to 3 points) are as large as the estimates. The method behaves the same way on new input: it rules out large effects, and it cannot find a small one.
Insights and limits
Section titled “Insights and limits”- The checks matter more than the estimate. Without the placebo dates, the regression’s −3.3% (p = 0.031) looks like a finding. With them, it is noise.
- One treated state is one treated unit. Clustering by county gave intervals that were too narrow. The placebo dates and the placebo counties showed it.
- There is no truth to check against. The placebo tests show how the methods fail on nothing. They do not show the true effect.
- We have only two fake dates. The data starts in 1990, the first year of these files. A longer history would give more fake dates and a better measure.
- Jobs, not hours, and one industry code. The data cannot see hours. Limited-service restaurants are close to fast food, but not the same.
- What a fair study would need. Many treated and untreated places, such as many pairs of neighbouring counties across state borders. Then one state’s luck does not decide the answer. Longer histories before the change. Hours and wages, not only jobs. For its own decisions, Larkfield can do better: a change in its own pay can roll out to its sites in a random order, and then it is a test.
Responsible use
Section titled “Responsible use”- The data has no people in it. It counts jobs by county and industry, and BLS hides counties where one employer could be seen. Do not try to find out which firm is behind a number.
- Minimum wages are a political subject. This page answers a narrow question about method: what this data can show. It gives no opinion about minimum-wage policy. A small effect on jobs says nothing about workers’ pay, prices or profits, which we did not study.
- Say what you could not check. A result like “−3.3%, p = 0.031” travels fast. Always report it with its placebo tests, or do not report it.
- Credit the sources. Public domain asks for nothing, but BLS asks to be cited. Cite Card and Krueger for their study, and do not copy their data.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows that you can study a change that nobody randomised, and that you test your own answer before you trust it. Make it yours with one change, such as another industry. Then make it public and easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
PLAN.md: the plan from “The plan before the estimates”, written before the estimates.results.csv: each method’s estimate, its fake-date estimates and its placebo error.extract_qcew.pyand the extract’s SHA-256, so that anyone can rebuild the data from BLS.- The source and licence, with the citation above.
A README template:
# A minimum wage without a test: New Jersey 1992, with public data
## ProblemNew Jersey raised its minimum wage on 1 April 1992; Pennsylvania did not.Did restaurant jobs change? No randomisation: a natural experiment.
## Data and licenceUS Bureau of Labor Statistics, Quarterly Census of Employment and Wages,1990-1993 (public domain). Extract: limited-service restaurants (NAICS 722211),21 NJ and 46 PA counties with all 16 quarters (95.8% of PA's jobs).
## ApproachPlan written first. Seven methods: before/after, 2 x 2, four regressions(clustered by county), synthetic control. One measure: the error at twofake dates a year early, where the true effect is zero.
## ResultsBest method (eastern PA): -2.5% (95% -5.7% to +0.8%), placebo error 5.0points. All PA: -3.3% (p = 0.031), but -7.0% at a fake date. Syntheticcontrol -5.1%, placebo rank 26 of 40. Card-Krueger window: +3.4%.
## LimitsOne treated state; jobs not hours; two fake dates; effects smaller thanabout 5% (1,900 jobs) cannot be told from noise.How to describe it in an interview, in three sentences: “I estimated the effect of a real minimum-wage rise on restaurant jobs with public data and seven methods. A regression gave a ‘significant’ loss of 3%, but the same method found a 7% loss at a fake date a year early. So I compared every method by its error on fake dates. The best method’s estimate was smaller than its own placebo error, so I concluded that the data rules out a large loss but cannot measure a small effect.”
Go further:
- Add 1994 and 1995 from BLS with
extract_qcew.py. Stop before 1 October 1996, when the federal minimum wage rose in Pennsylvania too. Does the estimate settle? - Add a third state next to New Jersey, such as Delaware or New York. Check its minimum wage in those years first. Do the placebo errors shrink?
- Compare with the other case studies, Did the mailing work, and for whom? and Real tests at an online shop. What did randomisation make easy there that was hard here?
Your turn: the 2 × 2 with eastern Pennsylvania
Section titled “Your turn: the 2 × 2 with eastern Pennsylvania”Before you run it, predict: the 2 × 2 on whole years gave −0.2% with all of Pennsylvania. With only eastern Pennsylvania, will it be higher or lower? In the notebook, section 8, use did_2x2 with the panel east, and put the estimate in east_2x2.
Show the answer
east_2x2 = round(did_2x2(east, START)["estimate"], 1)print(east_2x2)-0.9Eastern Pennsylvania grew 3.4% in the same year, a little faster than all of Pennsylvania (2.7%). So the difference is a little lower: −0.9%. The choice of comparison moves the answer by less than a point, far less than the placebo error.
Knowledge check
Section titled “Knowledge check”Check your understanding
The regression with all of Pennsylvania gives −3.3% (95% −6.2% to −0.3%, p = 0.031). Hana says: 'So the rise cost jobs.' What does Priya answer, and why?
Show the answer
Not from this evidence. Priya ran the same method on fake dates in 1991, when New Jersey’s minimum wage did not change. It found losses of 7.2% and 7.0%, with intervals that also excluded zero. So the method finds “significant” losses where there were none. Its error on nothing is larger than its real estimate. The interval is too narrow because the 21 New Jersey counties share one state economy, and clustering by county treats them as independent. The honest reading is that the data cannot tell a small effect from zero.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Why does Priya compare the methods by their estimates at fake dates, and not by their p-values or by how close they are to Card and Krueger?
Show the answer
Real data has no known true effect, so no method can be checked against the truth at the real date. At a fake date before the real rise, the truth is known: zero. A method’s estimate there is pure error, and its size is a fair measure that works the same way for every method. P-values only say how a method judges itself, and they were small even at the fake dates. Card and Krueger used different data and a different window, so being close to them does not prove that a method is right.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
In Module 5, the synthetic control was the most accurate method. Here, its placebo error is 9.4 points, one of the worst. Give two reasons from this case.
Show the answer
First, the data is short. There are only 9 quarters before the rise, and only 4 before the first fake date. The twin has too little history to learn New Jersey’s ups and downs, so it fails badly at the fake date 1991 Q1 (−18.1%). Second, the donors are small and noisy. A county’s restaurant jobs can jump by more than 80% when one employer opens, as Union County’s did. With all counties as donors, that county took a weight of 0.42, and the estimate moved to −11.6%. In Module 5, the simulator gave 78 weeks of smooth data. A method’s strength depends on the data that it gets.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- New Jersey’s 1992 rise is a natural experiment: one state changed, and its neighbour did not. Nobody randomised it.
- The data is public BLS data in the public domain, cut from 760 MB to a 0.5 MB extract with one script. For the hidden counties, we used the 46 Pennsylvania counties with all quarters (95.8% of the jobs) and checked the state totals.
- Before the rise, New Jersey’s restaurant jobs grew up to 17 points less than Pennsylvania’s: parallel trends failed in plain sight.
- A regression gave −3.3% with p = 0.031, but the same method found −7% at a fake date. With one treated state, intervals clustered by county are too narrow.
- By one measure, the placebo error, the best method was the regression with eastern Pennsylvania: −2.5%, with an error of 5.0 points. The data rules out a large loss, but it cannot measure a small effect.
Next, the final assignment: a new Larkfield scenario with planted problems, where you use every tool of the course.
Next: Final assignment