Skip to content
Français

A minimum wage without a test

  • Case study 3
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In Module 5, we estimated effects without a test on simulated data. In this case study, we use real public statistics: restaurant jobs in New Jersey and Pennsylvania around a minimum-wage rise. We estimate the effect with difference-in-differences and test our assumptions.

What you will be able to do

Estimate the effect of a real policy that nobody randomised, and say honestly how much the data can tell. You will build a comparison from public statistics, run seven methods, and test each one on fake dates where the true effect is zero. Then you choose the method you trust, give a recommendation, and say what a fair study would need.

Case studyPortfolio piece

Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • A natural experiment
  • Plan before the estimates
  • Seasons and year-over-year growth
  • Difference-in-differences with clustered errors
  • Pre-trends and placebo dates
  • Synthetic control and placebo units
  • One measure for many methods
  • A decision with honest limits

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus three tasks with checks. Its setup cells install the packages and get the data file (0.5 MB), so it needs no local setup. It runs in under a minute after the install.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study uses Modules 5 and 6 most:

What you need on your computer

Nothing: the notebook installs its own packages, so it can also run in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, SciPy 1.15.3, statsmodels 0.15.0 and matplotlib 3.11.2, as in the project larkfield-experiments. This case study does not change the project. The setup cell makes a new folder minimum-wage. To start again, delete that folder.

Tested on macOS with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield ships its parcels from one warehouse. A delivery partner packs and carries them, and most of the partner’s packers earn close to the minimum wage. The province has announced a rise in the minimum wage.

Hana owns the delivery budget. She asks TermPriyaLarkfield’s data scientist. She measures how well the assistant works, on test sets and not on guesses.: “Will the partner cut jobs? Should we plan for fewer packers and slower deliveries?” Hana has read two articles. One says that a minimum-wage rise destroys jobs. The other says that it does not.

Priya cannot test this. Nobody can give a minimum wage to random warehouses. So she looks for a natural experiment: a change that hit one place and not a similar place next to it. The most famous one is New Jersey in 1992. She practises on it with official public data, to learn what such data can and cannot say.

Hana and Priya are invented. The states, the dates and the job counts are real.

You will estimate the effect of New Jersey’s 1992 minimum-wage rise on jobs in fast-food-style restaurants, with Pennsylvania as the comparison. You write the plan, run seven methods, test each on fake dates, choose one with a single measure, and run it on new input. You end with a recommendation and what is still unknown.

On 1 April 1992, New Jersey raised its minimum wage from 4.25 to 5.05 dollars an hour: 19% more. Pennsylvania, next door, stayed at 4.25. Before that, both states followed the federal minimum wage, so both had the same rises in 1990 and 1991.

A timeline from 1990 to 1993 with two lanes. New Jersey: 3.35 dollars an hour in 1990 Q1, 3.80 from April 1990, 4.25 from April 1991, and 5.05 from 1 April 1992 in blue. Pennsylvania: the same 3.35, 3.80 and 4.25, then 4.25 with no change. Below: an orange box marks the fake dates for the placebo test, 1991 Q1 and Q2; a green box marks the real date, 1992 Q2, the first quarter with the new wage. Before: 9 quarters, 1990 Q1 to 1992 Q1. After: 7 quarters, to 1993 Q4. Units: 21 New Jersey counties and 46 Pennsylvania counties; outcome: jobs in limited-service restaurants each quarter.
The design. The two states had the same minimum wage until April 1992. Then only New Jersey's rose. We have 9 quarters before and 7 after.

This is like the two towns in Difference-in-differences. But nobody chose New Jersey at random. Anything else that hit New Jersey and not Pennsylvania at the same time will look like an effect. In 1994, David Card and Alan Krueger published a famous study of this rise. We come back to it near the end.

The data comes from the Quarterly Census of Employment and Wages of the US Bureau of Labor Statistics (BLS). Almost every employer reports its number of jobs to the state every quarter, for unemployment insurance. BLS publishes the totals by industry and county.

Why not Card and Krueger’s data? Their survey data is on David Card’s web page, but it states no licence. So we may cite it and link to it, but we may not share it. The BLS data is in the public domain.

The official files are large: about 190 MB per year, for every county and industry in the country. Learners should not download 760 MB. So we made a small extract with one short script, extract_qcew.py. It keeps New Jersey and Pennsylvania, private employers and four industries. It changes no values. The notebook gets the 0.5 MB extract and checks its checksum. An optional cell rebuilds the extract from the official files with the same script.

One row is one area (a state or a county), one industry and one quarter. It gives the jobs in each of the quarter’s three months. We use their average and call it jobs.

Industry code Industry Use
722211 limited-service restaurants: you order at a counter and pay first the outcome: the closest to fast food
7221 full-service restaurants new input, at the end
722 all food services and drinking places new input, at the end
10 all private jobs the whole economy, as a check

The industry codes are NAICS codes, the North American industry classification. In 1992, the US used an older system. In 2004, BLS rebuilt the old years with the new codes, from employers’ reports of that time. That is why we can separate limited-service restaurants from the others.

BLS does not publish a number that could reveal one employer’s data. For a small county with one or two restaurant firms, BLS leaves the row out. This is called suppression.

counties in the file: {'NJ': 21, 'PA': 47} | with all 16 quarters: {'NJ': 21, 'PA': 46}
share of the state's restaurant jobs in them before the rise: NJ 98.5%, PA 95.8%

What this tells us. All 21 New Jersey counties are there. Only 47 of Pennsylvania’s 67 counties are in the file, and one of them has a quarter with 0 jobs. We keep the 46 counties with all 16 quarters. They hold 95.8% of Pennsylvania’s restaurant jobs, so the hidden counties are small. The state totals include the hidden counties. As a check, the main 2 × 2 comparison on the state totals gives −1.1%, against −0.2% on our counties. The hidden counties do not change the story.

Two more limits. The data counts jobs, not hours: if a restaurant cuts hours but keeps its workers, we cannot see it. And a job counts if the person was paid in the pay period that includes the 12th of the month.

Priya writes the plan before she estimates anything. Hana agrees.

  • The question: did the rise change the number of jobs in limited-service restaurants in New Jersey?
  • The comparison: Pennsylvania, by county. 21 New Jersey counties and 46 Pennsylvania counties.
  • The outcome: jobs per county and quarter, on the log scale, so that differences read as percentages.
  • The methods: the naive before-and-after; a 2 × 2 on whole years; four regressions; a synthetic control.
  • One measure for all methods: each method’s error on fake dates, where the true effect is zero (explained below).
  • The decision: what Hana should plan for, and how sure we can be.

What we cannot check. In the course, the simulator knew every true effect. Here nobody knows it. So we check the methods on dates and places where we know that the effect is zero.

First, the two states’ restaurant jobs. Each state is set to 100 in the year before the rise (1991 Q2 to 1992 Q1). So we can compare a small state with a large one.

  • New Jersey
  • Pennsylvania
  • New Jersey's rise
Jobs in limited-service restaurants, one point per quarter, 1990 to 1993. Each state's year before the rise is 100. New Jersey has strong summer peaks. In 1990, New Jersey was about 10 points above its later level; Pennsylvania was below its own.
Text description of the chart

A line chart with 16 quarters from 1990 Q1 to 1993 Q4 and a dashed vertical line at 1 April 1992. New Jersey: 111.9, 108.7, 113.3 and 101.0 in 1990; 96.9, 99.2, 107.5 and 99.1 in 1991; 94.2, 102.3, 107.3 and 100.5 in 1992; 99.6, 109.8, 113.3 and 106.4 in 1993. Pennsylvania: 93.8, 96.6, 98.8 and 98.0 in 1990; 96.6, 102.0, 100.2 and 98.7 in 1991; 99.0, 102.9, 104.5 and 105.3 in 1992; 98.0, 104.8, 104.2 and 101.6 in 1993.

Two things stand out. First, seasons: New Jersey has its beaches, and its restaurant jobs peak every summer. Second, New Jersey fell, and Pennsylvania rose, before the rise. In 1990 and 1991, the US was in a recession, and New Jersey’s economy fell faster than Pennsylvania’s.

Compare each quarter with the same quarter a year before

Section titled “Compare each quarter with the same quarter a year before”

The seasons hide the trends. A simple way past them: compare each quarter with the same quarter a year before. Summer against summer, winter against winter. This is year-over-year growth. The chart shows New Jersey’s growth minus Pennsylvania’s. If the two states moved alike, the line would stay near zero.

  • Restaurant jobs
  • All private jobs
New Jersey's year-over-year growth minus Pennsylvania's, each quarter. Before the rise, New Jersey's restaurant jobs grew up to 17 points less than Pennsylvania's. All private jobs show the same direction, but smaller: about 2 to 3.5 points.
Text description of the chart

A line chart from 1991 Q1 to 1993 Q4 with a dashed vertical line at 1 April 1992. Restaurant jobs, in points: −17.3, −14.6, −6.7 and −2.6 in 1991; −5.4 in 1992 Q1; then +2.3, −4.4 and −5.1 in 1992; +6.6, +5.2, +5.7 and +9.4 in 1993. All private jobs: −3.4, −3.5, −2.9, −2.1, −2.0, then −1.0, −1.0, −0.4, −0.1, +0.7, +0.6 and +0.7.

What this tells us. The two states did not move alike before the rise. In early 1991, New Jersey’s restaurant jobs grew 17 points less than Pennsylvania’s. The gap shrank during 1991, long before any rise. This is the assumption of parallel trends, failing in plain sight. All private jobs show the same direction, so part of it is New Jersey’s whole economy. But the restaurant gap is about five times larger. Something else moves restaurant jobs too, and we do not know what.

After the rise, the restaurant line goes up. But it was already going up. Any method must separate the rise from this recovery.

The baseline: before and after, and the 2 × 2

Section titled “The baseline: before and after, and the 2 × 2”

The plan’s first two methods use whole years: the year after the rise (1992 Q2 to 1993 Q1) against the year before (1991 Q2 to 1992 Q1). Whole years contain every season once, so the seasons cancel.

naive, New Jersey before and after: +2.5%
2 x 2: New Jersey +2.5%, Pennsylvania +2.7%, difference -0.2%
Year before Year after Change
New Jersey 100 102.5 +2.5%
Pennsylvania 100 102.7 +2.7%
Difference-in-differences 2.5 − 2.7 ≈ −0.2%

What this tells us. The naive reading is “the minimum wage created jobs”: +2.5%. But Pennsylvania’s jobs rose 2.7% without any rise. The difference is almost zero, and the 2 × 2 has no TermConfidence intervalA range around a measured value, made by a method that holds the true value 95 times in 100. If the interval of a difference does not include zero, the difference is probably not chance.Learn more Estimate uncertaintyEssential Mathematics and Statistics for AI.

The regression uses every county and every quarter, as in Difference-in-differences. First, we take out each county’s own seasons: from each quarter, we subtract that county’s average for the same season. The model then has one effect per quarter (for what hit both states) and the effect of interest, nj:post. The errors are clustered by county.

def did_reg(p, start, end=16, trend=False):
q = p[p["t"] < end].copy()
q["post"] = (q["t"] >= start).astype(int)
q["y"] = q["y"] - q.groupby(["area_fips", "season"])["y"].transform("mean")
f = "y ~ nj:post + C(t)" + (" + nj:t" if trend else "")
m = smf.ols(f, q).fit(cov_type="cluster", cov_kwds={"groups": q["area_fips"].astype("category").cat.codes})
...
regression DiD, all PA (67 counties): -3.3% (95% -6.2% to -0.3%, p = 0.031)

The 67 counties are all the counties in the regression: 21 in New Jersey and 46 in Pennsylvania.

What this tells us. The regression says that New Jersey lost 3.3% of its restaurant jobs, and the interval does not contain zero. Hana would read: “the rise cost jobs, p = 0.031”. Before anyone says that, Priya tests the method.

A placebo test: a rise that never happened

Section titled “A placebo test: a rise that never happened”

At a market, you doubt a seller’s scale. You put an empty basket on it. If the scale shows 2 kg for nothing, you do not trust its 5 kg for your apples. You trust it when the empty basket weighs about zero.

A placebo date is the empty basket. We pretend that the rise happened a year early, in 1991, when nothing changed in New Jersey alone. We keep only the quarters before the real rise, so that the real rise cannot leak in. We run the method, unchanged. The true effect is zero, so any estimate is the method’s error.

Four steps from left to right. Keep only the quarters before the real rise, 1990 Q1 to 1992 Q1. Pretend the rise happened in 1991 Q1, then in 1991 Q2, in orange. Run the method, unchanged. Nothing changed then: the true effect is 0, so any estimate is error, in green. An arrow goes to a blue box: placebo error is the average size of the two fake estimates; the same measure for every method; the smaller, the more we trust it.
The placebo error: one measure for every method. We use two fake dates, a year before the real rise.

A tiny example with every value visible. Suppose that a method gives −6% at the first fake date and −4% at the second. Its placebo error is (6 + 4) ÷ 2 = 5 points. If the same method then says −3% at the real date, its answer is smaller than its own typical error on nothing. We cannot tell −3% from zero.

You met the placebo test in Test your own answer. There, a pass did not prove the answer. Here, we use the size of the failure as our one measure.

placebo, a fake rise in 1991 Q1: -7.2% (95% -9.8% to -4.5%)
placebo, a fake rise in 1991 Q2: -7.0% (95% -9.9% to -4.0%)

What this tells us. The method finds a “loss” of 7% in 1991, when New Jersey’s minimum wage did not change. Its interval is far from zero. So the empty basket weighs 7 kg. The real-date estimate (−3.3%) is smaller than the method’s error on nothing.

One more fact about 1991 Q2. The federal minimum wage rose on 1 April 1991, in both states, by the same amount. Restaurant pay was lower in Pennsylvania (138 dollars a week against 188 in New Jersey, in 1992 Q1). So that rise probably mattered more in Pennsylvania. That would make New Jersey look better, not worse. It cannot explain a fake loss of 7%.

Common mistake

An interval clustered by county, with only one treated state

What you see: a narrow interval and p = 0.031 at the real date. Placebo dates give “significant” effects too.

Why: the 21 New Jersey counties share one state economy. When New Jersey has a bad year, all of them have it together. Clustering by county treats them as 21 independent pieces of evidence. For the question “New Jersey against Pennsylvania”, there is really one treated unit and one comparison.

Fix: do not trust the interval alone. Test the method on placebo dates and placebo places, and compare the real estimate with those errors. More treated states, or many pairs of neighbouring counties, would give honest intervals.

Maybe all of Pennsylvania is the wrong comparison. The plan has three other regressions:

  • Eastern Pennsylvania only: 7 counties around Philadelphia and the Lehigh Valley, next to New Jersey (Bucks, Chester, Delaware, Montgomery, Philadelphia, Lehigh and Northampton). Neighbours may share more of New Jersey’s economy.
  • The share of private jobs: restaurant jobs divided by all private jobs in the same county. This takes out each county’s whole economy, such as the recession.
  • New Jersey’s own trend: a straight line through New Jersey’s gap before the rise, extended after it.
regression DiD, eastern PA -2.5% (95% -5.7% to +0.8%)
regression DiD, share of private jobs -2.7% (95% -5.8% to +0.6%)
regression DiD, NJ's own trend +2.0% (95% -5.4% to +10.1%)

The chart shows each method’s real estimate and its estimate at the fake date 1991 Q2, both with their 95% intervals.

The four regressions at the real date (blue) and at the fake date 1991 Q2 (orange), with 95% intervals clustered by county. Every fake-date interval is below zero: each method finds a loss where nothing changed.
Text description of the chart

Eight rows on an axis from −22% to +12%, with a no-effect line at 0. All PA: real −3.3% (−6.2 to −0.3); fake −7.0% (−9.9 to −4.0). Eastern PA: real −2.5% (−5.7 to +0.8); fake −5.1% (−8.0 to −2.1). Share of private jobs: real −2.7% (−5.8 to +0.6); fake −5.5% (−8.5 to −2.4). NJ’s own trend: real +2.0% (−5.4 to +10.1); fake −7.0% (−12.8 to −0.8).

What this tells us. The comparisons move the real estimate between −3.3% and +2.0%. Every one of them “finds” a loss of 5% to 7% at the fake date. Eastern Pennsylvania and the share of private jobs help a little. New Jersey’s own trend flips the sign, but its fake estimates are the worst: a straight line cannot follow a fall that slows down.

A synthetic control builds a twin for New Jersey from a weighted mix of Pennsylvania counties, as in Synthetic control. The weights make the mix follow New Jersey before the rise. After the rise, the mix stands for “New Jersey without the rise”.

The plan uses only counties with at least 300 restaurant jobs. Small counties jump when one restaurant opens or closes. We saw this when we first tried all 46 counties. The method gave Union County a weight of 0.42. Union County had 134 restaurant jobs in 1993 Q1 and 246 in 1993 Q2, probably because of one new employer. With that donor, the “effect” was −11.6%.

synthetic New Jersey from 39 donor counties: Monroe County 0.52, Northampton County 0.31, Butler County 0.17
fit before: RMSPE 0.0265; effect after: -5.1%

What this tells us. Three counties make the twin. Monroe County is in the Pocono Mountains, a holiday area with its own seasons, like the Jersey Shore. The twin follows New Jersey within about 2.7% a quarter before the rise (RMSPE, the typical gap, from Module 5). After the rise, New Jersey is 5.1% below its twin on average.

Is the gap unusual? Every county plays New Jersey

Section titled “Is the gap unusual? Every county plays New Jersey”

As in Module 5, we give every donor county the same treatment. Each one gets its own twin from the other donors, and we measure its gap after the “rise”. The chart shows the 32 donors whose twin fitted about as well as New Jersey’s (a fit error at most twice New Jersey’s).

  • New Jersey
  • A Pennsylvania county as a fake New Jersey
The gap between each unit and its synthetic twin. Orange: New Jersey, 5.1% below its twin after the rise on average. Grey: 32 Pennsylvania counties, each treated as a fake New Jersey. Many grey lines move further than New Jersey's after April 1992.
Text description of the chart

A line chart from 1990 to 1993 with a dashed vertical line at 1 April 1992. New Jersey’s gap stays between −3.6 and +5.5 before the rise. After it: −1.2, +3.6 and −3.5 in 1992; −11.8, −7.0, −7.3 and −9.6 in 1993. The 32 grey lines spread between about −31 and +28.

New Jersey's post/pre gap ratio 2.7: rank 26 of 40 (p = 0.65)
placebo effects: -18.6% to +32.1%; 18 of 39 at least as large as New Jersey's

What this tells us. The ratio compares each unit’s gap after the rise with its gap before. New Jersey ranks 26th of 40: 25 fake New Jerseys moved more, relative to their fit. Its placebo p-value is 0.65. A county’s restaurant jobs often move by 5% in a year for reasons that we do not know. New Jersey’s −5.1% is ordinary.

A fair comparison: one measure for seven methods

Section titled “A fair comparison: one measure for seven methods”

Now every method faces the same test: its estimate at the real date, and its error at the two fake dates.

Method Real date Fake 1991 Q1 Fake 1991 Q2 Placebo error (points)
New Jersey before and after +2.5% −7.4% −4.7% 6.0
2 × 2, whole years, all Pennsylvania −0.2% −9.8% −7.1% 8.4
Regression, all Pennsylvania −3.3% −7.2% −7.0% 7.1
Regression, eastern Pennsylvania −2.5% −4.9% −5.1% 5.0
Regression, share of private jobs −2.7% −5.9% −5.5% 5.7
Regression, New Jersey’s own trend +2.0% −14.8% −7.0% 10.9
Synthetic control −5.1% −18.1% +0.7% 9.4
The placebo error of each method: the average size of its two fake estimates, where the true effect is zero. The regression with eastern Pennsylvania (blue) has the smallest error, 5.0 points. No method gets close to zero.
Text description of the chart

Seven rows on an axis from 0 to 12 points. New Jersey before and after 6.0. 2 × 2, whole years, all PA 8.4. Regression DiD, all PA 7.1. Regression DiD, eastern PA 5.0. Regression DiD, share of private jobs 5.7. Regression DiD, NJ’s own trend 10.9. Synthetic control 9.4.

What this tells us. By our one measure, the regression with eastern Pennsylvania is the most trustworthy method here. Its error on fake dates is 5.0 points. Its real estimate is −2.5% (−5.7% to +0.8%). That is half its own placebo error. Even the best method cannot tell its answer from its error on nothing.

The synthetic control did well at one fake date (+0.7%) and badly at the other (−18.1%). With only four quarters before 1991 Q1, the twin had too little to learn from. In Module 5, it was the best method. Here, with short and noisy data, it is not. A method’s rank depends on the data.

Card and Krueger (1994) surveyed 410 fast-food restaurants in New Jersey and eastern Pennsylvania, in February–March and again in November–December 1992. They found that employment rose in New Jersey relative to Pennsylvania: +2.76 full-time-equivalent workers per restaurant (standard error 1.36). It was a surprise, because the usual theory predicts fewer jobs.

The study started a long debate. Neumark and Wascher (2000) used payroll records from a sample of restaurants and found the opposite. Card and Krueger replied (2000) with BLS data from the same programme as ours, under its earlier name. They found that New Jersey’s fast-food employment grew at “similar or slightly faster” rates than eastern Pennsylvania’s.

We cannot use their data, but we can use their window: 1992 Q1 to 1992 Q4, New Jersey against eastern Pennsylvania.

1992 Q1 to 1992 Q4: New Jersey +6.7%, eastern Pennsylvania +3.2%, difference +3.4%

What this tells us. With their window, our public data also shows New Jersey growing a little faster: +3.4%. With whole years and a regression, it shows −2.5%. A placebo error of 5 points covers both. So the honest summary agrees with the reply of 2000: no clear loss of jobs, and no precise answer either.

Sources (cited and linked only):

  • Card, D., and Krueger, A. B. (1994). “Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania.” American Economic Review 84(4), 772–793. Data: David Card’s data page (no licence stated).
  • Neumark, D., and Wascher, W. (2000). “… Comment.” American Economic Review 90(5), 1362–1396. doi:10.1257/aer.90.5.1362.
  • Card, D., and Krueger, A. B. (2000). “… Reply.” American Economic Review 90(5), 1397–1420. doi:10.1257/aer.90.5.1397. Working paper: NBER w6386.

Priya writes to Hana in her words.

  • This data cannot measure a small effect. The best method says −2.5%, with an error of 5 points on dates when nothing happened. A loss of about 2,200 restaurant jobs and a gain of about 300 are both inside its interval.
  • It does show that a large loss is unlikely. A loss of 15% would be three times the best method’s placebo error. No planned method came close: the most negative answer was −5.1%.
  • So do not plan for a large cut. Plan the partner’s capacity for about the same staff. Watch the partner’s hours, not only its head count: the data counts jobs, and a minimum wage can change hours.
  • Do not accept a single number from either article. Ask the writers for their placebo tests.

The cost of the rise to an employer is certain. The job effect is not. A worker on 25 hours a week (our assumption) costs 0.80 × 25 × 52 = 1,040 dollars a year more. That is the number that Hana can plan with today.

The study’s value is in jobs. New Jersey had about 38,300 limited-service restaurant jobs in the year before the rise. A placebo error of 5.0 points is about 1,900 jobs. Any effect smaller than that is inside the noise of this data.

A method is useful only if it works on data that it was not chosen on. Priya runs the regression with eastern Pennsylvania, unchanged, on two other industries, with the same placebo test.

full-service restaurants: +1.3% (95% -1.2% to +3.7%); placebo -2.0%, -1.7%
all food services and drinking places: -0.8% (95% -2.2% to +0.6%); placebo -3.1%, -2.8%

What this tells us. Both estimates are near zero, and the errors on fake dates (2 to 3 points) are as large as the estimates. The method behaves the same way on new input: it rules out large effects, and it cannot find a small one.

  • The checks matter more than the estimate. Without the placebo dates, the regression’s −3.3% (p = 0.031) looks like a finding. With them, it is noise.
  • One treated state is one treated unit. Clustering by county gave intervals that were too narrow. The placebo dates and the placebo counties showed it.
  • There is no truth to check against. The placebo tests show how the methods fail on nothing. They do not show the true effect.
  • We have only two fake dates. The data starts in 1990, the first year of these files. A longer history would give more fake dates and a better measure.
  • Jobs, not hours, and one industry code. The data cannot see hours. Limited-service restaurants are close to fast food, but not the same.
  • What a fair study would need. Many treated and untreated places, such as many pairs of neighbouring counties across state borders. Then one state’s luck does not decide the answer. Longer histories before the change. Hours and wages, not only jobs. For its own decisions, Larkfield can do better: a change in its own pay can roll out to its sites in a random order, and then it is a test.
  • The data has no people in it. It counts jobs by county and industry, and BLS hides counties where one employer could be seen. Do not try to find out which firm is behind a number.
  • Minimum wages are a political subject. This page answers a narrow question about method: what this data can show. It gives no opinion about minimum-wage policy. A small effect on jobs says nothing about workers’ pay, prices or profits, which we did not study.
  • Say what you could not check. A result like “−3.3%, p = 0.031” travels fast. Always report it with its placebo tests, or do not report it.
  • Credit the sources. Public domain asks for nothing, but BLS asks to be cited. Cite Card and Krueger for their study, and do not copy their data.

This case study shows that you can study a change that nobody randomised, and that you test your own answer before you trust it. Make it yours with one change, such as another industry. Then make it public and easy to read in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • PLAN.md: the plan from “The plan before the estimates”, written before the estimates.
  • results.csv: each method’s estimate, its fake-date estimates and its placebo error.
  • extract_qcew.py and the extract’s SHA-256, so that anyone can rebuild the data from BLS.
  • The source and licence, with the citation above.

A README template:

# A minimum wage without a test: New Jersey 1992, with public data
## Problem
New Jersey raised its minimum wage on 1 April 1992; Pennsylvania did not.
Did restaurant jobs change? No randomisation: a natural experiment.
## Data and licence
US Bureau of Labor Statistics, Quarterly Census of Employment and Wages,
1990-1993 (public domain). Extract: limited-service restaurants (NAICS 722211),
21 NJ and 46 PA counties with all 16 quarters (95.8% of PA's jobs).
## Approach
Plan written first. Seven methods: before/after, 2 x 2, four regressions
(clustered by county), synthetic control. One measure: the error at two
fake dates a year early, where the true effect is zero.
## Results
Best method (eastern PA): -2.5% (95% -5.7% to +0.8%), placebo error 5.0
points. All PA: -3.3% (p = 0.031), but -7.0% at a fake date. Synthetic
control -5.1%, placebo rank 26 of 40. Card-Krueger window: +3.4%.
## Limits
One treated state; jobs not hours; two fake dates; effects smaller than
about 5% (1,900 jobs) cannot be told from noise.

How to describe it in an interview, in three sentences: “I estimated the effect of a real minimum-wage rise on restaurant jobs with public data and seven methods. A regression gave a ‘significant’ loss of 3%, but the same method found a 7% loss at a fake date a year early. So I compared every method by its error on fake dates. The best method’s estimate was smaller than its own placebo error, so I concluded that the data rules out a large loss but cannot measure a small effect.”

Go further:

  • Add 1994 and 1995 from BLS with extract_qcew.py. Stop before 1 October 1996, when the federal minimum wage rose in Pennsylvania too. Does the estimate settle?
  • Add a third state next to New Jersey, such as Delaware or New York. Check its minimum wage in those years first. Do the placebo errors shrink?
  • Compare with the other case studies, Did the mailing work, and for whom? and Real tests at an online shop. What did randomisation make easy there that was hard here?

Your turn: the 2 × 2 with eastern Pennsylvania

Section titled “Your turn: the 2 × 2 with eastern Pennsylvania”

Before you run it, predict: the 2 × 2 on whole years gave −0.2% with all of Pennsylvania. With only eastern Pennsylvania, will it be higher or lower? In the notebook, section 8, use did_2x2 with the panel east, and put the estimate in east_2x2.

Show the answer
east_2x2 = round(did_2x2(east, START)["estimate"], 1)
print(east_2x2)
-0.9

Eastern Pennsylvania grew 3.4% in the same year, a little faster than all of Pennsylvania (2.7%). So the difference is a little lower: −0.9%. The choice of comparison moves the answer by less than a point, far less than the placebo error.

Check your understanding

The regression with all of Pennsylvania gives −3.3% (95% −6.2% to −0.3%, p = 0.031). Hana says: 'So the rise cost jobs.' What does Priya answer, and why?

Show the answer

Not from this evidence. Priya ran the same method on fake dates in 1991, when New Jersey’s minimum wage did not change. It found losses of 7.2% and 7.0%, with intervals that also excluded zero. So the method finds “significant” losses where there were none. Its error on nothing is larger than its real estimate. The interval is too narrow because the 21 New Jersey counties share one state economy, and clustering by county treats them as independent. The honest reading is that the data cannot tell a small effect from zero.

Check your understanding

Why does Priya compare the methods by their estimates at fake dates, and not by their p-values or by how close they are to Card and Krueger?

Show the answer

Real data has no known true effect, so no method can be checked against the truth at the real date. At a fake date before the real rise, the truth is known: zero. A method’s estimate there is pure error, and its size is a fair measure that works the same way for every method. P-values only say how a method judges itself, and they were small even at the fake dates. Card and Krueger used different data and a different window, so being close to them does not prove that a method is right.

Check your understanding

In Module 5, the synthetic control was the most accurate method. Here, its placebo error is 9.4 points, one of the worst. Give two reasons from this case.

Show the answer

First, the data is short. There are only 9 quarters before the rise, and only 4 before the first fake date. The twin has too little history to learn New Jersey’s ups and downs, so it fails badly at the fake date 1991 Q1 (−18.1%). Second, the donors are small and noisy. A county’s restaurant jobs can jump by more than 80% when one employer opens, as Union County’s did. With all counties as donors, that county took a weight of 0.42, and the estimate moved to −11.6%. In Module 5, the simulator gave 78 weeks of smooth data. A method’s strength depends on the data that it gets.

Recap

  • New Jersey’s 1992 rise is a natural experiment: one state changed, and its neighbour did not. Nobody randomised it.
  • The data is public BLS data in the public domain, cut from 760 MB to a 0.5 MB extract with one script. For the hidden counties, we used the 46 Pennsylvania counties with all quarters (95.8% of the jobs) and checked the state totals.
  • Before the rise, New Jersey’s restaurant jobs grew up to 17 points less than Pennsylvania’s: parallel trends failed in plain sight.
  • A regression gave −3.3% with p = 0.031, but the same method found −7% at a fake date. With one treated state, intervals clustered by county are too narrow.
  • By one measure, the placebo error, the best method was the regression with eastern Pennsylvania: −2.5%, with an error of 5.0 points. The data rules out a large loss, but it cannot measure a small effect.

Next, the final assignment: a new Larkfield scenario with planted problems, where you use every tool of the course.

Next: Final assignment