Monitor a model on public data that changes

What you will be able to do

Monitor a model while the world around it changes for real. You will train a small model on New York City’s 311 requests from early 2020. Then you watch it month by month through a lockdown, a storm and new kinds of complaint. You will find out when the drift mattered, when it did not, and what only the late labels could show.

Case studyPortfolio piece

Dataset

NYC 311 Service Requests (NYC Open Data, erm2-nwe9), 2020 and January 2021

Licence

NYC Open Data Law: no licence requirement, no restrictions on use

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Delayed labels
• Input and prediction drift (PSI)
• Estimate from the mix
• Accuracy with intervals
• Slices of errors
• Alarm policies
• Retraining policies
• Monthly monitor report

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is a notebook. The work is analysis: train, score, count and compare, month by month. The notebook runs every step of this page, with the same code and the same output, plus four tasks with checks. It installs pandas and scikit-learn and downloads 2.2 MB of data. It runs in about a minute on a laptop CPU, with no account, no key and no Docker.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• Data drift, prediction drift, concept drift and performance decay, and the PSI with its rule of thumb, from Data and concept drift.
• Labels that arrive late and the Wilson interval, from Delayed labels.
• An objective written before you measure, from Service objectives.
• Diagnose first, then choose the smallest change, from Retrain or revise. A new model is a release with a gate, from Evaluation gates.

What you need on your computer (only for the local route; the notebook needs none of it)

• A folder with a virtual environment that has pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The project ticket-api is not used.
• The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell makes a folder monitor-311 and downloads the data into it. To start again, delete that folder.

Tested on macOS (Apple silicon) with Python 3.12.13, pandas 3.0.6 and scikit-learn 1.9.1, in Jupyter from a fresh start. Windows and Linux are not tested.

The situation

Priya has watched drift in Larkfield’s practice runs, where the course made each change on purpose. Now she wants to see a monitor meet a change that nobody planned. A city’s open data has one: New York City’s 311 requests in 2020.

311 is the city’s line for problems that are not emergencies: noise, no heat, a broken street light, a damaged tree. A resident calls, writes online or uses the app. The city sends the request to an agency, and the agency closes it when it has acted.

Priya imagines a small, useful model for such a service. When a request arrives, the confirmation page says “most requests like yours are closed within a week”, or “this kind of request often takes longer”. Both errors cost something. A wrong “within a week” is a broken promise, and the resident calls again. A wrong “longer” makes people expect too little, or give up.

The team, the model and the confirmation page are constructed for this case study. NYC 311 does not use them. The requests, the dates and the changes are real.

What you will build

You will build a logistic regression that predicts “closed within 7 days” from what is known on arrival. You train it on January 2020 and check it on February. Then you build its monitor with the course’s methods: input drift, prediction drift, labels that arrive late, and accuracy with an interval. You run the monitor on every month from March to December 2020. At the end, you run its report on a month that it has never seen, January 2021.

The question is the course’s question, asked of a city: how would we know that the model is failing residents, before they tell us?

The data

The data is 311 Service Requests from 2020 to Present on NYC Open Data, dataset erm2-nwe9. It has over 22 million requests and grows every day. The course took a fixed 1-in-20 sample: every request whose unique_key is a multiple of 20, from January 2020 to January 2021. It kept 7 columns, with no addresses, coordinates or descriptions.

Source and terms

NYC 311 Service Requests from 2020 to Present (NYC Open Data, erm2-nwe9), City of New York, published by the Office of Technology and Innovation. The NYC Open Data Law, NYC Administrative Code §23-502, applies. Public data sets have no licence requirement and no restrictions on use. The City may ask users to name the source and version and to describe their changes. Version and changes: Nextia Learning downloaded a fixed 1-in-20 sample of January 2020 to January 2021 from the public API. It kept 7 columns and changed no value. The data comes “as is”, without warranties. The dataset card in the notebook’s folder has the checksums.

The source changes every day: agencies close old requests and correct rows. A new download can differ a little, so the notebook uses the fixed copy and checks its checksum. Priya loads it:

raw = load("data/nyc311_2020.csv.gz")
print(f"{len(raw):,} requests, {raw['month'].min()} to {raw['month'].max()}")
columns = ["created_date", "closed_date", "agency", "complaint_type", "borough", "open_data_channel_type"]
print(raw[columns].head(3).to_string(index=False))

147,015 requests, 2020-01 to 2020-12
           created_date             closed_date agency complaint_type   borough open_data_channel_type
2020-01-01T00:00:00.000 2020-01-02T09:51:29.000  DOHMH Food Poisoning     BRONX                 ONLINE
2020-01-01T00:00:00.000 2020-01-02T00:00:01.000  DOHMH Food Poisoning    QUEENS                 ONLINE
2020-01-01T00:09:39.000 2020-01-06T17:06:32.000    HPD       PLUMBING MANHATTAN                  PHONE

What this tells us. One row is one request. The model may use only what is known when the request arrives: the complaint type (174 types in 2020), the agency (17), the borough, the channel, and the hour and weekday of created_date. closed_date is the future: it makes the label.

First checks

Before anything is built on the dates, Priya checks them:

print("closed before it was created:", int((raw["days_to_close"] < 0).sum()))
print("closed in the same second:   ", int((raw["days_to_close"] == 0).sum()))
print("still open (no closed date): ", int(raw["closed"].isna().sum()))
print("no borough:                  ", int((raw["borough"] == "Unspecified").sum()))
print("channels:", raw["channel"].value_counts().to_dict())

closed before it was created: 1001
closed in the same second:    4073
still open (no closed date):  1768
no borough:                   2211
channels: {'PHONE': 57507, 'ONLINE': 45267, 'MOBILE': 26918, 'UNKNOWN': 17298, 'OTHER': 25}

What this tells us. 1,001 requests close before they were created. That is impossible, so prepare() drops them. The 4,073 that close in the same second often got an answer at once, with information; they stay. The 1,768 without a closed date are still open, more than five years later. They were not closed within a week, so their label is 0.

Frame the problem: a label that arrives late

The model answers when a request arrives: will it be closed within 7 days? The true answer comes later. It is known when the request closes, or 7 days after it arrived, whichever comes first. After 7 days, “not within a week” is certain, whatever happens next.

This is a delayed label, as in Delayed labels, with one difference: here every request gets its label, not only a sample. prepare() adds the label and the time when it becomes known:

requests = prepare(raw)
wait = (requests["label_known_at"] - requests["created"]).dt.total_seconds() / 86400
print(f"{len(requests):,} requests kept")
print(f"label known within 1 day: {(wait <= 1).mean():.1%}; median wait {wait.median():.2f} days; "
      f"waited the full 7 days: {(requests[TARGET] == 0).mean():.1%}")

146,014 requests kept
label known within 1 day: 56.8%; median wait 0.38 days; waited the full 7 days: 18.0%

What this tells us. Most labels come fast, because most requests close fast. 18% wait the full week: the requests that did not close in time. Remember this: the slow labels belong to the hard requests.

Explore what the monitor needs

Priya does not look at every column. She asks one question that the monitor needs: how does the mix of complaint types move, month by month?

• HEAT/HOT WATER (seasonal)
• Non-Emergency Police Matter
• Illegal Fireworks
• Damaged Tree

The share of four complaint types in each month of 2020, from the 1-in-20 sample. Heat complaints fall every summer. The other three are 2020's own changes: crowds reported in the lockdown, fireworks in June, and Tropical Storm Isaias in August.

Text description of the chart

The x axis shows the months of 2020. HEAT/HOT WATER, drawn dashed, is 13% in January, falls under 1% from June to September, and is back at 16% in December. Non-Emergency Police Matter is under 0.5% in January and February, jumps to 15% in April and 13.5% in May, and is back under 1% from August. Illegal Fireworks is near 0 except in June (9.4%) and July (5.7%). Damaged Tree is under 2.5% in every month except August, when it is 13.1%.

Three kinds of change are in this chart:

• A season. Heat complaints fall every summer and come back in October. That is normal, and it happens every year.
• Events. In April and May, residents reported crowds during the lockdown, and “Non-Emergency Police Matter” grew about tenfold. June brought illegal fireworks. On 4 August, Tropical Storm Isaias brought down trees across the city. The sample has 758 “Damaged Tree” requests on that one day.
• New kinds of request. Some types did not exist in February:

before = set(requests.loc[requests["month"] <= "2020-02", "complaint_type"])
new = requests[~requests["complaint_type"].isin(before)]
print(new.groupby("complaint_type")["created"].agg(first="min", requests="size")
      .query("requests >= 20").sort_values("first").to_string())

                                                  first  requests
complaint_type
COVID-19 Non-essential Construction 2020-04-02 08:42:57       321
Illegal Animal Kept as Pet          2020-04-06 15:59:55        27
Mosquitoes                          2020-06-04 13:27:52        20
NonCompliance with Phased Reopening 2020-06-08 10:31:05      2542
Outdoor Dining                      2020-07-01 18:35:50       197
Standing Water                      2020-07-22 00:00:00        37
Real Time Enforcement               2020-09-29 09:08:47        59
Private or Charter School Reopening 2020-10-13 11:04:13        51
Mass Gathering Complaint            2020-10-19 22:03:21       108

The model will meet types that it has never seen. Its encoder ignores an unknown type, so the model still answers from the agency, the borough, the channel and the time.

A baseline, alternatives and a fair comparison

Priya trains on January 2020 and compares the candidates on February 2020, a month that she trusts. February also becomes the reference for every drift check. All candidates get one measure: accuracy, the share of requests with a right prediction, with its 95% interval.

train = requests[requests["month"] == "2020-01"]
reference = requests[requests["month"] == "2020-02"]
model = make_model().fit(train[FEATURES], train[TARGET])
boosted = make_boosted().fit(train[FEATURES], train[TARGET])
candidates = {
    "always 'within a week'": np.ones(len(reference), dtype=int),
    "lookup by complaint type": lookup_predict(train, reference),
    "logistic regression": score(model, reference)["prediction"].to_numpy(),
    "boosted trees": (boosted.predict_proba(reference[FEATURES])[:, 1] >= 0.5).astype(int),
}
print(f"February 2020, {len(reference):,} requests: accuracy [95% interval]")
for name, prediction in candidates.items():
    good = int((prediction == reference[TARGET].to_numpy()).sum())
    low, high = wilson(good, len(reference))
    print(f"  {name:26s} {good / len(reference):.1%} [{low:.1%}, {high:.1%}]")

February 2020, 8,558 requests: accuracy [95% interval]
  always 'within a week'     83.2% [82.4%, 84.0%]
  lookup by complaint type   88.4% [87.7%, 89.0%]
  logistic regression        88.9% [88.3%, 89.6%]
  boosted trees              88.8% [88.1%, 89.4%]

What this tells us. The trivial baseline is already 83.2%, because most requests close within a week. The lookup table says “within a week” if most requests of the same type closed within a week in January. It adds 5 points: the complaint type does almost all the work. Logistic regression on six inputs adds half a point more, and the intervals overlap. Boosted trees add nothing.

Priya keeps the logistic regression. It is as good as the best, small and readable. It also has an answer for a type that it has never seen, where the lookup table has one overall guess. Then, before any monitoring, she writes the objective: a month’s accuracy should be at least 87%, February’s 88.9% minus 2 points. A month is below objective when its whole interval is under 87%, and watch when only the point is.

The monitor

When each signal is known

The monitor has four signals. They arrive at different times, and the time decides what each one can do:

August 2020: what the monitor knows, and when. Quick signals say where to look. Only the complete labels, a week after the month ends, measure the mistakes.

Text description of the diagram

For August’s requests, input drift and prediction drift are known as the requests arrive. At the end of the month, the estimate from the mix and the early labels are known too. Every label is known on 8 September, and only then is the accuracy complete. In August, the early signals pointed the right way. In September, they did not: only the complete labels showed the drop.

Input drift is the PSI of the complaint type, the agency and the channel against February. Prediction drift is the share predicted “within a week” (84.2% in February). The accuracy needs the complete labels.

A new signal: the estimate from the mix

Drift says that the mix moved. It does not say whether the move hurts. One cheap step helps.

Think of a delivery driver whose day gets harder: more stairs, fewer lifts. Her manager can guess the extra time from the new mix of stairs and lifts, before she comes back. He cannot guess that a lift broke.

The same idea works for the model. Take February’s accuracy for each complaint type, and weight it by this month’s mix. This is the estimate from the mix: the accuracy to expect if only the mix had changed. It needs no labels.

The estimate from the mix on a constructed example: the same model, a harder mix, a lower expected accuracy, before any label arrives.

Text description of the diagram

The model is right on 95% of noise requests and 60% of tree requests. With 90% noise and 10% trees, the expected accuracy is 91.5%. After a storm, with 60% noise and 40% trees, it is 81%. The estimate sees a harder mix. It cannot see a type that starts to behave differently.

Use it when one input explains most of the model’s errors, as the complaint type does here. It costs one table of February’s accuracy per type, and one weighted average per month.

Month by month

Predict. Which month will have the largest input drift? Will it also have the lowest accuracy?

reference_scored = score(model, reference)
months = [f"2020-{m:02d}" for m in range(3, 13)]
monthly = month_table(model, reference_scored, requests, months)
print(monthly.drop(columns="requests").to_string())

         psi_type  psi_agency  psi_channel  predicted_in_week  estimate_from_mix  accuracy    low   high           status
month
2020-03     0.234       0.122        0.009              0.808              0.867     0.878  0.871  0.885  meets objective
2020-04     1.339       0.330        0.138              0.855              0.881     0.894  0.887  0.900  meets objective
2020-05     1.291       0.296        0.208              0.873              0.907     0.911  0.906  0.916  meets objective
2020-06     1.821       0.387        0.210              0.853              0.905     0.895  0.890  0.900  meets objective
2020-07     1.500       0.409        0.136              0.818              0.882     0.869  0.863  0.874            watch
2020-08     1.454       0.651        0.140              0.720              0.830     0.839  0.833  0.844  below objective
2020-09     1.022       0.357        0.263              0.827              0.889     0.861  0.855  0.867  below objective
2020-10     0.684       0.360        0.210              0.831              0.890     0.869  0.864  0.875            watch
2020-11     0.560       0.313        0.103              0.830              0.888     0.887  0.881  0.893  meets objective
2020-12     0.559       0.332        0.086              0.833              0.891     0.880  0.874  0.886  meets objective

The two charts show the first column and the accuracy side by side:

Input drift of the complaint type in each month, against February 2020. Every month is above 0.2, the rule of thumb for a large change.

Text description of the chart

A column chart with ten columns, March to December 2020. March 0.23, April 1.34, May 1.29, June 1.82, July 1.50, August 1.45, September 1.02, October 0.68, November 0.56, December 0.56. Every column is above 0.2, the line for a large change.

• Meets objective
• Watch: the point under 87%
• Below objective: the whole interval under 87%

The accuracy of each month on all its labels, with its 95% interval and the number of requests. Only August and September are below the 87% objective.

Text description of the chart

Ten rows, one per month, with the accuracy and its 95% interval, and a dashed line at the 87% objective. March 87.8%, April 89.4%, May 91.1% and June 89.5% are right of the line. July 86.9% and October 86.9% have the point just left of the line, and the interval crosses it. August 83.9% (83.3% to 84.4%) and September 86.1% (85.5% to 86.7%) are fully left of the line. November 88.7% and December 88.0% are right of it.

What this tells us. The input drift is large in every month. The largest is June (1.82: fireworks and noise), and June’s accuracy, 89.5%, is better than February’s. April and May have a PSI above 1.2, and they are the model’s best months: the lockdown requests that grew were easy ones. Drift alone does not prove that the model is wrong.

The model really was worse in August (83.9%) and September (86.1%). Prediction drift moved in August, from 84% to 72% “within a week”, and hardly moved in September. Tomás would retrain now. Priya first asks why.

Where the model really was wrong

What the estimate saw, and what it missed

Put the estimate from the mix and the early labels next to the measured accuracy:

• Accuracy on all labels
• Estimate from the mix
• Early labels, at the end of the month
• Objective 87%

Three ways to read a month's accuracy. The estimate from the mix and the early labels are known at the end of the month; the accuracy on all labels a week later. In August all three fall below the objective. In September only the complete labels do.

Text description of the chart

The x axis shows March to December 2020. The accuracy on all labels goes 87.8%, 89.4%, 91.1%, 89.5%, 86.9%, 83.9%, 86.1%, 86.9%, 88.7%, 88.0%. The estimate from the mix goes 86.7%, 88.1%, 90.7%, 90.5%, 88.2%, 83.0%, 88.9%, 89.0%, 88.8%, 89.1%. The early labels go 88.2%, 90.1%, 92.4%, 91.4%, 88.3%, 84.8%, 88.0%, 87.7%, 89.8%, 89.2%. In August all three lines dip below the dashed 87% line. In September the estimate and the early labels stay above it, and only the accuracy on all labels is below.

In August, the estimate (83.0%) saw the drop at the end of the month, a week before the labels. The mix had moved toward types that were already hard. In September, the estimate said 88.9%, and the truth was 86.1%. The mix does not explain that drop.

The slices

The notebook’s wrong_by_type() lists the complaint types with the most wrong predictions in a month:

for month in ["2020-08", "2020-09"]:
    print(month); print(wrong_by_type(month).to_string()); print()

2020-08
                                     requests  wrong  closed_in_week  predicted_in_week  february_closed_in_week
complaint_type
Damaged Tree                             2279    772            0.33               0.03                     0.55
Dirty Conditions                          222    178            0.20               1.00                     0.91
NonCompliance with Phased Reopening       473    175            0.63               1.00                      NaN
Request Large Bulky Item Collection      1579    165            0.90               1.00                     0.86
UNSANITARY CONDITION                      349    126            0.36               0.01                     0.29

2020-09
                                     requests  wrong  closed_in_week  predicted_in_week  february_closed_in_week
complaint_type
Request Large Bulky Item Collection      1402    249            0.82               1.00                     0.86
NonCompliance with Phased Reopening       372    208            0.44               1.00                      NaN
Sidewalk Condition                        226    137            0.82               0.27                     0.45
UNSANITARY CONDITION                      363    133            0.36               0.01                     0.29
Dirty Conditions                          198    122            0.38               1.00                     0.91

What this tells us. August had three different problems:

• A hard slice that grew. The storm brought 2,279 “Damaged Tree” requests, and a third of them closed within a week. The model said “longer” for almost all of them, which is right two times in three. Without the trees, August’s accuracy is 86.5%: better, and still under the objective.
• A relationship that changed. “Dirty Conditions” closed within a week 91% of the time in February and 20% in August. The model still said “within a week”. This is concept drift: the same request, a different outcome. The data does not say why the agency got slower. It only shows that the old relationship stopped holding.
• A type that the model never saw. “NonCompliance with Phased Reopening” started in June. The model guessed “within a week” from its agency. In September, only 44% closed in time.

September has mostly the second and third kinds. The estimate from the mix cannot see them, because it assumes that each type behaves as in February. Only labels can.

Common mistake

Reading the labels too early

At the end of a month, about 93% of its labels are already known. It is tempting to judge the month then:

for month in ["2020-07", "2020-08", "2020-09", "2020-10"]:
    scored = score(model, requests[requests["month"] == month])
    end = pd.Period(month, "M").end_time.ceil("D")  # midnight after the last day
    early, full = accuracy(scored, as_of=end), accuracy(scored)
    print(f"{month}: labels known at month end {early['known'] / early['requests']:.1%}, "
          f"accuracy on them {early['accuracy']:.1%}; on all labels {full['accuracy']:.1%}")

2020-07: labels known at month end 93.7%, accuracy on them 88.3%; on all labels 86.9%
2020-08: labels known at month end 93.7%, accuracy on them 84.8%; on all labels 83.9%
2020-09: labels known at month end 93.4%, accuracy on them 88.0%; on all labels 86.1%
2020-10: labels known at month end 92.9%, accuracy on them 87.7%; on all labels 86.9%

At the end of September, the early labels say 88.0%, above the objective. All the labels say 86.1%. The labels that arrive first belong to the requests that closed fast, and the model is right on those more often. Early labels are not a random sample of the month.

The fix: judge a window only when every label in it can be known: here, 7 days after the window ends. Or measure only the requests that are old enough. Until then, the monitor’s report says “labels incomplete”.

Compare the monitors and the retraining policies

Which alarm to trust

Each signal can raise an alarm. The months that were really bad are those below objective on the complete labels: August and September. The notebook counts the alarms of each monitor from March to December:

Monitor	Known	Alarms	Bad months caught	False alarms
Drift: PSI of the type above 0.2	end of the month	10	2 of 2	8
Estimate from the mix under 87%	end of the month	2	1 of 2 (August)	1 (March)
Early labels under 87%	end of the month	1	1 of 2 (August)	0
Complete labels below objective	7 days later	2	2 of 2	0

The drift alarm “catches” both bad months only because it fires every month. A person who gets ten alarms for two problems soon stops reading them. The complete labels are right every time, and they are a week late. That week is the price of the truth.

Retrain on drift?

In Module 4, Tomás said: “Drift alert fired: retrain automatically.” Here the drift alarm fires every month, so that means a new model every month. Priya compares three policies with one measure: the share of all requests from March to December with a right prediction. A new model can start on the 8th of a month, when the labels of the month before are complete. bad is the set of months below objective:

for policy in ["keep", "every_month", "after_label_alarm"]:
    result = simulate_retraining(policy, model, requests, months, bad)
    under = [m for m, a in result["by_month"].items() if a < OBJECTIVE]
    print(f"{policy:18s} accuracy {result['overall']:.1%}  new models {result['releases']:2d}  months under 87%: {under}")

keep               accuracy 87.6%  new models  0  months under 87%: ['2020-07', '2020-08', '2020-09', '2020-10']
every_month        accuracy 88.6%  new models 10  months under 87%: ['2020-08']
after_label_alarm  accuracy 87.6%  new models  2  months under 87%: ['2020-07', '2020-08', '2020-12']

What this tells us. Here, a fresh model every month is 1.0 point better, with 10 new models. Tomás is right that a fresher model helps on this data. But the drift alarm is not the reason: it fires every month, so “retrain on drift” is a monthly schedule. August still fails, because no model trained on July knows about a storm.

Retraining once on the failing month helps September and October, and hurts November and December. A model trained on one odd month learns that month: a model of September knows nothing of winter heat complaints. Every new model is a release that needs a check before it goes live, as in Evaluation gates. This case study did not simulate that gate.

The recommendation

Priya writes it for the team, in the spirit of Retrain or revise:

• Act on the labels. Judge each month 7 days after it ends, on all its labels, against the 87% objective written in advance. Act when the whole interval is under it.
• Investigate on the quick signals. Drift and the estimate from the mix are known at the end of the month. When the estimate falls under the objective, or one type’s share jumps, look at that slice at once. A drift alarm opens an investigation, never a change.
• Never judge on early labels. The report says “labels incomplete” until the window is complete.
• Retrain on a schedule, with a gate. Monthly retraining gained 1.0 point here. Each new model must pass an evaluation on the newest complete month before it replaces the old one.
• Fix causes, not only models. The storm was an event: tell residents that tree requests take longer, for that week. Slower “Dirty Conditions” closures are a question for the agency, not for the model.
• Renew the reference. In 2021, the drift alarm still says “large change” against February 2020. Choose a new reference month whose labels you have checked.

The monitor on new input, end to end

January 2021 is new input: nothing so far used it. On 8 February 2021, every January label can be known. Priya runs the monitor’s report as it would read on that day:

january = prepare(load("data/nyc311_2021-01.csv.gz"))
print_report(month_report(model, reference_scored, january, "2021-02-08"), "2021-02-08")

Monthly report, read on 2021-02-08: 11,380 requests
Input drift against the reference month:
         input   psi  p_value        verdict
complaint_type 0.466 3.7e-210   large change
        agency 0.289 1.9e-171   large change
       channel 0.145 7.7e-147   small change
       borough 0.024  9.4e-24 no real change
          hour 0.012  5.9e-05 no real change
Prediction drift: predicted 'within a week' 84.0% (reference 84.2%)
Accuracy expected from the mix (no labels needed): 89.2%
Labels known: 100.0%; accuracy on them: 89.1% [88.5%, 89.7%]
Status: meets objective (objective 87%)

The same report a week earlier, on 1 February, says “labels incomplete”. 91.2% of the labels are known, and they say 90.8%: too high again.

What this tells us. A year after the reference month, the drift is still large, and the model meets its objective. The p-values are tiny everywhere, even for the hour, where nothing real changed. With thousands of requests, every small difference is “sure”. Read the PSI for how much and the p-value for how sure, as in Data and concept drift.

Insights and limits

• Drift points; labels decide. Drift was large in every month, and the model failed in two. In August, a hard slice grew, and the mix showed it. In September, a relationship changed and a new type came; only the labels showed it, a week late.
• Limits. One city, one year, one target and one model. Closure time is not quality: an agency can close a request without fixing the problem. The sample is 1 in 20, so small types have wide intervals. The 87% objective is an assumption. In the story, Priya writes it before she looks. The course chose it after a first look at the whole year, so read it as a teaching choice: with 86%, only August is below. The gate before a new model was not simulated.

Responsible use

• The data is about residents and their neighbourhoods. The sample has no addresses, coordinates or free text, and the method needs none. Do not join it with the full dataset to study a household.
• Reporting is not need. People in some neighbourhoods report more than others. A model can learn “requests from here close slowly”. It then turns a slow service into a lower promise for the same people. Check the errors by borough before you show a promise to residents.
• “Closed” is not “fixed”. Do not use this model to judge an agency’s work.
• Say what is constructed. The team, the model and the confirmation page are constructed for teaching. NYC 311 does not use them.

Make it a portfolio piece

This case study shows an employer that you can run a model after the launch. You can tell drift from failure, wait for the truth, and turn alarms into decisions. Make it yours with one change. Take other months (2021 to 2022, with a new reference), another target (“closed within 3 days”), or another city’s 311 data. Check that city’s licence first.

What to put in the repository:

• README.md (template below).
• The notebook, run from top to bottom, with its outputs and charts.
• MONITORING.md: the objective, the reference month, each signal with the time when it is known, and who acts on each alarm.
• The data source, the terms and the checksum of your copy. The NYC Open Data Law asks for the source, the version and your changes.

A README template:

# Monitoring a 311 "closed within a week" model through 2020

## Question
How would we know that a model trained before 2020 is failing residents,
and which signals can we trust, and when?

## Data and terms
NYC 311 Service Requests (NYC Open Data, erm2-nwe9), 1-in-20 sample,
Jan 2020 - Jan 2021, 7 columns, no addresses. NYC Open Data Law:
no restrictions on use; source, version and changes stated.

## Approach
Logistic regression on 6 inputs known on arrival; trained Jan 2020,
chosen on Feb 2020 (88.9%, lookup table 88.4%). Objective 87% per month,
written in advance. Monthly monitor: PSI (complaint type, agency, channel),
prediction drift, estimate from the mix, accuracy on complete labels
(Wilson interval). Alarm and retraining policies compared Mar-Dec 2020.

## Results
PSI of the type > 0.2 in all 10 months; accuracy below objective in 2:
Aug 83.9% (storm: a hard slice grew; the mix estimate saw it) and
Sep 86.1% (changed closure times, a new type; only labels saw it).
Early labels were optimistic (Sep: 88.0%). Monthly retraining +1.0 point.

## Limits
One city and year; closure time is not quality; gate not simulated.

How to describe it in an interview, in two sentences: “I monitored a 311 model through 2020. The input drift was large in every month, but the model failed its objective in only two. A storm explained August; a change in closure times, seen only in the late labels, explained September.”

Go further:

• Make the reference a rolling window (the last three complete months), and count the drift alarms that remain.
• Add the gate. Train a candidate on the first three weeks of a month, and test it against the current model on the last week. Replace the model only on a clear win.

Knowledge check

Check your understanding

In April and May 2020, the PSI of the complaint type was above 1.2, far over the 0.2 line. Was the model wrong in those months, and what does that teach about drift alarms?

Show the answer

No. April (89.4%) and May (91.1%) were the model’s best months, better than February’s 88.9%. The lockdown requests that grew, such as reports of crowds, were easy to predict. Drift says that the inputs moved, not that the predictions got worse. Here the drift alarm fired in all ten months, and only two were bad. So a drift alarm should open an investigation of the slice that moved. It should not page a person or trigger a change.

Check your understanding

The estimate from the mix warned about August but not about September. Explain both.

Show the answer

The estimate weights February’s accuracy for each complaint type by the month’s mix. In August, the storm made “Damaged Tree” 13% of requests, a type that the model was already weak on. So the estimate fell to 83.0%, close to the true 83.9%. In September, the drop came from types that started to behave differently (“Dirty Conditions” closed much more slowly) and from a new type. The estimate assumes that each type behaves as in February, so it said 88.9%. Only the complete labels showed 86.1%.

Check your understanding

At the end of September, 93% of September's labels are known, and they give 88.0%, above the 87% objective. Why does the monitor still say 'labels incomplete', and what was the true result?

Show the answer

The labels that arrive first belong to the requests that closed fast, and the model is right on those more often. So the early accuracy is too high: it is not a random sample of the month. September’s labels are complete 7 days after the month ends. Then the accuracy is 86.1% [85.5%, 86.7%], below the objective. Judge a window only when every label in it can be known.

Recap

• You trained a small model on January 2020 and chose it on February, with an objective written before any monitoring.
• You built a monthly monitor with four signals: input drift, prediction drift, the estimate from the mix and the accuracy on complete labels. Each one has its own time.
• Input drift was large in all ten months, and the model was below objective in two. August was a hard slice that grew. September was a changed relationship that only labels could show.
• Early labels were optimistic. Judge a window when all its labels can be known.
• Monthly retraining helped a little, as a schedule with a gate, not as a reaction to drift. A drift alarm opens an investigation, never a change.

Next, the other case study: a published outage, rebuilt in the course’s post-incident review format.

Next: Post-incident review of a public outage