Regression-test a classifier release by slice

What you will be able to do

Decide whether a new version of a classifier may replace the old one. It is better on average. You will check that claim the way a careful team does. You write rules and blockers first, use a frozen holdout, and read results per slice with paired intervals. Then the result, not the hope, decides.

Case studyPortfolio piece

Dataset

Civil Comments with identity labels (CivilComments-WILDS v1.0)

Licence

CC0 1.0 (public domain)

Time

About 2 hours

Runs in

Colab, Kaggle or VS Code

Skills used

• Release rules and blockers first
• Frozen holdout
• Slices of people
• Paired bootstrap intervals
• Saved outputs and run IDs
• Written release decision

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

The notebook runs every step of this page, with the same code and the same output, plus four tasks with checks. Its setup cells install pandas and scikit-learn and download the data (about 17 MB). It trains both model versions on the CPU and runs in about a minute on the course’s laptop. No account and no key are needed: no step calls a model service.

Content note

The data is real public comments from news sites. Some are insulting or hateful, also about groups of people. This page and the notebook never print a comment that the raters labelled toxic. They quote only harmless comments and sentences that the course wrote.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

This case study uses the whole course. These lessons matter most:

• Recall, false alarms and different error costs, from Prediction metrics.
• Dev and holdout sets, and why the holdout stays frozen, from Avoid contamination.
• Bootstrap intervals, the paired difference and slices with caution, from Uncertainty and slices.
• Blockers before averages, and a written decision, from Trade-offs and regressions and Establish a baseline.

You do not need to know how text classifiers work inside. The case study treats them as two versions that give each comment a score.

What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)

• Python 3.12 or later with pandas 3.0.6, scikit-learn 1.9.1 and matplotlib, in a new virtual environment, as in Set up the project.
• The notebook, downloaded from the link above, in VS Code or JupyterLab. Its setup cell downloads the data into a new folder classifier-release. To start again, delete that folder.

Tested on macOS (Apple silicon) with Python 3.12.13, pandas 3.0.6, scikit-learn 1.9.1 and numpy 2.5.3, in Jupyter from a fresh start. Windows and Linux are not tested.

The situation

The Harbourview Courier is a regional news site with a busy comment section. Every new comment gets a score from a text classifier. The comments with the highest scores are held: they wait for a moderator before anyone can read them. The rest appear at once. The model deletes nothing; a person decides every held comment.

Nadia leads the community desk and owns the comment policy. Her three moderators can read about one comment in ten. So the classifier’s job is to put the right 10% in front of them.

Each choice can go wrong in two ways:

• A missed toxic comment appears at once. Readers see an insult until somebody reports it.
• A harmless comment that is held waits for hours. Its writer often gives up. If this happens more to people who write about their own religion, race or sexuality, the site quietly silences them.

Kwame, the site’s machine-learning engineer, has built a new version. “v2 reads short pieces of words, not only whole words,” he says. “It catches disguised insults, and its overall score is higher. Let us ship it.”

Lena, the data scientist, asks the question of this course. How do we know the new version is better, and that it did not break anything that matters? This case study is her answer.

What you will build

You will build a release check for a classifier: written rules with blockers, two model versions trained as a team would, and a frozen holdout. You compare the versions overall and for eight groups of people, with paired bootstrap intervals, and write a decision for Nadia. You also look at the cases behind the numbers, and at what the old version already gets wrong.

It uses the whole course, from criteria before results (Map goals to evidence) to the order of a decision (Trade-offs and regressions).

The data

The data is Civil Comments: public comments from about 50 English-language news sites, written between 2015 and 2017. When the Civil Comments platform closed, it released the comments as an open archive. Jigsaw and Google’s Conversation AI team then had crowd raters label them. This case study uses the version of the WILDS benchmark: 448,000 comments that have labels for the groups of people they mention.

Source and licence

Civil Comments, with the toxicity and identity labels of Jigsaw / Conversation AI (Borkan, Dixon, Sorensen, Thain and Vasserman, “Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification”, 2019, arXiv:1903.04561), in the WILDS version (Koh et al., 2021). Released under CC0: “This dataset is released under CC0, as is the underlying comment text.” Nextia Learning took a fixed sample of 100,000 comments and kept 15 columns; nothing else changed. The dataset card is next to the notebook.

Who made the labels

Every label comes from crowd raters: paid workers who read a comment and answer questions about it, following the Perspective API’s rating guidelines. Two kinds of label matter here:

• Toxicity is the share of raters who chose “Toxic” or “Very Toxic”. A comment is toxic in this case study when that share is 0.5 or more.
• Identity labels say which groups of people a comment mentions. A comment is in a group when at least half of the raters chose that group.

These are the judgements of a few people per comment, not facts. Raters disagree, and each brings their own idea of what is toxic. The paper also says that earlier models chose some comments for identity labelling, so that raters would see identity content often. Keep both points in mind; the numbers below show how much they matter.

First checks

Lena loads the sample and counts the parts, the raters and the slices:

comments = load_sample("data/civil_comments_sample.csv.gz")
print(comments.groupby("part").agg(comments=("id", "size"), toxic=("toxic", "sum"), toxic_rate=("toxic", "mean")).round(3))
holdout = comments[comments["part"] == "holdout"]
print("\nraters per comment (holdout): median", holdout["toxicity_annotator_count"].median(),
      "| exactly 4 raters:", round((holdout["toxicity_annotator_count"] == 4).mean(), 3),
      "| toxicity exactly 0.5:", int((holdout["toxicity"] == 0.5).sum()), "of", int(holdout["toxic"].sum()), "toxic")
print("\nholdout slices:")
print(pd.DataFrame({g: {"comments": int(holdout[g].sum()), "toxic_rate": round(holdout.loc[holdout[g] == 1, "toxic"].mean(), 3)}
                    for g in GROUPS}).T)

The output:

         comments  toxic  toxic_rate
part
dev         10000   1088       0.109
holdout     30000   3393       0.113
train       60000   6708       0.112

raters per comment (holdout): median 4.0 | exactly 4 raters: 0.57 | toxicity exactly 0.5: 964 of 3393 toxic

holdout slices:
                 comments  toxic_rate
male               3270.0       0.158
female             3794.0       0.143
LGBTQ               999.0       0.269
christian          2991.0       0.096
muslim             1551.0       0.238
other_religions     792.0       0.144
black              1093.0       0.314
white              1771.0       0.290

What this tells us. About one comment in nine is toxic, in every part. Most comments have only 4 raters, and 964 of the 3,393 toxic holdout comments sit exactly on 0.5: often 2 raters of 4. On those, no version can be “right”. Both versions are measured against the same labels, so a comparison stays fair, but the absolute numbers carry this noise.

The slices differ a lot. Comments that mention Black or white people are toxic about 31% and 29% of the time, against 7.6% for comments that mention no group. Each slice has at least 792 comments in the holdout, so the slices are large enough to read with intervals.

The columns used

Column	What it means	Use
comment_text	The comment	The only input of the models
toxicity	Share of raters who said toxic	Target: toxic when 0.5 or more
toxicity_annotator_count	Number of toxicity raters	Label quality
male, female, LGBTQ, christian, muslim, other_religions, black, white	1 when at least half of the identity raters chose the group	Slices only, never an input
part	train, dev or holdout	The split

Frame the problem

One comment is one case. The decision is “hold for a moderator” or “show now”, made when the comment arrives, from its text alone. The operating point is fixed by the moderators’ time: each version holds the 10% of comments with its highest scores. Each version gets its own threshold, set on dev.

The identity columns are the most sensitive data here. They are never inputs, and they are never used to treat a comment differently. They exist only to check the models, slice by slice.

Lena measures two things per version, overall and in each slice:

• Recall: of the toxic comments, the share held.
• The benign-held rate: of the harmless comments, the share held. It is the false-alarm rate of Prediction metrics, read inside one slice.

Here is a tiny slice of 10 comments, constructed to show both measures. T1 to T3 are toxic and B1 to B7 are harmless.

A constructed tiny slice. v2 wins on recall and loses on the benign-held rate. Each measure reads only one row of comments.

Text description of the diagram

Ten comments in one slice: three toxic, T1 to T3, and seven harmless, B1 to B7. v1 holds T1, T2 and B1. Its recall is 2 of 3, 0.67, and its benign-held rate is 1 of 7, 0.14. v2 holds T1, T2, T3, B1, B2 and B3. Its recall is 3 of 3, 1.00, and its benign-held rate is 3 of 7, 0.43. v2 catches every toxic comment and holds three harmless ones.

v2 is better on recall and worse on the benign-held rate. In the real slices, the two measures can also move in opposite directions. So the rules watch both.

Rules first

Lena wrote the release rules before anybody trained a model. They are in release_rules.json, and the commit that added them is older than the training script. Here they are, in their fixed order:

1. B1, a blocker: identity statements. The course wrote 64 short, harmless sentences: four templates, such as “I am a … person.” and “… people come to this park every weekend.”, each with 16 words for groups. If v2 holds even one sentence that v1 shows, the release stops.
2. B2, a blocker: slice regression. The release stops if, in any of the 8 slices, v2 holds more harmless comments and the whole paired 95% interval of the difference is above +0.01.
3. G1, a gate: no overall loss. The lower end of the paired interval of recall, v2 − v1, must be at least −0.01.
4. G2, a gate: a measurable gain. That lower end must be above 0. The team replaces a working model only for a gain it can measure.
5. O1, a limit: speed. Scoring must take under 1 second per 1,000 comments.

Overall ROC AUC and per-slice AUC are reported, not gated. Why does B2 use the interval? A slice of a few hundred harmless comments moves by chance. A rule on the point value alone would stop good releases at random. “The whole interval above +0.01” means that we are fairly sure of a real rise. v2 then holds more harmless comments of this group, by more than one in a hundred.

The order of the work. The rules exist before any result, and the holdout is read once, after every choice.

Text description of the diagram

Five steps in order. Step 1: write the rules, before any model exists. Step 2: train v1 and v2 on 60,000 comments; the one change is character n-grams. Step 3: a first look on 10,000 dev comments, where each version’s threshold is set to hold 10% of them. Step 4: the holdout of 30,000 comments, read once, overall and for 8 slices, with paired intervals. Step 5: decide by the rules, in order; nothing changes after step 4. The rules come first so that no result can change them. The holdout comes last so that no choice can be tuned on it.

Two versions, one change

Both versions turn a comment into numbers with TF-IDF: each word or piece of text gets a weight. The weight is high when the piece is frequent in this comment and rare in others. A logistic regression then learns a weight per feature and gives each comment a score from 0 to 1. The machine learning course teaches these models; here they are just v1 and v2.

• v1, on the site today, reads words and pairs of words.
• v2 reads the same, plus character n-grams: runs of 2 to 5 letters, such as idio or diot. That is the one change. It should match insults that are disguised or misspelled.

def build(version):
    """v1: words and word pairs. v2: the same plus character n-grams. Same model."""
    features = words() if version == "v1" else FeatureUnion([("words", words()), ("chars", chars())])
    return Pipeline([("features", features), ("model", LogisticRegression(C=4, solver="liblinear"))])

Both train on the same 60,000 comments. On the course’s laptop, the notebook printed:

v1: trained in 5 s
v2: trained in 32 s

Saved outputs and run IDs

As in Establish a baseline, the evaluation never reads a live model. Lena saved the scores of both versions on dev and holdout once. Each run has a stable run ID: the version and the first characters of a hash of its saved scores, v1-13f555 and v2-8b2aeb. The notebook trains both versions again and compares its scores with the saved ones:

v1-13f555: largest difference 0.000000, held/shown decisions that differ: 0 of 40000
v2-8b2aeb: largest difference 0.000000, held/shown decisions that differ: 0 of 40000

What this tells us. The training is repeatable here. Another computer can differ in the last digits, and a comment right at a threshold could flip. So every number on this page comes from the saved outputs.

A baseline for the baseline

What would no model do? Holding 10% of comments at random catches about 10% of the toxic ones: recall 0.10. Both versions should be far above that.

A first look on dev

Each version’s threshold is the score above which it holds 10% of the dev comments: 0.254 for v1 and 0.291 for v2. Then Lena compares the versions on dev, with the paired bootstrap of Uncertainty and slices. It uses 2,000 resamples of the comments, seed 2026, and the same resample for both versions.

thresholds from dev: v1 0.253792 | v2 0.291032

recall (dev)        v1     v2     diff    95% interval
all              0.534  0.549  +0.015  [-0.007, +0.036]

ROC AUC (dev)
all              0.894  0.905  +0.011  [+0.006, +0.016]

What this tells us. Both versions are far above the random 0.10. On dev, v2 catches a few more toxic comments, but the interval of the difference includes 0. Its ROC AUC is higher, with an interval above 0. In the slices on dev, the LGBTQ benign-held rate rose from 0.148 to 0.193, with an interval from +0.004 to +0.086. That is a warning sign, but not a blocker by the rule. Lena writes the dev results in the run log and changes no rule. Dev is for looking; the decision is the holdout’s.

The holdout, once

Before you read on, predict: on the 30,000 holdout comments, will v2’s recall be better, with an interval above 0? Will every slice improve?

hold = holdout.merge(scores[["id", "v1", "v2"]], on="id")
result = compare(hold, hold["v1"].to_numpy(), hold["v2"].to_numpy(), t1, t2)
for m, title in (("recall", "recall: toxic comments held"), ("benign_held", "benign held: harmless comments held")):
    print(f"\n{title}\n{'':16s} v1     v2     diff    95% interval")
    for key in ["all", *GROUPS]:
        print(row(key, result[key], m))

The output:

recall: toxic comments held
                 v1     v2     diff    95% interval
all              0.512  0.532  +0.019  [+0.008, +0.031]
male             0.545  0.530  -0.015  [-0.045, +0.014]
female           0.486  0.523  +0.037  [+0.007, +0.065]
LGBTQ            0.506  0.569  +0.063  [+0.027, +0.101]
christian        0.411  0.436  +0.024  [-0.006, +0.054]
muslim           0.593  0.566  -0.027  [-0.060, +0.005]
other_religions  0.474  0.465  -0.009  [-0.072, +0.059]
black            0.676  0.650  -0.026  [-0.059, +0.006]
white            0.694  0.634  -0.060  [-0.089, -0.033]

benign held: harmless comments held
                 v1     v2     diff    95% interval
all              0.048  0.044  -0.003  [-0.006, -0.002]
male             0.080  0.073  -0.008  [-0.015, -0.001]
female           0.064  0.062  -0.002  [-0.008, +0.005]
LGBTQ            0.160  0.196  +0.036  [+0.015, +0.057]
christian        0.041  0.043  +0.001  [-0.004, +0.007]
muslim           0.172  0.149  -0.023  [-0.039, -0.007]
other_religions  0.080  0.075  -0.004  [-0.021, +0.012]
black            0.296  0.233  -0.063  [-0.086, -0.040]
white            0.279  0.219  -0.060  [-0.077, -0.042]

v2 − v1 on the 30,000 holdout comments, overall and per slice, with paired 95% bootstrap intervals (2,000 resamples, seed 2026). Top: recall. Bottom: the benign-held rate; the B2 line is at +0.01. Only the LGBTQ interval lies wholly above it.

Text description of the chart

Two panels, one row per slice, with the number of holdout comments in brackets and a dashed line at no difference.

Top, recall, v2 minus v1. All comments: +0.019, from +0.008 to +0.031. Female: +0.037. LGBTQ: +0.063, from +0.027 to +0.101. White: −0.060, from −0.089 to −0.033. The other slices have intervals that cross 0.

Bottom, the benign-held rate, v2 minus v1, with a second line at +0.01 for rule B2. All comments: −0.003. Black: −0.063 and white: −0.060, both with intervals well below 0. Muslim: −0.023. Male: −0.008. LGBTQ: +0.036, from +0.015 to +0.057, the only interval wholly above +0.01. The others cross 0.

What this tells us. Overall, v2 is better on both measures. It holds 1.9 points more of the toxic comments, and the interval is above 0. It also holds fewer harmless comments overall, and clearly fewer in the Black, white, Muslim and male slices. That is real progress.

One slice goes the other way. In the LGBTQ slice, v2 holds 19.6% of the harmless comments, against 16.0% for v1. The paired interval runs from +0.015 to +0.057: wholly above the B2 line. And in the white slice, v2 catches fewer toxic comments, −0.060. No rule covers that; it is reported.

Apply the rules, in order

The order matters. Blockers come first, because an average cannot pay for them.

B1, the identity statements. Each version scores the 64 harmless sentences:

held by v1: 11 | held by v2: 6 of 64

held by v2 only (B1):
                 text       v1       v2
I am a Muslim person. 0.248539 0.506179

held by v1 only:
                                          text       v1       v2
  My neighbour is gay, and she grows tomatoes. 0.365081 0.164753
   Gay people come to this park every weekend. 0.513222 0.215809
Muslim people come to this park every weekend. 0.261671 0.112739
My neighbour is Black, and she grows tomatoes. 0.255615 0.175698
 Black people come to this park every weekend. 0.515198 0.222521
 White people come to this park every weekend. 0.350157 0.113373

On this set, v2 is better as a whole: it holds 6 sentences, v1 holds 11. But v2 holds “I am a Muslim person.”, which v1 shows, just below its threshold of 0.254. The rule says one sentence is enough. B1 fires.

Is that rule too strict, now that v2 frees six other sentences? Perhaps. But the team wrote it before any result: a person who states their religion should never be held for it. A rule that you want to change after the result is a question for the next release, not a reason to ship this one.

B2, the slices. In code, B2 is one line. It is the second task in the notebook:

b2_slices = [g for g in GROUPS if result[g]["interval"]["benign_held"]["difference"][0] > 0.01]

It gives ['LGBTQ']. B2 fires.

The gates and the limit. The notebook applies every rule:

B1 identity statements   BLOCKER ['I am a Muslim person.']
B2 slice regression      BLOCKER ['LGBTQ']
G1 no overall loss       pass
G2 a measurable gain     pass
O1 scoring time          pass

Decision: DO NOT SHIP

The rules in their fixed order, with the holdout results. Two blockers stop the release before the gains are read.

Text description of the diagram

The rules, read from top to bottom on the 30,000 holdout comments. B1, identity statements: “I am a Muslim person.” is held by v2 only. v1 scores it 0.249, below its threshold of 0.254; v2 scores it 0.506, above its threshold of 0.291. Blocker. B2, slice regression: in the LGBTQ slice, the benign-held rate goes from 0.160 to 0.196. The difference is +0.036, with the interval +0.015 to +0.057, all above +0.010. Blocker. G1, no overall loss: recall goes from 0.512 to 0.532, +0.019, with the interval +0.008 to +0.031. Pass. G2, a measurable gain: the lower end, +0.008, is above 0. Pass. O1, scoring time: 250 milliseconds per 1,000 comments, against a limit of 1,000. Pass. Decision: do not ship v2. Two blockers, although its overall recall is better.

Read the cases behind the slice

An interval says how sure you are about a difference. It does not say which comments changed. So Lena opens the LGBTQ slice. Of its 730 harmless holdout comments, v1 and v2 both hold 98:

harmless LGBTQ comments: 730 | held by both: 98 | newly held by v2: 45 | released by v2: 19
newly held with 30% or more raters saying toxic: 14
gay          newly held  16 | released  17
lesbian      newly held   5 | released   0
homosexual   newly held  22 | released   2
transgender  newly held   8 | released   0

What this tells us. v2 newly holds 45 harmless comments of this slice and releases 19. The new holds lean on a few words. “Homosexual” appears in 22 of the 45, and in only 2 of the released ones; “transgender” and “lesbian” appear only in new holds. The character pieces of these words carry some of the toxicity that v2 learned from other comments. For “gay”, the two versions hold and release about the same number.

Not every new hold is a clear mistake: in 14 of the 45, at least 30% of the raters said “toxic”. In the other 31, few or no raters did. Here is one; one of its six raters said toxic:

“Since Jesus didn’t mention homosexuality, I would guess he would focus on what mattered - the poor, the vulnerable, the outcast.”

v1 gives it 0.138 and shows it. v2 gives it 0.295, just above its threshold, and holds it. This is the kind of comment that the site wants to publish at once.

The baseline is not fair either

A blocker stops a release that makes things worse. It does not say that the old version is good. Lena compares v1’s benign-held rate across the groups, with the comments that mention no group as a reference:

The share of harmless holdout comments that v1, the version on the site today, holds for a moderator, by the group they mention, with 95% bootstrap intervals. Comments that mention no group: a dot without an interval.

Text description of the chart

One row per group, with the number of holdout comments in brackets. No group: 0.021. Christian: 0.041. Female: 0.064. Male: 0.080. Other religions: 0.080. LGBTQ: 0.160. Muslim: 0.172. White: 0.279. Black: 0.296. The last four are far to the right of the others.

What this tells us. v1 holds 2.1% of the harmless comments that mention no group. It holds 9.3% of the harmless comments that mention at least one, and 29.6% of those that mention Black people. Someone who writes about their own identity waits for a moderator far more often. The model learned that these words often appear in toxic comments, and the raters’ labels may share some of that view.

This is the effect that the data’s authors set out to measure. The honest summary: “v2 is better than v1 for most groups and worse for one, and both hold harmless comments about identity far too often.”

The decision

Lena writes to Nadia in four parts, as in Trade-offs and regressions:

Decision: do not ship v2. Keep v1 on the site for now.

1. Blockers. v2 holds the harmless sentence “I am a Muslim person.”, which v1 shows (rule B1). On the holdout, v2 holds more harmless comments that mention LGBTQ people: 19.6% against 16.0%. The difference is +0.036, 95% interval +0.015 to +0.057 (rule B2). Of its 45 new holds in that slice, 22 contain “homosexual”.
2. Evidence. v2 is better overall: recall 0.532 against 0.512, +0.019 (interval +0.008 to +0.031), on 30,000 holdout comments. It also holds fewer harmless comments overall and for Black, white, Muslim and male writers.
3. Operations. v2 scores 1,000 comments in about 250 milliseconds, against about 30 for v1. That is within the limit.
4. Next step. Keep v2’s gains and fix its regression. Train the next candidate with more labelled comments that mention LGBTQ people and are harmless. Then run the same rules. Add disguised insults such as “1d1ot” to the test sentences, since that was v2’s purpose.

A prediction on new comments

The last step of every model is a new input. The course wrote six new comments, and both trained versions score them:

                                                     text    v1  held_v1    v2  held_v2
       Thanks for the clear report on the new bike lanes. 0.009    False 0.004    False
   As a gay man, I think the council made the right call. 0.165    False 0.159    False
My wife and I are Muslim, and we loved the food festival. 0.126    False 0.091    False
   The council should publish the budget before the vote. 0.030    False 0.012    False
  You are an idiot and nobody wants to read your garbage. 0.998     True 1.000     True
                                  What an 1d1ot. Go away. 0.124    False 0.033    False

What this tells us. Both versions show the four harmless comments and hold the plain insult. Neither holds “What an 1d1ot. Go away.”, although disguised words were the reason for v2. One example proves nothing. But it shows that the team never tested v2’s own promise; the next evaluation should.

Common mistake

The overall AUC went up, ship it

Symptom: Kwame shows one number: ROC AUC 0.887 → 0.902 on the holdout, with the interval +0.012 to +0.017. “It is not noise. Ship it.” Two weeks later, readers write in: their comments about same-sex marriage wait for hours.

Cause: the number is right and answers the wrong question. It averages over 30,000 comments; the LGBTQ slice is 999 of them. A gain on most comments hides a loss on a few, and nobody read the slices or the blockers.

Fix: decide in the fixed order: blockers on every rule, then the slices with paired intervals, then the overall gain. Write the rules before the results. Report the AUC on the scorecard as evidence, never as the gate.

Insights and limits

• Better on average is not better for everyone. v2 improves recall overall and for most groups, and it is worse for one group of writers. Only the slice view shows it.
• Rules written first protect the decision. The dev look showed the LGBTQ warning. Without fixed rules, the team could have argued it away, or tuned a rule until v2 passed.
• The operating point is part of the result. In the notebook’s “change one thing” task, a 5% queue moves both thresholds up. There, the LGBTQ difference shrinks to +0.004 (interval −0.009 to +0.018), and B2 does not fire. A result holds for the operating point at which it was measured.
• The labels are a few people’s judgement. The comparison is fair, but an absolute recall of about 0.5 partly reflects label noise, not only the models.
• The sample is not a real comment stream. WILDS keeps only comments with identity labels, and earlier models chose some of them for their identity content. In this holdout, 40% of comments mention a group. On a real site, the overall rates would differ.
• Only 8 groups. Disabled people, for example, are not a slice here, and “other religions” merges many faiths. The check cannot see what it does not measure.
• One run, English comments from 2015 to 2017. Language changes, and a model trained on these years meets new words.

Responsible use

This case study handles labels about religion, race, gender and sexuality, and comments that attack people for them. Some care is part of the method:

• Identity labels are for checking, never for deciding. They are not model inputs, and no comment gets a different threshold because of the group it mentions. On a real site, new comments have no such labels. You get them from a labelled evaluation set, as here.
• Hold, do not delete. Both versions only send comments to a person. A model that deletes comments or bans writers needs a much stricter bar, and a person must still decide.
• Protect the people who are discussed, and the people who discuss. A toxicity filter that holds “I am a Black person.” silences the people it is meant to protect. Measure the benign-held rate by group, not only recall.
• Do not read the slice table as facts about groups. “Comments that mention Black people are toxic 31% of the time” describes this sample and these raters. It says nothing about any group of people.
• Name the limits of the labels. Crowd raters labelled every comment, mostly four per comment, each with their own view. Credit them, and do not treat their majority as the truth.
• Show the data with care. Do not publish toxic examples in a portfolio or a slide. Counts and harmless examples are enough.

Make it a portfolio piece

This case study shows an employer that you can test a model release the way a careful team does. Make it yours with one change, for example your own identity sentences or a ninth slice. Then publish it, and make it easy to read in five minutes.

What to put in the repository:

• README.md, with the template below.
• release_rules.json, committed before your results, so the history shows the order.
• The notebook, run from top to bottom, with its outputs.
• The saved scores with their run IDs, and RUNS.md: what you ran, when, and what it showed.
• DECISION.md: the four-part decision, as above.
• The data source and licence. CC0 needs no credit, but credit the comment writers, the crowd raters and the dataset’s authors anyway. Link to the data and give the checksum, rather than copying 17 MB.

A README template:

# Regression test of a comment classifier, by slice

## Question
Can v2 (words + character n-grams) replace v1 (words) on a review queue that holds 10% of comments?

## Data and licence
Civil Comments with identity labels, WILDS version (Borkan et al. 2019; Koh et al. 2021), CC0.
Fixed sample: 60,000 train, 10,000 dev, 30,000 holdout. Labels by crowd raters (median 4 per comment).

## Method
Rules and blockers written first (release_rules.json). Thresholds from dev. Holdout read once.
Recall and benign-held rate, overall and for 8 identity slices, paired bootstrap 95% intervals
(2,000 resamples, seed 2026). Run IDs v1-13f555 and v2-8b2aeb.

## Result
Do not ship. v2 is better overall (recall +0.019 [+0.008, +0.031]) and fails two blockers:
a harmless identity sentence newly held, and more harmless LGBTQ comments held
(+0.036 [+0.015, +0.057]).

## Limits
Noisy labels, a sample rich in identity content, 8 groups only, one operating point.

How to describe it in an interview, in two sentences: “I regression-tested a new version of a toxicity classifier against the one in production, with blockers and slice rules written before any result. It was better on average but held more harmless comments from LGBTQ writers, so I advised in writing not to ship it.”

Go further:

• Train a v3 that fixes the regression, for example with more harmless LGBTQ comments in training. Use dev to develop it, and say how you would keep the holdout honest for a second decision.
• Repeat the decision at three queue sizes, 5%, 10% and 15%, and write how the operating point changes it.
• Compare with the judge case study, Calibrate a model judge against human ratings. There too, the labels are people’s judgements.

Knowledge check

Check your understanding

v2 holds 6 of the 64 identity sentences and v1 holds 11, yet rule B1 stops v2. Is that a bad rule? What should the team do about it?

Show the answer

B1 asks a narrow question: does the candidate hold any harmless identity sentence that the baseline shows? v2 does: “I am a Muslim person.” So B1 fires, although v2 is better on the set as a whole. The rule may well be too strict. A team can discuss a change, for example to “no more sentences held than the baseline”. But it must change the rule for the next release, in writing, before that release’s results. Changing a rule after you have seen the result that it blocks is the same as having no rule. For this release, the team keeps B1; B2 would stop v2 anyway.

Check your understanding

On dev, the LGBTQ benign-held rate rose by +0.045 with the interval +0.004 to +0.086, and B2 did not fire. On the holdout, the rise was +0.036 with +0.015 to +0.057, and B2 fired. Explain why, and why the team did not act on the dev result.

Show the answer

B2 fires only when the whole interval is above +0.01. On dev, the slice has 243 harmless comments, so the interval is wide, and its lower end, +0.004, is below the line. On the holdout, the slice has 730 harmless comments, so the interval is narrower, and its lower end, +0.015, is above the line. The point estimate was even a little larger on dev, but the evidence was weaker. The team changed nothing after the dev look. Dev is for looking, and the holdout is for the decision. It wrote the warning in the run log and waited.

Check your understanding

Nadia asks: 'v2 holds fewer harmless comments overall and for Black, white, Muslim and male writers. Is it not fairer than v1, then?' What do you answer?

Show the answer

v2 is better for most groups, and that is a real gain: the intervals for those slices are below 0. But it is worse for one group. It holds more harmless comments about LGBTQ people, 19.6% against 16.0%, and the interval is above the agreed line. A release that helps most writers and harms one group is not “fairer”; it moves the harm. Also, both versions hold harmless comments that mention a group far too often. v1 holds 9.3% of them, against 2.1% for comments with no group. The right answer is a v3 that keeps v2’s gains without the LGBTQ regression, and a plan to reduce the gap for every group.

Recap

• You wrote the rules before any model existed: two blockers, two gates, a limit and a fixed operating point.
• You trained two versions with one change, saved their scores with run IDs, and set each threshold on dev.
• On the frozen holdout, v2 was better overall: recall +0.019, interval +0.008 to +0.031.
• Two blockers stopped it. It newly held a harmless sentence, “I am a Muslim person.”, and it held more harmless LGBTQ comments: +0.036, interval +0.015 to +0.057.
• The cases behind the slice pointed to a few words. The baseline itself holds harmless comments about groups far more often than others.
• You wrote a decision that the result decided: do not ship v2, and what the next candidate must fix.

Next, the final assignment: build the whole harness for the support assistant, from the evaluation set to the release decision.

Next: Final assignment