When a simple model is better

What you will be able to do

Choose a model that people can read and check when the decision needs it, and use a complex model only when it earns its place. You will see where laws and standards ask for reasons, and test the trade-off on Larkfield’s data.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has the Larkfield part of this lesson, with checks. It compares the points score with two trained models, explains one ticket three ways, and tests small data. Its setup cells download the data and the course’s ticket_model.py, so it needs no local setup. The real-world cases and their sources are on this page.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 60 minutes

What you need to know

• A ticket’s contributions in logistic regression, and why they are not causes, from Explain predictions.
• What gradient boosting is, and what a complex model costs, from Ensembles: The cost of complexity.
• The one-standard-error rule, from Tune hyperparameters: The one-standard-error rule.
• The points score, a checklist that gives each ticket points, from Classification metrics and The precision–recall trade-off.
• The capacity threshold (flag at most 20% of tickets), from Thresholds and calibration.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment and the data in data/. You made them in Define a baseline: Set up your project. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows). If a data file is missing or changed, run python get_data.py --reset.
• ticket_model.py in its final form, with 11 features. You wrote it in Preprocess with a pipeline: Write ticket_model.py and added prior_escalations_90d in Select useful features: Change ticket_model.py. If it is missing or broken, download the course’s copy: curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py (curl.exe on Windows).
• A notebook in the project folder, in VS Code or JupyterLab, as in Define a baseline: Open a notebook in VS Code.

Tested on macOS with Python 3.14.6, pandas 3.0.6 and scikit-learn 1.9.1. The Windows commands are not tested. The notebook was tested in Jupyter.

“Why was I refused?”

Sara applies online for a small loan to repair her car. After ten seconds, the screen says: “Your application was not approved.” She calls the bank and asks the obvious question: “Why?”

Imagine two banks. At the first bank, the agent reads from a card: “Our system gives points. You had 41 points, and we need 50. You lost the most points for three late payments in the last year, and for a card balance close to its limit.” Sara now knows what happened. She can check it. If the late payments belong to another person with her name, she can ask for a correction. If they are hers, she knows what to change before she applies again.

At the second bank, the agent says: “A model with 400 decision trees gave you a score of 0.27. I cannot tell you more.” Nobody at the bank can say which facts about Sara mattered. Sara cannot check anything, and she cannot correct an error.

Both banks may be equally accurate. But only the first bank can answer “why?”. This lesson is about that difference: when it matters, what the law says about it, and what it costs.

Simple, readable and complex models

In this lesson, a simple model is one that a person can read completely and use to explain any single decision. You have met three:

• a points score: a checklist that adds fixed points for each fact (Sara’s bank card, or the ticket points score of Classification metrics);
• logistic regression: a weighted sum, where each ticket’s score splits exactly into contributions, as in Explain predictions;
• a small decision tree: a few yes/no questions that you can draw on one page.

A complex model, such as gradient boosting with 100 trees or a random forest, adds up many small parts. Each part is simple, but together they are too many to read. You can still study a complex model with extra tools, such as permutation importance or “what if?” questions. These tools describe the model from outside. They are useful, but they are not the model itself, and they can disagree with it.

People call the first kind interpretable models, and the extra tools explanation methods. The difference matters most when someone outside the team must trust a decision. That can be the person it affects, the people who use it every day, or a reviewer.

A points score on six tickets

Before the law, look again at how a points score works. You met Larkfield’s points score in Classification metrics. It gives each ticket points for three facts:

• priority 1: +2 points;
• team payment or warranty: +1 point;
• at least one escalation of the same customer in the last 90 days: +1 point.

Here are six made-up tickets. Add the points by hand before you read the output.

def points(df):
    """The points score: a checklist that a person can add up by hand."""
    return (2 * (df["priority"] == 1)
            + df["team"].isin(["payment", "warranty"]).astype(int)
            + (df["prior_escalations_90d"] >= 1).astype(int))


tiny = pd.DataFrame({"priority": [1, 1, 2, 3, 2, 3],
                     "team": ["payment", "delivery", "warranty", "returns", "payment", "account"],
                     "prior_escalations_90d": [1, 0, 2, 0, 0, 1]}, index=list("ABCDEF"))
tiny["points"] = points(tiny)
tiny

The output:

   priority      team  prior_escalations_90d  points
A         1   payment                      1       4
B         1  delivery                      0       2
C         2  warranty                      2       2
D         3   returns                      0       0
E         2   payment                      0       1
F         3   account                      1       1

Ticket A gets 2 + 1 + 1 = 4 points. Ticket C is not urgent, but it is a warranty ticket from a customer with earlier escalations: 0 + 1 + 1 = 2 points. Every point has a reason that you can say in one sentence. If Omar asks “why does ticket A have 4 points?”, the answer is the complete model, not a summary of it.

The rules are not guesses. As Classification metrics showed, each one comes from the train data. Priority 1 tickets escalate at 29.1%, against 8.9% for the others, so priority 1 gets 2 points. Payment and warranty tickets escalate at 18.5% against 8.9%. Tickets from customers with an earlier escalation escalate at 14.8% against 9.1%. So each of those rules gets 1 point. A real points score must then be tested on new data, as Priya does below.

Where a simple model is required or wiser

Many organisations make decisions about people: who gets a loan, a job interview, an insurance price, or a doctor’s attention first. In these areas, laws, supervisors and professional standards ask for reasons, for human review, or for models that a reviewer can check. The five cases below are real. Each one says what the rule or standard says, and where to read it.

This is not legal advice

This section describes what each source says, so that you understand why teams choose readable models. It is not legal advice, and it does not tell you whether a system complies. Laws and guidance change: three of the sources below changed between 2025 and 2026. For a real system, ask a qualified lawyer or compliance team.

Credit decisions in the United States

Sara’s case is the main example. In the United States, the Equal Credit Opportunity Act and its Regulation B apply when a lender refuses credit. Section 1002.9 says that the applicant must get a statement of reasons. The statement must be “specific and indicate the principal reason(s) for the adverse action”. It also says that one statement is not enough: that the applicant “failed to achieve a qualifying score on the creditor’s credit scoring system”. The official interpretation of Regulation B adds that the reasons must “relate to and accurately describe the factors actually considered or scored” by the lender.

So a lender must be able to name, for each refused applicant, the main facts that led to the refusal. With a points-based scorecard, those facts are the lines where the applicant lost the most points. This is one reason why many lenders use scorecards and logistic regression. In 2022, the Consumer Financial Protection Bureau (CFPB) published a circular that said a complex algorithm does not remove this duty. The CFPB withdrew that circular in May 2025, together with many other guidance documents. The withdrawal removed the guidance; the text of section 1002.9 that we quote here did not change.

Automated decisions in the European Union

The General Data Protection Regulation (GDPR) gives people in the EU rights about automated decisions. Article 22 gives a person “the right not to be subject to a decision based solely on automated processing”. This applies when the decision has legal effects or “similarly significantly affects” the person. There are three exceptions: the decision is necessary for a contract, a law allows it, or the person gives explicit consent. For a contract or consent, the organisation must offer at least three rights: to get human intervention, to express a point of view, and to contest the decision. Articles 13, 14 and 15 add a right to “meaningful information about the logic involved” in such decisions.

In December 2023, the EU Court of Justice applied Article 22 to credit scores. In the SCHUFA case (C-634/21), it held that a credit agency’s score is an automated decision when a bank attributes to it “a determining role” in granting credit. On 19 November 2025, the European Commission proposed changes to the GDPR, including Article 22. When we checked, that proposal was not yet law.

The EU AI Act (Regulation (EU) 2024/1689) adds rules for AI systems that it calls high-risk. Its Annex III lists several kinds. Among them are systems that evaluate a person’s creditworthiness or credit score, and systems for risk assessment and pricing in life and health insurance. Systems used to recruit or select people for jobs are also on the list. For high-risk systems, Article 13 asks for transparency for deployers, the organisations that use the system. They must be able to “interpret a system’s output and use it appropriately”. Article 14 asks for effective human oversight. Article 86 gives an affected person a right to explanations. They must be “clear and meaningful explanations of the role of the AI system” and of “the main elements of the decision taken”. In July 2026, the EU moved the date. The requirements for these listed high-risk systems, including Articles 13 and 14, now apply from 2 December 2027.

Model risk management in banks

Banks use models for loans, capital and fraud, and a wrong model can lose a lot of money. US bank supervisors publish guidance on model risk management: how a bank develops, checks and uses its models. For fifteen years, the main text was SR 11-7 (2011). On 17 April 2026, the Federal Reserve, the OCC and the FDIC replaced it with new guidance, SR 26-2. The new guidance says that a model’s inherent risk “increases with model complexity”. It says that the rigor of testing should match the model’s complexity and importance. It also names “interpretability measures” as one way to check that a model is conceptually sound.

The guidance is not a law, and it does not ban complex models. But its message for a model builder is clear. Every extra piece of complexity is extra work for the people who test, validate and monitor the model. A model that a validator can read is cheaper and safer to approve.

Early warning scores in hospitals

In many hospitals in England, nurses compute a National Early Warning Score (NEWS2) for each adult patient, several times a day. The Royal College of Physicians publishes it, and NHS England endorsed it as the early warning system for acutely ill patients in hospitals. The nurse measures six things, such as the breathing rate, the oxygen level and the pulse, and gives each one 0 to 3 points. A patient who needs extra oxygen gets 2 more points. A total of 5 or 6 is the key threshold for an urgent response. A total of 7 or more needs an urgent or emergency response by a team with critical-care skills.

No law forces a points score here. It is wiser. A nurse can compute NEWS2 at the bedside, with no computer, in a minute. Every nurse and doctor can see why the score is high, check each measurement, and act on the same rule. That is the kind of trust that a model needs when lives depend on a quick, shared decision.

Insurance prices with generalised linear models

Car and home insurers set their prices with models, and in the United States, state regulators review the rate filings. The National Association of Insurance Commissioners (NAIC) wrote a white paper on how regulators should review these models (adopted in 2020 and 2021). It says that the generalised linear model (GLM) “is a commonly used predictive model in insurance applications”. In statistics, GLMs are the family that includes linear and logistic regression. (The paper itself lists logistic regression among the more complex methods, so its words about GLMs are about rating plans.) The paper says that GLMs “are relatively transparent”, and that more complex methods make the regulators’ oversight “even more challenging”. It asks regulators to “thoroughly review all aspects of the model”, including its variables.

An insurer that files a GLM can show the regulator each factor and its effect on the price. An insurer that files a complex model must explain much more, and the review takes longer.

What these rules have in common

None of these sources bans complex models. They ask for three things: reasons for a single decision, a way for a person to review or contest it, and models that a checker can understand. A simple model makes each of these duties easier and more reliable, because the explanation is the model itself. The table sums up the five cases.

Area	Source	What it asks for	Why a simple model helps
Credit, United States	Regulation B, section 1002.9	Specific principal reasons for a refusal	Each refusal’s reasons are the points or contributions that it lost
Automated decisions, EU	GDPR Articles 22, 13–15	Human intervention, the right to contest, meaningful information about the logic	A person can review and explain the decision
High-risk AI, EU	AI Act Annex III; Articles 13, 14, 86	Interpretable output, human oversight, explanations of the main elements	The main elements are visible in the model
Banks, United States	SR 26-2 (replaced SR 11-7 in 2026)	Testing and validation that grow with complexity	Less to validate, easier to challenge
Hospitals, England	NEWS2	A score that every nurse computes and acts on	Computed by hand, checked at the bedside
Insurance prices, United States	NAIC white paper	Regulators review all aspects of the model	GLM factors can be read one by one

Two kinds of work follow from these sources, and they are different. The first kind is a control: something that an engineer can build and show. Examples are the reason list for each decision, a record of every score, a model card, or a step where a person reviews a refusal. The second kind is a conclusion, such as “this system complies with Article 22” or “these reasons meet Regulation B”. A conclusion needs a qualified review by lawyers or compliance experts, for the real system and the real country. As a model builder, you deliver the controls and the evidence. You do not write the conclusion.

Reasons that have nothing to do with law

Larkfield’s escalation model decides nothing about a customer’s money or rights: it only puts a ticket in a faster queue. No rule above applies to it. Priya still chose logistic regression in Tune hyperparameters. The other reasons for a simple model apply to almost every team.

Small data

A flexible model has many parts to learn, such as hundreds of splits in its trees. With few examples, it learns noise as if it were a pattern. That is overfitting, the student who memorises the practice exercises. A simple model has few parts, so it needs fewer examples.

Priya tests it. She trains both models on random samples of the train set, from 250 to 8,000 tickets, five samples at each size. Then she measures average precision (AP) on May.

• Logistic regression
• Boosting, tuned (depth 2)
• Boosting, default settings
• Points score (no training)

Validation average precision when each model learns from a random sample of the train set (mean of 5 samples; the last point uses all 16,939 tickets). The points score needs no training, so it is a flat line at 0.31.

Text description of the chart

A line chart. The horizontal axis is the number of train tickets, which doubles at each step: 250 to 8,000, then all 16,939. The vertical axis is average precision on May. Logistic regression: 0.315, 0.345, 0.366, 0.381, 0.385, 0.387, 0.391. Tuned boosting: 0.273, 0.312, 0.333, 0.369, 0.384, 0.397, 0.395. Boosting with default settings: 0.256, 0.275, 0.304, 0.320, 0.342, 0.357, 0.398. The points score is flat at 0.310. Logistic regression is best up to 2,000 tickets; from 4,000 tickets the lines meet.

With 250 to 2,000 tickets, logistic regression is clearly better. With 250 tickets, boosting with its default settings (0.256) is worse than the points score (0.310). The points score learned nothing from those tickets at all. Only from about 4,000 tickets do the lines meet. Many real projects start small: a new product, a rare event, a small shop.

Speed, size and cost

The final logistic regression has 28 coefficients and one intercept; saved with joblib, it is about 5 KB. The tuned boosting model has 100 trees with 400 leaves, about 74 KB. Both are small and fast on Larkfield’s data. But the difference grows with the data. A large boosting model or a deep network can need special hardware, and every prediction costs time and money. A points score needs a pencil.

Debugging and monitoring

When a simple model goes wrong, you can find out why. In Slice analysis, warranty tickets escalated more after the policy change of 1 May. With logistic regression, one coefficient (team_warranty, +0.28) shows how the model treats warranty, and you can compare it with the new rate. With 100 trees, the warranty effect is spread over many splits, mixed with other features. Watching a model after launch is the same: a change in one coefficient or one rule is easy to see and to explain.

Stability over time

A simple model changes little when you retrain it on new data, because each of its few parts is learned from many examples. A flexible model can change many splits after a small change in the data, as a single deep tree did in Ensembles. For the people who use the scores, a model that behaves the same next month is easier to trust.

Trust and adoption

A model is useful only if people act on it. Grace’s senior agents will look at the flagged tickets. Suppose a flag comes with “priority 1, payment team, an earlier escalation”. Then the agent sees at once why the ticket is there, and can disagree when it looks wrong. A flag that says only “0.41” invites the agent to ignore it. Omar adds: “If the agents ignore the flags, the model’s recall on paper means nothing.”

Handover

In Module 8, Priya hands the model to another team. A simple model is easier to hand over. Its model card can list every coefficient, and a new person can understand it in an hour. Fewer things can break when a package changes.

Larkfield: three models at the same capacity

Now the evidence. Priya compares three models on May, under the same rule as in Thresholds and calibration. She flags with the lowest threshold that keeps the flags at or under 20% of the tickets. A points score has only whole numbers, so its thresholds are 0 to 5.

First, she checks that the points score ranks May’s tickets. The points function is the one from the six tickets above.

valid["points"] = points(valid)
valid.groupby("points")[TARGET].agg(["size", "sum", "mean"]).round(3)

The output:

        size  sum   mean
points
0        320   17  0.053
1        532   63  0.118
2        211   50  0.237
3         84   35  0.417
4         32   18  0.562

The escalation rate (mean) goes up with every point: from 5.3% with no points to 56.2% with 4 points. A three-line checklist already ranks the tickets.

Then she fits logistic regression (the final model) and gradient boosting with the best setting of the tuning search: learning rate 0.1, depth 2. Before you read the output, predict: which model catches the most escalations at the capacity, and by how many?

from sklearn.ensemble import HistGradientBoostingClassifier

logistic = build_pipeline().fit(train[FEATURES], train[TARGET])
boosting = build_pipeline(HistGradientBoostingClassifier(max_depth=2, learning_rate=0.1, random_state=0))
boosting.fit(train[FEATURES], train[TARGET])
scores = {"points": valid["points"],
          "logistic": logistic.predict_proba(valid[FEATURES])[:, 1],
          "boosting": boosting.predict_proba(valid[FEATURES])[:, 1]}

def at_capacity(name, score, steps, capacity=0.20):
    """Flag with the lowest threshold that flags at most `capacity`, and print the result."""
    threshold = next(t for t in steps if (score >= t).mean() <= capacity)
    flags = score >= threshold
    caught = int((flags & (valid[TARGET] == 1)).sum())
    print(f"{name:9s} threshold {threshold:<5} flagged {flags.mean():.1%}  caught {caught}"
          f"  precision {caught / flags.sum():.3f}  recall {caught / valid[TARGET].sum():.3f}")


at_capacity("points", scores["points"], [0, 1, 2, 3, 4, 5])
steps = [s / 100 for s in range(1, 100)]
at_capacity("logistic", scores["logistic"], steps)
at_capacity("boosting", scores["boosting"], steps)

The output:

points    threshold 3     flagged 9.8%  caught 53  precision 0.457  recall 0.290
logistic  threshold 0.19  flagged 19.3%  caught 86  precision 0.379  recall 0.470
boosting  threshold 0.18  flagged 19.1%  caught 86  precision 0.382  recall 0.470

Read it in three steps.

1. Logistic regression and boosting catch the same 86 escalations of 183. Boosting’s precision is 0.003 higher, because it flags two tickets fewer. On the decision that matters to Grace, the two models tie.
2. The points score flags only 9.8%. Tickets with 3 or more points are 116; with 2 or more points they are 327, which is 27.7% and over the capacity. A score with only five values cannot stop at 20%. That is a real cost of a coarse checklist.
3. A tie rule fixes most of it. Suppose Grace’s team reviews all tickets with 3 or 4 points, then the 2-point tickets in order of arrival while there is room. Then the points score flags 235 tickets (19.9%) and catches 84, a recall of 0.459. The notebook runs this rule. That is two escalations fewer than the trained models, from a checklist that a new agent learns in a minute.

Over all thresholds, average precision tells the same story: points 0.310, logistic regression 0.391, boosting 0.395. The gap between the last two is 0.004. In Tune hyperparameters, cross-validation gave a gap of 0.0031 (0.3424 for boosting, 0.3393 for logistic regression). One standard error was 0.008. The gap is smaller than the noise, so it is not a difference.

Recall of each way to flag on the validation month under the capacity rule. Logistic regression and tuned boosting catch the same 86 of 183 escalations. The points score catches 53 at its cut-off of 3 points (9.8% flagged), or 84 with the tie rule.

Text description of the chart

A column chart of recall on May with at most 20% of the tickets flagged. The rule “flag priority 1”: 0.33 (61 caught, 13.8% flagged). Points score, 3 or more points (“pts 3+”): 0.29 (53 caught, 9.8% flagged). Points score with the tie rule (“pts + tie”): 0.46 (84 caught, 19.9% flagged). Logistic regression: 0.47 (86 caught, 19.3% flagged). Tuned boosting: 0.47 (86 caught, 19.1% flagged).

Tomás is not convinced: “Boosting has the higher AP. Higher is better.” Priya answers with the question from Ensembles: “Is the gain larger than the noise, and is it worth the cost?” Here the gain is 0.004 in AP and zero escalations at the capacity. The cost is a model that nobody can read.

One ticket, three explanations

Omar asks about T-117014, a priority 1 payment ticket from the phone channel that escalated: “Why was it flagged?” Priya shows what each model can say.

def point_reasons(ticket):
    """The points of one ticket, line by line."""
    row = ticket.iloc[0]
    return {"priority 1 (+2)": 2 * int(row["priority"] == 1),
            "payment or warranty (+1)": int(row["team"] in ("payment", "warranty")),
            "an escalation in the last 90 days (+1)": int(row["prior_escalations_90d"] >= 1)}


ticket = valid[valid["ticket_id"] == "T-117014"]
print(point_reasons(ticket))
print("logistic score:", round(logistic.predict_proba(ticket[FEATURES])[0, 1], 3))
print("boosting score:", round(boosting.predict_proba(ticket[FEATURES])[0, 1], 3),
      "from", boosting.named_steps["model"].n_iter_, "trees")

The output:

{'priority 1 (+2)': 2, 'payment or warranty (+1)': 1, 'an escalation in the last 90 days (+1)': 1}
logistic score: 0.38
boosting score: 0.409 from 100 trees

• The points score gives the complete reason: 2 points for priority 1, 1 for the payment team, 1 for the earlier escalation. Four points, so the ticket is flagged. Nothing is hidden, and Omar can check each line in the ticket.
• Logistic regression gives exact contributions, as in Explain predictions. Priority 1 adds +1.23 to the sum, and the payment team +0.38. The phone channel adds +0.23, and the home segment takes 0.71 away. With the intercept, they add up exactly to the sum that gives the score of 0.380. They compare with an average ticket, and they describe the model, not the cause.
• Boosting gives 0.409 from 100 trees. scikit-learn has no exact way to split that number into reasons. Priya can ask “what if?” questions, one at a time. With priority 3 instead of 1, boosting gives 0.095, and logistic regression gives 0.092. Each answer is useful. But the answers do not add up to the score, and Omar must know which questions to ask.

Other libraries can estimate a contribution for each feature of a boosting model. Those estimates are a second model of the first one. They can be good, but a reviewer must now trust two things instead of one.

When a complex model is worth it

A simple model is not always better. Tomás is right in several situations, and a good data scientist knows them:

• The inputs are images, text or sound. A photo has millions of pixel values, and no single pixel means anything. No points score can read a photo or a sentence. Neural networks, taught in the next course, Deep Learning and Transformers Explained, are the right tool here.
• The data is large and the pattern has strong interactions. Think of millions of rows, and a pattern such as “this feature matters only when that one is high”. A complex model can find what a weighted sum cannot. Larkfield’s pattern is mostly additive: each fact adds or removes some risk. That is why boosting only ties here.
• The gain is clearly larger than the noise, and it matters. For example, a fraud model that catches many more frauds, measured on validation data, with a difference of several standard errors.
• Errors are cheap or reviewed by a person, and nobody needs reasons. Film recommendations, the order of search results, or a draft that a person checks before it is sent.

Even then, keep the simple model as a benchmark: train it first, and report it next to the complex model. If the complex model is only a little better, you know the price of its complexity. If it is much better, look at why: often it found an interaction that you can add to the simple model.

A decision checklist

The questions of this lesson fit in one checklist. Use it before you choose a final model, and write the answers in the model card.

A checklist for choosing between a simple and a complex model. Start simple; a complex model must pass every question to replace it.

Text description of the diagram

1. Start with the simplest model that could meet the target, such as a points score or logistic regression.
2. Must a person get reasons, or must a reviewer check the model? If yes, prefer a model whose reasons are exact, and ask for qualified legal or compliance review.
3. Are the inputs images, text or sound? If yes, a complex model is usually needed.
4. Does a complex model win by more than one standard error? If no, keep the simple model.
5. Is the gain worth the cost: training and prediction time, size, monitoring, explaining and handover? If no, keep the simple model. If yes, use the complex model, write down why, and keep the simple one as a benchmark.

Guided practice: Priya’s model-choice note

Priya must write two to five sentences in Larkfield’s model card that say which model the team keeps and why. Do the same, with the evidence of this lesson.

1. Run the notebook to section 3, so that scores and at_capacity() exist.
2. Run the capacity comparison. You should see 86 caught for both logistic regression and boosting.
3. Compute the cross-validation gap from Tune hyperparameters: 0.3424 − 0.3393. You should see 0.0031.
4. Compare the gap with one standard error, 0.0080. You should see that the gap is smaller.
5. Answer each question of the checklist for Larkfield, in one line each.
6. Write the note: the model you keep, the evidence, and what the complex model would cost.

Show an example answer

The checklist for Larkfield: no customer gets a decision about money or rights. But Grace’s agents must see why a ticket is in their queue (question 1: reasons help; no legal duty found). The inputs are table columns, not images or text (question 2: no). Boosting wins by 0.0031 in cross-validation AP, less than one standard error of 0.008. It catches the same 86 escalations at the capacity (question 3: no). So the checklist stops: keep the simple model.

Priya’s note: “We keep logistic regression with the default C = 1.0. Tuned gradient boosting scored 0.0031 higher in cross-validation average precision, less than one standard error (0.008). Both models catch 86 of 183 escalations on May at the 20% capacity. Logistic regression gives each flagged ticket an exact list of contributions that agents can read. Its coefficients are easy to monitor after the warranty policy change. We keep the points score (recall 0.459 with the tie rule) as a fallback that works without a computer.”

Your turn: a smaller shop

Change one condition: the amount of data. Imagine a smaller shop with only 500 labelled tickets. In the notebook, section 6, train logistic regression and the same boosting model on train.sample(500, random_state=0), and measure AP on May. Before you run it, predict: which model loses more?

Show the answer

Logistic regression gets 0.336, and boosting 0.304. With all 16,939 tickets, they were 0.391 and 0.395. So on 500 tickets, logistic regression loses 0.055 and boosting loses 0.091, and the order changes. One sample can be lucky or unlucky. The chart in Small data averages five samples at each size and shows the same pattern: 0.345 against 0.312 at 500 tickets.

The lesson: when data is small, a simple model is not only easier to explain. It is usually more accurate too, because it has fewer parts to learn from few examples.

Failure case: the same reasons for everyone

Common mistake

Global importance used as the reasons for one ticket

Symptom: Omar wants each customer whose ticket is flagged to get a short note: “Your request went to a senior agent because of …”. Tomás suggests: “Keep boosting. Use its three most important features as the reasons.” He computes permutation importance on May and prints the reasons for two tickets:

from sklearn.inspection import permutation_importance

result = permutation_importance(boosting, valid[FEATURES], valid[TARGET], scoring="average_precision",
                                n_repeats=5, random_state=0)
top3 = pd.Series(result.importances_mean, index=FEATURES).nlargest(3).index.tolist()
for ticket_id in ["T-117014", "T-117033"]:
    row = valid[valid["ticket_id"] == ticket_id]
    score = boosting.predict_proba(row[FEATURES])[0, 1]
    print(ticket_id, "score", round(score, 3), "| reasons:", ", ".join(top3))

T-117014 score 0.409 | reasons: priority, team, word_count
T-117033 score 0.032 | reasons: priority, team, word_count

T-117014 is a priority 1 payment ticket, flagged. T-117033 is a priority 3 returns ticket, not flagged. They get the same three “reasons”.

Cause: permutation importance describes the whole model: “the model depends on priority”. It does not say whether this ticket’s priority pushed its score up or down, or by how much. So one ticket hears “priority” as the reason it was flagged, and the other as the reason it was not. These are not reasons for a single decision. Regulation B’s warning against the reason “failed to achieve a qualifying score” points at the same problem. A reason must say something about this applicant.

Fix: take the reasons of a decision from that decision. With the points score, they are the lines that gave the ticket its points. With logistic regression, they are the largest contributions of that ticket, with their sign, as in Explain predictions. If the team must use a complex model, use a method that gives each ticket its own contributions. Test that the reasons match the model. Then ask a qualified reviewer whether they meet any rule that applies.

Consequence in AI work: a reason that is the same for everyone looks like an explanation and is not one. A customer cannot check it or correct it. In credit, hiring or insurance, a team that sends such reasons may believe that it meets its duty when it does not. Only a qualified review can decide that, but the model builder can avoid the mistake.

See it end to endPredict credit-card defaultCase study

The credit case study keeps logistic regression because the bank must give each client exact reasons.

See it end to endChoose a model for hospital readmissionsCase study

The readmission case study chooses logistic regression over boosting for a pilot, and explains why.

See it end to endSegment online-shop customersCase study

The customer-segments case study finds that a simple RFM score is as good as the clusters.

Knowledge check

Check your understanding

Sara is refused a loan in the United States. The bank's notice says: you did not reach the qualifying score of our credit scoring system. What does Regulation B, section 1002.9, say about such a statement, and how does a scorecard help the bank?

Show the answer

Section 1002.9 says that the statement of reasons must be specific and indicate the principal reasons for the refusal. It also says that “failed to achieve a qualifying score” is insufficient. Its official interpretation adds that the reasons must relate to the factors that the lender actually considered or scored. A points-based scorecard helps because the principal reasons are visible in the model. They are the lines where Sara lost the most points, for example late payments. This describes the rule; whether a real notice complies is a question for a qualified reviewer, and rules can change.

Check your understanding

Predict: Larkfield had only 500 labelled tickets. Would you expect tuned boosting or logistic regression to give the higher validation average precision? Use the numbers of this lesson.

Show the answer

Logistic regression. With 500 train tickets (mean of five samples), logistic regression reaches 0.345 and tuned boosting 0.312. With one sample (random_state=0), they get 0.336 and 0.304. A flexible model has more parts to learn, so with few examples it learns more noise. The two meet at about 4,000 tickets (0.385 and 0.384). With all 16,939 tickets, they tie (0.391 and 0.395).

Check your understanding

Tomás says: boosting has a higher average precision on May, 0.395 against 0.391, so we should ship boosting. Give two pieces of evidence and two costs that support keeping logistic regression.

Show the answer

Evidence: at the 20% capacity, both models catch the same 86 of 183 escalations (recall 0.470). In cross-validation on train, the gap is 0.0031, smaller than one standard error (0.008). By the one-standard-error rule, it is not a difference. Costs of boosting: it has 100 trees and 400 leaves instead of 28 coefficients. So it is harder to read, debug and monitor, for example after the warranty policy change. And it cannot give each flagged ticket an exact list of reasons, which Grace’s agents use to trust the flags. It is also bigger (about 74 KB against 5 KB) and harder to hand over.

Sources

Laws and guidance change; read the current version before you rely on any of them.

• US Regulation B, 12 CFR 1002.9, Notifications, with its official interpretation (comments 9(b)(2)-2 and 9(b)(2)-4): consumerfinance.gov.
• CFPB, Interpretive Rules, Policy Statements, and Advisory Opinions; Withdrawal, 90 FR 20084 (12 May 2025), which lists Circular 2022-03 on complex algorithms: govinfo.gov.
• GDPR, Regulation (EU) 2016/679, Articles 13, 14, 15 and 22: eur-lex.europa.eu.
• Court of Justice of the EU, press release 186/23 on Case C-634/21, SCHUFA Holding (Scoring), 7 December 2023: curia.europa.eu.
• EU AI Act, Regulation (EU) 2024/1689, Articles 13, 14, 86 and Annex III: eur-lex.europa.eu. The new dates in the Digital Omnibus on AI, Regulation (EU) 2026/1744 of 8 July 2026: eur-lex.europa.eu, and its history on the European Parliament Legislative Train.
• Federal Reserve, SR 26-2, Revised Guidance on Model Risk Management, 17 April 2026, which replaces SR 11-7: the letter and the guidance, which has the quoted text.
• Royal College of Physicians, National Early Warning Score (NEWS) 2: rcp.ac.uk.
• NAIC, Regulatory Review of Predictive Models white paper: content.naic.org.

Recap

• A simple model (a points score, logistic regression, a small tree) can be read completely, so it can explain any single decision. A complex model needs extra tools, which describe it from outside.
• Rules ask for reasons and review, not for a model type. US Regulation B asks for specific principal reasons for a credit refusal. The GDPR gives rights about automated decisions. The EU AI Act lists credit scoring, life and health insurance pricing and hiring as high-risk. SR 26-2 ties validation work to complexity. NEWS2 and insurance GLMs show readable scores in daily use. This is not legal advice, and the rules change.
• Other reasons: small data (with 500 tickets, logistic regression 0.345 against boosting 0.312), speed and size, debugging and monitoring, stability, trust, handover.
• At Larkfield, logistic regression and tuned boosting both catch 86 of 183 escalations at the capacity. Their gap in cross-validation is 0.0031, less than one standard error. A three-line points score catches 84 with a tie rule.
• Choose a complex model for images, text and sound, or for large data with strong interactions. Also choose it when the gain is clearly larger than the noise and worth the cost. Keep the simple model as a benchmark.

Next, you leave the target behind: you look for groups of customers with k-means, and ask Grace if the groups mean anything. Then you finish Module 7 with the module practice.

Next: Unsupervised exploration