Unsupervised exploration

What you will be able to do

Look for groups in the data when there is no target to predict. A clustering method always returns groups, even when the data has none. This lesson shows how to make the groups fair (scale first), how to look at them (PCA) and how to choose their number honestly. It also shows why a person who knows the business must say what they mean.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every cell, task and check of this lesson and of the module practice, with the instructions. Its setup cells install the packages, download the data and the course’s ticket_model.py, so it needs no local setup. Read the explanations and the chart here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 70 minutes

What you need to know

• Unsupervised learning and clustering, from Unsupervised learning.
• Scaling and the Pipeline, from Preprocess with a pipeline.
• Why one unit can decide a distance, from Similarity and distance in Essential Mathematics and Statistics for AI.
• The four kinds of error, slice_table() and the counts next to rates, from Slice analysis.
• Contributions and their limits, from Explain predictions.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project folder ~/projects/ticket-model with its .venv environment and the data in data/. You made them in Define a baseline: Set up your project. matplotlib is in requirements.txt from that lesson. If python -c "import sklearn" fails, activate .venv again (source .venv/bin/activate on macOS and Linux, .venv\Scripts\Activate.ps1 on Windows). If a data file is missing or changed, run python get_data.py --reset.
• ticket_model.py in its final form, with 11 features. You wrote it in Preprocess with a pipeline: Write ticket_model.py and added prior_escalations_90d in Select useful features: Change ticket_model.py. If it is missing or broken, follow Reset to a known state at the end of this lesson.
• A notebook in the project folder, in VS Code or JupyterLab, as in Define a baseline: Open a notebook in VS Code.

Tested on macOS with Python 3.14.6, pandas 3.0.6, scikit-learn 1.9.1 and matplotlib 3.11.2. The Windows commands are not tested. The notebook was tested in Jupyter.

Groups without a target

In Unsupervised learning, you sorted a box of mixed buttons with no labels: nobody told you the groups, so you put similar things together. That is unsupervised learning: there is no target, and the method looks for structure in the features alone. Clustering is the main kind. It puts similar rows into groups, called clusters, so that rows in one cluster are closer to each other than to rows in other clusters.

The first three lessons of this module used the target, escalated_72h, everywhere. This lesson leaves it behind. Grace has a planning question, not a prediction question: “Do our customers fall into a few kinds, so that I can plan support for each kind?” There is no column that says what kind a customer is. So there is nothing to predict, and no right answer to compare with. A person must judge whether the groups are useful.

Priya clusters customers, not tickets. Grace can act on a group of customers (for example, give them one contact person), but not on a group of tickets.

She builds one row per customer from the train tickets only. Validation and test stay unseen, as for the model. Guests have no customer ID, so they are left out.

from ticket_model import FEATURES, TARGET, build_pipeline, load

train = load("train")
known = train[train["customer_id"].notna()]
customers = known.groupby("customer_id").agg(
    tickets=("ticket_id", "size"), order_value=("order_value", "median"),
    word_count=("word_count", "mean"), tenure_days=("customer_tenure_days", "max"))
print(len(customers), "customers;", train["customer_id"].isna().sum(), "guest tickets left out")
customers.describe().round(1)

The output:

847 customers; 202 guest tickets left out
       tickets  order_value  word_count  tenure_days
count    847.0        837.0       847.0        847.0
mean      19.8         80.9       104.0       1272.9
std       28.4         42.7        27.9        741.1
min        1.0         10.4         0.0          2.0
25%        4.0         52.1        90.2        623.5
50%       10.0         72.2       102.9       1300.0
75%       23.0        104.1       115.3       1905.0
max      262.0        448.4       314.7       2607.0

Column	What it means for one customer
tickets	Their tickets from July 2024 to April 2026 (the train period).
order_value	The median order value of their tickets. 10 customers have only account tickets, so it is missing.
word_count	The mean length of their first messages.
tenure_days	How long they were a customer at their last train ticket.

Look at the std row. tenure_days has an SD of 741.1 days; tickets has 28.4. Keep that in mind for the next section.

k-means on six customers

Imagine that you must place 3 ice-cream stands on a long beach, so that people walk as little as possible. A simple method works well. Put the stands anywhere. Each person goes to the nearest stand. Then move each stand to the middle of the people who came to it. Repeat until no stand moves.

k-means is that method, for rows of data. It is the most common clustering method. You choose k, the number of clusters (the number of stands). Then it repeats two steps:

1. Put each row in the cluster of its nearest centre (a point that stands for the cluster).
2. Move each centre to the mean of the rows in its cluster.

It stops when no row changes its cluster. “Nearest” means the straight-line distance, which adds up the squared differences of all columns. So the columns must be comparable.

Here are six made-up customers with two columns. A and E are small customers with few tickets. C is a big customer.

tiny = pd.DataFrame({"tickets": [2, 3, 40, 45, 4, 50],
                     "tenure_days": [300, 2400, 350, 2300, 1300, 1250]}, index=list("ABCDEF"))


def distance(df, a, b):
    """The straight-line distance between two rows."""
    return round(float(((df.loc[a] - df.loc[b]) ** 2).sum() ** 0.5), 2)


scaled_tiny = (tiny - tiny.mean()) / tiny.std(ddof=0)
print("as recorded: A to C", distance(tiny, "A", "C"), "| A to E", distance(tiny, "A", "E"))
print("scaled:      A to C", distance(scaled_tiny, "A", "C"), "| A to E", distance(scaled_tiny, "A", "E"))

The output:

as recorded: A to C 62.8 | A to E 1000.0
scaled:      A to C 1.79 | A to E 1.21

Check the first line by hand. A to C: √(38² + 50²) = √(1,444 + 2,500) = 62.8. A to E: √(2² + 1,000²) = 1,000.0. As recorded, a difference of 1,000 days hides a difference of 38 tickets, and A is “closer” to the big customer C. Scaled, each column counts in standard deviations ((value − mean) ÷ SD), and A is closer to E, the other small customer. This is the problem from Similarity and distance in the mathematics course, now inside a clustering method.

Run k-means with k = 2 on both versions:

from sklearn.cluster import KMeans

for name, data in [("as recorded", tiny), ("scaled", scaled_tiny)]:
    labels = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(data)
    groups = [list(tiny.index[labels == label]) for label in sorted(set(labels), key=list(labels).index)]
    print(name, groups)

The output:

as recorded [['A', 'C', 'E', 'F'], ['B', 'D']]
scaled [['A', 'B', 'E'], ['C', 'D', 'F']]

As recorded, the clusters are “shorter tenure” and “longer tenure”: only tenure_days counts. Scaled, they are “few tickets” (A, B, E) and “many tickets” (C, D, F). Both are valid clusters of numbers. Only the second uses both columns. Scale before you cluster.

Worked example: Priya clusters 847 customers

Priya puts three steps in one pipeline, as in Module 4. It fills the 10 missing order values with the median, scales, and runs k-means with k = 3. n_init=10 runs k-means from 10 different starting centres and keeps the best result. random_state=0 makes it the same every time. Before you look, predict: which column will separate the clusters most?

from sklearn.impute import SimpleImputer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

COLUMNS = ["tickets", "order_value", "word_count", "tenure_days"]
clusterer = make_pipeline(SimpleImputer(strategy="median"), StandardScaler(),
                          KMeans(n_clusters=3, n_init=10, random_state=0))
customers["cluster"] = clusterer.fit_predict(customers[COLUMNS])
profile = customers.groupby("cluster")[COLUMNS].mean().round(1)
profile["customers"] = customers["cluster"].value_counts()
profile

The output:

         tickets  order_value  word_count  tenure_days  customers
cluster
0           11.2         85.3       106.5        584.8        374
1           14.8         71.8       101.1       1858.2        407
2           98.5        111.8       107.2       1562.8         66

This table is the profile of each cluster: the mean of each column. The numbers 0, 1 and 2 are only names; k-means gives them in no particular order. Read it by rows:

• Cluster 0, 374 customers: about 11 tickets, a tenure of about 585 days (1.6 years).
• Cluster 1, 407 customers: about 15 tickets, a tenure of about 1,858 days (5 years).
• Cluster 2, 66 customers: about 99 tickets each, and larger orders (111.8).

word_count is almost the same in all three: it does not separate customers. Clustering did not use the target. Priya looks at it now, only to describe the clusters. She also counts the segments, a column the clustering did not use:

segment = known.groupby("customer_id")["segment"].first()
print(pd.crosstab(customers["cluster"], segment))
tickets = known.merge(customers["cluster"], left_on="customer_id", right_index=True)
print(tickets.groupby("cluster")[TARGET].agg(["size", "mean"]).round(3))

The output:

segment  business  home  trade
cluster
0              19   259     96
1              15   320     72
2              25     1     40
         size   mean
cluster
0        4196  0.102
1        6040  0.094
2        6501  0.148

Cluster 2 is almost all business and trade customers: 65 of its 66. Its 66 customers wrote 6,501 train tickets: 38.8% of all tickets from known customers. Those tickets escalate more (14.8% against about 10%). Clusters 0 and 1 have a similar mix of segments and a similar escalation rate (10.2% and 9.4%).

A picture in two dimensions: PCA

Hold a teapot in front of a lamp and look at its shadow on the wall. From one side, the shadow shows the spout and the handle, and you know it is a teapot. From above, it is only a circle. A good angle keeps most of the shape in two dimensions.

Four columns cannot be drawn on a flat page. PCA (principal component analysis) finds a good angle for the shadow of the data. It makes new axes, called principal components. Each one is a weighted mix of the scaled columns. The first component keeps as much of the spread of the data as possible; the second keeps as much of the rest as possible. Two components give a picture.

from sklearn.decomposition import PCA

scaled = clusterer[:-1].transform(customers[COLUMNS])  # the fitted imputer and scaler
pca = PCA(n_components=2).fit(scaled)
print("share of the spread kept:", pca.explained_variance_ratio_.round(3))
print(pd.DataFrame(pca.components_, index=["PC1", "PC2"], columns=COLUMNS).round(2))

The output:

share of the spread kept: [0.309 0.267]
     tickets  order_value  word_count  tenure_days
PC1     0.75         0.49        0.12         0.42
PC2     0.03        -0.67       -0.00         0.74

clusterer[:-1] is the pipeline without its last step: the imputer and the scaler, as fitted. The two components keep 30.9% + 26.7% = 57.5% of the spread. So the picture shows a little more than half of the differences between customers; the rest is in the two dimensions that are not drawn. Read the weights: PC1 grows with tickets, order value and tenure (“a bigger, older customer”). PC2 grows with tenure and falls with order value.

• Cluster 0: newer customers
• Cluster 1: long-standing customers
• Cluster 2: large accounts

400 of the 847 customers (a fixed random sample), drawn on the first two principal components and coloured by their k-means cluster (k = 3, scaled). The picture keeps 57.5% of the spread.

Text description of the chart

A scatter plot of 400 customers. The horizontal axis is the first principal component, which grows with tickets, order value and tenure; it runs from about −2 to 7. The vertical axis is the second principal component, which grows with tenure and falls with order value; it runs from about −7 to 2. Cluster 1, the long-standing customers (211 in the sample), forms a dense cloud at the upper left, around PC2 = 1. Cluster 0, the newer customers (151), forms a cloud just below it, around PC2 = −0.5. A few of its customers with large orders form a tail down to PC2 = −7. The two clouds touch: there is no gap between them. Cluster 2, the large accounts (38 in the sample), spreads to the right, from PC1 = 1.5 to 7. Its points are few, with much space between them.

What the picture shows:

• Clusters 0 and 1 touch. There is no empty space between them. k-means cut one cloud into two parts: shorter and longer tenure. It always cuts; that does not mean a natural border exists.
• Cluster 2 is spread out. The large accounts differ a lot from each other. They are one cluster mostly because they are far from everybody else.
• The colours come from k-means in four dimensions. PCA only draws them. A few points of one colour among another colour are normal: in the two hidden dimensions, those points are closer to their own cluster.

Choose k: hints, not answers

Priya chose k = 3. Two numbers can help to choose k, but neither gives the answer:

• Inertia is the sum of the squared distances from each row to its centre. Smaller means tighter clusters. It always falls when k grows: with one cluster per customer, it is 0.
• Silhouette compares, for each row, the mean distance to its own cluster with the mean distance to the nearest other cluster. The average goes from −1 to 1. Near 1 means clear, separate groups; near 0 means groups that overlap.

from sklearn.metrics import silhouette_score

for k in range(2, 9):
    km = KMeans(n_clusters=k, n_init=10, random_state=0).fit(scaled)
    print(f"k {k}: inertia {km.inertia_:.0f}, silhouette {silhouette_score(scaled, km.labels_):.3f}")

The output:

k 2: inertia 2720, silhouette 0.231
k 3: inertia 2181, silhouette 0.262
k 4: inertia 1821, silhouette 0.260
k 5: inertia 1504, silhouette 0.285
k 6: inertia 1360, silhouette 0.273
k 7: inertia 1245, silhouette 0.247
k 8: inertia 1134, silhouette 0.255

• Inertia falls by 539, 360, 317, 144, 115 and 111. People look for a bend (an “elbow”) where the fall slows down. Here it slows a little after k = 5, but there is no clear bend.
• Silhouette is highest at k = 5 (0.285), but k = 3, 4 and 6 are within 0.025. All values are below 0.3: the groups overlap.

The honest summary: these customers do not fall into clear, separate groups. k-means can still give useful groups, but the data does not decide k. The people who use the groups decide it. Priya keeps k = 3, because Grace can name and use three groups. Silhouette says that k = 5 is as good; you try it in the variation.

Grace names the clusters

A cluster is a set of numbers until someone who knows the business says what it means. Priya shows Grace the profile, the segments and the escalation rates. Grace names each cluster and says if it is useful.

Cluster	Grace’s name	Evidence	Useful?
0	Newer customers	374 customers, tenure about 1.6 years, 11 tickets	No, not for support. The escalation rate (10.2%) is the same as cluster 1’s.
1	Long-standing customers	407 customers, tenure about 5 years, 15 tickets	No, for the same reason. It is a split by tenure, which one line of pandas gives.
2	Large accounts	66 customers (65 trade or business), 98.5 tickets each, 38.8% of the known customers’ tickets, 14.8% escalate	Yes. 66 customers make almost 4 in 10 tickets. A named contact for each could solve problems before they become tickets.

Grace says: “Cluster 2 is real. I knew we had some big trade customers, but not that 66 of them write 39% of our tickets. Clusters 0 and 1 are only ‘new’ and ‘old’. They do not change what my team does, so I would not use them.”

Three lessons from this:

• k-means always returns k groups. It cannot say “there are no groups”. Cluster 0 against cluster 1 is a cut through one cloud.
• Only domain knowledge makes a cluster useful. The numbers said “cluster 2 has many tickets”. Grace turned it into a decision: a contact person for 66 customers.
• A cluster is not a new truth about the customers. Another k, other columns or other scaling give other clusters. Write down the choices with the result.

Is cluster 2 useful for the escalation model? Probably not: the model already has prior_tickets_90d and prior_escalations_90d, which carry most of the same information. The module practice checks it.

Guided practice: describe the clusters

1. Run the cells of the worked example, so that customers, profile and tickets exist.
2. Find the cluster with the highest mean tickets: big = profile["tickets"].idxmax(). You should see 2.
3. Compute the share of the known customers’ train tickets from that cluster: (tickets["cluster"] == big).mean(). You should see about 0.388.
4. Count the customers of each segment in each cluster with pd.crosstab. You should see 25 business and 40 trade customers in cluster 2.
5. Write a name for each cluster, and one sentence on whether Grace can use it.

Show an example answer

Cluster 2, “large accounts”, is 66 customers (7.8% of the 847), almost all trade or business. They write 38.8% of the known customers’ train tickets. Cluster 0 and cluster 1 are newer and long-standing customers. They escalate at about the same rate (10.2% and 9.4%), and they differ only in tenure. Your names can differ. Check that each name comes from the profile numbers, and that you say which clusters lead to no action.

Your turn: five clusters

Change one condition: k = 5, where silhouette was highest. Make the same pipeline with n_clusters=5, and make the profile. Before you run it, predict: does the large-accounts cluster survive? What do the new clusters split on?

Show the answer

   tickets  order_value  word_count  tenure_days
0     15.8         67.3        98.4       1931.1
1     16.8        145.1       105.3        994.2
2     10.3         60.6        89.6        592.7
3      7.0         65.8       154.4        936.1
4    100.7        110.2       106.9       1574.2

The large accounts survive almost unchanged: cluster 4 has 63 customers and 100.7 tickets each (it had 66 and 98.5). k-means cuts the other 784 customers in four instead of two:

• long tenure (cluster 0, 320 customers),
• large orders (cluster 1, 133 customers, an order value of 145.1),
• short tenure (cluster 2, 236 customers),
• long messages (cluster 3, 95 customers, 154.4 words).

A higher silhouette (0.285 against 0.262) did not create a more useful answer. Grace might find “large orders” or “long messages” interesting, but neither suggests a support action yet. A cluster that appears for every k, as the large accounts do, is more trustworthy than one that appears only for one k.

Failure case: no scaling

Common mistake

Clusters that are only one column

Symptom: Tomás leaves the scaler out of the pipeline. k-means runs without a warning and returns three clusters. Priya shows the smallest and largest values in each:

unscaled = make_pipeline(SimpleImputer(strategy="median"), KMeans(n_clusters=3, n_init=10, random_state=0))
raw = customers.assign(cluster=unscaled.fit_predict(customers[COLUMNS]))
raw.groupby("cluster")[["tickets", "tenure_days"]].agg(["min", "max"])

        tickets      tenure_days
            min  max         min     max
cluster
0             1  191         2.0   868.0
1             1  143       875.0  1737.0
2             1  262      1748.0  2607.0

The clusters are three bands of tenure_days that do not overlap: up to 868 days, 875 to 1,737, and from 1,748. Every cluster has customers with 1 ticket and customers with more than 140. The large accounts are gone.

Cause: k-means uses distances. tenure_days has an SD of 741 days and tickets an SD of 28, so a difference in tenure decides every distance. The other three columns hardly count. The result looks like clustering, but it is a sort by one column.

Fix: scale the columns first, in the same pipeline: make_pipeline(SimpleImputer(...), StandardScaler(), KMeans(...)). Then check the profile: if the clusters differ in only one column, ask whether that column dominates.

Consequence in AI work: a team presents “three customer types” that are only three age bands of the account, and plans services for them. Nothing in the output says that the scale was wrong. The same mistake affects every method that uses distances, such as nearest-neighbour search.

See it end to endSegment online-shop customersCase study

The customer-segments case study checks clusters against what customers did in the next six months.

See it end to endFind wholesale buyer typesCase study

The wholesale case study shows the silhouette trap and reads PCA axes in business words.

Knowledge check

Check your understanding

Why did Priya cluster customers and not tickets, and why only from the train tickets?

Show the answer

Customers, because Grace can act on a group of customers (for example, give them one contact person), but not on a group of single tickets. Train only, because the validation and test months must stay unseen, the same rule as for the model. The module practice then uses the clusters on May tickets, which is fair only if May did not shape the clusters.

Check your understanding

Silhouette is highest at k = 5 (0.285). Tomás says: so the data has five types of customer. What do you answer?

Show the answer

The silhouette values are all below 0.3, and k = 3 to 6 are within 0.025 of each other. So the groups overlap, and the data does not have a clear number of types. Silhouette and inertia are hints. k = 5 keeps the large accounts and cuts the rest into four overlapping parts. Choose k by what the groups let Grace do, and say in the report that the groups are not clearly separate.

Module practice

The module practice joins the lessons of this module: inspect the misclassified tickets, and check whether the clusters reveal a useful pattern. The completion check asks for at least one data problem and one model limitation, each with evidence.

Worked example: recall per cluster

Priya scores the validation month with the final model at 0.19, as in Slice analysis. She gives each ticket its customer’s cluster from the train clustering. Guests and customers who are new in May have no cluster: they get −1.

pipeline = build_pipeline().fit(train[FEATURES], train[TARGET])
valid = load("valid")
valid["score"] = pipeline.predict_proba(valid[FEATURES])[:, 1]
valid["flag"] = (valid["score"] >= 0.19).astype(int)
valid["cluster"] = valid["customer_id"].map(customers["cluster"]).fillna(-1).astype(int)
valid["caught"] = valid["flag"] * valid[TARGET]
by_cluster = valid.groupby("cluster").agg(tickets=(TARGET, "size"), escalated=(TARGET, "sum"),
                                          caught=("caught", "sum"))
by_cluster["recall"] = (by_cluster["caught"] / by_cluster["escalated"]).round(3)
by_cluster

The output:

         tickets  escalated  caught  recall
cluster
-1            24          2       1   0.500
 0           368         42      21   0.500
 1           382         62      24   0.387
 2           405         77      40   0.519

Priya reads it with the habits of this module:

• Counts first. Cluster −1 has 2 escalations: “too few to judge”. The others have 42 to 77.
• Is the difference real? The long-standing customers (cluster 1) have the lowest recall, 0.387, against 0.519 for the large accounts. The SE of each recall is about 0.06, so the SE of the difference is about 0.08. The difference, 0.13, is less than two SEs. It may be chance; Priya notes it as “watch”, not as a finding.
• Does the pattern help the model? The large accounts escalate more (77 of 405, 19.0%) and the model already catches more of them. It sees them through prior_tickets_90d and prior_escalations_90d. So the cluster adds little that the model does not have. It is useful to Grace for planning, not as a new feature.

Your task: inspect the missed escalations

Do this in the notebook, section 9.

1. Select the 55 escalated tickets that are not flagged and have a score below 0.10.
2. Count them by priority, team and cluster. Compare each count with the escalations of that slice.
3. Look at 10 of them, row by row. Look for anything that a senior agent would see but no column records.
4. Write your findings: at least one data problem and one model limitation, each with the numbers that show it. Use the evidence of the lessons of this module.

Show the model answer

The counts. By priority: 40 are priority 3 and 15 priority 2. By team: delivery 17, returns 17, warranty 10, account 7, payment 4. By cluster: 23 from long-standing customers (cluster 1), 19 from large accounts (cluster 2), 13 from newer customers (cluster 0). Compared with the escalations of each cluster (62, 77 and 42), that is 37%, 25% and 31%: no clear pattern by cluster.

The rows. Most look ordinary: priority 2 or 3, delivery or returns, an average message, often no escalation in the last 90 days. Some customers had earlier escalations (T-117033 had 3), which the model saw but outweighed with other columns. Nothing in the 11 columns separates these tickets from the many ordinary tickets that do not escalate.

Data problems (any one, with evidence):

• A label problem after the warranty policy. Warranty tickets escalate at 28.6% in May, but the model, trained before 1 May, gives them a mean score of 0.163; in train, warranty’s rate was 15.9%. The target changed meaning, and the training data does not show the new meaning.
• Unknown inputs. 22 May tickets have channel = unknown and 55 have region = unknown: 90 tickets have at least one unknown or missing input. They are too few to judge (5 escalations in the unknown channel), and they are not the main cause of the errors (14 of 238).
• Guests have no history. 9 May tickets come from guests: no customer ID, no tenure, no earlier tickets, and no cluster. The model cannot use the history features for them, and 9 tickets cannot show whether it works for them.

Model limitations (any one, with evidence):

• Missed escalations with a low score. 55 of the 97 missed escalations have a score below 0.10, so no threshold within the capacity catches them. The recall for priority 3 is 0.102 (6 of 59), with an SE of about 0.04: a reliable gap, not chance.
• Something that no column shows. The course data comes with a description, its data card (dataset.md in the labs repository). It says that about 1 in 4 tickets come from repeat customers with a hidden tendency to escalate. The rows of the missed tickets look ordinary in every column. The contributions of T-117033 show the model’s view of a ticket whose escalation no column predicts. A better model of the same columns cannot fix this; only new information can.
• The scores under-estimate after the policy change. The mean score in May is 0.120 against a rate of 0.155 (Thresholds and calibration). The ranking still works, but the scores must not be shown as “the chance”.

The clusters: one useful pattern for planning: 66 large accounts write 38.8% of the known customers’ tickets. No useful pattern for the escalation model: the recall differences by cluster are within the noise, and the model already has the history columns.

A complete answer names the problem and gives the numbers with their counts. It says where they come from (valid, not test), and what kind of problem it is: data, label or model.

Module check

Check your understanding

Retrieval: name the four kinds of error in the module's taxonomy, and the two questions to ask of every error.

Show the answer

The four outcomes of a decision that can be wrong or right: caught (flagged and escalated), false alarm (flagged, not escalated), missed near the threshold (escalated, score from 0.10 to under 0.19) and missed with a low score (escalated, score below 0.10). In May: 86, 141, 42 and 55, plus 855 correct passes. The two questions: is it a data problem (an input is unknown, missing or wrong)? Is it a label problem (the target is wrong or changed its meaning)?

Check your understanding

Predict and explain: Tomás adds a column in cents, order_value_cents (order_value × 100), to the customer table, and clusters again without scaling. What happens to the clusters? What happens with scaling?

Show the answer

Without scaling, order values in cents (a mean of about 8,000 and an SD of about 4,000) become the largest numbers. The clusters turn into bands of order value, and tenure, which decided before, now hardly counts. With scaling, order_value_cents becomes the same column as order_value in SDs. The result is close to clustering with order_value counted twice, so order value gets more weight than the other columns. Lesson: scale, and do not add a column twice in another unit.

Check your understanding

Completion check: from the evidence of this module, name at least one data problem and one model limitation of Larkfield's escalation model. Give the numbers, their counts, and the data set they come from.

Show the answer

A data problem (a label problem): after the warranty policy of 1 May, warranty tickets escalate at 28.6% in the validation month (48 of 168), but the model gives them a mean score of 0.163; in train, warranty’s rate was 15.9%. The target changed meaning after training. Other valid answers: 90 May tickets have an unknown channel or region, or a missing order value outside account; 9 guest tickets have no history.

A model limitation: 55 of the 97 missed escalations in May have a score below 0.10, so no threshold within the 20% capacity catches them. For priority 3, the model catches 6 of 59 escalations (recall 0.102, SE about 0.04). The missed tickets look ordinary in every column: the cause, such as a customer’s hidden tendency to escalate, is in no column.

All numbers come from the validation month. A complete answer gives each number with its count, and separates a data or label problem from a model limitation. It also says that no decision used a test slice.

Reset to a known state

If your project folder or notebook is in a strange state, go back to a known start. Do these steps in a terminal in ~/projects/ticket-model, with .venv active. Each step is safe to repeat.

1. Restore the data. Run python get_data.py --reset. The last line is Ready: train 16939 rows (1983 escalated), valid 1179 (183), test 1002 (157).
2. Restore ticket_model.py. Run curl -fsSL -O https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C05/project/ticket_model.py (curl.exe on Windows).
3. Check the model. Run python -c "import ticket_model as t; print(len(t.FEATURES))". You should see 11.
4. In your notebook, restart the kernel and run all cells from the top. In VS Code, select Restart, then Run All.

In Colab or Kaggle, there is nothing to restore on your computer. Delete the folder ticket-model in the file panel, then restart the session and run all cells. The setup cells download everything again. Your own notes in the notebook stay, as long as you saved a copy.

Module 7 recap

Recap

In this module you looked behind the score of the final model, using the validation month for every decision:

• Errors and slices: in May, the model catches 86 escalations, raises 141 false alarms and misses 97, of which 55 have a score below 0.10. Recall is 0.771 for payment, 0.083 for account and 0.102 for priority 3. Every rate goes with its count: a rate from 22 tickets has a 95% interval from 5% to 40%.
• Data and label problems: warranty escalates at 28.6% but has a mean score of 0.163, because the policy of 1 May changed the target. 90 tickets have an unknown or missing input; they explain only 14 of the 238 errors.
• Explanations: coefficients depend on the other features (prior_tickets_90d: +0.18 alone, −0.29 in the final model). Permutation importance ranks priority, prior escalations and team first. One ticket’s contributions add up exactly to its score. None of them is a reason or a cause.
• A simple model by choice: logistic regression and tuned boosting both catch 86 of 183 escalations at the capacity. So Larkfield keeps the model that people can read, check and hand over. Credit, hospital and insurance rules show why readable models matter when a person asks “why?”.
• Clusters: after scaling, k-means finds 66 large accounts with 38.8% of the known customers’ tickets, and a split by tenure that Grace cannot use. Inertia and silhouette (all below 0.3) are hints, not answers.

Next, in Module 8, you package the model and write its model card. It records the slices, the limits and the reason for a simple model that you found here. Then you plan the checks that keep it honest after launch.

Next: Package artifacts