Skip to content
Français

Normalise text

  • Lesson 11
  • 35 min
  • Module 4: Understand the query

Where we are

In Module 3, we built BM25 per field and a rule for product codes. BM25 still matches only the exact words that people type, and people type the same thing in many forms: plurals, capitals, accents and units. In this lesson, we normalise and stem queries and products in English and French, and we measure what each step fixes and what it breaks.

What you will be able to do

Make “Cushions”, “cushion” and “CUSHIONS” the same word for the search, in English and in French. Then measure what this fixes and what it breaks, query by query.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook writes a small normaliser with a stemmer and builds two BM25 indexes of the catalogue, plain and normalised. It asks you to measure both on the English validation queries. Then it shows which queries got worse. Its setup cells download the practice data, so it needs no local setup. About 35 minutes.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 35 minutes

What you need to know

What you need on your computer (only for the guided practice)

  • The project larkfield-search at the end of Module 3, with its virtual environment. search check must end with Setup check passed.
  • About 50 MB of disk space for two new packages: PyStemmer and scikit-learn.

A shop assistant who hears the word, not the spelling

Section titled “A shop assistant who hears the word, not the spelling”

A customer says to the shop assistant: “I need cushions for the garden.” Another customer writes “CUSHION” on a note. A third one says “a cushion, 18 inches”. The assistant does not care about the capital letters, the plural or how the size is written. She hears one thing: cushion.

A search engine does not hear. It compares strings of characters. For BM25, “cushions” and “cushion” are two different words, as different as “cushion” and “lamp”. So the same customer, with the same need, can get good results or bad results only because of an “s”.

This lesson teaches the search to hear the word, not the spelling. You change the text of the query and of the products into one standard form before you compare them.

Here is a real test query: “outdoor light fixtures” (an English WANDS query). Larkfield sells 110 outdoor wall lights. In their titles and descriptions, 23 say “fixture”, and only one says “fixtures”.

The run bm25-fields+codes from Module 3 compares exact words. “outdoor” and “light” are each in more than 3,000 products. “fixtures” is in 182 products, and only one of them is an outdoor wall light. So the top 10 is light bulbs: the first one is “40 watt equivalent e12 dimmable led edison candle light bulb”. Its NDCG@10 is 0.000.

The run bm25-norm, which you build in this lesson, turns “fixtures” into “fixtur”, and the products’ “fixture” into “fixtur” too. Now the word matches. Its first five results are outdoor wall lights, all graded exact, and NDCG@10 is 1.000.

One letter made the difference between a useless page and a perfect page. And “fixtures” is not a rare case. Plurals, capitals and units are in every query log.

To normalise text means to write it in one standard form, so that small differences in writing disappear. The word comes from “norm”, a standard. You apply the same steps to the query and to every product. If you change only one side, the two sides stop matching.

The project does it in five steps. The diagram follows one product title through them.

Five normalising steps for the text Grey Outdoor Cushions 18''. Lower case: grey outdoor cushions 18''. Remove accents (café becomes cafe): no change here. Write units one way: grey outdoor cushions 18 inch. Split into tokens: grey, outdoor, cushions, 18, inch. Stem each token: grey, outdoor, cushion, 18, inch. The result is the same set of tokens as a product with 18 inch outdoor cushion.
One text through the five steps of the project's normaliser. The query and every product go through the same steps.
Text description of the diagram

A row of boxes joined by arrows. “Typed: Grey Outdoor Cushions 18’‘” leads to “1. Lower case: grey outdoor cushions 18’’”, then “2. Remove accents (café becomes cafe)”, then “3. Write units one way: grey outdoor cushions 18 inch”, then “4. Split into tokens: grey · outdoor · cushions · 18 · inch”, then “5. Stem each token: grey · outdoor · cushion · 18 · inch”. The last box, with a thick border, says: “The same tokens as a product with 18 inch outdoor cushion”.

Lower case is the easy step: “Grey” becomes “grey”. The plain BM25 of Module 3 already did this.

Accents are harder than they look. French writes “café”, “extérieur” and “canapé”. People often type them without accents, because a keyboard makes accents slow. So the project removes accents on both sides: “café” becomes “cafe”. It does this with a standard Unicode method. Each accented letter is split into its base letter and the accent, and then the accent is dropped.

Removing accents has a cost in French. “pêche” (peach, fishing) and “péché” (sin) both become “peche”. For a shop of home and garden products, this rarely matters. For a search of legal texts, it could.

People write the same size in many ways: 18'', 18", 18 in, 18 inch and 18 inches. The product titles write 18 ''. The project rewrites every form as 18 inch, and every form of feet as ft. It also puts spaces in sizes such as 24x18, so they become 24 x 18.

This step is a list of rules that we wrote, one rule per pattern. Rules like these fix what you planned for, and nothing else. You see below that one of them breaks a real query.

A token is one word that the index stores, as in The inverted index. The project splits the text on every character that is not a letter or a digit. So “indoor/outdoor” becomes two tokens: “indoor” and “outdoor”.

The steps so far do not join “cushions” and “cushion”. For that, you need stemming: cutting a word down to its stem, the part that its forms share. “cushions”, “cushioned” and “cushion” all become “cushion”. The stem does not have to be a real word: “fixtures” and “fixture” both become “fixtur”.

A stemmer uses fixed rules about word endings, written for one language. The project uses the Snowball stemmers through the Python package PyStemmer. Snowball has rule sets for English, French and many other languages. Here is what the English stemmer does to some words from Larkfield’s queries. These are real outputs of the project’s function norm:

Typed Plain token Stem Joined with
cushions cushions cushion cushion
fixtures fixtures fixtur fixture
tables tables tabl table
lighting lighting light light, lights
living room living, room live, room live (a live plant)
oliver oliver oliv olive (a colour)

The first three rows are what you want. The last three show the risk. A stemmer does not know meanings. “Lighting” (a kind of product) and “light” (also a colour: “light blue”) get the same stem. “Oliver”, a name, gets the same stem as “olive”, a colour.

Lemmatisation is the careful version of stemming. It turns each word into its dictionary form, its lemma: “tables” becomes “table”, and “mice” becomes “mouse”. To do that, it needs a dictionary of the language. It often needs the role of the word in the sentence too. In “the lighting is soft”, “lighting” is a noun. In “she is lighting a candle”, it is a verb.

Lemmatisation makes fewer bad joins than stemming. But it is slower, and it needs a language model or a large dictionary for each language. Search queries are short and often have no grammar (“grey cushion 18”), so the role of a word is hard to guess. Most search engines use stemming for these reasons, and so does Larkfield’s project.

French: its own stemmer, and machine-translated titles

Section titled “French: its own stemmer, and machine-translated titles”

English rules do not fit French words. So the project has a second analyser, norm_fr, with the French Snowball stemmer: “Coussins gris extérieurs” becomes “coussin”, “gris”, “exterieur”.

But Larkfield’s product titles are in English. A French query matches nothing in an English title, however well you stem it. So the run bm25-norm also searches a French title field. Of the 40,066 products, 7,597 have a French title. These titles are machine-translated: a language model wrote them from the English titles.

Priya runs bm25-norm on the 243 test queries and compares it with bm25-fields+codes from Module 3. To separate the two changes, she also runs it without the French titles.

NDCG@10 with its 95% interval on Larkfield's test queries, in three groups. Stemming helps the English queries; the French titles help the French queries and cost the English queries a little.
Text description of the chart

Three panels with three rows each. English WANDS queries (120): exact words 0.641, with normalising and stemming 0.716, with French titles too 0.709. Misspelt queries (40): 0.362, 0.404, 0.416. French queries (60): 0.178, 0.206, 0.309. Each value has its 95% interval as a line.

Read the chart one group at a time.

  • English WANDS queries. Normalising and stemming take NDCG@10 from 0.641 to 0.716. The French titles then take it back a little, to 0.709. Probably some English words also appear in French titles, so a few wrong products move up.
  • Misspelt queries. These are queries with one wrong letter, a practice problem we made on purpose. Normalising helps them a little, because their other words now match better. It cannot fix the misspelt word itself. That is the next lesson.
  • French queries. Stemming alone helps a little. The French titles help most: from 0.206 to 0.309. But 0.309 is still far below the English queries. Only 7,597 products have a French title, and a French query cannot match the other products.

On all test queries, bm25-norm reaches NDCG@10 0.583 (95% interval 0.535 to 0.627), against 0.506 for bm25-fields+codes. The paired difference is +0.077 (95% interval +0.05 to +0.103). It is clear: the interval does not include 0. Without the French titles, the run reaches 0.559.

The same comparison says something else too. Normalising made 111 test queries better and 41 worse. The mean hides the 41. You must look at them before you keep a change.

The worst one is “oliver parsons”, a real WANDS query. Oliver is a name, and a Parsons chair is a style of chair. With exact words, the top 10 has eight Parsons chairs, graded partial or not judged. With stemming, “oliver” becomes “oliv”, the stem of “olive”. Now the first result is “soucy upholstered side chair in olive beige”, and the third is “olive branch wall décor”. NDCG@10 falls from 0.502 to 0.000.

The second one is “blk 18x18 seat cushions”. The unit rule turns “18x18” into “18 x 18”. So the query now has “18” twice, and BM25 adds the score of each query word, so “18” counts twice. Every product with “18 inch” in its title now scores high. The first result is an 18-inch storage ottoman, not a seat cushion. NDCG@10 falls from 0.433 to 0.155.

Both failures come from the same cause: a rule that knows letters, not meanings. You will meet this cause again in every lesson of this module.

Guided practice: add the module’s files and measure

Section titled “Guided practice: add the module’s files and measure”

This module adds five Python files to the project. You copy all of them now, from the snapshot end-of-m04. This lesson explains normalise.py. The next three lessons explain the other files, one per lesson.

  1. In larkfield-search, with the virtual environment active, copy the files of this module from the snapshot. They are the new code, its tests, the new list of packages and the README.

    Terminal window
    S=../nextia-learning-labs/C38/snapshots/end-of-m04
    for f in requirements.txt larkfield_search/normalise.py larkfield_search/spell.py larkfield_search/synonyms.py larkfield_search/detect.py larkfield_search/runs/understand.py tests/test_understand.py README.md; do
    cp "$S/$f" "$f"
    done
  2. Install the two new packages. The new requirements.txt adds PyStemmer (the stemmers) and scikit-learn (a classifier for the last lesson of this module):

    Terminal window
    python -m pip install -r requirements.txt
  3. Run the tests:

    Terminal window
    python -m pytest -q
    ........................ [100%]
    24 passed in 12.20s

    Module 3 ended with 19 tests. The five new ones test this module’s code. The first run takes some seconds, because Python loads scikit-learn for the first time. Your time will differ.

  4. Try the normaliser by hand. Start Python with python, and type the lines after >>>:

    >>> from larkfield_search.normalise import fold, norm, norm_fr
    >>> fold("Café table, 24 inches")
    'cafe table, 24 inch '
    >>> norm("Grey Outdoor Cushions 18''")
    ['grey', 'outdoor', 'cushion', '18', 'inch']
    >>> norm_fr("Coussins gris extérieurs")
    ['coussin', 'gris', 'exterieur']
    >>> norm("lighting"), norm("living room")
    (['light'], ['live', 'room'])

    fold does steps 1 to 3 (lower case, accents, units). norm adds the tokens and the English stems; norm_fr uses the French stemmer. Type exit() to leave Python.

  5. Measure the new run on the test queries:

    Terminal window
    search eval --run bm25-norm
    bm25-norm on 243 test queries: NDCG@10 0.583 (95% interval 0.535 to 0.627)
    recall@100 0.518, MRR 0.581, zero results 1.2%, judged in the top 10 74.0%; 0.2 ms a query (p50), 0.4 ms (p95)

    The first run builds BM25 indexes of the normalised fields. It takes a few seconds, and later runs reuse them.

  6. Compare it with Module 3’s run, by kind of query:

    Terminal window
    search compare bm25-fields+codes bm25-norm --by source
    bm25-norm minus bm25-fields+codes on 241 test queries: ndcg@10 +0.077 (95% interval +0.050 to +0.103); better on 111, worse on 41, the same on 89
    source queries bm25-fields+codes bm25-norm difference low high worse
    code 20 1.000 1.000 0.000 0.000 0.000 0
    planted 3 0.236 0.384 0.148 0.005 0.250 0
    translated 59 0.178 0.309 0.131 0.063 0.205 5
    typo 40 0.362 0.416 0.054 -0.005 0.116 12
    wands 119 0.641 0.709 0.068 0.034 0.103 24
  7. Look at the query that got worst:

    Terminal window
    search query "oliver parsons" --run bm25-norm -k 3
    bm25-norm: "oliver parsons" -> 3 results (query W-272: judged results are marked)
    1. LK-52242 soucy upholstered side chair in olive beige Dining Chairs 207.99 [not judged]
    2. LK-28242 magreb velvet upholstered parsons chair in olive Dining Chairs 221.99 [not judged]
    3. LK-98699 olive branch wall décor Wall Décor 14.99 [irrelevant]

Read step 6 row by row. The code queries do not change: the code rule of Module 3 still puts the product first. The worse column is the one to read twice. Normalising made 24 real English queries and 12 misspelt queries worse, although both groups got better on average.

In step 7, two of the three products are “not judged”. They count as irrelevant, so the measure may be a little unfair to bm25-norm. But a chair “in olive beige” is clearly not what a person who typed “oliver parsons” wanted.

Your turn: what do the French titles cost?

Section titled “Your turn: what do the French titles cost?”

Change one thing: turn off the French title field with a setting. Before you run it, predict: which language gets better without the French titles, and which gets worse?

Terminal window
search eval --run bm25-norm --set title_fr=0 --by language
The output, and what it shows
bm25-norm on 243 test queries: NDCG@10 0.559 (95% interval 0.510 to 0.605)
recall@100 0.509, MRR 0.554, zero results 2.5%, judged in the top 10 72.2%; 0.2 ms a query (p50), 0.4 ms (p95)
language queries ndcg@10 low high zero-result rate
en 182 0.677 0.628 0.724 0.016
fr 61 0.203 0.127 0.284 0.049

With the French titles, search eval --run bm25-norm --by language gives 0.674 for English and 0.307 for French. So the French titles cost the English queries 0.003 and give the French queries 0.104. The French queries are a quarter of the test set, and Larkfield wants French customers. So the trade is worth it. But it is a trade, and you only see it because you split the result by language.

The French queries are machine-translated from English queries, so they are cleaner than what real people type. Treat the French numbers as a first estimate, and measure again on real French queries when you have them.

Common mistake

Every search command fails after the copy

Symptom: Tomás copies the module’s files but skips step 2. Now even an old run fails. search eval --run bm25 stops with a long error that ends:

import Stemmer
ModuleNotFoundError: No module named 'Stemmer'

search info still works, because it does not load the runs.

Cause: the search command loads every file in larkfield_search/runs/ to find the runs. The new file understand.py loads normalise.py, and normalise.py needs the package PyStemmer, which is not installed. One missing package breaks every command that loads the runs, not only the new runs.

Fix: install the packages of the new requirements.txt: python -m pip install -r requirements.txt. Then run search eval --run bm25 again.

Consequence in search work: a search service loads all of its parts when it starts. A new part with a new dependency can stop the whole service, not only the new feature. Install and test every new dependency before you release the code that needs it.

Normalise almost always. Lower case, accents and a few unit rules cost nothing at query time, and they fix many small misses. On Larkfield’s test queries, normalising and stemming together are the largest step of this module’s rules.

Stem with care, and measure per query. Stemming joins forms of a word, but it also joins words that only look alike. Watch for names (people, brands, collections), colours and short words. If a stem makes a common query worse, you can protect the word. Keep a list of words that the stemmer must not change, such as “oliver”.

Lemmatise only when meaning matters more than speed, for example in long texts with full sentences. For short shop queries, stemming is the usual choice.

The cost is small but real. Every change of the analyser means a new index: the products must be normalised in the same way as the queries. In Larkfield’s project, the normalised indexes are built the first time a run needs them, in a few seconds. On a large catalogue, a re-index can take hours, so you plan it.

Check your understanding

Amira normalises the queries (lower case, stems) but forgets to rebuild the index, which still has plain tokens. What happens to the query “Cushions”, and why?

Show the answer

The query becomes the stem “cushion”. The plain index has the tokens “cushion” and “cushions” as they were written. So “Cushions” no longer matches the products that say “cushions”, although the person typed exactly that word. Normalising works only when both sides go through the same steps.

Check your understanding

Normalising raised NDCG@10 from 0.506 to 0.583 on the test queries. Omar says: “Then every query is better now.” What do you answer, with numbers?

Show the answer

No. The mean went up, but 41 of the 241 test queries got worse (111 got better). For “oliver parsons”, the stem “oliv” matches the colour olive, and NDCG@10 fell from 0.502 to 0.000. Look at the queries that got worse before you keep a change, and decide if they matter.

Check your understanding

Predict: a stemmer joins “lighting” and “light”. Name one query where this helps and one where it could hurt.

Show the answer

It helps a query like “outdoor lighting”, because products that say “outdoor light” now match. It could hurt a query like “light blue pillow”, where “light” is a colour: products about lighting now match the word too. A stemmer knows endings, not meanings.

Recap

  • To normalise text is to write it in one standard form: lower case, no accents, units written one way, tokens. Apply the same steps to the query and to the products.
  • Stemming cuts words to a shared stem with fixed rules (Snowball, through PyStemmer), in English and in French. Lemmatisation finds the dictionary form; it is more careful but slower, and rarely used for short queries.
  • On Larkfield’s test queries, bm25-norm reaches NDCG@10 0.583 (0.535 to 0.627), against 0.506 for bm25-fields+codes: +0.077 (+0.050 to +0.103). The machine-translated French titles give the French queries +0.104 and cost English 0.003.
  • 41 test queries got worse. “oliver parsons” fell from 0.502 to 0.000, because “oliver” and “olive” share a stem.

You can now match the forms of a word. But normalising cannot help when a word is spelt wrong: “outdor” is not a form of “outdoor”. Next, you correct spelling with a dictionary built from the catalogue and the query log.

Next: Fix spelling