Parse into parts
- Lesson 5
- 35 min
- Module 2: Clean first, then match exactly
Where we are
Normalising made equal values look equal. Some values hold several facts in one string, such as a full address or a product title with a pack size. In this lesson, we parse them into parts, and we decide what to do when parsing fails.
Lesson 5 of 29 · Entity Resolution and Record Matching: Is This the Same Thing?Parse into parts
1. The matching problem
2. Clean first, then match exactly
3. Blocking: do not compare everything
4. Compare records: similarity measures
5. Probabilistic linkage
6. Learned matching: classifiers, embeddings and language models
7. From pairs to entities
8. Run matching in production
What you will be able to do
Split values that hold several facts into separate fields. A name becomes first name and surname. An address becomes number, street, unit, town, province and postcode. A product title becomes words, size and pack. Then decide what a matcher does with a record that does not parse.
NotebookDo this lesson in a notebook
The notebook splits the garden centres’ names and addresses with pandas and regular expressions, and counts the records that do not parse. Then you write a parser for the pack size in product titles, and check your result. It downloads the data, so it needs no local setup.
Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.
Before you startWhat to know and to have ready
What you need to know
- Normal forms, and the file
larkfield_match/normalise.py, from Normalise. - A regular expression is a short pattern that describes text, such as “a letter, a digit, a letter”. Python’s
remodule uses them. You do not need to write one in this lesson: we explain each pattern in words.
Tested on macOS with Python 3.12.13. The Windows (PowerShell) and Linux commands are written the same way, but we did not test them.
The clerk fills in the new form
Section titled “The clerk fills in the new form”The clinic gets a new filing system. Its form has separate boxes: first name, surname, street, town, postcode. The old cards have one line for the whole address, and the name as “SURNAME, First”.
The clerk copies each old card into the new form. She knows that the postcode is at the end of the address. She knows that the part before the comma is the surname. When a card is unclear, she does not guess. She marks it and leaves the box empty.
After this work, she can compare one box at a time: this postcode with that postcode. That is what parsing does for a matcher.
The problem: one string, several facts
Section titled “The problem: one string, several facts”Look at the three lists from Module 1 again. Each one stores the address differently:
shop address '2709 boul. Saint-Joseph app. 27' city 'Saint-Ambroise-des-Monts' province 'QC' postcode 'g6z 3w2'loyalty (no address) postcode 'G6Z3W2'garden address '8983 Fern Lane, Ashbury Falls ON N2L 0V8' (no postcode column)The garden centres write the whole address in one field. The postcode is inside it. A rule such as “same name and same postcode” cannot use a garden-centre record until something takes the postcode out of the string. So all 11,943 garden-centre records are out of reach for that rule.
Names have the same problem. The garden centres write GREEN, SAMUEL. The loyalty list writes Samuel Green. As whole strings, green samuel and samuel green are different, even after normalising. To compare surnames, a matcher needs the surname as its own field.
Product titles hide numbers. In feed A, Mossway Vinyl Garden Hose, 30 ft, Brown (Pack of 2) holds a brand, a size, a colour and a pack size, all in one string.
The idea: parse each value into named parts
Section titled “The idea: parse each value into named parts”To parse a value is to split it into named parts, by its pattern. The result is a set of fields: one fact in each, such as the town or the postcode. After parsing, each field is normalised and compared on its own.
A parser uses what is certain first. In a Canadian address, the postcode is at the end and has a fixed shape: letter, digit, letter, digit, letter, digit. The province is a two-letter code just before it. So the parser reads from the end, and takes out the surest parts first.
A worked example with every step visible
Section titled “A worked example with every step visible”The project joins the shop’s four address columns into one line. Then the same parser reads every address, from all three lists. The diagram follows the line of shop record S-029535.
Text description of the diagram
The line is “2709 boul. Saint-Joseph app. 27, Saint-Ambroise-des-Monts QC g6z 3w2”. Step 1 takes the postcode from the end: G6Z3W2. Step 2 takes the province before it: QC. Step 3 takes the town after the last comma: “st ambroise des monts”. Step 4 finds the unit after the word “app.”: 27. Step 5 takes the house number at the start: 2709. Step 6 keeps the rest as the street, and normalises it to “blvd st joseph”. Nothing is left. The record has a number, a street and a town, so the parser marks it as parsed.
Step 4 knows several words for a flat: app. (French), apt, apartment, unit, suite, and the sign #. It also knows the form 3-12, where 3 is the flat and 12 the house number. Step 6 uses the street rules of the last lesson: boul. becomes blvd, and Saint becomes st.
Names: the comma decides
Section titled “Names: the comma decides”Names follow two patterns. With a comma, the part before it is the surname: TREMBLAY, MARIE-CLAUDE. Without a comma, the last word is the surname: Dr. Jean-François Desjardins. Titles go first, as in the last lesson.
The last-word rule is a guess, and it is sometimes wrong. A surname of two words, such as “Saint Pierre” without a hyphen, loses its first word. Our data writes such names with a hyphen, so the guess works here. On other data, check it.
Parse in the project
Section titled “Parse in the project”This lesson adds two modules and two commands to the project. larkfield_match/parse.py holds the parsers. larkfield_match/prepared.py normalises and parses every record into one table, the prepared records. The commands match parse and match normalise show the result.
-
Copy the four files from the snapshot
end-of-m02:Terminal window S=../nextia-learning-labs/C39/snapshots/end-of-m02for f in larkfield_match/parse.py larkfield_match/prepared.py larkfield_match/commands/parse.py larkfield_match/commands/normalise.py; docp "$S/$f" "$f"doneTerminal window $S = "..\nextia-learning-labs\C39\snapshots\end-of-m02"foreach ($f in 'larkfield_match\parse.py', 'larkfield_match\prepared.py', 'larkfield_match\commands\parse.py', 'larkfield_match\commands\normalise.py') { Copy-Item "$S\$f" $f } -
Parse one address and one name:
Terminal window match parse address "2709 boul. Saint-Joseph app. 27, Saint-Ambroise-des-Monts QC G6Z 3W2"number '2709'street 'blvd st joseph'unit '27'town 'st ambroise des monts'province 'QC'postcode 'G6Z3W2'ok TrueTerminal window match parse name "TREMBLAY, MARIE-CLAUDE"first 'marie claude'last 'tremblay'ok True -
Show one whole record, as written and prepared. The first run prepares all 60,108 records and saves them in
outputs/prepared_customers.pkl. That takes a few seconds (under ten on our laptop); later runs read the saved file.Terminal window match normalise S-029535As written:first_name 'Jean-François'last_name 'Desjardins'email 'desjardins.jean-francois77@mail.example'phone ''address '2709 boul. Saint-Joseph app. 27'city 'Saint-Ambroise-des-Monts'province 'QC'postcode 'g6z 3w2'date_of_birth ''created '2022-10-15'Prepared:source 'shop'first 'jean francois'last 'desjardins'full 'jean francois desjardins'email 'desjardins.jean-francois77@mail.example'phone ''number '2709'street 'blvd st joseph'unit '27'town 'st ambroise des monts'province 'QC'postcode 'G6Z3W2'dob ''created '2022-10-15'parse_ok True
The prepared record has the same columns for every list. The source values stay unchanged in data/. Try match normalise G-000002: the garden-centre record MARTIN, N. becomes first name n and surname martin, and its address gives the postcode N2L0V8. After parsing, all 11,943 garden-centre records have a postcode that a rule can compare.
When parsing fails
Section titled “When parsing fails”Most records parse. In our synthetic data, the share of records that parse is 1.000 for the shop. It is 0.996 for loyalty and 0.996 for the garden centres. That is 65 loyalty records and 46 garden-centre records, 111 in all.
Every one of the 111 has the same cause: the letter M. in the name. You met it in the last lesson’s failure case.
| Record | Name as written | First name | Surname | Why it fails |
|---|---|---|---|---|
| L-000315 | M. Pelletier |
(empty) | pelletier |
“M.” is read as the title Monsieur, so no first name is left |
| G-000256 | BROWN, M. |
(empty) | brown |
“M.” is an initial here, but the title rule removes it |
| G-001101 | M., LAVOIE |
lavoie |
(empty) | the list swapped the parts, so the comma rule takes “M.” as the surname |
The parser marks these records with parse_ok False. It does not stop, and it does not invent a value. It keeps the fields that it could fill. L-000315 still has its surname, email, phone, postcode and date of birth. A rule that needs the first name cannot use this record. A rule that uses the email still can.
Three ways to handle a record that does not parse
Section titled “Three ways to handle a record that does not parse”- Keep it, with the fields it has. This is what the project does. The record can still match on other fields. It is cheap and safe, and it may miss some pairs.
- Send it to a person. A data steward reads the 111 records and fixes the fields by hand. At TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield, that is Ivana’s job. It is accurate and slow. It works for 111 records, and not for 100,000.
- Improve the parser. For example, read
M.as an initial when a surname follows a comma. Each new rule needs a test, and it can break other records.
Never drop a record because it does not parse. A dropped record cannot match anything, so all its true pairs become missed matches, and nobody sees them.
Product titles: units and packs
Section titled “Product titles: units and packs”Hana, the merchandising lead, owns the supplier feeds. She knows that their titles hide two numbers that decide what a product is: the size and the pack. Feed A from Northfield Supply writes 4 ft. Feed B from Copperleaf Wholesale writes 1.2 m for the same product. A parser that finds the size can convert it to one unit, so that the numbers can be compared.
Text description of the diagram
The left pair is one product. Feed A writes “4 ft” and feed B writes “1.2 m”. The parser converts both to metres: 1.22 and 1.2. The brand and the words are the same, but the sizes are not exactly equal, so an exact rule says “different”. Module 4 compares numbers with a tolerance. The right pair is two products: a single metal trellis, and the same trellis in a pack of 2. Brand, words and size are equal. Only the pack field shows the difference.
The project’s title parser finds the pack first ((Pack of 2), 2-pack, x2, 2 pk), and then the size with its unit. It converts feet, inches and centimetres to metres, pounds to kilograms, and square feet to square metres. The words that are left are normalised, and a few spellings are made the same: gdn becomes garden, color becomes colour.
match parse title "Mossway Vinyl Garden Hose, 30 ft, Brown (Pack of 2)" size 9.14 size_kind 'length_m' pack 2 words 'mossway vinyl garden hose brown' ok TrueIn our synthetic feeds, the parser finds a size in 90.5% of feed A’s titles and 90.2% of feed B’s. The rest are not failures: gloves and hose nozzles have no size in their title. ok is False for them, but nothing is wrong. Read what a “failure” means before you count it.
Common mistake
A parse that is wrong, not empty
Symptom: a few records have a strange street, such as old mill blvd larkmere, and no town. A rule on the town never matches them.
Cause: the address has no comma before the town:
match parse address "5687 OLD MILL BOULEVARD LARKMERE ON N5C9Y7" number '5687' street 'old mill blvd larkmere' unit '' town '' province 'ON' postcode 'N5C9Y7' ok FalseThe parser finds the town after the last comma. Without a comma, it cannot see where the street ends and the town starts. So the town goes into the street. Here, ok is False because the town is empty, so the record is marked. A worse parser would guess a town, and the wrong value would look like a real one.
Fix: the same address with a comma, 5687 OLD MILL BOULEVARD, LARKMERE ON N5C9Y7, parses with town larkmere. If your data often has no comma, add a rule: a list of known towns, read from the end. In our data, every address has the comma, so the project does not need this rule. Test any new rule on records that parsed before, too.
Consequence in AI work: a parser can fail loudly (an empty field and a mark) or silently (a wrong value). Prefer loud failures, and count them for each source. A wrong value in a field is worse than an empty field, because a matcher trusts it.
When to parse, and what it costs
Section titled “When to parse, and what it costs”Parse when a field holds several facts that matter for matching: addresses, full names, product titles, and codes with a known structure. Parse before you block and compare: Module 3 builds blocking keys from the postcode and the surname.
Do not parse what you do not compare. Each parser needs rules, tests and care. The loyalty list has no street address, so there is nothing to parse there.
What it costs. A small parser with regular expressions is quick to run: the project prepares all 60,108 records in under ten seconds on our laptop. It works because our data follows a few patterns. For addresses from many countries, there are large trained address parsers, such as libpostal. They need much more memory and set-up, and this course does not use one.
Knowledge check
Section titled “Knowledge check”Check your understanding
Parse this garden-centre address by hand, in the order of the parser: “3-12, R. DES ERABLES, STE-ANNE-DES-PRES QC J0L1A0”. Give the postcode, province, town, unit, number and street.
Show the answer
- Postcode at the end:
J0L1A0. - Province before it:
QC. - Town after the last comma:
ste anne des pres. - Unit: the form
3-12means flat 3 of house number 12, so the unit is3. - Number:
12. - Street:
R. DES ERABLESbecomesrue des erables.
The project’s test test_split_address checks exactly this address. You can run match parse address with it.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
111 records do not parse, because of “M.” in the name. Amira wants to remove them from the matching job: “They are only 0.2% of the records.” What do you answer?
Show the answer
Do not remove them. A removed record cannot match, so all its true pairs become missed matches, and the customer count stays too high. These records still have useful fields: L-000315 has a surname, an email, a phone, a postcode and a date of birth. Keep them with the fields they have, and mark them with parse_ok False. Then decide. A person can fix 111 records in a short time. Or a new parser rule can read “M.” as an initial when it comes after a comma.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
Two titles parse to the same brand, words and size, but pack 1 and pack 2. Tomás says: “It is the same product, only the quantity differs. Merge them.” Is he right?
Show the answer
No. A single item and a pack of 2 are two products. They have different prices, different barcodes, and different stock. If they are merged, a customer who orders one trellis can get a pack of two, or the price comparison is wrong. That is why the parser keeps the pack as its own field, and a product rule must compare it. In our feeds, these pairs are a practice problem we made on purpose: 260 single items with a pack of 2.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- To parse is to split a value into named fields by its pattern. The parser takes the surest parts first: in an address, the postcode and the province at the end.
- The project’s
prepared.pynormalises and parses every record into the prepared records: the same columns for all three lists. After parsing, all 11,943 garden-centre records have a postcode. - 111 records do not parse, all because of “M.” in the name. The project keeps them with the fields it could fill, and marks them with
parse_okFalse. Never drop a record that does not parse. - Product titles hide a size and a pack. The parser converts sizes to one unit. The pack field keeps a single item and its pack of 2 apart.
- A wrong field is worse than an empty field. Prefer a parser that fails loudly.
Every record now has clean fields: first name, surname, email, phone, postcode, date of birth. Next, you combine them in rules, such as “same phone and same surname”, and you get the baseline numbers that every later method must beat.