Module 1The matching problem
3 lessonsThe matching problem. Count Larkfield's customers across three lists, name the two errors (false merges and missed matches) and their costs, and set up the project with the records.
Decide when two records describe the same customer or product. Clean and block the records, then compare rules, string measures, probabilistic linkage, pair classifiers, embeddings and a language model for the unsure pairs on the same data, and turn pairs into entities that people can review and undo.
Read every lesson as a guest, or sign in to enroll and track your progress.
Enroll to save the lessons that you complete. It is free.
You are enrolled. From now on, the lessons that you complete are saved.
This course teaches record linkage and duplicate removal: normalisation and parsing, exact and rule-based matching, blocking and what it loses, string and phonetic similarity, probabilistic linkage with Splink, pair classifiers and active learning, embeddings, a language model for unsure pairs only, clustering pairs into entities, golden records, human review, daily matching, scale and privacy. It does not teach SQL joins from the start, how to train a classifier from the start, or the law on personal data.
Take another course if you want to:
By the end of this course, you will be able to:
Larkfield, an online shop for home and garden products, has bought a chain of garden centres. Now three lists describe its customers: the shop’s accounts, the loyalty programme and the garden centres’ own list. The same person is in two or three of them, with a typo, a nickname, an old address or a family email. Larkfield sends two emails to one person, counts too many customers, and cannot see one person behind several accounts.
In this course, you follow Priya, Larkfield’s data scientist, Amira, the developer who owns the matching job, and Ivana, who owns the customer data and checks the unsure cases. You start by measuring: you learn the two errors, false merges and missed matches, and you measure pairs and entities with precision and recall. Then you change one thing at a time and measure it, on the same records, in one comparison table.
You go from clean data and exact rules to blocking, string and sound measures, probabilistic linkage with Splink, a trained pair classifier, embeddings and, only for the unsure pairs, a language model. You join pairs into customers, build a golden record that can be undone, and send unsure cases to a review queue that has a daily limit.
You build one project through the course: larkfield-match, a small matching tool with a match command. You finish with production work: new records every day, scale, and privacy. Every customer record in the course is synthetic, made by our code from known true people, so you can always check the answer. The case studies use real open data.
You completed SQL and Data Preparation for AI and Machine Learning, From Problem to Reliable Model, or you have the same skills (joins, cleaning, data quality checks, classification, precision, recall and thresholds). You can read and change Python and pandas code, use a notebook and a terminal. The mathematics and AI security courses help but are not required; lessons that use their ideas explain them briefly and link to them. You need Python 3.12 on your computer, or a free Google or Kaggle account for the notebooks. You need no key, no graphics card and no money.
Courses to take first: SQL and Data Preparation for AI, Machine Learning: From Problem to Reliable Model.
Data engineers, data scientists and AI engineers who must join, clean or remove duplicates from records about customers, companies or products, and who want to choose a matching method by measurement and by the cost of each error.
Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses
The matching problem. Count Larkfield's customers across three lists, name the two errors (false merges and missed matches) and their costs, and set up the project with the records.
Clean first, then match exactly. Normalise and parse names, addresses, phones and product titles, build exact keys and rules, and measure them with pair precision and recall.
Blocking. Why comparing every pair is impossible, blocking keys and sorted neighbourhood, MinHash and LSH for product titles, and how much each blocking loses.
Compare records. Levenshtein, Jaro–Winkler and token set ratio, sound codes and where they fail, and a comparison vector per pair with weights chosen by hand.
Probabilistic linkage. The Fellegi–Sunter idea with m and u probabilities, weights learned without labels in Splink on DuckDB, and two thresholds with a review band.
Learned matching. A pair classifier, active learning, embeddings for names and product titles, and a language model for the unsure pairs only, with recorded answers and their cost.
From pairs to entities. Connected components and the giant cluster, safer clustering, golden records with merge rules and undo, and cluster measures.
Run matching in production. New records every day with stable entity IDs, a human review queue, time and memory at a million records, and privacy.
Three end-to-end case studies on open data. Products between two shops, bibliographic records, and duplicate people and companies. Complete at least one for the certificate; each one is a portfolio piece.
Complete at least one case study for the certificate. The others are optional. What each one covers
Use every module in one practice project. You collect evidence, and check it against the course outcomes.
Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.
Match products between two shops2 hr
On an open product-matching benchmark with an allowed licence: blocking, string, supervised and embedding methods, and a language model for the unsure pairs; report precision at a fixed recall and the cost per thousand pairs.
Link bibliographic records2 hr
On open bibliographic records with an allowed licence: Splink against a pair classifier; error analysis of titles, author lists and venues.
Find duplicate people and companies2 hr 30 min
On simulated person records with an allowed licence and an open company register: rules, Splink and clustering into entities with a golden record.
Loading the reviews…
Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.