Skip to content
Français

Entity Resolution and Record Matching: Is This the Same Thing?

Decide when two records describe the same customer or product. Clean and block the records, then compare rules, string measures, probabilistic linkage, pair classifiers, embeddings and a language model for the unsure pairs on the same data, and turn pairs into entities that people can review and undo.

Read every lesson as a guest, or sign in to enroll and track your progress.

Is this the right course for you?What this course covers and does not cover, and 4 courses to take for other needs.

This course teaches record linkage and duplicate removal: normalisation and parsing, exact and rule-based matching, blocking and what it loses, string and phonetic similarity, probabilistic linkage with Splink, pair classifiers and active learning, embeddings, a language model for unsure pairs only, clustering pairs into entities, golden records, human review, daily matching, scale and privacy. It does not teach SQL joins from the start, how to train a classifier from the start, or the law on personal data.

Take another course if you want to:

What you'll learn

By the end of this course, you will be able to:

  • Frame a matching problem: what an entity is, what a false merge and a missed match each cost, and which one the business can accept more of.
  • Normalise and parse names, addresses, phone numbers and product titles, and build exact and rule-based matching as a baseline.
  • Make matching fast with blocking (keys, sorted neighbourhood, MinHash LSH) and measure what blocking loses.
  • Score pairs with string, phonetic, probabilistic (Fellegi–Sunter), supervised and embedding methods, and use a language model only for unsure pairs, with recorded and checked outputs.
  • Turn pair decisions into entities with clustering, build a golden record with merge rules, and send unsure cases to human review.
  • Measure pairs and clusters with precision and recall, run matching on new records over time, and protect personal data while doing it.

About this course

Larkfield, an online shop for home and garden products, has bought a chain of garden centres. Now three lists describe its customers: the shop’s accounts, the loyalty programme and the garden centres’ own list. The same person is in two or three of them, with a typo, a nickname, an old address or a family email. Larkfield sends two emails to one person, counts too many customers, and cannot see one person behind several accounts.

In this course, you follow Priya, Larkfield’s data scientist, Amira, the developer who owns the matching job, and Ivana, who owns the customer data and checks the unsure cases. You start by measuring: you learn the two errors, false merges and missed matches, and you measure pairs and entities with precision and recall. Then you change one thing at a time and measure it, on the same records, in one comparison table.

You go from clean data and exact rules to blocking, string and sound measures, probabilistic linkage with Splink, a trained pair classifier, embeddings and, only for the unsure pairs, a language model. You join pairs into customers, build a golden record that can be undone, and send unsure cases to a review queue that has a daily limit.

You build one project through the course: larkfield-match, a small matching tool with a match command. You finish with production work: new records every day, scale, and privacy. Every customer record in the course is synthetic, made by our code from known true people, so you can always check the answer. The case studies use real open data.

Before you start

You completed SQL and Data Preparation for AI and Machine Learning, From Problem to Reliable Model, or you have the same skills (joins, cleaning, data quality checks, classification, precision, recall and thresholds). You can read and change Python and pandas code, use a notebook and a terminal. The mathematics and AI security courses help but are not required; lessons that use their ideas explain them briefly and link to them. You need Python 3.12 on your computer, or a free Google or Kaggle account for the notebooks. You need no key, no graphics card and no money.

Courses to take first: SQL and Data Preparation for AI, Machine Learning: From Problem to Reliable Model.

Who this is for

Data engineers, data scientists and AI engineers who must join, clean or remove duplicates from records about customers, companies or products, and who want to choose a matching method by measurement and by the cost of each error.

How we made this course

Experienced AI practitioners designed this course. AI helped us draft the content; people chose the teaching approach and the examples, and reviewed every lesson. The outputs that you see are the actual results of running the code. How we make our courses

Inside the course

After the case studiesFinal assignment

Use every module in one practice project. You collect evidence, and check it against the course outcomes.

  1. Final assignment300 min

Case studies

Realistic projects from start to finish, on real data, with every output explained. Each one makes a portfolio piece. Complete at least one for the certificate; the others are optional.

  1. Match products between two shops2 hr

    On an open product-matching benchmark with an allowed licence: blocking, string, supervised and embedding methods, and a language model for the unsure pairs; report precision at a fixed recall and the cost per thousand pairs.

  2. Link bibliographic records2 hr

    On open bibliographic records with an allowed licence: Splink against a pair classifier; error analysis of titles, author lists and venues.

  3. Find duplicate people and companies2 hr 30 min

    On simulated person records with an allowed licence and an open company register: rules, Splink and clustering into entities with a golden record.

Reviews

Loading the reviews…

    Took this course? Sign in to review it. Learners can write a review after they complete 25% of the lessons.

    Share this course