Skip to content
Français

Scale

  • Lesson 28
  • 40 min
  • Module 8: Run matching in production

Where we are

Our daily run works on Larkfield's records. Records grow, and some companies have millions. In this lesson, we measure time and memory at 10,000, 100,000 and 1 million records, and we see when one machine is no longer enough.

What you will be able to do

Measure how time and memory grow from 10,000 to 1 million records. Find the step that grows too fast: one blocking key whose blocks grow with the records. Tighten that key and measure what it costs in true pairs. Then see what DuckDB and Splink can do on one machine, and when a cluster with Spark is the next step.

NotebookMeasure how blocking keys grow in a notebookColab · Kaggle · GitHub

The notebook counts the pairs that each blocking key makes for 5,000 to 60,108 records, without making the pairs. You see which key grows fastest, then tighten the surname key and measure its pair completeness. Its setup cells download the data, so it needs no local setup.

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

What you need on your computer

  • The project with the files of Module 8 and a trained model (match train), its virtual environment active. Or use the notebook only.

At a party of 10 guests, everyone shakes hands with everyone: 10 × 9 ÷ 2 = 45 handshakes. At a party of 100 guests: 4,950. Ten times the guests, about a hundred times the handshakes.

A clever host seats the guests at tables, and only people at one table shake hands. Twenty tables of five make 20 × 10 = 200 handshakes. Now ten times the guests means ten times the tables, and only ten times the handshakes.

There is one catch. The host seats guests by surname, and one table is for “Smith”. At a small party, three Smiths sit there. At a party of a million guests, thousands of Smiths sit at that one table, and they shake hands all night.

Blocking is that seating plan. A block of n records makes n × (n − 1) ÷ 2 pairs. Blocks that stay small keep the work small. A block that grows with the records brings back the problem that blocking was meant to solve.

The problem: from 60,000 to 1 million records

Section titled “The problem: from 60,000 to 1 million records”

TermLarkfieldA made-up Canadian online shop for home and garden products. It is not a real company. All Nextia courses use its help desk and data as the example.Learn more Meet Larkfield’s records are small. On this laptop, the whole run takes less than a minute. Other companies have far more records. To see what happens, the course team made two larger synthetic sets. They come from the same generator, with the same people model and error model: 109,830 and 1,099,849 records. They are not in labs, because they are large.

Records Candidate pairs Comparison vectors LightGBM Splink training Peak memory
11,067 51,879 0.3 s 1.2 s 3.9 s 0.45 GB
109,830 3,554,389 16.7 s 81.6 s 9.2 s 3.4 GB
1,099,849 about 340 million about 45 min (extrapolated) about 2.9 h (extrapolated) failed after 36 s about 24.5 GB for the vectors (extrapolated)

At the two smaller sizes, every step ran on the laptop: these are measured times. At 1.1 million records, we did not make all the candidate pairs: about 340 million pairs would not fit in memory. We counted them from the block sizes. We timed the comparison and LightGBM on 1 million of them, and multiplied. Those times are extrapolated: computed from a smaller run, never run at that size.

  • Measured
  • Extrapolated (dashed)
Time for the comparison vectors and LightGBM scoring of all candidate pairs, on one laptop CPU core. The solid line is measured; the dashed line to 1.1 million records is extrapolated from 1 million measured pairs. Both axes have a log scale.
Text description of the chart

A line chart with log scales on both axes. Records from 10,000 to about 1.6 million; minutes from 0.01 to 1,000. Measured: 11,067 records take 1.5 seconds, and 109,830 records take 1.6 minutes. Extrapolated, dashed: 1,099,849 records take about 218 minutes, about 3.6 hours. Ten times the records gave about 66 times the time, then about 133 times.

Ten times the records gave about 69 times the candidate pairs, then about 96 times. That is the party with the Smith table.

The five keys of the project make their pairs separately, before the duplicates are removed. Count them for each key:

  • Surname key
  • Postcode
  • Phone
Pairs made by three of the five blocking keys at three sizes of synthetic records, counted from the block sizes. Measured. Both axes have a log scale. The email and date of birth keys grow like the postcode.
Text description of the chart

Three lines with log scales on both axes. The surname key makes 38,775 pairs at 11,067 records, 3,418,193 at 109,830, and 337,158,879 at 1,099,849: a hundred times more for ten times the records. The postcode makes 15,997, 163,388 and 1,786,514: ten times more for ten times the records. The phone makes 4,143, 35,263 and 348,558. The surname line is steeper than the others.

The postcode, phone and email keys grow like the records. A postcode covers a few homes, a phone one home, and a bigger company does not put more people in one home. Their blocks stay small.

The surname key is different. Its key is the first 4 letters of the surname and the first 2 of the first name. More records means more people called “smit|jo”. The largest surname block had 71 records at 11,067 records, 633 at 109,830, and 5,487 at 1,099,849. One block of 5,487 records alone makes 15,050,841 pairs.

A key whose blocks grow with the records makes pairs that grow with the square of the records. Ten times the records, a hundred times the pairs.

DuckDB is a database that runs inside the Python program, with no server. It works on tables much larger than memory. When memory is full, it writes parts of the work to temporary files on the disk, and reads them back later. This is called spilling to disk. Splink uses DuckDB to compare and score the pairs.

Spilling has a limit: the disk. One uncapped run once filled a laptop’s disk. So the project caps DuckDB at 4 GB of memory and 4 GB of temporary files, as How Splink trains showed.

Splink trained in 3.9 s at 11,067 records, and 9.2 s at 109,830. At 1,099,849 records, it stopped after 36 seconds:

SplinkException: Error executing the following sql for table `__splink__df_comparison_vectors` …
… You can adjust this setting, by using (for example) PRAGMA max_temp_directory_size='10GiB'

This is the cap working. The error suggests a bigger cap, and with more disk the run would go on for longer. Our reading of the cause, not a measurement: Splink’s training compares the pairs of its own blocking rules, the date of birth and the full name. At this size, those blocks are large too.

The idea: tighten the keys before you buy a bigger machine

Section titled “The idea: tighten the keys before you buy a bigger machine”

A bigger machine makes each pair faster. It does not change how many pairs there are. The cheaper fix is a tighter key: a key with smaller blocks. We tried four changes to the surname key on the 1,099,849 records. The other four keys stayed the same:

Surname keyCandidate pairsLargest blockPair completenessVectors + LightGBM (extrapolated)
the project's key: 4 letters of the surname + 2 of the first name340,419,3765,4870.9938about 3.6 h
the whole surname + the whole first name158,342,3272,0780.9895about 1.7 h
the project's key + the first letter of the postcode67,983,4121,5830.9756about 44 min
the project's key + the first 3 characters of the postcode3,979,972260.9654about 3 min
no surname key at all3,260,497-0.9626about 2 min

The table shows a real choice, not a free fix:

  • Adding the first 3 characters of the postcode cuts the pairs from about 340 million to about 4 million: a few minutes instead of hours. But pair completeness falls from 0.9938 to 0.9654. That is 14,434 true pairs lost before any model sees them.
  • Why so many? The surname key was there for people who moved. They have different postcodes, phones and emails, so only the name brings their two records together. A key with the postcode in it loses exactly them, as in What blocking loses.
  • The whole name keeps almost everything (0.9895) and halves the pairs. The first letter of the postcode, a large postal region, sits in between.

So the answer depends on the business. Larkfield can lose about 1 in 30 true pairs and run in minutes. Or it can keep the whole name and run for about 1.7 hours. Either way, the pairs come first, and the machine second. The daily run of New records every day helps too: each day compares only the new records, a small part of these pairs.

The explorer from Module 3 shows the same trade-off on Larkfield’s 60,108 records. Compare keys with keys-wide, whose surname key uses only the first letter of the first name:

Explore

A wider surname key: more pairs, few more true pairs

Candidate pairs
1,108,591
Pair completeness
0.994
True pairs lost
174
Reduction ratio
0.999386

keys (postcode, email, phone, surname key or date of birth (the project’s default)): 1,108,591 candidate pairs out of 1,806,455,778. It keeps 27,503 of 27,677 true pairs and loses 174. True pairs with two different postcodes (mostly customers who moved): keys keeps 4,115 of 4,289.

  • One key
  • Several keys
All 4 measured blockings (synthetic records, 27,677 true pairs)
BlockingCandidate pairsTrue pairs keptPair completenessReduction ratio
surname1,035,49823,4890.8490.999427
postcode+surname1,103,62027,0190.9760.999389
keys1,108,59127,5030.9940.999386
keys-wide2,351,38227,5460.9950.998698
Candidate pairs against pair completeness for four blockings of the 60,108 synthetic customer records (1,806,455,778 possible pairs, 27,677 true pairs). Measured on the reference laptop.
Text description of the explorer

The explorer has four buttons: surname, postcode+surname, keys and keys-wide. It starts on keys. Four tiles show the candidate pairs, the pair completeness, the true pairs lost and the reduction ratio. A chart shows the four blockings as dots: candidate pairs on a log scale against pair completeness. A table lists them. Surname: 1,035,498 candidates, completeness 0.849. Postcode or surname: 1,103,620, 0.976. The five keys: 1,108,591, 0.994, 174 true pairs lost. The five keys with the wide surname key: 2,351,382, 0.995.

The wide key more than doubles the pairs and keeps only 0.001 more of the true pairs. At 60,000 records, that costs seconds. At a million records, the same choice costs hours.

match bench times each step on part of your records, or on all of them:

Terminal window
match bench
records candidates block s vectors s lgbm s splink train s splink predict s peak MB
10,000 30,369 0.0 0.1 0.4 2.7 0.1 421
30,000 273,828 0.1 0.7 3.5 2.5 0.5 1064
60,108 1,108,591 0.2 2.7 13.8 2.7 1.3 3111
Peak MB is the most memory this program used so far (it only grows).

Splink also prints its training log for each size. We left it out here. Your times will differ: ours ran on a laptop that was busy with other work. The counts do not change. On a quiet laptop, the course team measured 2.5 s for the vectors and 11.2 s for LightGBM at 60,108 records, with a peak of 4.0 GB.

Three times the records (10,000 to 30,000) gave nine times the candidate pairs: 273,828 ÷ 30,369 = 9.0. The square shows up on your own laptop.

Spark is a system that splits one job over many computers, a cluster. Splink can use Spark instead of DuckDB, with the same model and settings (Splink’s documentation; we did not run it). Move when one machine is not enough after you tightened the keys:

  • The candidate pairs still do not fit. For example, hundreds of millions of pairs that the business needs to keep, and a night is too short on one machine.
  • The data already lives in a cluster. Then copying it to one machine is the slow part.
  • Many runs at once. Several teams or many countries, each with its own matching.

A cluster costs money every hour, and it needs people who can run it. One laptop matched 110,000 records with every step under 2 minutes, in 3.4 GB of memory. Tighter keys took the extrapolated 1.1 million run from hours to minutes. Try both before a cluster.

Common mistake

Buy a bigger machine

Symptom: the matcher worked on 60,000 records in under a minute. On 1.1 million, Splink stops at the disk cap, and the comparison would take hours. TermAmiraLarkfield’s developer. She writes the code of the help-desk tools in the courses. asks for a machine with four times the memory.

Cause: the surname key’s blocks grow with the records. Its largest block went from 71 records to 5,487, and the pairs grew with the square: about 340 million. Four times the memory does not change the number of pairs.

Fix: count the pairs of each key from the block sizes first. Find the key that grows too fast, and tighten it. Measure the pair completeness of every option, and let the business choose the trade-off. Then size the machine.

Consequence in AI work: every step that compares items in pairs has this shape. Examples are near-duplicate search, clustering, pairwise ranking, and the evaluation of every pair of answers. Check the blocks, or the candidate lists, before you scale the machine.

Check your understanding

A blocking key makes 40,000 pairs on 10,000 records. Its blocks grow with the records. About how many pairs will it make on 100,000 records? And a key whose blocks stay the same size?

Show the answer

If the blocks grow with the records, the pairs grow with the square: 10 × 10 = 100 times more, about 4 million pairs. If the blocks stay the same size, there are only more blocks: 10 times more, about 400,000 pairs. Larkfield’s surname key went from 38,775 to 3,418,193 pairs; the postcode from 15,997 to 163,388.

Check your understanding

At 1.1 million records, adding the first 3 characters of the postcode to the surname key cut the time from hours to minutes. What did it cost, and which customers pay for it?

Show the answer

Pair completeness fell from 0.9938 to 0.9654: 14,434 true pairs were never compared. They are mostly customers who moved: their two records have different postcodes, phones and emails, and only the name could bring them together. They become missed matches.

Check your understanding

Splink stopped at 1.1 million records with a DuckDB error about the temporary directory. Is this a bug? What do you do?

Show the answer

No. It is the project’s cap. DuckDB may write at most 4 GB of temporary files, so a run that needs more stops before the disk is full. You can raise the cap if the disk has room. But first ask why it needs so much: the blocking rules of Splink’s training make large blocks at this size. Tighten them, or train on a sample of the records, then score.

Recap

  • Measured: 11,067 records → 51,879 candidates, every step under 4 s; 109,830 → 3,554,389, LightGBM 81.6 s, peak 3.4 GB.
  • Extrapolated: 1,099,849 records → about 340 million candidates, about 45 minutes for the vectors and 2.9 hours for LightGBM. Splink training stopped at the 4 GB disk cap.
  • A key whose blocks grow with the records makes pairs that grow with the square of the records. Here, it was the surname key.
  • Tighten the keys before you buy a bigger machine, and measure what each option loses: from about 340 million pairs to 4 million cost 14,434 true pairs.
  • DuckDB spills to disk on one machine. Move to Spark only when tightened keys still make too many pairs, or the data already lives in a cluster.

Next, the last lesson of the module: the personal data in all these pairs.

Next: Privacy and matching