Books for readers
- Case study 3
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, we recommended products. In this case study, we recommend books from public ratings and book data, and compare content-based, collaborative and hybrid methods on coverage and novelty.
Case study 3 of 4 · Recommendation and Personalisation: Show Each Person What MattersBooks for readers
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Build and compare recommenders on six million real book ratings, split by their position in the file because the data has no dates. You measure accuracy and also coverage, novelty and the share of famous books, by segment. Then you decide what a bookshop should show and to whom, what it costs, and what a person still checks. Last, you run the design once on ratings that no method has seen.
This case study is a notebook, not the larkfield-recs project. It has every code block of this page, in the same order, with three tasks and their checks. Its setup cells install the packages and download the data (29 MB) from its source. The content method downloads the course’s text model (about 200 MB, once). Read the explanations here, and run the code there.
Before you startWhat to know and to have ready
What you need to know
- The time split, from Split by time, and the baselines, from The baselines to beat.
- Recall@10 and NDCG@10, from Ranking metrics; coverage, novelty and diversity, from Beyond accuracy; segments and paired differences, from Evaluate by segment.
- Co-occurrence, lift and cosine, from People also bought, and ALS, from Hidden tastes.
- Text embeddings and content-based lists, from Content-based recommendations, and blends, from Hybrids.
What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)
- Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas, SciPy, matplotlib, implicit and sentence-transformers (with PyTorch).
- A new, empty folder. The setup cell downloads the data into a folder
books-for-readers. To start again, delete it and run the notebook from the top.
Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4, pandas 3.1.1, SciPy 1.18.1, implicit 0.7.3, sentence-transformers 6.2.0 and PyTorch 2.15.0. Windows, Linux, Colab and Kaggle were not run.
The situation
Section titled “The situation”A small online bookshop sells new and second-hand books. Its readers log in to buy and to rate what they read. The home page has space for a block of ten books: “Books for you”.
Mina chooses the books that the shop stocks. She has one wish and one fear. Her wish: “Help each reader find their next book.” Her fear: “Please do not show Harry Potter to everybody. Our readers know it already.”
Leo is the shop’s developer. He will build and run what you recommend. He asks: “Which method, for whom? How long does it take every night? What do I do when it breaks?”
The wish and the fear pull against each other. Famous books waste the best place on the page, because readers know them. Unknown books that nobody wants waste it too.
Mina, Leo and the bookshop are made up. The ratings are real.
What you will build
Section titled “What you will build”A comparison of seven methods on the same readers and the same window of ratings. They are popularity (popular, trending), neighbours (co-occurrence, item-cf), ALS (als), a content method (content) and a hybrid. The result is a recommendation for Mina and Leo: a design, its settings, its cost, and what a person still checks.
The data
Section titled “The data”Goodbooks-10k is a public dataset of ratings from Goodreads, a website where people record and rate the books they read. It has the 10,000 books with the most ratings, and the readers who rated them.
| Source | github.com/zygmuntz/goodbooks-10k, release v1.0 |
| Creator | Zygmunt Zając |
| Licence | Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) |
| Attribution | “goodbooks-10k” by Zygmunt Zając, https://github.com/zygmuntz/goodbooks-10k, licensed under CC BY-SA 4.0. |
| Files | ratings.zip, books.zip, book_tags.zip and tags.zip (29 MB together). The course does not keep a copy: the notebook downloads them from the dataset’s release page and checks their checksums. The dataset card has the details. |
ShareAlike means: if you share the data, or a changed version of it, you must use the same licence and give the same credit. So the course links the source and does not copy the files.
One row of ratings.csv is one reader’s rating of one book, from 1 to 5 stars. There are 5,976,479 rows, from 53,424 readers, on 10,000 books, with no dates.
The files we use:
| File | What it has |
|---|---|
ratings.csv |
user_id (the reader), book_id (1 to 10,000) and rating (1 to 5). |
books.csv |
The title (with the series in brackets, such as (Harry Potter, #1)), the authors and the year of first publication of each book. |
book_tags.csv and tags.csv |
The words that Goodreads users put on each book, such as fantasy, history or to-read, with how often. |
How this data differs from Larkfield
Section titled “How this data differs from Larkfield”Larkfield’s shop log is synthetic. This data is real behaviour of real (pseudonymous) readers. Four differences matter:
- No dates. We only know the order of the ratings (see the split below).
- Everybody has a history. Every reader has at least 19 ratings (median 111); at Larkfield, 53.7% of buyers ordered once. The dataset kept only the most rated books, and readers with many ratings.
- A few books are everywhere. The 100 most read books (1% of the books) get about 17% of all ratings. Harry Potter, The Hunger Games and Twilight are near the top.
- Real text. Every book has a title, an author and tags written by readers, not by a model.
A rating is also not a sale: many readers rate old books when they join. We predict which books a reader rates next, the closest signal to “which book they want next”.
Frame the problem
Section titled “Frame the problem”- When are the lists made? Every night, from all ratings before that night. A method may only use the past.
- What do we predict? For each reader who rates books in the next window, the books they rate in it. A book the reader has already rated is never shown.
- The main measure: recall@10, the share of the rated books in the first 10 suggestions, averaged over readers. NDCG@10 also counts places near the top more.
- Beyond accuracy, as in Beyond accuracy:
- coverage: the share of the books that appear in at least one list;
- novelty, in bits: a book that every reader rated has a novelty near 0;
- diversity: here, the share of pairs in a list by different authors;
- top 100: the share of suggestions from the 100 most read books;
- series: the share of suggestions that continue a series the reader started.
- The segments, counted in ratings before the window. New readers have none, light readers have 1 to 49, and heavy readers have 50 or more. We count ratings, not orders, so the names differ from Larkfield’s new, occasional and loyal.
A split without dates
Section titled “A split without dates”The dataset’s README says that the ratings are “sorted by time”. We checked. The file is not grouped by reader: the readers are mixed together, in 2,865,964 runs. Reader numbers grow with the position of their first rating, and new books appear only near the end of the file:
Text description of the chart
The line shows, for each tenth of the file, the share of ratings on books first published in 2015 or later. It is 0.00% in the first tenth and stays near zero until the seventh tenth. Then it rises: 0.9% in the eighth, 7.5% in the ninth and 19.5% in the last tenth.
So the position in the file works as a clock, but not as a date: we cannot say how many weeks a window covers. It keeps the rule of Split by time: a method only learns from ratings that came before the ones it predicts.
Text description of the diagram
Validation is 85% to 90% of the file: we try 9 ALS settings and 15 hybrid weightings there, and keep the best. The test is 90% to 95%: each method runs once, with every rating before 90%, and we report these numbers. The new input is the last 5%: only the chosen design runs, once.
To keep the notebook fast, we keep one reader in four (reader numbers divisible by 4): 13,356 readers and 1,493,771 ratings. Each rating keeps its position in the whole file, so the windows stay the same.
full, books, book_tags = read("data")ratings = sample(full)W = windows(len(full))test = Period(ratings, *W["test"])The test window has 5,591 readers: 180 new, 535 light and 4,876 heavy. They rated 13.4 books each on average (median 7).
Explore what the decision needs
Section titled “Explore what the decision needs”Popularity is strong, but it moves. In the test window, 9.6% of the ratings are on the 100 books with the most readers before the window. But 16.4% are on books first published in 2015 or later. The notebook’s first task computes this share. The top of popular and trending before the window shows it:
| Place | popular (all ratings before the window) | trending (the last 300,000 ratings) |
|---|---|---|
| 1 | The Hunger Games (The Hunger Games, #1) | The Girl on the Train |
| 2 | Harry Potter and the Sorcerer's Stone (Harry Potter, #1) | The Martian |
| 3 | To Kill a Mockingbird | Go Set a Watchman |
| 4 | Twilight (Twilight, #1) | All the Light We Cannot See |
| 5 | Catching Fire (The Hunger Games, #2) | To Kill a Mockingbird |
popular shows the famous books of all time. trending shows what readers rate now, such as The Girl on the Train. A method with no weight on the recent past will miss those books.
Series are everywhere. 4,422 of the 10,000 books are part of a series (their title ends with something like (Harry Potter, #3)). A reader of book 1 often rates book 2 next, but would probably find it without help. So we count these suggestions separately.
The baselines
Section titled “The baselines”These are the course’s baselines from The baselines to beat. popular ranks the books by the number of readers who ever rated them. trending does the same for the last 300,000 ratings before the window, about 5% of the file. Everyone gets the same list, minus the books they already rated:
def popular(period): """The books with the most readers before the cut.""" return np.argsort(-period.item_readers, kind="stable")
def trending(period): """The books with the most readers in the last 300,000 ratings before the cut (about 5% of the file).""" return np.argsort(-period.recent_readers, kind="stable")On the test window: popular 0.015 and trending 0.044 (95% interval 0.041 to 0.046). Trending is 3.0 times better than popular. Readers follow what is new and talked about, as Larkfield’s customers followed the season.
But look at what trending costs. Its lists use 0.41% of the books, and 55% of its suggestions are among the 100 most read books. That is exactly Mina’s fear.
The alternatives
Section titled “The alternatives”Neighbours, and the Harry Potter problem
Section titled “Neighbours, and the Harry Potter problem”Co-occurrence counts the readers who rated both books: “readers of this book also read that one”, as in People also bought. Item-cf divides that count by the size of each book (cosine). We keep the 100 nearest books of each book. A reader’s score for a book adds its similarity to each book they rated.
The Harry Potter problem is the trouble with counts. In a library where almost everyone has read Harry Potter, most readers of The Martian read Harry Potter too. The count says “Harry Potter is close to The Martian”, but that is chance, not taste. Larkfield had the same problem with a winter coat in Item-item collaborative filtering.
Here are the five nearest books of The Martian (937 readers in the sample), on the history before the test window:
| Place | By count (shared readers) | By cosine | By text (the content method, below) |
|---|---|---|---|
| 1 | The Hunger Games (The Hunger Games, #1) | Ready Player One | The Martian Chronicles |
| 2 | Gone Girl | Station Eleven | The Egg |
| 3 | Catching Fire (The Hunger Games, #2) | The Girl on the Train | Life of Pi |
| 4 | Mockingjay (The Hunger Games, #3) | All the Light We Cannot See | The Humans |
| 5 | Harry Potter and the Sorcerer's Stone (Harry Potter, #1) | Armada | Airman |
By count, the neighbours of The Martian are the famous books of the whole site. By cosine, they are books that the same kind of reader rates. The readers’ lists show it too:
Text description of the chart
Seven columns, one per method: the share of readers whose list of 10 has at least one Harry Potter book. popular 87%, trending 86%, co-occurrence 83%, item-cf 46%, ALS 15%, content 5%, hybrid 28%.
Co-occurrence gets 0.016 and item-cf 0.024. Cosine is more accurate and shows far fewer famous books (71% of suggestions from the top 100, against 98%). Both are far below trending: they learn from all the old ratings, so they know little about this year’s books.
ALS learns a vector for each reader and each book, as in Hidden tastes. The implicit library fits it, with every rating as one “read” signal (the stars are not used).
ALS has settings. We tried 9 on the validation window, as in Tune without fooling yourself:
Text description of the chart
One line, for regularisation 10, over 32, 64 and 128 factors. All 9 settings: 32 factors, regularisation 0.1: 0.031; 32 factors, regularisation 1: 0.031; 32 factors, regularisation 10: 0.031; 64 factors, regularisation 0.1: 0.035; 64 factors, regularisation 1: 0.034; 64 factors, regularisation 10: 0.034; 128 factors, regularisation 0.1: 0.038; 128 factors, regularisation 1: 0.038; 128 factors, regularisation 10: 0.039.
Bigger models were better. We stopped at 128 factors to keep the nightly job and the notebook fast; a larger model may be a little more accurate.
On the test window, ALS gets 0.034 (0.031 to 0.036). That is better than the neighbours, and below trending: the paired difference ALS minus trending is −0.010 (−0.013 to −0.006).
Content: books with similar texts
Section titled “Content: books with similar texts”A content method compares the texts of the books, as in Content-based recommendations. It needs no other readers, so it can suggest a book that few people have rated.
Each book’s text is its title, its first author and up to 8 tags that describe the book. We drop tags that describe what the reader did, such as to-read, owned or favorites. For example, book 2:
{c.text_examples['2']}The course’s text model is IBM Granite Embedding 97M multilingual R2 (Apache-2.0, as its model card states). It turns each text into a vector of 384 numbers. Texts with similar meaning get vectors that point the same way. A reader’s profile is the mean vector of their last 20 books rated 4 or 5 stars. The score of a book is the cosine between its vector and the profile.
def profiles(period, emb, rows, last=20): """A reader's taste as one vector: the mean embedding of their last `last` books rated 4 or 5 (or of their last books, if they rated none 4 or 5), scaled to length 1.""" P = np.zeros((len(rows), emb.shape[1]), np.float32) for i, r in enumerate(rows): books, order = period.O[r].indices, period.O[r].data # the reader's books and their order (1 = first) liked = period.R[r].toarray().ravel()[books] >= 4 if liked.any(): books, order = books[liked], order[liked] P[i] = emb[books[np.argsort(-order)[:last]]].mean(axis=0) # the newest `last` books return P / np.maximum(np.linalg.norm(P, axis=1, keepdims=True), 1e-9)
def content_scores(period, emb, rows): """Cosine between each book's embedding and the reader's profile.""" return profiles(period, emb, rows) @ emb.TIn the third column of the table of neighbours, the nearest text to The Martian is The Martian Chronicles: the same word in the title. Text similarity sees words, not quality or fashion.
Content gets 0.018, close to co-occurrence and far below ALS (paired difference −0.016 (−0.019 to −0.012)). Its lists explore the shelves: coverage 39.5%, novelty 7.62 bits, and only 3% of its suggestions from the top 100. But it does not know what readers want now.
A hybrid
Section titled “A hybrid”Trending knows the moment, ALS knows each reader’s taste, and content knows books that few people rated. A hybrid adds their scores, as in Hybrids: each score is scaled from 0 to 1 for each reader, multiplied by a weight, and added.
def hybrid_scores(period, model, emb, rows, w_als=1.0, w_content=1.0, w_trending=0.0): """A weighted sum of ALS, content and trending scores, each scaled 0..1 for each reader.""" t = np.tile(np.log1p(period.recent_readers), (len(rows), 1)) return (w_als * scale(als_scores(model, rows)) + w_content * scale(content_scores(period, emb, rows)) + w_trending * scale(t))We chose the weights on the validation window: ALS 1, and content and trending from small grids.
- content 0
- content 0.5
- content 1
Text description of the chart
Three lines, for content weights 0, 0.5 and 1, over trending weights 0, 0.5, 1, 2 and 4. content 0, trending 0: 0.039. content 0, trending 0.5: 0.049. content 0, trending 1: 0.058. content 0, trending 2: 0.061. content 0, trending 4: 0.058. content 0.5, trending 0: 0.042. content 0.5, trending 0.5: 0.054. content 0.5, trending 1: 0.061. content 0.5, trending 2: 0.064. content 0.5, trending 4: 0.059. content 1, trending 0: 0.041. content 1, trending 0.5: 0.055. content 1, trending 1: 0.061. content 1, trending 2: 0.065. content 1, trending 4: 0.061.
On the test window, the hybrid gets 0.057 (0.054 to 0.061).
A fair comparison
Section titled “A fair comparison”Every method ran on the same 5,591 readers of the test window, with the same history and the same measures. This is the case study’s comparison table:
| Method | recall@10 (95% interval) | NDCG@10 | coverage | novelty (bits) | diversity | top 100 | series | fit (s) |
|---|---|---|---|---|---|---|---|---|
popular | 0.015 (0.013 to 0.016) | 0.031 | 0.5% | 1.75 | 0.84 | 100% | 14% | 0.0 |
trending | 0.044 (0.041 to 0.046) | 0.063 | 0.4% | 2.82 | 0.88 | 55% | 11% | 0.0 |
co-occurrence | 0.016 (0.015 to 0.018) | 0.033 | 1.9% | 1.87 | 0.88 | 98% | 16% | 6.2 |
item-cf | 0.024 (0.022 to 0.026) | 0.042 | 21.0% | 2.95 | 0.89 | 71% | 18% | 4.0 |
als | 0.034 (0.031 to 0.036) | 0.052 | 14.6% | 3.87 | 0.91 | 25% | 23% | 27.5 |
content | 0.018 (0.016 to 0.020) | 0.026 | 39.5% | 7.62 | 0.92 | 3% | 7% | 0.0 |
hybrid | 0.057 (0.054 to 0.061) | 0.077 | 4.2% | 3.21 | 0.97 | 43% | 13% | 27.5 |
“Fit” is the training time on one thread (Apple M4) for one reader in four. The content method needs no training: its text vectors are made once, before (see the costs below).
How to read it:
- The hybrid is best on recall and NDCG. Against trending, the paired difference is +0.014 (+0.011 to +0.017). It is better for 822 readers, worse for 295 and the same for 4,474.
- Trending beats every single personal method. ALS minus trending is −0.010 (−0.013 to −0.006). The recent past is a strong signal, as on Larkfield’s test period (trending 0.156).
- Accuracy and novelty pull in opposite directions. The picture below puts the two side by side.
Text description of the chart
A scatter plot with one point per method. popular: novelty 1.75 bits, recall 0.015. trending: 2.82, 0.044. co-occurrence: 1.87, 0.016. item-cf: 2.95, 0.024. ALS: 3.87, 0.034. content: 7.62, 0.018. hybrid: 3.21, 0.057.
The hybrid’s novelty is between trending and ALS. Its coverage is small (4.2%), because trending pulls the same recent books into many lists. But only 28% of its lists have a Harry Potter book, against 86% for trending.
By segment
Section titled “By segment”The average hides the segments. Here is recall@10 by segment, with 95% intervals:
Text description of the chart
Three panels, one per segment, with five methods each. New readers (180): every method shows the trending list and gets 0.022. Light readers (535): trending 0.023, item-cf 0.050, ALS 0.041, content 0.024, hybrid 0.035. Heavy readers (4,876): trending 0.047, item-cf 0.021, ALS 0.033, content 0.017, hybrid 0.061.
- Heavy readers are most of the readers. For them, the hybrid is clearly best: 0.061 against 0.047 for trending (paired difference +0.014 (+0.011 to +0.018)).
- Light readers have 1 to 49 ratings. For them, the personal methods beat trending: ALS minus trending is +0.018 (+0.008 to +0.029). Item-cf is the best for them (0.050; against the hybrid +0.015 (+0.003 to +0.027)). The recommendation comes back to this.
- New readers have no ratings before the window, so every personal method falls back to trending. One surprise:
popularscores higher for them (0.026 against 0.022). New readers often start by rating many books they read years ago. In the test window, a new reader rated 77 books on average, 15% of them among the 100 most read books. A heavy reader rated 9 books, 7% of them in the top 100. Recall rewards a list that “predicts” those ratings, but the reader needs no suggestion for a book they have read. The measure and the goal disagree, so we do not follow the measure.
Failure case: counting makes a Harry Potter list
Section titled “Failure case: counting makes a Harry Potter list”Common mistake
People also read Harry Potter
Symptom: Leo builds “readers of your books also read” with plain counts, because it is the simplest method. Mina opens ten readers’ lists. 83% of all readers get at least one Harry Potter book, and 98% of all suggestions are among the 100 most read books. Recall@10 is only 0.016, far below trending (0.044).
Cause: the Harry Potter problem. A count of shared readers is large for any book that almost everyone read, so the famous books are “near” every book.
Diagnose: look at the nearest books of a few unrelated books, as in the table of The Martian. If the same famous books come back each time, the similarity measures popularity, not taste. Then measure the share of suggestions from the top 100.
Fix: divide by popularity. Cosine (item-cf) halves the Harry Potter lists (46%) and raises recall (0.024; paired difference +0.007 (+0.005 to +0.009)). ALS and content go further. Lift, from People also bought, divides by popularity even more, but it trusts small numbers; we did not measure it here.
The recommendation
Section titled “The recommendation”Text description of the diagram
A reader with no rating sees trending. A reader with at least one rating sees the hybrid list, made every night. A person checks both lists each week.
What we tell Mina and Leo:
-
Readers with history: the hybrid. It adds three scores:
- ALS: 128 factors, regularisation 10, 15 iterations;
- content: Granite text vectors of title, author and tags, and a profile of the last 20 books rated 4 or 5;
- trending: the last 300,000 ratings.
Weights: ALS 1, content 1, trending 2, chosen on the validation window. Books the reader already rated are never shown.
-
New readers: trending. We do not use
popular, although it scores higher for new readers, for the reason in the segment section. -
The fallback is trending, minus the books the reader rated, if the nightly job fails.
-
What a person checks each week:
- the share of lists with the most famous books (Mina’s fear; the hybrid has Harry Potter in 28% of its lists);
- two books of the same series in one list (the example below has book 1 and book 2 of one series);
- books on sensitive topics, as in Privacy and sensitive suggestions.
Two choices for Mina
Section titled “Two choices for Mina”Two variants were measured on the same readers. They are business choices, not mistakes:
- More discovery. With trending weight 1 instead of 2, recall@10 is 0.054 instead of 0.057 (paired difference −0.003 (−0.006 to −0.001)). Coverage rises from 4.2% to 9.4%, and novelty from 3.21 to 3.42 bits. If Mina values discovery most, this is the better setting; an online test can tell which earns more.
- One method per segment. Item-cf for light readers, the hybrid for the others (
switch): recall@10 0.059 against 0.057 (paired difference +0.001 (+0.000 to +0.003)). On the validation window, the two were close for light readers (0.045 against 0.043), so we did not plan a switch. We do not change the design because of the test: try the switch in the next validation round, then in an online test.
What it costs
Section titled “What it costs”Measured on one thread of an Apple M4, for one reader in four (13,099 readers with history). The full shop has four times as many readers. Costs in money are assumptions.
| Item | The hybrid | Trending |
|---|---|---|
| Training each night (ALS) | 27 s (about 4 times more for all readers) | 0 s |
| Text vectors for the 10,000 books | once; then only new books each night | none |
| Lists for all 13,099 readers with history | 12 s | under 1 s |
| Memory (the whole analysis, all methods, one process) | 1.6 GB peak | less |
| Server time, assumption: a 4 vCPU server at US$0.17 an hour, billed for 1 hour a night | US$0.17 a night, about US$62 a year | about the same |
| Leo’s time, assumption | a monthly check of the numbers and the weights, and a fix when the data changes | almost none |
The server is almost free. The real cost is Leo’s time: two models (ALS and the text model) to keep running. The hybrid adds +0.014 recall@10 over trending (+0.011 to +0.017). So the top 10 finds about 32% more of the rated books, with far fewer famous books. Whether that is worth Leo’s time depends on what a found book earns. Offline recall cannot say that.
So our advice: ship trending first, then the hybrid, and test it online against trending before you keep it. Offline wins are not online wins showed why, and Online tests and guardrails showed how.
End to end on new input
Section titled “End to end on new input”The last 5% of the file was never used. We run the whole design once: ALS learns on every rating before 95% of the file, and the hybrid ranks with the same weights. New readers get trending.
| Method | recall@10 (95% interval) | new | light | heavy | coverage |
|---|---|---|---|---|---|
trending | 0.041 (0.037 to 0.044) | 0.013 | 0.027 | 0.042 | 0.3% |
als | 0.029 (0.026 to 0.031) | 0.013 | 0.037 | 0.028 | 15.0% |
content | 0.016 (0.014 to 0.018) | 0.013 | 0.019 | 0.016 | 38.5% |
| the hybrid (the design) | 0.055 (0.051 to 0.058) | 0.013 | 0.033 | 0.057 | 4.4% |
5,568 readers rated books in the last window: 77 new, 294 light and 5,197 heavy. The order is the same as in the test: the hybrid, trending, ALS, then content. Trending has moved on (Me Before You (Me Before You, #1) and The Nightingale are now near the top), and the hybrid’s trending part follows it without retraining.
One reader’s list
Section titled “One reader’s list”Reader 1260 is a light reader: 40 ratings before the last window. Their last five books were *The God of Small Things* (5 stars), *The Glass Palace* (5 stars), *A Fine Balance* (3 stars), *White Teeth* (2 stars), *The Da Vinci Code (Robert Langdon, #2)* (4 stars). In the last window, they rated 117 books. Here is their hybrid list:
| Place | Book | Author | Readers before | Rated in the window |
|---|---|---|---|---|
| 1 | The Girl on the Train | Paula Hawkins | 1,855 | yes |
| 2 | The Nightingale | Kristin Hannah | 713 | yes |
| 3 | The Kite Runner | Khaled Hosseini | 3,171 | |
| 4 | Me Before You (Me Before You, #1) | Jojo Moyes | 1,215 | yes |
| 5 | After You (Me Before You, #2) | Jojo Moyes | 338 | |
| 6 | The Martian | Andy Weir | 1,228 | |
| 7 | The Goldfinch | Donna Tartt | 1,197 | yes |
| 8 | Room | Emma Donoghue | 1,286 | |
| 9 | The Lovely Bones | Alice Sebold | 2,844 | |
| 10 | Fates and Furies | Lauren Groff | 216 |
4 of the 10 were rated in the window, so recall@10 for this reader is 4 of 117 = 0.034. A list of 10 can find at most 10 of the 117 books. The list mixes recent novels from trending with books close to the reader’s literary fiction. Places 4 and 5 are book 1 and book 2 of the same series. A person would keep one and use the other place for something else (point 4 of the recommendation).
Insights and limits
Section titled “Insights and limits”What we learned:
- The recent past is the strongest single signal here. Trending beat every personal method on its own.
- A hybrid can be more accurate and less boring at once. It beat trending by +0.014. It showed Harry Potter to 28% of readers, against 86% for trending.
- Counting has a Harry Potter problem. Use cosine or a model (ALS) before you show “readers also read”.
- Segments need different methods. Light readers were served best by item-cf; heavy readers by the hybrid.
Against Larkfield’s synthetic shop: this data is easier for personal methods because every reader has a history. It is harder because it has no dates, popularity is extreme, and only the most rated books and readers were kept. Compare the order of the methods, not the values. On Larkfield’s test period, trending (0.156) also beat every single personal method. A tuned hybrid blend (0.193) also beat trending there, in every segment.
Limits:
- No dates. We do not know how long a window is. A real shop would split by date.
- Selected data. The 10,000 most rated books, and readers with 19 to 200 ratings. A real shop has many more rare books and many readers with one rating.
- A rating is not a sale. We predict ratings, which is close to, but not the same as, what the shop wants.
- Books with no rating before the window are never shown. They are 2.4% of the test-window ratings. Content could show them on day one, like Larkfield’s new products in Cold start for new products and new customers. We did not measure that.
- Offline only. Recall does not say whether the list caused a rating. Only an online test can measure what the block adds.
Responsible use
Section titled “Responsible use”- The readers are real. Do not try to identify a reader, and do not join this data with Goodreads profiles or other data.
- Reading is personal. Books can reveal health, religion, politics or sexuality. A list on a shared screen can expose them. Follow the checks of Privacy and sensitive suggestions.
- A recommender decides what people see. The hybrid’s coverage is small: most of the 10,000 books are never shown. Watch coverage, and give rare and new books a place on purpose.
- Say what is measured. Every number here is offline, on past ratings. Do not promise Mina more sales.
- Credit the data, and share alike. Give the attribution above and use the same licence for any shared copy. Link the source instead of copying the files.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can compare methods on more than accuracy. You explain a trade-off to the person who decides, and you handle a dataset without dates honestly.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
DECISION.md: the recommendation, the settings, the two choices for Mina, the costs with their assumptions, and the weekly checks.- The data source, the licence and the attribution. Do not commit the data, because of ShareAlike and size: link the source and give the SHA-256 of each file.
A README template:
# Book suggestions for readers
## QuestionWhat should a bookshop show in a block of 10 books, for new and returningreaders, without showing the same famous books to everybody?
## Data and licencegoodbooks-10k (Zygmunt Zajac, github.com/zygmuntz/goodbooks-10k), CC BY-SA 4.0.5,976,479 ratings, 53,424 readers, 10,000 books, with tags. Not copied:downloaded from the source and checked with SHA-256. One reader in four.
## ApproachNo dates: split by position in ratings.csv (sorted by time). Choose on 85-90%,test once on 90-95%, run the chosen design once on 95-100%. Methods: popular,trending, co-occurrence, item-cf, ALS (implicit), content (Granite embeddingsof title, author, tags), a hybrid of ALS + content + trending.Measures: recall@10 with bootstrap intervals, NDCG@10, coverage, novelty,diversity, share from the top 100 books, series share, by segment.
## Results(the comparison table, the recall-novelty chart and the segment chart)
## DecisionHybrid for readers with history, trending for new readers, trending as thefallback. Weekly checks: famous books, series, sensitive topics.
## LimitsNo dates, selected books and readers, ratings not sales, offline only.How to describe it in three sentences: “I compared seven recommendation methods on six million real book ratings, with a split by time and reader segments. A hybrid of ALS, text embeddings and trending was the most accurate and showed far fewer famous books than popularity. I explained the trade-off between accuracy and discovery to the person who decides.”
Go further:
- Try the switch (item-cf for light readers) in a new validation round, on another part of the file.
- Add a series rule: at most one book of a series in a list. Measure what it costs in recall.
- Let content show books with no ratings yet, in one place of the list, and measure recall on those books alone.
Knowledge check
Section titled “Knowledge check”Check your understanding
By count, the nearest books of The Martian are The Hunger Games and Harry Potter. Why, and how do you fix it?
Show the answer
A count of shared readers is large for any book that almost everyone read. So famous books are “near” every book by chance: the Harry Potter problem. Divide the count by the popularity of the books. Cosine (item-cf) does that: it halves the lists with a Harry Potter book (83% to 46%), and recall rises. ALS and the hybrid go further.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The content method has the highest novelty and coverage. Why is it not the recommended design?
Show the answer
Novelty and coverage say the lists are unusual, not that readers want the books. Content’s recall@10 is 0.018, far below ALS (0.034) and trending (0.044): text similarity sees words, not quality or what is popular now. It is useful inside the hybrid, the most accurate method.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On the test window, item-cf is better than the hybrid for light readers. Should you switch light readers to item-cf now?
Show the answer
No, not because of the test. The design was chosen on the validation window, where the two were close. If we change it after looking at the test, the test number no longer shows what the design does on new data. Try the switch in the next validation round, then in an online test.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- You can split data without dates honestly when the file is in time order: here, by position in the file.
- Trending (0.044) beat every single personal method; ALS (0.034), item-cf (0.024) and content (0.018) were below it.
- Counting has a Harry Potter problem: co-occurrence put a Harry Potter book in 83% of the lists. Cosine, ALS and the hybrid show far fewer famous books.
- The hybrid of ALS, content and trending was the most accurate (0.057) and showed far fewer famous books than trending. Content alone had the most novelty and low accuracy.
- The design: trending for new readers, the hybrid for readers with history, trending as the fallback, and a weekly check by a person. Mina chooses between accuracy and discovery; an online test decides.
Next comes an optional case study. We suggest the next lesson on a learning platform, where the order of lessons and what each one needs first matter more than popularity.
Next: What to learn next