What to learn next
- Case study 4
- 120 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, the order of purchases mattered a little. In this optional case study, we suggest the next lesson on a learning platform, where order and prerequisites matter more than popularity.
Case study 4 of 4 · Recommendation and Personalisation: Show Each Person What MattersWhat to learn next
1. The recommendation problem
2. Data, implicit feedback and an honest split
3. Measure recommendations offline
4. Neighbours and collaborative filtering
5. Matrix factorisation for implicit feedback
6. Content, hybrids and cold start
7. Two stages: candidates and ranking
8. From offline scores to the real shop
9. Run it in production
What you will be able to do
Suggest the next lesson to each learner of a learning platform, from synthetic progress data that we made for this case. You split the data by time. You compare popularity, co-occurrence of completions, a sequence method (Markov) and a rule that knows the prerequisites, on the same learners. You measure how often the real next lesson is among 3 suggestions. You also measure how often a suggestion is a lesson the learner is not ready for. Then you decide what the block should show, what it costs and what a person still checks.
This case study is optional. It is a notebook, and it does not use the larkfield-recs project. The notebook has the code blocks of this page, in the same order, with two tasks and their checks. Its setup cells install the packages and download the data (2 MB), so it needs no local setup. After the packages are installed, it runs in less than a minute on a laptop CPU. Read the explanations here, and run the code there.
Before you startWhat to know and to have ready
What you need to know
- The time split, from Split by time, and the baselines, from The baselines to beat.
- Recall@k and NDCG@k, from Ranking metrics; coverage, novelty and diversity, from Beyond accuracy; segments and paired differences, from Evaluate by segment.
- Co-occurrence and cosine, from People also bought and Item-item collaborative filtering.
- The Markov model, from Sessions, sequences and LLMs, briefly; rules after the ranker, from Business rules and re-ranking.
What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)
- Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas and matplotlib.
- A new, empty folder. The setup cell downloads the data into a folder
learn-next. To start again, delete it and run the notebook from the top.
Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4 and pandas 3.1.1. Windows, Linux, Colab and Kaggle were not run.
The situation
Section titled “The situation”Think of a good music teacher. A new pupil asks: “What should I play next?” The teacher does not hand them the most-played piece in the school. That piece may need skills the pupil does not have yet. The teacher knows two things: what the pupil has already learned, and which pieces need which skills. So the next piece is one the pupil is ready for.
A learning platform has the same problem with thousands of learners. Ridgeway Learning is an online learning platform that we invented for this case. It has 12 courses and 84 lessons, for data, machine learning and web development. After each lesson, a learner’s home page shows an Up next block with 3 lessons. Today the block shows the most popular lessons.
Nadia is the head of product. She asks: “Can the block show the lesson each learner will really take next? And can it never show a lesson they are not ready for?”
Kofi is the engineer who will build it. His questions: “What must we compute each night? How fast is one list?”
Nadia, Kofi and Ridgeway Learning are made up. So is the data.
What you will build
Section titled “What you will build”A comparison of seven lists on the same learners and the same two weeks. They come from four families, alone and combined: popularity, neighbours (co-occurrence of completions), a sequence method (Markov) and a prerequisite rule. The result is a recommendation for Nadia and Kofi: a design, its settings, its cost, and what a person still checks. Then you run the design once on two weeks that no method has seen.
The data
Section titled “The data”This data is synthetic. Our code made the catalogue, the learners and every completion, with a model of how learners behave. No real learner is in it. It does not come from Nextia Learning or from any other platform.
| Source | Made for this case study with make_data.py, in the labs folder of the notebook |
| Licence | CC0 1.0 (public domain dedication): use it as you like |
| Files | lessons.csv (the catalogue) and completions.csv.gz (1.7 MB), with a dataset card and their SHA-256 checksums |
| Size | 84 lessons in 12 courses; 259,760 completions by 24,000 learners, from 6 January 2025 to 15 February 2026 |
What one row means. In lessons.csv, one row is one lesson. It has an ID (for example PD-03, lesson 3 of Data analysis with pandas), a course, a position and a title. It also says whether the lesson is optional, and it lists the prerequisites. In completions.csv.gz, one row is one lesson that one learner completed, with the date and time.
Why synthetic data
Section titled “Why synthetic data”We looked for open progress data first. The Open University Learning Analytics Dataset is CC BY 4.0. It has seven modules and a planned week for each material, but it describes no prerequisites. It also has sensitive data (disability, deprivation) that this case does not need. EdNet is CC BY-NC 4.0: no commercial use, so not allowed. And we never use the real progress data of Nextia Learning’s own learners.
How the data was made
Section titled “How the data was made”The generator’s documentation lists every rule. In short:
- Each learner picks a goal when they join: data analyst, machine-learning engineer, backend developer, AI app builder, or none (a “browser”). A goal is a list of courses, in order.
- Learners take the lessons of a course in order. They take an optional lesson 45% of the time.
- 30% of the learners with a goal take two courses at a time.
- When they start a course, 7% jump ahead to a later course before its prerequisites are done. 10% take a detour to a popular course outside their goal. Browsers ignore prerequisites.
- After each lesson, some learners stop for good. Half of the learners completed 8 lessons or fewer; 17% completed only one.
How this data differs from Larkfield
Section titled “How this data differs from Larkfield”Both are synthetic, and both were made by our code. Larkfield’s shop log was made to look like a shop. This data was made to look like a learning platform. Three differences matter:
- Order is built in. Most learners take lessons in order, because we wrote the model that way. At Larkfield, the order of purchases mattered much less.
- The catalogue is tiny. 84 lessons, against Larkfield’s 2,994 products.
- There are rules. Prerequisites say which lessons a learner can take. Larkfield had business rules, but no product needed another one first.
So this case is easier than a real platform. The methods can only find what our behaviour model put in. Read the numbers as a test of the methods on known behaviour, not as a measurement of real learners.
The new idea: prerequisites and ready lessons
Section titled “The new idea: prerequisites and ready lessons”A prerequisite is a lesson you must complete before another one. Inside each course here, a lesson needs the lesson before it. Optional lessons are the exception: no lesson needs them. The first lesson of 8 courses needs lessons of other courses:
Text description of the diagram
Twelve courses. Python basics, How the web works, Git and the command line, and SQL for analysis need nothing. Statistics needs Python lesson 4. Data analysis with pandas needs Python lesson 8. Charts and dashboards needs pandas lesson 4. Machine learning needs pandas lesson 8 and Statistics lesson 5. Deep learning needs Machine learning lesson 6. Build APIs with Python needs Python lesson 8 and web lesson 4. Deploy to the cloud needs APIs lesson 5 and Git lesson 3. Build apps with language models needs APIs lesson 4.
A lesson is ready for a learner when they have not done it, and they have done all its prerequisites. Here is a tiny example: a learner who completed the first four lessons of Python basics.
| Lesson | Needs | Ready? |
|---|---|---|
PY-05 Dictionaries |
PY-04 |
yes |
STA-01 Averages and spread |
PY-04 |
yes |
SQL-01, WEB-01, GIT-01 |
nothing | yes |
PY-06 Files |
PY-05 |
no: PY-05 is not done |
PD-01 DataFrames |
PY-08 |
no |
A suggestion breaks a prerequisite when it is not ready. The break rate is the share of suggestions that break a prerequisite. It is a new measure for this case. We want it near 0.
Frame the problem
Section titled “Frame the problem”- When is the list made? When a learner opens their home page, from the lessons they have completed. Counts are rebuilt every night.
- What do we predict? The next lesson the learner completes.
- For whom? Every learner who completes a lesson in the test period, including new learners.
The measures
Section titled “The measures”- Recall@3: is the real next lesson among the 3 suggestions? There is one right answer per learner, so recall@3 is 1 or 0 for each learner. We average it over learners, with a 95% interval from the bootstrap.
- NDCG@3: the same, but a right answer in place 1 counts more than in place 3.
- Break rate: the share of the suggestions that break a prerequisite.
- Coverage, novelty and diversity, as in the harness. Diversity counts pairs of suggestions from different courses.
The segments are the learner’s state when the period starts. New: no lesson yet. In a course: they started a course and did not finish it. Between courses: every course they started is finished.
The time split
Section titled “The time split”Text description of the diagram
The history runs from 6 January 2025. Validation is 5 to 18 January 2026: we choose the design. The test is 19 January to 1 February: every method runs once with everything known before 19 January, and we report these numbers. The new input is 2 to 15 February: only the chosen design runs, once.
The test period has 4,963 learners: 1,053 new, 3,628 in a course and 282 between courses.
Explore what the decision needs
Section titled “Explore what the decision needs”One question decides which methods can work: what do learners do next?
Text description of the chart
Four columns, out of 4,963 learners. The next lesson was in the same course as the last lesson for 63.6%. It was a new learner’s first lesson for 21.2%. It was the first lesson of a new course for 10.6%. It was a lesson of another course they had already started for 4.6%.
Two facts follow:
- Most next lessons continue the last course. A method that knows the order should do well. A method that ignores order will guess lessons that are too far ahead.
- The hard cases are the changes: a new learner’s first lesson, and the next course after one ends. There, the learner’s goal matters, and we do not know it.
One more fact: 94.5% of the real next lessons were ready. The other 5.5% broke a prerequisite: some of our learners jump ahead. A method that never breaks a prerequisite can never find those. The notebook’s first task computes this share.
The baseline
Section titled “The baseline”popular ranks lessons by the number of learners who completed them in the 28 days before the period. Each learner gets the most popular lessons that they have not done. It is the “trending” list of The baselines to beat.
def popular(split, days=28): """Lessons ranked by the number of learners who completed them in the last `days` before the period.""" h = split.history recent = h[h["completed_at"] >= split.start - pd.Timedelta(days=days)] return np.bincount(recent["item"], minlength=split.cat.n).astype(float)On the test period, popular gets recall@3 0.415 (0.400 to 0.428), with a break rate of 58.5%. The most popular lessons are PY-01, PY-02, PY-03: the start of Python basics, which most goals begin with. So a new learner sees PY-01, PY-02, PY-03. They can take PY-01, but not the other two. At Larkfield, trending (0.156) beat every single personal method. Here, popularity ignores the one thing that matters most: where the learner is.
The alternatives
Section titled “The alternatives”Co-occurrence of completions
Section titled “Co-occurrence of completions”“Learners who completed this lesson also completed that one.” This is item-item collaborative filtering from Item-item collaborative filtering. For each pair of lessons, count the learners who completed both. Divide by the square root of the product of the two lessons’ counts: this is the cosine. A learner’s score for a lesson is the sum of its cosines with the lessons they completed.
def cooccurrence(split): """Item-item cosine between lessons: learners who completed both, divided by the square root of the product of the two counts (as in item-item collaborative filtering). The diagonal is 0.""" h = split.history.drop_duplicates(["learner", "item"]) users = {l: r for r, l in enumerate(h["learner"].unique())} m = np.zeros((len(users), split.cat.n)) m[h["learner"].map(users).to_numpy(), h["item"].to_numpy()] = 1 both = m.T @ m n = np.diag(both).copy() sim = both / np.sqrt(np.outer(n, n) + 1e-9) np.fill_diagonal(sim, 0) return simOn the test period, co-occurrence gets 0.726, with a break rate of 52.3%. It knows which lessons belong together. It does not know which one comes first. The failure case below shows what that means.
Markov: what learners took next
Section titled “Markov: what learners took next”Sessions, sequences and LLMs, briefly taught the Markov model. For each lesson, count which lesson the same learner completed directly after it. Suggest the lessons that most often followed the learner’s last lesson.
def transitions(split): """Markov: how often lesson b was the next lesson completed after lesson a, by the same learner.""" h = split.history.sort_values(["learner", "completed_at"], kind="stable") a, b = h["item"].to_numpy()[:-1], h["item"].to_numpy()[1:] same = h["learner"].to_numpy()[:-1] == h["learner"].to_numpy()[1:] T = np.zeros((split.cat.n, split.cat.n)) np.add.at(T, (a[same], b[same]), 1) return TMarkov gets 0.853: much better than co-occurrence (+0.126 (+0.114 to +0.138) paired difference). It knows the order. But its break rate is still 44.4%. After a lesson, one next lesson has a large count. The second and third places have small counts: a few learners who skipped a lesson or jumped ahead. Those places are often lessons the learner is not ready for. New learners have no last lesson, so Markov shows them popular lessons.
The prerequisite rule
Section titled “The prerequisite rule”The music teacher’s method: show only ready lessons, in a sensible order.
- The next ready lessons of the courses in progress, the most recent course first.
- The first lessons of new courses that are ready, the most popular first.
- Any other ready lesson (an optional lesson left behind), the most popular first.
If fewer than 3 lessons are ready, popular lessons fill the free places. The rule needs no training: only the catalogue and popularity.
def rule_scores(order, cat, rank): """The prerequisite-aware rule for one learner (`order`: the lessons done, oldest first). Only lessons whose prerequisites are done get a score: first the next lessons of the courses in progress (the most recent course first, then by position), then the first lessons of new courses (by `rank`), then the rest (optional lessons left behind), by `rank`. Other lessons get -inf.""" done = np.zeros((1, cat.n), bool) done[0, list(order)] = True ok = eligible(done, cat)[0] s = np.where(ok, rank, -np.inf) progress = in_progress(list(order), cat) for j in np.flatnonzero(ok): c = cat.course[j] if c in progress: s[j] = 100 - 10 * progress.index(c) - cat.position[j] / 100 elif j == cat.first[c]: s[j] = 2 + rank[j] return sThe rule gets 0.910, with a break rate of 0.2%. It beats Markov by +0.057 (+0.047 to +0.067). It also handles a learner who takes two courses at a time. Both courses are in progress, so both next lessons are in the list.
Combine them
Section titled “Combine them”The rule knows what a learner can take. Markov knows what learners did take. They can work together in two ways:
- Markov + prerequisites: Markov’s list, with every lesson that is not ready removed; the rule fills the free places.
- Rule + Markov: the rule’s order, but the new courses are ordered by Markov from the learner’s last lesson (popularity only breaks ties). After SQL for analysis, data analysts usually go on to Statistics: Markov has counted that.
We also tried co-occurrence + prerequisites. We chose the design on the validation weeks. The four designs got this recall@3:
| Design | Validation recall@3 |
|---|---|
| prerequisite rule | 0.896 |
| co-occurrence + prerequisites | 0.843 |
| Markov + prerequisites | 0.892 |
| rule + Markov | 0.906 |
We chose rule + Markov. The notebook’s second task builds it.
One honest note about this choice. Our first trial printed the numbers of the validation weeks and of the test period. The idea of rule + Markov came from that trial. So the test period is not fully clean for this one design. The validation weeks show the same pattern. The last two weeks, in End to end on new input, were not in that trial, so they are the cleaner check.
A fair comparison
Section titled “A fair comparison”Every method ran on the same 4,963 learners, with the same history and the same measures:
| Method | recall@3 (95% interval) | NDCG@3 | break rate | coverage | novelty | diversity |
|---|---|---|---|---|---|---|
| popular | 0.415 (0.400 to 0.428) | 0.407 | 58.5% | 30% | 1.23 | 0.15 |
| co-occurrence | 0.726 (0.714 to 0.740) | 0.666 | 52.3% | 99% | 2.08 | 0.29 |
| Markov | 0.853 (0.843 to 0.863) | 0.817 | 44.4% | 100% | 2.17 | 0.60 |
| prerequisite rule | 0.910 (0.901 to 0.919) | 0.840 | 0.2% | 99% | 2.06 | 0.96 |
| co-occurrence + prerequisites | 0.851 (0.839 to 0.861) | 0.743 | 0.2% | 100% | 2.13 | 0.97 |
| Markov + prerequisites | 0.906 (0.897 to 0.914) | 0.850 | 0.2% | 100% | 2.09 | 0.98 |
| rule + Markov | 0.919 (0.912 to 0.927) | 0.852 | 0.2% | 100% | 2.16 | 0.96 |
Coverage is the share of the 84 lessons that appear in at least one list. Novelty is in bits, as in the harness: a lesson that many learners completed has a low novelty.
Text description of the chart
Five columns. Popular: 58.5% of suggestions not ready. Co-occurrence: 52.3%. Markov: 44.4%. The prerequisite rule: 0.2%. Rule + Markov: 0.2%.
How to read it:
- Rule + Markov is best on recall@3 and NDCG@3. Against the rule alone, the paired difference is small but clear: +0.009 (+0.005 to +0.013). Against Markov alone: +0.066 (+0.057 to +0.075).
- Order matters more than co-occurrence. Markov beats co-occurrence by +0.126 (+0.114 to +0.138).
- Prerequisites matter most. The break rate falls from 44.4% (Markov) to 0.2%. The few breaks left come from the popular fill, when a learner has fewer than 3 ready lessons.
- Popularity is far behind, and over half of its suggestions are not ready.
By segment
Section titled “By segment”Text description of the chart
Three panels with five methods each. New learners: popular 0.622, co-occurrence 0.622, Markov 0.622, rule 0.846, rule + Markov 0.846. In a course: popular 0.368, co-occurrence 0.780, Markov 0.916, rule 0.938, rule + Markov 0.940. Between courses: popular 0.241, co-occurrence 0.426, Markov 0.894, rule 0.794, rule + Markov 0.926.
- New learners. Popularity, co-occurrence and Markov all show popular lessons, because the learner has no history: 0.622. The rule shows the first lessons of the courses that need nothing: 0.846.
- In a course. The next lesson is almost always the next one in the course. The rule and the design are almost equal here.
- Between courses. The rule alone orders new courses by popularity: 0.794. With Markov’s order, the design gets 0.926. This segment is small (282 learners), but it is where a learner may leave the platform.
Failure case: co-occurrence ignores order
Section titled “Failure case: co-occurrence ignores order”Common mistake
'Learners who completed this also completed that'
Symptom: a learner has just completed PD-03 (Clean data). Co-occurrence suggests PD-04 (ready), PD-05 (not ready), SQL-02 (not ready). Two of the three cards are lessons the learner cannot take yet.
Cause: learners who completed PD-03 also completed PD-05, but after PD-04. Co-occurrence counts the pair and forgets the order. It also suggests SQL-02: data analysts take both courses, so the two lessons are completed by the same people. But this learner has not started SQL. Over all learners in a course, 53% of co-occurrence’s suggestions from the learner’s own course were not ready.
Diagnose: for each method, measure the break rate next to recall. Then look at a few lists of real learners, with each lesson marked ready or not ready. The notebook prints this learner’s lists.
Fix: only show ready lessons. For this learner, rule + Markov suggests PD-04 (ready), SQL-01 (ready), GIT-01 (ready). The learner really completed {fl.next} next.
The same mistake happens with any method that does not know the rules of the catalogue. Markov made it less often, because order is part of its counts. But it still suggested PD-05 to this learner: PD-04 (ready), PD-05 (not ready), SQL-01 (ready).
The recommendation
Section titled “The recommendation”Text description of the diagram
A learner opens their home page. The service reads the lessons they completed, in order. It fills 3 places. First the next lessons of the courses in progress, the most recent course first. Then the first lessons of new courses whose prerequisites are done, ordered by what learners took next. Popular lessons fill the places only if fewer than 3 lessons are ready. A person checks the catalogue and the weekly numbers.
What we tell Nadia and Kofi:
- The Up next block: rule + Markov, 3 places. The next lessons of the courses in progress first. Then new courses whose prerequisites are done, ordered by Markov counts from the last lesson, with popularity to break ties.
- The fallback is the rule alone. If the Markov counts are missing, the rule orders new courses by popularity. It loses only 0.009 of recall@3 on the test period.
- Not now: co-occurrence. It was the weakest of the personal methods, and it suggested lessons out of order.
- What a person checks: the prerequisites in the catalogue are the design’s most important input. A wrong prerequisite blocks a lesson for everyone, and a missing one lets learners in too early. A course author checks them when a course is added or changed. A new course has no Markov counts, so it is ordered by popularity until learners take it. Each week, a person reviews recall@3 by segment and the break rate.
What it costs
Section titled “What it costs”Measured on one thread of an Apple M4. Costs in money are assumptions.
| Item | Rule + Markov | Popular |
|---|---|---|
| Nightly build | Markov counts: 26 ms | 1 ms |
| Table size | 1,934 pairs of lessons, 28 KB | 84 counts |
| One list, when the page opens | 0.2 ms (median), 0.2 ms (95th percentile), in Python | a copy |
| Lists for every learner of the period at once | 0.7 s for 4,963 learners (a Python loop) | less than a second |
| Server, assumption: the job runs on a server the platform already has | about US$0 extra | about US$0 extra |
| Engineering, assumption | prerequisites kept correct in the catalogue, a nightly job, a weekly check | almost none |
The real cost is not the computer. It is the catalogue: every prerequisite must be right, and every new course needs them before it goes live. That is work for course authors, not for engineers. In return, learners stop seeing lessons they are not ready for: the break rate falls from 58.5% to 0.2%.
Recall is offline. It says whether the learner took a suggested lesson, not whether the block caused it. Most learners would take the next lesson of their course anyway. So ship the design and test it online against the popular block, as in Online tests and guardrails. A good online measure is the share of learners who start a new course after finishing one.
End to end on new input
Section titled “End to end on new input”The two weeks from 2 to 15 February 2026 were not used to build or choose anything. We run the whole design once. Popularity and the Markov counts come from every completion before 2 February. Then we make the lists for the learners of those two weeks.
| Method | recall@3 (95% interval) | break rate |
|---|---|---|
| popular | 0.412 (0.400 to 0.425) | 57.6% |
| Markov | 0.845 (0.835 to 0.854) | 44.9% |
| prerequisite rule | 0.905 (0.897 to 0.914) | 0.3% |
| rule + Markov | 0.917 (0.909 to 0.925) | 0.3% |
5,000 learners completed a lesson in those two weeks. The order is the same as in the test period: the design first, the rule second, popular far behind.
One learner, as the service sees it
Section titled “One learner, as the service sees it”A service makes the list for one learner at a time, when the page opens. The notebook’s suggest_one does that, and gives the same list as the batch code. We show the first learner (by ID) who is between courses and has completed at least 8 lessons. We chose the rule before we looked at the list.
Learner {ex.learner} completed 19 lessons: How the web works (without the optional lesson), Python basics (without the optional lesson) and SQL for analysis. The last one, {ex.done[ex.done.length - 1]}, was on 2026-01-27. Every course they started is finished.
Ready lessons: GIT-01, PY-07, STA-01, PD-01, WEB-05, API-01. Among those, what did learners complete right after {ex.done[ex.done.length - 1]}? Markov counts: STA-01 857, PY-01 261, WEB-01 243, GIT-01 128, PD-01 109. (PY-01 and WEB-01 are done, so they do not count.)
| List | Lessons |
|---|---|
| Rule + Markov | STA-01, GIT-01, PD-01 |
| Popular | STA-01, STA-02, PY-07 |
| Markov | STA-01, GIT-01, PD-01 |
The learner really completed STA-01 (Statistics with Python, lesson 1) next. The design had it in place 1. The popular list also had it, but it showed STA-02, which needs STA-01 first. Markov’s list is the same as the design’s here, because its top lessons were ready.
Insights and limits
Section titled “Insights and limits”What we learned:
- Know the rules before you learn from the data. A list that respects prerequisites was the biggest gain, and it needed no model.
- Order beats togetherness. Markov (order) beat co-occurrence (togetherness) by a wide margin, because lessons have an order.
- Each family has its own job. The rule says what a learner can take. The sequence says what they will probably take. Popularity only fills gaps.
- Measure what must never happen. Recall alone would have hidden that Markov showed 44.4% lessons the learner was not ready for.
Against Larkfield’s shop log: at Larkfield, trending was hard to beat, and order mattered little. Here, popularity was far behind, and order and prerequisites decided almost everything. Both datasets are synthetic, so both results follow from how we built them. What carries over to a real platform is the method: measure a break rate next to recall, and look at each segment.
Limits:
- Synthetic data. The methods found the behaviour that we put in the model. The rule looks strong because most of our learners follow the prerequisites. Real learners repeat lessons, come back after months and arrive from search.
- The rule cannot follow learners who jump ahead. 5.5% of the real next lessons broke a prerequisite. The design never found those; Markov found 21% of them. If many real learners jump ahead, they may want a “skip ahead” choice, not a hidden lesson.
- We do not know the learner’s goal. It is in our generator but not in the data, as on most platforms. Asking learners for a goal might help most between courses.
- One test period, offline only. Two weeks in January. Recall says what learners did, not what the block made them do.
Responsible use
Section titled “Responsible use”- Suggestions steer people. A learning platform shapes careers, not only clicks. Show why a lesson is suggested (“next in Data analysis with pandas”), and let learners choose another one.
- Prerequisites are advice, not walls. The design never suggests a lesson that is not ready. That does not mean the platform should lock it. An experienced learner may know the material already.
- Do not judge learners by their speed. Progress data shows who stopped and who was slow. Do not use it to rank learners or to show them to others.
- Progress data is personal. A real platform’s completions belong to its learners. Use them for the learner’s own suggestions, and say so in the privacy notice. This case uses synthetic data only. It never used Nextia Learning’s real learners.
- Say what is measured. Every number here is offline, on synthetic data. Do not promise Nadia more finished courses.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows an employer that you can turn a business rule into a measure (the break rate). It also shows that you can combine a rule with a learned order, instead of choosing one.
What to put in the repository:
README.md(template below).- The notebook, run from top to bottom, with its outputs.
DECISION.md: the design, its settings, the costs with their assumptions, and what a person checks.- The data’s source and licence (CC0), with the SHA-256 checksums, and a sentence that the data is synthetic. You may commit the two files: CC0 allows it.
A README template:
# What to learn next: suggestions for a learning platform
## QuestionWhat should an "Up next" block show after each lesson, so that learnerssee the lesson they will take next, and never a lesson they are not ready for?
## Data and licenceSynthetic progress data (CC0 1.0), made by a documented behaviour model:12 courses, 84 lessons with prerequisites, 259,760 completions by24,000 learners. No real learners.
## ApproachTime split: choose on two validation weeks, test once on the next two,run the chosen design once on the last two. Methods: popular (28 days),co-occurrence (item-item cosine), Markov (the next lesson after the last),a prerequisite rule, and combinations. Measures: next-lesson recall@3 withbootstrap intervals, NDCG@3, the break rate (suggestions not ready),coverage, novelty, diversity, by segment (new, in a course, between courses).
## Results(the comparison table, the break-rate chart and the segment chart)
## DecisionThe prerequisite rule, with new courses ordered by Markov.Popular lessons only when fewer than 3 lessons are ready.
## LimitsSynthetic data, one test period, offline only, the learner's goal unknown.How to describe it in three sentences: “I compared popularity, co-occurrence, a Markov sequence model and a prerequisite rule for next-lesson suggestions, with a time split. I added a new measure for suggestions the learner is not ready for. A rule that only shows ready lessons, with new courses ordered by Markov, found the next lesson most often and almost never broke a prerequisite.”
Go further:
- Change
make_data.pyso that 30% of learners jump ahead, not 7%. Make the data again and run the notebook. Does the design still win? What happens to the rule between courses? - Give the design a “skip ahead” place: the third card may show Markov’s top lesson even if it is not ready, with a label. Measure recall@3 and the break rate.
- Add the learner’s goal to the data (the generator knows it). Order new courses by the goal. How much does it help between courses?
Knowledge check
Section titled “Knowledge check”Check your understanding
Markov finds the next lesson more often than co-occurrence. Why, and why is Markov still not enough on its own?
Show the answer
Markov counts which lesson came directly after the learner’s last lesson, so it knows the order. Co-occurrence only counts lessons completed by the same learners, so it also suggests lessons far ahead in the course. Markov gets 0.853 against 0.726. But Markov’s second and third places come from small counts, often learners who skipped or jumped ahead. So 44.4% of its suggestions are lessons the learner is not ready for. Only the prerequisites can remove those.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The design never shows a lesson whose prerequisites are not done. What does that cost, in this data?
Show the answer
In the test period, 5.5% of learners really completed a lesson whose prerequisites were not done: they jumped ahead. The design can never suggest that lesson, so it misses every one of them. Markov found 21% of them. On the whole test period the design is still clearly better, but the cost is real. A platform where many learners jump ahead might add a labelled “skip ahead” card instead of hiding those lessons.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The rule and the design are almost equal for learners in a course. Where does Markov's order help, and why there?
Show the answer
Between courses. In a course, the next lesson is almost always the next one in the course, and the rule already puts it first. When a course ends, the rule must choose among new courses, and alone it orders them by popularity: 0.794. Markov has counted which course learners started after that last lesson. For example, after SQL for analysis, learners usually started Statistics with Python. With that order, the design gets 0.926.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- When items have an order and prerequisites, popularity is a weak baseline:
populargets recall@3 0.415, and 58.5% of its suggestions are not ready. - Markov (0.853) beats co-occurrence (0.726) because it knows the order. But 44.4% of its suggestions still break a prerequisite.
- Measure what must never happen next to recall: here, the break rate.
- The design: the prerequisite rule, with new courses ordered by Markov (0.919, break rate 0.2%). The fallback is the rule alone.
- The real cost is a correct catalogue of prerequisites, not the computer. Test the block online before you keep it.
- The data is synthetic, so the methods only find the behaviour we put in. Use the method, not the numbers.
Next comes the final assignment: you build and defend a recommender for a new Larkfield category, on data that no lesson has used.
Next: Final assignment