Skip to content
Français

What to learn next

  • Case study 4
  • 120 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, the order of purchases mattered a little. In this optional case study, we suggest the next lesson on a learning platform, where order and prerequisites matter more than popularity.

What you will be able to do

Suggest the next lesson to each learner of a learning platform, from synthetic progress data that we made for this case. You split the data by time. You compare popularity, co-occurrence of completions, a sequence method (Markov) and a rule that knows the prerequisites, on the same learners. You measure how often the real next lesson is among 3 suggestions. You also measure how often a suggestion is a lesson the learner is not ready for. Then you decide what the block should show, what it costs and what a person still checks.

Case studyPortfolio pieceSynthetic data

Licence
CC0 1.0
Time
About 2 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Prerequisites
  • Time split
  • Popularity baseline
  • Co-occurrence
  • Markov transitions
  • A prerequisite rule
  • Recall@3 and a break rate
  • Fair comparison by segment
  • A design decision with costs

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is optional. It is a notebook, and it does not use the larkfield-recs project. The notebook has the code blocks of this page, in the same order, with two tasks and their checks. Its setup cells install the packages and download the data (2 MB), so it needs no local setup. After the packages are installed, it runs in less than a minute on a laptop CPU. Read the explanations here, and run the code there.

Before you startWhat to know and to have readyAbout 120 minutes

What you need to know

What you need on your computer (only for the local route; the notebook in Colab or Kaggle needs none of it)

  • Python 3.12 in a new virtual environment, with Jupyter. The notebook’s first cell installs NumPy, pandas and matplotlib.
  • A new, empty folder. The setup cell downloads the data into a folder learn-next. To start again, delete it and run the notebook from the top.

Tested on macOS (Apple M4, 16 GB) in Jupyter, with Python 3.12, NumPy 2.5.4 and pandas 3.1.1. Windows, Linux, Colab and Kaggle were not run.

Think of a good music teacher. A new pupil asks: “What should I play next?” The teacher does not hand them the most-played piece in the school. That piece may need skills the pupil does not have yet. The teacher knows two things: what the pupil has already learned, and which pieces need which skills. So the next piece is one the pupil is ready for.

A learning platform has the same problem with thousands of learners. Ridgeway Learning is an online learning platform that we invented for this case. It has 12 courses and 84 lessons, for data, machine learning and web development. After each lesson, a learner’s home page shows an Up next block with 3 lessons. Today the block shows the most popular lessons.

Nadia is the head of product. She asks: “Can the block show the lesson each learner will really take next? And can it never show a lesson they are not ready for?”

Kofi is the engineer who will build it. His questions: “What must we compute each night? How fast is one list?”

Nadia, Kofi and Ridgeway Learning are made up. So is the data.

A comparison of seven lists on the same learners and the same two weeks. They come from four families, alone and combined: popularity, neighbours (co-occurrence of completions), a sequence method (Markov) and a prerequisite rule. The result is a recommendation for Nadia and Kofi: a design, its settings, its cost, and what a person still checks. Then you run the design once on two weeks that no method has seen.

This data is synthetic. Our code made the catalogue, the learners and every completion, with a model of how learners behave. No real learner is in it. It does not come from Nextia Learning or from any other platform.

Source Made for this case study with make_data.py, in the labs folder of the notebook
Licence CC0 1.0 (public domain dedication): use it as you like
Files lessons.csv (the catalogue) and completions.csv.gz (1.7 MB), with a dataset card and their SHA-256 checksums
Size 84 lessons in 12 courses; 259,760 completions by 24,000 learners, from 6 January 2025 to 15 February 2026

What one row means. In lessons.csv, one row is one lesson. It has an ID (for example PD-03, lesson 3 of Data analysis with pandas), a course, a position and a title. It also says whether the lesson is optional, and it lists the prerequisites. In completions.csv.gz, one row is one lesson that one learner completed, with the date and time.

We looked for open progress data first. The Open University Learning Analytics Dataset is CC BY 4.0. It has seven modules and a planned week for each material, but it describes no prerequisites. It also has sensitive data (disability, deprivation) that this case does not need. EdNet is CC BY-NC 4.0: no commercial use, so not allowed. And we never use the real progress data of Nextia Learning’s own learners.

The generator’s documentation lists every rule. In short:

  • Each learner picks a goal when they join: data analyst, machine-learning engineer, backend developer, AI app builder, or none (a “browser”). A goal is a list of courses, in order.
  • Learners take the lessons of a course in order. They take an optional lesson 45% of the time.
  • 30% of the learners with a goal take two courses at a time.
  • When they start a course, 7% jump ahead to a later course before its prerequisites are done. 10% take a detour to a popular course outside their goal. Browsers ignore prerequisites.
  • After each lesson, some learners stop for good. Half of the learners completed 8 lessons or fewer; 17% completed only one.

Both are synthetic, and both were made by our code. Larkfield’s shop log was made to look like a shop. This data was made to look like a learning platform. Three differences matter:

  • Order is built in. Most learners take lessons in order, because we wrote the model that way. At Larkfield, the order of purchases mattered much less.
  • The catalogue is tiny. 84 lessons, against Larkfield’s 2,994 products.
  • There are rules. Prerequisites say which lessons a learner can take. Larkfield had business rules, but no product needed another one first.

So this case is easier than a real platform. The methods can only find what our behaviour model put in. Read the numbers as a test of the methods on known behaviour, not as a measurement of real learners.

The new idea: prerequisites and ready lessons

Section titled “The new idea: prerequisites and ready lessons”

A prerequisite is a lesson you must complete before another one. Inside each course here, a lesson needs the lesson before it. Optional lessons are the exception: no lesson needs them. The first lesson of 8 courses needs lessons of other courses:

A map of 12 courses. Four courses need nothing: Python basics, How the web works, Git and the command line, and SQL for analysis. Statistics needs Python lesson 4. Data analysis with pandas needs Python lesson 8. Charts and dashboards needs pandas lesson 4. Machine learning needs pandas lesson 8 and Statistics lesson 5. Deep learning needs Machine learning lesson 6. Build APIs with Python needs Python lesson 8 and web lesson 4. Deploy to the cloud needs APIs lesson 5 and Git lesson 3. Build apps with language models needs APIs lesson 4.
Ridgeway Learning's courses (invented). An arrow means that the first lesson of a course needs that lesson of another course. The courses with a thick border need nothing.
Text description of the diagram

Twelve courses. Python basics, How the web works, Git and the command line, and SQL for analysis need nothing. Statistics needs Python lesson 4. Data analysis with pandas needs Python lesson 8. Charts and dashboards needs pandas lesson 4. Machine learning needs pandas lesson 8 and Statistics lesson 5. Deep learning needs Machine learning lesson 6. Build APIs with Python needs Python lesson 8 and web lesson 4. Deploy to the cloud needs APIs lesson 5 and Git lesson 3. Build apps with language models needs APIs lesson 4.

A lesson is ready for a learner when they have not done it, and they have done all its prerequisites. Here is a tiny example: a learner who completed the first four lessons of Python basics.

Lesson Needs Ready?
PY-05 Dictionaries PY-04 yes
STA-01 Averages and spread PY-04 yes
SQL-01, WEB-01, GIT-01 nothing yes
PY-06 Files PY-05 no: PY-05 is not done
PD-01 DataFrames PY-08 no

A suggestion breaks a prerequisite when it is not ready. The break rate is the share of suggestions that break a prerequisite. It is a new measure for this case. We want it near 0.

  • When is the list made? When a learner opens their home page, from the lessons they have completed. Counts are rebuilt every night.
  • What do we predict? The next lesson the learner completes.
  • For whom? Every learner who completes a lesson in the test period, including new learners.
  • Recall@3: is the real next lesson among the 3 suggestions? There is one right answer per learner, so recall@3 is 1 or 0 for each learner. We average it over learners, with a 95% interval from the bootstrap.
  • NDCG@3: the same, but a right answer in place 1 counts more than in place 3.
  • Break rate: the share of the suggestions that break a prerequisite.
  • Coverage, novelty and diversity, as in the harness. Diversity counts pairs of suggestions from different courses.

The segments are the learner’s state when the period starts. New: no lesson yet. In a course: they started a course and did not finish it. Between courses: every course they started is finished.

Four boxes from left to right. 6 January 2025 to 4 January 2026: history only, the methods learn. 5 to 18 January 2026: validation, choose the design. 19 January to 1 February 2026: the test, every method once, with the history before 19 January; these numbers are reported. 2 to 15 February 2026: the new input, the design once, with the history before 2 February.
Choose on the validation weeks, test once, then run the chosen design once on the last two weeks. Each method learns only from completions before its period.
Text description of the diagram

The history runs from 6 January 2025. Validation is 5 to 18 January 2026: we choose the design. The test is 19 January to 1 February: every method runs once with everything known before 19 January, and we report these numbers. The new input is 2 to 15 February: only the chosen design runs, once.

The test period has 4,963 learners: 1,053 new, 3,628 in a course and 282 between courses.

One question decides which methods can work: what do learners do next?

What the test period's learners completed next. Same: a lesson of the course of their last lesson. First: a new learner's first lesson. New: the first lesson of a course they had not started. Other: a lesson of another course they had started.
Text description of the chart

Four columns, out of 4,963 learners. The next lesson was in the same course as the last lesson for 63.6%. It was a new learner’s first lesson for 21.2%. It was the first lesson of a new course for 10.6%. It was a lesson of another course they had already started for 4.6%.

Two facts follow:

  • Most next lessons continue the last course. A method that knows the order should do well. A method that ignores order will guess lessons that are too far ahead.
  • The hard cases are the changes: a new learner’s first lesson, and the next course after one ends. There, the learner’s goal matters, and we do not know it.

One more fact: 94.5% of the real next lessons were ready. The other 5.5% broke a prerequisite: some of our learners jump ahead. A method that never breaks a prerequisite can never find those. The notebook’s first task computes this share.

popular ranks lessons by the number of learners who completed them in the 28 days before the period. Each learner gets the most popular lessons that they have not done. It is the “trending” list of The baselines to beat.

def popular(split, days=28):
"""Lessons ranked by the number of learners who completed them in the last `days` before the period."""
h = split.history
recent = h[h["completed_at"] >= split.start - pd.Timedelta(days=days)]
return np.bincount(recent["item"], minlength=split.cat.n).astype(float)

On the test period, popular gets recall@3 0.415 (0.400 to 0.428), with a break rate of 58.5%. The most popular lessons are PY-01, PY-02, PY-03: the start of Python basics, which most goals begin with. So a new learner sees PY-01, PY-02, PY-03. They can take PY-01, but not the other two. At Larkfield, trending (0.156) beat every single personal method. Here, popularity ignores the one thing that matters most: where the learner is.

“Learners who completed this lesson also completed that one.” This is item-item collaborative filtering from Item-item collaborative filtering. For each pair of lessons, count the learners who completed both. Divide by the square root of the product of the two lessons’ counts: this is the cosine. A learner’s score for a lesson is the sum of its cosines with the lessons they completed.

def cooccurrence(split):
"""Item-item cosine between lessons: learners who completed both, divided by the square root of the product of
the two counts (as in item-item collaborative filtering). The diagonal is 0."""
h = split.history.drop_duplicates(["learner", "item"])
users = {l: r for r, l in enumerate(h["learner"].unique())}
m = np.zeros((len(users), split.cat.n))
m[h["learner"].map(users).to_numpy(), h["item"].to_numpy()] = 1
both = m.T @ m
n = np.diag(both).copy()
sim = both / np.sqrt(np.outer(n, n) + 1e-9)
np.fill_diagonal(sim, 0)
return sim

On the test period, co-occurrence gets 0.726, with a break rate of 52.3%. It knows which lessons belong together. It does not know which one comes first. The failure case below shows what that means.

Sessions, sequences and LLMs, briefly taught the Markov model. For each lesson, count which lesson the same learner completed directly after it. Suggest the lessons that most often followed the learner’s last lesson.

def transitions(split):
"""Markov: how often lesson b was the next lesson completed after lesson a, by the same learner."""
h = split.history.sort_values(["learner", "completed_at"], kind="stable")
a, b = h["item"].to_numpy()[:-1], h["item"].to_numpy()[1:]
same = h["learner"].to_numpy()[:-1] == h["learner"].to_numpy()[1:]
T = np.zeros((split.cat.n, split.cat.n))
np.add.at(T, (a[same], b[same]), 1)
return T

Markov gets 0.853: much better than co-occurrence (+0.126 (+0.114 to +0.138) paired difference). It knows the order. But its break rate is still 44.4%. After a lesson, one next lesson has a large count. The second and third places have small counts: a few learners who skipped a lesson or jumped ahead. Those places are often lessons the learner is not ready for. New learners have no last lesson, so Markov shows them popular lessons.

The music teacher’s method: show only ready lessons, in a sensible order.

  1. The next ready lessons of the courses in progress, the most recent course first.
  2. The first lessons of new courses that are ready, the most popular first.
  3. Any other ready lesson (an optional lesson left behind), the most popular first.

If fewer than 3 lessons are ready, popular lessons fill the free places. The rule needs no training: only the catalogue and popularity.

def rule_scores(order, cat, rank):
"""The prerequisite-aware rule for one learner (`order`: the lessons done, oldest first). Only lessons whose
prerequisites are done get a score: first the next lessons of the courses in progress (the most recent course
first, then by position), then the first lessons of new courses (by `rank`), then the rest (optional lessons
left behind), by `rank`. Other lessons get -inf."""
done = np.zeros((1, cat.n), bool)
done[0, list(order)] = True
ok = eligible(done, cat)[0]
s = np.where(ok, rank, -np.inf)
progress = in_progress(list(order), cat)
for j in np.flatnonzero(ok):
c = cat.course[j]
if c in progress:
s[j] = 100 - 10 * progress.index(c) - cat.position[j] / 100
elif j == cat.first[c]:
s[j] = 2 + rank[j]
return s

The rule gets 0.910, with a break rate of 0.2%. It beats Markov by +0.057 (+0.047 to +0.067). It also handles a learner who takes two courses at a time. Both courses are in progress, so both next lessons are in the list.

The rule knows what a learner can take. Markov knows what learners did take. They can work together in two ways:

  • Markov + prerequisites: Markov’s list, with every lesson that is not ready removed; the rule fills the free places.
  • Rule + Markov: the rule’s order, but the new courses are ordered by Markov from the learner’s last lesson (popularity only breaks ties). After SQL for analysis, data analysts usually go on to Statistics: Markov has counted that.

We also tried co-occurrence + prerequisites. We chose the design on the validation weeks. The four designs got this recall@3:

DesignValidation recall@3
prerequisite rule0.896
co-occurrence + prerequisites0.843
Markov + prerequisites0.892
rule + Markov0.906

We chose rule + Markov. The notebook’s second task builds it.

One honest note about this choice. Our first trial printed the numbers of the validation weeks and of the test period. The idea of rule + Markov came from that trial. So the test period is not fully clean for this one design. The validation weeks show the same pattern. The last two weeks, in End to end on new input, were not in that trial, so they are the cleaner check.

Every method ran on the same 4,963 learners, with the same history and the same measures:

Methodrecall@3 (95% interval)NDCG@3break ratecoveragenoveltydiversity
popular0.415 (0.400 to 0.428)0.40758.5%30%1.230.15
co-occurrence0.726 (0.714 to 0.740)0.66652.3%99%2.080.29
Markov0.853 (0.843 to 0.863)0.81744.4%100%2.170.60
prerequisite rule0.910 (0.901 to 0.919)0.8400.2%99%2.060.96
co-occurrence + prerequisites0.851 (0.839 to 0.861)0.7430.2%100%2.130.97
Markov + prerequisites0.906 (0.897 to 0.914)0.8500.2%100%2.090.98
rule + Markov0.919 (0.912 to 0.927)0.8520.2%100%2.160.96

Coverage is the share of the 84 lessons that appear in at least one list. Novelty is in bits, as in the harness: a lesson that many learners completed has a low novelty.

The break rate on the test period: the share of suggestions that the learner was not ready for. Co-occ. is co-occurrence; design is rule + Markov. Only the methods that know the prerequisites keep it near 0.
Text description of the chart

Five columns. Popular: 58.5% of suggestions not ready. Co-occurrence: 52.3%. Markov: 44.4%. The prerequisite rule: 0.2%. Rule + Markov: 0.2%.

How to read it:

  • Rule + Markov is best on recall@3 and NDCG@3. Against the rule alone, the paired difference is small but clear: +0.009 (+0.005 to +0.013). Against Markov alone: +0.066 (+0.057 to +0.075).
  • Order matters more than co-occurrence. Markov beats co-occurrence by +0.126 (+0.114 to +0.138).
  • Prerequisites matter most. The break rate falls from 44.4% (Markov) to 0.2%. The few breaks left come from the popular fill, when a learner has fewer than 3 ready lessons.
  • Popularity is far behind, and over half of its suggestions are not ready.
Next-lesson recall@3 by segment on the test period, with 95% intervals. In a course, every method that knows the order does well. Between courses, Markov's order helps the rule.
Text description of the chart

Three panels with five methods each. New learners: popular 0.622, co-occurrence 0.622, Markov 0.622, rule 0.846, rule + Markov 0.846. In a course: popular 0.368, co-occurrence 0.780, Markov 0.916, rule 0.938, rule + Markov 0.940. Between courses: popular 0.241, co-occurrence 0.426, Markov 0.894, rule 0.794, rule + Markov 0.926.

  • New learners. Popularity, co-occurrence and Markov all show popular lessons, because the learner has no history: 0.622. The rule shows the first lessons of the courses that need nothing: 0.846.
  • In a course. The next lesson is almost always the next one in the course. The rule and the design are almost equal here.
  • Between courses. The rule alone orders new courses by popularity: 0.794. With Markov’s order, the design gets 0.926. This segment is small (282 learners), but it is where a learner may leave the platform.

Common mistake

'Learners who completed this also completed that'

Symptom: a learner has just completed PD-03 (Clean data). Co-occurrence suggests PD-04 (ready), PD-05 (not ready), SQL-02 (not ready). Two of the three cards are lessons the learner cannot take yet.

Cause: learners who completed PD-03 also completed PD-05, but after PD-04. Co-occurrence counts the pair and forgets the order. It also suggests SQL-02: data analysts take both courses, so the two lessons are completed by the same people. But this learner has not started SQL. Over all learners in a course, 53% of co-occurrence’s suggestions from the learner’s own course were not ready.

Diagnose: for each method, measure the break rate next to recall. Then look at a few lists of real learners, with each lesson marked ready or not ready. The notebook prints this learner’s lists.

Fix: only show ready lessons. For this learner, rule + Markov suggests PD-04 (ready), SQL-01 (ready), GIT-01 (ready). The learner really completed {fl.next} next.

The same mistake happens with any method that does not know the rules of the catalogue. Markov made it less often, because order is part of its counts. But it still suggested PD-05 to this learner: PD-04 (ready), PD-05 (not ready), SQL-01 (ready).

A learner opens their home page. The service reads the lessons they completed, in order. It fills 3 places in order: 1. the next lessons of the courses in progress, the most recent course first; 2. the first lessons of new courses whose prerequisites are done, ordered by what learners took next (Markov, rebuilt each night); 3. popular lessons, only if fewer than 3 can be taken. Up next: 3 lessons. A person checks the prerequisites in the catalogue, new courses and the weekly numbers.
The recommended design: the prerequisite rule, with new courses ordered by Markov. The Markov counts are rebuilt every night; the list is made when the page opens.
Text description of the diagram

A learner opens their home page. The service reads the lessons they completed, in order. It fills 3 places. First the next lessons of the courses in progress, the most recent course first. Then the first lessons of new courses whose prerequisites are done, ordered by what learners took next. Popular lessons fill the places only if fewer than 3 lessons are ready. A person checks the catalogue and the weekly numbers.

What we tell Nadia and Kofi:

  1. The Up next block: rule + Markov, 3 places. The next lessons of the courses in progress first. Then new courses whose prerequisites are done, ordered by Markov counts from the last lesson, with popularity to break ties.
  2. The fallback is the rule alone. If the Markov counts are missing, the rule orders new courses by popularity. It loses only 0.009 of recall@3 on the test period.
  3. Not now: co-occurrence. It was the weakest of the personal methods, and it suggested lessons out of order.
  4. What a person checks: the prerequisites in the catalogue are the design’s most important input. A wrong prerequisite blocks a lesson for everyone, and a missing one lets learners in too early. A course author checks them when a course is added or changed. A new course has no Markov counts, so it is ordered by popularity until learners take it. Each week, a person reviews recall@3 by segment and the break rate.

Measured on one thread of an Apple M4. Costs in money are assumptions.

Item Rule + Markov Popular
Nightly build Markov counts: 26 ms 1 ms
Table size 1,934 pairs of lessons, 28 KB 84 counts
One list, when the page opens 0.2 ms (median), 0.2 ms (95th percentile), in Python a copy
Lists for every learner of the period at once 0.7 s for 4,963 learners (a Python loop) less than a second
Server, assumption: the job runs on a server the platform already has about US$0 extra about US$0 extra
Engineering, assumption prerequisites kept correct in the catalogue, a nightly job, a weekly check almost none

The real cost is not the computer. It is the catalogue: every prerequisite must be right, and every new course needs them before it goes live. That is work for course authors, not for engineers. In return, learners stop seeing lessons they are not ready for: the break rate falls from 58.5% to 0.2%.

Recall is offline. It says whether the learner took a suggested lesson, not whether the block caused it. Most learners would take the next lesson of their course anyway. So ship the design and test it online against the popular block, as in Online tests and guardrails. A good online measure is the share of learners who start a new course after finishing one.

The two weeks from 2 to 15 February 2026 were not used to build or choose anything. We run the whole design once. Popularity and the Markov counts come from every completion before 2 February. Then we make the lists for the learners of those two weeks.

Methodrecall@3 (95% interval)break rate
popular0.412 (0.400 to 0.425)57.6%
Markov0.845 (0.835 to 0.854)44.9%
prerequisite rule0.905 (0.897 to 0.914)0.3%
rule + Markov0.917 (0.909 to 0.925)0.3%

5,000 learners completed a lesson in those two weeks. The order is the same as in the test period: the design first, the rule second, popular far behind.

A service makes the list for one learner at a time, when the page opens. The notebook’s suggest_one does that, and gives the same list as the batch code. We show the first learner (by ID) who is between courses and has completed at least 8 lessons. We chose the rule before we looked at the list.

Learner {ex.learner} completed 19 lessons: How the web works (without the optional lesson), Python basics (without the optional lesson) and SQL for analysis. The last one, {ex.done[ex.done.length - 1]}, was on 2026-01-27. Every course they started is finished.

Ready lessons: GIT-01, PY-07, STA-01, PD-01, WEB-05, API-01. Among those, what did learners complete right after {ex.done[ex.done.length - 1]}? Markov counts: STA-01 857, PY-01 261, WEB-01 243, GIT-01 128, PD-01 109. (PY-01 and WEB-01 are done, so they do not count.)

List Lessons
Rule + Markov STA-01, GIT-01, PD-01
Popular STA-01, STA-02, PY-07
Markov STA-01, GIT-01, PD-01

The learner really completed STA-01 (Statistics with Python, lesson 1) next. The design had it in place 1. The popular list also had it, but it showed STA-02, which needs STA-01 first. Markov’s list is the same as the design’s here, because its top lessons were ready.

What we learned:

  • Know the rules before you learn from the data. A list that respects prerequisites was the biggest gain, and it needed no model.
  • Order beats togetherness. Markov (order) beat co-occurrence (togetherness) by a wide margin, because lessons have an order.
  • Each family has its own job. The rule says what a learner can take. The sequence says what they will probably take. Popularity only fills gaps.
  • Measure what must never happen. Recall alone would have hidden that Markov showed 44.4% lessons the learner was not ready for.

Against Larkfield’s shop log: at Larkfield, trending was hard to beat, and order mattered little. Here, popularity was far behind, and order and prerequisites decided almost everything. Both datasets are synthetic, so both results follow from how we built them. What carries over to a real platform is the method: measure a break rate next to recall, and look at each segment.

Limits:

  • Synthetic data. The methods found the behaviour that we put in the model. The rule looks strong because most of our learners follow the prerequisites. Real learners repeat lessons, come back after months and arrive from search.
  • The rule cannot follow learners who jump ahead. 5.5% of the real next lessons broke a prerequisite. The design never found those; Markov found 21% of them. If many real learners jump ahead, they may want a “skip ahead” choice, not a hidden lesson.
  • We do not know the learner’s goal. It is in our generator but not in the data, as on most platforms. Asking learners for a goal might help most between courses.
  • One test period, offline only. Two weeks in January. Recall says what learners did, not what the block made them do.
  • Suggestions steer people. A learning platform shapes careers, not only clicks. Show why a lesson is suggested (“next in Data analysis with pandas”), and let learners choose another one.
  • Prerequisites are advice, not walls. The design never suggests a lesson that is not ready. That does not mean the platform should lock it. An experienced learner may know the material already.
  • Do not judge learners by their speed. Progress data shows who stopped and who was slow. Do not use it to rank learners or to show them to others.
  • Progress data is personal. A real platform’s completions belong to its learners. Use them for the learner’s own suggestions, and say so in the privacy notice. This case uses synthetic data only. It never used Nextia Learning’s real learners.
  • Say what is measured. Every number here is offline, on synthetic data. Do not promise Nadia more finished courses.

This case study shows an employer that you can turn a business rule into a measure (the break rate). It also shows that you can combine a rule with a learned order, instead of choosing one.

What to put in the repository:

  • README.md (template below).
  • The notebook, run from top to bottom, with its outputs.
  • DECISION.md: the design, its settings, the costs with their assumptions, and what a person checks.
  • The data’s source and licence (CC0), with the SHA-256 checksums, and a sentence that the data is synthetic. You may commit the two files: CC0 allows it.

A README template:

# What to learn next: suggestions for a learning platform
## Question
What should an "Up next" block show after each lesson, so that learners
see the lesson they will take next, and never a lesson they are not ready for?
## Data and licence
Synthetic progress data (CC0 1.0), made by a documented behaviour model:
12 courses, 84 lessons with prerequisites, 259,760 completions by
24,000 learners. No real learners.
## Approach
Time split: choose on two validation weeks, test once on the next two,
run the chosen design once on the last two. Methods: popular (28 days),
co-occurrence (item-item cosine), Markov (the next lesson after the last),
a prerequisite rule, and combinations. Measures: next-lesson recall@3 with
bootstrap intervals, NDCG@3, the break rate (suggestions not ready),
coverage, novelty, diversity, by segment (new, in a course, between courses).
## Results
(the comparison table, the break-rate chart and the segment chart)
## Decision
The prerequisite rule, with new courses ordered by Markov.
Popular lessons only when fewer than 3 lessons are ready.
## Limits
Synthetic data, one test period, offline only, the learner's goal unknown.

How to describe it in three sentences: “I compared popularity, co-occurrence, a Markov sequence model and a prerequisite rule for next-lesson suggestions, with a time split. I added a new measure for suggestions the learner is not ready for. A rule that only shows ready lessons, with new courses ordered by Markov, found the next lesson most often and almost never broke a prerequisite.”

Go further:

  • Change make_data.py so that 30% of learners jump ahead, not 7%. Make the data again and run the notebook. Does the design still win? What happens to the rule between courses?
  • Give the design a “skip ahead” place: the third card may show Markov’s top lesson even if it is not ready, with a label. Measure recall@3 and the break rate.
  • Add the learner’s goal to the data (the generator knows it). Order new courses by the goal. How much does it help between courses?

Check your understanding

Markov finds the next lesson more often than co-occurrence. Why, and why is Markov still not enough on its own?

Show the answer

Markov counts which lesson came directly after the learner’s last lesson, so it knows the order. Co-occurrence only counts lessons completed by the same learners, so it also suggests lessons far ahead in the course. Markov gets 0.853 against 0.726. But Markov’s second and third places come from small counts, often learners who skipped or jumped ahead. So 44.4% of its suggestions are lessons the learner is not ready for. Only the prerequisites can remove those.

Check your understanding

The design never shows a lesson whose prerequisites are not done. What does that cost, in this data?

Show the answer

In the test period, 5.5% of learners really completed a lesson whose prerequisites were not done: they jumped ahead. The design can never suggest that lesson, so it misses every one of them. Markov found 21% of them. On the whole test period the design is still clearly better, but the cost is real. A platform where many learners jump ahead might add a labelled “skip ahead” card instead of hiding those lessons.

Check your understanding

The rule and the design are almost equal for learners in a course. Where does Markov's order help, and why there?

Show the answer

Between courses. In a course, the next lesson is almost always the next one in the course, and the rule already puts it first. When a course ends, the rule must choose among new courses, and alone it orders them by popularity: 0.794. Markov has counted which course learners started after that last lesson. For example, after SQL for analysis, learners usually started Statistics with Python. With that order, the design gets 0.926.

Recap

  • When items have an order and prerequisites, popularity is a weak baseline: popular gets recall@3 0.415, and 58.5% of its suggestions are not ready.
  • Markov (0.853) beats co-occurrence (0.726) because it knows the order. But 44.4% of its suggestions still break a prerequisite.
  • Measure what must never happen next to recall: here, the break rate.
  • The design: the prerequisite rule, with new courses ordered by Markov (0.919, break rate 0.2%). The fallback is the rule alone.
  • The real cost is a correct catalogue of prerequisites, not the computer. Test the block online before you keep it.
  • The data is synthetic, so the methods only find the behaviour we put in. Use the method, not the numbers.

Next comes the final assignment: you build and defend a recommender for a new Larkfield category, on data that no lesson has used.

Next: Final assignment