Skip to content
Français

Tag research papers with many labels

  • Case study 4
  • 150 min
  • Optional · 1 case study needed for the certificate

Where we are

In nine modules, most messages had one label. In this optional case study, we use real open paper abstracts that often have several subject labels at once. We train one model per label, choose a threshold for each, and report the results per label.

Case study 4 of 4 · Text Classification and Routing: Send Each Message to the Right PlaceTag research papers with many labels

What you will be able to do

Take 12,000 real paper abstracts to a decision for an assistant that suggests subject categories to a person. You will train one yes/no model per category, choose the thresholds from a cost table, study the long tail and two levels of labels, and compare three kinds of method fairly.

Case studyPortfolio piece

Time
About 2.5 hours
Runs in
Colab, Kaggle or VS Code
Skills used
  • Multi-label data checks
  • One-vs-rest TF-IDF
  • Thresholds from a cost table
  • A threshold per label
  • Micro- and macro-F1
  • The long tail
  • Two levels of labels
  • Frozen embeddings
  • Recorded LLM, zero- and few-shot

Colab and Kaggle open your own copy. To use VS Code, download the notebook from GitHub. Stuck? See the solution notebook.

This case study is optional. You can choose it for the certificate.

The notebook runs every step of this page, plus three tasks with checks. Its setup cells install the packages and download the data and the course’s files (6 MB). The embedding model (about 470 MB) downloads once. It needs no account and no key: the LLM’s answers were recorded. After the install, it runs in about 3 minutes on a laptop.

Before you startWhat to know and to have readyAbout 150 minutes

What you need to know

This case study builds on Module 7. These lessons matter most:

What you need on your computer

Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, sentence-transformers 6.2.0 and PyTorch 2.15.0 (CPU). The setup cell makes a folder arxiv-papers. To start again, delete that folder. This case study does not change your larkroute project.

Tested on macOS (Apple silicon) with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.

Imagine a university library that runs an open archive of research papers. It uses the subject categories of arXiv, the large open archive of science papers. Authors choose their paper’s categories from a long list, and volunteer moderators check the choice.

Nadia leads the moderators. Her problem: authors often choose too few categories, or the wrong ones. Then the readers of the missing lists never see the paper.

Kofi is a data scientist. Nadia asks him: “Can a model suggest the categories, so that a person only confirms them? How many papers get no useful suggestion? And what does it cost?”

We invented Nadia, Kofi and the library. The papers are real.

The important word is suggest. A person always looks at the paper, and the assistant saves that person time. Kofi writes the costs down with Nadia. They are assumptions:

  • A right category that the assistant does not suggest costs 1.00 dollar. The person must find it in a list of 70 categories.
  • A wrong suggestion costs 0.20 dollars. The person reads it and removes it, in about 10 seconds.
  • A paper with no suggestion at all costs 1.50 dollars, the course’s cost of a review: about a minute of the person’s time.

So “a person tags everything”, the library’s process today, costs 1,500 dollars per 1,000 papers.

You will build an assistant that reads a paper’s title and abstract and suggests every category that fits. When it is not sure of any category, it suggests nothing, and the person tags the paper alone. You will compare methods with one measure: the cost per 1,000 papers.

The data is a sample of arXiv’s paper metadata: the ID, the title, the abstract and the categories of each paper. arXiv dedicates its metadata to the public domain (CC0 1.0). We use a snapshot on Hugging Face, gfissore/arxiv-abstracts-2021, with about 2 million papers.

We took one of its five files (363,006 papers from 2020 and 2021). We kept the papers whose main category is in one of four areas: computer science (cs), mathematics (math), statistics (stat) and electrical engineering (eess). Then we drew a fixed sample: 5,000, 4,000, 1,500 and 1,500 papers. The sample is 5 MB, and it comes with the notebook.

Each paper has one primary category, the first in its list, and it can have more: cs.LG cs.CV means machine learning, then computer vision.

We also compared the full arXiv metadata dump and GoEmotions, Reddit comments with emotion labels. The dataset card says why this sample won.

The notebook loads the sample:

papers = pd.read_csv("data/papers.csv.gz", dtype={"id": str})

The first checks print:

papers: 12,000 | areas: {'cs': 5000, 'math': 4000, 'eess': 1500, 'stat': 1500}
missing values: 0 | repeated IDs: 0 | repeated abstracts: 1
words per abstract: median 150, 90% have at most 229, longest 328
categories per paper: {1: 5795, 2: 4161, 3: 1536, 4: 418, 5: 84, 6: 5, 8: 1}
categories without a name: []

What this tells us. The data is clean. The abstracts are about five times longer than Larkfield’s messages: a median of 150 words, against 28. More than half of the papers have two or more categories.

Note dtype={"id": str}. arXiv IDs look like numbers, but they are text. Read as a number, 1912.11930 becomes 1912.1193, and the paper is lost. Our first LLM recording had this bug.

arXiv gives a few categories two names. For example, statistics theory is both math.ST and stat.TH. The notebook checks the raw lists:

math.ST and stat.TH: 284 and 284 papers; always together: True
cs.IT and math.IT: 381 and 381 papers; always together: True
math.NA and cs.NA: 444 and 444 papers; always together: True
eess.SY and cs.SY: 609 and 609 papers; always together: True
papers that also had a category outside the four areas (dropped): 19.0%

What this tells us. The two names always come together. If we kept both, a model would get a “free” second label every time, and the scores would look too good. So we kept one name per pair.

Think of a library’s shelves. A few subjects fill whole walls, and many fill half a shelf. Some have three books. Labels work the same way: a few are very common, and many are rare. This shape is called a long tail. The notebook counts the training papers of each category:

train 8,002 | validation 1,999 | test 1,999 (used once, at the end)
categories: 78 in train | learned: 70 | too rare (< 20 papers): 8 | rare (20 to 49 papers): 16
most common: {'cs.LG': 1797, 'cs.CV': 1028, 'stat.ML': 792, 'cs.AI': 547, 'stat.ME': 536}
too rare: {'math.GM': 18, 'cs.ET': 18, 'math.HO': 16, 'cs.MS': 13, 'cs.SC': 10, 'cs.OS': 3, 'cs.OH': 2, 'cs.GL': 1}
The 78 categories of the training papers, grouped by how many papers they have. 15 categories have 300 papers or more; 8 have fewer than 20, too few to learn here.
Text description of the chart

Five columns, the number of categories in each size group, by training papers. 1 to 19 papers: 8 categories, in a second colour, too rare to learn. 20 to 49 papers: 16 categories. 50 to 99: 16. 100 to 299: 23. 300 or more: 15. The most common category, cs.LG, has 1,797 training papers; the rarest, cs.GL, has 1.

What this tells us. Machine learning (cs.LG) alone is on 1,797 of the 8,002 training papers. At the other end, 8 categories have fewer than 20 papers. A model cannot learn from 3 examples, so we do not train those 8, and the cost table counts them as missed. The 16 categories with 20 to 49 papers are the rare ones that we can learn, and we measure them on their own.

51.4% of the training papers have more than one category; at Larkfield, 12.6% of the messages held two requests.

Kofi writes down his answers before he trains anything:

  • Unit: one paper. It gets zero to five suggested categories.
  • Labels: the 70 categories with at least 20 training papers. We also look at the four areas, the first level of the labels.
  • One measure for the choice: the cost per 1,000 papers, with the cost table above.
  • The split: train 8,002, validation 1,999, test 1,999, stratified by area. All choices are made on validation; the test papers are used once, at the end.

A multi-label result needs new measures. Here are three, with a tiny example: 10 papers and three categories.

Category Right papers Found Wrong suggestions F1
cs.LG (common) 8 7 1 2 × 7 ÷ (2 × 7 + 1 + 1) = 0.875
stat.ML 4 2 2 2 × 2 ÷ (2 × 2 + 2 + 2) = 0.500
math.KT (rare) 1 0 0 0
  • Micro-F1 counts every suggestion of every category together: 9 found, 3 wrong, 4 missed. So 2 × 9 ÷ (2 × 9 + 3 + 4) = 0.72. Common categories decide it.
  • Macro-F1 averages the three F1 scores: (0.875 + 0.5 + 0) ÷ 3 = 0.458. The rare category counts as much as the common one.
  • Exact match is the share of papers whose whole set is exactly right, as in Multi-label messages.

The tables also show the recall of the rare categories, the share of papers with no suggestion (they go to the person), and the suggestions per paper.

Kofi starts with the method of Multi-label messages: TF-IDF of words and word pairs, and one logistic regression per category (one-vs-rest): 70 yes/no models. They train in 11 seconds. With the usual threshold of 0.50 for every category, on the validation papers:

method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_1000
0 TF-IDF, 0.50 for every category 0.198 0.459 0.196 0.0 0.494 0.65 1138.7
1 everything to a person 0.000 0.000 0.000 0.0 1.000 0.00 1500.0

What this tells us. At 0.50, half of the papers (49.4%) pass no threshold at all, and the person must tag them alone. The model suggests 0.65 categories per paper, but the papers have 1.7 on average. It saves only 361 dollars per 1,000 papers.

One validation paper, “On Tilings of Asymmetric Limited-Magnitude Balls”, has math.CO (combinatorics) and cs.IT (information theory).

The paper On Tilings of Asymmetric Limited-Magnitude Balls, whose true categories are math.CO and cs.IT, goes to 70 yes/no models: math.CO 0.280, cs.IT 0.126, math.DS 0.024, math.MP 0.023, and the 66 others 0.023 or less. With 0.50 for every category, nothing passes and the person tags the paper alone. With a threshold per category, math.CO 0.280 is above its threshold of 0.07 and cs.IT 0.126 is above 0.08, so the assistant suggests both, and both are right.
One validation paper. At 0.50, the assistant suggests nothing. With the thresholds chosen in this case study, it suggests exactly the two categories that the authors chose.
Text description of the diagram

On the left, the paper “On Tilings of Asymmetric Limited-Magnitude Balls”. Its authors chose math.CO and cs.IT. An arrow leads to the 70 yes/no models. Their scores: math.CO 0.280, cs.IT 0.126, math.DS 0.024, math.MP 0.023, and the 66 others 0.023 or less. One arrow, “the usual threshold”, leads to 0.50 for every category: nothing passes, and the person tags the paper alone. The other arrow, “this case study”, leads to a threshold per category: math.CO 0.280 is at least 0.07, yes; cs.IT 0.126 is at least 0.08, yes; math.MP 0.023 is below 0.05, no. The last box: suggest math.CO and cs.IT, both right, and the person confirms.

The model ranks the two right categories first and second, but both scores are far below 0.50. Each yes/no model saw few papers of its category, so its probabilities stay low. The problem is the threshold, not the ranking.

Should the assistant suggest a category with a probability of 0.28? Think of an umbrella. Carrying it costs a little; getting wet costs more. So you take it even when rain is not likely.

The same reasoning works for a suggestion. Let p be the chance that the category is right.

  • If the assistant suggests it, the cost is the chance that it is wrong times 0.20: (1 − p) × 0.20.
  • If it does not, the cost is the chance that it is right times 1.00: p × 1.00.

The two costs are equal when (1 − p) × 0.20 = p × 1.00, so p = 0.20 ÷ 1.20 = 0.167. Above that, suggesting is cheaper. This is the break-even threshold, and it comes from the cost table, not from the model.

p Suggest: (1 − p) × 0.20 Leave out: p × 1.00 Cheaper
0.10 0.18 0.10 leave out
0.167 0.167 0.167 equal
0.28 0.144 0.28 suggest
0.50 0.10 0.50 suggest

The break-even threshold holds only if the probabilities are true: 0.28 must mean “right 28 times in 100”. So Kofi tries every threshold from 0.05 to 0.90 on the validation papers:

  • Total cost per 1,000 papers
  • Missed categories (1.00 dollar each)
  • Wrong suggestions (0.20 dollars each)
  • No suggestion: the person tags it (1.50)
TF-IDF, 1,999 validation papers. As the threshold rises, wrong suggestions cost less, but more papers get no suggestion and go to the person. The total is lowest at 0.07.
Text description of the chart

Four lines against the threshold, from 0.05 to 0.60. The total cost: 760.3 dollars per 1,000 at 0.05, lowest at 0.07 with 735.1, then 749.6 at 0.10, 857.9 at 0.20, 965.9 at 0.30, 1,138.7 at 0.50 and 1,210.4 at 0.60. Wrong suggestions fall fast: 404.4 at 0.05, 299.6 at 0.07, 209.1 at 0.10, 93.0 at 0.20 and 21.9 at 0.50. Papers tagged by the person alone rise fast: 14.3 at 0.05, 45.8 at 0.07, 99.8 at 0.10, 307.7 at 0.20 and 740.6 at 0.50. Missed categories rise from 341.7 at 0.05 to 457.2 at 0.20, then fall to 376.2 at 0.50, because more and more papers go to the person, who finds every category.

one threshold for every category: lowest cost at 0.07, 735.1 dollars per 1,000, 3.1% to a person (at 0.50: 1138.7)

What this tells us. The cheapest threshold is 0.07, far below the break-even 0.167 and 0.50. The probabilities are too low: a category with 0.10 is right more often than 1 time in 10. At 0.07, the assistant suggests 2.79 categories per paper, 3.1% of the papers get no suggestion, and the cost falls from 1,138.7 to 735.1 dollars per 1,000.

Kofi also chooses a threshold per category on the validation papers, by the same cost. A category with fewer than 10 right validation papers keeps 0.07.

thresholds per category: from 0.05 to 0.22, median 0.07; 22 categories keep 0.07
method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_1000
0 TF-IDF, a threshold per category 0.210 0.622 0.455 0.090 0.040 2.36 673.6
1 TF-IDF, 0.07, at least one 0.139 0.573 0.421 0.062 0.000 2.82 731.0
2 TF-IDF, 0.07 for every category 0.135 0.573 0.421 0.062 0.031 2.79 735.1

What this tells us. On validation, the thresholds per category cost 673.6, about 60 dollars less than one threshold. But they were chosen on these same papers, so this row looks better than it will be. At Larkfield, thresholds per route lost on the test messages. The third row, “at least one”, always suggests the most likely category; it costs almost the same as sending those papers to the person.

If the person can tag at most 2% of the papers alone, the cheapest single threshold is 0.05, for 760.3 dollars per 1,000.

The first part of cs.LG is the area, cs: the “team” of Hierarchical labels, with the category as the “topic” inside it.

A paper goes to one or more of four areas. cs, computer science: 32 categories, for example cs.LG with 1,797 training papers, cs.CV 1,028, cs.AI 547. math, mathematics: 29 categories, for example math.OC 408, math.AP 353, math.CO 350. stat, statistics: 5 categories, for example stat.ML 792, stat.ME 536, stat.AP 301. eess, electrical engineering: 4 categories, for example eess.SP 430, eess.SY 409, eess.IV 397.
The two levels of the 70 learned categories, with the number of training papers of the largest ones. A quarter of the papers have categories in two areas.
Text description of the diagram

On the left, a paper, with its title and abstract. Four arrows lead to four area boxes. cs, computer science: 32 categories, for example cs.LG with 1,797 training papers, cs.CV with 1,028 and cs.AI with 547. math, mathematics: 29 categories, for example math.OC with 408, math.AP with 353 and math.CO with 350. stat, statistics: 5 categories, for example stat.ML with 792, stat.ME with 536 and stat.AP with 301. eess, electrical engineering: 4 categories, for example eess.SP with 430, eess.SY with 409 and eess.IV with 397.

24.6% of the training papers are in two areas, most often cs.LG with stat.ML. The notebook measures both levels with the threshold 0.07:

micro-F1 by category: 0.573 | by area: 0.820
wrong suggestions: 2995, of which in an area that the paper has: 2047 (68%)
method accuracy macro_f1 total_per_1000
0 area of the top category 0.812 0.752 750.4
1 a model for the 4 areas 0.842 0.794 632.3

What this tells us. The area level is much easier: a micro-F1 of 0.820 against 0.573. Two of every three wrong suggestions are in the right area. Those are near misses, and a person fixes them fast.

The table also scores a routing problem with the course’s routing.py: send each paper to the moderators of its primary area, where a wrong area costs 4 dollars. A model trained on the 4 areas (0.842) beats the area of the most likely category (0.812). This turns around the lesson of Route banking questions: here, the fine labels mix the areas, so a direct area model wins.

Two more methods learn on the same 8,002 training papers and are measured on the same 1,999 validation papers, with both kinds of threshold.

Frozen embeddings. The course’s older embedding model turns each text into 384 numbers, as in Sentence embeddings for classification, and one logistic regression per category learns on them. The model reads only the first 128 word pieces (parts of words): about the title and the first 80 words of the abstract. Embedding all 12,000 papers took 75 seconds on a laptop.

Both together. The mean of the TF-IDF and the embedding probabilities, for each category.

method exact_match micro_f1 macro_f1 rare_recall no_tag_share total_per_1000 ms_per_paper
0 TF-IDF, a threshold per category 0.210 0.622 0.455 0.090 0.040 673.6 3.08
1 TF-IDF + embeddings, a threshold per category 0.164 0.602 0.471 0.250 0.032 707.5 15.91
4 TF-IDF + embeddings, 0.12 for every category 0.133 0.571 0.444 0.208 0.027 752.2 15.91
5 embeddings, a threshold per category 0.117 0.543 0.427 0.271 0.046 842.5 12.83
6 embeddings, 0.17 for every category 0.118 0.529 0.403 0.229 0.049 877.7 12.83

What this tells us. Alone, the embeddings lose to TF-IDF, as on Larkfield’s inbox (0.920 against 0.962 there), but by more: 842.5 against 673.6 dollars. They read only the start of each abstract, and the deciding words can come later. But they find rare categories far better: a rare recall of 0.271 against 0.090. Together, the two models are the cheapest after TF-IDF alone, with almost three times its rare recall.

Among the trained routers, the cheapest on validation is still TF-IDF with a threshold per category. The notebook chooses it from the validation table.

A hosted LLM (Azure chat-small) chose categories for 400 validation and 400 test papers. The prompt lists the 70 categories with their names, such as cs.DL: Digital Libraries. The answer is JSON: a list of categories, each with a confidence from 0 to 1. We recorded each answer once, so you replay them with no key.

The zero-shot prompt has only the list and the paper. The few-shot prompt also shows the titles and categories of the 5 most similar training papers, found by TF-IDF, as in Few-shot examples and label descriptions. Here is the start of a few-shot prompt:

Similar papers and their categories:
- 'Elliptic fixed points with an invariant foliation: Some facts and more questions' -> math.DS
- 'Almost non-degenerate functions and a Zariski pair of links' -> math.CV math.AG
- 'Universality of the Hurwitz zeta-function on the half plane of absolute convergence' -> math.NT math.CV
...
Paper:
Local Versions of the Wiener-Levi Theorem. Let h be a real-analytic function in the neighborhood ...
zero-shot: 800 answers, all valid: True | tokens in 1195, out 39 | 0.139 dollars per 1,000 | median 1206 ms
few-shot: 800 answers, all valid: True | tokens in 1322, out 38 | 0.151 dollars per 1,000 | median 1562 ms

All 1,600 answers were valid. Each one costs about 0.015 cents and takes about 1.5 seconds.

The LLM’s written confidence gets a threshold too, chosen on the validation sample by cost. For the few-shot LLM, the cheapest is 0.01: keep every category that it names. For the zero-shot LLM, it is 0.7.

The LLM has answers only for the samples, so the trained methods are scored on the same 400 validation papers:

method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_1000
0 LLM few-shot, confidence >= 0.01 (recorded) 0.207 0.649 0.559 0.517 0.000 2.34 615.2
1 LLM zero-shot, confidence >= 0.7 (recorded) 0.230 0.645 0.568 0.552 0.003 2.14 661.4
2 TF-IDF, a threshold per category 0.207 0.598 0.411 0.069 0.037 2.46 719.3
3 TF-IDF, 0.07 for every category 0.140 0.556 0.380 0.034 0.030 2.88 773.0

What this tells us. Here the LLM wins, by a lot: 615.2 dollars per 1,000 papers for the few-shot LLM, against 719.3 for the best trained router. The largest gap is in the long tail: a rare recall of 0.517 against 0.069, and a macro-F1 of 0.559 against 0.411. And it never leaves a paper without a suggestion.

At Larkfield, the fine-tuned encoder beat the few-shot LLM. Here, the LLM has read about databases and K-theory long before this task. For a category with 30 training papers, TF-IDF has seen a few dozen examples; the LLM knows what the name means. The more labels you have, and the fewer examples each, the more that prior knowledge is worth.

400 papers is a small sample, but one missed category changes a row by only 2.5 dollars per 1,000. The gap is about 100 dollars, so it is not noise. The test sample below must still confirm it.

Now Kofi opens the test papers. Everything was chosen on validation; nothing changes after this run.

method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_1000
0 TF-IDF, a threshold per category 0.172 0.592 0.434 0.101 0.055 2.43 754.9
1 TF-IDF, 0.07, at least one 0.126 0.563 0.428 0.094 0.000 2.88 767.5
2 TF-IDF, 0.07 for every category 0.120 0.563 0.426 0.094 0.033 2.84 770.7
3 TF-IDF + embeddings, a threshold per category 0.131 0.565 0.422 0.237 0.042 2.47 814.0
4 TF-IDF + embeddings, 0.12 for every category 0.118 0.552 0.424 0.216 0.032 2.72 815.9
5 embeddings, 0.17 for every category 0.107 0.509 0.375 0.230 0.051 2.52 953.0
6 embeddings, a threshold per category 0.096 0.505 0.376 0.245 0.048 2.54 959.6
7 TF-IDF, 0.50 for every category 0.196 0.445 0.184 0.000 0.502 0.64 1157.9
8 everything to a person 0.000 0.000 0.000 0.000 1.000 0.00 1500.0
The 1,999 test papers, used once. Per cat.: TF-IDF with a threshold per category, the chosen router. All 0.07: TF-IDF with one threshold. Both: TF-IDF and embeddings together. Embed.: embeddings alone. A person tagging everything would cost 1,500.
Text description of the chart

A column chart with five columns, the cost per 1,000 test papers. TF-IDF with a threshold per category: 755 dollars, the lowest, in the first colour. TF-IDF with 0.07 for every category: 771. TF-IDF and embeddings together: 816. Embeddings alone: 953. TF-IDF with 0.50 for every category: 1,158. Everything to a person, not drawn, would be 1,500.

What this tells us. The choice holds, but the gap shrank. TF-IDF with a threshold per category costs 754.9 dollars per 1,000 papers, against 770.7 for one threshold: 16 dollars, not 60 as on validation. The thresholds fitted the validation papers a little too well. Against 1,500 for a person, the assistant halves the cost. It suggests 2.43 categories per paper and sends 5.5% of the papers to the person.

At Larkfield, exact match for the messages with two requests was 0.952. Here it is 0.172: fewer than 1 paper in 5 gets exactly the right set. Real labels chosen by thousands of authors are far harder than labels written from templates.

The samples with recorded LLM answers tell the same story:

method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_1000
0 LLM few-shot, confidence >= 0.01 (recorded) 0.255 0.676 0.576 0.500 0.000 2.27 586.7
1 LLM zero-shot, confidence >= 0.7 (recorded) 0.235 0.642 0.539 0.393 0.000 2.16 689.6
2 TF-IDF, a threshold per category 0.163 0.600 0.421 0.107 0.052 2.43 742.8
3 TF-IDF, 0.07 for every category 0.100 0.568 0.400 0.107 0.040 2.86 756.5
The 400 test papers with recorded LLM answers. Few-shot and Zero-shot: the hosted LLM. Per cat.: TF-IDF with a threshold per category. All 0.07: TF-IDF with one threshold. The few-shot LLM is the cheapest.
Text description of the chart

A column chart with four columns, the cost per 1,000 of the 400 test sample papers. The few-shot LLM: 587 dollars, the lowest. The zero-shot LLM: 690. TF-IDF with a threshold per category: 743. TF-IDF with 0.07 for every category: 757. The two LLM columns are in a third colour.

What this tells us. The few-shot LLM costs 586.7 dollars per 1,000 papers, 156 less than the best trained router, with a rare recall of 0.500 against 0.107. The zero-shot LLM, with the names only, costs 689.6.

The notebook gives the F1 score of each category on the test papers, grouped by the number of training papers:

20 to 49 training papers: 16 categories, mean F1 0.139
50 to 99 training papers: 16 categories, mean F1 0.371
100 to 299 training papers: 23 categories, mean F1 0.560
300 to 1999 training papers: 15 categories, mean F1 0.620
The chosen trained router's mean F1 per category, by how many training papers the category has. Rare categories are mostly missed.
Text description of the chart

Four columns, the mean F1 per category on the test papers, by the number of training papers. 20 to 49 papers: 0.14. 50 to 99: 0.37. 100 to 299: 0.56. 300 or more: 0.62.

What this tells us. Categories with 300 or more training papers reach a mean F1 of 0.620, and those with 20 to 49 papers 0.139. Several reach 0, such as cs.DB (databases) and cs.DL (digital libraries). This is why macro-F1 (0.434) is so far below micro-F1 (0.592).

The most missed categories tell a second story:

category missed suggested_instead times
0 cs.AI 74 cs.LG 56
1 math.OC 45 stat.ML 15
2 cs.LG 41 cs.CL 13
3 stat.ML 39 cs.LG 33

cs.AI (artificial intelligence) is missed 74 times, with cs.LG (machine learning) suggested instead in 56. stat.ML is missed 39 times, with cs.LG in 33. These categories overlap.

Common mistake

Labels that depend on the author

Symptom: Kofi reads 20 test papers where the assistant suggested cs.LG but not stat.ML. For most, he cannot say why the authors chose stat.ML too.

Cause: the categories are the authors’ own choices. In the training papers, cs.LG stat.ML appears 140 times and stat.ML cs.LG 178 times, and many similar papers have only one of the two. The label is partly a habit of the author.

Fix: do not chase these points with a bigger model. Treat the pair as one decision: show stat.ML as a second suggestion next to cs.LG, and let the person choose. If the library wants consistency, write a guideline for the border, as in Labelling guidelines.

Consequence in AI work: with labels chosen by many people, part of every “error” is disagreement. Measure it before you promise a score, as in Measure agreement.

The last step runs the assistant on three short abstracts that we wrote in the style of arXiv.

title suggested
0 Counting lattice paths with forbidden steps math.CO 0.76
1 A small transformer for detecting faults in power grids eess.SY 0.37, eess.SP 0.33
2 Confidence intervals after model selection stat.ME 0.72, stat.AP 0.19, cs.LG 0.15

What this tells us. The first suggestion is right. The second is half right: the abstract trains a transformer, and cs.LG is missing. For the third, the person would keep stat.ME, probably add math.ST, and remove cs.LG. A person stays in the loop.

Kofi writes to Nadia in her words:

  • Use the few-shot LLM to suggest categories, and keep every category that it names. On the test sample, it cost 586.7 dollars per 1,000 papers. The best trained router cost 742.8, and a person who tags everything 1,500. It finds the rare categories far better. It costs about 0.15 dollars per 1,000 papers to run and takes about 1.5 seconds.
  • Keep the trained router as the fallback: TF-IDF with a threshold per category (0.05 to 0.22). It runs inside the library, in 3 milliseconds. Use it when the LLM service is down, and for drafts that may not leave the library. On the full test set, it cost 754.9 dollars per 1,000 papers, with 5.5% sent to the person.
  • Show the suggestions as suggestions. With the trained router, only 51% of the suggestions are right, but they cover 71% of the right categories. A wrong suggestion costs a few seconds; a missed category costs a reader.
  • Show the area first. The area is right far more often (micro-F1 0.820 against 0.573), and most missing categories are in the suggested area.
  • Watch the rare categories. 24 categories have fewer than 50 training papers. The trained router almost never finds them; the LLM finds about half. Collect more examples.

And what not to use it for:

  • Not to tag papers automatically. Even with the LLM, only about 1 paper in 4 gets exactly the right set.
  • Not as a judge of the authors. A missing category is often a choice.
  • On real labels, the right threshold was far below 0.50. The cost table moved it from 0.50 to 0.07 and cut the cost by a third.
  • The long tail decides macro-F1. Big categories reach an F1 of 0.62; categories with 20 to 49 papers 0.14.
  • With many labels and few examples each, the LLM won. Its knowledge of the category names beat 70 trained models, most of all in the long tail.
  • TF-IDF still beat frozen embeddings, as at Larkfield, but every score is far lower.
  • Limits. Two years of papers, four areas, and a sample that is not a picture of arXiv as a whole. Titles and abstracts only. The labels are the authors’ choices. The cost table is an assumption, and the LLM was recorded for samples only.
  • A missed category hides a paper from its readers. The assistant can help early-career authors, but it can also make the habits of the big categories stronger. Watch which categories it never suggests.
  • Check who the assistant fails. We could only check areas and category sizes. With real submissions, also compare by the language of the abstract, as in Priority, urgency and language.
  • Abstracts are public, drafts are not. Before you send unpublished submissions to a hosted LLM, check the archive’s rules and the authors’ consent.
  • Credit the source. CC0 asks for nothing, but name arXiv and the snapshot, and say what you changed.

This case study shows that you can handle real multi-label data: label checks, the long tail, thresholds from a cost table, and an honest report per label. Make it yours with one change. For example, choose other areas, change the cost of a missed category, or add class weights for the rare categories. Then publish it so that someone can read it in five minutes.

What to put in the repository:

  • README.md, the page that people read first (template below).
  • The notebook, run from top to bottom, with its outputs.
  • thresholds.csv: your threshold for each category, and the number of validation papers behind it.
  • comparison.csv: every method on validation, and the test results apart.
  • per_category.csv: precision, recall and F1 per category on the test papers.
  • The data source and licence: arXiv’s metadata, CC0 1.0, the Hugging Face snapshot, and the SHA-256 of the sample.

A README template:

# Suggesting arXiv categories for paper abstracts, with thresholds chosen by cost
## Problem
An archive's authors choose categories for each paper; a person confirms. An assistant suggests
categories. A missed category costs $1.00, a wrong suggestion $0.20, a paper with no suggestion $1.50
(the person tags it alone). Assumptions.
## Data and licence
12,000 arXiv papers (2020-2021, areas cs, math, stat, eess) from gfissore/arxiv-abstracts-2021,
arXiv metadata, CC0 1.0. 70 learned categories; 51% of papers have more than one.
## Approach
TF-IDF + one logistic regression per category (one-vs-rest). Thresholds chosen on validation by cost:
one for all, and one per category. Compared with frozen MiniLM embeddings, their mean with TF-IDF,
and a hosted LLM (recorded, on a sample). Test used once.
## Results (test, used once)
TF-IDF, a threshold per category: $754.9 per 1,000 papers (a person tagging everything: $1,500),
micro-F1 0.592, macro-F1 0.434, 5.5% with no suggestion. Rare categories (20-49 papers): mean F1 0.14.
Test sample of 400: few-shot LLM $586.7 against $742.8 for TF-IDF; rare recall 0.500 against 0.107.
## Limits
Authors' own labels; abstracts only; four areas; assumed costs.

How to describe it in an interview, in two sentences: “I built an assistant that suggests arXiv categories for paper abstracts, with one yes/no model per category. Thresholds chosen from a cost table moved the best threshold from 0.50 to about 0.07 and halved the cost of tagging, and a few-shot LLM beat the trained models because it knows what the rare category names mean.”

Go further:

The notebook has three tasks with checks. Predict each answer before you run it.

1. How many papers have several labels? In section 2, compute the share of training papers with more than one category.

Show the answer
multi_share = round(float((train.tags.str.len() > 1).mean()), 3)
share of training papers with more than one category: 0.514

More than half, against 12.6% at Larkfield.

2. The break-even threshold. In section 5, compute the probability where suggesting and leaving out cost the same.

Show the answer
break_even = round(WRONG / (WRONG + MISSED), 3)
suggest a category when its probability is at least 0.167

0.20 ÷ (0.20 + 1.00) = 0.167. The cheapest threshold on validation is lower (0.07), because the model’s probabilities are too low.

3. A smaller review team. In section 6, find the cheapest single threshold when at most 2% of the papers may get no suggestion.

Show the answer
under = curve[curve.no_tag_share <= 0.02]
threshold_2 = float(under.loc[under.total_per_1000.idxmin(), "threshold"])
threshold with at most 2% to the person: 0.05
no_tag_share total_per_1000
0.01 760.3

0.05: 1.0% of the papers get no suggestion, for 760.3 dollars per 1,000, about 25 dollars more than at 0.07.

Check your understanding

With 0.50 as the threshold for every category, half of the validation papers get no suggestion. The cheapest threshold is 0.07. Why are the right thresholds so low here, when 0.50 worked for Larkfield's routes?

Show the answer

At Larkfield, the probabilities for the seven routes added up to 1, so the top route usually had a high score. Here, each of the 70 yes/no models learns one category against all the others. Most categories are on few papers, so “no” is usually right, and the probabilities stay low. The paper about tilings got 0.280 for math.CO and 0.126 for cs.IT, both right. And a missed category costs five times a wrong suggestion, so the break-even is already low (0.167). Choose the threshold on validation by cost, never by habit.

Check your understanding

The chosen router has a micro-F1 of 0.592 and a macro-F1 of 0.434 on the test papers. What does the gap tell you, and which number should Nadia look at?

Show the answer

Micro-F1 counts every suggestion together, so the big categories decide it. Macro-F1 weighs each of the 70 categories the same, so the rare ones pull it down. Categories with 20 to 49 training papers reach a mean F1 of only 0.139. The assistant works for the big lists and mostly fails for the small ones. Nadia should look at both, and at the results per category, because the average hides a small list that is almost never suggested.

Check your understanding

On validation, the thresholds per category beat one threshold by about 60 dollars per 1,000 papers. On the test papers, the gap is 16 dollars. Was choosing a threshold per category a mistake?

Show the answer

No, but the validation number was too good. The thresholds were chosen and scored on the same papers, so they fitted those papers’ noise. The test papers give the honest gap: still a gain, but a small one. Choose the thresholds on one set and score them on another, for example with cross-validation on the training papers. Keep the simpler rule when the gain is within the noise. Here, 16 dollars per 1,000 papers is about 32 missed categories on 1,999 papers: a real but modest difference.

Recap

  • Real multi-label data needs label checks: alias pairs, IDs that must stay text, categories outside the scope.
  • One yes/no model per category gave low probabilities. The cost table moved the threshold from 0.50 to 0.07 and cut the cost from 1,138.7 to 735.1 dollars per 1,000 validation papers.
  • A threshold per category won on the test papers too, but by 16 dollars, not 60: thresholds fitted on validation look better than they are.
  • The long tail decides macro-F1: categories with 300 or more papers reach an F1 of 0.62, those with 20 to 49 papers 0.14. The area level is far easier than the category level.
  • On the full test set, used once, the trained router cost 754.9 dollars per 1,000 papers and sent 5.5% to the person, against 1,500 for tagging everything.
  • On the test sample, the few-shot LLM was cheaper still (586.7 against 742.8), with a rare recall of 0.500 against 0.107. Fewer than 1 paper in 4 gets exactly the right set, so a person always confirms.

Next, the final assignment: a new channel and new products, and a routing system for the next quarter.

Next: Final assignment