Tag research papers with many labels
- Case study 4
- 150 min
- Optional · 1 case study needed for the certificate
Where we are
In nine modules, most messages had one label. In this optional case study, we use real open paper abstracts that often have several subject labels at once. We train one model per label, choose a threshold for each, and report the results per label.
Case study 4 of 4 · Text Classification and Routing: Send Each Message to the Right PlaceTag research papers with many labels
1. The routing problem and its cost
2. Labels you can trust
3. Baselines: rules and classical models
4. Embeddings and small fine-tuned models
5. LLMs as classifiers
6. Decision models, cascades and distillation
7. Harder label shapes
8. Confidence and people in the loop
9. Run it in production
What you will be able to do
Take 12,000 real paper abstracts to a decision for an assistant that suggests subject categories to a person. You will train one yes/no model per category, choose the thresholds from a cost table, study the long tail and two levels of labels, and compare three kinds of method fairly.
This case study is optional. You can choose it for the certificate.
The notebook runs every step of this page, plus three tasks with checks. Its setup cells install the packages and download the data and the course’s files (6 MB). The embedding model (about 470 MB) downloads once. It needs no account and no key: the LLM’s answers were recorded. After the install, it runs in about 3 minutes on a laptop.
Before you startWhat to know and to have ready
What you need to know
This case study builds on Module 7. These lessons matter most:
- One yes/no model per label, a threshold per label, and messages with no label, from Multi-label messages.
- Two levels of labels, from Hierarchical labels.
- Rare classes and macro-F1, from Imbalance and rare classes.
- TF-IDF, from Text as numbers: TF-IDF, and frozen embeddings, from Sentence embeddings for classification.
- The cost per 1,000 messages, from The cost of each mistake.
What you need on your computer
Nothing: the notebook runs in Colab or Kaggle. To run it locally, use a virtual environment with Python 3.12, pandas 3.1.1, scikit-learn 1.9.1, sentence-transformers 6.2.0 and PyTorch 2.15.0 (CPU). The setup cell makes a folder arxiv-papers. To start again, delete that folder. This case study does not change your larkroute project.
Tested on macOS (Apple silicon) with Python 3.12, in Jupyter from a fresh start. Colab, Kaggle, Windows and Linux are not tested.
The situation
Section titled “The situation”Imagine a university library that runs an open archive of research papers. It uses the subject categories of arXiv, the large open archive of science papers. Authors choose their paper’s categories from a long list, and volunteer moderators check the choice.
Nadia leads the moderators. Her problem: authors often choose too few categories, or the wrong ones. Then the readers of the missing lists never see the paper.
Kofi is a data scientist. Nadia asks him: “Can a model suggest the categories, so that a person only confirms them? How many papers get no useful suggestion? And what does it cost?”
We invented Nadia, Kofi and the library. The papers are real.
The important word is suggest. A person always looks at the paper, and the assistant saves that person time. Kofi writes the costs down with Nadia. They are assumptions:
- A right category that the assistant does not suggest costs 1.00 dollar. The person must find it in a list of 70 categories.
- A wrong suggestion costs 0.20 dollars. The person reads it and removes it, in about 10 seconds.
- A paper with no suggestion at all costs 1.50 dollars, the course’s cost of a review: about a minute of the person’s time.
So “a person tags everything”, the library’s process today, costs 1,500 dollars per 1,000 papers.
What you will build
Section titled “What you will build”You will build an assistant that reads a paper’s title and abstract and suggests every category that fits. When it is not sure of any category, it suggests nothing, and the person tags the paper alone. You will compare methods with one measure: the cost per 1,000 papers.
The data
Section titled “The data”The data is a sample of arXiv’s paper metadata: the ID, the title, the abstract and the categories of each paper. arXiv dedicates its metadata to the public domain (CC0 1.0). We use a snapshot on Hugging Face, gfissore/arxiv-abstracts-2021, with about 2 million papers.
We took one of its five files (363,006 papers from 2020 and 2021). We kept the papers whose main category is in one of four areas: computer science (cs), mathematics (math), statistics (stat) and electrical engineering (eess). Then we drew a fixed sample: 5,000, 4,000, 1,500 and 1,500 papers. The sample is 5 MB, and it comes with the notebook.
Each paper has one primary category, the first in its list, and it can have more: cs.LG cs.CV means machine learning, then computer vision.
We also compared the full arXiv metadata dump and GoEmotions, Reddit comments with emotion labels. The dataset card says why this sample won.
First checks
Section titled “First checks”The notebook loads the sample:
papers = pd.read_csv("data/papers.csv.gz", dtype={"id": str})The first checks print:
papers: 12,000 | areas: {'cs': 5000, 'math': 4000, 'eess': 1500, 'stat': 1500}missing values: 0 | repeated IDs: 0 | repeated abstracts: 1words per abstract: median 150, 90% have at most 229, longest 328categories per paper: {1: 5795, 2: 4161, 3: 1536, 4: 418, 5: 84, 6: 5, 8: 1}categories without a name: []What this tells us. The data is clean. The abstracts are about five times longer than Larkfield’s messages: a median of 150 words, against 28. More than half of the papers have two or more categories.
Note dtype={"id": str}. arXiv IDs look like numbers, but they are text. Read as a number, 1912.11930 becomes 1912.1193, and the paper is lost. Our first LLM recording had this bug.
Two names for one category
Section titled “Two names for one category”arXiv gives a few categories two names. For example, statistics theory is both math.ST and stat.TH. The notebook checks the raw lists:
math.ST and stat.TH: 284 and 284 papers; always together: Truecs.IT and math.IT: 381 and 381 papers; always together: Truemath.NA and cs.NA: 444 and 444 papers; always together: Trueeess.SY and cs.SY: 609 and 609 papers; always together: Truepapers that also had a category outside the four areas (dropped): 19.0%What this tells us. The two names always come together. If we kept both, a model would get a “free” second label every time, and the scores would look too good. So we kept one name per pair.
Many labels and a long tail
Section titled “Many labels and a long tail”Think of a library’s shelves. A few subjects fill whole walls, and many fill half a shelf. Some have three books. Labels work the same way: a few are very common, and many are rare. This shape is called a long tail. The notebook counts the training papers of each category:
train 8,002 | validation 1,999 | test 1,999 (used once, at the end)categories: 78 in train | learned: 70 | too rare (< 20 papers): 8 | rare (20 to 49 papers): 16most common: {'cs.LG': 1797, 'cs.CV': 1028, 'stat.ML': 792, 'cs.AI': 547, 'stat.ME': 536}too rare: {'math.GM': 18, 'cs.ET': 18, 'math.HO': 16, 'cs.MS': 13, 'cs.SC': 10, 'cs.OS': 3, 'cs.OH': 2, 'cs.GL': 1}Text description of the chart
Five columns, the number of categories in each size group, by training papers. 1 to 19 papers: 8 categories, in a second colour, too rare to learn. 20 to 49 papers: 16 categories. 50 to 99: 16. 100 to 299: 23. 300 or more: 15. The most common category, cs.LG, has 1,797 training papers; the rarest, cs.GL, has 1.
What this tells us. Machine learning (cs.LG) alone is on 1,797 of the 8,002 training papers. At the other end, 8 categories have fewer than 20 papers. A model cannot learn from 3 examples, so we do not train those 8, and the cost table counts them as missed. The 16 categories with 20 to 49 papers are the rare ones that we can learn, and we measure them on their own.
51.4% of the training papers have more than one category; at Larkfield, 12.6% of the messages held two requests.
Frame the problem
Section titled “Frame the problem”Kofi writes down his answers before he trains anything:
- Unit: one paper. It gets zero to five suggested categories.
- Labels: the 70 categories with at least 20 training papers. We also look at the four areas, the first level of the labels.
- One measure for the choice: the cost per 1,000 papers, with the cost table above.
- The split: train 8,002, validation 1,999, test 1,999, stratified by area. All choices are made on validation; the test papers are used once, at the end.
Micro-F1, macro-F1 and exact match
Section titled “Micro-F1, macro-F1 and exact match”A multi-label result needs new measures. Here are three, with a tiny example: 10 papers and three categories.
| Category | Right papers | Found | Wrong suggestions | F1 |
|---|---|---|---|---|
| cs.LG (common) | 8 | 7 | 1 | 2 × 7 ÷ (2 × 7 + 1 + 1) = 0.875 |
| stat.ML | 4 | 2 | 2 | 2 × 2 ÷ (2 × 2 + 2 + 2) = 0.500 |
| math.KT (rare) | 1 | 0 | 0 | 0 |
- Micro-F1 counts every suggestion of every category together: 9 found, 3 wrong, 4 missed. So 2 × 9 ÷ (2 × 9 + 3 + 4) = 0.72. Common categories decide it.
- Macro-F1 averages the three F1 scores: (0.875 + 0.5 + 0) ÷ 3 = 0.458. The rare category counts as much as the common one.
- Exact match is the share of papers whose whole set is exactly right, as in Multi-label messages.
The tables also show the recall of the rare categories, the share of papers with no suggestion (they go to the person), and the suggestions per paper.
A baseline: one yes/no model per category
Section titled “A baseline: one yes/no model per category”Kofi starts with the method of Multi-label messages: TF-IDF of words and word pairs, and one logistic regression per category (one-vs-rest): 70 yes/no models. They train in 11 seconds. With the usual threshold of 0.50 for every category, on the validation papers:
method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_10000 TF-IDF, 0.50 for every category 0.198 0.459 0.196 0.0 0.494 0.65 1138.71 everything to a person 0.000 0.000 0.000 0.0 1.000 0.00 1500.0What this tells us. At 0.50, half of the papers (49.4%) pass no threshold at all, and the person must tag them alone. The model suggests 0.65 categories per paper, but the papers have 1.7 on average. It saves only 361 dollars per 1,000 papers.
One paper, every value
Section titled “One paper, every value”One validation paper, “On Tilings of Asymmetric Limited-Magnitude Balls”, has math.CO (combinatorics) and cs.IT (information theory).
Text description of the diagram
On the left, the paper “On Tilings of Asymmetric Limited-Magnitude Balls”. Its authors chose math.CO and cs.IT. An arrow leads to the 70 yes/no models. Their scores: math.CO 0.280, cs.IT 0.126, math.DS 0.024, math.MP 0.023, and the 66 others 0.023 or less. One arrow, “the usual threshold”, leads to 0.50 for every category: nothing passes, and the person tags the paper alone. The other arrow, “this case study”, leads to a threshold per category: math.CO 0.280 is at least 0.07, yes; cs.IT 0.126 is at least 0.08, yes; math.MP 0.023 is below 0.05, no. The last box: suggest math.CO and cs.IT, both right, and the person confirms.
The model ranks the two right categories first and second, but both scores are far below 0.50. Each yes/no model saw few papers of its category, so its probabilities stay low. The problem is the threshold, not the ranking.
A threshold from the cost table
Section titled “A threshold from the cost table”Should the assistant suggest a category with a probability of 0.28? Think of an umbrella. Carrying it costs a little; getting wet costs more. So you take it even when rain is not likely.
The same reasoning works for a suggestion. Let p be the chance that the category is right.
- If the assistant suggests it, the cost is the chance that it is wrong times 0.20: (1 − p) × 0.20.
- If it does not, the cost is the chance that it is right times 1.00: p × 1.00.
The two costs are equal when (1 − p) × 0.20 = p × 1.00, so p = 0.20 ÷ 1.20 = 0.167. Above that, suggesting is cheaper. This is the break-even threshold, and it comes from the cost table, not from the model.
| p | Suggest: (1 − p) × 0.20 | Leave out: p × 1.00 | Cheaper |
|---|---|---|---|
| 0.10 | 0.18 | 0.10 | leave out |
| 0.167 | 0.167 | 0.167 | equal |
| 0.28 | 0.144 | 0.28 | suggest |
| 0.50 | 0.10 | 0.50 | suggest |
The break-even threshold holds only if the probabilities are true: 0.28 must mean “right 28 times in 100”. So Kofi tries every threshold from 0.05 to 0.90 on the validation papers:
- Total cost per 1,000 papers
- Missed categories (1.00 dollar each)
- Wrong suggestions (0.20 dollars each)
- No suggestion: the person tags it (1.50)
Text description of the chart
Four lines against the threshold, from 0.05 to 0.60. The total cost: 760.3 dollars per 1,000 at 0.05, lowest at 0.07 with 735.1, then 749.6 at 0.10, 857.9 at 0.20, 965.9 at 0.30, 1,138.7 at 0.50 and 1,210.4 at 0.60. Wrong suggestions fall fast: 404.4 at 0.05, 299.6 at 0.07, 209.1 at 0.10, 93.0 at 0.20 and 21.9 at 0.50. Papers tagged by the person alone rise fast: 14.3 at 0.05, 45.8 at 0.07, 99.8 at 0.10, 307.7 at 0.20 and 740.6 at 0.50. Missed categories rise from 341.7 at 0.05 to 457.2 at 0.20, then fall to 376.2 at 0.50, because more and more papers go to the person, who finds every category.
one threshold for every category: lowest cost at 0.07, 735.1 dollars per 1,000, 3.1% to a person (at 0.50: 1138.7)What this tells us. The cheapest threshold is 0.07, far below the break-even 0.167 and 0.50. The probabilities are too low: a category with 0.10 is right more often than 1 time in 10. At 0.07, the assistant suggests 2.79 categories per paper, 3.1% of the papers get no suggestion, and the cost falls from 1,138.7 to 735.1 dollars per 1,000.
A threshold per category
Section titled “A threshold per category”Kofi also chooses a threshold per category on the validation papers, by the same cost. A category with fewer than 10 right validation papers keeps 0.07.
thresholds per category: from 0.05 to 0.22, median 0.07; 22 categories keep 0.07 method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_10000 TF-IDF, a threshold per category 0.210 0.622 0.455 0.090 0.040 2.36 673.61 TF-IDF, 0.07, at least one 0.139 0.573 0.421 0.062 0.000 2.82 731.02 TF-IDF, 0.07 for every category 0.135 0.573 0.421 0.062 0.031 2.79 735.1What this tells us. On validation, the thresholds per category cost 673.6, about 60 dollars less than one threshold. But they were chosen on these same papers, so this row looks better than it will be. At Larkfield, thresholds per route lost on the test messages. The third row, “at least one”, always suggests the most likely category; it costs almost the same as sending those papers to the person.
If the person can tag at most 2% of the papers alone, the cheapest single threshold is 0.05, for 760.3 dollars per 1,000.
Two levels: area, then category
Section titled “Two levels: area, then category”The first part of cs.LG is the area, cs: the “team” of Hierarchical labels, with the category as the “topic” inside it.
Text description of the diagram
On the left, a paper, with its title and abstract. Four arrows lead to four area boxes. cs, computer science: 32 categories, for example cs.LG with 1,797 training papers, cs.CV with 1,028 and cs.AI with 547. math, mathematics: 29 categories, for example math.OC with 408, math.AP with 353 and math.CO with 350. stat, statistics: 5 categories, for example stat.ML with 792, stat.ME with 536 and stat.AP with 301. eess, electrical engineering: 4 categories, for example eess.SP with 430, eess.SY with 409 and eess.IV with 397.
24.6% of the training papers are in two areas, most often cs.LG with stat.ML. The notebook measures both levels with the threshold 0.07:
micro-F1 by category: 0.573 | by area: 0.820wrong suggestions: 2995, of which in an area that the paper has: 2047 (68%) method accuracy macro_f1 total_per_10000 area of the top category 0.812 0.752 750.41 a model for the 4 areas 0.842 0.794 632.3What this tells us. The area level is much easier: a micro-F1 of 0.820 against 0.573. Two of every three wrong suggestions are in the right area. Those are near misses, and a person fixes them fast.
The table also scores a routing problem with the course’s routing.py: send each paper to the moderators of its primary area, where a wrong area costs 4 dollars. A model trained on the 4 areas (0.842) beats the area of the most likely category (0.812). This turns around the lesson of Route banking questions: here, the fine labels mix the areas, so a direct area model wins.
Embeddings, and both together
Section titled “Embeddings, and both together”Two more methods learn on the same 8,002 training papers and are measured on the same 1,999 validation papers, with both kinds of threshold.
Frozen embeddings. The course’s older embedding model turns each text into 384 numbers, as in Sentence embeddings for classification, and one logistic regression per category learns on them. The model reads only the first 128 word pieces (parts of words): about the title and the first 80 words of the abstract. Embedding all 12,000 papers took 75 seconds on a laptop.
Both together. The mean of the TF-IDF and the embedding probabilities, for each category.
method exact_match micro_f1 macro_f1 rare_recall no_tag_share total_per_1000 ms_per_paper0 TF-IDF, a threshold per category 0.210 0.622 0.455 0.090 0.040 673.6 3.081 TF-IDF + embeddings, a threshold per category 0.164 0.602 0.471 0.250 0.032 707.5 15.914 TF-IDF + embeddings, 0.12 for every category 0.133 0.571 0.444 0.208 0.027 752.2 15.915 embeddings, a threshold per category 0.117 0.543 0.427 0.271 0.046 842.5 12.836 embeddings, 0.17 for every category 0.118 0.529 0.403 0.229 0.049 877.7 12.83What this tells us. Alone, the embeddings lose to TF-IDF, as on Larkfield’s inbox (0.920 against 0.962 there), but by more: 842.5 against 673.6 dollars. They read only the start of each abstract, and the deciding words can come later. But they find rare categories far better: a rare recall of 0.271 against 0.090. Together, the two models are the cheapest after TF-IDF alone, with almost three times its rare recall.
Among the trained routers, the cheapest on validation is still TF-IDF with a threshold per category. The notebook chooses it from the validation table.
The hosted LLM, recorded
Section titled “The hosted LLM, recorded”A hosted LLM (Azure chat-small) chose categories for 400 validation and 400 test papers. The prompt lists the 70 categories with their names, such as cs.DL: Digital Libraries. The answer is JSON: a list of categories, each with a confidence from 0 to 1. We recorded each answer once, so you replay them with no key.
The zero-shot prompt has only the list and the paper. The few-shot prompt also shows the titles and categories of the 5 most similar training papers, found by TF-IDF, as in Few-shot examples and label descriptions. Here is the start of a few-shot prompt:
Similar papers and their categories:- 'Elliptic fixed points with an invariant foliation: Some facts and more questions' -> math.DS- 'Almost non-degenerate functions and a Zariski pair of links' -> math.CV math.AG- 'Universality of the Hurwitz zeta-function on the half plane of absolute convergence' -> math.NT math.CV...Paper:Local Versions of the Wiener-Levi Theorem. Let h be a real-analytic function in the neighborhood ...zero-shot: 800 answers, all valid: True | tokens in 1195, out 39 | 0.139 dollars per 1,000 | median 1206 msfew-shot: 800 answers, all valid: True | tokens in 1322, out 38 | 0.151 dollars per 1,000 | median 1562 msAll 1,600 answers were valid. Each one costs about 0.015 cents and takes about 1.5 seconds.
The LLM’s written confidence gets a threshold too, chosen on the validation sample by cost. For the few-shot LLM, the cheapest is 0.01: keep every category that it names. For the zero-shot LLM, it is 0.7.
A fair comparison
Section titled “A fair comparison”The LLM has answers only for the samples, so the trained methods are scored on the same 400 validation papers:
method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_10000 LLM few-shot, confidence >= 0.01 (recorded) 0.207 0.649 0.559 0.517 0.000 2.34 615.21 LLM zero-shot, confidence >= 0.7 (recorded) 0.230 0.645 0.568 0.552 0.003 2.14 661.42 TF-IDF, a threshold per category 0.207 0.598 0.411 0.069 0.037 2.46 719.33 TF-IDF, 0.07 for every category 0.140 0.556 0.380 0.034 0.030 2.88 773.0What this tells us. Here the LLM wins, by a lot: 615.2 dollars per 1,000 papers for the few-shot LLM, against 719.3 for the best trained router. The largest gap is in the long tail: a rare recall of 0.517 against 0.069, and a macro-F1 of 0.559 against 0.411. And it never leaves a paper without a suggestion.
At Larkfield, the fine-tuned encoder beat the few-shot LLM. Here, the LLM has read about databases and K-theory long before this task. For a category with 30 training papers, TF-IDF has seen a few dozen examples; the LLM knows what the name means. The more labels you have, and the fewer examples each, the more that prior knowledge is worth.
400 papers is a small sample, but one missed category changes a row by only 2.5 dollars per 1,000. The gap is about 100 dollars, so it is not noise. The test sample below must still confirm it.
The test, once
Section titled “The test, once”Now Kofi opens the test papers. Everything was chosen on validation; nothing changes after this run.
method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_10000 TF-IDF, a threshold per category 0.172 0.592 0.434 0.101 0.055 2.43 754.91 TF-IDF, 0.07, at least one 0.126 0.563 0.428 0.094 0.000 2.88 767.52 TF-IDF, 0.07 for every category 0.120 0.563 0.426 0.094 0.033 2.84 770.73 TF-IDF + embeddings, a threshold per category 0.131 0.565 0.422 0.237 0.042 2.47 814.04 TF-IDF + embeddings, 0.12 for every category 0.118 0.552 0.424 0.216 0.032 2.72 815.95 embeddings, 0.17 for every category 0.107 0.509 0.375 0.230 0.051 2.52 953.06 embeddings, a threshold per category 0.096 0.505 0.376 0.245 0.048 2.54 959.67 TF-IDF, 0.50 for every category 0.196 0.445 0.184 0.000 0.502 0.64 1157.98 everything to a person 0.000 0.000 0.000 0.000 1.000 0.00 1500.0Text description of the chart
A column chart with five columns, the cost per 1,000 test papers. TF-IDF with a threshold per category: 755 dollars, the lowest, in the first colour. TF-IDF with 0.07 for every category: 771. TF-IDF and embeddings together: 816. Embeddings alone: 953. TF-IDF with 0.50 for every category: 1,158. Everything to a person, not drawn, would be 1,500.
What this tells us. The choice holds, but the gap shrank. TF-IDF with a threshold per category costs 754.9 dollars per 1,000 papers, against 770.7 for one threshold: 16 dollars, not 60 as on validation. The thresholds fitted the validation papers a little too well. Against 1,500 for a person, the assistant halves the cost. It suggests 2.43 categories per paper and sends 5.5% of the papers to the person.
At Larkfield, exact match for the messages with two requests was 0.952. Here it is 0.172: fewer than 1 paper in 5 gets exactly the right set. Real labels chosen by thousands of authors are far harder than labels written from templates.
The samples with recorded LLM answers tell the same story:
method exact_match micro_f1 macro_f1 rare_recall no_tag_share tags_per_paper total_per_10000 LLM few-shot, confidence >= 0.01 (recorded) 0.255 0.676 0.576 0.500 0.000 2.27 586.71 LLM zero-shot, confidence >= 0.7 (recorded) 0.235 0.642 0.539 0.393 0.000 2.16 689.62 TF-IDF, a threshold per category 0.163 0.600 0.421 0.107 0.052 2.43 742.83 TF-IDF, 0.07 for every category 0.100 0.568 0.400 0.107 0.040 2.86 756.5Text description of the chart
A column chart with four columns, the cost per 1,000 of the 400 test sample papers. The few-shot LLM: 587 dollars, the lowest. The zero-shot LLM: 690. TF-IDF with a threshold per category: 743. TF-IDF with 0.07 for every category: 757. The two LLM columns are in a third colour.
What this tells us. The few-shot LLM costs 586.7 dollars per 1,000 papers, 156 less than the best trained router, with a rare recall of 0.500 against 0.107. The zero-shot LLM, with the names only, costs 689.6.
Where the assistant still fails
Section titled “Where the assistant still fails”The notebook gives the F1 score of each category on the test papers, grouped by the number of training papers:
20 to 49 training papers: 16 categories, mean F1 0.139 50 to 99 training papers: 16 categories, mean F1 0.371 100 to 299 training papers: 23 categories, mean F1 0.560 300 to 1999 training papers: 15 categories, mean F1 0.620Text description of the chart
Four columns, the mean F1 per category on the test papers, by the number of training papers. 20 to 49 papers: 0.14. 50 to 99: 0.37. 100 to 299: 0.56. 300 or more: 0.62.
What this tells us. Categories with 300 or more training papers reach a mean F1 of 0.620, and those with 20 to 49 papers 0.139. Several reach 0, such as cs.DB (databases) and cs.DL (digital libraries). This is why macro-F1 (0.434) is so far below micro-F1 (0.592).
The most missed categories tell a second story:
category missed suggested_instead times0 cs.AI 74 cs.LG 561 math.OC 45 stat.ML 152 cs.LG 41 cs.CL 133 stat.ML 39 cs.LG 33cs.AI (artificial intelligence) is missed 74 times, with cs.LG (machine learning) suggested instead in 56. stat.ML is missed 39 times, with cs.LG in 33. These categories overlap.
Common mistake
Labels that depend on the author
Symptom: Kofi reads 20 test papers where the assistant suggested cs.LG but not stat.ML. For most, he cannot say why the authors chose stat.ML too.
Cause: the categories are the authors’ own choices. In the training papers, cs.LG stat.ML appears 140 times and stat.ML cs.LG 178 times, and many similar papers have only one of the two. The label is partly a habit of the author.
Fix: do not chase these points with a bigger model. Treat the pair as one decision: show stat.ML as a second suggestion next to cs.LG, and let the person choose. If the library wants consistency, write a guideline for the border, as in Labelling guidelines.
Consequence in AI work: with labels chosen by many people, part of every “error” is disagreement. Measure it before you promise a score, as in Measure agreement.
New abstracts
Section titled “New abstracts”The last step runs the assistant on three short abstracts that we wrote in the style of arXiv.
title suggested0 Counting lattice paths with forbidden steps math.CO 0.761 A small transformer for detecting faults in power grids eess.SY 0.37, eess.SP 0.332 Confidence intervals after model selection stat.ME 0.72, stat.AP 0.19, cs.LG 0.15What this tells us. The first suggestion is right. The second is half right: the abstract trains a transformer, and cs.LG is missing. For the third, the person would keep stat.ME, probably add math.ST, and remove cs.LG. A person stays in the loop.
Recommendation
Section titled “Recommendation”Kofi writes to Nadia in her words:
- Use the few-shot LLM to suggest categories, and keep every category that it names. On the test sample, it cost 586.7 dollars per 1,000 papers. The best trained router cost 742.8, and a person who tags everything 1,500. It finds the rare categories far better. It costs about 0.15 dollars per 1,000 papers to run and takes about 1.5 seconds.
- Keep the trained router as the fallback: TF-IDF with a threshold per category (0.05 to 0.22). It runs inside the library, in 3 milliseconds. Use it when the LLM service is down, and for drafts that may not leave the library. On the full test set, it cost 754.9 dollars per 1,000 papers, with 5.5% sent to the person.
- Show the suggestions as suggestions. With the trained router, only 51% of the suggestions are right, but they cover 71% of the right categories. A wrong suggestion costs a few seconds; a missed category costs a reader.
- Show the area first. The area is right far more often (micro-F1 0.820 against 0.573), and most missing categories are in the suggested area.
- Watch the rare categories. 24 categories have fewer than 50 training papers. The trained router almost never finds them; the LLM finds about half. Collect more examples.
And what not to use it for:
- Not to tag papers automatically. Even with the LLM, only about 1 paper in 4 gets exactly the right set.
- Not as a judge of the authors. A missing category is often a choice.
Insights and limits
Section titled “Insights and limits”- On real labels, the right threshold was far below 0.50. The cost table moved it from 0.50 to 0.07 and cut the cost by a third.
- The long tail decides macro-F1. Big categories reach an F1 of 0.62; categories with 20 to 49 papers 0.14.
- With many labels and few examples each, the LLM won. Its knowledge of the category names beat 70 trained models, most of all in the long tail.
- TF-IDF still beat frozen embeddings, as at Larkfield, but every score is far lower.
- Limits. Two years of papers, four areas, and a sample that is not a picture of arXiv as a whole. Titles and abstracts only. The labels are the authors’ choices. The cost table is an assumption, and the LLM was recorded for samples only.
Responsible use
Section titled “Responsible use”- A missed category hides a paper from its readers. The assistant can help early-career authors, but it can also make the habits of the big categories stronger. Watch which categories it never suggests.
- Check who the assistant fails. We could only check areas and category sizes. With real submissions, also compare by the language of the abstract, as in Priority, urgency and language.
- Abstracts are public, drafts are not. Before you send unpublished submissions to a hosted LLM, check the archive’s rules and the authors’ consent.
- Credit the source. CC0 asks for nothing, but name arXiv and the snapshot, and say what you changed.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This case study shows that you can handle real multi-label data: label checks, the long tail, thresholds from a cost table, and an honest report per label. Make it yours with one change. For example, choose other areas, change the cost of a missed category, or add class weights for the rare categories. Then publish it so that someone can read it in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
thresholds.csv: your threshold for each category, and the number of validation papers behind it.comparison.csv: every method on validation, and the test results apart.per_category.csv: precision, recall and F1 per category on the test papers.- The data source and licence: arXiv’s metadata, CC0 1.0, the Hugging Face snapshot, and the SHA-256 of the sample.
A README template:
# Suggesting arXiv categories for paper abstracts, with thresholds chosen by cost
## ProblemAn archive's authors choose categories for each paper; a person confirms. An assistant suggestscategories. A missed category costs $1.00, a wrong suggestion $0.20, a paper with no suggestion $1.50(the person tags it alone). Assumptions.
## Data and licence12,000 arXiv papers (2020-2021, areas cs, math, stat, eess) from gfissore/arxiv-abstracts-2021,arXiv metadata, CC0 1.0. 70 learned categories; 51% of papers have more than one.
## ApproachTF-IDF + one logistic regression per category (one-vs-rest). Thresholds chosen on validation by cost:one for all, and one per category. Compared with frozen MiniLM embeddings, their mean with TF-IDF,and a hosted LLM (recorded, on a sample). Test used once.
## Results (test, used once)TF-IDF, a threshold per category: $754.9 per 1,000 papers (a person tagging everything: $1,500),micro-F1 0.592, macro-F1 0.434, 5.5% with no suggestion. Rare categories (20-49 papers): mean F1 0.14.Test sample of 400: few-shot LLM $586.7 against $742.8 for TF-IDF; rare recall 0.500 against 0.107.
## LimitsAuthors' own labels; abstracts only; four areas; assumed costs.How to describe it in an interview, in two sentences: “I built an assistant that suggests arXiv categories for paper abstracts, with one yes/no model per category. Thresholds chosen from a cost table moved the best threshold from 0.50 to about 0.07 and halved the cost of tagging, and a few-shot LLM beat the trained models because it knows what the rare category names mean.”
Go further:
- Train the rare categories with class weights, as in Imbalance and rare classes. Does the rare recall rise, and what does it cost?
- Build the two-step version: the area model first, then the categories of the chosen areas only.
- Choose the thresholds per category with cross-validation on the training papers instead of on validation. Does the gap between validation and test shrink?
- Compare with the other case studies: Route banking questions, Find what the bot cannot handle and Sort requests with few labels.
Your turn
Section titled “Your turn”The notebook has three tasks with checks. Predict each answer before you run it.
1. How many papers have several labels? In section 2, compute the share of training papers with more than one category.
Show the answer
multi_share = round(float((train.tags.str.len() > 1).mean()), 3)share of training papers with more than one category: 0.514More than half, against 12.6% at Larkfield.
2. The break-even threshold. In section 5, compute the probability where suggesting and leaving out cost the same.
Show the answer
break_even = round(WRONG / (WRONG + MISSED), 3)suggest a category when its probability is at least 0.1670.20 ÷ (0.20 + 1.00) = 0.167. The cheapest threshold on validation is lower (0.07), because the model’s probabilities are too low.
3. A smaller review team. In section 6, find the cheapest single threshold when at most 2% of the papers may get no suggestion.
Show the answer
under = curve[curve.no_tag_share <= 0.02]threshold_2 = float(under.loc[under.total_per_1000.idxmin(), "threshold"])threshold with at most 2% to the person: 0.05 no_tag_share total_per_1000 0.01 760.30.05: 1.0% of the papers get no suggestion, for 760.3 dollars per 1,000, about 25 dollars more than at 0.07.
Knowledge check
Section titled “Knowledge check”Check your understanding
With 0.50 as the threshold for every category, half of the validation papers get no suggestion. The cheapest threshold is 0.07. Why are the right thresholds so low here, when 0.50 worked for Larkfield's routes?
Show the answer
At Larkfield, the probabilities for the seven routes added up to 1, so the top route usually had a high score. Here, each of the 70 yes/no models learns one category against all the others. Most categories are on few papers, so “no” is usually right, and the probabilities stay low. The paper about tilings got 0.280 for math.CO and 0.126 for cs.IT, both right. And a missed category costs five times a wrong suggestion, so the break-even is already low (0.167). Choose the threshold on validation by cost, never by habit.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The chosen router has a micro-F1 of 0.592 and a macro-F1 of 0.434 on the test papers. What does the gap tell you, and which number should Nadia look at?
Show the answer
Micro-F1 counts every suggestion together, so the big categories decide it. Macro-F1 weighs each of the 70 categories the same, so the rare ones pull it down. Categories with 20 to 49 training papers reach a mean F1 of only 0.139. The assistant works for the big lists and mostly fails for the small ones. Nadia should look at both, and at the results per category, because the average hides a small list that is almost never suggested.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
On validation, the thresholds per category beat one threshold by about 60 dollars per 1,000 papers. On the test papers, the gap is 16 dollars. Was choosing a threshold per category a mistake?
Show the answer
No, but the validation number was too good. The thresholds were chosen and scored on the same papers, so they fitted those papers’ noise. The test papers give the honest gap: still a gain, but a small one. Choose the thresholds on one set and score them on another, for example with cross-validation on the training papers. Keep the simpler rule when the gain is within the noise. Here, 16 dollars per 1,000 papers is about 32 missed categories on 1,999 papers: a real but modest difference.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- Real multi-label data needs label checks: alias pairs, IDs that must stay text, categories outside the scope.
- One yes/no model per category gave low probabilities. The cost table moved the threshold from 0.50 to 0.07 and cut the cost from 1,138.7 to 735.1 dollars per 1,000 validation papers.
- A threshold per category won on the test papers too, but by 16 dollars, not 60: thresholds fitted on validation look better than they are.
- The long tail decides macro-F1: categories with 300 or more papers reach an F1 of 0.62, those with 20 to 49 papers 0.14. The area level is far easier than the category level.
- On the full test set, used once, the trained router cost 754.9 dollars per 1,000 papers and sent 5.5% to the person, against 1,500 for tagging everything.
- On the test sample, the few-shot LLM was cheaper still (586.7 against 742.8), with a rare recall of 0.500 against 0.107. Fewer than 1 paper in 4 gets exactly the right set, so a person always confirms.
Next, the final assignment: a new channel and new products, and a routing system for the next quarter.
Next: Final assignment