Agent-ready reports from real open-source issues
- Case study 2
- 120 min
- Optional · 1 case study needed for the certificate
Where we are
This case study takes the agent-ready report from Module 5 to real open-source issues. You measure how often real reports say enough for a coding agent, and you rewrite the weak ones.
Case study 2 of 3 · Testing AI Applications: Manual and AI-Assisted QAAgent-ready reports from real open-source issues
1. How AI features fail
2. Plan and judge without one right answer
3. Explore and reproduce
4. Report bugs people can act on
5. Report bugs a coding agent can fix
6. Test retrieval, agents and safety
7. AI-assisted QA and regression protection
What you will be able to do
Find out how often real issues, as their reporters wrote them, say enough for a coding agent to make the fix that the maintainers accepted. You rate 30 real issues with a short rubric and check each one against its merged fix and test. Then you compare ways to decide which issues can go to an agent as written. Last, you rewrite weak issues in the agent-ready format, and check one rewrite against the real test by running it.
The notebook runs every step of this page, with the same code and the same output, plus four tasks with checks. It downloads nine small files and uses only the Python standard library. It runs in Colab or Kaggle with no setup, in a few seconds. You need no account and no key: the model ratings were recorded once, and the notebook replays them. One optional part downloads Django and runs the real test.
Before you startWhat to know and to have ready
What you need to know
This case study uses Module 5 most. These lessons matter most:
- The agent-ready template and its fields, from Agent-ready issue templates.
- What an agent cannot guess, from What a coding agent needs.
- A failing case that fails today and must pass after the fix, from Turn a bug into a failing case.
- Neighbours and shortcuts, from Verify the agent’s fix.
- Rubrics that two people apply the same way, from Rubrics and property checks.
- Checking a model rater against your own labels, from Triage many outputs with a model judge.
What you need on your computer (only for the local route; Colab and Kaggle need none of it)
- Python 3.10 or newer, and the notebook, downloaded from GitHub, in VS Code or JupyterLab. The practice app and your QA workspace are not needed.
- The notebook’s first cell downloads the data into a new folder
agent-ready-issues. To start again, delete that folder. - For the optional Django part only: Python 3.10 to 3.12 and the
patchprogram. macOS and Linux have it; on Windows, use WSL or skip the part.
Tested on macOS with Python 3.12 and 3.13, in Jupyter from a fresh start. Windows, Linux, Colab and Kaggle are not tested.
The situation
Section titled “The situation”Amara (she) is the QA lead of a small team that maintains open-source Python libraries. The team has a long backlog of issues from users. Her manager, Jonas (he), has a plan: “We have a coding agent now. Let us give it the open issues as they are, and review what comes back.”
Amara has seen what an agent does with a vague report: it guesses. She asks for evidence first. “Before we hand over hundreds of issues, let me measure two things. How often does an issue, as written, say enough to make the fix that maintainers accept? And can a short rubric find those issues?”
Each wrong decision costs something. An issue that goes to the agent too early wastes an agent run, a review and a reopened issue. An issue that is held back for a rewrite costs a tester a few minutes. Jonas wants speed; Amara wants fewer wasted runs. Amara, Jonas and their team are made up for this course; the issues are real.
What you will build
Section titled “What you will build”You will build a readiness study of real issues and a triage rule from it, in the steps of the picture below. At the end, you run the whole method on one new issue.
The data
Section titled “The data”SWE-bench is a public set of real issues from GitHub. Each issue comes with the change that fixed it, as the maintainers merged it, and with the test that came with the change. Researchers use it to measure coding agents: an agent gets the issue text, and the hidden test decides whether its fix works. SWE-bench Lite is a smaller subset: 300 issues from 12 well-known Python projects. Among them are Django, SymPy, pytest and Flask.
The course compared Lite with SWE-bench Multilingual (nine languages) and SWE-Gym (2,438 unfiltered Python issues). Lite won because it is small, Python only and pinned to one version. The dataset card records the comparison and the checksums.
The sample has 31 issues. Thirty are for the study, drawn so that every project appears: 6 from Django, 5 from SymPy, and 1 to 3 from each other project. One more Django issue is kept apart until the end, as the new input.
| Field | What it means |
|---|---|
instance_id, repo, pull_request |
The issue’s ID, its project, and a link to the merged pull request |
problem_statement |
The issue text, as the reporter wrote it |
patch |
The fix that the maintainers merged (a diff) |
test_patch |
The test change that came with the fix (a diff) |
fail_to_pass |
The tests that fail before the fix and pass after it |
base_commit, version |
The commit before the fix, and the project’s version line |
First checks
Section titled “First checks”The notebook loads the sample and counts it:
issues = read_jsonl("issues_sample.jsonl")by_id = {i["instance_id"]: i for i in issues}sample = [i for i in issues if i["role"] == "sample"]new_input = next(i for i in issues if i["role"] == "new-input")print(len(sample), "issues to rate, 1 kept apart:", new_input["instance_id"])print("words per issue text, median:", median(len(i["problem_statement"].split()) for i in sample))The output:
30 issues to rate, 1 kept apart: django__django-11133words per issue text, median: 120.5What this tells us. A typical issue has about 120 words: a few paragraphs, often with a code snippet or an error. Thirty of them are quick to rate by hand.
Frame the problem
Section titled “Frame the problem”Amara and Jonas agree on four things before anyone rates an issue.
- The unit: one issue, as written, before any discussion.
- The decision: send it to the agent as written, or hold it for a rewrite.
- What counts as “ready”. The merged test is the maintainers’ own acceptance check. An issue is ready when that test checks what the issue text expects. If the test checks something the text never says, an agent that follows the text can fail it.
- Two passes. First, rate every issue from its text alone. Only then open the fixes and tests. Otherwise the answer key would leak into the ratings.
The idea: the merged test is the answer key
Section titled “The idea: the merged test is the answer key”Think of a tailor’s order slip. The customer writes “shorten the trousers”. The tailor shortens them by 3 cm. At the fitting, the customer wanted 5 cm. The slip was not wrong, but the real check, the fitting, tested something the slip did not say. A merged test is the fitting. The maintainers wrote it when they fixed the issue. The reporter never saw it. So it shows, after the fact, what “fixed” really meant. When the issue text and the test agree, an agent had a fair chance. When they do not, even a perfect reader of the text could fail.
Text description of the diagram
Eight steps. The data has 300 issues; the sample takes 30, plus 1 kept apart. Pass 1 rates the text alone: the course author once, a model three times. Pass 2 opens the answer key. Then come the comparison of triage rules, three rewrites, a real run of one rewrite, and the new input.
The rubric
Section titled “The rubric”The rubric turns the agent-ready template of Agent-ready issue templates into five criteria. Each one scores 0, 1 or 2, from the text alone. The template was written for AI features, so three fields change for a code library. “Model and prompt version” becomes “library version or commit”. “The attached conversation” becomes “the code that shows the bug”. “Suspected layer” becomes “suspected place in the code”.
| Criterion | The question | 2 | 1 | 0 |
|---|---|---|---|---|
| Reproduction | Can a fixer run the failing case from the text? | Runs as written, and the actual output is shown | Something to start from, but the fixer must guess | None |
| Versions | Which build did it happen on? | Library version or commit, and the environment that matters | Partly, or vague (“current”) | None |
| Expected | What must happen instead? | An exact value, or a clear Must / Must not rule | Implied, vague or undecided | Not stated |
| Acceptance | How do we know it is done? | A done condition beyond the one example | Only the one example | Nothing to check |
| Scope | Where is the problem; what must not change? | The suspected cause or code place, or limits | Only the feature or API | Only the symptom |
The total runs from 0 to 10. Before she looks at any test, Amara sets a bar. An issue with 7 or more has mostly full marks, and it may go to the agent as written.
One issue, by hand
Section titled “One issue, by hand”Here is a short Django issue, django__django-15851, in full:
dbshell additional parameters should be passed before dbname on PostgreSQL.Descriptionpsql expects all options to proceed the database name, if provided. So, if doing something like`./manage.py dbshell -- -c "select * from some_table;" one will get this:$ ./manage.py dbshell -- -c "select * from some_table;"psql: warning: extra command-line argument "-c" ignoredpsql: warning: extra command-line argument "select * from some_table;" ignoredpsql (10.21)Type "help" for help.some_database=>It appears the args list just need to be constructed in the proper order, leaving the database namefor the end of the args list.Rate it before you read on. The course author’s scores, with each value visible:
- Reproduction 2: an exact command, and the actual output.
- Versions 1: psql 10.21, but no Django version.
- Expected 2: a clear rule: the options come before the database name.
- Acceptance 1: only this one example can serve as the check.
- Scope 2: it names the cause: the order of the argument list.
The total is 8 of 10, so the issue clears the bar. Later, the answer key agrees: the merged fix moves the extra arguments before the database name, and the merged test checks exactly that order.
Pass 1: what the texts say
Section titled “Pass 1: what the texts say”The course author rated all 30 issues from their text alone, with a one-line reason for each score. The author committed the ratings before opening any fix. The notebook counts the scores:
for c in CRITERIA: counts = Counter(author[i][c] for i in ids) print(f"{c:13} 2: {counts[2]:2} 1: {counts[1]:2} 0: {counts[0]:2}")totals = [total(author[i]) for i in ids]print("total: median", median(totals), "| 7 or more:", sum(t >= 7 for t in totals), "of", len(ids))The output:
reproduction 2: 21 1: 5 0: 4versions 2: 9 1: 5 0: 16expected 2: 16 1: 13 0: 1acceptance 2: 2 1: 24 0: 4scope 2: 12 1: 18 0: 0total: median 6.0 | 7 or more: 13 of 30Text description of the chart
A column chart with five columns, one per criterion: reproduction 21 of 30, versions 9, expected 16, acceptance 2, scope 12.
What this tells us. Reporters are good at showing the problem: 21 of 30 texts have a runnable example with its output. They are poor at the parts a fixer needs next. 16 texts give no version at all, and only 2 give a done condition beyond their one example. The expected result is clear in about half (16). The median total is 6, and 13 issues reach the bar of 7.
Pass 2: the answer key
Section titled “Pass 2: the answer key”Now the course author opened each merged fix and test, and answered two questions per issue. Does the text point to the place of the fix? Does the merged test check what the text expects? Each answer is yes, partly or no, with a reason.
One partly shows the problem well. In pallets__flask-4992, a user asks for a way to open a config file in binary mode:
flask = by_id["pallets__flask-4992"]print([line for line in flask["problem_statement"].splitlines() if "mode=" in line])print([line.strip() for line in added_lines(flask["test_patch"]) if "from_file" in line or "text=" in line])The output:
['app.config.from_file("config.toml", tomllib.load, mode="b")']['def test_config_from_file_json():', 'def test_config_from_file_toml():', 'app.config.from_file(', 'os.path.join(current_dir, "static", "config.toml"), tomllib.load, text=False']What this tells us. The text proposes mode="b". The maintainers chose text=False, and their test calls it that way. An agent that follows the text exactly writes a working feature and fails the test. The issue scored 8: it was clear, but it stated a detail that the maintainers decided differently.
Over all 30 issues:
does the test check what the text expects? Counter({'partly': 16, 'yes': 10, 'no': 4})does the text point to the fix? Counter({'partly': 14, 'yes': 14, 'no': 2})yes 10 issues, rubric totals [6, 6, 7, 7, 7, 8, 8, 9, 9, 9]partly 16 issues, rubric totals [3, 4, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 7, 8, 8, 9]no 4 issues, rubric totals [2, 2, 3, 8]Text description of the chart
Three rows of dots. Test matches the text, 10 issues: totals 6, 6, 7, 7, 7, 8, 8, 9, 9, 9 (median 7.5). Partly, 16 issues: totals from 3 to 9 (median 6). No, 4 issues: totals 2, 2, 3 and 8 (median 2.5).
What this tells us. Only 10 of 30 issues are ready: their merged test checks what the text expects. On 16, the test checks more, or a detail that the text does not give. Examples are a parameter name, an output format, an error type or an extra case. On 4, it checks something else. The rubric totals rise from “no” to “yes”, but the groups overlap. The one high “no” is an xarray issue (8 points). Its text was clear, but the maintainers fixed a different problem: they added a warning instead of keeping the data lazy.
The expected criterion separates best: 9 of the 10 ready issues have full marks on it, against 7 of the other 20. The acceptance criterion is almost never met, by ready issues or not.
A computer can check part of “points to the fix”: does the text name a file, function or class that the merged diff changes? It named one in 20 issues. Where the author judged “points to the fix: yes”, 12 of 14 texts named such a place.
Common mistake
A name in the text is not a pointer to the fix
The automatic check counts a name when the same word appears in the text. In the new input of this study, it finds response: the fix changes django/http/response.py, and the reporter wrote response = HttpResponse(...). That is a variable in a code example, not a pointer to a file. Treat the automatic check as a hint for a person, not as a verdict.
Triage rules under one measure
Section titled “Triage rules under one measure”Now the decision. A triage rule sends some issues to the agent as written and holds the others for a rewrite. Each rule makes two kinds of mistake: it sends an issue that is not ready, or it holds an issue that was ready. One measure for every rule: the number of misrouted issues, both kinds added together.
Two baselines frame the comparison. “Send everything” is Jonas’s plan. “Rewrite everything first” is the cautious opposite. Two cheap rules need no rubric at all: “has code or a traceback”, and “long text”. Then come two rubric rules: the total of 7 or more, and full marks on expected alone.
for name, flags in rules.items(): r = triage(flags, ready) print(f"{name:29} sent {r['sent']:2} ({r['sent_ready']:2} ready) | misrouted {r['misrouted']:2}")The output:
send everything sent 30 (10 ready) | misrouted 20rewrite everything first sent 0 ( 0 ready) | misrouted 10has code or a traceback sent 19 ( 7 ready) | misrouted 15long text (above the median) sent 15 ( 5 ready) | misrouted 15rubric total 7+ sent 13 ( 8 ready) | misrouted 7expected = 2 sent 16 ( 9 ready) | misrouted 8What this tells us. Sending everything misroutes 20 issues. Two in three would reach the agent with a text that does not match the maintainers’ test. A code block or a long text does not help much (15). The rubric total misroutes 7: it sends 13 issues, and 8 of them are ready. Full marks on expected alone does almost as well (8). With 30 issues, a difference of one or two is within chance, so read these two as equal.
Was the bar of 7 a lucky choice? The notebook moves it from 3 to 10:
- Course author
- Model
- Send everything
- Rewrite everything first
Text description of the chart
Two solid lines and two dashed lines. For bars from 3 to 10, the course author’s ratings misroute 18, 16, 15, 10, 7, 9, 8 and 10 issues. The model’s ratings misroute 18, 15, 14, 11, 9, 7, 10 and 10. The dashed baselines stay at 20 (send everything) and 10 (rewrite everything first).
What this tells us. Bars of 7 to 9 all misroute 7 to 9 issues; 7 was a reasonable choice, not a lucky one. Below 6, the rule sends too much. And no bar beats “rewrite everything first” by much: 7 against 10.
Change one thing: the cost of each mistake
Section titled “Change one thing: the cost of each mistake”Equal weights treat a wasted agent run like a few minutes of rewriting. Amara thinks a wasted run costs more: the agent’s change, a review, a reopened issue. As an assumption, she counts a wasted run as 3 and an unneeded rewrite as 1:
send everything weighted cost 60rewrite everything first weighted cost 10has code or a traceback weighted cost 39long text (above the median) weighted cost 35rubric total 7+ weighted cost 17expected = 2 weighted cost 22What this tells us. With these weights, rewriting every issue first is the cheapest rule (10), and the rubric gate costs 17. The reason is the base rate: only 10 of 30 issues were ready, so a gate that sends 13 still sends 5 that are not. This changes the recommendation. When agent runs are expensive and rewrites are cheap, the rubric is more useful as a checklist for the rewrite than as a gate.
A model applies the rubric
Section titled “A model applies the rubric”Amara will not rate hundreds of issues by hand. Can a model apply the rubric? The course recorded chat-small, a small hosted model on Azure, on the same 30 issues. It got the rubric and the issue text only. The course asked each issue three times, because a model’s answer can change between runs. The 93 calls cost about US$0.03; prices change, so check the current page.
for c in CRITERIA: same = sum(model_median[i][c] == author[i][c] for i in ids) lower = sum(model_median[i][c] < author[i][c] for i in ids) stable = sum(len({run[c] for run in model[i]}) == 1 for i in ids) print(f"{c:13} same as the author {same:2}/30, model lower {lower}, same in all 3 runs {stable:2}/30")changed = sum(len({total(run) for run in model[i]}) > 1 for i in ids)print("issues whose model total changed between runs:", changed)r = triage([total(model_median[i]) >= 7 for i in ids], ready)print("model rubric total 7+: sent", r["sent"], "| misrouted", r["misrouted"])The output:
reproduction same as the author 21/30, model lower 8, same in all 3 runs 27/30versions same as the author 26/30, model lower 3, same in all 3 runs 28/30expected same as the author 26/30, model lower 2, same in all 3 runs 27/30acceptance same as the author 26/30, model lower 3, same in all 3 runs 25/30scope same as the author 26/30, model lower 4, same in all 3 runs 28/30issues whose model total changed between runs: 10model rubric total 7+: sent 7 | misrouted 9Text description of the chart
A column chart with five columns: reproduction 21 of 30, versions 26, expected 26, acceptance 26, scope 26.
What this tells us. The model agrees with the author on 26 of 30 issues for four criteria. Reproduction is the exception, at 21: the model is stricter, and it scored lower on 8 issues. Its reasons show why: it wanted the setup written out, such as a Django project, a database or a config file. The author had counted that setup as trivial. The model is not fully stable either: its total changed between runs on 10 of 30 issues. As a gate, the model’s ratings misroute 9 issues, close to the author’s 7. As in Triage many outputs with a model judge, these numbers come from checking the model against your own labels first. Without that check, you would not know that it is strict on reproduction.
Rewrite three weak issues
Section titled “Rewrite three weak issues”The course author rewrote three weak issues in the agent-ready format. They are a Django crash (django__django-15213, total 5), a missing Flask check (pallets__flask-4045, total 5) and a pytest feature request (pytest-dev__pytest-5221, total 3). The author wrote them after pass 2, and so knew the merged fixes. To limit that hindsight, each rewrite uses only the issue text and a real run at the issue’s base commit. Whatever the maintainers had to decide stays an open question, in a field the template does not have: Decision needed.
Here is the Django issue, side by side with its rewrite. The actual output in the rewrite comes from a real run on Django at the base commit.
| Template field | The original issue | The rewrite |
|---|---|---|
| Summary | “ExpressionWrapper for ~Q(pk__in=[]) crashes.” | The annotation builds invalid SQL (SELECT AS "selected"), and the query fails. |
| Environment | None | Django main at commit 03cadb9 (4.1 in development), Python 3.12, SQLite |
| Steps | queryset.annotate(...), with “table” as the model |
The same annotation on the Author model of Django’s own annotations test app |
| Actual | SELECT AS "foo" FROM "table" |
SELECT AS "selected", then OperationalError: near "AS": syntax error |
| Expected | Not stated: only “the query breaks” | Must: selected is True for every row. Must: Q(pk__in=[]) stays False. Must not: change what filter() returns. |
| Reproduction rate | None | 3 of 3 runs |
| Suspected place | None | A guess: the negated empty condition compiles to an empty SQL string |
| Acceptance criteria | None | The attached failing case passes (3 of 3 tests); the annotations suite passes; no existing test changed |
| Constraints | None | Change only django/db/models/; do not special-case pk__in=[]; do not edit existing tests |
The two other rewrites hit a limit that no tester can work around. The Flask issue asks for “an error” when a blueprint name contains a dot. The existing check for endpoint names uses assert, which raises AssertionError. Should the new check do the same, or raise ValueError? The rewrite does not guess: it asks the maintainers under Decision needed. The pytest rewrite leaves the exact output format and “show the default scope or not” as decisions in the same way.
Rated with the same rubric, the totals rise: Django from 5 to 10, Flask from 5 to 9, pytest from 3 to 8. The two open decisions keep expected at 1. The model’s median totals for the rewrites are 9, 10 and 7. It gave the Flask rewrite full marks on expected, although its own reason said that the error type was open.
Check one rewrite against the merged fix and test
Section titled “Check one rewrite against the merged fix and test”Is the Django rewrite right? Its acceptance criteria became a failing case, as in Turn a bug into a failing case: three Django tests, one per rule. The course author downloaded Django at the issue’s base commit. Then the author ran the case, the merged test, the merged fix, and both again. The recorded output, shortened:
### 0. Reproduce: the SQL on the base commitQ(pk__in=[]) -> SELECT 0 AS "selected" FROM "auth_user"~Q(pk__in=[]) -> SELECT AS "selected" FROM "auth_user"### 1. The rewrite's case on the base commitRan 3 tests in 0.002sFAILED (errors=1)### 2. The merged test on the base commitRan 3 tests in 0.006sFAILED (errors=3)### 3. The SQL after the merged fixQ(pk__in=[]) -> SELECT 0 AS "selected" FROM "auth_user"~Q(pk__in=[]) -> SELECT 1 AS "selected" FROM "auth_user"### 3a. The rewrite's case after the merged fixRan 3 tests in 0.001sOK### 3b. The merged test after the merged fixRan 3 tests in 0.006sOK### 3c. The whole annotations suite after the fixRan 80 tests in 0.028sOK (skipped=3)What this tells us. The rewrite’s case behaves as a failing case must. It fails on the base commit, with the same error as the merged test, and it passes after the merged fix. The two passing tests before the fix are the “must keep working” rules: they guard against a fix that breaks something else. After the fix, the whole annotations suite passes too.
Does the rewrite’s acceptance criterion match what the real test checks? Rule by rule:
Text description of the diagram
Three rows and one group. The rewrite’s first Must is the same check as the merged test test_full_expression_annotation. Its second Must is the same check as an existing test. Its Must not is checked only by the rewrite. Two merged tests, the annotation inside Count() and Sum() over the annotation, are neighbours that the rewrite did not name.
What this tells us. The core matches: the rewrite’s first Must is the merged test’s main assertion, “every row is selected”. The maintainers also tested two neighbours: the same annotation inside an aggregation (Count(), which adds a GROUP BY) and summed (Sum()). The rewrite did not name them. An agent’s fix could pass the rewrite’s case and still break inside a GROUP BY. That is why Verify the agent’s fix runs the neighbours, not only the failing case. A better rewrite adds one line: “Must: the same annotation works inside Count() and Sum().”
End to end: a new issue
Section titled “End to end: a new issue”The issue kept apart, django__django-11133, now goes through the whole method. Its text says that HttpResponse(memoryview(b"My Content")).content gives b'<memory at 0x7fcc47ab2648>', and the reporter expects b'My Content'. String and bytes content work.
Rate it. The course author: reproduction 2, versions 0, expected 2, acceptance 1, scope 1, so 6 of 10. The model’s median gives the same five scores.
Decide. Below the bar of 7: rewrite first. The rewrite adds the version, the attached failing case, and “string and bytes content keep their current results”.
Open the answer key.
author total 6 | model total 6 | decision with the bar of 7: rewrite firstmerged test: ['def test_memoryview_content(self):', "r = HttpResponse(memoryview(b'memoryview'))", "self.assertEqual(r.content, b'memoryview')"]names in the text that the fix changes: ['response']What this tells us. The merged test checks exactly what the text expects: memoryview content gives the same bytes. So this issue was ready, and the rule held it back, because the version was missing. That is the cheap mistake, “ready but held”. The rewrite still adds an attached case and a “must keep working” rule.
Recommendations
Section titled “Recommendations”Amara writes to Jonas in plain words:
- Do not send issues to the agent as written. In this study, only 10 of 30 real issues said what the maintainers’ test checked. Sending everything would misroute two in three.
- Make the agent-ready rewrite the normal step. If a wasted run costs three rewrites (an assumption), rewriting every issue first was the cheapest rule.
- If rewrite time is short, let the rubric choose the order. A total of 7 or more, or full marks on “expected”, found most ready issues.
- Ask the maintainers first when the text leaves a decision open. Parameter names, output formats and error types were the most common gaps. No tester can guess them, and no agent should.
- Name the neighbours in the acceptance criteria. The merged tests often checked related cases, such as the same feature inside another call.
- A model may pre-rate issues. It agreed with the author on most criteria. Check it on a sample of your own issues first, and expect it to be stricter on reproduction.
And what not to use the study for:
- Not to judge reporters. Issue authors wrote for maintainers, not for agents. A low score is about fit for an agent, not about a person.
- Not as a number for every project. Measure your own backlog before you quote one.
Limits
Section titled “Limits”- Filtered to be easy. SWE-bench Lite removed issues with images, links, very short texts, and fixes that touch more than one file. A real backlog has more weak issues.
- Small and one rater. Thirty issues and one person’s ratings and checks. A second rater would disagree on some scores, and differences of one or two issues are within chance.
- One fix is one answer. The merged test shows what the maintainers chose. A different, correct fix could fail it. “The test checks something the text does not say” can mean a weak text, or maintainers who chose more.
- Hindsight in the rewrites. The author knew the fixes when writing the rewrites. Open decisions were kept open, but a tester without that knowledge might write a weaker rewrite.
- One model, three runs. Another model or prompt would rate differently.
- The agent was not run. This study checks the issue texts, not an agent’s fixes. What a coding agent needs shows a recorded agent experiment.
Responsible use
Section titled “Responsible use”- These are real people’s words. The issue texts are public, but each was written by a person who wanted help. Quote them to learn, credit the project, and do not mock a weak report.
- Say who rated what. The rubric scores and the checks are the course author’s judgement. The model’s scores are model outputs. Keep them in separate columns, and never call a model’s score a human label.
- An agent-ready report is not a licence to skip review. A rewrite with a failing case makes an agent’s fix easier to check. People still read the change, as in Verify the agent’s fix.
- Keep private data out of model calls. Here, the issues were public. Before you send your own team’s issues to a hosted model, check what they contain and what your organisation allows.
Make it a portfolio piece
Section titled “Make it a portfolio piece”This study shows an employer that you can turn “the agent will fix it” into a measured decision. Make it yours with one change. For example, rate 30 issues from your own project or from SWE-Gym. Or add a second human rater, or run a coding agent on the original text and on your rewrite. Then make it public, and easy to read in five minutes.
What to put in the repository:
README.md, the page that people read first (template below).- The notebook, run from top to bottom, with its outputs.
RUBRIC.md: the five criteria with their 0, 1 and 2 levels, and one example per level.ratings.csvandchecks.csv, with a reason per score, and the commit that shows pass 1 came before pass 2.- Your rewrites, each with its failing case, and the run log of the check against the merged test.
- The data source and licence: SWE-bench Lite, MIT, with credit to the projects.
A README template:
# Are real issues ready for a coding agent?
## QuestionHow often does an issue, as written, state what the maintainers' merged test checks?Can a short rubric pick out the ready ones?
## Data and licenceSWE-bench Lite (Jimenez et al., 2024), MIT: 30 issues from 12 Python projects,each with its merged fix and test; 1 more issue kept apart.
## MethodPass 1: a 5-criterion rubric (0-2 each) from the text alone. Pass 2: the mergedfix and test as the answer key. Triage rules compared by misrouted issues.A hosted model rated the same issues 3 times. 3 rewrites; 1 checked by running it.
## ResultsReady (test matches the text): 10 of 30. Misrouted: send everything 20,rewrite everything 10, rubric total 7+ 7. Most often missing: versions (16 of 30),acceptance criteria beyond one example (28 of 30). Model vs me: 21-26 of 30.Django rewrite: fails before the merged fix, passes after; the maintainers'test added 2 neighbours that my rewrite did not name.
## LimitsFiltered, easy issues; one rater; 30 issues; the agent itself was not run.How to describe it in an interview, in two sentences: “I measured 30 real open-source issues against the tests that their maintainers merged: only a third stated what the test checked, and a five-criterion rubric cut the misrouted issues from 20 to 7. I rewrote weak issues in an agent-ready format and proved one by running it: it failed before the real fix and passed after it.”
Go further:
- Run the optional Django part in the notebook, then add the two neighbours to the rewrite’s case. Predict first: do they fail on the base commit?
- Ask a colleague to rate the same 30 issues with the rubric. Where do you disagree most?
- Rate 30 issues from SWE-Gym, which was not filtered for clear texts. Predict first: more or fewer ready issues?
- Give a coding agent the original text and your rewrite of the same issue, and run the merged test on both fixes.
Knowledge check
Section titled “Knowledge check”Check your understanding
The Flask issue about from_file() scored 8 of 10, yet the merged test checks only 'partly' what the text expects. How can a clear issue fail the answer key, and what should the reporter or tester have done?
Show the answer
The text was clear about the problem and proposed a solution: a mode="b" parameter. The maintainers chose a different design, text=False, and wrote the test for it. An agent that follows the text builds a working feature that fails the test. The rubric judges how clear the text is, not whether its proposal is the one the maintainers want. The fix is to agree the interface before the agent runs. Mark the parameter name as a decision for the maintainers, as the rewrites do. Then turn the agreed answer into the acceptance criterion.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
With equal weights, the rubric gate misroutes 7 issues and 'rewrite everything first' misroutes 10. With a wasted agent run counted as 3, the gate costs 17 and rewriting everything costs 10. What decides which rule is better, and what would you recommend?
Show the answer
Two things decide it: the base rate of ready issues and the cost of each mistake. Only 10 of 30 issues were ready, so any gate still sends some issues that are not, and each costs three times a rewrite. When a wasted run is expensive and a rewrite is cheap, rewriting everything first wins. Recommend the rewrite as the normal step, and use the rubric to order the work when time is short. If agent runs became cheap and reliable, or the backlog had more ready issues, the gate could win; measure again before changing the rule.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Check your understanding
The Django rewrite's failing case fails before the merged fix and passes after it. Does that prove the rewrite's acceptance criteria were complete? Explain with the merged tests.
Show the answer
No. It proves the case is a real failing case: it detects this bug and accepts this fix. Completeness is a different question. The maintainers’ tests also checked two neighbours: the annotation inside Count() (a GROUP BY) and Sum() over it. The rewrite did not name them, so a different fix could pass the rewrite’s case and break those queries. Acceptance criteria should name the neighbours, and the verification should run them, as well as the whole suite.
Does your answer have the main points of ours?
Sign in and enroll in the course to save your answers. Every check counts toward the certificate.
Recap
- A five-criterion rubric from the agent-ready template rated 30 real issues from their text alone. Versions (missing in 16) and acceptance criteria (full marks in 2) were the weakest parts.
- The merged test is an answer key: it shows what “fixed” meant. It checked what the text expects on only 10 of 30 issues.
- Under one measure, misrouted issues, the rubric total of 7 or more (7) beat sending everything (20). When a wasted agent run costs more than a rewrite, rewriting every issue first was cheapest.
- A model applied the rubric close to the author, but more strictly on reproduction, and its total changed between runs on 10 issues.
- A rewrite with Must rules and a failing case matched the core of the real test. It failed before the merged fix and passed after it, but it missed the maintainers’ neighbours.
Next, a different kind of test pass: open attack sets against a recorded assistant, with rates by category and severity-rated reports.