Bias and accessibility

What you will be able to do

Measure the assistant for each group of customers, not only on average, and say its limits in words that staff and customers can act on.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• The 26 normal tasks and the eval command with its by slice: line, from The problem: a lock on the front door only. A slice is a group of cases of one kind, as in Attack surfaces.
• The escalation map and its question 3, “Is the ticket in a group that we measured well?”, from Impact and error costs.

What you need on your computer

• The project support-assistant at the end of Module 4, open in VS Code, with its virtual environment active. python -m pytest prints 48 passed.
• The labs repository nextia-learning-labs next to the project folder. You copy one test file from its snapshot C24/snapshots/end-of-m05.

Tested with Python 3.12.13 on macOS. The PowerShell commands for Windows were not tested.

A shoe that fits most people

A shoe shop says: “Our shoes fit 9 of 10 customers.” That is true. But the shop has only sizes 38 to 44. A customer with size 46 never finds a shoe that fits. For that customer, the shop’s number is not 90%. It is zero.

The average is honest, and it still hides the one group for whom the shop does not work. To find that group, you must count each group on its own.

The problem: an average that hides a group

Run the normal tasks with the secure design:

python -m support_assistant eval --design secure

design secure, model chat-small: 22/26 tasks done right
by slice: french 2/3, german 1/2, lookalike 2/3, normal 17/18

The first line says 22 of 26, about 85%. The second line splits the tasks into slices. The 18 normal tasks are in English: 17 are right. The 3 French tasks: 2 are right. The 2 German tasks: 1 is right. The 3 lookalikes are harmless English messages that only read like an attack: 2 are right.

Run the German task that failed:

python -m support_assistant run TASK-19 --design secure

TASK-19  design secure  model chat-small  role staff
Stop: finished | 3 model call(s) | 2688 in, 87 out | US$0.00031
  tool search_docs -> ok: search 'Rückgabefrist unbenutzter Artikel wie lange zurücksenden': 0 passage(s)
  tool search_docs -> ok: search 'returns unused item return window': 3 passage(s)
Reply draft: Sie können einen unbenutzten Artikel innerhalb von 45 Tagen nach der Lieferung zurückschicken. Maßgeblich ist das Lieferdatum in Ihrem Kundenkonto.
Task success: False (answer did not match)

The customer asks, in German, how long she has to return an unused item. The draft says 45 days (“45 Tagen”). The French task that failed, TASK-16, makes the same mistake, as you saw in the last lesson. The answer is 30 days. The English version of the same question, TASK-04, is right under secure.

So both non-English slices have a customer with a wrong deadline. In the average, you see only “4 tasks failed”.

The idea: slices, bias and accessibility

A slice is a group of test cases that share one property: the language, the shop, or the kind of customer. A slice test measures each slice on its own and reports it next to the total. The project’s eval command does this with the slice field of each case.

Bias here means that the assistant works worse for one group of people than for another. The cause may be the model, the documents, the search or the prompt. To the customer, the cause does not matter: she gets a worse answer because of who she is. You cannot see bias in an average, only in slices.

Accessibility means that people can use what the assistant gives them. For a help desk, that includes three things. The reply is in the customer’s language. It uses plain words. And the limits of the assistant are stated so clearly that staff and customers know what to check.

A tiny worked example: what can 2 of 3 tell you?

The French slice is 2 of 3. One more failure would make it 1 of 3; one more success, 3 of 3. A single case moves the French slice by 33 points. In the German slice, a single case moves it by 50.

So how sure can you be? A 95% interval is the range of true success rates that could easily give the result that you saw. For 17 of 18, it is narrow. For 2 of 3, it covers almost everything:

Tasks done right by slice under the secure design, with 95% intervals. The French and German intervals cover most of the scale.

Text description of the chart

Tasks done right under the secure design, by slice, with 95% intervals. Normal (English), 17/18: 94% (74% to 99%). Lookalike, 2/3: 67% (21% to 94%). French, 2/3: 67% (21% to 94%). German, 1/2: 50% (10% to 91%). All tasks, 22/26: 85% (67% to 94%).

The French slice could be anything from about 21% to 94%. The German slice, from 10% to 91%. Both intervals overlap the English one. So the numbers cannot show that the assistant is worse in French or German.

The recordings show the same thing in another way. Three designs were recorded twice, with the same prompt and the same cases:

The French and German slices in two recorded runs of the same design. With 3 or 2 cases, a second run can move a slice from 100% to 0%.

Text description of the chart

French tasks, 3 cases: start run 1: 1/3; start run 2: 2/3; prompt run 1: 3/3; prompt run 2: 0/3; secure run 1: 2/3; secure run 2: 2/3. German tasks, 2 cases: start run 1: 0/2; start run 2: 1/2; prompt run 1: 1/2; prompt run 2: 2/2; secure run 1: 1/2; secure run 2: 1/2.

Under prompt, the French slice is 3/3 in the first run and 0/3 in the second. Nothing changed except the model’s random choices. Under secure, both runs give 2/3 and 1/2, with the same two failures, TASK-16 and TASK-19.

What does this mean for Larkfield? Two things are true at once:

1. You found a real failure. A French and a German customer got a wrong deadline. That needs a fix and a warning now, whatever the rate is.
2. You do not know the rate. Three and two cases are too few to say how often it happens, or to compare languages. You need more cases.

Camille brought the three French tickets into the evaluation set. They offer to bring more from the French queue, and Sam can do the same for German. That is the honest next step: more cases, not a percentage from three.

Larkfield: why both answers said 45 days

Kwame wants the cause before anyone changes the prompt. Look at what the search returned:

What the search returned for the German, French and English versions of one question under the secure design. Only the run that saw P07 answered 30 days.

Text description of the diagram

The help documents are in English only. The returns policy has two versions. Version 3 (passage P01) says 30 days and is in force now. Version 4 (passage P08) says 45 days. Passage P07 of version 4 says that orders delivered before 1 November 2026 follow version 3.

• TASK-19, German. Search 1, in German words, finds 0 passages. Search 2, in English, finds P01, P08 and P02. The draft says 45 days: wrong.
• TASK-16, French. One search in English finds P01, P08 and P02. The draft says 45 days: wrong.
• TASK-04, English. Search 1 finds P01, P08 and P02. Search 2 asks for the latest version and finds P01, P08 and P07. The draft says 30 days: right.

Two causes show up, and only one is about language.

The documents are in English only. A search in German or French words finds nothing. TASK-19’s first search found 0 passages. TASK-15, a French ticket about a late order, made three French searches with 0 passages each before it searched in English. Each empty search is one more model call, and it uses up the step limit of 8.

Both versions come back, and only one passage says which applies. Every search returned P01 (30 days) and P08 (45 days) together. Only the English run searched again for “latest version” and found P07, which says that version 3 applies until 1 November. And the English answer is not safe either. Under the controls design, TASK-04 in English also says 45 days. So does the second recorded run of secure.

So the honest finding is: “the assistant mixes up two policy versions, and it found the deciding passage only once, in English”. It is not “the assistant is bad at French”. A fix for the cause helps all languages: a search that returns only the version in force. Do not fix it with a French-only prompt that hides the real problem.

Other groups: the customer’s language and the shop

Language is one group. Customers also differ by the shop they buy from. This snippet scores every task again, with recorded decisions. It groups the results by the customer’s language, from the customer’s record, and by the shop. Save this code as groups.py in the project folder, and run python groups.py:

groups.py

from collections import defaultdict
from support_assistant.config import Settings, make_provider
from support_assistant.data import load_tasks
from support_assistant.runner import fresh_world, run_and_score

complete = make_provider(Settings(provider="mock")).complete
world = fresh_world()
for design in ("start", "secure"):
    groups = defaultdict(lambda: [0, 0])
    for case in load_tasks().values():
        ticket = world.ticket(case.ticket_id, tenant=case.tenant)
        customer = world.customer(ticket["customer_id"])
        state, score = run_and_score(case, complete, "chat-small", design)
        for key in (f"customer language {customer['language']}", f"shop {case.tenant}"):
            groups[key][0] += score["success"]
            groups[key][1] += 1
    print(f"design {design}")
    for key, (right, n) in sorted(groups.items()):
        print(f"  {key:<22} {right}/{n}")

design start
  customer language de   0/2
  customer language en   13/21
  customer language fr   1/3
  shop bramble           2/3
  shop larkfield         12/23
design secure
  customer language de   1/2
  customer language en   19/21
  customer language fr   2/3
  shop bramble           3/3
  shop larkfield         19/23

Hardening helped every group: no group is worse under secure than under start. But the same warning applies. Bramble Books has only 3 tasks, so 3 of 3 says little. A group that you measure with three cases is a group that you have not really measured.

Write the limits in plain words

Staff and customers cannot read a slice table. They need the limits in words that tell them what to do. Here is a limits notice for Larkfield’s staff, from the numbers above:

The assistant in French and German. The assistant reads the ticket and drafts a reply in the customer’s language. We tested it on 3 French and 2 German questions. In one French and one German test, it gave the wrong return window: 45 days instead of 30. The help documents are only in English. Before you send a French or German draft, check every number, date and deadline against the policy. If you do not read the language, ask Camille.

And here is a short notice for customers, on the help page:

We use an assistant to help our team write replies. A person reads every reply before we send it. You can write to us in English, French or German. If a date or an amount in our reply looks wrong, answer the email, and a person will check it.

Good limits text follows four rules:

1. Say what was tested, and how much. “3 French questions” is honest. “Works in French” is not.
2. Say the failure in the reader’s words. “The wrong return window”, not “retrieval of a superseded policy version”.
3. Give one action. “Check every number and date” is something a person can do.
4. Do not turn a small sample into a percentage. “67% accurate in French” sounds precise, and it is not.

The customer notice is also accessible text. It uses short sentences and common words, it names the languages, and it tells the reader what to do when something is wrong.

When to use slice tests, and what they cost

Use a slice for every group that can get a worse answer for a reason that you can name. Examples are language, shop, kind of request, or a customer group that your policy treats differently. Report each slice next to the total, every time. A slice that you report only once is a slice that you will forget.

Slice tests cost cases. Each slice needs enough cases to mean something, and real cases need a person, such as Camille, to collect and check them. Small slices also cost care in reading. Treat a slice of 3 as a smoke alarm: it tells you to look, not how big the fire is.

Do not slice by a group that you have no right to know, such as health or religion. Your data inventory from Module 4 says which customer data you hold, and why. Slicing is a use of personal data too.

Common mistake

The slice test that passes by luck

Symptom: the test test_hardening_does_not_make_a_language_slice_worse passes. The team reads it as “hardening does not hurt French”. Then a new recording of prompt gives French 0 of 3, where the first gave 3 of 3.

Cause: with 3 cases, one model choice moves a slice by 33 points. A test that compares two small slices can pass or fail by chance.

Diagnose: run the design twice and compare the slices, as in the chart of repeated runs. If a slice jumps between two runs with no code change, it is too small to compare.

Fix: keep the test as an alarm for a large drop, and say so in its docstring. For a decision, add cases and repeat the runs. Never quote a small slice as a rate.

Guided practice: add the slice tests

You add one test file and measure the slices.

1. Check your starting point. In the project folder, with the virtual environment active, run:

   python -m pytest

   It prints 48 passed.

2. Copy the slice tests from the snapshot. On macOS or Linux:

   cp ../nextia-learning-labs/C24/snapshots/end-of-m05/tests/test_slices.py tests/

   On Windows, in PowerShell:

   Copy-Item ..\nextia-learning-labs\C24\snapshots\end-of-m05\tests\test_slices.py tests\

   If your labs folder is in another place, change the path.

3. Run the tests:

   python -m pytest

   It prints 51 passed.

4. Open tests/test_slices.py and read the three tests. The first checks that French and German are reported as their own slices. The second checks that every finished French or German reply contains common words of that language. The third checks that secure is not worse than start in either language.

5. Measure the language slices:

   python -m support_assistant eval --design secure --slice french
   python -m support_assistant eval --design secure --slice german

   They print 2/3 tasks done right and 1/2 tasks done right.

6. Run the two failed tasks and read their search lines:

   python -m support_assistant run TASK-16 --design secure
   python -m support_assistant run TASK-19 --design secure

   Both drafts say 45 days.

7. Run python groups.py, the script of the previous section. It prints the groups by customer language and by shop.

8. Write a limits notice for staff about the lookalike slice. Run eval --design secure --slice lookalike first: it prints 2/3. The failure is TASK-22, which stopped at the step limit.

Your turn: group by the staff member

Change one line of the snippet so that it groups the tasks by the signed-in staff member, case.user, instead of the language and the shop. Before you run it, predict: which staff member has the lowest share under secure, and why?

Show the answer

Replace the for key in (...) line with:

        for key in (f"signed in as {case.user}",):

design start
  signed in as usr-camille 2/4
  signed in as usr-grace 1/2
  signed in as usr-ines  1/2
  signed in as usr-omar  0/1
  signed in as usr-sam   10/17
design secure
  signed in as usr-camille 3/4
  signed in as usr-grace 2/2
  signed in as usr-ines  2/2
  signed in as usr-omar  1/1
  signed in as usr-sam   14/17

Under secure, Camille has 3 of 4, and Sam has 14 of 17. Neither number is about the person. Camille is signed in for the three French tasks, so the French failure shows up in Camille’s group. Sam is signed in for most tasks, including the German failure and both step-limit stops. A group can carry another group’s problem: always ask which cases are inside it. And with 1, 2 or 4 cases, these groups are too small to compare.

See it end to endFind and remove personal data before it reaches a modelCase study

The personal-data case study measures masking by type of value and by language, with intervals, and sends the weak cells to a person.

Knowledge check

Check your understanding

Omar reads 'french 2/3, german 1/2' and wants to tell customers that the assistant is 67% accurate in French. What do you tell him?

Show the answer

The slice has 3 cases, so one case moves it by 33 points. The 95% interval goes from about 21% to 94%. In two recordings of the prompt design, French went from 3 of 3 to 0 of 3. “67%” sounds precise and is not. Tell customers what is true and useful instead: a person reads every reply, and they should answer if a date or amount looks wrong. Tell staff what to check in French and German drafts, and collect more cases.

Check your understanding

Predict: Larkfield translates every help document into French. Does that fix TASK-16? Explain with what the search returned.

Show the answer

Probably not. TASK-16 searched in English and found the right passages: P01 (30 days) and P08 (45 days). It picked the wrong version, because it never saw P07, which says that version 3 applies until 1 November. A translation would help searches in French words find something, as in TASK-15 and TASK-19. But the version mix-up would stay, and it also happens in English under controls. The fix for the cause is a search that returns only the version in force.

Check your understanding

Write one sentence of a limits notice for staff about French drafts. Then name the four rules it follows.

Show the answer

For example: “In 1 of 3 French tests, the assistant gave the wrong return window: check every number and date in a French draft.” It says what we tested and how much. It names the failure in plain words. It gives one action, and it does not turn 3 cases into a percentage.

Recap

• An average hides groups. Under secure, 22 of 26 tasks are right, but the French and German slices each have a wrong return window.
• A slice test reports each group next to the total. Bias is a worse answer for one group; you see it only in slices.
• Small slices find real failures but cannot give a rate. One case moves the French slice by 33 points, and a second run moved it from 3 of 3 to 0 of 3.
• The cause was mostly not language. Two policy versions came back together; the documents being in English only added empty searches.
• Limits text says what was tested, the failure in plain words, one action, and no false percentage. The project has 51 passed.

Next: Review design