Negative testing

What you will be able to do

Test what must fail, not only what must work. Run one test matrix that sends every person, in every organization, to every protected route, and checks the exact refusal. Add the bad identities and the record IDs of the other shop. Then break the API on purpose and watch the matrix catch it.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• The access matrix and its rules, 404 for outsiders and 403 for members, from Users and organizations and Authorization in the API.
• The 15 made-up bad tokens and the checks that refuse them, from Token meaning.
• pytest and its parametrize marker, from Build and Deploy Your First API.

What you need on your computer

• The project after Lifecycle changes: it matches the snapshot end-of-m05, and python -m pytest prints 234 passed, 1 deselected.
• PostgreSQL and Azurite running, and the data of both shops loaded. The tests start the practice provider inside the test process, so you need no provider terminal in this lesson.

Every bug in this lesson is a constructed case: written on purpose, run, and then removed. Tested on macOS (Apple silicon); the Windows commands are written but not tested.

The night round in a museum

A museum closes at six. The night guard does not only check that the main door opens with the staff key. He walks the whole building. He tries every door with the wrong key. He tries the storeroom with a visitor’s ticket. He pushes the emergency exit from the outside. A door that opens when it should not is the one that he is there to find.

Most tests are like the day staff: they check that the right key opens the right door. A guard checks the other direction. In this lesson, you give the ticket API a night round.

The problem: 234 green tests and a hole

Your project has 234 tests. Many of them check access: Omar may not add a message, Sam gets 404 for Bramble’s ticket, Camille gets two answers. Each one was written by hand, for the case that the lesson taught.

Now make one small mistake on purpose. In ticket_api/members.py, the route that lists the members asks for the action member.read. Change it to ticket.read:

ticket_api/members.py (constructed bug: do not keep)

def members(caller: Annotated[Caller, Depends(require("ticket.read"))], db: DB) -> MemberList:

A developer could make this mistake when they copy a route from tickets.py. Run every test:

python -m pytest -q

The last line:

234 passed, 1 deselected in 20.62s

Every test passes. But Omar, who is read-only, can now list Larkfield’s members: their user IDs, names, roles and start dates. Camille can list Bramble’s members, where they are read-only too. No test asked, because no test sent a read-only person to the member list. Hand-picked tests cover the cases that somebody thought of. A privilege escalation lives in a case that nobody thought of. Leave the bug in for now: you use it again in a few minutes.

The idea: test the whole matrix

The access matrix already says what every person may do in every organization. So the tests do not need to pick cases. They can send every person to every route in every organization and compare the answer with the matrix. This is a test matrix: one parametrised test, with one test case per cell.

Testing what must fail has a name: negative testing. A positive test checks that Grace can invite a person. A negative test checks that Sam cannot, and that he gets the right refusal: 403, not 500, not 201. For access control, negative tests matter more than positive ones. A missing permission annoys a user, who reports it at once. An extra permission hurts silently.

The course’s test matrix has three parts:

The 213 checks of the test matrix: 154 cells for people and organizations, 4 cases with the other shop's record ID, and 55 bad identities.

Text description of the diagram

The test file has three groups. First, 7 people times 2 organizations times 11 routes: 154 checks. Second, 4 cases where a member uses the right path with the other organization’s record ID; each case sends 3 requests. Third, 5 kinds of bad identity on each of the 11 routes: 55 checks. Together: 213.

Each group tests some of the five kinds of failure:

Kind of failure	Where in the matrix	The answer it needs
Missing identity	bad identity: no identity	401
Expired credentials	bad identity: expired	401
Wrong audience	bad identity: wrong audience, and an ID token (its audience is the application)	401
Insufficient role	the cells of read-only and staff members; also no scope	403
Wrong organization	the cells of outsiders, and the other shop’s record ID	404

A tiny matrix first

Before the big file, look at one row with every value visible. Omar is read-only at Larkfield. The matrix in ticket_api/access.py gives read_only three actions: ticket.read, file.read and document.read. So, for Omar at Larkfield:

Route	Action	Omar’s role allows it?	Expected
read a ticket	ticket.read	yes	200
add a message	ticket.write	no	403
list members	member.read	no	403
read the audit	audit.read	no	403

At Bramble Books, Omar has no membership, so every route expects 404, with the same body as an organization that does not exist. The test does not hard-code these numbers. It asks decide() for the decision and turns it into the expected status:

tests/test_access_matrix.py (part)

def expected(user: str, tenant: str, route: str) -> int:
    action, *_, ok = ROUTES[route]
    decision = decide(MEMBERSHIPS.get((tenant, user)), action)
    return ok if decision.allowed else decision.status

ok is the status of a successful call: 200 for a read, 201 for a new message, 202 for an export that starts. Then one test function runs for every cell. pytest.mark.parametrize makes 154 test cases from one function, and gives each one a readable ID such as usr-omar|larkfield|upload a file.

Add the test matrix

1. Copy the test file from the snapshot end-of-m06:

   macOS / Linux

   S=../nextia-learning-labs/C23/snapshots/end-of-m06
   cp "$S/tests/test_access_matrix.py" tests/

   Windows (PowerShell)

   $S = "..\nextia-learning-labs\C23\snapshots\end-of-m06"
   Copy-Item "$S\tests\test_access_matrix.py" tests\

2. Your API still has the member-list bug. Run only the new file:

   python -m pytest tests/test_access_matrix.py -q

   E       AssertionError: {"items":[{"user_id":"usr-grace","name":"Grace","role":"owner","since":"2024-01-08T09:00:00Z"},…
   E       assert 200 == 403
   E        +  where 200 = <Response [200 OK]>.status_code
   E        +  and   403 = expected('usr-omar', 'larkfield', 'list members')
   …
   FAILED tests/test_access_matrix.py::test_matrix[usr-omar|larkfield|list members] - AssertionError: …
   FAILED tests/test_access_matrix.py::test_matrix[usr-camille|bramble|list members] - AssertionError: …
   2 failed, 211 passed in 19.84s

   The matrix finds what 234 tests missed. Each failure names the cell, and the assertion shows what the API answered: assert 200 == 403. The output is shortened here. A 200 means that the API sent the member list to a read-only person.

3. Remove the bug: change ticket.read back to member.read in ticket_api/members.py. Run the matrix again:

   python -m pytest tests/test_access_matrix.py

   collected 213 items
   
   tests/test_access_matrix.py ............................................ [ 20%]
   ........................................................................ [ 54%]
   ........................................................................ [ 88%]
   .........................                                                [100%]
   
   ============================= 213 passed in 18.77s =============================

   The time differs on each computer. Each case makes a new test client, so 213 cases take some seconds.

4. Run the rows of one person, to see the test IDs:

   python -m pytest tests/test_access_matrix.py -k "usr-omar" -v

   Each line names a person, an organization and a route, and every line says PASSED.

Read the matrix as a picture

A list of 213 test names is hard to read. The picture below draws the same results as a grid: one row for each person in each organization, one column for each route. The colour and the number in a cell are the status that the access matrix requires. Every cell here passed: the API answered exactly that status.

• allowed (200, 201, 202)
• 403: member, role too small
• 404: not a member
• 401: no valid identity

The test matrix at the end of the course: 213 of 213 checks passed. Rows: each person in each organization, then five bad identities. Columns: the 11 protected routes.

Text description of the chart

Fourteen rows show the 7 people in the 2 organizations, with their role there. Grace (owner) and Ines (owner) get 200, 201 or 202 on all 11 routes in their own organization. Sam and Camille at Larkfield are staff: they get 403 for “invite a person” and “read the audit”, and success on the rest. Omar at Larkfield and Camille at Bramble Books are read-only. They get success for list tickets, read a ticket, similar tickets, download a file and search documents. They get 403 on the other six routes. Every row without a membership, including Tomás and Kwame in both organizations, is 404 on all 11 routes. Under the people are five rows of bad identity, sent to Larkfield’s routes. No identity, expired, wrong audience and ID token are 401 on all 11 routes. A token with no scope gets 403 on all 11. The other organization’s record ID: 4 of 4 cases pass. 213 of 213 checks passed.

Read it like a map. The grey blocks are the outsiders: whole rows of 404. The orange blocks are the members with a smaller role. The blue blocks are the work that each role exists for. Tomás and Kwame have no membership, so both of their rows are grey. Kwame is a platform administrator, but that is not a membership: without a support grant, he gets 404 like anybody else.

Now the same picture for the API with the member-list bug. The data is the real result of the constructed run:

• allowed (200, 201, 202)
• 403: member, role too small
• 404: not a member

Constructed bug: the member list asks for ticket.read. 211 of 213 checks passed; the failed cells have a thick outline.

Text description of the chart

The same 14 rows and 11 columns. Two cells have a thick outline: Omar at Larkfield, list members, expected 403; and Camille at Bramble Books, list members, expected 403. In both cells the API answered 200. Every other cell passed. 211 of 213 checks passed.

The two failed cells are in the read-only rows, in one column. That pattern tells you where to look: one route, and the check for one role. A failed cell in a grey row would mean a much worse bug: an outsider got in.

The bad identities and the other shop’s IDs

The last two groups use the same idea. Five kinds of bad identity go to every route:

tests/test_access_matrix.py (part)

BAD = ["no identity", "expired", "wrong audience", "ID token", "no scope"]

The first four must get 401, and the last one 403: the person is known, but the token does not allow the action. In Token meaning, the 15 made-up tokens went to one endpoint, /v1/me. Here, five of them go to all 11 routes. That checks that no route forgot the identity check, for example a new route without require().

The other shop’s record ID is the object-level attack of Threat examples. Sam uses his own Larkfield path, but with Bramble’s ticket T-40001. Reading the ticket and its similar tickets must give 404. Adding a message must give 404 too, or 403 for a read-only member, because the role check runs before the lookup.

Who checks the expected answers?

Kwame asks an awkward question: “The matrix takes its expected answers from decide(). What if decide() itself is wrong?”

Try it. Put a second constructed bug in ticket_api/access.py: give read_only the action ticket.write too. Then run the matrix and the policy test:

python -m pytest tests/test_access_matrix.py tests/test_access_policy.py -q

FAILED tests/test_access_matrix.py::test_another_organizations_record_id[usr-camille|bramble|T-30002]
FAILED tests/test_access_policy.py::test_the_code_is_the_matrix[2-read_only-ticket.write]
FAILED tests/test_access_policy.py::test_a_role_that_cannot_is_403_and_names_the_rule
FAILED tests/test_access_policy.py::test_camille_gets_two_answers_for_one_action
4 failed, 244 passed in 19.18s

The failure lines are shortened here. All 154 cells of the matrix passed: the API and decide() agreed, and both were wrong. Omar could write, and the matrix expected exactly that. Only one hand-written expectation in the matrix noticed, in a case with the other shop’s ID. Three tests of test_access_policy.py failed. That file has the paper matrix copied by hand, row by row, and it compares decide() with it.

Two tests, two questions. The policy test asks: is decide() the agreed matrix? The test matrix asks: does every route obey decide()?

Text description of the diagram

The rules start on paper, in docs/access-matrix.md. The policy test is a hand-written copy of the paper, and it checks decide(). The test matrix asks decide() for each expected status and checks the API’s 213 answers. A bug in a route fails the test matrix. A bug in decide() passes the cells of the test matrix and fails the policy test.

So you need both. The test matrix finds a route that does not obey the rules. The policy test finds rules that are not the agreed rules. Remove the second bug before you continue.

A third constructed bug: when the second lock hides the first

The author removed the organization from one query on purpose: get_ticket() in ticket_api/repository.py without t.tenant_id = %(tenant_id)s. The test matrix still passed, 213 of 213. Row-level security from Tenant-safe queries held: as ticket_app for Larkfield, the database showed no Bramble row. With a copy of the matrix that sets db_row_security=False, the 4 cases with the other shop’s ID failed with assert 200 == 404. The second lock hid a broken first lock. A test run without the second lock shows whether the first one still works.

Run every test

Run the whole suite once more, with no bug in the code:

python -m pytest -q

The last line:

447 passed, 1 deselected in 38.67s

The matrix costs some seconds on each run, and more when you add a route or a person: the cells multiply. It is still cheap. A new route needs one line in ROUTES, and it gets 14 cells and 5 bad identities at once. A new person or membership needs one line in PEOPLE or MEMBERSHIPS. When not to use a full matrix: for rules that depend on the data, not on the role. An example is “the author may edit their own message for 5 minutes”. Test those with a few hand-written cases, next to the matrix.

Common mistakes

Common mistake

The matrix passes in no time

Symptom: python -m pytest tests/test_access_matrix.py prints 213 skipped in 0.01s.

Cause: PostgreSQL is not running. The tests skip themselves when they cannot reach the database. A skipped test is not a passed test.

Fix: start the database (docker compose up -d db azurite) and run the tests again. Add -rs to see why tests were skipped: PostgreSQL is not running (with Docker: docker compose up -d db).

Common mistake

A new route has no cells

Symptom: you add a route, and the matrix still has 213 checks.

Cause: the matrix tests only the routes in its ROUTES table. A route that is not in it is not tested.

Fix: add every new protected route to ROUTES, with its action, method, path, body and success status. A review of a new route should ask “is it in the matrix?”.

Your turn: predict a row

Camille is staff at Larkfield and read-only at Bramble Books. Without the picture, write their two rows: 11 statuses each. Then run python -m pytest tests/test_access_matrix.py -k "usr-camille" -v and compare.

Show the answer

At Larkfield (staff): 200, 200, 200, 201, 200, 201, 200, 200, 403 (invite a person), 202, 403 (read the audit). At Bramble Books (read-only): 200, 200, 200, 403, 200, 403, 200, 403 (list members), 403, 403, 403. All 22 cells say PASSED, and their two cases with the other shop’s ID too.

See it end to endAudit access control in an open-source multi-tenant appCase study

The Gitea case study writes a test matrix for a real app: 7 people, 2 organizations and 10 routes. That is 140 cells, and the crossings are their own tests.

See it end to endAdd sign-in and roles to an existing APICase study

The case study tests another API with an access matrix: 225 of 225 checks pass. The attempts that all worked with a shared key now all fail.

Check your understanding

Name the five kinds of failure that the test matrix checks, and the status each one must get.

Show the answer

Missing identity: 401. Expired credentials: 401. Wrong audience (also an ID token sent as an access token): 401. Insufficient role or scope: 403, because the person is a member and already sees the organization. Wrong organization: 404, for an outsider and for another organization’s record ID, so that nothing is revealed.

Check your understanding

Predict: a developer adds a new route, DELETE /v1/tenants/{tenant}/tickets/{id}, and forgets require(). They add it to ROUTES with the action ticket.write. Which cells of the matrix fail?

Show the answer

Without require(), the route checks neither the identity nor the membership. The four bad identities that need 401 fail, and the no-scope row that needs 403. Every outsider row (Grace, Sam and Omar at Bramble Books, Ines at Larkfield, Tomás and Kwame in both) expects 404 and fails. The read-only rows expect 403 and fail. Only the cells of owners and staff, who may write, pass.

Check your understanding

Why does the project need test_access_policy.py when the test matrix already has 213 checks?

Show the answer

The test matrix takes its expected answers from decide(). If decide() is wrong, the API and the expectation are wrong together, and the matrix cells pass. With read_only allowed to write, all 154 cells passed. The policy test compares decide() with a hand-written copy of the paper matrix, so it catches a wrong rule.

Check your understanding

Practical check: run the test matrix. How many checks pass, and what does a skipped check mean?

Show the answer

python -m pytest tests/test_access_matrix.py prints 213 passed. A skipped check did not run, usually because PostgreSQL is not running. It proves nothing: count only passed checks.

Recap

• Negative testing checks what must fail, and the exact refusal: 401, 403 or 404.
• A test matrix sends every person, in every organization, to every route: 7 × 2 × 11 = 154 cells. With 4 cases of the other shop’s ID and 5 bad identities × 11 routes, that makes 213 checks.
• A constructed bug in the member list passed all 234 earlier tests and failed 2 checks of the matrix.
• The matrix trusts decide(); test_access_policy.py checks decide() against the paper. Keep both.
• A skipped test is not a passed test.
• Next: record who did what, and who tried.

Next: Auditability