Tests and repeatable runs

What you will be able to do

Turn the collection into a check that anyone can run with one command. It checks every request, every expected status code and every field, in under a second. You use the same check again in Module 7, to prove that the deployed API works.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Variables and environments, and the difference between local and shared values, from Collections, variables and environments: Variables and Local values and shared values.

What you need on your computer

• The Ticket Classifier API collection, from First requests in Postman: Import the OpenAPI document, with the three failure requests from First requests in Postman: Your turn: requests for every row.
• The Postman environments Local and Deployed, and the API key authorization of the collection, from Collections, variables and environments: Worked example: two Postman environments. Local is the active environment.
• The API runs locally with fastapi dev, as in Environment and first endpoint: Step 3: start the server.
• If your project is missing or broken, copy the snapshot end-of-m04, the project at the end of Module 4, as in Use a snapshot. Then do the guided practice and the “Your turn” task of First requests in Postman, and the guided practice of Collections, variables and environments again.

Tested with the Postman CLI 1.69.0 on macOS. The steps for the app follow Postman’s documentation for Postman 12.

Tests in Postman

Until now, you read each response and decided if it was correct. A test in Postman is a short JavaScript script that runs after the response arrives, and checks it for you. You write it in the request’s Scripts tab, under Post-response:

Post-response script of Classify a support ticket

pm.test("status is 200", () => pm.response.to.have.status(200));

const result = pm.response.json();
pm.test("category is billing", () => pm.expect(result.category).to.equal("billing"));
pm.test("priority is 1 to 3", () => pm.expect(result.priority).to.be.within(1, 3));
pm.test("answers in under 500 ms", () => pm.expect(pm.response.responseTime).to.be.below(500));

Part	Meaning
pm.test("name", () => ...)	One test, with a name that says what it checks. It passes if the code inside does not fail.
pm.response.to.have.status(200)	The status code must be 200.
pm.response.json()	The response body, read as JSON.
pm.expect(value).to.equal(...)	The value must be equal. Also .to.be.within(1, 3), .to.include(...), .to.be.below(...), .to.include.all.keys(...).
pm.response.responseTime	How long the response took, in milliseconds.

You do not need to know JavaScript for this. The same few lines, with different values, cover almost every test in this course. After you send the request, the response pane’s Test Results tab shows each test as passed or failed.

Tests for every request, and one for all

Some checks are the same for every request: in this API, every response is JSON, and every error has the shape from Module 3. Write those once, in the collection’s Post-response script. Postman runs it after every request in the collection:

Post-response script of the collection

pm.test("response is JSON", () => pm.response.to.be.json);
if (pm.response.code >= 400) {
    pm.test("error has the standard shape", () => {
        const error = pm.response.json().error;
        pm.expect(error).to.include.all.keys("code", "message", "fields");
    });
}

to.include.all.keys means “has at least these keys”. A stricter check, to.have.keys, means “exactly these keys”. It would fail as soon as the API adds a field to its errors, which is not a breaking change. Check what clients rely on, and allow what the contract allows.

Then each request only checks what is special about it. Here are the tests for the failure requests:

Reject an empty subject

pm.test("status is 422", () => pm.response.to.have.status(422));
const error = pm.response.json().error;
pm.test("code is invalid_request", () => pm.expect(error.code).to.equal("invalid_request"));
pm.test("names the subject field", () => pm.expect(error.fields).to.include("subject"));

Reject an unknown field

pm.test("status is 422", () => pm.response.to.have.status(422));
pm.test("names the colour field", () => pm.expect(pm.response.json().error.fields).to.include("colour"));

Reject a GET on classify

pm.test("status is 405", () => pm.response.to.have.status(405));
pm.test("code is method_not_allowed", () => pm.expect(pm.response.json().error.code).to.equal("method_not_allowed"));

Run the whole collection

The Collection Runner sends every request in the collection, in order, and shows every test result:

1. Select the collection in the sidebar, then the Run icon.
2. Under Run type, choose Functional, then Local.
3. Choose the Postman environment Local.
4. Select Start run.

The run shows each request with its status code and its tests. A run of the whole collection takes less than a second, so you can run it after every change to the API.

Save the collection in the project

The collection lives in your Postman account. Make the check part of the project, so that it is reviewed with the code and runs on any computer. To do this, export it into the project:

1. In the sidebar, next to the collection, select View more actions (the …), then More › Export collection, then Export JSON.
2. Save the file in a new folder postman in the project, as ticket-api.postman_collection.json.
3. In Environments, next to Local, select View more actions, then Export. Save it as postman/local.postman_environment.json.
4. Open the environment file in VS Code, and check that apiKey has an empty "value". If it shows a real key, delete it from the file, and read the common mistake in the last lesson.

Run it from the terminal: the Postman CLI

The Postman CLI runs a collection file from the terminal. It does not need a Postman account to run a file from your project, so anyone who has your project can run your checks.

1. Install the Postman CLI:

   macOS

   curl -o- "https://dl-cli.pstmn.io/install/unix.sh" | sh

   Windows

   irm https://dl-cli.pstmn.io/install/win64.ps1 | iex

   Linux

   curl -o- "https://dl-cli.pstmn.io/install/unix.sh" | sh

   If you have Node.js, npm install -g postman-cli also works.

2. Open a new terminal, so it finds the new command. Check it: postman --version. The output is a version number, such as 1.69.0.

3. In the project folder, with the API running, run the collection with the Local environment:

   postman collection run postman/ticket-api.postman_collection.json -e postman/local.postman_environment.json

Output (start and end)

No authorization data found. Please use the `postman login` command.

Running your collection...

Ticket Classifier API

Folder tickets
Sub-folder List the ticket categories

  GET http://127.0.0.1:8000/v1/categories [200 OK, 202 B, 8 ms]
  Pass  response is JSON
  Pass  status is 200
  Pass  lists 5 categories
...
|                requests |                 6 |                0 |
|              assertions |                24 |                0 |

The first line only says that you are not signed in. The CLI runs the local file anyway. Six requests and 24 checks, all passed. The exit code is 0.

To run one folder only, add -i with its name: -i health runs only Check that the service runs.

When the API is down

Stop the API (Ctrl+C in its terminal), and run the collection again:

Output (shortened)

  GET http://127.0.0.1:8000/v1/categories [errored]
     connect ECONNREFUSED 127.0.0.1:8000
...
   #  failure         detail
 01.  Error           connect ECONNREFUSED 127.0.0.1:8000
                      inside "tickets / List the ticket categories"
 02.  AssertionError  response is JSON
...
|                requests |        6 |        6 |

Every request fails, and the exit code is 1. That is what a check must do: fail loudly, with a non-zero exit code, so that a script or an automatic job stops. The first failure (ECONNREFUSED) tells you why: the connection step, from Module 1. Start the API again.

Postman tests or pytest?

Both test the API. They test it from different places, so a good project has both:

	Postman collection	pytest (Module 6)
Sends real HTTP to	A running API: local, or deployed	The application in memory, no server
Sees	Only requests and responses, as a client does	Also the inside: replace the classifier, change settings
Best for	Exploring, examples for other developers, checking a deployed API	Every rule and every failure path, on every change, fast
Can test “the classifier is down”?	Only if the deployed API is really down	Yes, by giving the route a broken classifier

Use pytest for the detailed tests of the code. Use the collection to check the contract from the outside, and especially to check a deployment, where pytest cannot reach.

Guided practice: a tested collection

1. Add the collection-level script, and the tests for each request in the worked examples. Add tests of your own for List the ticket categories (status 200, five categories) and Check that the service runs (status 200, "status": "ok").
2. Send each request, and read its Test Results tab.
3. Run the whole collection in the Collection Runner with Local. Every test passes.
4. Export the collection and the Local environment into postman/. Check that the environment file has no key.
5. Install the Postman CLI, and run the collection from the terminal. Check the summary and the exit code (echo $?, or $LASTEXITCODE in PowerShell).
6. Stop the API, and run it again. The exit code is 1. Start the API again.

Your turn: break the contract, and see who notices

In models.py, change the limit of subject from max_length=120 to max_length=10, and save. Then run, in this order:

1. The Postman collection from the terminal.
2. python -m pytest.

Which checks fail? Which check would fail if you had changed the limit to 200 instead? Undo the change afterwards.

Show the answer

With 10, the collection fails on Classify a support ticket: "Charged twice for March" is longer than 10 characters, so the API answers 422. The pytest contract test from Module 4 also fails, because maxLength changed in the OpenAPI document, and so does the example test.

With 200, every Postman test passes: no request in the collection sends a subject longer than 120 characters. Only the pytest contract test notices that the documented limit changed. That is why you need both: the collection checks behaviour from the outside, and the contract test checks the promise. To make the collection notice too, add a request that sends exactly 121 characters, and expects 422.

Common mistake

Every request errors in the terminal, but works in the app

Symptom: the terminal run marks every request as [errored], with one of these, although the same collection works in the app:

  GET {{baseUrl}}/v1/categories [errored]
     getaddrinfo ENOTFOUND {{baseurl}}
  GET //v1/categories [errored]
     Invalid URI "http:////v1/categories"

Cause: the run has no usable value for baseUrl. In the first case, nothing replaced {{baseUrl}}: the command has no -e option, or points to the wrong file. In the second case, the run used the collection variable baseUrl = / from the import. This is because the environment file has no variable with exactly that name. Variable names are case-sensitive: baseURL is not baseUrl.

Fix:

1. Check the command: -e postman/local.postman_environment.json, with the correct path.
2. Open the environment file, and check the exact name baseUrl and its value.
3. For a quick check, set the value in the command: --env-var "baseUrl=http://127.0.0.1:8000".

Check your understanding

Why does the collection-level script check the error shape only when the status code is 400 or higher?

Show the answer

Only error responses have the error shape. A 200 response from /v1/classify has category, priority and so on, so the check would fail for every success. The condition makes one script correct for every request in the collection. All responses must be JSON, and all errors must have code, message and fields.

Check your understanding

Module check: the collection runs from the terminal with every test passing. Then you stop the API. What does the run show, what is its exit code, and why does the exit code matter?

Show the answer

Every request is marked [errored] with connect ECONNREFUSED 127.0.0.1:8000, the tests after it fail, and the exit code is 1. The exit code matters because programs read it, not people: a script or an automatic deployment check stops on a non-zero exit code. A check that printed errors but returned 0 would let a broken deployment pass.

See it end to endBuild a data API for city service requestsCase study

The data-API case study runs its Postman collection from the terminal, and two of its requests test the paging across requests.

Recap

• A Postman test is a pm.test in a request’s Post-response script. Test Results shows the outcome.
• Checks that apply to every request go in the collection’s script, once.
• The Collection Runner runs every request and test with the chosen Postman environment.
• Export the collection and the environment (without secrets) into the project’s postman folder.
• postman collection run <file> -e <environment> runs them from the terminal, without signing in, and exits with 1 if anything fails.
• Use the collection to check the API from the outside, and pytest to test the code in detail.

You can now explore, document and check the API with Postman. In the next module, you make the API robust: automated tests for every path, useful logs, and protection.

Next: Automated endpoint tests