Basic HTTP client use

What you will be able to do

Get the ticket records from a web service instead of a file, and make the program fail clearly when the service has a problem. Most data in AI work arrives this way, including the answers from AI models.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every step, task and check of this lesson, with the instructions. Its first cells make the ticket-cleaner folder as at the start of this lesson. They also install httpx and download the data files. So the notebook works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• read_rows, split_records and build_summary, from Reading and writing files.
• JSON in Python, from Reading and writing files: CSV and JSON.
• Packages and dependencies, from Isolate dependencies.

What you need on your computer (only for the local route; the notebook needs none of it)

• The project reads data/tickets.csv and writes reports/summary.json. You made it in Reading and writing files: Guided practice. If it is missing or different, do that section again. You can also copy the files from the setup cell of this lesson’s notebook. It has every file as at the start of this lesson.
• data/tickets.csv and data/tickets.json. You downloaded them in Reading and writing files: Get the data file. If one is missing or changed, download it again with the same curl command. The checksums are in the dataset card.
• The project’s environment .venv, active in your terminal (the prompt starts with (.venv)). You made it in Isolate dependencies: Guided practice. If it is missing or broken, delete it and make it again from the lock file, as in Make the environment again from nothing.
• httpx 0.28.1 in that environment. You installed it in Isolate dependencies: Worked example. Check it with python -c "import httpx; print(httpx.__version__)". If it is missing, run python -m pip install -r requirements-lock.txt.
• An internet connection. The lesson downloads one small public file. There is no account and no cost.

Tested with Python 3.14.6 and httpx 0.28.1 on macOS. The notebook was tested in Jupyter.

Request and response

When your program gets data from a web service, it is the client, and the service is the server. They talk with HTTP, the same method that your browser uses for web pages. Each exchange has two parts:

Part	Contains	Example
Request (client → server)	A method, a URL, and headers.	GET https://…/tickets.json, with the header Accept: application/json
Response (server → client)	A status code, headers, and a body.	200 OK, the header Content-Type: application/json, and the JSON text

The method says what the client wants. GET means “send me this resource”. Other methods, such as POST, send data to the server. This lesson uses GET only. Headers are extra facts as name–value pairs, for example the type of the body. The body is the data itself, here JSON text.

The status code is a three-digit number that says how the request went. The first digit gives the group:

Code	Name	Meaning for your program
200	OK	Success. The body has the data.
400	Bad Request	Your request is wrong. Retrying the same request does not help.
401	Unauthorized	The service needs a valid key or token. See the next lesson.
404	Not Found	No resource at this URL. Check the URL.
429	Too Many Requests	You sent too many requests. Wait, then try again.
500	Internal Server Error	The server has a problem. Not your fault.
503	Service Unavailable	The server is busy or down. Try again later.

2xx means success, 4xx means a problem with the request, and 5xx means a problem on the server. Your program must check the code before it uses the body. A 404 response also has a body, but the body is an error page, not your data.

Worked example: get the tickets

The labs repository publishes the same 15 tickets as JSON at a public URL. Amira tests the request in Python’s interactive mode first. In the terminal, with the environment active, she runs python and types each line after >>>:

Interactive Python

>>> import httpx
>>> url = "https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets.json"
>>> response = httpx.get(url, timeout=10.0)
>>> response.status_code
200
>>> response.headers["content-type"]
'text/plain; charset=utf-8'
>>> rows = response.json()
>>> len(rows)
15
>>> rows[0]
{'id': 'T-1001', 'status': 'open', 'category': 'billing', 'priority': 2, 'subject': 'Charged twice for March'}

response.json() decodes the JSON body into Python values, as json.load does for a file. The result is the same list of dictionaries that read_rows gives for tickets.json. (This server says text/plain for every file. A real web service says application/json.)

timeout=10.0 means: if the server does not answer within 10 seconds, stop waiting and raise an error. Without a limit, a program can wait for a very long time for a server that never answers.

Then she asks for a file that does not exist:

Interactive Python

>>> missing = httpx.get(url.replace("tickets.json", "nothing.json"), timeout=10.0)
>>> missing.status_code
404
>>> missing.text
'404: Not Found'
>>> missing.raise_for_status()
Traceback (most recent call last):
  ...
httpx.HTTPStatusError: Client error '404 Not Found' for url 'https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/nothing.json'

raise_for_status() does nothing for a 2xx code, and raises an error for a 4xx or 5xx code. Call it before you read the body. Then a wrong URL or a broken server stops the program with a clear message, instead of becoming wrong data later. To leave interactive mode, she types exit().

The fetch function

Amira puts the request into a module, remote.py. Its contract matches read_rows: it returns a list of raw records, so the rest of the program does not care where they came from.

ticket_cleaner/remote.py

import httpx


def fetch_rows(url: str, client: httpx.Client | None = None) -> list[dict]:
    """Download raw records as JSON from a ticket service."""
    client = client or httpx.Client(timeout=10.0)
    with client:
        response = client.get(url)
        response.raise_for_status()
        return response.json()

Line	Why
client: httpx.Client | None = None	The caller can give a ready-made client. Normal code does not, and gets a new one. Tests give a client with a fake network. See the next section.
client or httpx.Client(timeout=10.0)	or gives the left value if it is not None, or else the right one. The new client has a 10-second timeout for every request.
with client:	A client keeps network connections open. The with block closes them at the end, as for a file.
response.raise_for_status()	Stop with an error for 4xx and 5xx, before the body is used.

This diagram shows where the new function fits. The rest of the program does not change:

Two sources, one shape: read_rows and fetch_rows both return raw rows, so the validation and the report work the same for a file and for a web service.

Text description of the diagram

There are three ways to get raw rows (a list[dict]). read_rows reads a CSV or JSON file. fetch_rows downloads JSON from a web service. In a test, a mock response replaces the network for fetch_rows. split_records then checks every row. Rows with no problems become Ticket objects in the tickets list. Rows with problems go to the rejected list with their reasons. build_summary uses both lists, and write_json saves reports/summary.json.

Test without the network: mocked responses

How do you check that your program handles a 503 from the server? You cannot ask a real server to fail when you want. And a check that needs the internet fails on a train, and is slow.

A mock is a stand-in for a real part, which you control. httpx has MockTransport: a client that does not use the network, and calls your function to make each response instead. Amira writes a small script to see both cases:

try_mock.py

import httpx

from ticket_cleaner.remote import fetch_rows


def answer_ok(request: httpx.Request) -> httpx.Response:
    return httpx.Response(200, json=[{"id": "T-2001", "status": "open"}])


def answer_unavailable(request: httpx.Request) -> httpx.Response:
    return httpx.Response(503, text="Service Unavailable")


def main() -> None:
    client = httpx.Client(transport=httpx.MockTransport(answer_ok))
    print(fetch_rows("https://tickets.example/api/tickets", client=client))

    client = httpx.Client(transport=httpx.MockTransport(answer_unavailable))
    fetch_rows("https://tickets.example/api/tickets", client=client)


if __name__ == "__main__":
    main()

Output (last lines)

[{'id': 'T-2001', 'status': 'open'}]
...
httpx.HTTPStatusError: Server error '503 Service Unavailable' for url 'https://tickets.example/api/tickets'

No request left the computer: the domain tickets.example does not even exist. Each mock function gets the request, and returns the response that you want to test. The first call gives data. The second stops with a clear error, before any bad data reaches split_records. In Module 5, checks like these become automatic tests.

Note

In code that you write for a project, mark mock data clearly, as here: an invented domain and invented IDs (T-2001). Then nobody mistakes a mocked result for a real one.

Guided practice: a second source

1. Make ticket_cleaner/remote.py with the code above.

2. Make try_mock.py in the project folder, and run it: python try_mock.py. You see the list, then the 503 error.

3. In count_tickets.py, add a choice of source. Add these imports and constants at the top:

   count_tickets.py (top)

   from pathlib import Path
   
   from ticket_cleaner.files import read_rows, write_json
   from ticket_cleaner.parsing import split_records
   from ticket_cleaner.remote import fetch_rows
   from ticket_cleaner.report import build_summary
   
   SOURCE = "remote"  # a file path, or "remote"
   TICKETS_URL = "https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets.json"
   OUTPUT_FILE = Path("reports/summary.json")

4. Add a function that loads the rows from the source, and use it in main() instead of read_rows(INPUT_FILE):

   count_tickets.py

   def load_rows(source: str) -> list[dict]:
       """Return raw records from a file, or from the web service for "remote"."""
       if source == "remote":
           return fetch_rows(TICKETS_URL)
       return read_rows(Path(source))
   
   
   def main() -> None:
       rows = load_rows(SOURCE)
       # ... the rest of main() does not change

5. Run python count_tickets.py. The output is the same as for the CSV file: 15 rows: 9 valid, 6 rejected.

6. Change SOURCE to "data/tickets.csv", and run it again. The result is the same.

Your turn: test a 401 and a 200 with bad data

Add two mock functions to try_mock.py, and call fetch_rows with each one. Predict the result first.

1. answer_unauthorized: status 401, with the text "missing token".
2. answer_bad_rows: status 200, with the JSON list [{"id": "T-2002", "status": "opne"}].

Show the expected result

The 401 raises httpx.HTTPStatusError: Client error '401 Unauthorized' ..., in the same way as the 503. The 200 returns the list without an error: fetch_rows checks only that the request worked. It does not check the records. That is the job of split_records, which rejects the record for missing category, missing priority and unknown status 'opne'. Each function has one job, and the diagram shows where each check happens.

Common mistake

A JSON error from a missing page

Symptom: the program stops in response.json() with an error that does not mention HTTP:

json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)

Cause: the server answered with an error, for example 404, and the body is an error page such as 404: Not Found, not JSON. The code read the body without checking the status code first. The message talks about JSON, so it hides the real problem.

Fix:

1. Call response.raise_for_status() before response.json(). The error now names the real problem: 404 Not Found and the URL.
2. Check the URL for a typing mistake. Open it in a browser to see what the server sends.
3. For a 5xx code, the problem is on the server. Try again later.

ConnectTimeout and ConnectError

httpx.ConnectTimeout: timed out means that the server did not answer within the timeout. httpx.ConnectError: [Errno 8] nodename nor servname provided, or not known (the words depend on your system) means that the computer cannot find the server’s name: check the URL and your internet connection. Both errors come from client.get, before any status code exists.

Check your understanding

A request returns status 503. Is the problem in your program or on the server? What should the program do?

Show the answer

On the server: 5xx codes are server problems, and 503 means busy or down. The program must not use the body as data. raise_for_status() stops it with a clear error. A person, or a later version of the program, can try again later. A 4xx code is different: the same request fails again until you change it.

Check your understanding

Why does fetch_rows accept a client argument, if normal code never passes one?

Show the answer

So that a test can pass a client with MockTransport. Then the test controls every response, including errors such as 503, without the network. The normal code path stays simple: with no argument, the function makes a real client with a timeout.

Recap

• A request has a method, a URL and headers. A response has a status code, headers and a body.
• 2xx is success, 4xx is a problem with the request, and 5xx is a problem on the server.
• Call raise_for_status() before response.json(), so an error page never becomes data.
• Always set a timeout. httpx clients take timeout=10.0.
• A function that accepts a client can be tested with MockTransport, without the network.

Next, you move the URL and the choice of source out of the code. Then you add a secret token, and keep it out of every file that others can see.

Next: Configuration and secrets