Configuration and secrets

What you will be able to do

Run the same code with a different source, URL or token, without editing it. And make sure that a secret token never appears in your code, your Git history or your output. The AI services in later courses all need a secret key, and this lesson is how you handle one.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The notebook has every step, task and check of this lesson, with the instructions. Its first cells make the ticket-cleaner folder as at the start of this lesson. They also install httpx and download the data files. So the notebook works without your local project. Read the explanations here, and run the code there.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 45 minutes

What you need to know

• fetch_rows and the choice of source, from Basic HTTP client use.
• Mock services with MockTransport, from Basic HTTP client use: Test without the network.
• Dataclasses, from Structured records.

What you need on your computer (only for the local route; the notebook and the boxes on this page need none of it)

• count_tickets.py that loads rows from a file or from the web service, and ticket_cleaner/remote.py. You made them in Basic HTTP client use: Guided practice. If they are missing or different, do that section again. You can also copy the files from the setup cell of this lesson’s notebook. It has every file as at the start of this lesson.
• data/tickets.csv and data/tickets.json. You downloaded them in Reading and writing files: Get the data file. If one is missing or changed, download it again with the same curl command. The checksums are in the dataset card.
• The project’s environment .venv, active in your terminal (the prompt starts with (.venv)). You made it in Isolate dependencies: Guided practice. If it is missing or broken, delete it and make it again from the lock file, as in Make the environment again from nothing.
• httpx 0.28.1 in that environment. You installed it in Isolate dependencies: Worked example. Check it with python -c "import httpx; print(httpx.__version__)". If it is missing, run python -m pip install -r requirements-lock.txt.

Tested with Python 3.14.6 and httpx 0.28.1 on macOS, and Pyodide 314.0 (browser). The notebook was tested in Jupyter.

Configuration is not code

At the end of the last lesson, count_tickets.py had two constants that you edited by hand: SOURCE and TICKETS_URL. They are configuration: values that change from one place or one run to the next, while the code stays the same. A colleague wants the CSV file, the test server needs a different URL, and the real service needs a token.

If configuration is in the code, every change is an edit to the code. Edits get committed by mistake, and a secret in the code is a secret in the Git history. So you separate them:

Belongs in the code	Belongs in configuration
How to validate a record	Which file or URL to read
The allowed statuses	The address of a server
The report format	A token or password (a secret)
A safe default value	A value that is different on your computer and a colleague’s

Environment variables

An environment variable is a name and a text value that the terminal gives to every program that it starts. You have seen one already: PATH. Programs read them, and they are not in any file of your project, so they cannot be committed by accident.

Set one in the terminal. It stays set until you close that terminal:

macOS

Terminal

export TICKETS_SOURCE=remote
echo $TICKETS_SOURCE

Windows

PowerShell

$env:TICKETS_SOURCE = "remote"
$env:TICKETS_SOURCE

Linux

Terminal

export TICKETS_SOURCE=remote
echo $TICKETS_SOURCE

Output

remote

To remove it again, run unset TICKETS_SOURCE on macOS and Linux, or Remove-Item Env:TICKETS_SOURCE in PowerShell. On macOS and Linux, you can also set a variable for one command only: TICKETS_SOURCE=remote python count_tickets.py.

In Python, the os module reads environment variables. There are two ways, and the difference matters:

Code	If the variable is not set
os.environ.get("TICKETS_SOURCE", "data/tickets.csv")	Returns the default, "data/tickets.csv".
os.environ["TICKETS_SOURCE"]	Stops with KeyError: 'TICKETS_SOURCE'.

Use .get() with a default when there is a safe default. Use [ ] only for a setting that the program cannot work without, so that it stops at once with the name of what is missing.

Secrets

A secret is a value that gives access: a token, a password or an API key. Anyone who has it can act as you, and some services charge you for what they do. Four rules:

1. Not in the code. Read it from an environment variable.
2. Not in a file that you commit. If you keep secrets in a local file, add the file name to .gitignore first.
3. Not in the output. Do not print it, and do not log it. A screenshot or a shared log is enough to leak it.
4. If it leaks, replace it. Go to the service, cancel the old token, and make a new one. Deleting the commit is not enough: copies of the history can exist elsewhere. The Git course shows how to keep secrets out of a repository.

A web service usually takes a token in a request header called Authorization, as Bearer followed by the token. The token travels in the request, and the server checks it. The service answers 401 Unauthorized if the token is missing or wrong.

Worked example: a settings module

Amira puts all the configuration in one module, config.py. Every other module gets settings from it. So one place reads the environment, and one list shows every setting that the program has.

ticket_cleaner/config.py

import os
from dataclasses import dataclass, field

DEFAULT_URL = "https://raw.githubusercontent.com/Nextia-Intelligence/nextia-learning-labs/main/C02/data/tickets.json"


@dataclass(frozen=True)
class Settings:
    source: str
    tickets_url: str
    api_token: str | None = field(repr=False)


def load_settings() -> Settings:
    """Read the settings from environment variables, with safe defaults."""
    return Settings(
        source=os.environ.get("TICKETS_SOURCE", "data/tickets.csv"),
        tickets_url=os.environ.get("TICKETS_URL", DEFAULT_URL),
        api_token=os.environ.get("TICKETS_API_TOKEN") or None,
    )

Part	Why
field(repr=False)	When you print a Settings object, the token is left out. Printing the settings to check them is common, and must not leak the secret.
os.environ.get("TICKETS_API_TOKEN") or None	A variable that is set to an empty string counts as “no token”.
No default for the token	There is no safe default for a secret. Without one, the program sends no token.

Run Python in your browser

Settings without leaks

The token is in the settings, but the printed line does not show it. Delete = field(repr=False) from the api_token line, run it again, and see what a log line would leak.

Python code (you can change it)import os from dataclasses import dataclass, field @dataclass(frozen=True) class Settings: source: str tickets_url: str api_token: str | None = field(repr=False) def load_settings() -> Settings: return Settings( source=os.environ.get("TICKETS_SOURCE", "data/tickets.csv"), tickets_url=os.environ.get("TICKETS_URL", "https://tickets.example/api"), api_token=os.environ.get("TICKETS_API_TOKEN") or None, ) print("Defaults:", load_settings()) # In the terminal, you set these before you start the program. os.environ["TICKETS_SOURCE"] = "remote" os.environ["TICKETS_API_TOKEN"] = "test-token-123" settings = load_settings() print("Changed: ", settings) print("Token is set:", settings.api_token is not None)

fetch_rows gets a token parameter, and sends the header only when there is a token:

ticket_cleaner/remote.py

import httpx


def fetch_rows(
    url: str, token: str | None = None, client: httpx.Client | None = None
) -> list[dict]:
    """Download raw records as JSON from a ticket service."""
    headers = {"Authorization": f"Bearer {token}"} if token else {}
    client = client or httpx.Client(timeout=10.0)
    with client:
        response = client.get(url, headers=headers)
        response.raise_for_status()
        return response.json()

{...} if token else {} is a conditional expression: one value if the condition is true, another one if not. It is a short form of if and else that gives a value.

The script loses its constants, and gets everything from the settings:

count_tickets.py

from pathlib import Path

from ticket_cleaner.config import Settings, load_settings
from ticket_cleaner.files import read_rows, write_json
from ticket_cleaner.parsing import split_records
from ticket_cleaner.remote import fetch_rows
from ticket_cleaner.report import build_summary

OUTPUT_FILE = Path("reports/summary.json")


def load_rows(settings: Settings) -> list[dict]:
    """Return raw records from a file, or from the web service for "remote"."""
    if settings.source == "remote":
        return fetch_rows(settings.tickets_url, token=settings.api_token)
    return read_rows(Path(settings.source))


def main() -> None:
    settings = load_settings()
    print(f"Source: {settings.source}")
    rows = load_rows(settings)
    tickets, rejected = split_records(rows)
    for record in rejected:
        print(f"Row {record.row} rejected: {'; '.join(record.problems)}")

    summary = build_summary(tickets, rejected, status="open")
    write_json(summary, OUTPUT_FILE)
    print(f"{len(rows)} rows: {len(tickets)} valid, {len(rejected)} rejected.")
    print(f"Report: {OUTPUT_FILE}")


if __name__ == "__main__":
    main()

Document the settings

A setting that only exists in your terminal is invisible to the next person. Document the names, but not the secret values, in a file called .env.example in the project folder:

.env.example

# Settings for ticket-cleaner. Set them as environment variables.
# TICKETS_SOURCE: a .csv or .json file, or "remote". Default: data/tickets.csv
TICKETS_SOURCE=data/tickets.csv
# TICKETS_URL: the web service for "remote". Default: the course's public file.
TICKETS_URL=
# TICKETS_API_TOKEN: secret. Never write a real token in this file.
TICKETS_API_TOKEN=

Many projects also keep their real values in a file called .env, which a tool loads when the program starts. If you do that, put .env in .gitignore before you write anything in it. This course sets the variables in the terminal, so it does not need that tool.

Guided practice: one program, three sources

1. Make ticket_cleaner/config.py, update remote.py, and replace count_tickets.py with the code above.

2. Make .env.example.

3. Make sure that TICKETS_SOURCE is not set (run the remove command from above), then run the script:

   python count_tickets.py

   Output (first and last lines)

   Source: data/tickets.csv
   Row 7 rejected: unknown status 'opne'
   ...
   15 rows: 9 valid, 6 rejected.
   Report: reports/summary.json

4. Set TICKETS_SOURCE to data/tickets.json, and run it again. The first line is Source: data/tickets.json, and the counts are the same.

5. Set TICKETS_SOURCE to remote, and run it again. The program downloads the JSON. The counts are the same.

6. Check that nothing was lost in any run: in reports/summary.json, valid_records plus the number of entries in rejected is 15.

Your turn: a service that needs a token

The real ticket service of a company checks a token. Write try_token.py: a mock service that answers 401 without the header Authorization: Bearer test-token-123, and the tickets with it. Then call fetch_rows two times, without and with the token. Read the token from the environment variable TICKETS_API_TOKEN, not from the code.

Show one answer

try_token.py

import httpx

from ticket_cleaner.config import load_settings
from ticket_cleaner.remote import fetch_rows

URL = "https://tickets.example/api/tickets"


def mock_service(request: httpx.Request) -> httpx.Response:
    if request.headers.get("Authorization") != "Bearer test-token-123":
        return httpx.Response(401, text="missing or wrong token")
    return httpx.Response(200, json=[{"id": "T-2001", "status": "open"}])


def main() -> None:
    settings = load_settings()
    client = httpx.Client(transport=httpx.MockTransport(mock_service))
    print(fetch_rows(URL, token=settings.api_token, client=client))


if __name__ == "__main__":
    main()

Without the variable, the program stops with HTTPStatusError: Client error '401 Unauthorized'. Set TICKETS_API_TOKEN to test-token-123 (with export or $env:), and run it again: it prints the list. The token value is in your terminal only. It is not in the code, the output, or a committed file.

Common mistake

A setting that does not change anything

Symptom: you set TICKETS_SOURCE, but the program still prints Source: data/tickets.csv.

Cause: an environment variable belongs to one terminal. Usually it was set in a different terminal window or tab, for example the terminal in VS Code instead of a separate one. Other causes: a typing mistake in the name (TICKET_SOURCE), or, in PowerShell, TICKETS_SOURCE=remote without $env:, which does not set anything.

Fix:

1. In the terminal where you run the program, show the value: echo $TICKETS_SOURCE (macOS and Linux) or $env:TICKETS_SOURCE (PowerShell). If the output is empty, the variable is not set in this terminal.
2. Set it again in this terminal, with the exact name from config.py.
3. Run the program again. The Source: line shows the value that the program really used, which is why main() prints it.

Check your understanding

A teammate asks for your API token in a chat so they can run the program. What do you do instead?

Show the answer

Do not send it. A token in a chat is a leaked token. The teammate gets their own token from the service, and sets it as TICKETS_API_TOKEN in their own terminal. .env.example tells them the name of the variable. If you sent it already, cancel the token at the service and make a new one.

Check your understanding

Module check: with each of the three sources, show that the program keeps every valid record and reports every invalid one. What numbers do you compare?

Show the answer

Run the program for each source: data/tickets.csv, data/tickets.json and remote. Then compare three numbers: the input rows (15), valid_records in the report (9), and the entries in rejected (6). Valid plus rejected must equal the input rows, and each rejected entry must have a row number and a reason. The same numbers for all three sources show that the source does not change the result.

Recap

• Configuration is what changes between runs or places: sources, addresses and secrets. Keep it out of the code.
• Environment variables belong to one terminal. Read them with os.environ.get(name, default), in one settings module.
• Secrets are never in code, committed files or output. field(repr=False) keeps a token out of printed settings.
• Document every setting in .env.example, without real secret values.
• If a secret leaks, cancel it at the service and make a new one.

You can now read data from files and services, and configure the program safely. In the next module, you find and fix failures, and write tests that run automatically.

Next: Read failures