Notebook discipline

What you will be able to do

Use a notebook for what it is good at: trying ideas quickly. But do not trust a result that exists only because of the order in which you ran the cells. Then move the code that you want to keep into the project, where it has tests.

NotebookDo this lesson in a notebookColab · Kaggle · GitHub

The lab notebook makes each kind of hidden state happen on purpose, so that you can see it. It downloads the course’s 15 tickets and checks them first. It needs no install and no local project. About 30 minutes. You use it in the guided practice below.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 40 minutes

What you need to know

• Modules and imports, and that an import runs a module once, from Modules and imports.
• Function contracts, from Function contracts: The contract, and tests, from Focused tests. You move code from a notebook into the project with both.

What you need on your computer

• For the lab notebook: a free Google account (for Colab) or Kaggle account. Or use VS Code with the Jupyter extension, as in the box under the guided practice. The notebook needs no packages beyond Python itself.
• For Your turn, on your computer, ticket-cleaner with its tests. You made them in Focused tests: Guided practice and Your turn. You downloaded data/tickets.csv in Reading and writing files. If the tests do not pass, do those sections again.
• The project’s environment .venv, active in your terminal (the prompt starts with (.venv)). You made it in Isolate dependencies: Guided practice. If it is missing or broken, delete it and make it again from the lock file, as in Make the environment again from nothing. VS Code installs ipykernel in it when you select it as the kernel. Do not add ipykernel to requirements.txt.

Tested with Python 3.12 and 3.14 in JupyterLab 4.6. The notebook was tested in Jupyter.

What a notebook is

A notebook (a .ipynb file) is a page of cells. A code cell holds Python code, and its output appears under it. A text cell holds explanation. You run one cell at a time, look at the result, and change the next step. For exploring data, that is faster than running a whole script again and again. Data scientists and AI engineers use notebooks every day, for example in Google Colab, Kaggle or JupyterLab.

The code runs in a kernel: a Python program that starts with the notebook, and keeps every variable in memory until it restarts. This is the cause of the problems in this lesson. The page and the kernel are two different things:

The page shows	The kernel holds
The cells in their order, top to bottom.	The variables from every cell that ran, in the order that you ran them.
The current text of each cell.	The result of the text that the cell had when it ran.
Only the cells that exist.	Variables from cells that you deleted.

The number in [ ] next to a cell, the execution count, shows the order. [7] above [3] on the page means that the upper cell ran later.

Hidden state

When the kernel holds values that the page does not explain, the notebook has hidden state. Three ways to get it:

1. A changed cell that did not run again. You change wanted_status = "open" to "closed" and run it, but not the cell that filters the rows. The page says “closed”, and the result is still for “open”.
2. A deleted cell. You delete the cell that set priority_limit. Every later cell still works, until the kernel restarts.
3. A module that changed. You edit a .py file that the notebook imported. Python imports a module only once per kernel, so the notebook still uses the old version.

In each case, the notebook works for you today. It fails for a colleague who opens it, and for you after a restart. Worse, it can show a wrong number with no error at all.

Restart and run all

The test is simple: restart the kernel, and run all cells from the top. Now the kernel has only what the page says, in the order that the page says. If the notebook runs without an error and gives the same results, it has no hidden state. Do this before you trust a result, and every time before you share a notebook.

Tool	Menu
JupyterLab	Kernel › Restart Kernel and Run All Cells
VS Code	Restart, then Run All, in the bar above the notebook
Colab	Runtime › Restart session and run all
Kaggle	Run › Restart & clear cell outputs, then Run › Run all

Guided practice: the lab notebook

The lab notebook makes each kind of hidden state happen on purpose, so you can see it. It downloads the course’s 15 tickets and checks them first. About 30 minutes. Each code cell has a text cell before it, which says what to do and what you should see.

1. Open the notebook in Colab or Kaggle, from the box Do this lesson in a notebook at the top of this page. If you prefer to work on your computer, see the box below the steps.
2. Run the cells one at a time, from the top. Read the text cell before each code cell.
3. In Predict, then run, write your prediction before you run the cell. You see closed 9, although only 3 tickets are closed.
4. In A deleted cell that still counts, delete the cell, and run the next one. It still works.
5. Restart and run all. The run stops with NameError: name 'priority_limit' is not defined. Put the value back above the cell that uses it, and restart and run all again, until it ends without an error.
6. In Change one thing, predict the version number, then run. The import shows 1 False, and importlib.reload shows 2 True.
7. Do the two exercises. Each one ends with a check cell, which prints Check passed. when your answer is right.
8. Restart and run all one last time. Every cell runs, and both check cells print Check passed.

Run the notebook on your computer

In VS Code, install the Jupyter extension from Microsoft. Download the notebook with View on GitHub, then the download button, and save it in a new folder. Open it in VS Code, and select Select Kernel › Python Environments, then a Python 3.12 or newer. If VS Code asks to install ipykernel, accept. The notebook uses only Python’s own modules, so it needs nothing else.

Explore in the notebook, keep in the project

A notebook is a good place to find out what to do. It is a bad place to keep code that the project depends on. You cannot import a function from a cell, and the cells have no tests. Hidden state can change a result without warning.

So when an idea in the notebook works and you want to keep it, move it:

The notebook is where you find the rule. The project is where you keep it, test it, and run it.

Text description of the diagram

In the notebook, you load the data and look at it, try a cleaning rule, then count and compare the results. A rule that you want to keep moves to ticket_cleaner/parsing.py as a function. A calculation that you want to keep moves to ticket_cleaner/report.py. Each one gets a test in tests/. The command-line interface in ticket_cleaner/cli.py uses both, and README.md says how to set up and run it. After that, the notebook imports the functions from the package instead of keeping its own copy.

The rules for the move:

1. Make it a function with a contract. A cell that changes global variables becomes a function with parameters and a return value. See Function contracts.
2. Give it a test, made from the example that you tried in the notebook.
3. Import it in the notebook, do not copy it. Then the notebook and the program always use the same version.

ticket-cleaner already follows this pattern. In this module, you add the last parts. The next lesson adds a command to run it. The last lesson adds the tools and the README that let another person use it.

Your turn: an exploration that you keep

Open a notebook on your computer, in your ticket-cleaner folder, with the project’s .venv as the kernel. (In VS Code: Select Kernel › Python Environments › .venv. Accept the installation of ipykernel. It is a tool for you, not a dependency of the project, so do not add it to requirements.txt.)

1. In the first cell, import from the package, and read the data:

   from pathlib import Path
   
   from ticket_cleaner.files import read_rows
   from ticket_cleaner.parsing import split_records
   
   tickets, rejected = split_records(read_rows(Path("data/tickets.csv")))
   len(tickets), len(rejected)

   The output is (9, 6).

2. Explore one question in the next cells: how many urgent tickets (priority 1) are in each category?

3. Restart and run all.

4. Decide: is the answer something the report should always include? If yes, move it into report.py as a function, with a test in tests/test_report.py, and import it in the notebook.

Show one answer

Exploration in the notebook:

urgent = [ticket for ticket in tickets if ticket.is_urgent()]
[(ticket.id, ticket.category) for ticket in urgent]

The output is [('T-1003', 'login'), ('T-1006', 'shipping'), ('T-1010', 'login')]. So the answer is login: 2, shipping: 1, and count_by_category(urgent) already calculates it. No new function is needed: the notebook found that an existing function answers the question. If the help desk wants it in every report, add one line to build_summary, "urgent_by_category": count_by_category([t for t in selected if t.is_urgent()]), and a test for it.

Common mistake

It worked yesterday

Symptom: you open a notebook that worked yesterday, and it stops with NameError or ModuleNotFoundError. Or a colleague gets a different number from the same notebook.

Cause: hidden state. Yesterday, the kernel had a variable from a cell that is now deleted or changed, or a module version that is no longer on disk. Opening the notebook again started a new kernel, which has only what is on the page.

Fix:

1. Restart and run all. Note the first cell that fails.
2. Find where its missing value came from. Put that code back, in a cell above the cell that fails.
3. Restart and run all again, until it runs from top to bottom.
4. If the cell uses your own module, restart the kernel after every change to the module, or use importlib.reload.

Check your understanding

A notebook shows cells with the execution counts [1], [2], [9], [4], [5] from top to bottom. What does that tell you, and what do you do before you trust its last output?

Show the answer

The third cell ran last, after the fourth and fifth. So the fourth and fifth cells used values from before the third cell’s newest run, and the third cell’s change is not in their output. Restart and run all, then read the results again. After that, the counts are [1] to [5] in order.

Check your understanding

Why import clean_text from ticket_cleaner in a notebook, instead of copying the function into a cell?

Show the answer

With a copy, there are two versions. The function in the package changes, for example after a bug fix with a regression test. The copy in the notebook keeps the old behaviour, so the notebook gives different results from the program. An import always uses the version that has the tests.

Recap

• The kernel keeps every variable until it restarts. The order in which you ran the cells counts, not their order on the page.
• Hidden state comes from changed cells that did not run again, deleted cells, and changed modules.
• Restart and run all before you trust a result or share a notebook.
• Explore in the notebook. Keep in the project: a function with a contract, a test, and an import in the notebook.

Next, you give ticket-cleaner a command-line interface, so anyone can run it with options instead of environment variables and code changes.

Next: Command-line interface