Neurons and layers

What you will be able to do

Open a neural network and see what is inside: neurons, each one a weighted sum that you already know, arranged in layers. Then you can read a network diagram and count the numbers that training must learn.

NotebookA neuron by hand, in a notebook (optional)Colab · Kaggle · GitHub

The notebook computes one neuron with plain Python and checks it against scikit-learn’s logistic regression. Then it computes a layer and counts the weights of the ticket network. You can follow the lesson without code. About 20 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 50 minutes

What you need to know

• Why Larkfield needs a router, and what a neural network adds, from What deep learning is.
• Logistic regression: a weighted sum z, then the S-curve, from Classification intuition, and the model with four features from its section Several features.

What you need on your computer

• A browser and a calculator. The optional notebook opens in Colab or Kaggle and needs no installation.

The notebook was tested with Python 3.12, NumPy 2.5, scikit-learn 1.9.1 and PyTorch 2.14.1.

Taking an umbrella: a decision from weighted clues

Before you leave home, you decide whether to take an umbrella. You do not follow a written rule. You weigh a few clues. Dark clouds count a lot. The forecast counts a lot too. A strong wind counts a little, and a blue sky counts against. You add it all up in your head, and if the total is high enough, you take the umbrella.

Two things make this decision yours. The weights: how much each clue counts for you. And your starting point: a careful person takes an umbrella at the first cloud, a relaxed person only in a storm. Change either one, and the same sky gives a different decision.

That is all one artificial neuron does. It takes some numbers, gives each one a weight, adds them up with a starting point, and turns the total into an answer.

One neuron: a weighted sum, a bias and an activation

A neuron computes its output in two steps:

1. A weighted sum plus a bias. It multiplies each input by its own weight and adds the results. Then it adds one more number, the bias: the starting point, which moves the total up or down for every input. The result is called z.
2. An activation. A fixed function turns z into the neuron’s output. One common choice is the logistic function, the S-curve that turns any z into a number between 0 and 1.

You know these two steps already. In the machine learning course, logistic regression made its score in exactly this way: z = intercept + weight × feature + …, then the S-curve. The intercept there is the bias here. A neuron with the logistic activation is a logistic regression. The two fields grew up apart, so the same numbers have two names.

Why is it called a neuron?

The name comes from the brain. A brain cell, a neuron, receives signals from many other cells. Some signals push it to fire, some hold it back, and it fires when the total is strong enough. In the 1940s and 1950s, researchers drew a simple model of this: inputs, weights, a sum and a threshold.

Do not take the link far. A real brain cell is a living cell with chemistry, timing and thousands of connections that change all the time. An artificial neuron is three lines of arithmetic. Deep learning borrowed the name and the picture, not the biology. When someone says a network “thinks like a brain”, they are speaking very loosely.

Worked example: the escalation score is one neuron

In the machine learning course, Priya trained a logistic regression that scores how likely a ticket is to be escalated. It used four features. These are its real, learned numbers, rounded to four decimals as the course printed them:

• bias (the intercept): −0.7279
• weights: priority −0.8938, word_count +0.0052, prior_tickets_90d +0.0233, created_hour −0.0096

Take one ticket: priority 1, 250 words, 3 earlier tickets in the last 90 days, written at 10 in the morning. Draw the model as a neuron, and every number has a place.

The machine learning course's escalation model drawn as one neuron. Step 1 is the weighted sum plus the bias; step 2 is the activation. The formula below it is the same model, written as logistic regression.

Text description of the diagram

Four input boxes on the left hold one ticket: priority = 1, word_count = 250, prior_tickets_90d = 3 and created_hour = 10. Each input goes through a weight: × −0.8938, × +0.0052, × +0.0233 and × −0.0096. Four arrows lead into a circle labelled “sum + bias (−0.7279)”. This is step 1, the weighted sum plus the bias. Its output is z = −0.348. The next box is the logistic S-curve: step 2, the activation. The output is the score 0.414.

A box under the picture gives the same model as logistic regression. It reads: z = −0.7279 − 0.8938 × priority + 0.0052 × word_count + 0.0233 × prior_tickets_90d − 0.0096 × created_hour, and score = 1 ÷ (1 + e^(−z)).

Now compute it. First multiply each input by its weight:

Input	Value	× Weight	=
priority	1	−0.8938	−0.894
word_count	250	+0.0052	+1.300
prior_tickets_90d	3	+0.0233	+0.070
created_hour	10	−0.0096	−0.096

Then add the four products and the bias: −0.894 + 1.300 + 0.070 − 0.096 − 0.728 = −0.348. That is z. Finally, the S-curve: z = −0.348 gives a score of 0.414. Check it with a calculator: 1 ÷ (1 + e^0.348) = 1 ÷ 2.416 = 0.414.

This is what the machine learning course’s predict_proba computes for such a ticket, up to the rounding of the weights. Nothing new happened; you only drew it differently. The new idea comes next: many neurons at once.

The problem: one neuron learns one thing

One neuron makes one weighted sum. So it can learn one feature: one combination of the inputs. For escalation, that was enough, because one score was the whole answer.

Routing a ticket needs more. Is this ticket about money? About something broken? About a broken item that arrived broken, or one that broke after a month? About signing in? Each of these is a different combination of words. One weighted sum cannot track them all at once, because it has only one weight per word. “Refund” cannot push toward payment and, at the same time, count as a hint about returns.

A layer: many neurons reading the same inputs

The answer is simple: use several neurons side by side. Each one reads all the same inputs, but each has its own weights and its own bias. So each one can learn a different combination. A group of neurons that read the same inputs is a layer.

Here is a tiny layer: two inputs and three neurons, with made-up weights. The inputs are 2 and 1.

Neuron	Weights	Bias	Weighted sum + bias
1	1 and 1	−1	1 × 2 + 1 × 1 − 1 = 2
2	1 and −1	0	1 × 2 − 1 × 1 + 0 = 1
3	−0.5 and 2	0.5	−0.5 × 2 + 2 × 1 + 0.5 = 1.5

Two numbers went in; three numbers come out: 2, 1 and 1.5. Each output is one neuron’s view of the same input. Neuron 1 adds the two inputs; neuron 2 looks at their difference; neuron 3 cares mostly about the second input. In a trained network, nobody chooses these weights. Training finds weights that make each neuron’s view useful.

This lesson leaves out the activation inside the layer for now. The next lesson shows why it is essential.

A network: layers in a chain

A network puts layers in a chain: the outputs of one layer are the inputs of the next. The diagram shows a small one, with two input numbers, a layer of four neurons, and one output neuron.

A network with 2 inputs, a hidden layer of 4 neurons and 1 output neuron. Every line is one weight, and every neuron has one bias: 17 numbers to learn.

Text description of the diagram

On the left, the input layer: two numbers, feature 1 and feature 2. In the middle, the hidden layer: four neurons, h1 to h4. On the right, the output layer: one neuron, “out”.

Each input has a line to each hidden neuron: 2 × 4 = 8 lines, so 8 weights, plus 4 biases, one per hidden neuron: 12 numbers. Each hidden neuron has a line to the output neuron: 4 weights, plus 1 bias: 5 numbers. Every line is one weight. In total, 12 + 5 = 17 numbers to learn.

The layers have names. The input layer is not really a layer of neurons: it is the input numbers themselves. The output layer gives the answer: one number here, one score per team for Larkfield. Every layer between them is a hidden layer. It is “hidden” because its values appear nowhere in the data. Nobody labels them, and nobody tells the network what they should be. Training decides what each hidden neuron computes.

Count the weights

Every line between two neurons is one weight, and every neuron in a hidden or output layer has one bias. So a layer with n inputs and m neurons has n × m weights and m biases. For the network in the diagram:

• input → hidden: 2 × 4 = 8 weights, plus 4 biases = 12;
• hidden → output: 4 × 1 = 4 weights, plus 1 bias = 5;
• total: 12 + 5 = 17 numbers.

Deep learning calls all of them, weights and biases together, the network’s parameters: the numbers that training learns. You met the word in Parameters and hyperparameters, for the intercept and the coefficients.

The running example: the ticket network

In Module 2, you build the course’s first ticket network. It reads a ticket as word counts, as the linear model of the last lesson did. It has one input for each entry of a fixed word list, built from the training tickets. That is 2,864 inputs. A hidden layer of 64 neurons follows. The output layer has 5 neurons: one score for each team.

Count its parameters with the same rule:

• input → hidden: 2,864 × 64 = 183,296 weights, plus 64 biases = 183,360;
• hidden → output: 64 × 5 = 320 weights, plus 5 biases = 325;
• total: 183,360 + 325 = 183,685 parameters.

Tomás whistles. “183,685 numbers. Who sets them all?” Nobody, by hand. Before training, they are small random numbers. With random weights, the network routes only 18.2% of the validation tickets right: no better than a random guess among five teams. Module 3 shows how training moves all 183,685 numbers, a little at a time, until the network routes 98.3% right.

Amira, the developer who will run the router in the help-desk app, notes the other side of the count. Every parameter must be stored and used for every ticket. 183,685 numbers are about 0.7 MB at 4 bytes each: tiny today. The language models of Module 7 have hundreds of millions.

When to add neurons and layers, and what it costs

More neurons in a layer let the network learn more features at once. More layers let it build features out of features. That is where a network’s power comes from.

But each parameter must be learned from examples. With many parameters and few examples, a network can learn the training data by heart, mistakes included. This is overfitting, which you know from Train, validation and test sets. More parameters also cost training time and memory. So start small, and grow only when the validation results show that the network needs it. The next lesson’s explorer shows this on a picture, and Module 4 returns to it.

Guided practice: one neuron and one count, by hand

1. A neuron has two weights, 0.5 and −1, and a bias of 0.2. Its inputs are 3 and 1. Compute the weighted sum plus the bias. You should get z = 0.7.
2. Apply the logistic function: 1 ÷ (1 + e^(−0.7)). You should get about 0.668.
3. A network has 4 inputs, a hidden layer of 3 neurons and an output layer of 2 neurons. Count the parameters of each layer, then the total.
4. Compare your answers with the example answer.

Show the example answer

1. z = 0.5 × 3 + (−1) × 1 + 0.2 = 1.5 − 1 + 0.2 = 0.7.
2. e^(−0.7) = 0.497, so the output is 1 ÷ 1.497 = 0.668.
3. Input → hidden: 4 × 3 = 12 weights + 3 biases = 15. Hidden → output: 3 × 2 = 6 weights + 2 biases = 8. Total: 23 parameters.

In the notebook, section 2 checks steps 1 and 2, and section 4 checks the count.

Your turn: a wider hidden layer

Change one condition in the ticket network: Tomás wants 128 hidden neurons instead of 64. Before you count, predict: does the number of parameters double, more than double, or less than double? Then count it.

Show the answer

Input → hidden: 2,864 × 128 = 366,592 weights + 128 biases = 366,720. Hidden → output: 128 × 5 = 640 weights + 5 biases = 645. Total: 367,365 parameters.

That is almost exactly double (183,685 × 2 = 367,370). Almost every parameter sits between the inputs and the hidden layer, and that part grows in step with the number of hidden neurons. The 2,864 inputs make the first layer expensive. Doubling it is worth it only if the validation results get better.

Common mistake

A small weight read as a small effect

Symptom: Tomás looks at the escalation neuron and says: “word_count has the smallest weight, 0.0052. It hardly matters.” But in the worked example, word_count adds +1.300 to z, the largest term of the four, larger than priority’s −0.894.

Cause: a weight is “per unit of the input”. A ticket has hundreds of words, so a small weight per word adds up. Priority goes only from 1 to 3, so it needs a big weight to matter. The size of a weight says nothing alone; its effect is weight × input.

Fix: to see what drives one output, look at the products, weight × input, as in the table of the worked example. To compare weights with each other, put the inputs on one scale first, as Preprocess with a pipeline did. Networks need this scaling too, because inputs on very different scales also make training slow.

Knowledge check

Check your understanding

In your own words: what does one neuron compute, and how is it related to logistic regression?

Show the answer

A neuron multiplies each input by its weight, adds the results and a bias, and gets z. Then an activation function turns z into the output. With the logistic function as the activation, this is exactly logistic regression. The bias is the intercept, the weights are the coefficients, and the output is the score.

Check your understanding

A network has 3 inputs, one hidden layer of 5 neurons and an output layer of 2 neurons. How many parameters does it have? Show the count per layer.

Show the answer

Input → hidden: 3 × 5 = 15 weights + 5 biases = 20. Hidden → output: 5 × 2 = 10 weights + 2 biases = 12. Total: 20 + 12 = 32 parameters.

Check your understanding

Why is a hidden layer called hidden, and who decides what its neurons compute?

Show the answer

Its values appear nowhere in the data: the data has inputs and right answers, but no values for the neurons in between. Nobody labels them or tells the network what they should be. Training decides: it sets the weights so that the hidden neurons compute whatever features help the output layer give right answers.

Recap

• A neuron computes a weighted sum of its inputs plus a bias, then an activation. With the logistic activation, it is a logistic regression. For the example ticket, the escalation model of the machine learning course gives z = −0.348 and a score of 0.414.
• The name comes from brain cells, but only as a loose picture: an artificial neuron is a few lines of arithmetic.
• A layer is many neurons that read the same inputs, each with its own weights. A network is layers in a chain: an input layer, hidden layers, and an output layer.
• A layer with n inputs and m neurons has n × m weights and m biases. The ticket network, 2,864 → 64 → 5, has 183,685 parameters; with random weights it routes 18.2% right, as chance does.

Next: so far the layer had no activation inside. The next lesson shows that without one, all those layers collapse into a single weighted sum.

Next: Why layers need activations