Why layers need activations

What you will be able to do

See why every hidden layer needs an activation. Without one, a deep network is no more powerful than one neuron: its boundary stays a straight line. With one, the boundary can bend around the data.

NotebookThe toy boundaries, in a notebook (optional)Colab · Kaggle · GitHub

The notebook trains the explorer’s networks with scikit-learn and draws their boundaries. It also shows the collapse of two layers, and computes the module practice. You can follow the lesson without code. About 20 minutes.

Open in ColabOpen in KaggleView on GitHub

Colab and Kaggle open your own copy. They need a free Google or Kaggle account. Stuck? See the solution notebook.

Before you startWhat to know and to have readyAbout 55 minutes

What you need to know

• A neuron, a layer, a hidden layer and how to count parameters, from Neurons and layers.
• The logistic function, the S-curve that turns z into a score from 0 to 1, from Classification intuition.
• Overfitting: a model that learns its training data by heart, from Train, validation and test sets.

What you need on your computer

• A browser. The optional notebook opens in Colab or Kaggle and needs no installation.

The explorer’s networks were trained with scikit-learn 1.9.1.

A straight ruler and a folding ruler

Coins and buttons lie on a table. The buttons form a ring, and the coins lie in the middle. Separate them with one straight ruler. However you turn the ruler, some buttons stay on the coins’ side. A straight line cannot go around a ring.

Now take a folding ruler, the carpenter’s kind with joints. With one joint, you can make a corner. With four joints, you can make a square around the coins. With many short pieces, you can follow almost any curve. Each joint adds one bend, and the pieces between the joints stay straight.

A neural network works like the folding ruler. This lesson shows what makes the joints: the activation. And it shows that without activations, a network is a straight ruler, however many layers it has.

The problem: stacked layers collapse

The last lesson built layers from weighted sums and left the activation out. Here is what goes wrong. Take the smallest possible network: one input x, one hidden neuron and one output neuron, with no activation.

• The hidden neuron computes h = 2x + 1.
• The output neuron computes y = 3h − 4.

Follow two inputs through it. For x = 0: h = 1, so y = 3 − 4 = −1. For x = 1: h = 3, so y = 9 − 4 = 5. Each step of 1 in x adds 6 to y.

Now put the first line into the second: y = 3 × (2x + 1) − 4 = 6x + 3 − 4 = 6x − 1. Two layers with four parameters give one straight line, which one neuron with two parameters could give too. The middle layer added nothing.

This is not an accident of these numbers. A weighted sum of weighted sums is always one weighted sum. With 2 inputs, 16 hidden neurons and no activation, the 65 parameters still describe one straight line. You can stack 100 such layers, and the whole network is still one neuron in disguise.

The idea: an activation bends the line

The fix is to put a function that is not a straight line after each hidden neuron. This is the neuron’s activation. The most common one is very simple:

ReLU(z) = max(0, z): keep z if it is positive; replace it with 0 if it is negative.

The name is short for rectified linear unit. In electronics, a rectifier lets current through in one direction only. ReLU lets positive values through and blocks negative ones.

Put ReLU after the hidden neuron of the tiny network: h = max(0, 2x + 1), and y = 3h − 4.

x	2x + 1	h = max(0, 2x + 1)	y = 3h − 4	without ReLU: 6x − 1
−2	−3	0	−4	−13
−1	−1	0	−4	−7
−0.5	0	0	−4	−4
0	1	1	−1	−1
1	3	3	5	5
2	5	5	11	11

Read the y column. Left of x = −0.5, y stays at −4: the hidden neuron is “off”. Right of it, y rises by 6 for each step: the neuron is “on”. So the output is two straight pieces with a bend at x = −0.5. One ReLU neuron made one joint in the ruler.

• No activation: y = 6x − 1
• With ReLU: y = 3 × max(0, 2x + 1) − 4

The same two-layer network, with and without ReLU in the hidden neuron. Without it, the result is one straight line. With it, the line bends where the hidden neuron switches on.

Text description of the chart

The horizontal axis is the input x, from −3 to 3; the vertical axis is the output y. The dashed line is the network without an activation: the straight line y = 6x − 1, from −19 at x = −3 to 17 at x = 3. The solid line is the network with ReLU. It is flat at y = −4 for every x below −0.5. Then it rises with the same slope as the dashed line, and the two lines meet from x = −0.5 on. The bend is at x = −0.5.

With more hidden neurons, each one has its own switch point, so the output gets more bends. With two inputs instead of one, each ReLU neuron folds the plane along a line, and the pieces form a shape. That is what the explorer below shows.

Decision boundaries on toy data

A decision boundary is where a classifier changes its answer: on one side it says group A, on the other side group B. For a model that gives a score from 0 to 1, it is where the score is exactly 0.5. One neuron with the logistic activation is a logistic regression. Its boundary is always a straight line, because z = 0 along a straight line.

To see boundaries, the data must fit on a screen. So the toy data has two features, two groups and 200 points. Group B (triangles) lies in a disc in the middle; group A (circles) lies in a ring around it. A second set of 200 new points, made the same way, checks each network on data it did not learn from.

Priya really trained each network in the explorer. Each one has 2 inputs, one hidden layer of 0, 2, 4, 8 or 16 neurons, and one logistic output neuron. The activation of the hidden layer is none or ReLU. The shading and the boundary come from each network’s real scores.

Explore

Bend the boundary: hidden neurons and the activation

Weights and biases

65

Right on the 200 training points

51.0%

Right on 200 new points

52.0%

16 hidden neurons and no activation: the boundary is still one straight line. Without an activation, the two layers add up to one weighted sum, so 65 weights do the work of 3.

• Group A (circles)
• Group B (triangles)
• Boundary: the network is 50:50

Real decision boundaries of small networks on 200 toy points: scikit-learn MLPClassifier. The shading shows the network's answer; the line is where it is 50:50.

Text description of the explorer

The explorer shows 200 training points: group B, 100 triangles, in a disc in the middle; group A, 100 circles, in a ring around it. Shading shows where the network says group A and where it says group B. A dark line marks the boundary. Two controls choose the number of hidden neurons (0, 2, 4, 8, 16) and the activation of the hidden layer (none or ReLU). The readout gives the number of parameters and the share of right answers on the training points and on 200 new points.

No activation, with 0, 2, 4, 8 or 16 hidden neurons: the boundary is always one straight line across the disc. About half of the points are right. With 0 neurons: 3 parameters, 51.0% of the training points and 52.0% of the new points right. With 16 neurons and no activation: 65 parameters, 51.0% of the training points and 52.0% of the new points right.

ReLU: with 2 neurons, the boundary has two straight pieces and a corner: 9 parameters, 66.0% of the training points and 65.0% of the new points right. With 4 neurons, it is a four-sided shape around the disc: 17 parameters, 97.5% of the training points and 96.5% of the new points right. With 8 neurons, a shape with more sides: 33 parameters, 99.0% of the training points and 98.0% of the new points right. With 16 neurons, almost a circle: 65 parameters, 100.0% of the training points and 100.0% of the new points right.

What did you see? With no activation, the boundary is one straight line for every number of neurons. Sixteen neurons get 51.0% of the training points right, the same as one neuron, 51.0%: a coin toss. The extra 62 parameters bought nothing.

With ReLU, each hidden neuron adds one straight piece. Two neurons make a corner (66.0%). Four make a closed shape with four sides (97.5%). Eight and sixteen make shapes that follow the disc closely (99.0% and 100.0%). The new points agree: 100.0% with 16 neurons. Same data, same number of parameters as the straight line; the activation alone made the difference.

The activations you will meet

ReLU is the usual choice for hidden layers: it is cheap to compute, and networks with ReLU train well. You will see others, but they share one job: they are not straight lines, so the layers cannot collapse into one.

The output layer has a different job: it must give an answer in the right form. For a yes-or-no answer, the output neuron uses the logistic function, which deep learning usually calls the sigmoid (“S-shaped”). For one answer out of several, such as Larkfield’s five teams, the output layer gives five scores. A function called softmax turns them into five probabilities that add up to 1. Module 3 teaches softmax, in Loss: how wrong is the network?.

• ReLU
• Logistic (sigmoid)

Two activations. ReLU is 0 for every negative z and z itself for a positive z. The logistic function (sigmoid) bends every z into a value between 0 and 1.

Text description of the chart

The horizontal axis is z, from −4 to 4. ReLU is a flat line at 0 for every negative z. From z = 0 on, it rises with slope 1 and reaches 4 at z = 4. The logistic function is an S-curve: 0.018 at z = −4, 0.5 at z = 0 and 0.982 at z = 4. It never goes below 0 or above 1.

The running example: the ticket network

The course’s ticket network has a ReLU after its hidden layer: 2,864 word counts → 64 neurons → ReLU → 5 team scores. The ReLU is what lets the 64 hidden neurons become 64 different features. Without it, the whole network would be one weighted sum of the word counts: a linear model, like the logistic regression of the first lesson.

Here is an honest note. On the tickets, that linear model already routes 98.8% right, so the bends matter much less there than on the rings. They matter when the pattern is curved: when a word means one thing alone and another thing next to other words. Modules 5 and 6 look for such patterns in the tickets.

More neurons: more bends, more risk

If 16 neurons fit the rings so well, are more neurons always better? Look at a second toy dataset, the “two moons”. Its two groups are shaped like two curved bananas that reach into each other, with more noise than the rings. The points of the two groups overlap a little, so no boundary can be perfect.

Explore

Two moons: more bends, more risk

Weights and biases

33

Right on the 200 training points

96.5%

Right on 200 new points

94.0%

8 hidden neurons with ReLU: the boundary is made of straight pieces that meet at bends. It gets 96.5% of the training points right, against 83.5% for one neuron.

• Group A (circles)
• Group B (triangles)
• Boundary: the network is 50:50

Real decision boundaries on the two-moons toy data: scikit-learn MLPClassifier. Compare the training points with the new points.

Text description of the explorer

The same explorer on the two-moons data. 100 circles (group A) form an upper moon; 100 triangles (group B) form a lower moon that reaches into it. The groups overlap a little near the middle.

No activation: always a straight diagonal line, 65 parameters, 83.5% of the training points and 85.0% of the new points right, the same as one neuron. ReLU with 2 neurons: almost the same line, 9 parameters, 83.0% of the training points and 85.5% of the new points right. ReLU with 4 neurons: the boundary bends around the tips of the moons, 17 parameters, 95.5% of the training points and 93.5% of the new points right. ReLU with 8 neurons: 33 parameters, 96.5% of the training points and 94.0% of the new points right. ReLU with 16 neurons: the boundary has sharp spikes that reach toward single training points, 65 parameters, 97.0% of the training points and 90.5% of the new points right.

With 8 ReLU neurons, the network gets 96.5% of the training points and 94.0% of the new points right. With 16, it gets more training points right, 97.0%, but fewer new points, 90.5%. Its extra bends reach out to single noisy training points. They follow chance details of these 200 points, not the real shape of the moons. That is overfitting, now in a picture.

Also look at 2 ReLU neurons: 83.0%, no better than one neuron. Two bends were not enough here, and this run found almost the same line. Results like this also depend on the random starting weights: a network is not guaranteed to use every neuron it has.

So the rule for width and depth is: more neurons and more layers give more bends, and more risk of learning noise. Choose the size with the validation data, as Control complexity in the machine learning course chose a tree’s depth. Module 4 adds tools that let a large network learn without overfitting.

Guided practice: read the explorer

1. In the first explorer (the rings), make your prediction, then select I have made my prediction.
2. Keep None. Select 0, 2, 4, 8 and 16 neurons. Each time, read the line and the share of training points right. You should see one straight line every time, and 50.5% to 51.0%.
3. Select ReLU and 2. You should see a corner and 66.0%.
4. Select 4, 8 and 16. You should see 97.5%, 99.0% and 100.0% on the training points.
5. Select 0 with ReLU. You should see the straight line again: with no hidden layer, there is nowhere to put the activation.
6. Write two sentences: what does the activation change, and what do extra neurons change?

Show an example answer

“Without an activation, the boundary stays one straight line, however many neurons I add, because the layers add up to one weighted sum. With ReLU, each hidden neuron adds a straight piece with a bend, so more neurons let the boundary go around the disc.”

Your turn: choose the size on the moons

Change one condition: the data. In the second explorer, you predicted which number of ReLU neurons gets the most new points right. Now check it: select ReLU, then 2, 4, 8 and 16 neurons, and write both accuracies for each. Which size would you choose, when only the answers on new points matter?

Show the answer

ReLU neurons	Training points right	New points right
2	83.0%	85.5%
4	95.5%	93.5%
8	96.5%	94.0%
16	97.0%	90.5%

Choose 8 neurons: it gets the most new points right, 94.0%. Sixteen neurons look better on the training points but worse on new points; that gap is the sign of overfitting. Four neurons are almost as good with half the parameters, so they are a fair choice too. With only 200 new points, the difference between 4 and 8 is too small to matter: half a point, 1 point in 200. The drop with 16 is larger.

Common mistake

A deep network without activations

Symptom: Tomás builds a network for the rings with three hidden layers of 64 neurons each: 8,577 parameters. It trains without an error, but it gets only 49.5% of the training points right, no better than one neuron. With ReLU after each hidden layer, the same network gets 100%.

Cause: he forgot the activations. A layer in code, such as PyTorch’s nn.Linear in Module 2, computes only the weighted sum plus the bias. It has no activation unless you add one. So his three layers collapse into one weighted sum, and the boundary is a straight line.

Fix: put an activation, such as ReLU, after every hidden layer. Do not put ReLU after the output layer: the output gets the sigmoid or softmax for its answer. To check a network, write down its layers in order and look for an activation between every two weight layers.

Knowledge check

Check your understanding

Two layers without an activation compute h = 4x − 2 and then y = −0.5h + 3. Write y as one line in x. What does this show?

Show the answer

y = −0.5 × (4x − 2) + 3 = −2x + 1 + 3 = −2x + 4. The two layers give one straight line, which one neuron could give too. A weighted sum of weighted sums is one weighted sum, so layers without activations add no power.

Check your understanding

Predict: on the rings, what boundary does a network with 8 hidden neurons and no activation draw? And with 8 hidden ReLU neurons?

Show the answer

Without an activation: one straight line, and about half of the points right (50.5% in the explorer). The network is one neuron in disguise. With 8 ReLU neurons: a closed shape of straight pieces around the disc. It gets 99.0% of the training points and 98.0% of the new points right.

Module practice

The module practice joins the three lessons. First compute a tiny network by hand. Then use the explorer to explain why a network without activations cannot separate curved data.

Worked example: a tiny network by hand

The network has 2 inputs, 2 hidden neurons with ReLU, and 1 output neuron with the logistic function. The weights are made up so that you can follow every number.

• hidden neuron 1: weights 2 and −1, bias 0;
• hidden neuron 2: weights −1 and 1, bias −2;
• output neuron: weights 1.5 and −1 (for hidden neurons 1 and 2), bias −1, then the logistic function.

The input is (2, 1).

1. Hidden neuron 1: 2 × 2 + (−1) × 1 + 0 = 3. ReLU keeps it: 3.
2. Hidden neuron 2: (−1) × 2 + 1 × 1 − 2 = −3. ReLU replaces it with 0: this neuron is off.
3. Output: z = 1.5 × 3 + (−1) × 0 − 1 = 3.5.
4. Logistic: 1 ÷ (1 + e^(−3.5)) = 0.971. The network is 97% sure of group B.

What did the ReLU change? Without it, hidden neuron 2 would pass on −3, and z = 1.5 × 3 − 1 × (−3) − 1 = 6.5, a score of 0.998. The ReLU switched one neuron off for this input. For another input, another neuron is off. That switching is what makes the bends.

Your task

1. Compute the same network for the input (0, 3): both hidden neurons before and after ReLU, then z, then the score.
2. Use the first explorer to explain to Omar, in three or four sentences, why a network without activations cannot separate the rings. Use the numbers that you see.

Show the model answer

1. The input (0, 3). Hidden neuron 1: 2 × 0 + (−1) × 3 + 0 = −3, and ReLU gives 0. Hidden neuron 2: (−1) × 0 + 1 × 3 − 2 = 1, and ReLU keeps 1. Output: z = 1.5 × 0 + (−1) × 1 − 1 = −2. Score: 1 ÷ (1 + e²) = 1 ÷ 8.389 = 0.119, so the network says group A. This time, neuron 1 is off and neuron 2 is on: the opposite of the worked example.

2. For Omar. “Without activations, every layer is a weighted sum, and a weighted sum of weighted sums is one weighted sum. So the whole network is one neuron, and its boundary is one straight line. In the explorer, 16 neurons without an activation get 51.0% of the points right, the same as 0 neurons. A straight line cannot go around a disc. With ReLU, the same 16 neurons get 100%, because each neuron adds a bend.”

A complete answer computes both hidden neurons before and after ReLU, and explains the collapse with the explorer’s numbers, not only with words.

Module check

Check your understanding

Retrieval: name the two steps that one neuron computes, and give the number of parameters of a layer with 10 inputs and 4 neurons.

Show the answer

Step 1: the weighted sum of the inputs plus the bias, z. Step 2: the activation, a fixed function of z, such as ReLU or the logistic function. A layer with 10 inputs and 4 neurons has 10 × 4 = 40 weights and 4 biases: 44 parameters.

Check your understanding

Explain or predict: Tomás doubles the number of hidden layers of a network without activations, from 2 to 4. What happens to its decision boundary? What one change would let it bend?

Show the answer

Nothing changes in what it can draw. Without activations, 4 layers collapse into one weighted sum, just as 2 layers do. So the boundary stays one straight line. It only has more parameters to train. Adding an activation such as ReLU after every hidden layer lets the boundary bend: each hidden neuron then adds a straight piece with a corner.

Check your understanding

Completion check: a neuron has weights 0.4 and −0.2, a bias of 0.1 and the logistic activation. Compute its output for the inputs 5 and 3, and explain why stacking linear layers adds no power.

Show the answer

z = 0.4 × 5 + (−0.2) × 3 + 0.1 = 2 − 0.6 + 0.1 = 1.5. The output is 1 ÷ (1 + e^(−1.5)) = 1 ÷ 1.223 = 0.818.

Stacking linear layers adds no power because a weighted sum of weighted sums is again one weighted sum. For example, h = 2x + 1 and y = 3h − 4 give y = 6x − 1. Any number of layers without activations equals one layer, so the decision boundary stays a straight line. An activation between the layers, such as ReLU, breaks the collapse.

Recap

Module 1 recap

• Deep learning learns its own features from raw input in layers. On Larkfield’s tickets, the machine learning course’s linear model routes 98.8% right and remains the router to use. The course uses the task to see how networks learn.
• A neuron is a weighted sum plus a bias, then an activation. With the logistic activation it is a logistic regression. A layer is many neurons reading the same inputs; a network is layers in a chain. The ticket network, 2,864 → 64 → 5, has 183,685 parameters.
• Without activations, layers collapse: h = 2x + 1 and y = 3h − 4 are one line, y = 6x − 1. With ReLU, max(0, z), each hidden neuron adds a bend. On the rings, 16 neurons give 51.0% without an activation and 100% with ReLU.
• More neurons give more bends and more risk of overfitting. On the moons, 16 ReLU neurons get 97.0% of the training points, but only 90.5% of new points; 8 neurons get 94.0%.

Reset: this module installs nothing and saves nothing on your computer. To start an explorer again, reload the page. If you used a notebook, restart it: in Colab, choose Runtime › Restart session; in Kaggle, Run › Restart & clear cell outputs. Then run all cells again. To start completely fresh, open the notebook again from the link at the top of the lesson.

Next, Module 2 turns these pictures into code. The numbers of a network live in tensors, and you must read each tensor’s shape before you run anything.

Next: Tensors and shapes