Why can two layers solve a pattern that one linear neuron cannot?

examplepredictionlossgradientchanged parameters

01 · Smallest useful mechanism

Hidden layers learn representations that can make a difficult boundary simple.

One linear neuron can draw only one boundary. A hidden layer can compute intermediate features such as OR-like and AND-like responses. The next layer no longer sees the raw inputs; it sees a transformed representation. Composition changes the geometry of the problem, allowing a final weighted sum to separate cases that previously overlapped.

XOR is not linearly separable in its input coordinates, but two hidden features can remap it into a space where one final linear decision succeeds.

02 · Experiment

Test the prediction

Exact XOR construction · handcrafted hidden features

The network is intentionally small and inspectable. Its features are declared to reveal why representation changes can make a boundary linear.

Input representation

x₁ and x₂

Before layer 1

layer 1

Hidden representation

hOR and hAND

Linearly separable

Choose an XOR example

prediction 1 · target 1

Inputs

[0, 1]

The original coordinates make XOR non-linear.

→

Hidden features

OR score

0+1−0.5 = 0.5

hOR = 1

AND score

0+1−1.5 = -0.5

hAND = 0

→

Output layer

1 − 2(0) − 0.5 = 0.5

XOR = 1

Complete truth table

x₁	x₂	target	hOR	hAND	output
0	0	0	0	0	0
0	1	1	1	0	1
1	0	1	1	0	1
1	1	0	1	1	0

Depth composes functions

The second layer does not see raw x₁ and x₂. It receives features created by the first layer.

Representation changes geometry

A problem that needs multiple boundaries in input space can become linearly separable in hidden space.

Real networks learn the features

These hard steps and weights are handcrafted for inspection. Backpropagation normally learns differentiable features from data.

Loading the learning experiment…

03 · Reason from evidence

A “deep” XOR classifier behaves exactly like one linear layer and never fits all four examples.

Your brief: Determine whether depth actually changed the hypothesis class. Show the smallest evidence that would confirm or refute your diagnosis.

Evidence your answer should use

The operations between consecutive weight matrices
Hidden activations for all four XOR inputs
A comparison of W₂(W₁x) with the composed matrix (W₂W₁)x
A rerun with a declared nonlinearity and otherwise fixed settings

There is no reveal button here. A strong response names the claim, the evidence that would test it, and what the evidence cannot establish.

Your learning artifact

Write the argument you would defend

Stored only in this browser. No account required; course reset does not delete it.

Your response

04 · Check your understanding

Why does the second layer solve XOR in the activity?

Optional referenceDeep dive / reference chapterOpen the complete essay, diagrams, mathematics, exercises, glossary, and sources when you want more depth.on demand

Next: The same idea scales to images: early layers create local features and later layers compose them.