How does a neural network work?

A network is a composed function

A feed-forward neural network takes a numeric input and transforms it through a sequence of layers. A layer usually computes many weighted sums in parallel and then applies an activation function. The output of one layer becomes the input to the next.

Read left to right: each layer remixes the current representation, the nonlinear gate changes what the next layer can express, and the final space makes a simple decision possible.

flowchart LR
  X[Numeric input] --> L1[Weighted mixing]
  L1 --> A1[Nonlinear activation]
  A1 --> H[Hidden representation]
  H --> L2[Weighted mixing]
  L2 --> Y[Output]

If every layer performed only a linear operation, the whole stack would collapse mathematically into one linear operation. Nonlinear activations prevent that collapse. They let the network represent curved boundaries and conditional patterns that a single perceptron cannot.

Hidden layers learn representations

The engineer normally specifies the input encoding, layer types, connections, and output contract, but not the meaning of every hidden unit. During training, the weights change together to reduce the loss for the complete task. Hidden layers can therefore develop intermediate features useful to later layers.

“Representation” means the pattern of numeric activations produced for an input. It is not necessarily a human-readable concept or a clean slot such as “contains a wheel.” Some directions can correlate with useful properties while the representation remains distributed, contextual, and difficult to interpret.

Training coordinates the parameters

A forward pass produces predictions and a loss. Backpropagation computes how a small change in each parameter would affect that loss, and an optimizer chooses updates. The process is local in calculation but global in effect: a useful feature in an early layer is useful because it helps the final objective across training examples.

Depth, width, connectivity, activation functions, normalization, initialization, loss, optimizer, and data all influence what can be learned and how reliably. More parameters increase capacity; they do not by themselves guarantee better generalization, truthful outputs, or an appropriate product.

From networks to Transformers

A Transformer is a neural-network architecture. It combines attention, which mixes information across token positions, with feed-forward networks that transform each position, plus residual paths and normalization. Token embeddings provide the initial numeric representations; repeated Transformer blocks update them; an output layer converts the final representation into token scores.

This connects the vocabulary without collapsing it: a perceptron is one simple linear threshold unit; a multilayer network composes learned transformations; a Transformer specifies a particular composition; a language model defines a sequence objective and output behaviour; an agent adds a harness, tools, and a loop outside the network.