Deep learning, stated precisely

Most people carry a roughly-correct picture of neural networks. The gaps are usually terminology and ordering, and they matter once you start reading real training code.

What a neuron computes

z = w1·x1 + w2·x2 + ... + b        weighted sum, then add bias
a = activation(z)                  non-linearity

x = inputs   w = weights   b = bias
z = pre-activation value   a = neuron output

The bias is added before the activation. ReLU is the "if z < 0, output 0" step — the bias is not the if/else; it just shifts where that threshold sits. Weights and biases start random and are the only things training changes.

Backprop and gradient descent are two steps

1. 01Forward pass: run inputs through the network, produce a prediction.
2. 02Loss: compare prediction to target.
3. 03Backpropagation: apply the chain rule from the last layer backward to get the gradient of the loss with respect to every parameter.
4. 04Gradient descent: nudge each parameter in the direction that reduces loss.
5. 05Repeat.

Backpropagation computes gradients. Gradient descent uses them to update parameters. They are often said in one breath, but they are separate operations.

Each layer builds on the last layer's output

Layer 2 does not see the raw input again. It sees Layer 1's activations. Every layer works on the representation the previous layer produced, which is why depth buys abstraction:

pixels -> edges -> curves / corners -> shapes -> object parts -> full object

No neuron is told "you detect edges" or "you detect eyes". If a weight pattern reduces loss, training keeps nudging it that way, and useful feature detectors can emerge. More precisely: layers learn useful representations, and some of those happen to line up with features a human would name — especially in CNNs.

Deeper is not automatically better

More layers mean more capacity and a harder optimization problem: vanishing or exploding gradients, overfitting, and diminishing returns past a point. This is the problem residual connections and normalization exist to manage — the subject of the next note.

Solid here: neurons, weights, biases, activations, loss, backprop, gradient descent, layer chaining, feature hierarchy. Still practising: writing training loops from scratch, reading tensor shapes fluently, debugging real training runs, regularization mechanics.