Neural Networks & Deep Learning • Module A: Inside the Black BoxLesson 1: Activation Functions & Backpropagation
How does a neural network convert simple mathematical operations into high-level intelligence? The secret lies in non-linear activation functions and weight adjustments through backpropagation.
Without activation functions, a neural network is just a giant stack of linear transformations (y = wx + b). No matter how many layers you stack, a combination of linear functions is always just another linear function. Activation functions introduce non-linearity, allowing neural networks to learn complex, non-linear decision boundaries.
Toggle between activation functions and drag the slider to see how the output and derivative change across different inputs.
f(x) = max(0, x)f'(x) = x > 0 ? 1 : 0Backpropagation is how a neural network learns. After making a prediction (the forward pass), the network calculates the prediction error using a loss function. During the backward pass, we use the mathematical Chain Rule to compute the gradient of the loss function with respect to each individual weight in the network. These gradients tell us how much to adjust each weight to minimize error.
Watch data flow forward through the network, then see gradients propagate backward via the chain rule. Inputs: x₁ = 1, x₂ = 0.5 | Target: 0.8 | Activation: ReLU
Calculate the forward pass for a single neuron with a ReLU activation function.
In the next lesson, we will see how we use these calculated gradients to update our weights using optimized descent algorithms!