Learning path

Artificial neural networks and deep learning (ANN/DL)

Build up from one neuron to multilayer networks: perceptrons, activations, forward passes, backpropagation, loss functions, convolutions, and regularization.

About this subject

A neural network is a function built from layers of simple units, each computing a weighted sum followed by a nonlinear activation. Its weights are learned by gradient descent, with backpropagation computing the gradients. Deep learning refers to networks with many layers.

This path covers the perceptron and its limits, activation functions and softmax, a multilayer forward pass, backpropagation with a gradient check, cross-entropy loss, convolutional shapes and parameter counts, and regularization with dropout. The programs use plain code without a framework so that nothing hides the mathematics. Each lesson uses a small, complete program that runs on its own. Python has the full sequence. The other languages listed below add lessons that use that language's own idioms and standard library, such as Java threads, C++ smart pointers, C# LINQ, or Node.js streams, and appear only for the topics the language supports. The programs are simplified models chosen to make an idea visible, so they are a foundation for further study rather than a replacement for a full course or for production-grade software.

Before you start: The machine-learning lessons on gradient descent, and basic derivatives.

1
Python
Readable code for automation, data, and first programs.
The perceptron and linear separability
Train a single artificial neuron with the perceptron rule, and see why it cannot learn XOR.
Activation functions and softmax
Compare sigmoid, tanh, and ReLU, and compute a numerically stable softmax.
Multilayer networks: a forward pass that computes XOR
Combine two layers of neurons with hand-chosen weights to compute XOR, which one neuron cannot.
Backpropagation: learning XOR by gradient descent
Compute gradients with the chain rule, verify them numerically, and train a small network on XOR.
Cross-entropy loss and the softmax gradient
See why classification networks use cross-entropy, and verify its simple gradient numerically.
Convolutional networks: output shapes and parameter counts
Compute how convolution and pooling change the size of an image, and why sharing weights saves parameters.
Regularization: weight decay and dropout
Shrink weights with an L2 penalty, and see why inverted dropout keeps the expected activation unchanged.
2
C
Learn memory, pointers, and efficient systems programming.
C: training a perceptron until it makes no mistakes
Use integer weights and the perceptron rule to learn the AND function.
C: softmax and why the largest score is subtracted
Convert scores to probabilities and show that the naive formula overflows for large scores.
3
C++
Build fast applications with modern types and classes.
C++: a perceptron learns OR but not XOR
Train the same neuron on two gates and compare the accuracy it can reach.
C++: stable softmax and cross-entropy loss
Turn raw scores into probabilities, measure the loss, and avoid overflow.
4
Java
Understand object oriented programs and the JVM.
Java: training a perceptron to learn AND
Update the weights of a single neuron with the perceptron rule until it classifies all four inputs correctly.
Java: a two-layer network that computes XOR
Combine two hidden neurons so that a network solves the problem a single perceptron cannot.
5
C#
Build typed applications on .NET.
C#: backpropagation learns XOR
Train a 2-2-1 sigmoid network by gradient descent until it computes XOR.
C#: activation functions and their ranges
Compare sigmoid, tanh, and ReLU on the same inputs.
6
JavaScript
Add behavior to the web and build applications.
JavaScript: a forward pass through two dense layers
Multiply by weight matrices, add biases, apply ReLU, and finish with softmax probabilities.
JavaScript: checking a gradient numerically
Compare a hand-derived gradient with a finite-difference estimate, the standard test for backpropagation code.
7
TypeScript
JavaScript with static types for dependable code.
TypeScript: composing neural-network layers behind an interface
Build a small network from Dense and Sigmoid layers that share one Layer interface.

About this subject

Python

The perceptron and linear separability

Activation functions and softmax

Multilayer networks: a forward pass that computes XOR

Backpropagation: learning XOR by gradient descent

Cross-entropy loss and the softmax gradient

Convolutional networks: output shapes and parameter counts

Regularization: weight decay and dropout

C

C: training a perceptron until it makes no mistakes

C: softmax and why the largest score is subtracted

C++

C++: a perceptron learns OR but not XOR

C++: stable softmax and cross-entropy loss

Java

Java: training a perceptron to learn AND

Java: a two-layer network that computes XOR

C#

C#: backpropagation learns XOR

C#: activation functions and their ranges

JavaScript

JavaScript: a forward pass through two dense layers

JavaScript: checking a gradient numerically

TypeScript

TypeScript: composing neural-network layers behind an interface