Table of Contents
Preface vii
1 Foundations 1
Functions 2
Derivatives 7
Nested Functions 9
The Chain Rule 11
A Slightly Longer Example 14
Functions with Multiple Inputs 17
Derivatives of Functions with Multiple Inputs 19
Functions with Multiple Vector Inputs 20
Creating New Features from Existing Features 21
Derivatives of Functions with Multiple Vector Inputs 23
Vector Functions and Their Derivatives: One Step Further 26
Computational Graph with Two 2D Matrix Inputs 29
The Fun Part: The Backward Pass 33
Conclusion 39
2 Fundamentals 41
Supervised Learning Overview 42
Supervised Learning Models 44
Linear Regression 45
Training the Model 50
Assessing Our Model: Training Set Versus Testing Set 54
Assessing Our Model: The Code 55
Neural Networks from Scratch 58
Training and Assessing Our First Neural Network 65
Conclusion 68
3 Deep Learning from Scratch 71
Deep Learning Definition: A First Pass 72
The Building Blocks of Neural Networks: Operations 73
The Building Blocks of Neural Networks: Layers 77
Building Blocks on Building Blocks 79
The NeuralNetwork Class, and Maybe Others 85
Deep Learning from Scratch 88
Trainer and Optimizer 91
Putting Everything Together 96
Conclusion and Next Steps 98
4 Extensions 99
Some Intuition About Neural Networks 100
The Softmax Cross Entropy Loss Function 102
Experiments 110
Momentum 113
Learning Rate Decay 116
Weight Initialization 118
Dropout 122
Conclusion 125
5 Convolutional Neural Networks 127
Neural Networks and Representation Learning 127
Convolutional Layers 132
Implementing the Multichannel Convolution Operation 138
Using This Operation to Train a CNN 153
Conclusion 156
6 Recurrent Neural Networks 159
The Key Limitation: Handling Branching 160
Automatic Differentiation 162
Motivation for Recurrent Neural Networks 167
Introduction to Recurrent Neural Networks 168
RNNs: The Code 175
Conclusion 193
7 PyTorch 195
PyTorch Tensors 195
Deep Learning with PyTorch 197
Convolutional Neural Networks in PyTorch 205
Postscript: Unsupervised Learning via Autoencoders 212
Conclusion 220
A Deep Dives 221
Index 231