xixu-me/d2l-pytorch-notebooks
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit

Browse Source
This commit is contained in:
xixu-me committed 2024-08-20 16:25:10 +08:00
1 parent 1fb7b586eb
commit 4c20ec342f
549 files changed
+805499
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

chapter_multilayer-perceptrons/backprop.ipynb
+319
View File
@@ -0,0 +1,319 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "40d6b8e7",
"metadata": {
"origin_pos": 0
},
"source": [
"# Forward Propagation, Backward Propagation, and Computational Graphs\n",
":label:`sec_backprop`\n",
"\n",
"So far, we have trained our models\n",
"with minibatch stochastic gradient descent.\n",
"However, when we implemented the algorithm,\n",
"we only worried about the calculations involved\n",
"in *forward propagation* through the model.\n",
"When it came time to calculate the gradients,\n",
"we just invoked the backpropagation function provided by the deep learning framework.\n",
"\n",
"The automatic calculation of gradients\n",
"profoundly simplifies\n",
"the implementation of deep learning algorithms.\n",
"Before automatic differentiation,\n",
"even small changes to complicated models required\n",
"recalculating complicated derivatives by hand.\n",
"Surprisingly often, academic papers had to allocate\n",
"numerous pages to deriving update rules.\n",
"While we must continue to rely on automatic differentiation\n",
"so we can focus on the interesting parts,\n",
"you ought to know how these gradients\n",
"are calculated under the hood\n",
"if you want to go beyond a shallow\n",
"understanding of deep learning.\n",
"\n",
"In this section, we take a deep dive\n",
"into the details of *backward propagation*\n",
"(more commonly called *backpropagation*).\n",
"To convey some insight for both the\n",
"techniques and their implementations,\n",
"we rely on some basic mathematics and computational graphs.\n",
"To start, we focus our exposition on\n",
"a one-hidden-layer MLP\n",
"with weight decay ($\\ell_2$ regularization, to be described in subsequent chapters).\n",
"\n",
"## Forward Propagation\n",
"\n",
"*Forward propagation* (or *forward pass*) refers to the calculation and storage\n",
"of intermediate variables (including outputs)\n",
"for a neural network in order\n",
"from the input layer to the output layer.\n",
"We now work step-by-step through the mechanics\n",
"of a neural network with one hidden layer.\n",
"This may seem tedious but in the eternal words\n",
"of funk virtuoso James Brown,\n",
"you must \"pay the cost to be the boss\".\n",
"\n",
"\n",
"For the sake of simplicity, let's assume\n",
"that the input example is $\\mathbf{x}\\in \\mathbb{R}^d$\n",
"and that our hidden layer does not include a bias term.\n",
"Here the intermediate variable is:\n",
"\n",
"$$\\mathbf{z}= \\mathbf{W}^{(1)} \\mathbf{x},$$\n",
"\n",
"where $\\mathbf{W}^{(1)} \\in \\mathbb{R}^{h \\times d}$\n",
"is the weight parameter of the hidden layer.\n",
"After running the intermediate variable\n",
"$\\mathbf{z}\\in \\mathbb{R}^h$ through the\n",
"activation function $\\phi$\n",
"we obtain our hidden activation vector of length $h$:\n",
"\n",
"$$\\mathbf{h}= \\phi (\\mathbf{z}).$$\n",
"\n",
"The hidden layer output $\\mathbf{h}$\n",
"is also an intermediate variable.\n",
"Assuming that the parameters of the output layer\n",
"possess only a weight of\n",
"$\\mathbf{W}^{(2)} \\in \\mathbb{R}^{q \\times h}$,\n",
"we can obtain an output layer variable\n",
"with a vector of length $q$:\n",
"\n",
"$$\\mathbf{o}= \\mathbf{W}^{(2)} \\mathbf{h}.$$\n",
"\n",
"Assuming that the loss function is $l$\n",
"and the example label is $y$,\n",
"we can then calculate the loss term\n",
"for a single data example,\n",
"\n",
"$$L = l(\\mathbf{o}, y).$$\n",
"\n",
"As we will see the definition of $\\ell_2$ regularization\n",
"to be introduced later,\n",
"given the hyperparameter $\\lambda$,\n",
"the regularization term is\n",
"\n",
"$$s = \\frac{\\lambda}{2} \\left(\\|\\mathbf{W}^{(1)}\\|_\\textrm{F}^2 + \\|\\mathbf{W}^{(2)}\\|_\\textrm{F}^2\\right),$$\n",
":eqlabel:`eq_forward-s`\n",
"\n",
"where the Frobenius norm of the matrix\n",
"is simply the $\\ell_2$ norm applied\n",
"after flattening the matrix into a vector.\n",
"Finally, the model's regularized loss\n",
"on a given data example is:\n",
"\n",
"$$J = L + s.$$\n",
"\n",
"We refer to $J$ as the *objective function*\n",
"in the following discussion.\n",
"\n",
"\n",
"## Computational Graph of Forward Propagation\n",
"\n",
"Plotting *computational graphs* helps us visualize\n",
"the dependencies of operators\n",
"and variables within the calculation.\n",
":numref:`fig_forward` contains the graph associated\n",
"with the simple network described above,\n",
"where squares denote variables and circles denote operators.\n",
"The lower-left corner signifies the input\n",
"and the upper-right corner is the output.\n",
"Notice that the directions of the arrows\n",
"(which illustrate data flow)\n",
"are primarily rightward and upward.\n",
"\n",
"![Computational graph of forward propagation.](../img/forward.svg)\n",
":label:`fig_forward`\n",
"\n",
"## Backpropagation\n",
"\n",
"*Backpropagation* refers to the method of calculating\n",
"the gradient of neural network parameters.\n",
"In short, the method traverses the network in reverse order,\n",
"from the output to the input layer,\n",
"according to the *chain rule* from calculus.\n",
"The algorithm stores any intermediate variables\n",
"(partial derivatives)\n",
"required while calculating the gradient\n",
"with respect to some parameters.\n",
"Assume that we have functions\n",
"$\\mathsf{Y}=f(\\mathsf{X})$\n",
"and $\\mathsf{Z}=g(\\mathsf{Y})$,\n",
"in which the input and the output\n",
"$\\mathsf{X}, \\mathsf{Y}, \\mathsf{Z}$\n",
"are tensors of arbitrary shapes.\n",
"By using the chain rule,\n",
"we can compute the derivative\n",
"of $\\mathsf{Z}$ with respect to $\\mathsf{X}$ via\n",
"\n",
"$$\\frac{\\partial \\mathsf{Z}}{\\partial \\mathsf{X}} = \\textrm{prod}\\left(\\frac{\\partial \\mathsf{Z}}{\\partial \\mathsf{Y}}, \\frac{\\partial \\mathsf{Y}}{\\partial \\mathsf{X}}\\right).$$\n",
"\n",
"Here we use the $\\textrm{prod}$ operator\n",
"to multiply its arguments\n",
"after the necessary operations,\n",
"such as transposition and swapping input positions,\n",
"have been carried out.\n",
"For vectors, this is straightforward:\n",
"it is simply matrix--matrix multiplication.\n",
"For higher dimensional tensors,\n",
"we use the appropriate counterpart.\n",
"The operator $\\textrm{prod}$ hides all the notational overhead.\n",
"\n",
"Recall that\n",
"the parameters of the simple network with one hidden layer,\n",
"whose computational graph is in :numref:`fig_forward`,\n",
"are $\\mathbf{W}^{(1)}$ and $\\mathbf{W}^{(2)}$.\n",
"The objective of backpropagation is to\n",
"calculate the gradients $\\partial J/\\partial \\mathbf{W}^{(1)}$\n",
"and $\\partial J/\\partial \\mathbf{W}^{(2)}$.\n",
"To accomplish this, we apply the chain rule\n",
"and calculate, in turn, the gradient of\n",
"each intermediate variable and parameter.\n",
"The order of calculations are reversed\n",
"relative to those performed in forward propagation,\n",
"since we need to start with the outcome of the computational graph\n",
"and work our way towards the parameters.\n",
"The first step is to calculate the gradients\n",
"of the objective function $J=L+s$\n",
"with respect to the loss term $L$\n",
"and the regularization term $s$:\n",
"\n",
"$$\\frac{\\partial J}{\\partial L} = 1 \\; \\textrm{and} \\; \\frac{\\partial J}{\\partial s} = 1.$$\n",
"\n",
"Next, we compute the gradient of the objective function\n",
"with respect to variable of the output layer $\\mathbf{o}$\n",
"according to the chain rule:\n",
"\n",
"$$\n",
"\\frac{\\partial J}{\\partial \\mathbf{o}}\n",
"= \\textrm{prod}\\left(\\frac{\\partial J}{\\partial L}, \\frac{\\partial L}{\\partial \\mathbf{o}}\\right)\n",
"= \\frac{\\partial L}{\\partial \\mathbf{o}}\n",
"\\in \\mathbb{R}^q.\n",
"$$\n",
"\n",
"Next, we calculate the gradients\n",
"of the regularization term\n",
"with respect to both parameters:\n",
"\n",
"$$\\frac{\\partial s}{\\partial \\mathbf{W}^{(1)}} = \\lambda \\mathbf{W}^{(1)}\n",
"\\; \\textrm{and} \\;\n",
"\\frac{\\partial s}{\\partial \\mathbf{W}^{(2)}} = \\lambda \\mathbf{W}^{(2)}.$$\n",
"\n",
"Now we are able to calculate the gradient\n",
"$\\partial J/\\partial \\mathbf{W}^{(2)} \\in \\mathbb{R}^{q \\times h}$\n",
"of the model parameters closest to the output layer.\n",
"Using the chain rule yields:\n",
"\n",
"$$\\frac{\\partial J}{\\partial \\mathbf{W}^{(2)}}= \\textrm{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{o}}, \\frac{\\partial \\mathbf{o}}{\\partial \\mathbf{W}^{(2)}}\\right) + \\textrm{prod}\\left(\\frac{\\partial J}{\\partial s}, \\frac{\\partial s}{\\partial \\mathbf{W}^{(2)}}\\right)= \\frac{\\partial J}{\\partial \\mathbf{o}} \\mathbf{h}^\\top + \\lambda \\mathbf{W}^{(2)}.$$\n",
":eqlabel:`eq_backprop-J-h`\n",
"\n",
"To obtain the gradient with respect to $\\mathbf{W}^{(1)}$\n",
"we need to continue backpropagation\n",
"along the output layer to the hidden layer.\n",
"The gradient with respect to the hidden layer output\n",
"$\\partial J/\\partial \\mathbf{h} \\in \\mathbb{R}^h$ is given by\n",
"\n",
"\n",
"$$\n",
"\\frac{\\partial J}{\\partial \\mathbf{h}}\n",
"= \\textrm{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{o}}, \\frac{\\partial \\mathbf{o}}{\\partial \\mathbf{h}}\\right)\n",
"= {\\mathbf{W}^{(2)}}^\\top \\frac{\\partial J}{\\partial \\mathbf{o}}.\n",
"$$\n",
"\n",
"Since the activation function $\\phi$ applies elementwise,\n",
"calculating the gradient $\\partial J/\\partial \\mathbf{z} \\in \\mathbb{R}^h$\n",
"of the intermediate variable $\\mathbf{z}$\n",
"requires that we use the elementwise multiplication operator,\n",
"which we denote by $\\odot$:\n",
"\n",
"$$\n",
"\\frac{\\partial J}{\\partial \\mathbf{z}}\n",
"= \\textrm{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{h}}, \\frac{\\partial \\mathbf{h}}{\\partial \\mathbf{z}}\\right)\n",
"= \\frac{\\partial J}{\\partial \\mathbf{h}} \\odot \\phi'\\left(\\mathbf{z}\\right).\n",
"$$\n",
"\n",
"Finally, we can obtain the gradient\n",
"$\\partial J/\\partial \\mathbf{W}^{(1)} \\in \\mathbb{R}^{h \\times d}$\n",
"of the model parameters closest to the input layer.\n",
"According to the chain rule, we get\n",
"\n",
"$$\n",
"\\frac{\\partial J}{\\partial \\mathbf{W}^{(1)}}\n",
"= \\textrm{prod}\\left(\\frac{\\partial J}{\\partial \\mathbf{z}}, \\frac{\\partial \\mathbf{z}}{\\partial \\mathbf{W}^{(1)}}\\right) + \\textrm{prod}\\left(\\frac{\\partial J}{\\partial s}, \\frac{\\partial s}{\\partial \\mathbf{W}^{(1)}}\\right)\n",
"= \\frac{\\partial J}{\\partial \\mathbf{z}} \\mathbf{x}^\\top + \\lambda \\mathbf{W}^{(1)}.\n",
"$$\n",
"\n",
"\n",
"\n",
"## Training Neural Networks\n",
"\n",
"When training neural networks,\n",
"forward and backward propagation depend on each other.\n",
"In particular, for forward propagation,\n",
"we traverse the computational graph in the direction of dependencies\n",
"and compute all the variables on its path.\n",
"These are then used for backpropagation\n",
"where the compute order on the graph is reversed.\n",
"\n",
"Take the aforementioned simple network as an illustrative example.\n",
"On the one hand,\n",
"computing the regularization term :eqref:`eq_forward-s`\n",
"during forward propagation\n",
"depends on the current values of model parameters $\\mathbf{W}^{(1)}$ and $\\mathbf{W}^{(2)}$.\n",
"They are given by the optimization algorithm according to backpropagation in the most recent iteration.\n",
"On the other hand,\n",
"the gradient calculation for the parameter\n",
":eqref:`eq_backprop-J-h` during backpropagation\n",
"depends on the current value of the hidden layer output $\\mathbf{h}$,\n",
"which is given by forward propagation.\n",
"\n",
"\n",
"Therefore when training neural networks, once model parameters are initialized,\n",
"we alternate forward propagation with backpropagation,\n",
"updating model parameters using gradients given by backpropagation.\n",
"Note that backpropagation reuses the stored intermediate values from forward propagation to avoid duplicate calculations.\n",
"One of the consequences is that we need to retain\n",
"the intermediate values until backpropagation is complete.\n",
"This is also one of the reasons why training\n",
"requires significantly more memory than plain prediction.\n",
"Besides, the size of such intermediate values is roughly\n",
"proportional to the number of network layers and the batch size.\n",
"Thus,\n",
"training deeper networks using larger batch sizes\n",
"more easily leads to *out-of-memory* errors.\n",
"\n",
"\n",
"## Summary\n",
"\n",
"Forward propagation sequentially calculates and stores intermediate variables within the computational graph defined by the neural network. It proceeds from the input to the output layer.\n",
"Backpropagation sequentially calculates and stores the gradients of intermediate variables and parameters within the neural network in the reversed order.\n",
"When training deep learning models, forward propagation and backpropagation are interdependent,\n",
"and training requires significantly more memory than prediction.\n",
"\n",
"\n",
"## Exercises\n",
"\n",
"1. Assume that the inputs $\\mathbf{X}$ to some scalar function $f$ are $n \\times m$ matrices. What is the dimensionality of the gradient of $f$ with respect to $\\mathbf{X}$?\n",
"1. Add a bias to the hidden layer of the model described in this section (you do not need to include bias in the regularization term).\n",
" 1. Draw the corresponding computational graph.\n",
" 1. Derive the forward and backward propagation equations.\n",
"1. Compute the memory footprint for training and prediction in the model described in this section.\n",
"1. Assume that you want to compute second derivatives. What happens to the computational graph? How long do you expect the calculation to take?\n",
"1. Assume that the computational graph is too large for your GPU.\n",
" 1. Can you partition it over more than one GPU?\n",
" 1. What are the advantages and disadvantages over training on a smaller minibatch?\n",
"\n",
"[Discussions](https://discuss.d2l.ai/t/102)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}
chapter_multilayer-perceptrons/dropout.ipynb
+4401
View File
File diff suppressed because it is too large. Load diff
chapter_multilayer-perceptrons/generalization-deep.ipynb
+393
View File
@@ -0,0 +1,393 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "00ea4628",
"metadata": {
"origin_pos": 0
},
"source": [
"# Generalization in Deep Learning\n",
"\n",
"\n",
"In :numref:`chap_regression` and :numref:`chap_classification`,\n",
"we tackled regression and classification problems\n",
"by fitting linear models to training data.\n",
"In both cases, we provided practical algorithms\n",
"for finding the parameters that maximized\n",
"the likelihood of the observed training labels.\n",
"And then, towards the end of each chapter,\n",
"we recalled that fitting the training data\n",
"was only an intermediate goal.\n",
"Our real quest all along was to discover *general patterns*\n",
"on the basis of which we can make accurate predictions\n",
"even on new examples drawn from the same underlying population.\n",
"Machine learning researchers are *consumers* of optimization algorithms.\n",
"Sometimes, we must even develop new optimization algorithms.\n",
"But at the end of the day, optimization is merely a means to an end.\n",
"At its core, machine learning is a statistical discipline\n",
"and we wish to optimize training loss only insofar\n",
"as some statistical principle (known or unknown)\n",
"leads the resulting models to generalize beyond the training set.\n",
"\n",
"\n",
"On the bright side, it turns out that deep neural networks\n",
"trained by stochastic gradient descent generalize remarkably well\n",
"across myriad prediction problems, spanning computer vision;\n",
"natural language processing; time series data; recommender systems;\n",
"electronic health records; protein folding;\n",
"value function approximation in video games\n",
"and board games; and numerous other domains.\n",
"On the downside, if you were looking\n",
"for a straightforward account\n",
"of either the optimization story\n",
"(why we can fit them to training data)\n",
"or the generalization story\n",
"(why the resulting models generalize to unseen examples),\n",
"then you might want to pour yourself a drink.\n",
"While our procedures for optimizing linear models\n",
"and the statistical properties of the solutions\n",
"are both described well by a comprehensive body of theory,\n",
"our understanding of deep learning\n",
"still resembles the wild west on both fronts.\n",
"\n",
"Both the theory and practice of deep learning\n",
"are rapidly evolving,\n",
"with theorists adopting new strategies\n",
"to explain what's going on,\n",
"even as practitioners continue\n",
"to innovate at a blistering pace,\n",
"building arsenals of heuristics for training deep networks\n",
"and a body of intuitions and folk knowledge\n",
"that provide guidance for deciding\n",
"which techniques to apply in which situations.\n",
"\n",
"The summary of the present moment is that the theory of deep learning\n",
"has produced promising lines of attack and scattered fascinating results,\n",
"but still appears far from a comprehensive account\n",
"of both (i) why we are able to optimize neural networks\n",
"and (ii) how models learned by gradient descent\n",
"manage to generalize so well, even on high-dimensional tasks.\n",
"However, in practice, (i) is seldom a problem\n",
"(we can always find parameters that will fit all of our training data)\n",
"and thus understanding generalization is far the bigger problem.\n",
"On the other hand, even absent the comfort of a coherent scientific theory,\n",
"practitioners have developed a large collection of techniques\n",
"that may help you to produce models that generalize well in practice.\n",
"While no pithy summary can possibly do justice\n",
"to the vast topic of generalization in deep learning,\n",
"and while the overall state of research is far from resolved,\n",
"we hope, in this section, to present a broad overview\n",
"of the state of research and practice.\n",
"\n",
"\n",
"## Revisiting Overfitting and Regularization\n",
"\n",
"According to the \"no free lunch\" theorem of :citet:`wolpert1995no`,\n",
"any learning algorithm generalizes better on data with certain distributions, and worse with other distributions.\n",
"Thus, given a finite training set,\n",
"a model relies on certain assumptions: \n",
"to achieve human-level performance\n",
"it may be useful to identify *inductive biases* \n",
"that reflect how humans think about the world.\n",
"Such inductive biases show preferences \n",
"for solutions with certain properties.\n",
"For example,\n",
"a deep MLP has an inductive bias\n",
"towards building up a complicated function by the composition of simpler functions.\n",
"\n",
"With machine learning models encoding inductive biases,\n",
"our approach to training them\n",
"typically consists of two phases: (i) fit the training data;\n",
"and (ii) estimate the *generalization error*\n",
"(the true error on the underlying population)\n",
"by evaluating the model on holdout data.\n",
"The difference between our fit on the training data\n",
"and our fit on the test data is called the *generalization gap* and when this is large,\n",
"we say that our models *overfit* to the training data.\n",
"In extreme cases of overfitting,\n",
"we might exactly fit the training data,\n",
"even when the test error remains significant.\n",
"And in the classical view,\n",
"the interpretation is that our models are too complex,\n",
"requiring that we either shrink the number of features,\n",
"the number of nonzero parameters learned,\n",
"or the size of the parameters as quantified.\n",
"Recall the plot of model complexity compared with loss\n",
"(:numref:`fig_capacity_vs_error`)\n",
"from :numref:`sec_generalization_basics`.\n",
"\n",
"\n",
"However deep learning complicates this picture in counterintuitive ways.\n",
"First, for classification problems,\n",
"our models are typically expressive enough\n",
"to perfectly fit every training example,\n",
"even in datasets consisting of millions\n",
":cite:`zhang2021understanding`.\n",
"In the classical picture, we might think\n",
"that this setting lies on the far right extreme\n",
"of the model complexity axis,\n",
"and that any improvements in generalization error\n",
"must come by way of regularization,\n",
"either by reducing the complexity of the model class,\n",
"or by applying a penalty, severely constraining\n",
"the set of values that our parameters might take.\n",
"But that is where things start to get weird.\n",
"\n",
"Strangely, for many deep learning tasks\n",
"(e.g., image recognition and text classification)\n",
"we are typically choosing among model architectures,\n",
"all of which can achieve arbitrarily low training loss\n",
"(and zero training error).\n",
"Because all models under consideration achieve zero training error,\n",
"*the only avenue for further gains is to reduce overfitting*.\n",
"Even stranger, it is often the case that\n",
"despite fitting the training data perfectly,\n",
"we can actually *reduce the generalization error*\n",
"further by making the model *even more expressive*,\n",
"e.g., adding layers, nodes, or training\n",
"for a larger number of epochs.\n",
"Stranger yet, the pattern relating the generalization gap\n",
"to the *complexity* of the model (as captured, for example, in the depth or width of the networks)\n",
"can be non-monotonic,\n",
"with greater complexity hurting at first\n",
"but subsequently helping in a so-called \"double-descent\" pattern\n",
":cite:`nakkiran2021deep`.\n",
"Thus the deep learning practitioner possesses a bag of tricks,\n",
"some of which seemingly restrict the model in some fashion\n",
"and others that seemingly make it even more expressive,\n",
"and all of which, in some sense, are applied to mitigate overfitting.\n",
"\n",
"Complicating things even further,\n",
"while the guarantees provided by classical learning theory\n",
"can be conservative even for classical models,\n",
"they appear powerless to explain why it is\n",
"that deep neural networks generalize in the first place.\n",
"Because deep neural networks are capable of fitting\n",
"arbitrary labels even for large datasets,\n",
"and despite the use of familiar methods such as $\\ell_2$ regularization,\n",
"traditional complexity-based generalization bounds,\n",
"e.g., those based on the VC dimension\n",
"or Rademacher complexity of a hypothesis class\n",
"cannot explain why neural networks generalize.\n",
"\n",
"## Inspiration from Nonparametrics\n",
"\n",
"Approaching deep learning for the first time,\n",
"it is tempting to think of them as parametric models.\n",
"After all, the models *do* have millions of parameters.\n",
"When we update the models, we update their parameters.\n",
"When we save the models, we write their parameters to disk.\n",
"However, mathematics and computer science are riddled\n",
"with counterintuitive changes of perspective,\n",
"and surprising isomorphisms between seemingly different problems.\n",
"While neural networks clearly *have* parameters,\n",
"in some ways it can be more fruitful\n",
"to think of them as behaving like nonparametric models.\n",
"So what precisely makes a model nonparametric?\n",
"While the name covers a diverse set of approaches,\n",
"one common theme is that nonparametric methods\n",
"tend to have a level of complexity that grows\n",
"as the amount of available data grows.\n",
"\n",
"Perhaps the simplest example of a nonparametric model\n",
"is the $k$-nearest neighbor algorithm (we will cover more nonparametric models later, for example in :numref:`sec_attention-pooling`).\n",
"Here, at training time,\n",
"the learner simply memorizes the dataset.\n",
"Then, at prediction time,\n",
"when confronted with a new point $\\mathbf{x}$,\n",
"the learner looks up the $k$ nearest neighbors\n",
"(the $k$ points $\\mathbf{x}_i'$ that minimize\n",
"some distance $d(\\mathbf{x}, \\mathbf{x}_i')$).\n",
"When $k=1$, this algorithm is called $1$-nearest neighbors,\n",
"and the algorithm will always achieve a training error of zero.\n",
"That however, does not mean that the algorithm will not generalize.\n",
"In fact, it turns out that under some mild conditions,\n",
"the 1-nearest neighbor algorithm is consistent\n",
"(eventually converging to the optimal predictor).\n",
"\n",
"\n",
"Note that $1$-nearest neighbor requires that we specify\n",
"some distance function $d$, or equivalently,\n",
"that we specify some vector-valued basis function $\\phi(\\mathbf{x})$\n",
"for featurizing our data.\n",
"For any choice of the distance metric,\n",
"we will achieve zero training error\n",
"and eventually reach an optimal predictor,\n",
"but different distance metrics $d$\n",
"encode different inductive biases\n",
"and with a finite amount of available data\n",
"will yield different predictors.\n",
"Different choices of the distance metric $d$\n",
"represent different assumptions about the underlying patterns\n",
"and the performance of the different predictors\n",
"will depend on how compatible the assumptions\n",
"are with the observed data.\n",
"\n",
"In a sense, because neural networks are over-parametrized,\n",
"possessing many more parameters than are needed to fit the training data,\n",
"they tend to *interpolate* the training data (fitting it perfectly)\n",
"and thus behave, in some ways, more like nonparametric models.\n",
"More recent theoretical research has established\n",
"deep connection between large neural networks\n",
"and nonparametric methods, notably kernel methods.\n",
"In particular, :citet:`Jacot.Grabriel.Hongler.2018`\n",
"demonstrated that in the limit, as multilayer perceptrons\n",
"with randomly initialized weights grow infinitely wide,\n",
"they become equivalent to (nonparametric) kernel methods\n",
"for a specific choice of the kernel function\n",
"(essentially, a distance function),\n",
"which they call the neural tangent kernel.\n",
"While current neural tangent kernel models may not fully explain\n",
"the behavior of modern deep networks,\n",
"their success as an analytical tool\n",
"underscores the usefulness of nonparametric modeling\n",
"for understanding the behavior of over-parametrized deep networks.\n",
"\n",
"\n",
"## Early Stopping\n",
"\n",
"While deep neural networks are capable of fitting arbitrary labels,\n",
"even when labels are assigned incorrectly or randomly\n",
":cite:`zhang2021understanding`,\n",
"this capability only emerges over many iterations of training.\n",
"A new line of work :cite:`Rolnick.Veit.Belongie.Shavit.2017`\n",
"has revealed that in the setting of label noise,\n",
"neural networks tend to fit cleanly labeled data first\n",
"and only subsequently to interpolate the mislabeled data.\n",
"Moreover, it has been established that this phenomenon\n",
"translates directly into a guarantee on generalization:\n",
"whenever a model has fitted the cleanly labeled data\n",
"but not randomly labeled examples included in the training set,\n",
"it has in fact generalized :cite:`Garg.Balakrishnan.Kolter.Lipton.2021`.\n",
"\n",
"Together these findings help to motivate *early stopping*,\n",
"a classic technique for regularizing deep neural networks.\n",
"Here, rather than directly constraining the values of the weights,\n",
"one constrains the number of epochs of training.\n",
"The most common way to determine the stopping criterion\n",
"is to monitor validation error throughout training\n",
"(typically by checking once after each epoch)\n",
"and to cut off training when the validation error\n",
"has not decreased by more than some small amount $\\epsilon$\n",
"for some number of epochs.\n",
"This is sometimes called a *patience criterion*.\n",
"As well as the potential to lead to better generalization\n",
"in the setting of noisy labels,\n",
"another benefit of early stopping is the time saved.\n",
"Once the patience criterion is met, one can terminate training.\n",
"For large models that might require days of training\n",
"simultaneously across eight or more GPUs,\n",
"well-tuned early stopping can save researchers days of time\n",
"and can save their employers many thousands of dollars.\n",
"\n",
"Notably, when there is no label noise and datasets are *realizable*\n",
"(the classes are truly separable, e.g., distinguishing cats from dogs),\n",
"early stopping tends not to lead to significant improvements in generalization.\n",
"On the other hand, when there is label noise,\n",
"or intrinsic variability in the label\n",
"(e.g., predicting mortality among patients),\n",
"early stopping is crucial.\n",
"Training models until they interpolate noisy data is typically a bad idea.\n",
"\n",
"\n",
"## Classical Regularization Methods for Deep Networks\n",
"\n",
"In :numref:`chap_regression`, we described\n",
"several classical regularization techniques\n",
"for constraining the complexity of our models.\n",
"In particular, :numref:`sec_weight_decay`\n",
"introduced a method called weight decay,\n",
"which consists of adding a regularization term to the loss function\n",
"in order to penalize large values of the weights.\n",
"Depending on which weight norm is penalized\n",
"this technique is known either as ridge regularization (for $\\ell_2$ penalty)\n",
"or lasso regularization (for an $\\ell_1$ penalty).\n",
"In the classical analysis of these regularizers,\n",
"they are considered as sufficiently restrictive on the values\n",
"that the weights can take to prevent the model from fitting arbitrary labels.\n",
"\n",
"In deep learning implementations,\n",
"weight decay remains a popular tool.\n",
"However, researchers have noted\n",
"that typical strengths of $\\ell_2$ regularization\n",
"are insufficient to prevent the networks\n",
"from interpolating the data :cite:`zhang2021understanding` and thus the benefits if interpreted\n",
"as regularization might only make sense\n",
"in combination with the early stopping criterion.\n",
"Absent early stopping, it is possible\n",
"that just like the number of layers\n",
"or number of nodes (in deep learning)\n",
"or the distance metric (in 1-nearest neighbor),\n",
"these methods may lead to better generalization\n",
"not because they meaningfully constrain\n",
"the power of the neural network\n",
"but rather because they somehow encode inductive biases\n",
"that are better compatible with the patterns\n",
"found in datasets of interests.\n",
"Thus, classical regularizers remain popular\n",
"in deep learning implementations,\n",
"even if the theoretical rationale\n",
"for their efficacy may be radically different.\n",
"\n",
"Notably, deep learning researchers have also built\n",
"on techniques first popularized\n",
"in classical regularization contexts,\n",
"such as adding noise to model inputs.\n",
"In the next section we will introduce\n",
"the famous dropout technique\n",
"(invented by :citet:`Srivastava.Hinton.Krizhevsky.ea.2014`),\n",
"which has become a mainstay of deep learning,\n",
"even as the theoretical basis for its efficacy\n",
"remains similarly mysterious.\n",
"\n",
"\n",
"## Summary\n",
"\n",
"Unlike classical linear models,\n",
"which tend to have fewer parameters than examples,\n",
"deep networks tend to be over-parametrized,\n",
"and for most tasks are capable\n",
"of perfectly fitting the training set.\n",
"This *interpolation regime* challenges\n",
"many hard fast-held intuitions.\n",
"Functionally, neural networks look like parametric models.\n",
"But thinking of them as nonparametric models\n",
"can sometimes be a more reliable source of intuition.\n",
"Because it is often the case that all deep networks under consideration\n",
"are capable of fitting all of the training labels,\n",
"nearly all gains must come by mitigating overfitting\n",
"(closing the *generalization gap*).\n",
"Paradoxically, the interventions\n",
"that reduce the generalization gap\n",
"sometimes appear to increase model complexity\n",
"and at other times appear to decrease complexity.\n",
"However, these methods seldom decrease complexity\n",
"sufficiently for classical theory\n",
"to explain the generalization of deep networks,\n",
"and *why certain choices lead to improved generalization*\n",
"remains for the most part a massive open question\n",
"despite the concerted efforts of many brilliant researchers.\n",
"\n",
"\n",
"## Exercises\n",
"\n",
"1. In what sense do traditional complexity-based measures fail to account for generalization of deep neural networks?\n",
"1. Why might *early stopping* be considered a regularization technique?\n",
"1. How do researchers typically determine the stopping criterion?\n",
"1. What important factor seems to differentiate cases when early stopping leads to big improvements in generalization?\n",
"1. Beyond generalization, describe another benefit of early stopping.\n",
"\n",
"[Discussions](https://discuss.d2l.ai/t/7473)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}
chapter_multilayer-perceptrons/index.ipynb
+56
View File
@@ -0,0 +1,56 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "34b05043",
"metadata": {
"origin_pos": 0
},
"source": [
"# Multilayer Perceptrons\n",
":label:`chap_perceptrons`\n",
"\n",
"In this chapter, we will introduce your first truly *deep* network.\n",
"The simplest deep networks are called *multilayer perceptrons*,\n",
"and they consist of multiple layers of neurons\n",
"each fully connected to those in the layer below\n",
"(from which they receive input)\n",
"and those above (which they, in turn, influence).\n",
"Although automatic differentiation\n",
"significantly simplifies the implementation of deep learning algorithms,\n",
"we will dive deep into how these gradients\n",
"are calculated in deep networks.\n",
"Then we will\n",
"be ready to\n",
"discuss issues relating to numerical stability and parameter initialization\n",
"that are key to successfully training deep networks.\n",
"When we train such high-capacity models we run the risk of overfitting. Thus, we will\n",
"revisit regularization and generalization\n",
"for deep networks.\n",
"Throughout, we aim\n",
"to give you a firm grasp not just of the concepts but also of the practice of using deep networks.\n",
"At the end of this chapter, we apply what we have introduced so far to a real case: house price\n",
"prediction. We punt matters relating to the computational performance, scalability, and efficiency\n",
"of our models to subsequent chapters.\n",
"\n",
":begin_tab:toc\n",
" - [mlp](mlp.ipynb)\n",
" - [mlp-implementation](mlp-implementation.ipynb)\n",
" - [backprop](backprop.ipynb)\n",
" - [numerical-stability-and-init](numerical-stability-and-init.ipynb)\n",
" - [generalization-deep](generalization-deep.ipynb)\n",
" - [dropout](dropout.ipynb)\n",
" - [kaggle-house-price](kaggle-house-price.ipynb)\n",
":end_tab:\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}
chapter_multilayer-perceptrons/kaggle-house-price.ipynb
+2322
View File
File diff suppressed because it is too large. Load diff
chapter_multilayer-perceptrons/mlp-implementation.ipynb
+4254
View File
File diff suppressed because it is too large. Load diff
chapter_multilayer-perceptrons/mlp.ipynb
+5542
View File
File diff suppressed because it is too large. Load diff
chapter_multilayer-perceptrons/numerical-stability-and-init.ipynb
+1439
View File
File diff suppressed because it is too large. Load diff
chapter_multilayer-perceptrons/submission.csv
+1460
View File
File diff suppressed because it is too large. Load diff