Initial commit
This commit is contained in:
xixu-me committed
1 parent
1fb7b586eb
commit
4c20ec342f
549 files changed
+805499
No files matched your search
@@ -0,0 +1,453 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3727da18",
|
||||
"metadata": {
|
||||
"origin_pos": 0
|
||||
},
|
||||
"source": [
|
||||
"# Backpropagation Through Time\n",
|
||||
":label:`sec_bptt`\n",
|
||||
"\n",
|
||||
"If you completed the exercises in :numref:`sec_rnn-scratch`,\n",
|
||||
"you would have seen that gradient clipping is vital \n",
|
||||
"for preventing the occasional massive gradients\n",
|
||||
"from destabilizing training.\n",
|
||||
"We hinted that the exploding gradients\n",
|
||||
"stem from backpropagating across long sequences.\n",
|
||||
"Before introducing a slew of modern RNN architectures,\n",
|
||||
"let's take a closer look at how *backpropagation*\n",
|
||||
"works in sequence models in mathematical detail.\n",
|
||||
"Hopefully, this discussion will bring some precision \n",
|
||||
"to the notion of *vanishing* and *exploding* gradients.\n",
|
||||
"If you recall our discussion of forward and backward \n",
|
||||
"propagation through computational graphs\n",
|
||||
"when we introduced MLPs in :numref:`sec_backprop`,\n",
|
||||
"then forward propagation in RNNs\n",
|
||||
"should be relatively straightforward.\n",
|
||||
"Applying backpropagation in RNNs \n",
|
||||
"is called *backpropagation through time* :cite:`Werbos.1990`.\n",
|
||||
"This procedure requires us to expand (or unroll) \n",
|
||||
"the computational graph of an RNN\n",
|
||||
"one time step at a time.\n",
|
||||
"The unrolled RNN is essentially \n",
|
||||
"a feedforward neural network \n",
|
||||
"with the special property \n",
|
||||
"that the same parameters \n",
|
||||
"are repeated throughout the unrolled network,\n",
|
||||
"appearing at each time step.\n",
|
||||
"Then, just as in any feedforward neural network,\n",
|
||||
"we can apply the chain rule, \n",
|
||||
"backpropagating gradients through the unrolled net.\n",
|
||||
"The gradient with respect to each parameter\n",
|
||||
"must be summed across all places \n",
|
||||
"that the parameter occurs in the unrolled net.\n",
|
||||
"Handling such weight tying should be familiar \n",
|
||||
"from our chapters on convolutional neural networks.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Complications arise because sequences\n",
|
||||
"can be rather long.\n",
|
||||
"It is not unusual to work with text sequences\n",
|
||||
"consisting of over a thousand tokens. \n",
|
||||
"Note that this poses problems both from \n",
|
||||
"a computational (too much memory)\n",
|
||||
"and optimization (numerical instability)\n",
|
||||
"standpoint. \n",
|
||||
"Input from the first step passes through\n",
|
||||
"over 1000 matrix products before arriving at the output, \n",
|
||||
"and another 1000 matrix products \n",
|
||||
"are required to compute the gradient. \n",
|
||||
"We now analyze what can go wrong and \n",
|
||||
"how to address it in practice.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Analysis of Gradients in RNNs\n",
|
||||
":label:`subsec_bptt_analysis`\n",
|
||||
"\n",
|
||||
"We start with a simplified model of how an RNN works.\n",
|
||||
"This model ignores details about the specifics \n",
|
||||
"of the hidden state and how it is updated.\n",
|
||||
"The mathematical notation here\n",
|
||||
"does not explicitly distinguish\n",
|
||||
"scalars, vectors, and matrices.\n",
|
||||
"We are just trying to develop some intuition.\n",
|
||||
"In this simplified model,\n",
|
||||
"we denote $h_t$ as the hidden state,\n",
|
||||
"$x_t$ as input, and $o_t$ as output\n",
|
||||
"at time step $t$.\n",
|
||||
"Recall our discussions in\n",
|
||||
":numref:`subsec_rnn_w_hidden_states`\n",
|
||||
"that the input and the hidden state\n",
|
||||
"can be concatenated before being multiplied \n",
|
||||
"by one weight variable in the hidden layer.\n",
|
||||
"Thus, we use $w_\\textrm{h}$ and $w_\\textrm{o}$ to indicate the weights \n",
|
||||
"of the hidden layer and the output layer, respectively.\n",
|
||||
"As a result, the hidden states and outputs \n",
|
||||
"at each time step are\n",
|
||||
"\n",
|
||||
"$$\\begin{aligned}h_t &= f(x_t, h_{t-1}, w_\\textrm{h}),\\\\o_t &= g(h_t, w_\\textrm{o}),\\end{aligned}$$\n",
|
||||
":eqlabel:`eq_bptt_ht_ot`\n",
|
||||
"\n",
|
||||
"where $f$ and $g$ are transformations\n",
|
||||
"of the hidden layer and the output layer, respectively.\n",
|
||||
"Hence, we have a chain of values \n",
|
||||
"$\\{\\ldots, (x_{t-1}, h_{t-1}, o_{t-1}), (x_{t}, h_{t}, o_t), \\ldots\\}$ \n",
|
||||
"that depend on each other via recurrent computation.\n",
|
||||
"The forward propagation is fairly straightforward.\n",
|
||||
"All we need is to loop through the $(x_t, h_t, o_t)$ triples one time step at a time.\n",
|
||||
"The discrepancy between output $o_t$ and the desired target $y_t$ \n",
|
||||
"is then evaluated by an objective function \n",
|
||||
"across all the $T$ time steps as\n",
|
||||
"\n",
|
||||
"$$L(x_1, \\ldots, x_T, y_1, \\ldots, y_T, w_\\textrm{h}, w_\\textrm{o}) = \\frac{1}{T}\\sum_{t=1}^T l(y_t, o_t).$$\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For backpropagation, matters are a bit trickier, \n",
|
||||
"especially when we compute the gradients \n",
|
||||
"with regard to the parameters $w_\\textrm{h}$ of the objective function $L$. \n",
|
||||
"To be specific, by the chain rule,\n",
|
||||
"\n",
|
||||
"$$\\begin{aligned}\\frac{\\partial L}{\\partial w_\\textrm{h}} & = \\frac{1}{T}\\sum_{t=1}^T \\frac{\\partial l(y_t, o_t)}{\\partial w_\\textrm{h}} \\\\& = \\frac{1}{T}\\sum_{t=1}^T \\frac{\\partial l(y_t, o_t)}{\\partial o_t} \\frac{\\partial g(h_t, w_\\textrm{o})}{\\partial h_t} \\frac{\\partial h_t}{\\partial w_\\textrm{h}}.\\end{aligned}$$\n",
|
||||
":eqlabel:`eq_bptt_partial_L_wh`\n",
|
||||
"\n",
|
||||
"The first and the second factors of the\n",
|
||||
"product in :eqref:`eq_bptt_partial_L_wh`\n",
|
||||
"are easy to compute.\n",
|
||||
"The third factor $\\partial h_t/\\partial w_\\textrm{h}$ is where things get tricky, \n",
|
||||
"since we need to recurrently compute the effect of the parameter $w_\\textrm{h}$ on $h_t$.\n",
|
||||
"According to the recurrent computation\n",
|
||||
"in :eqref:`eq_bptt_ht_ot`,\n",
|
||||
"$h_t$ depends on both $h_{t-1}$ and $w_\\textrm{h}$,\n",
|
||||
"where computation of $h_{t-1}$\n",
|
||||
"also depends on $w_\\textrm{h}$.\n",
|
||||
"Thus, evaluating the total derivate of $h_t$ \n",
|
||||
"with respect to $w_\\textrm{h}$ using the chain rule yields\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial h_t}{\\partial w_\\textrm{h}}= \\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial w_\\textrm{h}} +\\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial h_{t-1}} \\frac{\\partial h_{t-1}}{\\partial w_\\textrm{h}}.$$\n",
|
||||
":eqlabel:`eq_bptt_partial_ht_wh_recur`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"To derive the above gradient, assume that we have \n",
|
||||
"three sequences $\\{a_{t}\\},\\{b_{t}\\},\\{c_{t}\\}$ \n",
|
||||
"satisfying $a_{0}=0$ and $a_{t}=b_{t}+c_{t}a_{t-1}$ for $t=1, 2,\\ldots$.\n",
|
||||
"Then for $t\\geq 1$, it is easy to show\n",
|
||||
"\n",
|
||||
"$$a_{t}=b_{t}+\\sum_{i=1}^{t-1}\\left(\\prod_{j=i+1}^{t}c_{j}\\right)b_{i}.$$\n",
|
||||
":eqlabel:`eq_bptt_at`\n",
|
||||
"\n",
|
||||
"By substituting $a_t$, $b_t$, and $c_t$ according to\n",
|
||||
"\n",
|
||||
"$$\\begin{aligned}a_t &= \\frac{\\partial h_t}{\\partial w_\\textrm{h}},\\\\\n",
|
||||
"b_t &= \\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial w_\\textrm{h}}, \\\\\n",
|
||||
"c_t &= \\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial h_{t-1}},\\end{aligned}$$\n",
|
||||
"\n",
|
||||
"the gradient computation in :eqref:`eq_bptt_partial_ht_wh_recur` satisfies\n",
|
||||
"$a_{t}=b_{t}+c_{t}a_{t-1}$.\n",
|
||||
"Thus, per :eqref:`eq_bptt_at`, \n",
|
||||
"we can remove the recurrent computation \n",
|
||||
"in :eqref:`eq_bptt_partial_ht_wh_recur` with\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial h_t}{\\partial w_\\textrm{h}}=\\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial w_\\textrm{h}}+\\sum_{i=1}^{t-1}\\left(\\prod_{j=i+1}^{t} \\frac{\\partial f(x_{j},h_{j-1},w_\\textrm{h})}{\\partial h_{j-1}} \\right) \\frac{\\partial f(x_{i},h_{i-1},w_\\textrm{h})}{\\partial w_\\textrm{h}}.$$\n",
|
||||
":eqlabel:`eq_bptt_partial_ht_wh_gen`\n",
|
||||
"\n",
|
||||
"While we can use the chain rule to compute $\\partial h_t/\\partial w_\\textrm{h}$ recursively, \n",
|
||||
"this chain can get very long whenever $t$ is large.\n",
|
||||
"Let's discuss a number of strategies for dealing with this problem.\n",
|
||||
"\n",
|
||||
"### Full Computation ### \n",
|
||||
"\n",
|
||||
"One idea might be to compute the full sum in :eqref:`eq_bptt_partial_ht_wh_gen`.\n",
|
||||
"However, this is very slow and gradients can blow up,\n",
|
||||
"since subtle changes in the initial conditions\n",
|
||||
"can potentially affect the outcome a lot.\n",
|
||||
"That is, we could see things similar to the butterfly effect,\n",
|
||||
"where minimal changes in the initial conditions \n",
|
||||
"lead to disproportionate changes in the outcome.\n",
|
||||
"This is generally undesirable.\n",
|
||||
"After all, we are looking for robust estimators that generalize well. \n",
|
||||
"Hence this strategy is almost never used in practice.\n",
|
||||
"\n",
|
||||
"### Truncating Time Steps###\n",
|
||||
"\n",
|
||||
"Alternatively,\n",
|
||||
"we can truncate the sum in\n",
|
||||
":eqref:`eq_bptt_partial_ht_wh_gen`\n",
|
||||
"after $\\tau$ steps. \n",
|
||||
"This is what we have been discussing so far. \n",
|
||||
"This leads to an *approximation* of the true gradient,\n",
|
||||
"simply by terminating the sum at $\\partial h_{t-\\tau}/\\partial w_\\textrm{h}$. \n",
|
||||
"In practice this works quite well. \n",
|
||||
"It is what is commonly referred to as truncated \n",
|
||||
"backpropgation through time :cite:`Jaeger.2002`.\n",
|
||||
"One of the consequences of this is that the model \n",
|
||||
"focuses primarily on short-term influence \n",
|
||||
"rather than long-term consequences. \n",
|
||||
"This is actually *desirable*, since it biases the estimate \n",
|
||||
"towards simpler and more stable models.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Randomized Truncation ### \n",
|
||||
"\n",
|
||||
"Last, we can replace $\\partial h_t/\\partial w_\\textrm{h}$\n",
|
||||
"by a random variable which is correct in expectation \n",
|
||||
"but truncates the sequence.\n",
|
||||
"This is achieved by using a sequence of $\\xi_t$\n",
|
||||
"with predefined $0 \\leq \\pi_t \\leq 1$,\n",
|
||||
"where $P(\\xi_t = 0) = 1-\\pi_t$ and \n",
|
||||
"$P(\\xi_t = \\pi_t^{-1}) = \\pi_t$, thus $E[\\xi_t] = 1$.\n",
|
||||
"We use this to replace the gradient\n",
|
||||
"$\\partial h_t/\\partial w_\\textrm{h}$\n",
|
||||
"in :eqref:`eq_bptt_partial_ht_wh_recur`\n",
|
||||
"with\n",
|
||||
"\n",
|
||||
"$$z_t= \\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial w_\\textrm{h}} +\\xi_t \\frac{\\partial f(x_{t},h_{t-1},w_\\textrm{h})}{\\partial h_{t-1}} \\frac{\\partial h_{t-1}}{\\partial w_\\textrm{h}}.$$\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"It follows from the definition of $\\xi_t$ \n",
|
||||
"that $E[z_t] = \\partial h_t/\\partial w_\\textrm{h}$.\n",
|
||||
"Whenever $\\xi_t = 0$ the recurrent computation\n",
|
||||
"terminates at that time step $t$.\n",
|
||||
"This leads to a weighted sum of sequences of varying lengths,\n",
|
||||
"where long sequences are rare but appropriately overweighted. \n",
|
||||
"This idea was proposed by \n",
|
||||
":citet:`Tallec.Ollivier.2017`.\n",
|
||||
"\n",
|
||||
"### Comparing Strategies\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":label:`fig_truncated_bptt`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":numref:`fig_truncated_bptt` illustrates the three strategies \n",
|
||||
"when analyzing the first few characters of *The Time Machine* \n",
|
||||
"using backpropagation through time for RNNs:\n",
|
||||
"\n",
|
||||
"* The first row is the randomized truncation that partitions the text into segments of varying lengths.\n",
|
||||
"* The second row is the regular truncation that breaks the text into subsequences of the same length. This is what we have been doing in RNN experiments.\n",
|
||||
"* The third row is the full backpropagation through time that leads to a computationally infeasible expression.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Unfortunately, while appealing in theory, \n",
|
||||
"randomized truncation does not work \n",
|
||||
"much better than regular truncation, \n",
|
||||
"most likely due to a number of factors.\n",
|
||||
"First, the effect of an observation\n",
|
||||
"after a number of backpropagation steps \n",
|
||||
"into the past is quite sufficient \n",
|
||||
"to capture dependencies in practice. \n",
|
||||
"Second, the increased variance counteracts the fact \n",
|
||||
"that the gradient is more accurate with more steps. \n",
|
||||
"Third, we actually *want* models that have only \n",
|
||||
"a short range of interactions. \n",
|
||||
"Hence, regularly truncated backpropagation through time \n",
|
||||
"has a slight regularizing effect that can be desirable.\n",
|
||||
"\n",
|
||||
"## Backpropagation Through Time in Detail\n",
|
||||
"\n",
|
||||
"After discussing the general principle,\n",
|
||||
"let's discuss backpropagation through time in detail.\n",
|
||||
"In contrast to the analysis in :numref:`subsec_bptt_analysis`,\n",
|
||||
"in the following we will show how to compute\n",
|
||||
"the gradients of the objective function\n",
|
||||
"with respect to all the decomposed model parameters.\n",
|
||||
"To keep things simple, we consider \n",
|
||||
"an RNN without bias parameters,\n",
|
||||
"whose activation function in the hidden layer\n",
|
||||
"uses the identity mapping ($\\phi(x)=x$).\n",
|
||||
"For time step $t$, let the single example input \n",
|
||||
"and the target be $\\mathbf{x}_t \\in \\mathbb{R}^d$ and $y_t$, respectively. \n",
|
||||
"The hidden state $\\mathbf{h}_t \\in \\mathbb{R}^h$ \n",
|
||||
"and the output $\\mathbf{o}_t \\in \\mathbb{R}^q$\n",
|
||||
"are computed as\n",
|
||||
"\n",
|
||||
"$$\\begin{aligned}\\mathbf{h}_t &= \\mathbf{W}_\\textrm{hx} \\mathbf{x}_t + \\mathbf{W}_\\textrm{hh} \\mathbf{h}_{t-1},\\\\\n",
|
||||
"\\mathbf{o}_t &= \\mathbf{W}_\\textrm{qh} \\mathbf{h}_{t},\\end{aligned}$$\n",
|
||||
"\n",
|
||||
"where $\\mathbf{W}_\\textrm{hx} \\in \\mathbb{R}^{h \\times d}$, $\\mathbf{W}_\\textrm{hh} \\in \\mathbb{R}^{h \\times h}$, and\n",
|
||||
"$\\mathbf{W}_\\textrm{qh} \\in \\mathbb{R}^{q \\times h}$\n",
|
||||
"are the weight parameters.\n",
|
||||
"Denote by $l(\\mathbf{o}_t, y_t)$\n",
|
||||
"the loss at time step $t$. \n",
|
||||
"Our objective function,\n",
|
||||
"the loss over $T$ time steps\n",
|
||||
"from the beginning of the sequence is thus\n",
|
||||
"\n",
|
||||
"$$L = \\frac{1}{T} \\sum_{t=1}^T l(\\mathbf{o}_t, y_t).$$\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"In order to visualize the dependencies among\n",
|
||||
"model variables and parameters during computation\n",
|
||||
"of the RNN,\n",
|
||||
"we can draw a computational graph for the model,\n",
|
||||
"as shown in :numref:`fig_rnn_bptt`.\n",
|
||||
"For example, the computation of the hidden states of time step 3,\n",
|
||||
"$\\mathbf{h}_3$, depends on the model parameters\n",
|
||||
"$\\mathbf{W}_\\textrm{hx}$ and $\\mathbf{W}_\\textrm{hh}$,\n",
|
||||
"the hidden state of the previous time step $\\mathbf{h}_2$,\n",
|
||||
"and the input of the current time step $\\mathbf{x}_3$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":label:`fig_rnn_bptt`\n",
|
||||
"\n",
|
||||
"As just mentioned, the model parameters in :numref:`fig_rnn_bptt` \n",
|
||||
"are $\\mathbf{W}_\\textrm{hx}$, $\\mathbf{W}_\\textrm{hh}$, and $\\mathbf{W}_\\textrm{qh}$. \n",
|
||||
"Generally, training this model requires \n",
|
||||
"gradient computation with respect to these parameters\n",
|
||||
"$\\partial L/\\partial \\mathbf{W}_\\textrm{hx}$, $\\partial L/\\partial \\mathbf{W}_\\textrm{hh}$, and $\\partial L/\\partial \\mathbf{W}_\\textrm{qh}$.\n",
|
||||
"According to the dependencies in :numref:`fig_rnn_bptt`,\n",
|
||||
"we can traverse in the opposite direction of the arrows\n",
|
||||
"to calculate and store the gradients in turn.\n",
|
||||
"To flexibly express the multiplication of \n",
|
||||
"matrices, vectors, and scalars of different shapes\n",
|
||||
"in the chain rule,\n",
|
||||
"we continue to use the $\\textrm{prod}$ operator \n",
|
||||
"as described in :numref:`sec_backprop`.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"First of all, differentiating the objective function\n",
|
||||
"with respect to the model output at any time step $t$\n",
|
||||
"is fairly straightforward:\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial L}{\\partial \\mathbf{o}_t} = \\frac{\\partial l (\\mathbf{o}_t, y_t)}{T \\cdot \\partial \\mathbf{o}_t} \\in \\mathbb{R}^q.$$\n",
|
||||
":eqlabel:`eq_bptt_partial_L_ot`\n",
|
||||
"\n",
|
||||
"Now we can calculate the gradient of the objective \n",
|
||||
"with respect to the parameter $\\mathbf{W}_\\textrm{qh}$\n",
|
||||
"in the output layer:\n",
|
||||
"$\\partial L/\\partial \\mathbf{W}_\\textrm{qh} \\in \\mathbb{R}^{q \\times h}$. \n",
|
||||
"Based on :numref:`fig_rnn_bptt`, \n",
|
||||
"the objective $L$ depends on $\\mathbf{W}_\\textrm{qh}$ \n",
|
||||
"via $\\mathbf{o}_1, \\ldots, \\mathbf{o}_T$. \n",
|
||||
"Using the chain rule yields\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\frac{\\partial L}{\\partial \\mathbf{W}_\\textrm{qh}}\n",
|
||||
"= \\sum_{t=1}^T \\textrm{prod}\\left(\\frac{\\partial L}{\\partial \\mathbf{o}_t}, \\frac{\\partial \\mathbf{o}_t}{\\partial \\mathbf{W}_\\textrm{qh}}\\right)\n",
|
||||
"= \\sum_{t=1}^T \\frac{\\partial L}{\\partial \\mathbf{o}_t} \\mathbf{h}_t^\\top,\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"where $\\partial L/\\partial \\mathbf{o}_t$\n",
|
||||
"is given by :eqref:`eq_bptt_partial_L_ot`.\n",
|
||||
"\n",
|
||||
"Next, as shown in :numref:`fig_rnn_bptt`,\n",
|
||||
"at the final time step $T$,\n",
|
||||
"the objective function\n",
|
||||
"$L$ depends on the hidden state $\\mathbf{h}_T$ \n",
|
||||
"only via $\\mathbf{o}_T$.\n",
|
||||
"Therefore, we can easily find the gradient \n",
|
||||
"$\\partial L/\\partial \\mathbf{h}_T \\in \\mathbb{R}^h$\n",
|
||||
"using the chain rule:\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial L}{\\partial \\mathbf{h}_T} = \\textrm{prod}\\left(\\frac{\\partial L}{\\partial \\mathbf{o}_T}, \\frac{\\partial \\mathbf{o}_T}{\\partial \\mathbf{h}_T} \\right) = \\mathbf{W}_\\textrm{qh}^\\top \\frac{\\partial L}{\\partial \\mathbf{o}_T}.$$\n",
|
||||
":eqlabel:`eq_bptt_partial_L_hT_final_step`\n",
|
||||
"\n",
|
||||
"It gets trickier for any time step $t < T$,\n",
|
||||
"where the objective function $L$ depends on \n",
|
||||
"$\\mathbf{h}_t$ via $\\mathbf{h}_{t+1}$ and $\\mathbf{o}_t$.\n",
|
||||
"According to the chain rule,\n",
|
||||
"the gradient of the hidden state\n",
|
||||
"$\\partial L/\\partial \\mathbf{h}_t \\in \\mathbb{R}^h$\n",
|
||||
"at any time step $t < T$ can be recurrently computed as:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial L}{\\partial \\mathbf{h}_t} = \\textrm{prod}\\left(\\frac{\\partial L}{\\partial \\mathbf{h}_{t+1}}, \\frac{\\partial \\mathbf{h}_{t+1}}{\\partial \\mathbf{h}_t} \\right) + \\textrm{prod}\\left(\\frac{\\partial L}{\\partial \\mathbf{o}_t}, \\frac{\\partial \\mathbf{o}_t}{\\partial \\mathbf{h}_t} \\right) = \\mathbf{W}_\\textrm{hh}^\\top \\frac{\\partial L}{\\partial \\mathbf{h}_{t+1}} + \\mathbf{W}_\\textrm{qh}^\\top \\frac{\\partial L}{\\partial \\mathbf{o}_t}.$$\n",
|
||||
":eqlabel:`eq_bptt_partial_L_ht_recur`\n",
|
||||
"\n",
|
||||
"For analysis, expanding the recurrent computation\n",
|
||||
"for any time step $1 \\leq t \\leq T$ gives\n",
|
||||
"\n",
|
||||
"$$\\frac{\\partial L}{\\partial \\mathbf{h}_t}= \\sum_{i=t}^T {\\left(\\mathbf{W}_\\textrm{hh}^\\top\\right)}^{T-i} \\mathbf{W}_\\textrm{qh}^\\top \\frac{\\partial L}{\\partial \\mathbf{o}_{T+t-i}}.$$\n",
|
||||
":eqlabel:`eq_bptt_partial_L_ht`\n",
|
||||
"\n",
|
||||
"We can see from :eqref:`eq_bptt_partial_L_ht` \n",
|
||||
"that this simple linear example already\n",
|
||||
"exhibits some key problems of long sequence models:\n",
|
||||
"it involves potentially very large powers of $\\mathbf{W}_\\textrm{hh}^\\top$.\n",
|
||||
"In it, eigenvalues smaller than 1 vanish\n",
|
||||
"and eigenvalues larger than 1 diverge.\n",
|
||||
"This is numerically unstable,\n",
|
||||
"which manifests itself in the form of vanishing \n",
|
||||
"and exploding gradients.\n",
|
||||
"One way to address this is to truncate the time steps\n",
|
||||
"at a computationally convenient size \n",
|
||||
"as discussed in :numref:`subsec_bptt_analysis`. \n",
|
||||
"In practice, this truncation can also be effected \n",
|
||||
"by detaching the gradient after a given number of time steps.\n",
|
||||
"Later on, we will see how more sophisticated sequence models \n",
|
||||
"such as long short-term memory can alleviate this further. \n",
|
||||
"\n",
|
||||
"Finally, :numref:`fig_rnn_bptt` shows \n",
|
||||
"that the objective function $L$ \n",
|
||||
"depends on model parameters $\\mathbf{W}_\\textrm{hx}$ and $\\mathbf{W}_\\textrm{hh}$\n",
|
||||
"in the hidden layer via hidden states\n",
|
||||
"$\\mathbf{h}_1, \\ldots, \\mathbf{h}_T$.\n",
|
||||
"To compute gradients with respect to such parameters\n",
|
||||
"$\\partial L / \\partial \\mathbf{W}_\\textrm{hx} \\in \\mathbb{R}^{h \\times d}$ and $\\partial L / \\partial \\mathbf{W}_\\textrm{hh} \\in \\mathbb{R}^{h \\times h}$,\n",
|
||||
"we apply the chain rule giving\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"\\frac{\\partial L}{\\partial \\mathbf{W}_\\textrm{hx}}\n",
|
||||
"&= \\sum_{t=1}^T \\textrm{prod}\\left(\\frac{\\partial L}{\\partial \\mathbf{h}_t}, \\frac{\\partial \\mathbf{h}_t}{\\partial \\mathbf{W}_\\textrm{hx}}\\right)\n",
|
||||
"= \\sum_{t=1}^T \\frac{\\partial L}{\\partial \\mathbf{h}_t} \\mathbf{x}_t^\\top,\\\\\n",
|
||||
"\\frac{\\partial L}{\\partial \\mathbf{W}_\\textrm{hh}}\n",
|
||||
"&= \\sum_{t=1}^T \\textrm{prod}\\left(\\frac{\\partial L}{\\partial \\mathbf{h}_t}, \\frac{\\partial \\mathbf{h}_t}{\\partial \\mathbf{W}_\\textrm{hh}}\\right)\n",
|
||||
"= \\sum_{t=1}^T \\frac{\\partial L}{\\partial \\mathbf{h}_t} \\mathbf{h}_{t-1}^\\top,\n",
|
||||
"\\end{aligned}\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"where $\\partial L/\\partial \\mathbf{h}_t$\n",
|
||||
"which is recurrently computed by\n",
|
||||
":eqref:`eq_bptt_partial_L_hT_final_step`\n",
|
||||
"and :eqref:`eq_bptt_partial_L_ht_recur`\n",
|
||||
"is the key quantity that affects the numerical stability.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Since backpropagation through time is the application of backpropagation in RNNs,\n",
|
||||
"as we have explained in :numref:`sec_backprop`,\n",
|
||||
"training RNNs alternates forward propagation with\n",
|
||||
"backpropagation through time.\n",
|
||||
"Moreover, backpropagation through time\n",
|
||||
"computes and stores the above gradients in turn.\n",
|
||||
"Specifically, stored intermediate values\n",
|
||||
"are reused to avoid duplicate calculations,\n",
|
||||
"such as storing $\\partial L/\\partial \\mathbf{h}_t$\n",
|
||||
"to be used in computation of both $\\partial L / \\partial \\mathbf{W}_\\textrm{hx}$ \n",
|
||||
"and $\\partial L / \\partial \\mathbf{W}_\\textrm{hh}$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"Backpropagation through time is merely an application of backpropagation to sequence models with a hidden state.\n",
|
||||
"Truncation, such as regular or randomized, is needed for computational convenience and numerical stability.\n",
|
||||
"High powers of matrices can lead to divergent or vanishing eigenvalues. This manifests itself in the form of exploding or vanishing gradients.\n",
|
||||
"For efficient computation, intermediate values are cached during backpropagation through time.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Exercises\n",
|
||||
"\n",
|
||||
"1. Assume that we have a symmetric matrix $\\mathbf{M} \\in \\mathbb{R}^{n \\times n}$ with eigenvalues $\\lambda_i$ whose corresponding eigenvectors are $\\mathbf{v}_i$ ($i = 1, \\ldots, n$). Without loss of generality, assume that they are ordered in the order $|\\lambda_i| \\geq |\\lambda_{i+1}|$. \n",
|
||||
" 1. Show that $\\mathbf{M}^k$ has eigenvalues $\\lambda_i^k$.\n",
|
||||
" 1. Prove that for a random vector $\\mathbf{x} \\in \\mathbb{R}^n$, with high probability $\\mathbf{M}^k \\mathbf{x}$ will be very much aligned with the eigenvector $\\mathbf{v}_1$ \n",
|
||||
"of $\\mathbf{M}$. Formalize this statement.\n",
|
||||
" 1. What does the above result mean for gradients in RNNs?\n",
|
||||
"1. Besides gradient clipping, can you think of any other methods to cope with gradient explosion in recurrent neural networks?\n",
|
||||
"\n",
|
||||
"[Discussions](https://discuss.d2l.ai/t/334)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"required_libs": []
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "26d2f7a4",
|
||||
"metadata": {
|
||||
"origin_pos": 0
|
||||
},
|
||||
"source": [
|
||||
"# Recurrent Neural Networks\n",
|
||||
":label:`chap_rnn`\n",
|
||||
"\n",
|
||||
"Up until now, we have focused primarily on fixed-length data.\n",
|
||||
"When introducing linear and logistic regression\n",
|
||||
"in :numref:`chap_regression` and :numref:`chap_classification`\n",
|
||||
"and multilayer perceptrons in :numref:`chap_perceptrons`,\n",
|
||||
"we were happy to assume that each feature vector $\\mathbf{x}_i$\n",
|
||||
"consisted of a fixed number of components $x_1, \\dots, x_d$,\n",
|
||||
"where each numerical feature $x_j$\n",
|
||||
"corresponded to a particular attribute.\n",
|
||||
"These datasets are sometimes called *tabular*,\n",
|
||||
"because they can be arranged in tables,\n",
|
||||
"where each example $i$ gets its own row,\n",
|
||||
"and each attribute gets its own column.\n",
|
||||
"Crucially, with tabular data, we seldom\n",
|
||||
"assume any particular structure over the columns.\n",
|
||||
"\n",
|
||||
"Subsequently, in :numref:`chap_cnn`,\n",
|
||||
"we moved on to image data, where inputs consist\n",
|
||||
"of the raw pixel values at each coordinate in an image.\n",
|
||||
"Image data hardly fitted the bill\n",
|
||||
"of a protypical tabular dataset.\n",
|
||||
"There, we needed to call upon convolutional neural networks (CNNs)\n",
|
||||
"to handle the hierarchical structure and invariances.\n",
|
||||
"However, our data were still of fixed length.\n",
|
||||
"Every Fashion-MNIST image is represented\n",
|
||||
"as a $28 \\times 28$ grid of pixel values.\n",
|
||||
"Moreover, our goal was to develop a model\n",
|
||||
"that looked at just one image and then\n",
|
||||
"outputted a single prediction.\n",
|
||||
"But what should we do when faced with a\n",
|
||||
"sequence of images, as in a video,\n",
|
||||
"or when tasked with producing\n",
|
||||
"a sequentially structured prediction,\n",
|
||||
"as in the case of image captioning?\n",
|
||||
"\n",
|
||||
"A great many learning tasks require dealing with sequential data.\n",
|
||||
"Image captioning, speech synthesis, and music generation\n",
|
||||
"all require that models produce outputs consisting of sequences.\n",
|
||||
"In other domains, such as time series prediction,\n",
|
||||
"video analysis, and musical information retrieval,\n",
|
||||
"a model must learn from inputs that are sequences.\n",
|
||||
"These demands often arise simultaneously:\n",
|
||||
"tasks such as translating passages of text\n",
|
||||
"from one natural language to another,\n",
|
||||
"engaging in dialogue, or controlling a robot,\n",
|
||||
"demand that models both ingest and output\n",
|
||||
"sequentially structured data.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Recurrent neural networks (RNNs) are deep learning models\n",
|
||||
"that capture the dynamics of sequences via\n",
|
||||
"*recurrent* connections, which can be thought of\n",
|
||||
"as cycles in the network of nodes.\n",
|
||||
"This might seem counterintuitive at first.\n",
|
||||
"After all, it is the feedforward nature of neural networks\n",
|
||||
"that makes the order of computation unambiguous.\n",
|
||||
"However, recurrent edges are defined in a precise way\n",
|
||||
"that ensures that no such ambiguity can arise.\n",
|
||||
"Recurrent neural networks are *unrolled* across time steps (or sequence steps),\n",
|
||||
"with the *same* underlying parameters applied at each step.\n",
|
||||
"While the standard connections are applied *synchronously*\n",
|
||||
"to propagate each layer's activations\n",
|
||||
"to the subsequent layer *at the same time step*,\n",
|
||||
"the recurrent connections are *dynamic*,\n",
|
||||
"passing information across adjacent time steps.\n",
|
||||
"As the unfolded view in :numref:`fig_unfolded-rnn` reveals,\n",
|
||||
"RNNs can be thought of as feedforward neural networks\n",
|
||||
"where each layer's parameters (both conventional and recurrent)\n",
|
||||
"are shared across time steps.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":label:`fig_unfolded-rnn`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Like neural networks more broadly,\n",
|
||||
"RNNs have a long discipline-spanning history,\n",
|
||||
"originating as models of the brain popularized\n",
|
||||
"by cognitive scientists and subsequently adopted\n",
|
||||
"as practical modeling tools employed\n",
|
||||
"by the machine learning community.\n",
|
||||
"As we do for deep learning more broadly,\n",
|
||||
"in this book we adopt the machine learning perspective,\n",
|
||||
"focusing on RNNs as practical tools that rose\n",
|
||||
"to popularity in the 2010s owing to\n",
|
||||
"breakthrough results on such diverse tasks\n",
|
||||
"as handwriting recognition :cite:`graves2008novel`,\n",
|
||||
"machine translation :cite:`Sutskever.Vinyals.Le.2014`,\n",
|
||||
"and recognizing medical diagnoses :cite:`Lipton.Kale.2016`.\n",
|
||||
"We point the reader interested in more\n",
|
||||
"background material to a publicly available\n",
|
||||
"comprehensive review :cite:`Lipton.Berkowitz.Elkan.2015`.\n",
|
||||
"We also note that sequentiality is not unique to RNNs.\n",
|
||||
"For example, the CNNs that we already introduced\n",
|
||||
"can be adapted to handle data of varying length,\n",
|
||||
"e.g., images of varying resolution.\n",
|
||||
"Moreover, RNNs have recently ceded considerable\n",
|
||||
"market share to Transformer models,\n",
|
||||
"which will be covered in :numref:`chap_attention-and-transformers`.\n",
|
||||
"However, RNNs rose to prominence as the default models\n",
|
||||
"for handling complex sequential structure in deep learning,\n",
|
||||
"and remain staple models for sequential modeling to this day.\n",
|
||||
"The stories of RNNs and of sequence modeling\n",
|
||||
"are inextricably linked, and this is as much\n",
|
||||
"a chapter about the ABCs of sequence modeling problems\n",
|
||||
"as it is a chapter about RNNs.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"One key insight paved the way for a revolution in sequence modeling.\n",
|
||||
"While the inputs and targets for many fundamental tasks in machine learning\n",
|
||||
"cannot easily be represented as fixed-length vectors,\n",
|
||||
"they can often nevertheless be represented as\n",
|
||||
"varying-length sequences of fixed-length vectors.\n",
|
||||
"For example, documents can be represented as sequences of words;\n",
|
||||
"medical records can often be represented as sequences of events\n",
|
||||
"(encounters, medications, procedures, lab tests, diagnoses);\n",
|
||||
"videos can be represented as varying-length sequences of still images.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"While sequence models have popped up in numerous application areas,\n",
|
||||
"basic research in the area has been driven predominantly\n",
|
||||
"by advances on core tasks in natural language processing.\n",
|
||||
"Thus, throughout this chapter, we will focus\n",
|
||||
"our exposition and examples on text data.\n",
|
||||
"If you get the hang of these examples,\n",
|
||||
"then applying the models to other data modalities\n",
|
||||
"should be relatively straightforward.\n",
|
||||
"In the next few sections, we introduce basic\n",
|
||||
"notation for sequences and some evaluation measures\n",
|
||||
"for assessing the quality of sequentially structured model outputs.\n",
|
||||
"After that, we discuss basic concepts of a language model\n",
|
||||
"and use this discussion to motivate our first RNN models.\n",
|
||||
"Finally, we describe the method for calculating gradients\n",
|
||||
"when backpropagating through RNNs and explore some challenges\n",
|
||||
"that are often encountered when training such networks,\n",
|
||||
"motivating the modern RNN architectures that will follow\n",
|
||||
"in :numref:`chap_modern_rnn`.\n",
|
||||
"\n",
|
||||
":begin_tab:toc\n",
|
||||
" - [sequence](sequence.ipynb)\n",
|
||||
" - [text-sequence](text-sequence.ipynb)\n",
|
||||
" - [language-model](language-model.ipynb)\n",
|
||||
" - [rnn](rnn.ipynb)\n",
|
||||
" - [rnn-scratch](rnn-scratch.ipynb)\n",
|
||||
" - [rnn-concise](rnn-concise.ipynb)\n",
|
||||
" - [bptt](bptt.ipynb)\n",
|
||||
":end_tab:\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"required_libs": []
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@@ -0,0 +1,470 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "741b627f",
|
||||
"metadata": {
|
||||
"origin_pos": 0
|
||||
},
|
||||
"source": [
|
||||
"# Language Models\n",
|
||||
":label:`sec_language-model`\n",
|
||||
"\n",
|
||||
"In :numref:`sec_text-sequence`, we saw how to map text sequences into tokens, where these tokens can be viewed as a sequence of discrete observations such as words or characters. Assume that the tokens in a text sequence of length $T$ are in turn $x_1, x_2, \\ldots, x_T$.\n",
|
||||
"The goal of *language models*\n",
|
||||
"is to estimate the joint probability of the whole sequence:\n",
|
||||
"\n",
|
||||
"$$P(x_1, x_2, \\ldots, x_T),$$\n",
|
||||
"\n",
|
||||
"where statistical tools\n",
|
||||
"in :numref:`sec_sequence`\n",
|
||||
"can be applied.\n",
|
||||
"\n",
|
||||
"Language models are incredibly useful. For instance, an ideal language model should generate natural text on its own, simply by drawing one token at a time $x_t \\sim P(x_t \\mid x_{t-1}, \\ldots, x_1)$.\n",
|
||||
"Quite unlike the monkey using a typewriter, all text emerging from such a model would pass as natural language, e.g., English text. Furthermore, it would be sufficient for generating a meaningful dialog, simply by conditioning the text on previous dialog fragments.\n",
|
||||
"Clearly we are still very far from designing such a system, since it would need to *understand* the text rather than just generate grammatically sensible content.\n",
|
||||
"\n",
|
||||
"Nonetheless, language models are of great service even in their limited form.\n",
|
||||
"For instance, the phrases \"to recognize speech\" and \"to wreck a nice beach\" sound very similar.\n",
|
||||
"This can cause ambiguity in speech recognition,\n",
|
||||
"which is easily resolved through a language model that rejects the second translation as outlandish.\n",
|
||||
"Likewise, in a document summarization algorithm\n",
|
||||
"it is worthwhile knowing that \"dog bites man\" is much more frequent than \"man bites dog\", or that \"I want to eat grandma\" is a rather disturbing statement, whereas \"I want to eat, grandma\" is much more benign.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "8b60dead",
|
||||
"metadata": {
|
||||
"attributes": {
|
||||
"classes": [],
|
||||
"id": "",
|
||||
"n": "3"
|
||||
},
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:28:41.375771Z",
|
||||
"iopub.status.busy": "2023-08-18T19:28:41.375138Z",
|
||||
"iopub.status.idle": "2023-08-18T19:28:44.315880Z",
|
||||
"shell.execute_reply": "2023-08-18T19:28:44.314922Z"
|
||||
},
|
||||
"origin_pos": 3,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"from d2l import torch as d2l"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "54c6ae2a",
|
||||
"metadata": {
|
||||
"origin_pos": 6
|
||||
},
|
||||
"source": [
|
||||
"## Learning Language Models\n",
|
||||
"\n",
|
||||
"The obvious question is how we should model a document, or even a sequence of tokens. \n",
|
||||
"Suppose that we tokenize text data at the word level.\n",
|
||||
"Let's start by applying basic probability rules:\n",
|
||||
"\n",
|
||||
"$$P(x_1, x_2, \\ldots, x_T) = \\prod_{t=1}^T P(x_t \\mid x_1, \\ldots, x_{t-1}).$$\n",
|
||||
"\n",
|
||||
"For example, \n",
|
||||
"the probability of a text sequence containing four words would be given as:\n",
|
||||
"\n",
|
||||
"$$\\begin{aligned}&P(\\textrm{deep}, \\textrm{learning}, \\textrm{is}, \\textrm{fun}) \\\\\n",
|
||||
"=&P(\\textrm{deep}) P(\\textrm{learning} \\mid \\textrm{deep}) P(\\textrm{is} \\mid \\textrm{deep}, \\textrm{learning}) P(\\textrm{fun} \\mid \\textrm{deep}, \\textrm{learning}, \\textrm{is}).\\end{aligned}$$\n",
|
||||
"\n",
|
||||
"### Markov Models and $n$-grams\n",
|
||||
":label:`subsec_markov-models-and-n-grams`\n",
|
||||
"\n",
|
||||
"Among those sequence model analyses in :numref:`sec_sequence`,\n",
|
||||
"let's apply Markov models to language modeling.\n",
|
||||
"A distribution over sequences satisfies the Markov property of first order if $P(x_{t+1} \\mid x_t, \\ldots, x_1) = P(x_{t+1} \\mid x_t)$. Higher orders correspond to longer dependencies. This leads to a number of approximations that we could apply to model a sequence:\n",
|
||||
"\n",
|
||||
"$$\n",
|
||||
"\\begin{aligned}\n",
|
||||
"P(x_1, x_2, x_3, x_4) &= P(x_1) P(x_2) P(x_3) P(x_4),\\\\\n",
|
||||
"P(x_1, x_2, x_3, x_4) &= P(x_1) P(x_2 \\mid x_1) P(x_3 \\mid x_2) P(x_4 \\mid x_3),\\\\\n",
|
||||
"P(x_1, x_2, x_3, x_4) &= P(x_1) P(x_2 \\mid x_1) P(x_3 \\mid x_1, x_2) P(x_4 \\mid x_2, x_3).\n",
|
||||
"\\end{aligned}\n",
|
||||
"$$\n",
|
||||
"\n",
|
||||
"The probability formulae that involve one, two, and three variables are typically referred to as *unigram*, *bigram*, and *trigram* models, respectively. \n",
|
||||
"In order to compute the language model, we need to calculate the\n",
|
||||
"probability of words and the conditional probability of a word given\n",
|
||||
"the previous few words.\n",
|
||||
"Note that\n",
|
||||
"such probabilities are\n",
|
||||
"language model parameters.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Word Frequency\n",
|
||||
"\n",
|
||||
"Here, we\n",
|
||||
"assume that the training dataset is a large text corpus, such as all\n",
|
||||
"Wikipedia entries, [Project Gutenberg](https://en.wikipedia.org/wiki/Project_Gutenberg),\n",
|
||||
"and all text posted on the\n",
|
||||
"web.\n",
|
||||
"The probability of words can be calculated from the relative word\n",
|
||||
"frequency of a given word in the training dataset.\n",
|
||||
"For example, the estimate $\\hat{P}(\\textrm{deep})$ can be calculated as the\n",
|
||||
"probability of any sentence starting with the word \"deep\". A\n",
|
||||
"slightly less accurate approach would be to count all occurrences of\n",
|
||||
"the word \"deep\" and divide it by the total number of words in\n",
|
||||
"the corpus.\n",
|
||||
"This works fairly well, particularly for frequent\n",
|
||||
"words. Moving on, we could attempt to estimate\n",
|
||||
"\n",
|
||||
"$$\\hat{P}(\\textrm{learning} \\mid \\textrm{deep}) = \\frac{n(\\textrm{deep, learning})}{n(\\textrm{deep})},$$\n",
|
||||
"\n",
|
||||
"where $n(x)$ and $n(x, x')$ are the number of occurrences of singletons\n",
|
||||
"and consecutive word pairs, respectively.\n",
|
||||
"Unfortunately, \n",
|
||||
"estimating the\n",
|
||||
"probability of a word pair is somewhat more difficult, since the\n",
|
||||
"occurrences of \"deep learning\" are a lot less frequent. \n",
|
||||
"In particular, for some unusual word combinations it may be tricky to\n",
|
||||
"find enough occurrences to get accurate estimates.\n",
|
||||
"As suggested by the empirical results in :numref:`subsec_natural-lang-stat`,\n",
|
||||
"things take a turn for the worse for three-word combinations and beyond.\n",
|
||||
"There will be many plausible three-word combinations that we likely will not see in our dataset.\n",
|
||||
"Unless we provide some solution to assign such word combinations a nonzero count, we will not be able to use them in a language model. If the dataset is small or if the words are very rare, we might not find even a single one of them.\n",
|
||||
"\n",
|
||||
"### Laplace Smoothing\n",
|
||||
"\n",
|
||||
"A common strategy is to perform some form of *Laplace smoothing*.\n",
|
||||
"The solution is to\n",
|
||||
"add a small constant to all counts. \n",
|
||||
"Denote by $n$ the total number of words in\n",
|
||||
"the training set\n",
|
||||
"and $m$ the number of unique words.\n",
|
||||
"This solution helps with singletons, e.g., via\n",
|
||||
"\n",
|
||||
"$$\\begin{aligned}\n",
|
||||
"\t\\hat{P}(x) & = \\frac{n(x) + \\epsilon_1/m}{n + \\epsilon_1}, \\\\\n",
|
||||
"\t\\hat{P}(x' \\mid x) & = \\frac{n(x, x') + \\epsilon_2 \\hat{P}(x')}{n(x) + \\epsilon_2}, \\\\\n",
|
||||
"\t\\hat{P}(x'' \\mid x,x') & = \\frac{n(x, x',x'') + \\epsilon_3 \\hat{P}(x'')}{n(x, x') + \\epsilon_3}.\n",
|
||||
"\\end{aligned}$$\n",
|
||||
"\n",
|
||||
"Here $\\epsilon_1,\\epsilon_2$, and $\\epsilon_3$ are hyperparameters.\n",
|
||||
"Take $\\epsilon_1$ as an example:\n",
|
||||
"when $\\epsilon_1 = 0$, no smoothing is applied;\n",
|
||||
"when $\\epsilon_1$ approaches positive infinity,\n",
|
||||
"$\\hat{P}(x)$ approaches the uniform probability $1/m$. \n",
|
||||
"The above is a rather primitive variant of what\n",
|
||||
"other techniques can accomplish :cite:`Wood.Gasthaus.Archambeau.ea.2011`.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Unfortunately, models like this get unwieldy rather quickly\n",
|
||||
"for the following reasons. \n",
|
||||
"First, \n",
|
||||
"as discussed in :numref:`subsec_natural-lang-stat`,\n",
|
||||
"many $n$-grams occur very rarely, \n",
|
||||
"making Laplace smoothing rather unsuitable for language modeling.\n",
|
||||
"Second, we need to store all counts.\n",
|
||||
"Third, this entirely ignores the meaning of the words. For\n",
|
||||
"instance, \"cat\" and \"feline\" should occur in related contexts.\n",
|
||||
"It is quite difficult to adjust such models to additional contexts,\n",
|
||||
"whereas, deep learning based language models are well suited to\n",
|
||||
"take this into account.\n",
|
||||
"Last, long word\n",
|
||||
"sequences are almost certain to be novel, hence a model that simply\n",
|
||||
"counts the frequency of previously seen word sequences is bound to perform poorly there.\n",
|
||||
"Therefore, we focus on using neural networks for language modeling\n",
|
||||
"in the rest of the chapter.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Perplexity\n",
|
||||
":label:`subsec_perplexity`\n",
|
||||
"\n",
|
||||
"Next, let's discuss about how to measure the quality of the language model, which we will then use to evaluate our models in the subsequent sections.\n",
|
||||
"One way is to check how surprising the text is.\n",
|
||||
"A good language model is able to predict, with high accuracy, the tokens that come next.\n",
|
||||
"Consider the following continuations of the phrase \"It is raining\", as proposed by different language models:\n",
|
||||
"\n",
|
||||
"1. \"It is raining outside\"\n",
|
||||
"1. \"It is raining banana tree\"\n",
|
||||
"1. \"It is raining piouw;kcj pwepoiut\"\n",
|
||||
"\n",
|
||||
"In terms of quality, Example 1 is clearly the best. The words are sensible and logically coherent.\n",
|
||||
"While it might not quite accurately reflect which word follows semantically (\"in San Francisco\" and \"in winter\" would have been perfectly reasonable extensions), the model is able to capture which kind of word follows.\n",
|
||||
"Example 2 is considerably worse by producing a nonsensical extension. Nonetheless, at least the model has learned how to spell words and some degree of correlation between words. Last, Example 3 indicates a poorly trained model that does not fit data properly.\n",
|
||||
"\n",
|
||||
"We might measure the quality of the model by computing the likelihood of the sequence.\n",
|
||||
"Unfortunately this is a number that is hard to understand and difficult to compare.\n",
|
||||
"After all, shorter sequences are much more likely to occur than the longer ones,\n",
|
||||
"hence evaluating the model on Tolstoy's magnum opus\n",
|
||||
"*War and Peace* will inevitably produce a much smaller likelihood than, say, on Saint-Exupery's novella *The Little Prince*. What is missing is the equivalent of an average.\n",
|
||||
"\n",
|
||||
"Information theory comes handy here.\n",
|
||||
"We defined entropy, surprisal, and cross-entropy\n",
|
||||
"when we introduced the softmax regression\n",
|
||||
"(:numref:`subsec_info_theory_basics`).\n",
|
||||
"If we want to compress text, we can ask about\n",
|
||||
"predicting the next token given the current set of tokens.\n",
|
||||
"A better language model should allow us to predict the next token more accurately.\n",
|
||||
"Thus, it should allow us to spend fewer bits in compressing the sequence.\n",
|
||||
"So we can measure it by the cross-entropy loss averaged\n",
|
||||
"over all the $n$ tokens of a sequence:\n",
|
||||
"\n",
|
||||
"$$\\frac{1}{n} \\sum_{t=1}^n -\\log P(x_t \\mid x_{t-1}, \\ldots, x_1),$$\n",
|
||||
":eqlabel:`eq_avg_ce_for_lm`\n",
|
||||
"\n",
|
||||
"where $P$ is given by a language model and $x_t$ is the actual token observed at time step $t$ from the sequence.\n",
|
||||
"This makes the performance on documents of different lengths comparable. For historical reasons, scientists in natural language processing prefer to use a quantity called *perplexity*. In a nutshell, it is the exponential of :eqref:`eq_avg_ce_for_lm`:\n",
|
||||
"\n",
|
||||
"$$\\exp\\left(-\\frac{1}{n} \\sum_{t=1}^n \\log P(x_t \\mid x_{t-1}, \\ldots, x_1)\\right).$$\n",
|
||||
"\n",
|
||||
"Perplexity can be best understood as the reciprocal of the geometric mean of the number of real choices that we have when deciding which token to pick next. Let's look at a number of cases:\n",
|
||||
"\n",
|
||||
"* In the best case scenario, the model always perfectly estimates the probability of the target token as 1. In this case the perplexity of the model is 1.\n",
|
||||
"* In the worst case scenario, the model always predicts the probability of the target token as 0. In this situation, the perplexity is positive infinity.\n",
|
||||
"* At the baseline, the model predicts a uniform distribution over all the available tokens of the vocabulary. In this case, the perplexity equals the number of unique tokens of the vocabulary. In fact, if we were to store the sequence without any compression, this would be the best we could do for encoding it. Hence, this provides a nontrivial upper bound that any useful model must beat.\n",
|
||||
"\n",
|
||||
"## Partitioning Sequences\n",
|
||||
":label:`subsec_partitioning-seqs`\n",
|
||||
"\n",
|
||||
"We will design language models using neural networks\n",
|
||||
"and use perplexity to evaluate \n",
|
||||
"how good the model is at \n",
|
||||
"predicting the next token given the current set of tokens\n",
|
||||
"in text sequences.\n",
|
||||
"Before introducing the model,\n",
|
||||
"let's assume that it\n",
|
||||
"processes a minibatch of sequences with predefined length\n",
|
||||
"at a time.\n",
|
||||
"Now the question is how to [**read minibatches of input sequences and target sequences at random**].\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Suppose that the dataset takes the form of a sequence of $T$ token indices in `corpus`.\n",
|
||||
"We will\n",
|
||||
"partition it\n",
|
||||
"into subsequences, where each subsequence has $n$ tokens (time steps).\n",
|
||||
"To iterate over \n",
|
||||
"(almost) all the tokens of the entire dataset \n",
|
||||
"for each epoch\n",
|
||||
"and obtain all possible length-$n$ subsequences,\n",
|
||||
"we can introduce randomness.\n",
|
||||
"More concretely,\n",
|
||||
"at the beginning of each epoch,\n",
|
||||
"discard the first $d$ tokens,\n",
|
||||
"where $d\\in [0,n)$ is uniformly sampled at random.\n",
|
||||
"The rest of the sequence\n",
|
||||
"is then partitioned\n",
|
||||
"into $m=\\lfloor (T-d)/n \\rfloor$ subsequences.\n",
|
||||
"Denote by $\\mathbf x_t = [x_t, \\ldots, x_{t+n-1}]$ the length-$n$ subsequence starting from token $x_t$ at time step $t$. \n",
|
||||
"The resulting $m$ partitioned subsequences\n",
|
||||
"are \n",
|
||||
"$\\mathbf x_d, \\mathbf x_{d+n}, \\ldots, \\mathbf x_{d+n(m-1)}.$\n",
|
||||
"Each subsequence will be used as an input sequence into the language model.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"For language modeling,\n",
|
||||
"the goal is to predict the next token based on the tokens we have seen so far; hence the targets (labels) are the original sequence, shifted by one token.\n",
|
||||
"The target sequence for any input sequence $\\mathbf x_t$\n",
|
||||
"is $\\mathbf x_{t+1}$ with length $n$.\n",
|
||||
"\n",
|
||||
" \n",
|
||||
":label:`fig_lang_model_data`\n",
|
||||
"\n",
|
||||
":numref:`fig_lang_model_data` shows an example of obtaining five pairs of input sequences and target sequences with $n=5$ and $d=2$.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "a09a2d6b",
|
||||
"metadata": {
|
||||
"attributes": {
|
||||
"classes": [],
|
||||
"id": "",
|
||||
"n": "5"
|
||||
},
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:28:44.321194Z",
|
||||
"iopub.status.busy": "2023-08-18T19:28:44.320794Z",
|
||||
"iopub.status.idle": "2023-08-18T19:28:44.326575Z",
|
||||
"shell.execute_reply": "2023-08-18T19:28:44.325777Z"
|
||||
},
|
||||
"origin_pos": 7,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@d2l.add_to_class(d2l.TimeMachine) #@save\n",
|
||||
"def __init__(self, batch_size, num_steps, num_train=10000, num_val=5000):\n",
|
||||
" super(d2l.TimeMachine, self).__init__()\n",
|
||||
" self.save_hyperparameters()\n",
|
||||
" corpus, self.vocab = self.build(self._download())\n",
|
||||
" array = torch.tensor([corpus[i:i+num_steps+1]\n",
|
||||
" for i in range(len(corpus)-num_steps)])\n",
|
||||
" self.X, self.Y = array[:,:-1], array[:,1:]"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "5d5393db",
|
||||
"metadata": {
|
||||
"origin_pos": 8
|
||||
},
|
||||
"source": [
|
||||
"To train language models,\n",
|
||||
"we will randomly sample \n",
|
||||
"pairs of input sequences and target sequences\n",
|
||||
"in minibatches.\n",
|
||||
"The following data loader randomly generates a minibatch from the dataset each time.\n",
|
||||
"The argument `batch_size` specifies the number of subsequence examples in each minibatch\n",
|
||||
"and `num_steps` is the subsequence length in tokens.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "06c824e8",
|
||||
"metadata": {
|
||||
"attributes": {
|
||||
"classes": [],
|
||||
"id": "",
|
||||
"n": "6"
|
||||
},
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:28:44.329887Z",
|
||||
"iopub.status.busy": "2023-08-18T19:28:44.329335Z",
|
||||
"iopub.status.idle": "2023-08-18T19:28:44.334517Z",
|
||||
"shell.execute_reply": "2023-08-18T19:28:44.333506Z"
|
||||
},
|
||||
"origin_pos": 9,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"@d2l.add_to_class(d2l.TimeMachine) #@save\n",
|
||||
"def get_dataloader(self, train):\n",
|
||||
" idx = slice(0, self.num_train) if train else slice(\n",
|
||||
" self.num_train, self.num_train + self.num_val)\n",
|
||||
" return self.get_tensorloader([self.X, self.Y], train, idx)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d6608ee2",
|
||||
"metadata": {
|
||||
"origin_pos": 10
|
||||
},
|
||||
"source": [
|
||||
"As we can see in the following, \n",
|
||||
"a minibatch of target sequences\n",
|
||||
"can be obtained \n",
|
||||
"by shifting the input sequences\n",
|
||||
"by one token.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "7cbf8e11",
|
||||
"metadata": {
|
||||
"attributes": {
|
||||
"classes": [],
|
||||
"id": "",
|
||||
"n": "7"
|
||||
},
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:28:44.338600Z",
|
||||
"iopub.status.busy": "2023-08-18T19:28:44.338043Z",
|
||||
"iopub.status.idle": "2023-08-18T19:28:45.209913Z",
|
||||
"shell.execute_reply": "2023-08-18T19:28:45.208778Z"
|
||||
},
|
||||
"origin_pos": 11,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"Downloading ../data/timemachine.txt from http://d2l-data.s3-accelerate.amazonaws.com/timemachine.txt...\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"X: tensor([[10, 4, 2, 21, 10, 16, 15, 0, 20, 2],\n",
|
||||
" [21, 9, 6, 19, 0, 24, 2, 26, 0, 16]]) \n",
|
||||
"Y: tensor([[ 4, 2, 21, 10, 16, 15, 0, 20, 2, 10],\n",
|
||||
" [ 9, 6, 19, 0, 24, 2, 26, 0, 16, 9]])\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"data = d2l.TimeMachine(batch_size=2, num_steps=10)\n",
|
||||
"for X, Y in data.train_dataloader():\n",
|
||||
" print('X:', X, '\\nY:', Y)\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "7f4c8710",
|
||||
"metadata": {
|
||||
"origin_pos": 12
|
||||
},
|
||||
"source": [
|
||||
"## Summary and Discussion\n",
|
||||
"\n",
|
||||
"Language models estimate the joint probability of a text sequence. For long sequences, $n$-grams provide a convenient model by truncating the dependence. However, there is a lot of structure but not enough frequency to deal efficiently with infrequent word combinations via Laplace smoothing. Thus, we will focus on neural language modeling in subsequent sections.\n",
|
||||
"To train language models, we can randomly sample pairs of input sequences and target sequences in minibatches. After training, we will use perplexity to measure the language model quality.\n",
|
||||
"\n",
|
||||
"Language models can be scaled up with increased data size, model size, and amount in training compute. Large language models can perform desired tasks by predicting output text given input text instructions. As we will discuss later (e.g., :numref:`sec_large-pretraining-transformers`),\n",
|
||||
"at the present moment\n",
|
||||
"large language models form the basis of state-of-the-art systems across diverse tasks.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Exercises\n",
|
||||
"\n",
|
||||
"1. Suppose there are 100,000 words in the training dataset. How much word frequency and multi-word adjacent frequency does a four-gram need to store?\n",
|
||||
"1. How would you model a dialogue?\n",
|
||||
"1. What other methods can you think of for reading long sequence data?\n",
|
||||
"1. Consider our method for discarding a uniformly random number of the first few tokens at the beginning of each epoch.\n",
|
||||
" 1. Does it really lead to a perfectly uniform distribution over the sequences on the document?\n",
|
||||
" 1. What would you have to do to make things even more uniform? \n",
|
||||
"1. If we want a sequence example to be a complete sentence, what kind of problem does this introduce in minibatch sampling? How can we fix it?\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "59140a53",
|
||||
"metadata": {
|
||||
"origin_pos": 14,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"source": [
|
||||
"[Discussions](https://discuss.d2l.ai/t/118)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"required_libs": []
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,335 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "36609bcf",
|
||||
"metadata": {
|
||||
"origin_pos": 0
|
||||
},
|
||||
"source": [
|
||||
"# Recurrent Neural Networks\n",
|
||||
":label:`sec_rnn`\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"In :numref:`sec_language-model` we described Markov models and $n$-grams for language modeling, where the conditional probability of token $x_t$ at time step $t$ only depends on the $n-1$ previous tokens.\n",
|
||||
"If we want to incorporate the possible effect of tokens earlier than time step $t-(n-1)$ on $x_t$,\n",
|
||||
"we need to increase $n$.\n",
|
||||
"However, the number of model parameters would also increase exponentially with it, as we need to store $|\\mathcal{V}|^n$ numbers for a vocabulary set $\\mathcal{V}$.\n",
|
||||
"Hence, rather than modeling $P(x_t \\mid x_{t-1}, \\ldots, x_{t-n+1})$ it is preferable to use a latent variable model,\n",
|
||||
"\n",
|
||||
"$$P(x_t \\mid x_{t-1}, \\ldots, x_1) \\approx P(x_t \\mid h_{t-1}),$$\n",
|
||||
"\n",
|
||||
"where $h_{t-1}$ is a *hidden state* that stores the sequence information up to time step $t-1$.\n",
|
||||
"In general,\n",
|
||||
"the hidden state at any time step $t$ could be computed based on both the current input $x_{t}$ and the previous hidden state $h_{t-1}$:\n",
|
||||
"\n",
|
||||
"$$h_t = f(x_{t}, h_{t-1}).$$\n",
|
||||
":eqlabel:`eq_ht_xt`\n",
|
||||
"\n",
|
||||
"For a sufficiently powerful function $f$ in :eqref:`eq_ht_xt`, the latent variable model is not an approximation. After all, $h_t$ may simply store all the data it has observed so far.\n",
|
||||
"However, it could potentially make both computation and storage expensive.\n",
|
||||
"\n",
|
||||
"Recall that we have discussed hidden layers with hidden units in :numref:`chap_perceptrons`.\n",
|
||||
"It is noteworthy that\n",
|
||||
"hidden layers and hidden states refer to two very different concepts.\n",
|
||||
"Hidden layers are, as explained, layers that are hidden from view on the path from input to output.\n",
|
||||
"Hidden states are technically speaking *inputs* to whatever we do at a given step,\n",
|
||||
"and they can only be computed by looking at data at previous time steps.\n",
|
||||
"\n",
|
||||
"*Recurrent neural networks* (RNNs) are neural networks with hidden states. Before introducing the RNN model, we first revisit the MLP model introduced in :numref:`sec_mlp`.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "deee527d",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:27:13.858746Z",
|
||||
"iopub.status.busy": "2023-08-18T19:27:13.858238Z",
|
||||
"iopub.status.idle": "2023-08-18T19:27:16.785485Z",
|
||||
"shell.execute_reply": "2023-08-18T19:27:16.784592Z"
|
||||
},
|
||||
"origin_pos": 3,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import torch\n",
|
||||
"from d2l import torch as d2l"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3cbba908",
|
||||
"metadata": {
|
||||
"origin_pos": 6
|
||||
},
|
||||
"source": [
|
||||
"## Neural Networks without Hidden States\n",
|
||||
"\n",
|
||||
"Let's take a look at an MLP with a single hidden layer.\n",
|
||||
"Let the hidden layer's activation function be $\\phi$.\n",
|
||||
"Given a minibatch of examples $\\mathbf{X} \\in \\mathbb{R}^{n \\times d}$ with batch size $n$ and $d$ inputs, the hidden layer output $\\mathbf{H} \\in \\mathbb{R}^{n \\times h}$ is calculated as\n",
|
||||
"\n",
|
||||
"$$\\mathbf{H} = \\phi(\\mathbf{X} \\mathbf{W}_{\\textrm{xh}} + \\mathbf{b}_\\textrm{h}).$$\n",
|
||||
":eqlabel:`rnn_h_without_state`\n",
|
||||
"\n",
|
||||
"In :eqref:`rnn_h_without_state`, we have the weight parameter $\\mathbf{W}_{\\textrm{xh}} \\in \\mathbb{R}^{d \\times h}$, the bias parameter $\\mathbf{b}_\\textrm{h} \\in \\mathbb{R}^{1 \\times h}$, and the number of hidden units $h$, for the hidden layer.\n",
|
||||
"So armed, we apply broadcasting (see :numref:`subsec_broadcasting`) during the summation.\n",
|
||||
"Next, the hidden layer output $\\mathbf{H}$ is used as input of the output layer, which is given by\n",
|
||||
"\n",
|
||||
"$$\\mathbf{O} = \\mathbf{H} \\mathbf{W}_{\\textrm{hq}} + \\mathbf{b}_\\textrm{q},$$\n",
|
||||
"\n",
|
||||
"where $\\mathbf{O} \\in \\mathbb{R}^{n \\times q}$ is the output variable, $\\mathbf{W}_{\\textrm{hq}} \\in \\mathbb{R}^{h \\times q}$ is the weight parameter, and $\\mathbf{b}_\\textrm{q} \\in \\mathbb{R}^{1 \\times q}$ is the bias parameter of the output layer. If it is a classification problem, we can use $\\mathrm{softmax}(\\mathbf{O})$ to compute the probability distribution of the output categories.\n",
|
||||
"\n",
|
||||
"This is entirely analogous to the regression problem we solved previously in :numref:`sec_sequence`, hence we omit details.\n",
|
||||
"Suffice it to say that we can pick feature-label pairs at random and learn the parameters of our network via automatic differentiation and stochastic gradient descent.\n",
|
||||
"\n",
|
||||
"## Recurrent Neural Networks with Hidden States\n",
|
||||
":label:`subsec_rnn_w_hidden_states`\n",
|
||||
"\n",
|
||||
"Matters are entirely different when we have hidden states. Let's look at the structure in some more detail.\n",
|
||||
"\n",
|
||||
"Assume that we have\n",
|
||||
"a minibatch of inputs\n",
|
||||
"$\\mathbf{X}_t \\in \\mathbb{R}^{n \\times d}$\n",
|
||||
"at time step $t$.\n",
|
||||
"In other words,\n",
|
||||
"for a minibatch of $n$ sequence examples,\n",
|
||||
"each row of $\\mathbf{X}_t$ corresponds to one example at time step $t$ from the sequence.\n",
|
||||
"Next,\n",
|
||||
"denote by $\\mathbf{H}_t \\in \\mathbb{R}^{n \\times h}$ the hidden layer output of time step $t$.\n",
|
||||
"Unlike with MLP, here we save the hidden layer output $\\mathbf{H}_{t-1}$ from the previous time step and introduce a new weight parameter $\\mathbf{W}_{\\textrm{hh}} \\in \\mathbb{R}^{h \\times h}$ to describe how to use the hidden layer output of the previous time step in the current time step. Specifically, the calculation of the hidden layer output of the current time step is determined by the input of the current time step together with the hidden layer output of the previous time step:\n",
|
||||
"\n",
|
||||
"$$\\mathbf{H}_t = \\phi(\\mathbf{X}_t \\mathbf{W}_{\\textrm{xh}} + \\mathbf{H}_{t-1} \\mathbf{W}_{\\textrm{hh}} + \\mathbf{b}_\\textrm{h}).$$\n",
|
||||
":eqlabel:`rnn_h_with_state`\n",
|
||||
"\n",
|
||||
"Compared with :eqref:`rnn_h_without_state`, :eqref:`rnn_h_with_state` adds one more term $\\mathbf{H}_{t-1} \\mathbf{W}_{\\textrm{hh}}$ and thus\n",
|
||||
"instantiates :eqref:`eq_ht_xt`.\n",
|
||||
"From the relationship between hidden layer outputs $\\mathbf{H}_t$ and $\\mathbf{H}_{t-1}$ of adjacent time steps,\n",
|
||||
"we know that these variables captured and retained the sequence's historical information up to their current time step, just like the state or memory of the neural network's current time step. Therefore, such a hidden layer output is called a *hidden state*.\n",
|
||||
"Since the hidden state uses the same definition of the previous time step in the current time step, the computation of :eqref:`rnn_h_with_state` is *recurrent*. Hence, as we said, neural networks with hidden states\n",
|
||||
"based on recurrent computation are named\n",
|
||||
"*recurrent neural networks*.\n",
|
||||
"Layers that perform\n",
|
||||
"the computation of :eqref:`rnn_h_with_state`\n",
|
||||
"in RNNs\n",
|
||||
"are called *recurrent layers*.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"There are many different ways for constructing RNNs.\n",
|
||||
"Those with a hidden state defined by :eqref:`rnn_h_with_state` are very common.\n",
|
||||
"For time step $t$,\n",
|
||||
"the output of the output layer is similar to the computation in the MLP:\n",
|
||||
"\n",
|
||||
"$$\\mathbf{O}_t = \\mathbf{H}_t \\mathbf{W}_{\\textrm{hq}} + \\mathbf{b}_\\textrm{q}.$$\n",
|
||||
"\n",
|
||||
"Parameters of the RNN\n",
|
||||
"include the weights $\\mathbf{W}_{\\textrm{xh}} \\in \\mathbb{R}^{d \\times h}, \\mathbf{W}_{\\textrm{hh}} \\in \\mathbb{R}^{h \\times h}$,\n",
|
||||
"and the bias $\\mathbf{b}_\\textrm{h} \\in \\mathbb{R}^{1 \\times h}$\n",
|
||||
"of the hidden layer,\n",
|
||||
"together with the weights $\\mathbf{W}_{\\textrm{hq}} \\in \\mathbb{R}^{h \\times q}$\n",
|
||||
"and the bias $\\mathbf{b}_\\textrm{q} \\in \\mathbb{R}^{1 \\times q}$\n",
|
||||
"of the output layer.\n",
|
||||
"It is worth mentioning that\n",
|
||||
"even at different time steps,\n",
|
||||
"RNNs always use these model parameters.\n",
|
||||
"Therefore, the parametrization cost of an RNN\n",
|
||||
"does not grow as the number of time steps increases.\n",
|
||||
"\n",
|
||||
":numref:`fig_rnn` illustrates the computational logic of an RNN at three adjacent time steps.\n",
|
||||
"At any time step $t$,\n",
|
||||
"the computation of the hidden state can be treated as:\n",
|
||||
"(i) concatenating the input $\\mathbf{X}_t$ at the current time step $t$ and the hidden state $\\mathbf{H}_{t-1}$ at the previous time step $t-1$;\n",
|
||||
"(ii) feeding the concatenation result into a fully connected layer with the activation function $\\phi$.\n",
|
||||
"The output of such a fully connected layer is the hidden state $\\mathbf{H}_t$ of the current time step $t$.\n",
|
||||
"In this case,\n",
|
||||
"the model parameters are the concatenation of $\\mathbf{W}_{\\textrm{xh}}$ and $\\mathbf{W}_{\\textrm{hh}}$, and a bias of $\\mathbf{b}_\\textrm{h}$, all from :eqref:`rnn_h_with_state`.\n",
|
||||
"The hidden state of the current time step $t$, $\\mathbf{H}_t$, will participate in computing the hidden state $\\mathbf{H}_{t+1}$ of the next time step $t+1$.\n",
|
||||
"What is more, $\\mathbf{H}_t$ will also be\n",
|
||||
"fed into the fully connected output layer\n",
|
||||
"to compute the output\n",
|
||||
"$\\mathbf{O}_t$ of the current time step $t$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":label:`fig_rnn`\n",
|
||||
"\n",
|
||||
"We just mentioned that the calculation of $\\mathbf{X}_t \\mathbf{W}_{\\textrm{xh}} + \\mathbf{H}_{t-1} \\mathbf{W}_{\\textrm{hh}}$ for the hidden state is equivalent to\n",
|
||||
"matrix multiplication of the\n",
|
||||
"concatenation of $\\mathbf{X}_t$ and $\\mathbf{H}_{t-1}$\n",
|
||||
"and the\n",
|
||||
"concatenation of $\\mathbf{W}_{\\textrm{xh}}$ and $\\mathbf{W}_{\\textrm{hh}}$.\n",
|
||||
"Though this can be proven mathematically,\n",
|
||||
"in the following we just use a simple code snippet as a demonstration.\n",
|
||||
"To begin with,\n",
|
||||
"we define matrices `X`, `W_xh`, `H`, and `W_hh`, whose shapes are (3, 1), (1, 4), (3, 4), and (4, 4), respectively.\n",
|
||||
"Multiplying `X` by `W_xh`, and `H` by `W_hh`, and then adding these two products,\n",
|
||||
"we obtain a matrix of shape (3, 4).\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "09b186bf",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:27:16.789724Z",
|
||||
"iopub.status.busy": "2023-08-18T19:27:16.789025Z",
|
||||
"iopub.status.idle": "2023-08-18T19:27:16.822423Z",
|
||||
"shell.execute_reply": "2023-08-18T19:27:16.821267Z"
|
||||
},
|
||||
"origin_pos": 7,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"tensor([[ 1.2526, 0.0580, -3.3460, -0.2519],\n",
|
||||
" [-1.3064, 1.4132, -0.1435, 0.3482],\n",
|
||||
" [ 3.1495, 0.8172, 1.5167, -0.9038]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 2,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"X, W_xh = torch.randn(3, 1), torch.randn(1, 4)\n",
|
||||
"H, W_hh = torch.randn(3, 4), torch.randn(4, 4)\n",
|
||||
"torch.matmul(X, W_xh) + torch.matmul(H, W_hh)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "59eb3429",
|
||||
"metadata": {
|
||||
"origin_pos": 10
|
||||
},
|
||||
"source": [
|
||||
"Now we concatenate the matrices `X` and `H`\n",
|
||||
"along columns (axis 1),\n",
|
||||
"and the matrices\n",
|
||||
"`W_xh` and `W_hh` along rows (axis 0).\n",
|
||||
"These two concatenations\n",
|
||||
"result in\n",
|
||||
"matrices of shape (3, 5)\n",
|
||||
"and of shape (5, 4), respectively.\n",
|
||||
"Multiplying these two concatenated matrices,\n",
|
||||
"we obtain the same output matrix of shape (3, 4)\n",
|
||||
"as above.\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "52ac9b29",
|
||||
"metadata": {
|
||||
"execution": {
|
||||
"iopub.execute_input": "2023-08-18T19:27:16.827070Z",
|
||||
"iopub.status.busy": "2023-08-18T19:27:16.826785Z",
|
||||
"iopub.status.idle": "2023-08-18T19:27:16.833496Z",
|
||||
"shell.execute_reply": "2023-08-18T19:27:16.832705Z"
|
||||
},
|
||||
"origin_pos": 11,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"tensor([[ 1.2526, 0.0580, -3.3460, -0.2519],\n",
|
||||
" [-1.3064, 1.4132, -0.1435, 0.3482],\n",
|
||||
" [ 3.1495, 0.8172, 1.5167, -0.9038]])"
|
||||
]
|
||||
},
|
||||
"execution_count": 3,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"torch.matmul(torch.cat((X, H), 1), torch.cat((W_xh, W_hh), 0))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ce7632f2",
|
||||
"metadata": {
|
||||
"origin_pos": 12
|
||||
},
|
||||
"source": [
|
||||
"## RNN-Based Character-Level Language Models\n",
|
||||
"\n",
|
||||
"Recall that for language modeling in :numref:`sec_language-model`,\n",
|
||||
"we aim to predict the next token based on\n",
|
||||
"the current and past tokens;\n",
|
||||
"thus we shift the original sequence by one token\n",
|
||||
"as the targets (labels).\n",
|
||||
":citet:`Bengio.Ducharme.Vincent.ea.2003` first proposed\n",
|
||||
"to use a neural network for language modeling.\n",
|
||||
"In the following we illustrate how RNNs can be used to build a language model.\n",
|
||||
"Let the minibatch size be one, and the sequence of the text be \"machine\".\n",
|
||||
"To simplify training in subsequent sections,\n",
|
||||
"we tokenize text into characters rather than words\n",
|
||||
"and consider a *character-level language model*.\n",
|
||||
":numref:`fig_rnn_train` demonstrates how to predict the next character based on the current and previous characters via an RNN for character-level language modeling.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
":label:`fig_rnn_train`\n",
|
||||
"\n",
|
||||
"During the training process,\n",
|
||||
"we run a softmax operation on the output from the output layer for each time step, and then use the cross-entropy loss to compute the error between the model output and the target.\n",
|
||||
"Because of the recurrent computation of the hidden state in the hidden layer, the output, $\\mathbf{O}_3$, of time step 3 in :numref:`fig_rnn_train` is determined by the text sequence \"m\", \"a\", and \"c\". Since the next character of the sequence in the training data is \"h\", the loss of time step 3 will depend on the probability distribution of the next character generated based on the feature sequence \"m\", \"a\", \"c\" and the target \"h\" of this time step.\n",
|
||||
"\n",
|
||||
"In practice, each token is represented by a $d$-dimensional vector, and we use a batch size $n>1$. Therefore, the input $\\mathbf X_t$ at time step $t$ will be an $n\\times d$ matrix, which is identical to what we discussed in :numref:`subsec_rnn_w_hidden_states`.\n",
|
||||
"\n",
|
||||
"In the following sections, we will implement RNNs\n",
|
||||
"for character-level language models.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Summary\n",
|
||||
"\n",
|
||||
"A neural network that uses recurrent computation for hidden states is called a recurrent neural network (RNN).\n",
|
||||
"The hidden state of an RNN can capture historical information of the sequence up to the current time step. With recurrent computation, the number of RNN model parameters does not grow as the number of time steps increases. As for applications, an RNN can be used to create character-level language models.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Exercises\n",
|
||||
"\n",
|
||||
"1. If we use an RNN to predict the next character in a text sequence, what is the required dimension for any output?\n",
|
||||
"1. Why can RNNs express the conditional probability of a token at some time step based on all the previous tokens in the text sequence?\n",
|
||||
"1. What happens to the gradient if you backpropagate through a long sequence?\n",
|
||||
"1. What are some of the problems associated with the language model described in this section?\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1c7bc477",
|
||||
"metadata": {
|
||||
"origin_pos": 14,
|
||||
"tab": [
|
||||
"pytorch"
|
||||
]
|
||||
},
|
||||
"source": [
|
||||
"[Discussions](https://discuss.d2l.ai/t/1050)\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"language_info": {
|
||||
"name": "python"
|
||||
},
|
||||
"required_libs": []
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user