xixu-me/d2l-pytorch-notebooks
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
91fc65dcf00a1f997917edbb250edda5ff7bf4d7
d2l-pytorch-notebooks/chapter_optimization/optimization-intro.ipynb
T
xixu-me 4c20ec342f Initial commit
2024-08-20 16:25:10 +08:00

6435 lines
234 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{
"cells": [
{
"cell_type": "markdown",
"id": "848845cf",
"metadata": {
"origin_pos": 0
},
"source": [
"# Optimization and Deep Learning\n",
":label:`sec_optimization-intro`\n",
"\n",
"In this section, we will discuss the relationship between optimization and deep learning as well as the challenges of using optimization in deep learning.\n",
"For a deep learning problem, we will usually define a *loss function* first. Once we have the loss function, we can use an optimization algorithm in attempt to minimize the loss.\n",
"In optimization, a loss function is often referred to as the *objective function* of the optimization problem. By tradition and convention most optimization algorithms are concerned with *minimization*. If we ever need to maximize an objective there is a simple solution: just flip the sign on the objective.\n",
"\n",
"## Goal of Optimization\n",
"\n",
"Although optimization provides a way to minimize the loss function for deep\n",
"learning, in essence, the goals of optimization and deep learning are\n",
"fundamentally different.\n",
"The former is primarily concerned with minimizing an\n",
"objective whereas the latter is concerned with finding a suitable model, given a\n",
"finite amount of data.\n",
"In :numref:`sec_generalization_basics`,\n",
"we discussed the difference between these two goals in detail.\n",
"For instance,\n",
"training error and generalization error generally differ: since the objective\n",
"function of the optimization algorithm is usually a loss function based on the\n",
"training dataset, the goal of optimization is to reduce the training error.\n",
"However, the goal of deep learning (or more broadly, statistical inference) is to\n",
"reduce the generalization error.\n",
"To accomplish the latter we need to pay\n",
"attention to overfitting in addition to using the optimization algorithm to\n",
"reduce the training error.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "14cc4e80",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:41:57.543191Z",
"iopub.status.busy": "2023-08-18T19:41:57.542597Z",
"iopub.status.idle": "2023-08-18T19:42:00.769262Z",
"shell.execute_reply": "2023-08-18T19:42:00.768380Z"
},
"origin_pos": 2,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import numpy as np\n",
"import torch\n",
"from mpl_toolkits import mplot3d\n",
"from d2l import torch as d2l"
]
},
{
"cell_type": "markdown",
"id": "57d2e15b",
"metadata": {
"origin_pos": 4
},
"source": [
"To illustrate the aforementioned different goals,\n",
"let's consider\n",
"the empirical risk and the risk.\n",
"As described\n",
"in :numref:`subsec_empirical-risk-and-risk`,\n",
"the empirical risk\n",
"is an average loss\n",
"on the training dataset\n",
"while the risk is the expected loss\n",
"on the entire population of data.\n",
"Below we define two functions:\n",
"the risk function `f`\n",
"and the empirical risk function `g`.\n",
"Suppose that we have only a finite amount of training data.\n",
"As a result, here `g` is less smooth than `f`.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f29da23f",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:42:00.773480Z",
"iopub.status.busy": "2023-08-18T19:42:00.772792Z",
"iopub.status.idle": "2023-08-18T19:42:00.777413Z",
"shell.execute_reply": "2023-08-18T19:42:00.776663Z"
},
"origin_pos": 5,
"tab": [
"pytorch"
]
},
"outputs": [],
"source": [
"def f(x):\n",
" return x * torch.cos(np.pi * x)\n",
"\n",
"def g(x):\n",
" return f(x) + 0.2 * torch.cos(5 * np.pi * x)"
]
},
{
"cell_type": "markdown",
"id": "14067cd6",
"metadata": {
"origin_pos": 6
},
"source": [
"The graph below illustrates that the minimum of the empirical risk on a training dataset may be at a different location from the minimum of the risk (generalization error).\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "db4011c0",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:42:00.780794Z",
"iopub.status.busy": "2023-08-18T19:42:00.780104Z",
"iopub.status.idle": "2023-08-18T19:42:01.146519Z",
"shell.execute_reply": "2023-08-18T19:42:01.145342Z"
},
"origin_pos": 7,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"261.023438pt\" height=\"183.35625pt\" viewBox=\"0 0 261.023438 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
" <metadata>\n",
" <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
" <cc:Work>\n",
" <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
" <dc:date>2023-08-18T19:42:01.081082</dc:date>\n",
" <dc:format>image/svg+xml</dc:format>\n",
" <dc:creator>\n",
" <cc:Agent>\n",
" <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
" </cc:Agent>\n",
" </dc:creator>\n",
" </cc:Work>\n",
" </rdf:RDF>\n",
" </metadata>\n",
" <defs>\n",
" <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
" </defs>\n",
" <g id=\"figure_1\">\n",
" <g id=\"patch_1\">\n",
" <path d=\"M 0 183.35625 \n",
"L 261.023438 183.35625 \n",
"L 261.023438 0 \n",
"L 0 0 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"axes_1\">\n",
" <g id=\"patch_2\">\n",
" <path d=\"M 58.523438 145.8 \n",
"L 253.823438 145.8 \n",
"L 253.823438 7.2 \n",
"L 58.523438 7.2 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"matplotlib.axis_1\">\n",
" <g id=\"xtick_1\">\n",
" <g id=\"line2d_1\">\n",
" <path d=\"M 85.334594 145.8 \n",
"L 85.334594 7.2 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_2\">\n",
" <defs>\n",
" <path id=\"m0443ea7734\" d=\"M 0 0 \n",
"L 0 3.5 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#m0443ea7734\" x=\"85.334594\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_1\">\n",
" <!-- 0.6 -->\n",
" <g transform=\"translate(77.383032 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
"Q 1547 4250 1301 3770 \n",
"Q 1056 3291 1056 2328 \n",
"Q 1056 1369 1301 889 \n",
"Q 1547 409 2034 409 \n",
"Q 2525 409 2770 889 \n",
"Q 3016 1369 3016 2328 \n",
"Q 3016 3291 2770 3770 \n",
"Q 2525 4250 2034 4250 \n",
"z\n",
"M 2034 4750 \n",
"Q 2819 4750 3233 4129 \n",
"Q 3647 3509 3647 2328 \n",
"Q 3647 1150 3233 529 \n",
"Q 2819 -91 2034 -91 \n",
"Q 1250 -91 836 529 \n",
"Q 422 1150 422 2328 \n",
"Q 422 3509 836 4129 \n",
"Q 1250 4750 2034 4750 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
"L 1344 794 \n",
"L 1344 0 \n",
"L 684 0 \n",
"L 684 794 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
"Q 1688 2584 1439 2293 \n",
"Q 1191 2003 1191 1497 \n",
"Q 1191 994 1439 701 \n",
"Q 1688 409 2113 409 \n",
"Q 2538 409 2786 701 \n",
"Q 3034 994 3034 1497 \n",
"Q 3034 2003 2786 2293 \n",
"Q 2538 2584 2113 2584 \n",
"z\n",
"M 3366 4563 \n",
"L 3366 3988 \n",
"Q 3128 4100 2886 4159 \n",
"Q 2644 4219 2406 4219 \n",
"Q 1781 4219 1451 3797 \n",
"Q 1122 3375 1075 2522 \n",
"Q 1259 2794 1537 2939 \n",
"Q 1816 3084 2150 3084 \n",
"Q 2853 3084 3261 2657 \n",
"Q 3669 2231 3669 1497 \n",
"Q 3669 778 3244 343 \n",
"Q 2819 -91 2113 -91 \n",
"Q 1303 -91 875 529 \n",
"Q 447 1150 447 2328 \n",
"Q 447 3434 972 4092 \n",
"Q 1497 4750 2381 4750 \n",
"Q 2619 4750 2861 4703 \n",
"Q 3103 4656 3366 4563 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_2\">\n",
" <g id=\"line2d_3\">\n",
" <path d=\"M 121.202363 145.8 \n",
"L 121.202363 7.2 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_4\">\n",
" <g>\n",
" <use xlink:href=\"#m0443ea7734\" x=\"121.202363\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_2\">\n",
" <!-- 0.8 -->\n",
" <g transform=\"translate(113.2508 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n",
"Q 1584 2216 1326 1975 \n",
"Q 1069 1734 1069 1313 \n",
"Q 1069 891 1326 650 \n",
"Q 1584 409 2034 409 \n",
"Q 2484 409 2743 651 \n",
"Q 3003 894 3003 1313 \n",
"Q 3003 1734 2745 1975 \n",
"Q 2488 2216 2034 2216 \n",
"z\n",
"M 1403 2484 \n",
"Q 997 2584 770 2862 \n",
"Q 544 3141 544 3541 \n",
"Q 544 4100 942 4425 \n",
"Q 1341 4750 2034 4750 \n",
"Q 2731 4750 3128 4425 \n",
"Q 3525 4100 3525 3541 \n",
"Q 3525 3141 3298 2862 \n",
"Q 3072 2584 2669 2484 \n",
"Q 3125 2378 3379 2068 \n",
"Q 3634 1759 3634 1313 \n",
"Q 3634 634 3220 271 \n",
"Q 2806 -91 2034 -91 \n",
"Q 1263 -91 848 271 \n",
"Q 434 634 434 1313 \n",
"Q 434 1759 690 2068 \n",
"Q 947 2378 1403 2484 \n",
"z\n",
"M 1172 3481 \n",
"Q 1172 3119 1398 2916 \n",
"Q 1625 2713 2034 2713 \n",
"Q 2441 2713 2670 2916 \n",
"Q 2900 3119 2900 3481 \n",
"Q 2900 3844 2670 4047 \n",
"Q 2441 4250 2034 4250 \n",
"Q 1625 4250 1398 4047 \n",
"Q 1172 3844 1172 3481 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_3\">\n",
" <g id=\"line2d_5\">\n",
" <path d=\"M 157.070131 145.8 \n",
"L 157.070131 7.2 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_6\">\n",
" <g>\n",
" <use xlink:href=\"#m0443ea7734\" x=\"157.070131\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_3\">\n",
" <!-- 1.0 -->\n",
" <g transform=\"translate(149.118568 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
"L 1825 531 \n",
"L 1825 4091 \n",
"L 703 3866 \n",
"L 703 4441 \n",
"L 1819 4666 \n",
"L 2450 4666 \n",
"L 2450 531 \n",
"L 3481 531 \n",
"L 3481 0 \n",
"L 794 0 \n",
"L 794 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_4\">\n",
" <g id=\"line2d_7\">\n",
" <path d=\"M 192.937899 145.8 \n",
"L 192.937899 7.2 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_8\">\n",
" <g>\n",
" <use xlink:href=\"#m0443ea7734\" x=\"192.937899\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_4\">\n",
" <!-- 1.2 -->\n",
" <g transform=\"translate(184.986337 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
"L 3431 531 \n",
"L 3431 0 \n",
"L 469 0 \n",
"L 469 531 \n",
"Q 828 903 1448 1529 \n",
"Q 2069 2156 2228 2338 \n",
"Q 2531 2678 2651 2914 \n",
"Q 2772 3150 2772 3378 \n",
"Q 2772 3750 2511 3984 \n",
"Q 2250 4219 1831 4219 \n",
"Q 1534 4219 1204 4116 \n",
"Q 875 4013 500 3803 \n",
"L 500 4441 \n",
"Q 881 4594 1212 4672 \n",
"Q 1544 4750 1819 4750 \n",
"Q 2544 4750 2975 4387 \n",
"Q 3406 4025 3406 3419 \n",
"Q 3406 3131 3298 2873 \n",
"Q 3191 2616 2906 2266 \n",
"Q 2828 2175 2409 1742 \n",
"Q 1991 1309 1228 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_5\">\n",
" <g id=\"line2d_9\">\n",
" <path d=\"M 228.805667 145.8 \n",
"L 228.805667 7.2 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_10\">\n",
" <g>\n",
" <use xlink:href=\"#m0443ea7734\" x=\"228.805667\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_5\">\n",
" <!-- 1.4 -->\n",
" <g transform=\"translate(220.854105 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
"L 825 1625 \n",
"L 2419 1625 \n",
"L 2419 4116 \n",
"z\n",
"M 2253 4666 \n",
"L 3047 4666 \n",
"L 3047 1625 \n",
"L 3713 1625 \n",
"L 3713 1100 \n",
"L 3047 1100 \n",
"L 3047 0 \n",
"L 2419 0 \n",
"L 2419 1100 \n",
"L 313 1100 \n",
"L 313 1709 \n",
"L 2253 4666 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_6\">\n",
" <!-- x -->\n",
" <g transform=\"translate(153.214063 174.076563) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
"L 2247 1797 \n",
"L 3578 0 \n",
"L 2900 0 \n",
"L 1881 1375 \n",
"L 863 0 \n",
"L 184 0 \n",
"L 1544 1831 \n",
"L 300 3500 \n",
"L 978 3500 \n",
"L 1906 2253 \n",
"L 2834 3500 \n",
"L 3513 3500 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-78\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"matplotlib.axis_2\">\n",
" <g id=\"ytick_1\">\n",
" <g id=\"line2d_11\">\n",
" <path d=\"M 58.523438 143.859356 \n",
"L 253.823438 143.859356 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_12\">\n",
" <defs>\n",
" <path id=\"m4c573ff2b3\" d=\"M 0 0 \n",
"L -3.5 0 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#m4c573ff2b3\" x=\"58.523438\" y=\"143.859356\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_7\">\n",
" <!-- −1.25 -->\n",
" <g transform=\"translate(20.878125 147.658575) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
"L 4684 2272 \n",
"L 4684 1741 \n",
"L 678 1741 \n",
"L 678 2272 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
"L 3169 4666 \n",
"L 3169 4134 \n",
"L 1269 4134 \n",
"L 1269 2991 \n",
"Q 1406 3038 1543 3061 \n",
"Q 1681 3084 1819 3084 \n",
"Q 2600 3084 3056 2656 \n",
"Q 3513 2228 3513 1497 \n",
"Q 3513 744 3044 326 \n",
"Q 2575 -91 1722 -91 \n",
"Q 1428 -91 1123 -41 \n",
"Q 819 9 494 109 \n",
"L 494 744 \n",
"Q 775 591 1075 516 \n",
"Q 1375 441 1709 441 \n",
"Q 2250 441 2565 725 \n",
"Q 2881 1009 2881 1497 \n",
"Q 2881 1984 2565 2268 \n",
"Q 2250 2553 1709 2553 \n",
"Q 1456 2553 1204 2497 \n",
"Q 953 2441 691 2322 \n",
"L 691 4666 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-32\" x=\"179.199219\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"242.822266\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_2\">\n",
" <g id=\"line2d_13\">\n",
" <path d=\"M 58.523438 117.787485 \n",
"L 253.823438 117.787485 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_14\">\n",
" <g>\n",
" <use xlink:href=\"#m4c573ff2b3\" x=\"58.523438\" y=\"117.787485\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_8\">\n",
" <!-- −1.00 -->\n",
" <g transform=\"translate(20.878125 121.586704) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"179.199219\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"242.822266\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_3\">\n",
" <g id=\"line2d_15\">\n",
" <path d=\"M 58.523438 91.715614 \n",
"L 253.823438 91.715614 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_16\">\n",
" <g>\n",
" <use xlink:href=\"#m4c573ff2b3\" x=\"58.523438\" y=\"91.715614\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_9\">\n",
" <!-- −0.75 -->\n",
" <g transform=\"translate(20.878125 95.514833) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n",
"L 3525 4666 \n",
"L 3525 4397 \n",
"L 1831 0 \n",
"L 1172 0 \n",
"L 2766 4134 \n",
"L 525 4134 \n",
"L 525 4666 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-37\" x=\"179.199219\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"242.822266\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_4\">\n",
" <g id=\"line2d_17\">\n",
" <path d=\"M 58.523438 65.643743 \n",
"L 253.823438 65.643743 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_18\">\n",
" <g>\n",
" <use xlink:href=\"#m4c573ff2b3\" x=\"58.523438\" y=\"65.643743\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_10\">\n",
" <!-- −0.50 -->\n",
" <g transform=\"translate(20.878125 69.442961) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"242.822266\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_5\">\n",
" <g id=\"line2d_19\">\n",
" <path d=\"M 58.523438 39.571872 \n",
"L 253.823438 39.571872 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_20\">\n",
" <g>\n",
" <use xlink:href=\"#m4c573ff2b3\" x=\"58.523438\" y=\"39.571872\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_11\">\n",
" <!-- −0.25 -->\n",
" <g transform=\"translate(20.878125 43.37109) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-32\" x=\"179.199219\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"242.822266\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_6\">\n",
" <g id=\"line2d_21\">\n",
" <path d=\"M 58.523438 13.500001 \n",
"L 253.823438 13.500001 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_22\">\n",
" <g>\n",
" <use xlink:href=\"#m4c573ff2b3\" x=\"58.523438\" y=\"13.500001\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_12\">\n",
" <!-- 0.00 -->\n",
" <g transform=\"translate(29.257812 17.299219) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_13\">\n",
" <!-- risk -->\n",
" <g transform=\"translate(14.798437 85.444531) rotate(-90) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
"Q 2534 3019 2420 3045 \n",
"Q 2306 3072 2169 3072 \n",
"Q 1681 3072 1420 2755 \n",
"Q 1159 2438 1159 1844 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1341 3275 1631 3429 \n",
"Q 1922 3584 2338 3584 \n",
"Q 2397 3584 2469 3576 \n",
"Q 2541 3569 2628 3553 \n",
"L 2631 2963 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
"L 1178 3500 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 3500 \n",
"z\n",
"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 4134 \n",
"L 603 4134 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
"L 2834 2853 \n",
"Q 2591 2978 2328 3040 \n",
"Q 2066 3103 1784 3103 \n",
"Q 1356 3103 1142 2972 \n",
"Q 928 2841 928 2578 \n",
"Q 928 2378 1081 2264 \n",
"Q 1234 2150 1697 2047 \n",
"L 1894 2003 \n",
"Q 2506 1872 2764 1633 \n",
"Q 3022 1394 3022 966 \n",
"Q 3022 478 2636 193 \n",
"Q 2250 -91 1575 -91 \n",
"Q 1294 -91 989 -36 \n",
"Q 684 19 347 128 \n",
"L 347 722 \n",
"Q 666 556 975 473 \n",
"Q 1284 391 1588 391 \n",
"Q 1994 391 2212 530 \n",
"Q 2431 669 2431 922 \n",
"Q 2431 1156 2273 1281 \n",
"Q 2116 1406 1581 1522 \n",
"L 1381 1569 \n",
"Q 847 1681 609 1914 \n",
"Q 372 2147 372 2553 \n",
"Q 372 3047 722 3315 \n",
"Q 1072 3584 1716 3584 \n",
"Q 2034 3584 2315 3537 \n",
"Q 2597 3491 2834 3397 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
"L 1159 4863 \n",
"L 1159 1991 \n",
"L 2875 3500 \n",
"L 3609 3500 \n",
"L 1753 1863 \n",
"L 3688 0 \n",
"L 2938 0 \n",
"L 1159 1709 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-72\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"41.113281\"/>\n",
" <use xlink:href=\"#DejaVuSans-73\" x=\"68.896484\"/>\n",
" <use xlink:href=\"#DejaVuSans-6b\" x=\"120.996094\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"line2d_23\">\n",
" <path d=\"M 67.40071 13.500003 \n",
"L 69.194097 15.170636 \n",
"L 70.987484 16.905096 \n",
"L 72.78087 18.701587 \n",
"L 74.574268 20.558178 \n",
"L 76.367654 22.472793 \n",
"L 78.161041 24.443255 \n",
"L 79.954428 26.467279 \n",
"L 81.747815 28.542444 \n",
"L 83.541201 30.666212 \n",
"L 85.334588 32.835959 \n",
"L 87.127975 35.048933 \n",
"L 88.921372 37.302291 \n",
"L 90.714759 39.593065 \n",
"L 92.508145 41.918217 \n",
"L 94.301532 44.274585 \n",
"L 96.09493 46.658991 \n",
"L 97.888316 49.068064 \n",
"L 99.681703 51.498428 \n",
"L 101.47509 53.946613 \n",
"L 103.268487 56.409067 \n",
"L 105.061874 58.882175 \n",
"L 106.85526 61.362264 \n",
"L 108.648647 63.845598 \n",
"L 110.442034 66.328375 \n",
"L 112.235421 68.806789 \n",
"L 114.028807 71.276954 \n",
"L 115.822194 73.734941 \n",
"L 117.615591 76.17683 \n",
"L 119.408978 78.598597 \n",
"L 121.202365 80.996283 \n",
"L 122.995751 83.365866 \n",
"L 124.789138 85.703332 \n",
"L 126.582525 88.004638 \n",
"L 128.375912 90.265759 \n",
"L 130.169298 92.482695 \n",
"L 131.962696 94.651446 \n",
"L 133.756082 96.767987 \n",
"L 135.549469 98.828367 \n",
"L 137.342856 100.828657 \n",
"L 139.136242 102.76496 \n",
"L 140.929629 104.63342 \n",
"L 142.723016 106.43021 \n",
"L 144.516403 108.151587 \n",
"L 146.3098 109.793847 \n",
"L 148.103187 111.353359 \n",
"L 149.896573 112.826543 \n",
"L 151.68996 114.209911 \n",
"L 153.483357 115.500065 \n",
"L 155.276744 116.693665 \n",
"L 157.070131 117.787485 \n",
"L 158.863518 118.77838 \n",
"L 160.656904 119.66333 \n",
"L 162.450291 120.439388 \n",
"L 164.243678 121.103756 \n",
"L 166.037086 121.653712 \n",
"L 167.830451 122.086682 \n",
"L 169.623838 122.400218 \n",
"L 171.417224 122.591982 \n",
"L 173.210611 122.659811 \n",
"L 175.003998 122.601617 \n",
"L 176.797385 122.415522 \n",
"L 178.590771 122.099736 \n",
"L 180.384179 121.652667 \n",
"L 182.177566 121.072837 \n",
"L 183.970953 120.358952 \n",
"L 185.764339 119.509869 \n",
"L 187.557726 118.524593 \n",
"L 189.351113 117.402347 \n",
"L 191.1445 116.142447 \n",
"L 192.937886 114.744421 \n",
"L 194.731294 113.207964 \n",
"L 196.524681 111.532958 \n",
"L 198.318068 109.719441 \n",
"L 200.111454 107.767642 \n",
"L 201.904841 105.677966 \n",
"L 203.698228 103.450997 \n",
"L 205.491615 101.087523 \n",
"L 207.285001 98.588417 \n",
"L 209.078409 95.954889 \n",
"L 210.871775 93.188257 \n",
"L 212.665161 90.289945 \n",
"L 214.458548 87.261663 \n",
"L 216.251935 84.105268 \n",
"L 218.045321 80.8228 \n",
"L 219.838708 77.416484 \n",
"L 221.632095 73.888707 \n",
"L 223.425503 70.242011 \n",
"L 225.21889 66.479238 \n",
"L 227.012276 62.603333 \n",
"L 228.805663 58.617261 \n",
"L 230.59905 54.524358 \n",
"L 232.392436 50.328058 \n",
"L 234.185823 46.031953 \n",
"L 235.97921 41.639804 \n",
"L 237.772618 37.155464 \n",
"L 239.566005 32.58314 \n",
"L 241.359391 27.926982 \n",
"L 243.152778 23.191436 \n",
"L 244.946165 18.380867 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_24\">\n",
" <path d=\"M 67.40071 13.5 \n",
"L 69.194097 18.433456 \n",
"L 70.987484 23.350407 \n",
"L 72.78087 28.170685 \n",
"L 74.574268 32.817916 \n",
"L 76.367654 37.221266 \n",
"L 78.161041 41.31732 \n",
"L 79.954428 45.051444 \n",
"L 81.747815 48.3791 \n",
"L 83.541201 51.266916 \n",
"L 85.334588 53.693457 \n",
"L 87.127975 55.649644 \n",
"L 88.921372 57.138949 \n",
"L 90.714759 58.177229 \n",
"L 92.508145 58.792292 \n",
"L 94.301532 59.023071 \n",
"L 96.09493 58.918713 \n",
"L 97.888316 58.537162 \n",
"L 99.681703 57.943759 \n",
"L 101.47509 57.209451 \n",
"L 103.268487 56.409055 \n",
"L 105.061874 55.619328 \n",
"L 106.85526 54.916946 \n",
"L 108.648647 54.376494 \n",
"L 110.442034 54.068646 \n",
"L 112.235421 54.058309 \n",
"L 114.028807 54.402881 \n",
"L 115.822194 55.15078 \n",
"L 117.615591 56.340169 \n",
"L 119.408978 57.997891 \n",
"L 121.202365 60.138787 \n",
"L 122.995751 62.765157 \n",
"L 124.789138 65.866668 \n",
"L 126.582525 69.420471 \n",
"L 128.375912 73.391686 \n",
"L 130.169298 77.734197 \n",
"L 131.962696 82.39171 \n",
"L 133.756082 87.298878 \n",
"L 135.549469 92.383046 \n",
"L 137.342856 97.565826 \n",
"L 139.136242 102.764941 \n",
"L 140.929629 107.896233 \n",
"L 142.723016 112.875512 \n",
"L 144.516403 117.620678 \n",
"L 146.3098 122.053563 \n",
"L 148.103187 126.101832 \n",
"L 149.896573 129.700609 \n",
"L 151.68996 132.794072 \n",
"L 153.483357 135.336722 \n",
"L 155.276744 137.29437 \n",
"L 157.070131 138.644987 \n",
"L 158.863518 139.379086 \n",
"L 160.656904 139.5 \n",
"L 162.450291 139.023567 \n",
"L 164.243678 137.977834 \n",
"L 166.037086 136.402173 \n",
"L 167.830451 134.346423 \n",
"L 169.623838 131.86934 \n",
"L 171.417224 129.037322 \n",
"L 173.210611 125.922699 \n",
"L 175.003998 122.601667 \n",
"L 176.797385 119.152746 \n",
"L 178.590771 115.654464 \n",
"L 180.384179 112.18357 \n",
"L 182.177566 108.813114 \n",
"L 183.970953 105.610479 \n",
"L 185.764339 102.635803 \n",
"L 187.557726 99.940426 \n",
"L 189.351113 97.565701 \n",
"L 191.1445 95.541747 \n",
"L 192.937886 93.886925 \n",
"L 194.731294 92.607258 \n",
"L 196.524681 91.696301 \n",
"L 198.318068 91.135274 \n",
"L 200.111454 90.893576 \n",
"L 201.904841 90.929493 \n",
"L 203.698228 91.191274 \n",
"L 205.491615 91.618389 \n",
"L 207.285001 92.143071 \n",
"L 209.078409 92.692069 \n",
"L 210.871775 93.188232 \n",
"L 212.665161 93.552752 \n",
"L 214.458548 93.706965 \n",
"L 216.251935 93.574352 \n",
"L 218.045321 93.08251 \n",
"L 219.838708 92.16492 \n",
"L 221.632095 90.762742 \n",
"L 223.425503 88.826172 \n",
"L 225.21889 86.315895 \n",
"L 227.012276 83.204038 \n",
"L 228.805663 79.474757 \n",
"L 230.59905 75.125067 \n",
"L 232.392436 70.164721 \n",
"L 234.185823 64.616123 \n",
"L 235.97921 58.513901 \n",
"L 237.772618 51.903946 \n",
"L 239.566005 44.842872 \n",
"L 241.359391 37.39609 \n",
"L 243.152778 29.636758 \n",
"L 244.946165 21.643698 \n",
"\" clip-path=\"url(#p1b44431415)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n",
" </g>\n",
" <g id=\"patch_3\">\n",
" <path d=\"M 58.523438 145.8 \n",
"L 58.523438 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_4\">\n",
" <path d=\"M 253.823438 145.8 \n",
"L 253.823438 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_5\">\n",
" <path d=\"M 58.523438 145.8 \n",
"L 253.823438 145.8 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_6\">\n",
" <path d=\"M 58.523438 7.2 \n",
"L 253.823438 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_7\">\n",
" <path d=\"M 138.679605 132.472942 \n",
"Q 146.92526 135.240264 154.11098 137.651862 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" <path d=\"M 150.955183 134.483116 \n",
"L 154.11098 137.651862 \n",
"L 149.682506 138.275252 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" </g>\n",
" <g id=\"text_14\">\n",
" <!-- min of -->\n",
" <g transform=\"translate(67.40071 117.018421) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
"Q 3544 3216 3844 3400 \n",
"Q 4144 3584 4550 3584 \n",
"Q 5097 3584 5394 3201 \n",
"Q 5691 2819 5691 2113 \n",
"L 5691 0 \n",
"L 5113 0 \n",
"L 5113 2094 \n",
"Q 5113 2597 4934 2840 \n",
"Q 4756 3084 4391 3084 \n",
"Q 3944 3084 3684 2787 \n",
"Q 3425 2491 3425 1978 \n",
"L 3425 0 \n",
"L 2847 0 \n",
"L 2847 2094 \n",
"Q 2847 2600 2669 2842 \n",
"Q 2491 3084 2119 3084 \n",
"Q 1678 3084 1418 2786 \n",
"Q 1159 2488 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1356 3278 1631 3431 \n",
"Q 1906 3584 2284 3584 \n",
"Q 2666 3584 2933 3390 \n",
"Q 3200 3197 3328 2828 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
"L 3513 0 \n",
"L 2938 0 \n",
"L 2938 2094 \n",
"Q 2938 2591 2744 2837 \n",
"Q 2550 3084 2163 3084 \n",
"Q 1697 3084 1428 2787 \n",
"Q 1159 2491 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1366 3272 1645 3428 \n",
"Q 1925 3584 2291 3584 \n",
"Q 2894 3584 3203 3211 \n",
"Q 3513 2838 3513 2113 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
"Q 1497 3097 1228 2736 \n",
"Q 959 2375 959 1747 \n",
"Q 959 1119 1226 758 \n",
"Q 1494 397 1959 397 \n",
"Q 2419 397 2687 759 \n",
"Q 2956 1122 2956 1747 \n",
"Q 2956 2369 2687 2733 \n",
"Q 2419 3097 1959 3097 \n",
"z\n",
"M 1959 3584 \n",
"Q 2709 3584 3137 3096 \n",
"Q 3566 2609 3566 1747 \n",
"Q 3566 888 3137 398 \n",
"Q 2709 -91 1959 -91 \n",
"Q 1206 -91 779 398 \n",
"Q 353 888 353 1747 \n",
"Q 353 2609 779 3096 \n",
"Q 1206 3584 1959 3584 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
"L 2375 4384 \n",
"L 1825 4384 \n",
"Q 1516 4384 1395 4259 \n",
"Q 1275 4134 1275 3809 \n",
"L 1275 3500 \n",
"L 2222 3500 \n",
"L 2222 3053 \n",
"L 1275 3053 \n",
"L 1275 0 \n",
"L 697 0 \n",
"L 697 3053 \n",
"L 147 3053 \n",
"L 147 3500 \n",
"L 697 3500 \n",
"L 697 3744 \n",
"Q 697 4328 969 4595 \n",
"Q 1241 4863 1831 4863 \n",
"L 2375 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-6d\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"97.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"125.195312\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"188.574219\"/>\n",
" <use xlink:href=\"#DejaVuSans-6f\" x=\"220.361328\"/>\n",
" <use xlink:href=\"#DejaVuSans-66\" x=\"281.542969\"/>\n",
" </g>\n",
" <!-- empirical risk -->\n",
" <g transform=\"translate(67.40071 128.216233) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
"L 3597 1613 \n",
"L 953 1613 \n",
"Q 991 1019 1311 708 \n",
"Q 1631 397 2203 397 \n",
"Q 2534 397 2845 478 \n",
"Q 3156 559 3463 722 \n",
"L 3463 178 \n",
"Q 3153 47 2828 -22 \n",
"Q 2503 -91 2169 -91 \n",
"Q 1331 -91 842 396 \n",
"Q 353 884 353 1716 \n",
"Q 353 2575 817 3079 \n",
"Q 1281 3584 2069 3584 \n",
"Q 2775 3584 3186 3129 \n",
"Q 3597 2675 3597 1894 \n",
"z\n",
"M 3022 2063 \n",
"Q 3016 2534 2758 2815 \n",
"Q 2500 3097 2075 3097 \n",
"Q 1594 3097 1305 2825 \n",
"Q 1016 2553 972 2059 \n",
"L 3022 2063 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
"L 1159 -1331 \n",
"L 581 -1331 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2969 \n",
"Q 1341 3281 1617 3432 \n",
"Q 1894 3584 2278 3584 \n",
"Q 2916 3584 3314 3078 \n",
"Q 3713 2572 3713 1747 \n",
"Q 3713 922 3314 415 \n",
"Q 2916 -91 2278 -91 \n",
"Q 1894 -91 1617 61 \n",
"Q 1341 213 1159 525 \n",
"z\n",
"M 3116 1747 \n",
"Q 3116 2381 2855 2742 \n",
"Q 2594 3103 2138 3103 \n",
"Q 1681 3103 1420 2742 \n",
"Q 1159 2381 1159 1747 \n",
"Q 1159 1113 1420 752 \n",
"Q 1681 391 2138 391 \n",
"Q 2594 391 2855 752 \n",
"Q 3116 1113 3116 1747 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
"L 3122 2828 \n",
"Q 2878 2963 2633 3030 \n",
"Q 2388 3097 2138 3097 \n",
"Q 1578 3097 1268 2742 \n",
"Q 959 2388 959 1747 \n",
"Q 959 1106 1268 751 \n",
"Q 1578 397 2138 397 \n",
"Q 2388 397 2633 464 \n",
"Q 2878 531 3122 666 \n",
"L 3122 134 \n",
"Q 2881 22 2623 -34 \n",
"Q 2366 -91 2075 -91 \n",
"Q 1284 -91 818 406 \n",
"Q 353 903 353 1747 \n",
"Q 353 2603 823 3093 \n",
"Q 1294 3584 2113 3584 \n",
"Q 2378 3584 2631 3529 \n",
"Q 2884 3475 3122 3366 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
"Q 1497 1759 1228 1600 \n",
"Q 959 1441 959 1056 \n",
"Q 959 750 1161 570 \n",
"Q 1363 391 1709 391 \n",
"Q 2188 391 2477 730 \n",
"Q 2766 1069 2766 1631 \n",
"L 2766 1759 \n",
"L 2194 1759 \n",
"z\n",
"M 3341 1997 \n",
"L 3341 0 \n",
"L 2766 0 \n",
"L 2766 531 \n",
"Q 2569 213 2275 61 \n",
"Q 1981 -91 1556 -91 \n",
"Q 1019 -91 701 211 \n",
"Q 384 513 384 1019 \n",
"Q 384 1609 779 1909 \n",
"Q 1175 2209 1959 2209 \n",
"L 2766 2209 \n",
"L 2766 2266 \n",
"Q 2766 2663 2505 2880 \n",
"Q 2244 3097 1772 3097 \n",
"Q 1472 3097 1187 3025 \n",
"Q 903 2953 641 2809 \n",
"L 641 3341 \n",
"Q 956 3463 1253 3523 \n",
"Q 1550 3584 1831 3584 \n",
"Q 2591 3584 2966 3190 \n",
"Q 3341 2797 3341 1997 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-65\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"61.523438\"/>\n",
" <use xlink:href=\"#DejaVuSans-70\" x=\"158.935547\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"222.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-72\" x=\"250.195312\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"291.308594\"/>\n",
" <use xlink:href=\"#DejaVuSans-63\" x=\"319.091797\"/>\n",
" <use xlink:href=\"#DejaVuSans-61\" x=\"374.072266\"/>\n",
" <use xlink:href=\"#DejaVuSans-6c\" x=\"435.351562\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"463.134766\"/>\n",
" <use xlink:href=\"#DejaVuSans-72\" x=\"494.921875\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"536.035156\"/>\n",
" <use xlink:href=\"#DejaVuSans-73\" x=\"563.818359\"/>\n",
" <use xlink:href=\"#DejaVuSans-6b\" x=\"615.917969\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"patch_8\">\n",
" <path d=\"M 174.551475 71.720249 \n",
"Q 174.76892 96.361025 174.976499 119.883811 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" <path d=\"M 176.941124 115.866318 \n",
"L 174.976499 119.883811 \n",
"L 172.94128 115.901615 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" </g>\n",
" <g id=\"text_15\">\n",
" <!-- min of risk -->\n",
" <g transform=\"translate(148.103189 65.643743) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-6d\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"97.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"125.195312\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"188.574219\"/>\n",
" <use xlink:href=\"#DejaVuSans-6f\" x=\"220.361328\"/>\n",
" <use xlink:href=\"#DejaVuSans-66\" x=\"281.542969\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"316.748047\"/>\n",
" <use xlink:href=\"#DejaVuSans-72\" x=\"348.535156\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"389.648438\"/>\n",
" <use xlink:href=\"#DejaVuSans-73\" x=\"417.431641\"/>\n",
" <use xlink:href=\"#DejaVuSans-6b\" x=\"469.53125\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <defs>\n",
" <clipPath id=\"p1b44431415\">\n",
" <rect x=\"58.523438\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
" </clipPath>\n",
" </defs>\n",
"</svg>\n"
],
"text/plain": [
"<Figure size 350x250 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"def annotate(text, xy, xytext): #@save\n",
" d2l.plt.gca().annotate(text, xy=xy, xytext=xytext,\n",
" arrowprops=dict(arrowstyle='->'))\n",
"\n",
"x = torch.arange(0.5, 1.5, 0.01)\n",
"d2l.set_figsize((4.5, 2.5))\n",
"d2l.plot(x, [f(x), g(x)], 'x', 'risk')\n",
"annotate('min of\\nempirical risk', (1.0, -1.2), (0.5, -1.1))\n",
"annotate('min of risk', (1.1, -1.05), (0.95, -0.5))"
]
},
{
"cell_type": "markdown",
"id": "026c7044",
"metadata": {
"origin_pos": 8
},
"source": [
"## Optimization Challenges in Deep Learning\n",
"\n",
"In this chapter, we are going to focus specifically on the performance of optimization algorithms in minimizing the objective function, rather than a\n",
"model's generalization error.\n",
"In :numref:`sec_linear_regression`\n",
"we distinguished between analytical solutions and numerical solutions in\n",
"optimization problems.\n",
"In deep learning, most objective functions are\n",
"complicated and do not have analytical solutions. Instead, we must use numerical\n",
"optimization algorithms.\n",
"The optimization algorithms in this chapter\n",
"all fall into this\n",
"category.\n",
"\n",
"There are many challenges in deep learning optimization. Some of the most vexing ones are local minima, saddle points, and vanishing gradients.\n",
"Let's have a look at them.\n",
"\n",
"\n",
"### Local Minima\n",
"\n",
"For any objective function $f(x)$,\n",
"if the value of $f(x)$ at $x$ is smaller than the values of $f(x)$ at any other points in the vicinity of $x$, then $f(x)$ could be a local minimum.\n",
"If the value of $f(x)$ at $x$ is the minimum of the objective function over the entire domain,\n",
"then $f(x)$ is the global minimum.\n",
"\n",
"For example, given the function\n",
"\n",
"$$f(x) = x \\cdot \\textrm{cos}(\\pi x) \\textrm{ for } -1.0 \\leq x \\leq 2.0,$$\n",
"\n",
"we can approximate the local minimum and global minimum of this function.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "b436be00",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:42:01.150879Z",
"iopub.status.busy": "2023-08-18T19:42:01.150295Z",
"iopub.status.idle": "2023-08-18T19:42:01.430105Z",
"shell.execute_reply": "2023-08-18T19:42:01.429024Z"
},
"origin_pos": 9,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.120313pt\" height=\"183.35625pt\" viewBox=\"0 0 245.120313 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
" <metadata>\n",
" <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
" <cc:Work>\n",
" <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
" <dc:date>2023-08-18T19:42:01.374477</dc:date>\n",
" <dc:format>image/svg+xml</dc:format>\n",
" <dc:creator>\n",
" <cc:Agent>\n",
" <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
" </cc:Agent>\n",
" </dc:creator>\n",
" </cc:Work>\n",
" </rdf:RDF>\n",
" </metadata>\n",
" <defs>\n",
" <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
" </defs>\n",
" <g id=\"figure_1\">\n",
" <g id=\"patch_1\">\n",
" <path d=\"M 0 183.35625 \n",
"L 245.120313 183.35625 \n",
"L 245.120313 0 \n",
"L 0 0 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"axes_1\">\n",
" <g id=\"patch_2\">\n",
" <path d=\"M 42.620312 145.8 \n",
"L 237.920313 145.8 \n",
"L 237.920313 7.2 \n",
"L 42.620312 7.2 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"matplotlib.axis_1\">\n",
" <g id=\"xtick_1\">\n",
" <g id=\"line2d_1\">\n",
" <path d=\"M 51.497585 145.8 \n",
"L 51.497585 7.2 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_2\">\n",
" <defs>\n",
" <path id=\"m41f7a55300\" d=\"M 0 0 \n",
"L 0 3.5 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#m41f7a55300\" x=\"51.497585\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_1\">\n",
" <!-- −1 -->\n",
" <g transform=\"translate(44.126491 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
"L 4684 2272 \n",
"L 4684 1741 \n",
"L 678 1741 \n",
"L 678 2272 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
"L 1825 531 \n",
"L 1825 4091 \n",
"L 703 3866 \n",
"L 703 4441 \n",
"L 1819 4666 \n",
"L 2450 4666 \n",
"L 2450 531 \n",
"L 3481 531 \n",
"L 3481 0 \n",
"L 794 0 \n",
"L 794 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_2\">\n",
" <g id=\"line2d_3\">\n",
" <path d=\"M 110.877336 145.8 \n",
"L 110.877336 7.2 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_4\">\n",
" <g>\n",
" <use xlink:href=\"#m41f7a55300\" x=\"110.877336\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_2\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(107.696086 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
"Q 1547 4250 1301 3770 \n",
"Q 1056 3291 1056 2328 \n",
"Q 1056 1369 1301 889 \n",
"Q 1547 409 2034 409 \n",
"Q 2525 409 2770 889 \n",
"Q 3016 1369 3016 2328 \n",
"Q 3016 3291 2770 3770 \n",
"Q 2525 4250 2034 4250 \n",
"z\n",
"M 2034 4750 \n",
"Q 2819 4750 3233 4129 \n",
"Q 3647 3509 3647 2328 \n",
"Q 3647 1150 3233 529 \n",
"Q 2819 -91 2034 -91 \n",
"Q 1250 -91 836 529 \n",
"Q 422 1150 422 2328 \n",
"Q 422 3509 836 4129 \n",
"Q 1250 4750 2034 4750 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_3\">\n",
" <g id=\"line2d_5\">\n",
" <path d=\"M 170.257086 145.8 \n",
"L 170.257086 7.2 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_6\">\n",
" <g>\n",
" <use xlink:href=\"#m41f7a55300\" x=\"170.257086\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_3\">\n",
" <!-- 1 -->\n",
" <g transform=\"translate(167.075836 160.398438) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_4\">\n",
" <g id=\"line2d_7\">\n",
" <path d=\"M 229.636837 145.8 \n",
"L 229.636837 7.2 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_8\">\n",
" <g>\n",
" <use xlink:href=\"#m41f7a55300\" x=\"229.636837\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_4\">\n",
" <!-- 2 -->\n",
" <g transform=\"translate(226.455587 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
"L 3431 531 \n",
"L 3431 0 \n",
"L 469 0 \n",
"L 469 531 \n",
"Q 828 903 1448 1529 \n",
"Q 2069 2156 2228 2338 \n",
"Q 2531 2678 2651 2914 \n",
"Q 2772 3150 2772 3378 \n",
"Q 2772 3750 2511 3984 \n",
"Q 2250 4219 1831 4219 \n",
"Q 1534 4219 1204 4116 \n",
"Q 875 4013 500 3803 \n",
"L 500 4441 \n",
"Q 881 4594 1212 4672 \n",
"Q 1544 4750 1819 4750 \n",
"Q 2544 4750 2975 4387 \n",
"Q 3406 4025 3406 3419 \n",
"Q 3406 3131 3298 2873 \n",
"Q 3191 2616 2906 2266 \n",
"Q 2828 2175 2409 1742 \n",
"Q 1991 1309 1228 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-32\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_5\">\n",
" <!-- x -->\n",
" <g transform=\"translate(137.310937 174.076563) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
"L 2247 1797 \n",
"L 3578 0 \n",
"L 2900 0 \n",
"L 1881 1375 \n",
"L 863 0 \n",
"L 184 0 \n",
"L 1544 1831 \n",
"L 300 3500 \n",
"L 978 3500 \n",
"L 1906 2253 \n",
"L 2834 3500 \n",
"L 3513 3500 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-78\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"matplotlib.axis_2\">\n",
" <g id=\"ytick_1\">\n",
" <g id=\"line2d_9\">\n",
" <path d=\"M 42.620312 137.560854 \n",
"L 237.920313 137.560854 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_10\">\n",
" <defs>\n",
" <path id=\"mbba31e2d6c\" d=\"M 0 0 \n",
"L -3.5 0 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#mbba31e2d6c\" x=\"42.620312\" y=\"137.560854\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_6\">\n",
" <!-- −1 -->\n",
" <g transform=\"translate(20.878125 141.360073) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_2\">\n",
" <g id=\"line2d_11\">\n",
" <path d=\"M 42.620312 96.055298 \n",
"L 237.920313 96.055298 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_12\">\n",
" <g>\n",
" <use xlink:href=\"#mbba31e2d6c\" x=\"42.620312\" y=\"96.055298\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_7\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(29.257812 99.854516) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_3\">\n",
" <g id=\"line2d_13\">\n",
" <path d=\"M 42.620312 54.549741 \n",
"L 237.920313 54.549741 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_14\">\n",
" <g>\n",
" <use xlink:href=\"#mbba31e2d6c\" x=\"42.620312\" y=\"54.549741\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_8\">\n",
" <!-- 1 -->\n",
" <g transform=\"translate(29.257812 58.34896) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_4\">\n",
" <g id=\"line2d_15\">\n",
" <path d=\"M 42.620312 13.044184 \n",
"L 237.920313 13.044184 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_16\">\n",
" <g>\n",
" <use xlink:href=\"#mbba31e2d6c\" x=\"42.620312\" y=\"13.044184\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_9\">\n",
" <!-- 2 -->\n",
" <g transform=\"translate(29.257812 16.843403) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-32\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_10\">\n",
" <!-- f(x) -->\n",
" <g transform=\"translate(14.798437 85.121094) rotate(-90) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
"L 2375 4384 \n",
"L 1825 4384 \n",
"Q 1516 4384 1395 4259 \n",
"Q 1275 4134 1275 3809 \n",
"L 1275 3500 \n",
"L 2222 3500 \n",
"L 2222 3053 \n",
"L 1275 3053 \n",
"L 1275 0 \n",
"L 697 0 \n",
"L 697 3053 \n",
"L 147 3053 \n",
"L 147 3500 \n",
"L 697 3500 \n",
"L 697 3744 \n",
"Q 697 4328 969 4595 \n",
"Q 1241 4863 1831 4863 \n",
"L 2375 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
"Q 1566 4138 1362 3434 \n",
"Q 1159 2731 1159 2009 \n",
"Q 1159 1288 1364 580 \n",
"Q 1569 -128 1984 -844 \n",
"L 1484 -844 \n",
"Q 1016 -109 783 600 \n",
"Q 550 1309 550 2009 \n",
"Q 550 2706 781 3412 \n",
"Q 1013 4119 1484 4856 \n",
"L 1984 4856 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
"L 1013 4856 \n",
"Q 1481 4119 1714 3412 \n",
"Q 1947 2706 1947 2009 \n",
"Q 1947 1309 1714 600 \n",
"Q 1481 -109 1013 -844 \n",
"L 513 -844 \n",
"Q 928 -128 1133 580 \n",
"Q 1338 1288 1338 2009 \n",
"Q 1338 2731 1133 3434 \n",
"Q 928 4138 513 4856 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-66\"/>\n",
" <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
" <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
" <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"line2d_17\">\n",
" <path d=\"M 51.497585 54.549741 \n",
"L 53.278976 55.973583 \n",
"L 55.06037 57.731143 \n",
"L 56.841761 59.784948 \n",
"L 59.216953 62.915327 \n",
"L 62.185941 67.318969 \n",
"L 65.748726 73.060548 \n",
"L 73.468093 85.670474 \n",
"L 76.437078 90.068527 \n",
"L 78.812269 93.246202 \n",
"L 81.187462 96.055299 \n",
"L 83.562652 98.448229 \n",
"L 85.344043 99.948584 \n",
"L 87.125435 101.185666 \n",
"L 88.906828 102.15432 \n",
"L 90.68822 102.853751 \n",
"L 92.469613 103.287469 \n",
"L 94.251006 103.463156 \n",
"L 96.032398 103.392513 \n",
"L 97.813791 103.091025 \n",
"L 99.595183 102.577701 \n",
"L 101.970373 101.602556 \n",
"L 104.345563 100.350994 \n",
"L 107.314551 98.50152 \n",
"L 113.252526 94.408167 \n",
"L 116.815311 92.107885 \n",
"L 119.190501 90.797549 \n",
"L 121.56569 89.747324 \n",
"L 123.347084 89.168175 \n",
"L 125.128476 88.793798 \n",
"L 126.909869 88.644307 \n",
"L 128.691262 88.736392 \n",
"L 130.472654 89.083043 \n",
"L 132.254047 89.693305 \n",
"L 134.035439 90.572093 \n",
"L 135.816832 91.720053 \n",
"L 137.598223 93.133492 \n",
"L 139.379615 94.804343 \n",
"L 141.754805 97.410497 \n",
"L 144.129996 100.410623 \n",
"L 146.505187 103.750855 \n",
"L 149.474176 108.303335 \n",
"L 154.224555 116.092429 \n",
"L 159.568731 124.791627 \n",
"L 162.537715 129.195261 \n",
"L 164.912907 132.325645 \n",
"L 166.694301 134.379452 \n",
"L 168.475695 136.137012 \n",
"L 170.257086 137.560854 \n",
"L 172.038477 138.61629 \n",
"L 173.226071 139.099581 \n",
"L 174.413672 139.396684 \n",
"L 175.601266 139.5 \n",
"L 176.78886 139.40277 \n",
"L 177.976453 139.099165 \n",
"L 179.164054 138.584272 \n",
"L 180.351641 137.854217 \n",
"L 181.539235 136.906143 \n",
"L 182.726829 135.738243 \n",
"L 184.508227 133.57304 \n",
"L 186.289618 130.914408 \n",
"L 188.071009 127.770565 \n",
"L 189.852406 124.155588 \n",
"L 191.633797 120.089484 \n",
"L 194.008985 114.011594 \n",
"L 196.384173 107.254707 \n",
"L 199.353165 97.997842 \n",
"L 202.915953 85.991267 \n",
"L 210.041516 60.771447 \n",
"L 214.198102 46.617515 \n",
"L 217.167094 37.350755 \n",
"L 219.542275 30.677498 \n",
"L 221.323665 26.202348 \n",
"L 223.105063 22.247434 \n",
"L 224.886454 18.868262 \n",
"L 226.074055 16.960752 \n",
"L 227.261649 15.345879 \n",
"L 228.449243 14.036461 \n",
"L 229.04304 13.5 \n",
"L 229.04304 13.5 \n",
"\" clip-path=\"url(#pea7382aa85)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_3\">\n",
" <path d=\"M 42.620312 145.8 \n",
"L 42.620312 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_4\">\n",
" <path d=\"M 237.920313 145.8 \n",
"L 237.920313 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_5\">\n",
" <path d=\"M 42.620312 145.8 \n",
"L 237.920313 145.8 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_6\">\n",
" <path d=\"M 42.620312 7.2 \n",
"L 237.920313 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_7\">\n",
" <path d=\"M 99.3292 126.056399 \n",
"Q 96.50034 117.196292 94.011534 109.401249 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" <path d=\"M 93.322904 113.820049 \n",
"L 94.011534 109.401249 \n",
"L 97.133396 112.603433 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" </g>\n",
" <g id=\"text_11\">\n",
" <!-- local minimum -->\n",
" <g transform=\"translate(65.154928 137.560854) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
"Q 1497 3097 1228 2736 \n",
"Q 959 2375 959 1747 \n",
"Q 959 1119 1226 758 \n",
"Q 1494 397 1959 397 \n",
"Q 2419 397 2687 759 \n",
"Q 2956 1122 2956 1747 \n",
"Q 2956 2369 2687 2733 \n",
"Q 2419 3097 1959 3097 \n",
"z\n",
"M 1959 3584 \n",
"Q 2709 3584 3137 3096 \n",
"Q 3566 2609 3566 1747 \n",
"Q 3566 888 3137 398 \n",
"Q 2709 -91 1959 -91 \n",
"Q 1206 -91 779 398 \n",
"Q 353 888 353 1747 \n",
"Q 353 2609 779 3096 \n",
"Q 1206 3584 1959 3584 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
"L 3122 2828 \n",
"Q 2878 2963 2633 3030 \n",
"Q 2388 3097 2138 3097 \n",
"Q 1578 3097 1268 2742 \n",
"Q 959 2388 959 1747 \n",
"Q 959 1106 1268 751 \n",
"Q 1578 397 2138 397 \n",
"Q 2388 397 2633 464 \n",
"Q 2878 531 3122 666 \n",
"L 3122 134 \n",
"Q 2881 22 2623 -34 \n",
"Q 2366 -91 2075 -91 \n",
"Q 1284 -91 818 406 \n",
"Q 353 903 353 1747 \n",
"Q 353 2603 823 3093 \n",
"Q 1294 3584 2113 3584 \n",
"Q 2378 3584 2631 3529 \n",
"Q 2884 3475 3122 3366 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
"Q 1497 1759 1228 1600 \n",
"Q 959 1441 959 1056 \n",
"Q 959 750 1161 570 \n",
"Q 1363 391 1709 391 \n",
"Q 2188 391 2477 730 \n",
"Q 2766 1069 2766 1631 \n",
"L 2766 1759 \n",
"L 2194 1759 \n",
"z\n",
"M 3341 1997 \n",
"L 3341 0 \n",
"L 2766 0 \n",
"L 2766 531 \n",
"Q 2569 213 2275 61 \n",
"Q 1981 -91 1556 -91 \n",
"Q 1019 -91 701 211 \n",
"Q 384 513 384 1019 \n",
"Q 384 1609 779 1909 \n",
"Q 1175 2209 1959 2209 \n",
"L 2766 2209 \n",
"L 2766 2266 \n",
"Q 2766 2663 2505 2880 \n",
"Q 2244 3097 1772 3097 \n",
"Q 1472 3097 1187 3025 \n",
"Q 903 2953 641 2809 \n",
"L 641 3341 \n",
"Q 956 3463 1253 3523 \n",
"Q 1550 3584 1831 3584 \n",
"Q 2591 3584 2966 3190 \n",
"Q 3341 2797 3341 1997 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
"Q 3544 3216 3844 3400 \n",
"Q 4144 3584 4550 3584 \n",
"Q 5097 3584 5394 3201 \n",
"Q 5691 2819 5691 2113 \n",
"L 5691 0 \n",
"L 5113 0 \n",
"L 5113 2094 \n",
"Q 5113 2597 4934 2840 \n",
"Q 4756 3084 4391 3084 \n",
"Q 3944 3084 3684 2787 \n",
"Q 3425 2491 3425 1978 \n",
"L 3425 0 \n",
"L 2847 0 \n",
"L 2847 2094 \n",
"Q 2847 2600 2669 2842 \n",
"Q 2491 3084 2119 3084 \n",
"Q 1678 3084 1418 2786 \n",
"Q 1159 2488 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1356 3278 1631 3431 \n",
"Q 1906 3584 2284 3584 \n",
"Q 2666 3584 2933 3390 \n",
"Q 3200 3197 3328 2828 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
"L 1178 3500 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 3500 \n",
"z\n",
"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 4134 \n",
"L 603 4134 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
"L 3513 0 \n",
"L 2938 0 \n",
"L 2938 2094 \n",
"Q 2938 2591 2744 2837 \n",
"Q 2550 3084 2163 3084 \n",
"Q 1697 3084 1428 2787 \n",
"Q 1159 2491 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1366 3272 1645 3428 \n",
"Q 1925 3584 2291 3584 \n",
"Q 2894 3584 3203 3211 \n",
"Q 3513 2838 3513 2113 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
"L 544 3500 \n",
"L 1119 3500 \n",
"L 1119 1403 \n",
"Q 1119 906 1312 657 \n",
"Q 1506 409 1894 409 \n",
"Q 2359 409 2629 706 \n",
"Q 2900 1003 2900 1516 \n",
"L 2900 3500 \n",
"L 3475 3500 \n",
"L 3475 0 \n",
"L 2900 0 \n",
"L 2900 538 \n",
"Q 2691 219 2414 64 \n",
"Q 2138 -91 1772 -91 \n",
"Q 1169 -91 856 284 \n",
"Q 544 659 544 1381 \n",
"z\n",
"M 1991 3584 \n",
"L 1991 3584 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-6c\"/>\n",
" <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n",
" <use xlink:href=\"#DejaVuSans-63\" x=\"88.964844\"/>\n",
" <use xlink:href=\"#DejaVuSans-61\" x=\"143.945312\"/>\n",
" <use xlink:href=\"#DejaVuSans-6c\" x=\"205.224609\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"233.007812\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"264.794922\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"362.207031\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"389.990234\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"453.369141\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"481.152344\"/>\n",
" <use xlink:href=\"#DejaVuSans-75\" x=\"578.564453\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"641.943359\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"patch_8\">\n",
" <path d=\"M 185.798253 68.913244 \n",
"Q 181.139381 101.210008 176.640134 132.400192 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" <path d=\"M 179.190741 128.72672 \n",
"L 176.640134 132.400192 \n",
"L 175.23172 128.155623 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" </g>\n",
" <g id=\"text_12\">\n",
" <!-- global minimum -->\n",
" <g transform=\"translate(146.505186 62.850852) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
"Q 2906 2416 2648 2759 \n",
"Q 2391 3103 1925 3103 \n",
"Q 1463 3103 1205 2759 \n",
"Q 947 2416 947 1791 \n",
"Q 947 1169 1205 825 \n",
"Q 1463 481 1925 481 \n",
"Q 2391 481 2648 825 \n",
"Q 2906 1169 2906 1791 \n",
"z\n",
"M 3481 434 \n",
"Q 3481 -459 3084 -895 \n",
"Q 2688 -1331 1869 -1331 \n",
"Q 1566 -1331 1297 -1286 \n",
"Q 1028 -1241 775 -1147 \n",
"L 775 -588 \n",
"Q 1028 -725 1275 -790 \n",
"Q 1522 -856 1778 -856 \n",
"Q 2344 -856 2625 -561 \n",
"Q 2906 -266 2906 331 \n",
"L 2906 616 \n",
"Q 2728 306 2450 153 \n",
"Q 2172 0 1784 0 \n",
"Q 1141 0 747 490 \n",
"Q 353 981 353 1791 \n",
"Q 353 2603 747 3093 \n",
"Q 1141 3584 1784 3584 \n",
"Q 2172 3584 2450 3431 \n",
"Q 2728 3278 2906 2969 \n",
"L 2906 3500 \n",
"L 3481 3500 \n",
"L 3481 434 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
"Q 3116 2381 2855 2742 \n",
"Q 2594 3103 2138 3103 \n",
"Q 1681 3103 1420 2742 \n",
"Q 1159 2381 1159 1747 \n",
"Q 1159 1113 1420 752 \n",
"Q 1681 391 2138 391 \n",
"Q 2594 391 2855 752 \n",
"Q 3116 1113 3116 1747 \n",
"z\n",
"M 1159 2969 \n",
"Q 1341 3281 1617 3432 \n",
"Q 1894 3584 2278 3584 \n",
"Q 2916 3584 3314 3078 \n",
"Q 3713 2572 3713 1747 \n",
"Q 3713 922 3314 415 \n",
"Q 2916 -91 2278 -91 \n",
"Q 1894 -91 1617 61 \n",
"Q 1341 213 1159 525 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 4863 \n",
"L 1159 4863 \n",
"L 1159 2969 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-67\"/>\n",
" <use xlink:href=\"#DejaVuSans-6c\" x=\"63.476562\"/>\n",
" <use xlink:href=\"#DejaVuSans-6f\" x=\"91.259766\"/>\n",
" <use xlink:href=\"#DejaVuSans-62\" x=\"152.441406\"/>\n",
" <use xlink:href=\"#DejaVuSans-61\" x=\"215.917969\"/>\n",
" <use xlink:href=\"#DejaVuSans-6c\" x=\"277.197266\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"304.980469\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"336.767578\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"434.179688\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"461.962891\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"525.341797\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"553.125\"/>\n",
" <use xlink:href=\"#DejaVuSans-75\" x=\"650.537109\"/>\n",
" <use xlink:href=\"#DejaVuSans-6d\" x=\"713.916016\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <defs>\n",
" <clipPath id=\"pea7382aa85\">\n",
" <rect x=\"42.620312\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
" </clipPath>\n",
" </defs>\n",
"</svg>\n"
],
"text/plain": [
"<Figure size 350x250 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x = torch.arange(-1.0, 2.0, 0.01)\n",
"d2l.plot(x, [f(x), ], 'x', 'f(x)')\n",
"annotate('local minimum', (-0.3, -0.25), (-0.77, -1.0))\n",
"annotate('global minimum', (1.1, -0.95), (0.6, 0.8))"
]
},
{
"cell_type": "markdown",
"id": "f571535c",
"metadata": {
"origin_pos": 10
},
"source": [
"The objective function of deep learning models usually has many local optima.\n",
"When the numerical solution of an optimization problem is near the local optimum, the numerical solution obtained by the final iteration may only minimize the objective function *locally*, rather than *globally*, as the gradient of the objective function's solutions approaches or becomes zero.\n",
"Only some degree of noise might knock the parameter out of the local minimum. In fact, this is one of the beneficial properties of\n",
"minibatch stochastic gradient descent where the natural variation of gradients over minibatches is able to dislodge the parameters from local minima.\n",
"\n",
"\n",
"### Saddle Points\n",
"\n",
"Besides local minima, saddle points are another reason for gradients to vanish. A *saddle point* is any location where all gradients of a function vanish but which is neither a global nor a local minimum.\n",
"Consider the function $f(x) = x^3$. Its first and second derivative vanish for $x=0$. Optimization might stall at this point, even though it is not a minimum.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "60015b8b",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:42:01.433863Z",
"iopub.status.busy": "2023-08-18T19:42:01.433582Z",
"iopub.status.idle": "2023-08-18T19:42:01.688916Z",
"shell.execute_reply": "2023-08-18T19:42:01.687512Z"
},
"origin_pos": 11,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"245.120313pt\" height=\"183.35625pt\" viewBox=\"0 0 245.120313 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
" <metadata>\n",
" <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
" <cc:Work>\n",
" <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
" <dc:date>2023-08-18T19:42:01.644563</dc:date>\n",
" <dc:format>image/svg+xml</dc:format>\n",
" <dc:creator>\n",
" <cc:Agent>\n",
" <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
" </cc:Agent>\n",
" </dc:creator>\n",
" </cc:Work>\n",
" </rdf:RDF>\n",
" </metadata>\n",
" <defs>\n",
" <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
" </defs>\n",
" <g id=\"figure_1\">\n",
" <g id=\"patch_1\">\n",
" <path d=\"M 0 183.35625 \n",
"L 245.120313 183.35625 \n",
"L 245.120313 0 \n",
"L 0 0 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"axes_1\">\n",
" <g id=\"patch_2\">\n",
" <path d=\"M 42.620312 145.8 \n",
"L 237.920313 145.8 \n",
"L 237.920313 7.2 \n",
"L 42.620312 7.2 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"matplotlib.axis_1\">\n",
" <g id=\"xtick_1\">\n",
" <g id=\"line2d_1\">\n",
" <path d=\"M 51.497585 145.8 \n",
"L 51.497585 7.2 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_2\">\n",
" <defs>\n",
" <path id=\"m8175d2a92f\" d=\"M 0 0 \n",
"L 0 3.5 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#m8175d2a92f\" x=\"51.497585\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_1\">\n",
" <!-- −2 -->\n",
" <g transform=\"translate(44.126491 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
"L 4684 2272 \n",
"L 4684 1741 \n",
"L 678 1741 \n",
"L 678 2272 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
"L 3431 531 \n",
"L 3431 0 \n",
"L 469 0 \n",
"L 469 531 \n",
"Q 828 903 1448 1529 \n",
"Q 2069 2156 2228 2338 \n",
"Q 2531 2678 2651 2914 \n",
"Q 2772 3150 2772 3378 \n",
"Q 2772 3750 2511 3984 \n",
"Q 2250 4219 1831 4219 \n",
"Q 1534 4219 1204 4116 \n",
"Q 875 4013 500 3803 \n",
"L 500 4441 \n",
"Q 881 4594 1212 4672 \n",
"Q 1544 4750 1819 4750 \n",
"Q 2544 4750 2975 4387 \n",
"Q 3406 4025 3406 3419 \n",
"Q 3406 3131 3298 2873 \n",
"Q 3191 2616 2906 2266 \n",
"Q 2828 2175 2409 1742 \n",
"Q 1991 1309 1228 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_2\">\n",
" <g id=\"line2d_3\">\n",
" <path d=\"M 95.995193 145.8 \n",
"L 95.995193 7.2 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_4\">\n",
" <g>\n",
" <use xlink:href=\"#m8175d2a92f\" x=\"95.995193\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_2\">\n",
" <!-- −1 -->\n",
" <g transform=\"translate(88.624099 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
"L 1825 531 \n",
"L 1825 4091 \n",
"L 703 3866 \n",
"L 703 4441 \n",
"L 1819 4666 \n",
"L 2450 4666 \n",
"L 2450 531 \n",
"L 3481 531 \n",
"L 3481 0 \n",
"L 794 0 \n",
"L 794 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_3\">\n",
" <g id=\"line2d_5\">\n",
" <path d=\"M 140.4928 145.8 \n",
"L 140.4928 7.2 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_6\">\n",
" <g>\n",
" <use xlink:href=\"#m8175d2a92f\" x=\"140.4928\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_3\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(137.31155 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
"Q 1547 4250 1301 3770 \n",
"Q 1056 3291 1056 2328 \n",
"Q 1056 1369 1301 889 \n",
"Q 1547 409 2034 409 \n",
"Q 2525 409 2770 889 \n",
"Q 3016 1369 3016 2328 \n",
"Q 3016 3291 2770 3770 \n",
"Q 2525 4250 2034 4250 \n",
"z\n",
"M 2034 4750 \n",
"Q 2819 4750 3233 4129 \n",
"Q 3647 3509 3647 2328 \n",
"Q 3647 1150 3233 529 \n",
"Q 2819 -91 2034 -91 \n",
"Q 1250 -91 836 529 \n",
"Q 422 1150 422 2328 \n",
"Q 422 3509 836 4129 \n",
"Q 1250 4750 2034 4750 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_4\">\n",
" <g id=\"line2d_7\">\n",
" <path d=\"M 184.990408 145.8 \n",
"L 184.990408 7.2 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_8\">\n",
" <g>\n",
" <use xlink:href=\"#m8175d2a92f\" x=\"184.990408\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_4\">\n",
" <!-- 1 -->\n",
" <g transform=\"translate(181.809158 160.398438) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_5\">\n",
" <g id=\"line2d_9\">\n",
" <path d=\"M 229.488015 145.8 \n",
"L 229.488015 7.2 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_10\">\n",
" <g>\n",
" <use xlink:href=\"#m8175d2a92f\" x=\"229.488015\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_5\">\n",
" <!-- 2 -->\n",
" <g transform=\"translate(226.306765 160.398438) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-32\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_6\">\n",
" <!-- x -->\n",
" <g transform=\"translate(137.310937 174.076563) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
"L 2247 1797 \n",
"L 3578 0 \n",
"L 2900 0 \n",
"L 1881 1375 \n",
"L 863 0 \n",
"L 184 0 \n",
"L 1544 1831 \n",
"L 300 3500 \n",
"L 978 3500 \n",
"L 1906 2253 \n",
"L 2834 3500 \n",
"L 3513 3500 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-78\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"matplotlib.axis_2\">\n",
" <g id=\"ytick_1\">\n",
" <g id=\"line2d_11\">\n",
" <path d=\"M 42.620312 115.697371 \n",
"L 237.920313 115.697371 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_12\">\n",
" <defs>\n",
" <path id=\"m3b5c865722\" d=\"M 0 0 \n",
"L -3.5 0 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#m3b5c865722\" x=\"42.620312\" y=\"115.697371\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_7\">\n",
" <!-- −5 -->\n",
" <g transform=\"translate(20.878125 119.49659) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
"L 3169 4666 \n",
"L 3169 4134 \n",
"L 1269 4134 \n",
"L 1269 2991 \n",
"Q 1406 3038 1543 3061 \n",
"Q 1681 3084 1819 3084 \n",
"Q 2600 3084 3056 2656 \n",
"Q 3513 2228 3513 1497 \n",
"Q 3513 744 3044 326 \n",
"Q 2575 -91 1722 -91 \n",
"Q 1428 -91 1123 -41 \n",
"Q 819 9 494 109 \n",
"L 494 744 \n",
"Q 775 591 1075 516 \n",
"Q 1375 441 1709 441 \n",
"Q 2250 441 2565 725 \n",
"Q 2881 1009 2881 1497 \n",
"Q 2881 1984 2565 2268 \n",
"Q 2250 2553 1709 2553 \n",
"Q 1456 2553 1204 2497 \n",
"Q 953 2441 691 2322 \n",
"L 691 4666 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_2\">\n",
" <g id=\"line2d_13\">\n",
" <path d=\"M 42.620312 76.026324 \n",
"L 237.920313 76.026324 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_14\">\n",
" <g>\n",
" <use xlink:href=\"#m3b5c865722\" x=\"42.620312\" y=\"76.026324\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_8\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(29.257812 79.825543) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_3\">\n",
" <g id=\"line2d_15\">\n",
" <path d=\"M 42.620312 36.355276 \n",
"L 237.920313 36.355276 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_16\">\n",
" <g>\n",
" <use xlink:href=\"#m3b5c865722\" x=\"42.620312\" y=\"36.355276\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_9\">\n",
" <!-- 5 -->\n",
" <g transform=\"translate(29.257812 40.154495) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-35\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_10\">\n",
" <!-- f(x) -->\n",
" <g transform=\"translate(14.798437 85.121094) rotate(-90) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
"L 2375 4384 \n",
"L 1825 4384 \n",
"Q 1516 4384 1395 4259 \n",
"Q 1275 4134 1275 3809 \n",
"L 1275 3500 \n",
"L 2222 3500 \n",
"L 2222 3053 \n",
"L 1275 3053 \n",
"L 1275 0 \n",
"L 697 0 \n",
"L 697 3053 \n",
"L 147 3053 \n",
"L 147 3500 \n",
"L 697 3500 \n",
"L 697 3744 \n",
"Q 697 4328 969 4595 \n",
"Q 1241 4863 1831 4863 \n",
"L 2375 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
"Q 1566 4138 1362 3434 \n",
"Q 1159 2731 1159 2009 \n",
"Q 1159 1288 1364 580 \n",
"Q 1569 -128 1984 -844 \n",
"L 1484 -844 \n",
"Q 1016 -109 783 600 \n",
"Q 550 1309 550 2009 \n",
"Q 550 2706 781 3412 \n",
"Q 1013 4119 1484 4856 \n",
"L 1984 4856 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
"L 1013 4856 \n",
"Q 1481 4119 1714 3412 \n",
"Q 1947 2706 1947 2009 \n",
"Q 1947 1309 1714 600 \n",
"Q 1481 -109 1013 -844 \n",
"L 513 -844 \n",
"Q 928 -128 1133 580 \n",
"Q 1338 1288 1338 2009 \n",
"Q 1338 2731 1133 3434 \n",
"Q 928 4138 513 4856 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-66\"/>\n",
" <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
" <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
" <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"line2d_17\">\n",
" <path d=\"M 51.497585 139.5 \n",
"L 54.61242 133.065804 \n",
"L 57.72725 127.081819 \n",
"L 60.842079 121.531714 \n",
"L 63.956914 116.399137 \n",
"L 67.071749 111.667781 \n",
"L 70.186578 107.321324 \n",
"L 73.301413 103.343416 \n",
"L 76.416243 99.717747 \n",
"L 79.531078 96.427977 \n",
"L 82.645907 93.457784 \n",
"L 85.760742 90.790833 \n",
"L 88.875572 88.410803 \n",
"L 91.990407 86.301357 \n",
"L 95.105241 84.44617 \n",
"L 98.220074 82.828916 \n",
"L 101.779882 81.251025 \n",
"L 105.339689 79.938199 \n",
"L 108.899497 78.866065 \n",
"L 112.459308 78.010249 \n",
"L 116.464091 77.275676 \n",
"L 120.913853 76.702192 \n",
"L 125.808589 76.311456 \n",
"L 131.593279 76.089797 \n",
"L 140.047824 76.026332 \n",
"L 151.172226 75.916641 \n",
"L 156.956915 75.624432 \n",
"L 161.851651 75.148864 \n",
"L 166.301412 74.478265 \n",
"L 170.306195 73.640008 \n",
"L 173.866006 72.679079 \n",
"L 177.425814 71.489646 \n",
"L 180.985622 70.047334 \n",
"L 184.100454 68.55871 \n",
"L 187.215286 66.841486 \n",
"L 190.330121 64.879331 \n",
"L 193.444956 62.655918 \n",
"L 196.559785 60.154922 \n",
"L 199.674615 57.360012 \n",
"L 202.78945 54.254853 \n",
"L 205.904285 50.823123 \n",
"L 209.019114 47.048497 \n",
"L 212.133949 42.914636 \n",
"L 215.248779 38.405226 \n",
"L 218.363614 33.50392 \n",
"L 221.478443 28.194408 \n",
"L 224.593278 22.460342 \n",
"L 227.708108 16.285413 \n",
"L 229.04304 13.5 \n",
"L 229.04304 13.5 \n",
"\" clip-path=\"url(#peeee3efc73)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_3\">\n",
" <path d=\"M 42.620312 145.8 \n",
"L 42.620312 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_4\">\n",
" <path d=\"M 237.920313 145.8 \n",
"L 237.920313 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_5\">\n",
" <path d=\"M 42.620313 145.8 \n",
"L 237.920313 145.8 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_6\">\n",
" <path d=\"M 42.620313 7.2 \n",
"L 237.920313 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_7\">\n",
" <path d=\"M 146.262941 104.137265 \n",
"Q 143.590236 91.851413 141.155193 80.658044 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" <path d=\"M 140.051188 84.991769 \n",
"L 141.155193 80.658044 \n",
"L 143.95977 84.141483 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" </g>\n",
" <g id=\"text_11\">\n",
" <!-- saddle point -->\n",
" <g transform=\"translate(117.354044 115.697371) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
"L 2834 2853 \n",
"Q 2591 2978 2328 3040 \n",
"Q 2066 3103 1784 3103 \n",
"Q 1356 3103 1142 2972 \n",
"Q 928 2841 928 2578 \n",
"Q 928 2378 1081 2264 \n",
"Q 1234 2150 1697 2047 \n",
"L 1894 2003 \n",
"Q 2506 1872 2764 1633 \n",
"Q 3022 1394 3022 966 \n",
"Q 3022 478 2636 193 \n",
"Q 2250 -91 1575 -91 \n",
"Q 1294 -91 989 -36 \n",
"Q 684 19 347 128 \n",
"L 347 722 \n",
"Q 666 556 975 473 \n",
"Q 1284 391 1588 391 \n",
"Q 1994 391 2212 530 \n",
"Q 2431 669 2431 922 \n",
"Q 2431 1156 2273 1281 \n",
"Q 2116 1406 1581 1522 \n",
"L 1381 1569 \n",
"Q 847 1681 609 1914 \n",
"Q 372 2147 372 2553 \n",
"Q 372 3047 722 3315 \n",
"Q 1072 3584 1716 3584 \n",
"Q 2034 3584 2315 3537 \n",
"Q 2597 3491 2834 3397 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
"Q 1497 1759 1228 1600 \n",
"Q 959 1441 959 1056 \n",
"Q 959 750 1161 570 \n",
"Q 1363 391 1709 391 \n",
"Q 2188 391 2477 730 \n",
"Q 2766 1069 2766 1631 \n",
"L 2766 1759 \n",
"L 2194 1759 \n",
"z\n",
"M 3341 1997 \n",
"L 3341 0 \n",
"L 2766 0 \n",
"L 2766 531 \n",
"Q 2569 213 2275 61 \n",
"Q 1981 -91 1556 -91 \n",
"Q 1019 -91 701 211 \n",
"Q 384 513 384 1019 \n",
"Q 384 1609 779 1909 \n",
"Q 1175 2209 1959 2209 \n",
"L 2766 2209 \n",
"L 2766 2266 \n",
"Q 2766 2663 2505 2880 \n",
"Q 2244 3097 1772 3097 \n",
"Q 1472 3097 1187 3025 \n",
"Q 903 2953 641 2809 \n",
"L 641 3341 \n",
"Q 956 3463 1253 3523 \n",
"Q 1550 3584 1831 3584 \n",
"Q 2591 3584 2966 3190 \n",
"Q 3341 2797 3341 1997 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
"L 2906 4863 \n",
"L 3481 4863 \n",
"L 3481 0 \n",
"L 2906 0 \n",
"L 2906 525 \n",
"Q 2725 213 2448 61 \n",
"Q 2172 -91 1784 -91 \n",
"Q 1150 -91 751 415 \n",
"Q 353 922 353 1747 \n",
"Q 353 2572 751 3078 \n",
"Q 1150 3584 1784 3584 \n",
"Q 2172 3584 2448 3432 \n",
"Q 2725 3281 2906 2969 \n",
"z\n",
"M 947 1747 \n",
"Q 947 1113 1208 752 \n",
"Q 1469 391 1925 391 \n",
"Q 2381 391 2643 752 \n",
"Q 2906 1113 2906 1747 \n",
"Q 2906 2381 2643 2742 \n",
"Q 2381 3103 1925 3103 \n",
"Q 1469 3103 1208 2742 \n",
"Q 947 2381 947 1747 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
"L 3597 1613 \n",
"L 953 1613 \n",
"Q 991 1019 1311 708 \n",
"Q 1631 397 2203 397 \n",
"Q 2534 397 2845 478 \n",
"Q 3156 559 3463 722 \n",
"L 3463 178 \n",
"Q 3153 47 2828 -22 \n",
"Q 2503 -91 2169 -91 \n",
"Q 1331 -91 842 396 \n",
"Q 353 884 353 1716 \n",
"Q 353 2575 817 3079 \n",
"Q 1281 3584 2069 3584 \n",
"Q 2775 3584 3186 3129 \n",
"Q 3597 2675 3597 1894 \n",
"z\n",
"M 3022 2063 \n",
"Q 3016 2534 2758 2815 \n",
"Q 2500 3097 2075 3097 \n",
"Q 1594 3097 1305 2825 \n",
"Q 1016 2553 972 2059 \n",
"L 3022 2063 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
"L 1159 -1331 \n",
"L 581 -1331 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2969 \n",
"Q 1341 3281 1617 3432 \n",
"Q 1894 3584 2278 3584 \n",
"Q 2916 3584 3314 3078 \n",
"Q 3713 2572 3713 1747 \n",
"Q 3713 922 3314 415 \n",
"Q 2916 -91 2278 -91 \n",
"Q 1894 -91 1617 61 \n",
"Q 1341 213 1159 525 \n",
"z\n",
"M 3116 1747 \n",
"Q 3116 2381 2855 2742 \n",
"Q 2594 3103 2138 3103 \n",
"Q 1681 3103 1420 2742 \n",
"Q 1159 2381 1159 1747 \n",
"Q 1159 1113 1420 752 \n",
"Q 1681 391 2138 391 \n",
"Q 2594 391 2855 752 \n",
"Q 3116 1113 3116 1747 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
"Q 1497 3097 1228 2736 \n",
"Q 959 2375 959 1747 \n",
"Q 959 1119 1226 758 \n",
"Q 1494 397 1959 397 \n",
"Q 2419 397 2687 759 \n",
"Q 2956 1122 2956 1747 \n",
"Q 2956 2369 2687 2733 \n",
"Q 2419 3097 1959 3097 \n",
"z\n",
"M 1959 3584 \n",
"Q 2709 3584 3137 3096 \n",
"Q 3566 2609 3566 1747 \n",
"Q 3566 888 3137 398 \n",
"Q 2709 -91 1959 -91 \n",
"Q 1206 -91 779 398 \n",
"Q 353 888 353 1747 \n",
"Q 353 2609 779 3096 \n",
"Q 1206 3584 1959 3584 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
"L 1178 3500 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 3500 \n",
"z\n",
"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 4134 \n",
"L 603 4134 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
"L 3513 0 \n",
"L 2938 0 \n",
"L 2938 2094 \n",
"Q 2938 2591 2744 2837 \n",
"Q 2550 3084 2163 3084 \n",
"Q 1697 3084 1428 2787 \n",
"Q 1159 2491 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1366 3272 1645 3428 \n",
"Q 1925 3584 2291 3584 \n",
"Q 2894 3584 3203 3211 \n",
"Q 3513 2838 3513 2113 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
"L 1172 3500 \n",
"L 2356 3500 \n",
"L 2356 3053 \n",
"L 1172 3053 \n",
"L 1172 1153 \n",
"Q 1172 725 1289 603 \n",
"Q 1406 481 1766 481 \n",
"L 2356 481 \n",
"L 2356 0 \n",
"L 1766 0 \n",
"Q 1100 0 847 248 \n",
"Q 594 497 594 1153 \n",
"L 594 3053 \n",
"L 172 3053 \n",
"L 172 3500 \n",
"L 594 3500 \n",
"L 594 4494 \n",
"L 1172 4494 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-73\"/>\n",
" <use xlink:href=\"#DejaVuSans-61\" x=\"52.099609\"/>\n",
" <use xlink:href=\"#DejaVuSans-64\" x=\"113.378906\"/>\n",
" <use xlink:href=\"#DejaVuSans-64\" x=\"176.855469\"/>\n",
" <use xlink:href=\"#DejaVuSans-6c\" x=\"240.332031\"/>\n",
" <use xlink:href=\"#DejaVuSans-65\" x=\"268.115234\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"329.638672\"/>\n",
" <use xlink:href=\"#DejaVuSans-70\" x=\"361.425781\"/>\n",
" <use xlink:href=\"#DejaVuSans-6f\" x=\"424.902344\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"486.083984\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"513.867188\"/>\n",
" <use xlink:href=\"#DejaVuSans-74\" x=\"577.246094\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <defs>\n",
" <clipPath id=\"peeee3efc73\">\n",
" <rect x=\"42.620312\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
" </clipPath>\n",
" </defs>\n",
"</svg>\n"
],
"text/plain": [
"<Figure size 350x250 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x = torch.arange(-2.0, 2.0, 0.01)\n",
"d2l.plot(x, [x**3], 'x', 'f(x)')\n",
"annotate('saddle point', (0, -0.2), (-0.52, -5.0))"
]
},
{
"cell_type": "markdown",
"id": "293d3a4c",
"metadata": {
"origin_pos": 12
},
"source": [
"Saddle points in higher dimensions are even more insidious, as the example below shows. Consider the function $f(x, y) = x^2 - y^2$. It has its saddle point at $(0, 0)$. This is a maximum with respect to $y$ and a minimum with respect to $x$. Moreover, it *looks* like a saddle, which is where this mathematical property got its name.\n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "7f52e2d2",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:42:01.692526Z",
"iopub.status.busy": "2023-08-18T19:42:01.692229Z",
"iopub.status.idle": "2023-08-18T19:42:01.948705Z",
"shell.execute_reply": "2023-08-18T19:42:01.947510Z"
},
"origin_pos": 14,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"169.715408pt\" height=\"168.910565pt\" viewBox=\"0 0 169.715408 168.910565\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
" <metadata>\n",
" <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
" <cc:Work>\n",
" <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
" <dc:date>2023-08-18T19:42:01.910406</dc:date>\n",
" <dc:format>image/svg+xml</dc:format>\n",
" <dc:creator>\n",
" <cc:Agent>\n",
" <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
" </cc:Agent>\n",
" </dc:creator>\n",
" </cc:Work>\n",
" </rdf:RDF>\n",
" </metadata>\n",
" <defs>\n",
" <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
" </defs>\n",
" <g id=\"figure_1\">\n",
" <g id=\"patch_1\">\n",
" <path d=\"M 0 168.910565 \n",
"L 169.715408 168.910565 \n",
"L 169.715408 0 \n",
"L 0 0 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"patch_2\">\n",
" <path d=\"M 7.2 145.8 \n",
"L 145.8 145.8 \n",
"L 145.8 7.2 \n",
"L 7.2 7.2 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"pane3d_1\">\n",
" <g id=\"patch_3\">\n",
" <path d=\"M 17.665256 111.625682 \n",
"L 63.435502 73.260154 \n",
"L 62.799252 17.930204 \n",
"L 14.838663 52.929657 \n",
"\" style=\"fill: #f2f2f2; opacity: 0.5; stroke: #f2f2f2; stroke-linejoin: miter\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"pane3d_2\">\n",
" <g id=\"patch_4\">\n",
" <path d=\"M 63.435502 73.260154 \n",
"L 136.880417 94.607691 \n",
"L 139.501407 37.371964 \n",
"L 62.799252 17.930204 \n",
"\" style=\"fill: #e6e6e6; opacity: 0.5; stroke: #e6e6e6; stroke-linejoin: miter\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"pane3d_3\">\n",
" <g id=\"patch_5\">\n",
" <path d=\"M 17.665256 111.625682 \n",
"L 95.520453 137.053189 \n",
"L 136.880417 94.607691 \n",
"L 63.435502 73.260154 \n",
"\" style=\"fill: #ececec; opacity: 0.5; stroke: #ececec; stroke-linejoin: miter\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"axis3d_1\">\n",
" <g id=\"line2d_1\">\n",
" <path d=\"M 17.665256 111.625682 \n",
"L 95.520453 137.053189 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_1\">\n",
" <!-- x -->\n",
" <g transform=\"translate(36.190346 162.307234) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
"L 2247 1797 \n",
"L 3578 0 \n",
"L 2900 0 \n",
"L 1881 1375 \n",
"L 863 0 \n",
"L 184 0 \n",
"L 1544 1831 \n",
"L 300 3500 \n",
"L 978 3500 \n",
"L 1906 2253 \n",
"L 2834 3500 \n",
"L 3513 3500 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-78\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"Line3DCollection_1\">\n",
" <path d=\"M 22.380575 113.165705 \n",
"L 67.902197 74.558445 \n",
"L 67.454802 19.11025 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" <path d=\"M 55.566605 124.004262 \n",
"L 99.270858 83.676077 \n",
"L 100.18343 27.406003 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" <path d=\"M 90.315667 135.353306 \n",
"L 131.990658 93.186431 \n",
"L 134.384718 36.075033 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" </g>\n",
" <g id=\"xtick_1\">\n",
" <g id=\"line2d_2\">\n",
" <path d=\"M 22.776975 112.829515 \n",
"L 21.586075 113.839527 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_2\">\n",
" <!-- −1 -->\n",
" <g transform=\"translate(7.454595 136.528767) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
"L 4684 2272 \n",
"L 4684 1741 \n",
"L 678 1741 \n",
"L 678 2272 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
"L 1825 531 \n",
"L 1825 4091 \n",
"L 703 3866 \n",
"L 703 4441 \n",
"L 1819 4666 \n",
"L 2450 4666 \n",
"L 2450 531 \n",
"L 3481 531 \n",
"L 3481 0 \n",
"L 794 0 \n",
"L 794 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_2\">\n",
" <g id=\"line2d_3\">\n",
" <path d=\"M 55.94789 123.65243 \n",
"L 54.80236 124.709471 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_3\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(44.898548 147.812111) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
"Q 1547 4250 1301 3770 \n",
"Q 1056 3291 1056 2328 \n",
"Q 1056 1369 1301 889 \n",
"Q 1547 409 2034 409 \n",
"Q 2525 409 2770 889 \n",
"Q 3016 1369 3016 2328 \n",
"Q 3016 3291 2770 3770 \n",
"Q 2525 4250 2034 4250 \n",
"z\n",
"M 2034 4750 \n",
"Q 2819 4750 3233 4129 \n",
"Q 3647 3509 3647 2328 \n",
"Q 3647 1150 3233 529 \n",
"Q 2819 -91 2034 -91 \n",
"Q 1250 -91 836 529 \n",
"Q 422 1150 422 2328 \n",
"Q 422 3509 836 4129 \n",
"Q 1250 4750 2034 4750 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_3\">\n",
" <g id=\"line2d_4\">\n",
" <path d=\"M 90.679959 134.984714 \n",
"L 89.585446 136.092146 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_4\">\n",
" <!-- 1 -->\n",
" <g transform=\"translate(79.730646 159.630877) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"axis3d_2\">\n",
" <g id=\"line2d_5\">\n",
" <path d=\"M 136.880417 94.607691 \n",
"L 95.520453 137.053189 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_5\">\n",
" <!-- y -->\n",
" <g transform=\"translate(141.167797 145.683654) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
"Q 1816 -950 1584 -1140 \n",
"Q 1353 -1331 966 -1331 \n",
"L 506 -1331 \n",
"L 506 -850 \n",
"L 844 -850 \n",
"Q 1081 -850 1212 -737 \n",
"Q 1344 -625 1503 -206 \n",
"L 1606 56 \n",
"L 191 3500 \n",
"L 800 3500 \n",
"L 1894 763 \n",
"L 2988 3500 \n",
"L 3597 3500 \n",
"L 2059 -325 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-79\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"Line3DCollection_2\">\n",
" <path d=\"M 18.155257 50.509358 \n",
"L 20.819213 108.981973 \n",
"L 98.382242 134.116289 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" <path d=\"M 39.885822 34.651381 \n",
"L 41.525154 91.625842 \n",
"L 117.126847 114.879713 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" <path d=\"M 59.991995 19.978812 \n",
"L 60.746858 75.513829 \n",
"L 134.460919 97.090691 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" </g>\n",
" <g id=\"xtick_4\">\n",
" <g id=\"line2d_6\">\n",
" <path d=\"M 97.728616 133.904481 \n",
"L 99.691184 134.540452 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_6\">\n",
" <!-- −1 -->\n",
" <g transform=\"translate(103.898882 155.145417) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_5\">\n",
" <g id=\"line2d_7\">\n",
" <path d=\"M 116.491011 114.68414 \n",
"L 118.400099 115.271345 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_7\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(126.349632 135.361913) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_6\">\n",
" <g id=\"line2d_8\">\n",
" <path d=\"M 133.842096 96.909555 \n",
"L 135.700045 97.453396 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_8\">\n",
" <!-- 1 -->\n",
" <g transform=\"translate(143.234942 117.068702) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"axis3d_3\">\n",
" <g id=\"line2d_9\">\n",
" <path d=\"M 136.880417 94.607691 \n",
"L 139.501407 37.371964 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"Line3DCollection_3\">\n",
" <path d=\"M 136.930632 93.511118 \n",
"L 63.423287 72.197896 \n",
"L 17.611191 110.502989 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" <path d=\"M 138.162202 66.616789 \n",
"L 63.124011 46.172077 \n",
"L 16.284113 82.945357 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" <path d=\"M 139.446688 38.566888 \n",
"L 62.812508 19.082955 \n",
"L 14.897772 54.157093 \n",
"\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8\"/>\n",
" </g>\n",
" <g id=\"xtick_7\">\n",
" <g id=\"line2d_10\">\n",
" <path d=\"M 136.313678 93.332234 \n",
"L 138.166009 93.869311 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_9\">\n",
" <!-- −1 -->\n",
" <g transform=\"translate(147.773221 98.515807) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_8\">\n",
" <g id=\"line2d_11\">\n",
" <path d=\"M 137.531779 66.445026 \n",
"L 139.424579 66.960733 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_10\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(153.568228 71.725004) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_9\">\n",
" <g id=\"line2d_12\">\n",
" <path d=\"M 138.802197 38.403029 \n",
"L 140.737271 38.895014 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"text_11\">\n",
" <!-- 1 -->\n",
" <g transform=\"translate(155.241948 43.789973) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"axes_1\">\n",
" <g id=\"line2d_13\">\n",
" <defs>\n",
" <path id=\"mbee6ed6cb2\" d=\"M -3 3 \n",
"L 3 -3 \n",
"M -3 -3 \n",
"L 3 3 \n",
"\" style=\"stroke: #ff0000\"/>\n",
" </defs>\n",
" <g clip-path=\"url(#p366f7efb7b)\">\n",
" <use xlink:href=\"#mbee6ed6cb2\" x=\"78.372973\" y=\"74.627027\" style=\"fill: #ff0000; stroke: #ff0000\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"Line3DCollection_4\">\n",
" <path d=\"M 24.318635 81.868935 \n",
"L 24.710556 80.407083 \n",
"L 25.103462 78.967565 \n",
"L 25.497331 77.550469 \n",
"L 25.892142 76.155867 \n",
"L 26.287877 74.783829 \n",
"L 26.684512 73.434437 \n",
"L 27.082029 72.107753 \n",
"L 27.480405 70.803848 \n",
"L 27.879622 69.522778 \n",
"L 28.279656 68.264617 \n",
"L 28.680486 67.029418 \n",
"L 29.082094 65.817238 \n",
"L 29.484454 64.628137 \n",
"L 29.887548 63.462163 \n",
"L 30.291355 62.319367 \n",
"L 30.69585 61.199798 \n",
"L 31.101015 60.103501 \n",
"L 31.506827 59.030517 \n",
"L 31.913264 57.980887 \n",
"L 32.320305 56.954651 \n",
"L 32.727929 55.951839 \n",
"L 33.136113 54.972491 \n",
"L 33.544835 54.016632 \n",
"L 33.954075 53.084291 \n",
"L 34.363809 52.175495 \n",
"L 34.774015 51.290266 \n",
"L 35.184674 50.428623 \n",
"L 35.595762 49.590584 \n",
"L 36.007257 48.776167 \n",
"L 36.419138 47.985383 \n",
"L 36.831382 47.218242 \n",
"L 37.243968 46.474752 \n",
"L 37.656874 45.75492 \n",
"L 38.070078 45.058749 \n",
"L 38.483558 44.386238 \n",
"L 38.897292 43.737386 \n",
"L 39.311258 43.112188 \n",
"L 39.725435 42.510639 \n",
"L 40.1398 41.932727 \n",
"L 40.554333 41.378444 \n",
"L 40.96901 40.847773 \n",
"L 41.383811 40.340698 \n",
"L 41.798714 39.857202 \n",
"L 42.213697 39.39726 \n",
"L 42.628739 38.960852 \n",
"L 43.043818 38.547951 \n",
"L 43.458913 38.15853 \n",
"L 43.874002 37.792556 \n",
"L 44.289064 37.449996 \n",
"L 44.704078 37.130815 \n",
"L 45.119023 36.834978 \n",
"L 45.533876 36.562443 \n",
"L 45.948618 36.313167 \n",
"L 46.363228 36.087107 \n",
"L 46.777684 35.884217 \n",
"L 47.191965 35.704448 \n",
"L 47.606052 35.547748 \n",
"L 48.019923 35.414064 \n",
"L 48.433558 35.30334 \n",
"L 48.846937 35.215521 \n",
"L 49.260038 35.150544 \n",
"L 49.672843 35.108352 \n",
"L 50.08533 35.088876 \n",
"L 50.497481 35.092054 \n",
"L 50.909274 35.117817 \n",
"L 51.32069 35.166094 \n",
"L 51.73171 35.236814 \n",
"L 52.142314 35.329904 \n",
"L 52.552483 35.445287 \n",
"L 52.962197 35.582887 \n",
"L 53.371437 35.742621 \n",
"L 53.780184 35.924412 \n",
"L 54.18842 36.128173 \n",
"L 54.596125 36.353821 \n",
"L 55.003281 36.601267 \n",
"L 55.40987 36.870424 \n",
"L 55.815873 37.161201 \n",
"L 56.221272 37.473507 \n",
"L 56.626048 37.807245 \n",
"L 57.030186 38.162323 \n",
"L 57.433665 38.538641 \n",
"L 57.836469 38.936102 \n",
"L 58.238581 39.354604 \n",
"L 58.639982 39.794045 \n",
"L 59.040657 40.254322 \n",
"L 59.440588 40.73533 \n",
"L 59.839758 41.236963 \n",
"L 60.23815 41.759111 \n",
"L 60.63575 42.301667 \n",
"L 61.032539 42.864518 \n",
"L 61.428501 43.447552 \n",
"L 61.823623 44.050657 \n",
"L 62.217886 44.673714 \n",
"L 62.611275 45.316612 \n",
"L 63.003776 45.979229 \n",
"L 63.395374 46.661452 \n",
"L 63.786051 47.363154 \n",
"L 64.175795 48.084218 \n",
"L 64.564591 48.824522 \n",
"L 64.952423 49.583939 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 31.326259 93.962164 \n",
"L 31.717396 92.505925 \n",
"L 32.109392 91.071739 \n",
"L 32.502226 89.659692 \n",
"L 32.895877 88.269855 \n",
"L 33.290328 86.902299 \n",
"L 33.685555 85.557103 \n",
"L 34.081538 84.234331 \n",
"L 34.478257 82.934053 \n",
"L 34.875692 81.656324 \n",
"L 35.27382 80.401219 \n",
"L 35.672621 79.168789 \n",
"L 36.072076 77.959091 \n",
"L 36.47216 76.772186 \n",
"L 36.872855 75.608121 \n",
"L 37.27414 74.466946 \n",
"L 37.675991 73.348714 \n",
"L 38.078388 72.253465 \n",
"L 38.481312 71.18124 \n",
"L 38.884739 70.132086 \n",
"L 39.288648 69.106036 \n",
"L 39.69302 68.103124 \n",
"L 40.09783 67.123389 \n",
"L 40.503058 66.166858 \n",
"L 40.908684 65.233556 \n",
"L 41.314685 64.323514 \n",
"L 41.721039 63.436753 \n",
"L 42.127727 62.573292 \n",
"L 42.534725 61.733151 \n",
"L 42.942012 60.916346 \n",
"L 43.349568 60.12289 \n",
"L 43.75737 59.352793 \n",
"L 44.165398 58.606065 \n",
"L 44.573629 57.882711 \n",
"L 44.982042 57.182736 \n",
"L 45.390616 56.506138 \n",
"L 45.799329 55.852921 \n",
"L 46.208161 55.223076 \n",
"L 46.61709 54.6166 \n",
"L 47.026094 54.033483 \n",
"L 47.435153 53.473716 \n",
"L 47.844244 52.937285 \n",
"L 48.253348 52.424173 \n",
"L 48.662443 51.934363 \n",
"L 49.071508 51.467837 \n",
"L 49.480521 51.024568 \n",
"L 49.889463 50.604535 \n",
"L 50.298312 50.207709 \n",
"L 50.707048 49.83406 \n",
"L 51.115649 49.483556 \n",
"L 51.524096 49.156164 \n",
"L 51.932367 48.851847 \n",
"L 52.340443 48.570567 \n",
"L 52.748302 48.312281 \n",
"L 53.155924 48.076948 \n",
"L 53.56329 47.864523 \n",
"L 53.970379 47.674958 \n",
"L 54.377171 47.508201 \n",
"L 54.783646 47.364204 \n",
"L 55.189785 47.242911 \n",
"L 55.595568 47.144266 \n",
"L 56.000974 47.068209 \n",
"L 56.405986 47.014685 \n",
"L 56.810583 46.983627 \n",
"L 57.214746 46.974972 \n",
"L 57.618456 46.988654 \n",
"L 58.021694 47.024606 \n",
"L 58.42444 47.082756 \n",
"L 58.826678 47.163032 \n",
"L 59.228387 47.265361 \n",
"L 59.629548 47.389666 \n",
"L 60.030146 47.535868 \n",
"L 60.430159 47.70389 \n",
"L 60.829571 47.893648 \n",
"L 61.228364 48.10506 \n",
"L 61.626519 48.33804 \n",
"L 62.02402 48.592501 \n",
"L 62.420849 48.868355 \n",
"L 62.816987 49.165512 \n",
"L 63.212419 49.483878 \n",
"L 63.607128 49.823362 \n",
"L 64.001094 50.183866 \n",
"L 64.394303 50.565295 \n",
"L 64.78674 50.96755 \n",
"L 65.178384 51.39053 \n",
"L 65.569222 51.834133 \n",
"L 65.959238 52.298258 \n",
"L 66.348414 52.782799 \n",
"L 66.736736 53.287649 \n",
"L 67.124188 53.812704 \n",
"L 67.510754 54.357851 \n",
"L 67.896419 54.922981 \n",
"L 68.28117 55.507985 \n",
"L 68.664989 56.112745 \n",
"L 69.047862 56.737151 \n",
"L 69.429775 57.381086 \n",
"L 69.810715 58.044436 \n",
"L 70.190665 58.727079 \n",
"L 70.569613 59.428898 \n",
"L 70.947545 60.149774 \n",
"L 71.324447 60.889583 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 38.218828 103.780253 \n",
"L 38.609854 102.326915 \n",
"L 39.001619 100.895427 \n",
"L 39.3941 99.485873 \n",
"L 39.787279 98.098326 \n",
"L 40.181136 96.732856 \n",
"L 40.575648 95.389541 \n",
"L 40.970796 94.068445 \n",
"L 41.366559 92.769638 \n",
"L 41.762918 91.493176 \n",
"L 42.159849 90.239131 \n",
"L 42.557334 89.007557 \n",
"L 42.955351 87.79851 \n",
"L 43.353879 86.61205 \n",
"L 43.752897 85.448224 \n",
"L 44.152385 84.307084 \n",
"L 44.552321 83.18868 \n",
"L 44.952684 82.093055 \n",
"L 45.353454 81.020249 \n",
"L 45.754608 79.970308 \n",
"L 46.156126 78.943266 \n",
"L 46.557988 77.939158 \n",
"L 46.960171 76.958022 \n",
"L 47.362654 75.999885 \n",
"L 47.765417 75.064773 \n",
"L 48.168438 74.152717 \n",
"L 48.571696 73.263739 \n",
"L 48.97517 72.397857 \n",
"L 49.378839 71.555092 \n",
"L 49.782681 70.735462 \n",
"L 50.186676 69.938976 \n",
"L 50.590802 69.165649 \n",
"L 50.995038 68.415489 \n",
"L 51.399364 67.688502 \n",
"L 51.803758 66.984692 \n",
"L 52.208199 66.304062 \n",
"L 52.612667 65.646611 \n",
"L 53.01714 65.012334 \n",
"L 53.421598 64.401226 \n",
"L 53.826019 63.813282 \n",
"L 54.230384 63.248489 \n",
"L 54.63467 62.706835 \n",
"L 55.038858 62.188305 \n",
"L 55.442928 61.692882 \n",
"L 55.846858 61.220547 \n",
"L 56.250628 60.771277 \n",
"L 56.654218 60.345048 \n",
"L 57.057607 59.941835 \n",
"L 57.460775 59.561607 \n",
"L 57.863703 59.204334 \n",
"L 58.26637 58.869981 \n",
"L 58.668755 58.558516 \n",
"L 59.07084 58.269898 \n",
"L 59.472603 58.004089 \n",
"L 59.874027 57.761046 \n",
"L 60.27509 57.540724 \n",
"L 60.675774 57.343078 \n",
"L 61.076059 57.168058 \n",
"L 61.475927 57.015614 \n",
"L 61.875356 56.885692 \n",
"L 62.27433 56.778238 \n",
"L 62.672828 56.693194 \n",
"L 63.070832 56.630501 \n",
"L 63.468323 56.590099 \n",
"L 63.865283 56.571923 \n",
"L 64.261693 56.57591 \n",
"L 64.657535 56.601991 \n",
"L 65.05279 56.650098 \n",
"L 65.447442 56.720159 \n",
"L 65.841471 56.812102 \n",
"L 66.23486 56.925851 \n",
"L 66.627591 57.06133 \n",
"L 67.019647 57.218461 \n",
"L 67.41101 57.397162 \n",
"L 67.801664 57.597353 \n",
"L 68.191591 57.818949 \n",
"L 68.580774 58.061863 \n",
"L 68.969198 58.32601 \n",
"L 69.356843 58.6113 \n",
"L 69.743695 58.917641 \n",
"L 70.129739 59.244942 \n",
"L 70.514955 59.593109 \n",
"L 70.89933 59.962046 \n",
"L 71.282848 60.351656 \n",
"L 71.665492 60.761839 \n",
"L 72.047247 61.192495 \n",
"L 72.428099 61.643524 \n",
"L 72.808031 62.114823 \n",
"L 73.187029 62.606284 \n",
"L 73.565079 63.117804 \n",
"L 73.942165 63.649275 \n",
"L 74.318273 64.200585 \n",
"L 74.693389 64.77163 \n",
"L 75.067499 65.362292 \n",
"L 75.440588 65.972463 \n",
"L 75.812644 66.602026 \n",
"L 76.183653 67.25087 \n",
"L 76.553601 67.918873 \n",
"L 76.922474 68.605921 \n",
"L 77.290262 69.311896 \n",
"L 77.656949 70.036674 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 45.032743 111.384673 \n",
"L 45.424331 109.931557 \n",
"L 45.816541 108.500162 \n",
"L 46.209349 107.090573 \n",
"L 46.602736 105.702865 \n",
"L 46.996683 104.337106 \n",
"L 47.391168 102.993376 \n",
"L 47.78617 101.671737 \n",
"L 48.181669 100.37226 \n",
"L 48.577646 99.095 \n",
"L 48.974077 97.840032 \n",
"L 49.370943 96.607407 \n",
"L 49.768225 95.397182 \n",
"L 50.165898 94.209418 \n",
"L 50.563945 93.044162 \n",
"L 50.962344 91.901466 \n",
"L 51.361072 90.781378 \n",
"L 51.760111 89.683944 \n",
"L 52.159439 88.609203 \n",
"L 52.559035 87.557201 \n",
"L 52.958877 86.527974 \n",
"L 53.358946 85.521555 \n",
"L 53.75922 84.537982 \n",
"L 54.159677 83.577284 \n",
"L 54.560299 82.639487 \n",
"L 54.961062 81.724622 \n",
"L 55.361946 80.83271 \n",
"L 55.762931 79.963771 \n",
"L 56.163996 79.117826 \n",
"L 56.565118 78.294892 \n",
"L 56.966279 77.49498 \n",
"L 57.367457 76.718105 \n",
"L 57.768631 75.964274 \n",
"L 58.16978 75.233494 \n",
"L 58.570884 74.525772 \n",
"L 58.971923 73.841107 \n",
"L 59.372874 73.179501 \n",
"L 59.773719 72.540949 \n",
"L 60.174437 71.925447 \n",
"L 60.575006 71.332988 \n",
"L 60.975407 70.763563 \n",
"L 61.375619 70.217158 \n",
"L 61.775623 69.693759 \n",
"L 62.175398 69.19335 \n",
"L 62.574923 68.715911 \n",
"L 62.974179 68.261422 \n",
"L 63.373146 67.829858 \n",
"L 63.771805 67.421193 \n",
"L 64.170135 67.035398 \n",
"L 64.568116 66.672445 \n",
"L 64.96573 66.332299 \n",
"L 65.362957 66.014926 \n",
"L 65.759776 65.720288 \n",
"L 66.156169 65.448346 \n",
"L 66.552118 65.199059 \n",
"L 66.947602 64.972382 \n",
"L 67.342603 64.76827 \n",
"L 67.737101 64.586675 \n",
"L 68.13108 64.427546 \n",
"L 68.524519 64.290831 \n",
"L 68.9174 64.176476 \n",
"L 69.309704 64.084423 \n",
"L 69.701415 64.014616 \n",
"L 70.092513 63.966993 \n",
"L 70.482981 63.941491 \n",
"L 70.8728 63.938047 \n",
"L 71.261954 63.956593 \n",
"L 71.650423 63.997061 \n",
"L 72.038193 64.059381 \n",
"L 72.425244 64.143481 \n",
"L 72.811559 64.249287 \n",
"L 73.197122 64.376722 \n",
"L 73.581916 64.525708 \n",
"L 73.965925 64.696166 \n",
"L 74.349131 64.888015 \n",
"L 74.731518 65.101171 \n",
"L 75.113071 65.33555 \n",
"L 75.493773 65.591065 \n",
"L 75.873608 65.867627 \n",
"L 76.25256 66.165145 \n",
"L 76.630615 66.483531 \n",
"L 77.007755 66.822689 \n",
"L 77.383967 67.182524 \n",
"L 77.759236 67.562941 \n",
"L 78.133545 67.96384 \n",
"L 78.506881 68.385124 \n",
"L 78.87923 68.82669 \n",
"L 79.250576 69.288438 \n",
"L 79.620905 69.770261 \n",
"L 79.990205 70.272056 \n",
"L 80.358459 70.793715 \n",
"L 80.725656 71.33513 \n",
"L 81.091782 71.896195 \n",
"L 81.456821 72.476793 \n",
"L 81.820763 73.076817 \n",
"L 82.183594 73.696152 \n",
"L 82.545303 74.334687 \n",
"L 82.905874 74.992301 \n",
"L 83.265297 75.668881 \n",
"L 83.623559 76.364308 \n",
"L 83.980648 77.078462 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 51.803488 116.813651 \n",
"L 52.196318 115.358076 \n",
"L 52.589653 113.92417 \n",
"L 52.983469 112.512019 \n",
"L 53.377749 111.121697 \n",
"L 53.772472 109.753272 \n",
"L 54.167615 108.406825 \n",
"L 54.563159 107.082419 \n",
"L 54.959083 105.780123 \n",
"L 55.355367 104.499996 \n",
"L 55.75199 103.242109 \n",
"L 56.14893 102.006517 \n",
"L 56.546169 100.793275 \n",
"L 56.943682 99.602444 \n",
"L 57.341452 98.434073 \n",
"L 57.739457 97.288212 \n",
"L 58.137675 96.164913 \n",
"L 58.536086 95.064219 \n",
"L 58.93467 93.986171 \n",
"L 59.333404 92.930813 \n",
"L 59.732268 91.898183 \n",
"L 60.131244 90.888315 \n",
"L 60.530306 89.901247 \n",
"L 60.929437 88.937008 \n",
"L 61.328615 87.995623 \n",
"L 61.727819 87.077124 \n",
"L 62.127028 86.181533 \n",
"L 62.526222 85.30887 \n",
"L 62.925381 84.459155 \n",
"L 63.324482 83.632407 \n",
"L 63.723506 82.828638 \n",
"L 64.122432 82.047861 \n",
"L 64.52124 81.290084 \n",
"L 64.919909 80.555315 \n",
"L 65.318418 79.84356 \n",
"L 65.716748 79.15482 \n",
"L 66.114878 78.489096 \n",
"L 66.512787 77.846384 \n",
"L 66.910456 77.22668 \n",
"L 67.307865 76.629977 \n",
"L 67.704992 76.056265 \n",
"L 68.101819 75.505534 \n",
"L 68.498325 74.977767 \n",
"L 68.894491 74.472949 \n",
"L 69.290297 73.991062 \n",
"L 69.685723 73.532083 \n",
"L 70.08075 73.095989 \n",
"L 70.475358 72.682755 \n",
"L 70.869528 72.292353 \n",
"L 71.263241 71.924752 \n",
"L 71.656478 71.579919 \n",
"L 72.049218 71.257822 \n",
"L 72.441445 70.958421 \n",
"L 72.833138 70.681678 \n",
"L 73.224279 70.427552 \n",
"L 73.61485 70.195998 \n",
"L 74.004831 69.986973 \n",
"L 74.394205 69.800426 \n",
"L 74.782954 69.636309 \n",
"L 75.171059 69.49457 \n",
"L 75.558503 69.375154 \n",
"L 75.945267 69.278006 \n",
"L 76.331334 69.203066 \n",
"L 76.716687 69.150275 \n",
"L 77.101307 69.11957 \n",
"L 77.485178 69.110887 \n",
"L 77.868283 69.124161 \n",
"L 78.250604 69.159321 \n",
"L 78.632125 69.2163 \n",
"L 79.01283 69.295024 \n",
"L 79.3927 69.395419 \n",
"L 79.771721 69.517411 \n",
"L 80.149876 69.66092 \n",
"L 80.52715 69.825869 \n",
"L 80.903525 70.012175 \n",
"L 81.278986 70.219756 \n",
"L 81.653518 70.448527 \n",
"L 82.027107 70.698402 \n",
"L 82.399735 70.969291 \n",
"L 82.771387 71.261106 \n",
"L 83.142052 71.573757 \n",
"L 83.51171 71.907148 \n",
"L 83.88035 72.261186 \n",
"L 84.247957 72.635775 \n",
"L 84.614516 73.030816 \n",
"L 84.980013 73.44621 \n",
"L 85.344435 73.881857 \n",
"L 85.707768 74.337655 \n",
"L 86.069998 74.813499 \n",
"L 86.431113 75.309286 \n",
"L 86.791098 75.824908 \n",
"L 87.149941 76.360256 \n",
"L 87.50763 76.915225 \n",
"L 87.864152 77.489699 \n",
"L 88.219493 78.083571 \n",
"L 88.573643 78.696726 \n",
"L 88.92659 79.329052 \n",
"L 89.27832 79.98043 \n",
"L 89.628823 80.650746 \n",
"L 89.978089 81.339882 \n",
"L 90.326103 82.047717 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 58.566254 120.082803 \n",
"L 58.961017 118.622062 \n",
"L 59.35617 117.183014 \n",
"L 59.751688 115.765745 \n",
"L 60.147553 114.370327 \n",
"L 60.543744 112.996832 \n",
"L 60.940239 111.645339 \n",
"L 61.337019 110.315912 \n",
"L 61.734061 109.008623 \n",
"L 62.131347 107.723528 \n",
"L 62.528854 106.4607 \n",
"L 62.926561 105.220194 \n",
"L 63.32445 104.002066 \n",
"L 63.722496 102.806378 \n",
"L 64.12068 101.633179 \n",
"L 64.518983 100.482518 \n",
"L 64.917381 99.35445 \n",
"L 65.315854 98.249016 \n",
"L 65.714383 97.166258 \n",
"L 66.112945 96.106223 \n",
"L 66.511519 95.068946 \n",
"L 66.910087 94.054461 \n",
"L 67.308625 93.062809 \n",
"L 67.707113 92.094017 \n",
"L 68.105532 91.148113 \n",
"L 68.503859 90.225128 \n",
"L 68.902074 89.325083 \n",
"L 69.300157 88.448 \n",
"L 69.698088 87.593899 \n",
"L 70.095844 86.762798 \n",
"L 70.493407 85.95471 \n",
"L 70.890755 85.169648 \n",
"L 71.287869 84.407621 \n",
"L 71.684728 83.668638 \n",
"L 72.08131 82.952702 \n",
"L 72.477598 82.259817 \n",
"L 72.87357 81.589983 \n",
"L 73.269207 80.943196 \n",
"L 73.664487 80.319453 \n",
"L 74.059393 79.718747 \n",
"L 74.453902 79.141068 \n",
"L 74.847997 78.586404 \n",
"L 75.241658 78.054742 \n",
"L 75.634864 77.546064 \n",
"L 76.027597 77.060352 \n",
"L 76.419836 76.597586 \n",
"L 76.811564 76.157741 \n",
"L 77.20276 75.740792 \n",
"L 77.593407 75.34671 \n",
"L 77.983484 74.975466 \n",
"L 78.372974 74.627027 \n",
"L 78.761857 74.301358 \n",
"L 79.150115 73.998423 \n",
"L 79.537729 73.718182 \n",
"L 79.924682 73.460593 \n",
"L 80.310954 73.225613 \n",
"L 80.696529 73.013197 \n",
"L 81.081388 72.823295 \n",
"L 81.465513 72.655859 \n",
"L 81.848887 72.510836 \n",
"L 82.231492 72.388172 \n",
"L 82.613311 72.28781 \n",
"L 82.994327 72.209693 \n",
"L 83.374523 72.153759 \n",
"L 83.753882 72.119946 \n",
"L 84.132387 72.108189 \n",
"L 84.510021 72.118423 \n",
"L 84.886768 72.15058 \n",
"L 85.262612 72.204587 \n",
"L 85.637537 72.280374 \n",
"L 86.011526 72.377866 \n",
"L 86.384564 72.496988 \n",
"L 86.756636 72.63766 \n",
"L 87.127725 72.799804 \n",
"L 87.497818 72.983339 \n",
"L 87.866897 73.188181 \n",
"L 88.234948 73.414245 \n",
"L 88.601958 73.661444 \n",
"L 88.967911 73.92969 \n",
"L 89.332792 74.218893 \n",
"L 89.696588 74.528962 \n",
"L 90.059284 74.859802 \n",
"L 90.420866 75.211319 \n",
"L 90.781323 75.583417 \n",
"L 91.140637 75.975996 \n",
"L 91.498797 76.388958 \n",
"L 91.855791 76.822202 \n",
"L 92.211604 77.275626 \n",
"L 92.566223 77.749123 \n",
"L 92.919638 78.242591 \n",
"L 93.271834 78.755922 \n",
"L 93.6228 79.289005 \n",
"L 93.972524 79.841737 \n",
"L 94.320994 80.414 \n",
"L 94.668197 81.005686 \n",
"L 95.014123 81.616681 \n",
"L 95.358762 82.246872 \n",
"L 95.7021 82.89614 \n",
"L 96.044127 83.564369 \n",
"L 96.384835 84.251444 \n",
"L 96.724209 84.957241 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 65.356543 121.185182 \n",
"L 65.753953 119.716513 \n",
"L 66.151634 118.269634 \n",
"L 66.549565 116.844634 \n",
"L 66.947725 115.441586 \n",
"L 67.346094 114.060561 \n",
"L 67.744649 112.701641 \n",
"L 68.14337 111.36489 \n",
"L 68.542236 110.05038 \n",
"L 68.941227 108.758168 \n",
"L 69.340321 107.488329 \n",
"L 69.739496 106.240917 \n",
"L 70.138734 105.01599 \n",
"L 70.538011 103.813609 \n",
"L 70.937306 102.633825 \n",
"L 71.336601 101.476688 \n",
"L 71.735871 100.34225 \n",
"L 72.135098 99.230558 \n",
"L 72.534261 98.141651 \n",
"L 72.933337 97.075575 \n",
"L 73.332306 96.032369 \n",
"L 73.731148 95.012067 \n",
"L 74.129841 94.014709 \n",
"L 74.528365 93.040324 \n",
"L 74.926699 92.088936 \n",
"L 75.324822 91.160581 \n",
"L 75.722714 90.255279 \n",
"L 76.120354 89.373052 \n",
"L 76.517722 88.51392 \n",
"L 76.914796 87.677902 \n",
"L 77.311557 86.86501 \n",
"L 77.707984 86.075258 \n",
"L 78.104057 85.308656 \n",
"L 78.499756 84.56521 \n",
"L 78.89506 83.844928 \n",
"L 79.289951 83.147809 \n",
"L 79.684406 82.473856 \n",
"L 80.078408 81.823065 \n",
"L 80.471936 81.195431 \n",
"L 80.864969 80.590949 \n",
"L 81.25749 80.009608 \n",
"L 81.649477 79.451397 \n",
"L 82.040912 78.916301 \n",
"L 82.431777 78.404304 \n",
"L 82.82205 77.915387 \n",
"L 83.211713 77.449528 \n",
"L 83.600748 77.006704 \n",
"L 83.989136 76.58689 \n",
"L 84.376857 76.190055 \n",
"L 84.763893 75.816172 \n",
"L 85.150227 75.465204 \n",
"L 85.535839 75.13712 \n",
"L 85.920711 74.831881 \n",
"L 86.304825 74.549447 \n",
"L 86.688163 74.289777 \n",
"L 87.070708 74.052826 \n",
"L 87.452442 73.838548 \n",
"L 87.833347 73.646896 \n",
"L 88.213406 73.477817 \n",
"L 88.592602 73.331261 \n",
"L 88.970917 73.207172 \n",
"L 89.348336 73.105493 \n",
"L 89.72484 73.026164 \n",
"L 90.100414 72.969126 \n",
"L 90.475042 72.934316 \n",
"L 90.848705 72.921667 \n",
"L 91.22139 72.931113 \n",
"L 91.593079 72.962585 \n",
"L 91.963758 73.016012 \n",
"L 92.33341 73.091321 \n",
"L 92.70202 73.188438 \n",
"L 93.069573 73.307285 \n",
"L 93.436053 73.447784 \n",
"L 93.801447 73.609854 \n",
"L 94.165738 73.793414 \n",
"L 94.528913 73.998379 \n",
"L 94.890957 74.224663 \n",
"L 95.251856 74.472181 \n",
"L 95.611596 74.740839 \n",
"L 95.970163 75.03055 \n",
"L 96.327545 75.341221 \n",
"L 96.683725 75.672757 \n",
"L 97.038692 76.025062 \n",
"L 97.392435 76.398039 \n",
"L 97.744937 76.791588 \n",
"L 98.096187 77.20561 \n",
"L 98.446174 77.640002 \n",
"L 98.794883 78.094661 \n",
"L 99.142304 78.569482 \n",
"L 99.488425 79.064358 \n",
"L 99.833233 79.579182 \n",
"L 100.176717 80.113844 \n",
"L 100.518867 80.668236 \n",
"L 100.859669 81.24224 \n",
"L 101.199115 81.83575 \n",
"L 101.537192 82.448647 \n",
"L 101.873891 83.08082 \n",
"L 102.209201 83.732148 \n",
"L 102.543112 84.402514 \n",
"L 102.875615 85.0918 \n",
"L 103.206697 85.799883 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 72.210812 120.090746 \n",
"L 72.611609 118.611293 \n",
"L 73.012559 117.153808 \n",
"L 73.413638 115.718379 \n",
"L 73.814827 114.305082 \n",
"L 74.216106 112.913989 \n",
"L 74.61745 111.545182 \n",
"L 75.01884 110.198727 \n",
"L 75.420255 108.874696 \n",
"L 75.821674 107.573148 \n",
"L 76.223074 106.294157 \n",
"L 76.624434 105.037781 \n",
"L 77.025735 103.804076 \n",
"L 77.426952 102.593106 \n",
"L 77.828067 101.404921 \n",
"L 78.229059 100.239573 \n",
"L 78.629903 99.097114 \n",
"L 79.030582 97.977591 \n",
"L 79.431073 96.881045 \n",
"L 79.831355 95.807523 \n",
"L 80.231407 94.757063 \n",
"L 80.631209 93.729699 \n",
"L 81.030738 92.725473 \n",
"L 81.429975 91.744413 \n",
"L 81.828899 90.786546 \n",
"L 82.227488 89.851906 \n",
"L 82.625722 88.940514 \n",
"L 83.023582 88.052392 \n",
"L 83.421045 87.18756 \n",
"L 83.818091 86.346038 \n",
"L 84.2147 85.527838 \n",
"L 84.610852 84.732975 \n",
"L 85.006526 83.961456 \n",
"L 85.401703 83.21329 \n",
"L 85.796362 82.488483 \n",
"L 86.190483 81.787036 \n",
"L 86.584046 81.108951 \n",
"L 86.977033 80.454223 \n",
"L 87.369422 79.822848 \n",
"L 87.761195 79.21482 \n",
"L 88.152331 78.630129 \n",
"L 88.542812 78.068763 \n",
"L 88.932618 77.530706 \n",
"L 89.321731 77.015942 \n",
"L 89.710131 76.524452 \n",
"L 90.0978 76.056214 \n",
"L 90.484718 75.611203 \n",
"L 90.870868 75.189395 \n",
"L 91.25623 74.790759 \n",
"L 91.640786 74.415265 \n",
"L 92.02452 74.062878 \n",
"L 92.407411 73.733565 \n",
"L 92.789442 73.427286 \n",
"L 93.170596 73.144001 \n",
"L 93.550854 72.883668 \n",
"L 93.9302 72.646242 \n",
"L 94.308616 72.431676 \n",
"L 94.686085 72.239921 \n",
"L 95.062591 72.070925 \n",
"L 95.438115 71.924635 \n",
"L 95.812642 71.800995 \n",
"L 96.186155 71.699948 \n",
"L 96.558638 71.621433 \n",
"L 96.930075 71.565388 \n",
"L 97.300449 71.531749 \n",
"L 97.669745 71.520451 \n",
"L 98.037947 71.531425 \n",
"L 98.40504 71.5646 \n",
"L 98.771008 71.619906 \n",
"L 99.135837 71.697267 \n",
"L 99.499511 71.796608 \n",
"L 99.862016 71.91785 \n",
"L 100.223337 72.060914 \n",
"L 100.58346 72.225719 \n",
"L 100.94237 72.41218 \n",
"L 101.300054 72.620212 \n",
"L 101.656498 72.849729 \n",
"L 102.011689 73.100641 \n",
"L 102.365612 73.372858 \n",
"L 102.718254 73.666287 \n",
"L 103.069604 73.980835 \n",
"L 103.419647 74.316406 \n",
"L 103.76837 74.672902 \n",
"L 104.115764 75.050224 \n",
"L 104.461813 75.448273 \n",
"L 104.806506 75.866945 \n",
"L 105.149833 76.306137 \n",
"L 105.49178 76.765746 \n",
"L 105.832337 77.245663 \n",
"L 106.171493 77.745781 \n",
"L 106.509235 78.265991 \n",
"L 106.845553 78.80618 \n",
"L 107.180439 79.36624 \n",
"L 107.513878 79.946052 \n",
"L 107.845863 80.545506 \n",
"L 108.176383 81.164483 \n",
"L 108.505429 81.802869 \n",
"L 108.83299 82.460542 \n",
"L 109.159057 83.137385 \n",
"L 109.483623 83.833276 \n",
"L 109.806675 84.548092 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 79.167147 116.745165 \n",
"L 79.572108 115.251949 \n",
"L 79.977101 113.78096 \n",
"L 80.3821 112.332288 \n",
"L 80.787086 110.90601 \n",
"L 81.192038 109.502199 \n",
"L 81.596932 108.120939 \n",
"L 82.001748 106.762297 \n",
"L 82.406463 105.426347 \n",
"L 82.811058 104.113147 \n",
"L 83.215508 102.822776 \n",
"L 83.619793 101.555288 \n",
"L 84.023892 100.310745 \n",
"L 84.427783 99.089208 \n",
"L 84.831443 97.890731 \n",
"L 85.234854 96.715364 \n",
"L 85.637991 95.563164 \n",
"L 86.040834 94.434174 \n",
"L 86.443364 93.328438 \n",
"L 86.845555 92.246005 \n",
"L 87.247389 91.186912 \n",
"L 87.648844 90.151194 \n",
"L 88.049899 89.138893 \n",
"L 88.450533 88.150038 \n",
"L 88.850726 87.184657 \n",
"L 89.250454 86.242783 \n",
"L 89.649699 85.324438 \n",
"L 90.04844 84.429644 \n",
"L 90.446656 83.558422 \n",
"L 90.844326 82.710792 \n",
"L 91.24143 81.886765 \n",
"L 91.637948 81.086357 \n",
"L 92.033859 80.309574 \n",
"L 92.429143 79.556427 \n",
"L 92.82378 78.82692 \n",
"L 93.21775 78.121055 \n",
"L 93.611033 77.438832 \n",
"L 94.00361 76.780248 \n",
"L 94.395461 76.145297 \n",
"L 94.786566 75.533975 \n",
"L 95.176906 74.946268 \n",
"L 95.566462 74.382165 \n",
"L 95.955215 73.841651 \n",
"L 96.343146 73.324708 \n",
"L 96.730235 72.831317 \n",
"L 97.116465 72.361454 \n",
"L 97.501817 71.915096 \n",
"L 97.886272 71.492216 \n",
"L 98.269812 71.092782 \n",
"L 98.652419 70.716765 \n",
"L 99.034076 70.364127 \n",
"L 99.414763 70.034836 \n",
"L 99.794464 69.728849 \n",
"L 100.173161 69.446127 \n",
"L 100.550837 69.186625 \n",
"L 100.927475 68.950298 \n",
"L 101.303057 68.737098 \n",
"L 101.677567 68.546974 \n",
"L 102.050988 68.379874 \n",
"L 102.423304 68.235741 \n",
"L 102.794499 68.114521 \n",
"L 103.164556 68.016153 \n",
"L 103.533459 67.940576 \n",
"L 103.901194 67.887727 \n",
"L 104.267743 67.857539 \n",
"L 104.633092 67.849946 \n",
"L 104.997226 67.864878 \n",
"L 105.360129 67.902262 \n",
"L 105.721788 67.962025 \n",
"L 106.082186 68.044092 \n",
"L 106.44131 68.148384 \n",
"L 106.799146 68.274822 \n",
"L 107.155678 68.423324 \n",
"L 107.510895 68.593807 \n",
"L 107.864782 68.786185 \n",
"L 108.217325 69.000371 \n",
"L 108.568512 69.236276 \n",
"L 108.918329 69.493809 \n",
"L 109.266764 69.772878 \n",
"L 109.613803 70.073388 \n",
"L 109.959435 70.395243 \n",
"L 110.303647 70.738347 \n",
"L 110.646427 71.102597 \n",
"L 110.987765 71.487896 \n",
"L 111.327646 71.894138 \n",
"L 111.666061 72.321219 \n",
"L 112.002999 72.769037 \n",
"L 112.338447 73.237481 \n",
"L 112.672396 73.726443 \n",
"L 113.004836 74.235814 \n",
"L 113.335755 74.765482 \n",
"L 113.665143 75.315331 \n",
"L 113.992992 75.885252 \n",
"L 114.31929 76.475123 \n",
"L 114.644028 77.084831 \n",
"L 114.967197 77.714256 \n",
"L 115.28879 78.363282 \n",
"L 115.608795 79.031783 \n",
"L 115.927205 79.719639 \n",
"L 116.244011 80.426727 \n",
"L 116.559205 81.152921 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 86.266001 111.06794 \n",
"L 86.675953 109.55782 \n",
"L 87.085808 108.070273 \n",
"L 87.495543 106.605391 \n",
"L 87.905136 105.163252 \n",
"L 88.314566 103.743932 \n",
"L 88.723809 102.347516 \n",
"L 89.132844 100.974072 \n",
"L 89.541649 99.623676 \n",
"L 89.950203 98.29639 \n",
"L 90.358481 96.99229 \n",
"L 90.766463 95.711434 \n",
"L 91.174128 94.453885 \n",
"L 91.581451 93.219705 \n",
"L 91.988412 92.008948 \n",
"L 92.39499 90.821668 \n",
"L 92.801162 89.65792 \n",
"L 93.206906 88.51775 \n",
"L 93.612202 87.401202 \n",
"L 94.017026 86.308325 \n",
"L 94.421359 85.239157 \n",
"L 94.825179 84.193735 \n",
"L 95.228462 83.172101 \n",
"L 95.63119 82.174284 \n",
"L 96.033342 81.200313 \n",
"L 96.434894 80.25022 \n",
"L 96.835826 79.32403 \n",
"L 97.236119 78.421763 \n",
"L 97.635751 77.543441 \n",
"L 98.0347 76.689083 \n",
"L 98.432948 75.858703 \n",
"L 98.830472 75.052313 \n",
"L 99.227253 74.269922 \n",
"L 99.623271 73.51154 \n",
"L 100.018505 72.777171 \n",
"L 100.412937 72.066814 \n",
"L 100.806544 71.380474 \n",
"L 101.199308 70.718143 \n",
"L 101.59121 70.079817 \n",
"L 101.98223 69.465487 \n",
"L 102.372347 68.875146 \n",
"L 102.761545 68.308776 \n",
"L 103.149802 67.766364 \n",
"L 103.537102 67.247889 \n",
"L 103.923424 66.753335 \n",
"L 104.308749 66.282674 \n",
"L 104.693061 65.835883 \n",
"L 105.076341 65.412932 \n",
"L 105.458569 65.013793 \n",
"L 105.839729 64.638428 \n",
"L 106.219804 64.286805 \n",
"L 106.598774 63.958886 \n",
"L 106.976623 63.654631 \n",
"L 107.353334 63.373994 \n",
"L 107.728889 63.116933 \n",
"L 108.103271 62.8834 \n",
"L 108.476465 62.673345 \n",
"L 108.848452 62.486715 \n",
"L 109.219218 62.323458 \n",
"L 109.588746 62.183515 \n",
"L 109.95702 62.066829 \n",
"L 110.324025 61.973336 \n",
"L 110.689743 61.902977 \n",
"L 111.054162 61.855682 \n",
"L 111.417265 61.831387 \n",
"L 111.779037 61.830019 \n",
"L 112.139463 61.851511 \n",
"L 112.498529 61.895784 \n",
"L 112.856222 61.962765 \n",
"L 113.212526 62.052374 \n",
"L 113.567426 62.164534 \n",
"L 113.920912 62.299159 \n",
"L 114.272967 62.456168 \n",
"L 114.623579 62.635473 \n",
"L 114.972736 62.836988 \n",
"L 115.320422 63.060622 \n",
"L 115.666628 63.306284 \n",
"L 116.01134 63.573881 \n",
"L 116.354545 63.863316 \n",
"L 116.696232 64.174493 \n",
"L 117.036389 64.507314 \n",
"L 117.375004 64.861678 \n",
"L 117.712065 65.237483 \n",
"L 118.047564 65.634625 \n",
"L 118.381486 66.052997 \n",
"L 118.713822 66.492494 \n",
"L 119.044563 66.953007 \n",
"L 119.373696 67.434426 \n",
"L 119.701212 67.936638 \n",
"L 120.027103 68.459531 \n",
"L 120.351357 69.00299 \n",
"L 120.673965 69.566897 \n",
"L 120.99492 70.15114 \n",
"L 121.31421 70.755592 \n",
"L 121.631827 71.380139 \n",
"L 121.947764 72.024657 \n",
"L 122.262012 72.689026 \n",
"L 122.574562 73.373118 \n",
"L 122.885407 74.07681 \n",
"L 123.19454 74.799976 \n",
"L 123.501952 75.542486 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 93.551058 102.94972 \n",
"L 93.966883 101.419348 \n",
"L 94.382478 99.911988 \n",
"L 94.797818 98.427736 \n",
"L 95.212882 96.966671 \n",
"L 95.627647 95.528871 \n",
"L 96.042089 94.114424 \n",
"L 96.456186 92.7234 \n",
"L 96.869916 91.355875 \n",
"L 97.283257 90.011914 \n",
"L 97.696184 88.691595 \n",
"L 98.108676 87.394978 \n",
"L 98.520711 86.122125 \n",
"L 98.932266 84.873103 \n",
"L 99.343318 83.647965 \n",
"L 99.753846 82.446767 \n",
"L 100.163827 81.269564 \n",
"L 100.573239 80.116405 \n",
"L 100.982061 78.987334 \n",
"L 101.390269 77.8824 \n",
"L 101.797842 76.801644 \n",
"L 102.20476 75.745103 \n",
"L 102.610999 74.712818 \n",
"L 103.016538 73.70482 \n",
"L 103.421357 72.721139 \n",
"L 103.825432 71.761807 \n",
"L 104.228744 70.826849 \n",
"L 104.631271 69.916284 \n",
"L 105.032992 69.030137 \n",
"L 105.433886 68.168425 \n",
"L 105.833933 67.331163 \n",
"L 106.233112 66.518362 \n",
"L 106.631402 65.730031 \n",
"L 107.028783 64.96618 \n",
"L 107.425234 64.226813 \n",
"L 107.820736 63.51193 \n",
"L 108.215269 62.821529 \n",
"L 108.608813 62.155609 \n",
"L 109.001348 61.514162 \n",
"L 109.392855 60.897179 \n",
"L 109.783314 60.30465 \n",
"L 110.172707 59.73656 \n",
"L 110.561014 59.192891 \n",
"L 110.948216 58.673626 \n",
"L 111.334296 58.17874 \n",
"L 111.719233 57.70821 \n",
"L 112.103011 57.262009 \n",
"L 112.48561 56.840108 \n",
"L 112.867014 56.442475 \n",
"L 113.247203 56.069072 \n",
"L 113.626163 55.719865 \n",
"L 114.003872 55.394814 \n",
"L 114.380315 55.093876 \n",
"L 114.755476 54.817006 \n",
"L 115.129337 54.564158 \n",
"L 115.501881 54.335283 \n",
"L 115.873092 54.130328 \n",
"L 116.242954 53.94924 \n",
"L 116.611452 53.79196 \n",
"L 116.978568 53.658431 \n",
"L 117.344288 53.548592 \n",
"L 117.708596 53.462378 \n",
"L 118.071477 53.399726 \n",
"L 118.432916 53.360565 \n",
"L 118.792898 53.344827 \n",
"L 119.151409 53.352439 \n",
"L 119.508434 53.383325 \n",
"L 119.86396 53.437409 \n",
"L 120.217972 53.514613 \n",
"L 120.570457 53.614857 \n",
"L 120.9214 53.738056 \n",
"L 121.27079 53.884125 \n",
"L 121.618613 54.052978 \n",
"L 121.964857 54.244526 \n",
"L 122.309508 54.458677 \n",
"L 122.652554 54.695338 \n",
"L 122.993984 54.954415 \n",
"L 123.333786 55.23581 \n",
"L 123.671947 55.539428 \n",
"L 124.008457 55.865164 \n",
"L 124.343305 56.212918 \n",
"L 124.676478 56.582587 \n",
"L 125.007967 56.974064 \n",
"L 125.337763 57.387241 \n",
"L 125.665852 57.822009 \n",
"L 125.992226 58.278259 \n",
"L 126.316876 58.755876 \n",
"L 126.639791 59.25475 \n",
"L 126.960962 59.774762 \n",
"L 127.280382 60.315797 \n",
"L 127.598039 60.877736 \n",
"L 127.913925 61.460457 \n",
"L 128.228034 62.063844 \n",
"L 128.540355 62.687766 \n",
"L 128.850882 63.332106 \n",
"L 129.159606 63.996735 \n",
"L 129.466521 64.68153 \n",
"L 129.771618 65.386359 \n",
"L 130.074891 66.111094 \n",
"L 130.376334 66.855605 \n",
"L 130.675939 67.619758 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 24.318635 81.868935 \n",
"L 25.025641 83.182889 \n",
"L 25.731159 84.473354 \n",
"L 26.435225 85.740419 \n",
"L 27.137878 86.984178 \n",
"L 27.839158 88.204725 \n",
"L 28.5391 89.402139 \n",
"L 29.237744 90.57651 \n",
"L 29.935126 91.727916 \n",
"L 30.631287 92.856445 \n",
"L 31.326259 93.962164 \n",
"L 32.020082 95.045152 \n",
"L 32.712795 96.105484 \n",
"L 33.40443 97.143223 \n",
"L 34.095027 98.158442 \n",
"L 34.784623 99.151203 \n",
"L 35.47325 100.121569 \n",
"L 36.160949 101.069599 \n",
"L 36.847755 101.995355 \n",
"L 37.533702 102.898888 \n",
"L 38.218828 103.780253 \n",
"L 38.903169 104.639501 \n",
"L 39.586759 105.476678 \n",
"L 40.269634 106.291834 \n",
"L 40.951832 107.08501 \n",
"L 41.633386 107.856249 \n",
"L 42.314331 108.605588 \n",
"L 42.994704 109.333068 \n",
"L 43.674541 110.038721 \n",
"L 44.353875 110.722579 \n",
"L 45.032743 111.384673 \n",
"L 45.711178 112.025033 \n",
"L 46.389219 112.643682 \n",
"L 47.066898 113.240644 \n",
"L 47.74425 113.815941 \n",
"L 48.421312 114.36959 \n",
"L 49.098118 114.901611 \n",
"L 49.774704 115.412016 \n",
"L 50.451104 115.900818 \n",
"L 51.127353 116.368029 \n",
"L 51.803488 116.813651 \n",
"L 52.479542 117.237696 \n",
"L 53.155551 117.640163 \n",
"L 53.831551 118.021055 \n",
"L 54.507577 118.380372 \n",
"L 55.183663 118.718108 \n",
"L 55.859846 119.034257 \n",
"L 56.536161 119.328813 \n",
"L 57.212643 119.601764 \n",
"L 57.889328 119.853099 \n",
"L 58.566254 120.082803 \n",
"L 59.243452 120.290857 \n",
"L 59.920961 120.477243 \n",
"L 60.598816 120.64194 \n",
"L 61.277053 120.784923 \n",
"L 61.955709 120.906165 \n",
"L 62.63482 121.005638 \n",
"L 63.314422 121.08331 \n",
"L 63.994552 121.13915 \n",
"L 64.675247 121.17312 \n",
"L 65.356543 121.185182 \n",
"L 66.038477 121.175298 \n",
"L 66.721087 121.14342 \n",
"L 67.404411 121.089506 \n",
"L 68.088485 121.013508 \n",
"L 68.773346 120.915374 \n",
"L 69.459035 120.795054 \n",
"L 70.145586 120.65249 \n",
"L 70.833041 120.487625 \n",
"L 71.521438 120.300398 \n",
"L 72.210812 120.090746 \n",
"L 72.901207 119.858606 \n",
"L 73.592658 119.603907 \n",
"L 74.285208 119.326579 \n",
"L 74.978895 119.026548 \n",
"L 75.673757 118.703739 \n",
"L 76.369837 118.358074 \n",
"L 77.067177 117.989469 \n",
"L 77.765813 117.597841 \n",
"L 78.465788 117.183103 \n",
"L 79.167147 116.745165 \n",
"L 79.869926 116.283934 \n",
"L 80.574169 115.799314 \n",
"L 81.279922 115.291208 \n",
"L 81.987221 114.759515 \n",
"L 82.696114 114.204128 \n",
"L 83.406645 113.624942 \n",
"L 84.118853 113.021844 \n",
"L 84.832785 112.394724 \n",
"L 85.548487 111.743462 \n",
"L 86.266001 111.06794 \n",
"L 86.985373 110.368036 \n",
"L 87.706652 109.64362 \n",
"L 88.429879 108.894568 \n",
"L 89.155103 108.120743 \n",
"L 89.882371 107.322012 \n",
"L 90.611733 106.498229 \n",
"L 91.343233 105.649259 \n",
"L 92.07692 104.774951 \n",
"L 92.812847 103.875155 \n",
"L 93.551058 102.94972 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 28.279656 68.264617 \n",
"L 28.985001 69.583979 \n",
"L 29.688929 70.879613 \n",
"L 30.391474 72.151607 \n",
"L 31.092677 73.400058 \n",
"L 31.792577 74.62506 \n",
"L 32.491208 75.82669 \n",
"L 33.188611 77.005041 \n",
"L 33.884822 78.160191 \n",
"L 34.579881 79.292228 \n",
"L 35.27382 80.401219 \n",
"L 35.96668 81.487244 \n",
"L 36.658498 82.550377 \n",
"L 37.349307 83.590684 \n",
"L 38.039147 84.608235 \n",
"L 38.728055 85.603096 \n",
"L 39.416063 86.575325 \n",
"L 40.103211 87.524986 \n",
"L 40.789535 88.45214 \n",
"L 41.475069 89.356836 \n",
"L 42.159849 90.239131 \n",
"L 42.843914 91.099077 \n",
"L 43.527296 91.936718 \n",
"L 44.210031 92.752104 \n",
"L 44.892158 93.545279 \n",
"L 45.573708 94.316282 \n",
"L 46.254719 95.065153 \n",
"L 46.935227 95.791931 \n",
"L 47.615265 96.496649 \n",
"L 48.29487 97.179339 \n",
"L 48.974077 97.840032 \n",
"L 49.65292 98.478756 \n",
"L 50.331436 99.095536 \n",
"L 51.00966 99.690396 \n",
"L 51.687626 100.263355 \n",
"L 52.36537 100.814434 \n",
"L 53.042927 101.343647 \n",
"L 53.720332 101.851011 \n",
"L 54.397621 102.336537 \n",
"L 55.074829 102.800235 \n",
"L 55.75199 103.242109 \n",
"L 56.42914 103.662169 \n",
"L 57.106315 104.060415 \n",
"L 57.78355 104.43685 \n",
"L 58.46088 104.791468 \n",
"L 59.13834 105.12427 \n",
"L 59.815968 105.435246 \n",
"L 60.493797 105.724389 \n",
"L 61.171863 105.991687 \n",
"L 61.850203 106.23713 \n",
"L 62.528854 106.4607 \n",
"L 63.207848 106.662379 \n",
"L 63.887224 106.842147 \n",
"L 64.567017 106.999983 \n",
"L 65.247264 107.135861 \n",
"L 65.928001 107.249754 \n",
"L 66.609265 107.341632 \n",
"L 67.291091 107.411465 \n",
"L 67.973519 107.459216 \n",
"L 68.656583 107.484851 \n",
"L 69.340321 107.488329 \n",
"L 70.02477 107.46961 \n",
"L 70.709969 107.428648 \n",
"L 71.395954 107.365399 \n",
"L 72.082764 107.279812 \n",
"L 72.770435 107.171838 \n",
"L 73.459008 107.04142 \n",
"L 74.148519 106.888504 \n",
"L 74.839008 106.71303 \n",
"L 75.530514 106.514936 \n",
"L 76.223074 106.294157 \n",
"L 76.91673 106.05063 \n",
"L 77.611519 105.784283 \n",
"L 78.307483 105.495044 \n",
"L 79.004662 105.182837 \n",
"L 79.703094 104.847588 \n",
"L 80.402821 104.489214 \n",
"L 81.103886 104.107633 \n",
"L 81.806327 103.702759 \n",
"L 82.510186 103.274504 \n",
"L 83.215508 102.822776 \n",
"L 83.92233 102.34748 \n",
"L 84.630698 101.848521 \n",
"L 85.340655 101.325797 \n",
"L 86.052242 100.779206 \n",
"L 86.765503 100.208643 \n",
"L 87.480485 99.613995 \n",
"L 88.197227 98.995153 \n",
"L 88.915777 98.352002 \n",
"L 89.636181 97.68442 \n",
"L 90.358481 96.99229 \n",
"L 91.082725 96.275485 \n",
"L 91.808961 95.533874 \n",
"L 92.537231 94.767333 \n",
"L 93.267586 93.97572 \n",
"L 94.000072 93.1589 \n",
"L 94.73474 92.316728 \n",
"L 95.471634 91.449063 \n",
"L 96.210806 90.555755 \n",
"L 96.952308 89.63665 \n",
"L 97.696184 88.691595 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 32.320305 56.954651 \n",
"L 33.022734 58.276188 \n",
"L 33.72382 59.573842 \n",
"L 34.423599 60.847702 \n",
"L 35.122109 62.097864 \n",
"L 35.819392 63.32442 \n",
"L 36.515479 64.527451 \n",
"L 37.210411 65.707048 \n",
"L 37.904225 66.86329 \n",
"L 38.59696 67.996263 \n",
"L 39.288648 69.106036 \n",
"L 39.979329 70.192689 \n",
"L 40.669041 71.256295 \n",
"L 41.357817 72.29692 \n",
"L 42.045695 73.314635 \n",
"L 42.732712 74.309505 \n",
"L 43.418902 75.281589 \n",
"L 44.104303 76.23095 \n",
"L 44.788951 77.157649 \n",
"L 45.47288 78.061736 \n",
"L 46.156126 78.943266 \n",
"L 46.838728 79.802293 \n",
"L 47.520716 80.638859 \n",
"L 48.202129 81.453014 \n",
"L 48.883004 82.244804 \n",
"L 49.563372 83.014266 \n",
"L 50.24327 83.761441 \n",
"L 50.922735 84.486367 \n",
"L 51.601802 85.189076 \n",
"L 52.280503 85.869601 \n",
"L 52.958877 86.527974 \n",
"L 53.636956 87.16422 \n",
"L 54.314778 87.778365 \n",
"L 54.992377 88.370433 \n",
"L 55.669787 88.940443 \n",
"L 56.347045 89.488416 \n",
"L 57.024184 90.014365 \n",
"L 57.701241 90.518306 \n",
"L 58.378251 91.000251 \n",
"L 59.055248 91.460207 \n",
"L 59.732268 91.898183 \n",
"L 60.409347 92.314184 \n",
"L 61.086518 92.708212 \n",
"L 61.763819 93.080267 \n",
"L 62.441284 93.430347 \n",
"L 63.118949 93.758448 \n",
"L 63.796849 94.064562 \n",
"L 64.475021 94.348681 \n",
"L 65.153499 94.610794 \n",
"L 65.832319 94.850887 \n",
"L 66.511519 95.068946 \n",
"L 67.191133 95.264949 \n",
"L 67.871197 95.438878 \n",
"L 68.551748 95.590709 \n",
"L 69.232822 95.720418 \n",
"L 69.914455 95.827975 \n",
"L 70.596685 95.913352 \n",
"L 71.279548 95.976516 \n",
"L 71.963081 96.017432 \n",
"L 72.647321 96.036063 \n",
"L 73.332306 96.032369 \n",
"L 74.018071 96.006309 \n",
"L 74.704657 95.957837 \n",
"L 75.392099 95.886905 \n",
"L 76.080437 95.793466 \n",
"L 76.769707 95.677467 \n",
"L 77.459949 95.538852 \n",
"L 78.151201 95.377565 \n",
"L 78.843502 95.193547 \n",
"L 79.536891 94.986734 \n",
"L 80.231407 94.757063 \n",
"L 80.92709 94.504465 \n",
"L 81.623979 94.22887 \n",
"L 82.322115 93.930206 \n",
"L 83.021538 93.608396 \n",
"L 83.722287 93.263363 \n",
"L 84.424405 92.895026 \n",
"L 85.127934 92.503299 \n",
"L 85.832911 92.088099 \n",
"L 86.539382 91.649335 \n",
"L 87.247389 91.186912 \n",
"L 87.956971 90.700738 \n",
"L 88.668173 90.190715 \n",
"L 89.381039 89.656739 \n",
"L 90.095611 89.09871 \n",
"L 90.811932 88.516518 \n",
"L 91.53005 87.910053 \n",
"L 92.250005 87.279202 \n",
"L 92.971844 86.62385 \n",
"L 93.695615 85.943876 \n",
"L 94.421359 85.239157 \n",
"L 95.149124 84.50957 \n",
"L 95.878959 83.754979 \n",
"L 96.610908 82.97526 \n",
"L 97.34502 82.170271 \n",
"L 98.081342 81.339876 \n",
"L 98.819927 80.483926 \n",
"L 99.560818 79.602281 \n",
"L 100.304066 78.694789 \n",
"L 101.049726 77.761295 \n",
"L 101.797842 76.801644 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 36.419138 47.985383 \n",
"L 37.117425 49.30585 \n",
"L 37.814449 50.602365 \n",
"L 38.510244 51.875016 \n",
"L 39.204848 53.123897 \n",
"L 39.898302 54.349103 \n",
"L 40.590639 55.550712 \n",
"L 41.281897 56.728815 \n",
"L 41.972113 57.883491 \n",
"L 42.661326 59.014827 \n",
"L 43.349568 60.12289 \n",
"L 44.036879 61.207761 \n",
"L 44.723295 62.269512 \n",
"L 45.40885 63.308208 \n",
"L 46.093581 64.32392 \n",
"L 46.777526 65.316713 \n",
"L 47.460718 66.286648 \n",
"L 48.143192 67.233785 \n",
"L 48.824988 68.158184 \n",
"L 49.506137 69.059896 \n",
"L 50.186676 69.938976 \n",
"L 50.866642 70.795477 \n",
"L 51.546067 71.629442 \n",
"L 52.224988 72.44092 \n",
"L 52.903443 73.229956 \n",
"L 53.581462 73.996587 \n",
"L 54.259082 74.740855 \n",
"L 54.936339 75.462795 \n",
"L 55.613268 76.162442 \n",
"L 56.289903 76.839826 \n",
"L 56.966279 77.49498 \n",
"L 57.642431 78.127929 \n",
"L 58.318396 78.738698 \n",
"L 58.994206 79.327311 \n",
"L 59.669897 79.893787 \n",
"L 60.345504 80.438145 \n",
"L 61.021062 80.9604 \n",
"L 61.696607 81.460567 \n",
"L 62.372172 81.938657 \n",
"L 63.047794 82.394678 \n",
"L 63.723506 82.828638 \n",
"L 64.399345 83.240541 \n",
"L 65.075345 83.630389 \n",
"L 65.751542 83.998182 \n",
"L 66.427971 84.343918 \n",
"L 67.104668 84.667592 \n",
"L 67.781667 84.969197 \n",
"L 68.459005 85.248724 \n",
"L 69.136717 85.506162 \n",
"L 69.814838 85.741495 \n",
"L 70.493407 85.95471 \n",
"L 71.172456 86.145785 \n",
"L 71.852022 86.314702 \n",
"L 72.532142 86.461435 \n",
"L 73.212852 86.585961 \n",
"L 73.894189 86.68825 \n",
"L 74.576189 86.768273 \n",
"L 75.258889 86.825997 \n",
"L 75.942326 86.861387 \n",
"L 76.626536 86.874404 \n",
"L 77.311557 86.86501 \n",
"L 77.997426 86.833161 \n",
"L 78.684182 86.778813 \n",
"L 79.371862 86.701918 \n",
"L 80.060504 86.602427 \n",
"L 80.750144 86.480287 \n",
"L 81.440824 86.335442 \n",
"L 82.13258 86.167837 \n",
"L 82.825452 85.97741 \n",
"L 83.51948 85.764099 \n",
"L 84.2147 85.527838 \n",
"L 84.911155 85.268561 \n",
"L 85.608883 84.986196 \n",
"L 86.307926 84.68067 \n",
"L 87.008322 84.351908 \n",
"L 87.710112 83.99983 \n",
"L 88.413338 83.624356 \n",
"L 89.118042 83.2254 \n",
"L 89.824263 82.802877 \n",
"L 90.532045 82.356697 \n",
"L 91.24143 81.886765 \n",
"L 91.952459 81.392988 \n",
"L 92.665176 80.875268 \n",
"L 93.379626 80.333501 \n",
"L 94.095848 79.767585 \n",
"L 94.81389 79.177411 \n",
"L 95.533796 78.56287 \n",
"L 96.255608 77.923846 \n",
"L 96.979373 77.260225 \n",
"L 97.705139 76.571884 \n",
"L 98.432948 75.858703 \n",
"L 99.162847 75.120556 \n",
"L 99.894886 74.357308 \n",
"L 100.629109 73.568834 \n",
"L 101.365564 72.754992 \n",
"L 102.104301 71.915644 \n",
"L 102.84537 71.050643 \n",
"L 103.588816 70.159849 \n",
"L 104.334691 69.243107 \n",
"L 105.083048 68.300263 \n",
"L 105.833933 67.331163 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 40.554333 41.378444 \n",
"L 41.247296 42.694628 \n",
"L 41.939079 43.986875 \n",
"L 42.629714 45.255272 \n",
"L 43.319239 46.499913 \n",
"L 44.007694 47.720892 \n",
"L 44.695112 48.918287 \n",
"L 45.38153 50.092187 \n",
"L 46.066985 51.242671 \n",
"L 46.751515 52.369825 \n",
"L 47.435153 53.473716 \n",
"L 48.117935 54.554424 \n",
"L 48.799901 55.61202 \n",
"L 49.481081 56.64657 \n",
"L 50.161513 57.658144 \n",
"L 50.841235 58.646804 \n",
"L 51.520278 59.612613 \n",
"L 52.198679 60.555629 \n",
"L 52.876475 61.475912 \n",
"L 53.553697 62.373515 \n",
"L 54.230384 63.248489 \n",
"L 54.90657 64.100887 \n",
"L 55.582288 64.930753 \n",
"L 56.257574 65.738134 \n",
"L 56.932465 66.523076 \n",
"L 57.606991 67.285615 \n",
"L 58.28119 68.025792 \n",
"L 58.955098 68.743643 \n",
"L 59.628746 69.439201 \n",
"L 60.302171 70.112497 \n",
"L 60.975407 70.763563 \n",
"L 61.648488 71.392422 \n",
"L 62.32145 71.999103 \n",
"L 62.994327 72.583625 \n",
"L 63.667153 73.146009 \n",
"L 64.339963 73.686274 \n",
"L 65.012792 74.204433 \n",
"L 65.685675 74.700503 \n",
"L 66.358646 75.174493 \n",
"L 67.03174 75.626411 \n",
"L 67.704992 76.056265 \n",
"L 68.378436 76.464059 \n",
"L 69.052109 76.849795 \n",
"L 69.726044 77.213472 \n",
"L 70.400277 77.555088 \n",
"L 71.074842 77.874638 \n",
"L 71.749776 78.172114 \n",
"L 72.425114 78.447509 \n",
"L 73.10089 78.700809 \n",
"L 73.77714 78.932 \n",
"L 74.453902 79.141068 \n",
"L 75.131209 79.327991 \n",
"L 75.809097 79.49275 \n",
"L 76.487603 79.635322 \n",
"L 77.166762 79.75568 \n",
"L 77.846612 79.853796 \n",
"L 78.527187 79.929641 \n",
"L 79.208526 79.983181 \n",
"L 79.890665 80.014381 \n",
"L 80.573641 80.023203 \n",
"L 81.25749 80.009608 \n",
"L 81.94225 79.973553 \n",
"L 82.627958 79.914993 \n",
"L 83.314653 79.833881 \n",
"L 84.002372 79.730167 \n",
"L 84.691152 79.603799 \n",
"L 85.381033 79.454721 \n",
"L 86.072052 79.282876 \n",
"L 86.764249 79.088205 \n",
"L 87.457663 78.870644 \n",
"L 88.152331 78.630129 \n",
"L 88.848296 78.366592 \n",
"L 89.545594 78.079963 \n",
"L 90.244268 77.770169 \n",
"L 90.944358 77.437133 \n",
"L 91.645902 77.080777 \n",
"L 92.348943 76.701021 \n",
"L 93.053524 76.297779 \n",
"L 93.759682 75.870968 \n",
"L 94.467461 75.420496 \n",
"L 95.176906 74.946268 \n",
"L 95.888054 74.448193 \n",
"L 96.600951 73.926171 \n",
"L 97.315642 73.380101 \n",
"L 98.032166 72.809879 \n",
"L 98.75057 72.215399 \n",
"L 99.470899 71.596549 \n",
"L 100.193195 70.953216 \n",
"L 100.917504 70.285286 \n",
"L 101.643874 69.592636 \n",
"L 102.372347 68.875146 \n",
"L 103.102972 68.13269 \n",
"L 103.835798 67.365136 \n",
"L 104.570867 66.572357 \n",
"L 105.308229 65.754212 \n",
"L 106.047934 64.910565 \n",
"L 106.790031 64.041268 \n",
"L 107.534566 63.146182 \n",
"L 108.281591 62.225153 \n",
"L 109.031158 61.278027 \n",
"L 109.783314 60.30465 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 44.704078 37.130815 \n",
"L 45.390591 38.439576 \n",
"L 46.076007 39.724498 \n",
"L 46.760358 40.985668 \n",
"L 47.443683 42.223178 \n",
"L 48.12602 43.437122 \n",
"L 48.807401 44.627575 \n",
"L 49.487863 45.794628 \n",
"L 50.167442 46.938356 \n",
"L 50.846176 48.058846 \n",
"L 51.524096 49.156164 \n",
"L 52.20124 50.230387 \n",
"L 52.877644 51.281588 \n",
"L 53.55334 52.309829 \n",
"L 54.228365 53.315182 \n",
"L 54.902756 54.297708 \n",
"L 55.576543 55.257466 \n",
"L 56.249763 56.194517 \n",
"L 56.922453 57.10892 \n",
"L 57.594643 58.000723 \n",
"L 58.26637 58.869981 \n",
"L 58.93767 59.716746 \n",
"L 59.608574 60.541059 \n",
"L 60.279118 61.342968 \n",
"L 60.949338 62.122517 \n",
"L 61.619265 62.879743 \n",
"L 62.288935 63.614686 \n",
"L 62.958383 64.327381 \n",
"L 63.627642 65.017861 \n",
"L 64.296746 65.686157 \n",
"L 64.96573 66.332299 \n",
"L 65.634628 66.956311 \n",
"L 66.303474 67.558221 \n",
"L 66.972302 68.138048 \n",
"L 67.641146 68.695812 \n",
"L 68.310041 69.231533 \n",
"L 68.979021 69.745223 \n",
"L 69.64812 70.236897 \n",
"L 70.317374 70.706567 \n",
"L 70.986814 71.154238 \n",
"L 71.656478 71.579919 \n",
"L 72.326398 71.983614 \n",
"L 72.996609 72.365324 \n",
"L 73.667148 72.725049 \n",
"L 74.338046 73.062786 \n",
"L 75.009341 73.37853 \n",
"L 75.681066 73.672274 \n",
"L 76.353256 73.944009 \n",
"L 77.025948 74.193722 \n",
"L 77.699175 74.4214 \n",
"L 78.372974 74.627027 \n",
"L 79.047379 74.810583 \n",
"L 79.722425 74.972048 \n",
"L 80.39815 75.111398 \n",
"L 81.074588 75.228609 \n",
"L 81.751776 75.323652 \n",
"L 82.429749 75.396496 \n",
"L 83.108544 75.44711 \n",
"L 83.788198 75.475458 \n",
"L 84.468747 75.481502 \n",
"L 85.150227 75.465204 \n",
"L 85.832677 75.426521 \n",
"L 86.516132 75.365409 \n",
"L 87.200631 75.281819 \n",
"L 87.886211 75.175704 \n",
"L 88.572909 75.04701 \n",
"L 89.260764 74.895684 \n",
"L 89.949814 74.721669 \n",
"L 90.640098 74.524905 \n",
"L 91.331654 74.305329 \n",
"L 92.02452 74.062878 \n",
"L 92.718738 73.797484 \n",
"L 93.414344 73.509079 \n",
"L 94.111382 73.197588 \n",
"L 94.809889 72.862937 \n",
"L 95.509905 72.505048 \n",
"L 96.211472 72.123842 \n",
"L 96.914633 71.719233 \n",
"L 97.619425 71.291138 \n",
"L 98.325891 70.839467 \n",
"L 99.034076 70.364127 \n",
"L 99.744018 69.865026 \n",
"L 100.455761 69.342067 \n",
"L 101.169351 68.795146 \n",
"L 101.884826 68.224164 \n",
"L 102.602233 67.629014 \n",
"L 103.321618 67.009585 \n",
"L 104.043021 66.365766 \n",
"L 104.766489 65.697444 \n",
"L 105.492069 65.004496 \n",
"L 106.219804 64.286805 \n",
"L 106.949742 63.544245 \n",
"L 107.681931 62.776685 \n",
"L 108.416414 61.984 \n",
"L 109.153241 61.16605 \n",
"L 109.89246 60.322701 \n",
"L 110.634122 59.453806 \n",
"L 111.378272 58.559226 \n",
"L 112.124962 57.638812 \n",
"L 112.874243 56.692408 \n",
"L 113.626163 55.719865 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 48.846937 35.215521 \n",
"L 49.525936 36.513831 \n",
"L 50.203924 37.788482 \n",
"L 50.880931 39.039559 \n",
"L 51.556996 40.267153 \n",
"L 52.232156 41.471354 \n",
"L 52.906442 42.65224 \n",
"L 53.579891 43.809896 \n",
"L 54.252538 44.9444 \n",
"L 54.92442 46.055835 \n",
"L 55.595568 47.144266 \n",
"L 56.266017 48.20977 \n",
"L 56.935806 49.252417 \n",
"L 57.604963 50.27227 \n",
"L 58.273526 51.269398 \n",
"L 58.94153 52.243862 \n",
"L 59.609006 53.195721 \n",
"L 60.27599 54.125034 \n",
"L 60.942517 55.031857 \n",
"L 61.608619 55.916241 \n",
"L 62.27433 56.778238 \n",
"L 62.939686 57.617898 \n",
"L 63.604717 58.435263 \n",
"L 64.26946 59.23038 \n",
"L 64.933949 60.003293 \n",
"L 65.598215 60.754036 \n",
"L 66.262293 61.48265 \n",
"L 66.926217 62.18917 \n",
"L 67.590021 62.873628 \n",
"L 68.253736 63.536053 \n",
"L 68.9174 64.176476 \n",
"L 69.581042 64.79492 \n",
"L 70.244699 65.391412 \n",
"L 70.908404 65.965972 \n",
"L 71.57219 66.51862 \n",
"L 72.236091 67.049372 \n",
"L 72.90014 67.558244 \n",
"L 73.564372 68.045249 \n",
"L 74.228821 68.510397 \n",
"L 74.89352 68.953697 \n",
"L 75.558503 69.375154 \n",
"L 76.223804 69.774773 \n",
"L 76.889458 70.152556 \n",
"L 77.555498 70.508501 \n",
"L 78.221959 70.842607 \n",
"L 78.888875 71.154868 \n",
"L 79.556281 71.445277 \n",
"L 80.22421 71.713824 \n",
"L 80.892699 71.960499 \n",
"L 81.561781 72.185287 \n",
"L 82.231492 72.388172 \n",
"L 82.901866 72.569135 \n",
"L 83.572938 72.728157 \n",
"L 84.244743 72.865213 \n",
"L 84.917318 72.980279 \n",
"L 85.590697 73.073326 \n",
"L 86.264916 73.144327 \n",
"L 86.940012 73.193247 \n",
"L 87.61602 73.220053 \n",
"L 88.292976 73.224708 \n",
"L 88.970917 73.207172 \n",
"L 89.64988 73.167404 \n",
"L 90.3299 73.10536 \n",
"L 91.011016 73.020993 \n",
"L 91.693265 72.914256 \n",
"L 92.376682 72.785096 \n",
"L 93.061308 72.63346 \n",
"L 93.747177 72.459292 \n",
"L 94.434331 72.262533 \n",
"L 95.122807 72.043122 \n",
"L 95.812642 71.800995 \n",
"L 96.503877 71.536087 \n",
"L 97.196549 71.248328 \n",
"L 97.8907 70.937647 \n",
"L 98.586368 70.603971 \n",
"L 99.283593 70.247222 \n",
"L 99.982415 69.867321 \n",
"L 100.682877 69.464186 \n",
"L 101.385016 69.037734 \n",
"L 102.088876 68.587877 \n",
"L 102.794499 68.114521 \n",
"L 103.501924 67.617578 \n",
"L 104.211195 67.09695 \n",
"L 104.922357 66.552539 \n",
"L 105.635448 65.984243 \n",
"L 106.350514 65.391958 \n",
"L 107.067601 64.775577 \n",
"L 107.786749 64.134987 \n",
"L 108.508005 63.470079 \n",
"L 109.231415 62.780732 \n",
"L 109.95702 62.066829 \n",
"L 110.68487 61.328248 \n",
"L 111.415011 60.564859 \n",
"L 112.147488 59.77654 \n",
"L 112.882348 58.963152 \n",
"L 113.61964 58.124563 \n",
"L 114.359415 57.26063 \n",
"L 115.101716 56.371215 \n",
"L 115.846594 55.456171 \n",
"L 116.594103 54.515346 \n",
"L 117.344288 53.548592 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 52.962197 35.582887 \n",
"L 53.632693 36.867869 \n",
"L 54.302262 38.129447 \n",
"L 54.970937 39.367706 \n",
"L 55.638754 40.582734 \n",
"L 56.305748 41.774621 \n",
"L 56.97195 42.943442 \n",
"L 57.637397 44.089281 \n",
"L 58.302123 45.212213 \n",
"L 58.966163 46.31232 \n",
"L 59.629548 47.389666 \n",
"L 60.292314 48.444327 \n",
"L 60.954495 49.476371 \n",
"L 61.616122 50.48586 \n",
"L 62.277231 51.47286 \n",
"L 62.937856 52.437433 \n",
"L 63.598027 53.379636 \n",
"L 64.257779 54.299525 \n",
"L 64.917148 55.197158 \n",
"L 65.576163 56.072583 \n",
"L 66.23486 56.925851 \n",
"L 66.893272 57.757012 \n",
"L 67.551429 58.566107 \n",
"L 68.209367 59.353182 \n",
"L 68.86712 60.118278 \n",
"L 69.524718 60.861432 \n",
"L 70.182195 61.582681 \n",
"L 70.839585 62.282061 \n",
"L 71.496921 62.959602 \n",
"L 72.154234 63.615334 \n",
"L 72.811559 64.249287 \n",
"L 73.468927 64.861484 \n",
"L 74.126374 65.451949 \n",
"L 74.783931 66.020704 \n",
"L 75.44163 66.567767 \n",
"L 76.099507 67.093156 \n",
"L 76.757593 67.596885 \n",
"L 77.415923 68.078966 \n",
"L 78.074528 68.539411 \n",
"L 78.733443 68.978227 \n",
"L 79.3927 69.395419 \n",
"L 80.052334 69.790993 \n",
"L 80.712377 70.16495 \n",
"L 81.372864 70.517289 \n",
"L 82.033828 70.848007 \n",
"L 82.695302 71.1571 \n",
"L 83.357321 71.444561 \n",
"L 84.019919 71.710381 \n",
"L 84.683129 71.954548 \n",
"L 85.346986 72.177048 \n",
"L 86.011526 72.377866 \n",
"L 86.67678 72.556983 \n",
"L 87.342784 72.71438 \n",
"L 88.009573 72.850034 \n",
"L 88.677182 72.96392 \n",
"L 89.345645 73.056011 \n",
"L 90.014998 73.126277 \n",
"L 90.685277 73.174688 \n",
"L 91.356516 73.20121 \n",
"L 92.028752 73.205806 \n",
"L 92.70202 73.188438 \n",
"L 93.376356 73.149065 \n",
"L 94.051797 73.087645 \n",
"L 94.72838 73.004131 \n",
"L 95.40614 72.898476 \n",
"L 96.085113 72.77063 \n",
"L 96.76534 72.62054 \n",
"L 97.446854 72.448152 \n",
"L 98.129696 72.253408 \n",
"L 98.813903 72.036247 \n",
"L 99.499511 71.796608 \n",
"L 100.186561 71.534425 \n",
"L 100.875089 71.249633 \n",
"L 101.565137 70.94216 \n",
"L 102.256742 70.611934 \n",
"L 102.949943 70.25888 \n",
"L 103.644781 69.882921 \n",
"L 104.341298 69.483975 \n",
"L 105.039529 69.061962 \n",
"L 105.739519 68.616795 \n",
"L 106.44131 68.148384 \n",
"L 107.144939 67.656641 \n",
"L 107.85045 67.14147 \n",
"L 108.557887 66.602775 \n",
"L 109.267288 66.040458 \n",
"L 109.978699 65.454414 \n",
"L 110.692165 64.844541 \n",
"L 111.407725 64.210728 \n",
"L 112.125425 63.552867 \n",
"L 112.845312 62.87084 \n",
"L 113.567426 62.164534 \n",
"L 114.291816 61.433829 \n",
"L 115.018528 60.678596 \n",
"L 115.747605 59.898717 \n",
"L 116.479095 59.094057 \n",
"L 117.213046 58.264485 \n",
"L 117.949507 57.409862 \n",
"L 118.688523 56.530054 \n",
"L 119.430144 55.624914 \n",
"L 120.174421 54.694297 \n",
"L 120.9214 53.738056 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 57.030186 38.162323 \n",
"L 57.691268 39.431278 \n",
"L 58.35151 40.677158 \n",
"L 59.010941 41.900045 \n",
"L 59.669597 43.100025 \n",
"L 60.327513 44.277186 \n",
"L 60.984719 45.4316 \n",
"L 61.64125 46.563351 \n",
"L 62.297139 47.672511 \n",
"L 62.952422 48.759161 \n",
"L 63.607128 49.823362 \n",
"L 64.26129 50.865188 \n",
"L 64.914945 51.884707 \n",
"L 65.568121 52.881979 \n",
"L 66.220853 53.857068 \n",
"L 66.873175 54.810035 \n",
"L 67.525116 55.740933 \n",
"L 68.17671 56.649821 \n",
"L 68.827992 57.536754 \n",
"L 69.47899 58.401777 \n",
"L 70.129739 59.244942 \n",
"L 70.780272 60.066297 \n",
"L 71.430619 60.865882 \n",
"L 72.080813 61.643742 \n",
"L 72.730889 62.399918 \n",
"L 73.380875 63.134444 \n",
"L 74.030805 63.847358 \n",
"L 74.680712 64.538694 \n",
"L 75.330629 65.208482 \n",
"L 75.980585 65.856752 \n",
"L 76.630615 66.483531 \n",
"L 77.280749 67.088844 \n",
"L 77.931023 67.672714 \n",
"L 78.581466 68.235161 \n",
"L 79.232111 68.776204 \n",
"L 79.882991 69.29586 \n",
"L 80.534138 69.794142 \n",
"L 81.185586 70.271064 \n",
"L 81.837365 70.726636 \n",
"L 82.48951 71.160864 \n",
"L 83.142052 71.573757 \n",
"L 83.795024 71.965316 \n",
"L 84.448459 72.335545 \n",
"L 85.102391 72.684442 \n",
"L 85.756851 73.012006 \n",
"L 86.411874 73.318231 \n",
"L 87.067492 73.60311 \n",
"L 87.723739 73.866634 \n",
"L 88.380648 74.108794 \n",
"L 89.038253 74.329574 \n",
"L 89.696588 74.528962 \n",
"L 90.355685 74.706937 \n",
"L 91.015579 74.863481 \n",
"L 91.676305 74.998572 \n",
"L 92.337895 75.112186 \n",
"L 93.000385 75.204296 \n",
"L 93.663809 75.274874 \n",
"L 94.328202 75.32389 \n",
"L 94.9936 75.35131 \n",
"L 95.660035 75.3571 \n",
"L 96.327545 75.341221 \n",
"L 96.996163 75.303635 \n",
"L 97.665927 75.2443 \n",
"L 98.336873 75.16317 \n",
"L 99.009035 75.0602 \n",
"L 99.682449 74.935342 \n",
"L 100.357154 74.788542 \n",
"L 101.033184 74.619749 \n",
"L 101.710578 74.428906 \n",
"L 102.389372 74.215955 \n",
"L 103.069604 73.980835 \n",
"L 103.751312 73.723484 \n",
"L 104.434532 73.443835 \n",
"L 105.119305 73.14182 \n",
"L 105.805669 72.81737 \n",
"L 106.49366 72.47041 \n",
"L 107.183319 72.100867 \n",
"L 107.874688 71.708659 \n",
"L 108.567803 71.29371 \n",
"L 109.262704 70.855934 \n",
"L 109.959435 70.395243 \n",
"L 110.658033 69.911553 \n",
"L 111.35854 69.40477 \n",
"L 112.061 68.8748 \n",
"L 112.76545 68.321548 \n",
"L 113.471935 67.744913 \n",
"L 114.1805 67.144793 \n",
"L 114.891182 66.521082 \n",
"L 115.604029 65.873674 \n",
"L 116.319084 65.202455 \n",
"L 117.036389 64.507314 \n",
"L 117.75599 63.788135 \n",
"L 118.477934 63.044793 \n",
"L 119.202262 62.277172 \n",
"L 119.929023 61.485141 \n",
"L 120.658262 60.668573 \n",
"L 121.39003 59.827333 \n",
"L 122.124368 58.961289 \n",
"L 122.861327 58.0703 \n",
"L 123.600958 57.154224 \n",
"L 124.343305 56.212918 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 61.032539 42.864518 \n",
"L 61.683382 44.114955 \n",
"L 62.333469 45.342712 \n",
"L 62.982828 46.547867 \n",
"L 63.631494 47.730506 \n",
"L 64.279502 48.890712 \n",
"L 64.926879 50.028557 \n",
"L 65.573661 51.144122 \n",
"L 66.219879 52.237477 \n",
"L 66.865567 53.3087 \n",
"L 67.510754 54.357851 \n",
"L 68.155474 55.385003 \n",
"L 68.79976 56.39022 \n",
"L 69.443641 57.373561 \n",
"L 70.087149 58.33509 \n",
"L 70.730319 59.274864 \n",
"L 71.373179 60.192936 \n",
"L 72.015762 61.089362 \n",
"L 72.658101 61.964197 \n",
"L 73.300224 62.817484 \n",
"L 73.942165 63.649275 \n",
"L 74.583957 64.459614 \n",
"L 75.225627 65.248542 \n",
"L 75.867209 66.016102 \n",
"L 76.508736 66.762334 \n",
"L 77.150237 67.487272 \n",
"L 77.791743 68.190952 \n",
"L 78.433287 68.873407 \n",
"L 79.074901 69.534668 \n",
"L 79.716614 70.174762 \n",
"L 80.358459 70.793715 \n",
"L 81.000467 71.391555 \n",
"L 81.642671 71.968301 \n",
"L 82.2851 72.523973 \n",
"L 82.927787 73.05859 \n",
"L 83.570764 73.572171 \n",
"L 84.214061 74.064724 \n",
"L 84.857712 74.536266 \n",
"L 85.501747 74.986806 \n",
"L 86.146198 75.416352 \n",
"L 86.791098 75.824908 \n",
"L 87.436478 76.21248 \n",
"L 88.08237 76.579068 \n",
"L 88.728807 76.924675 \n",
"L 89.375821 77.249294 \n",
"L 90.023443 77.552924 \n",
"L 90.671707 77.835557 \n",
"L 91.320646 78.097184 \n",
"L 91.970291 78.337796 \n",
"L 92.620676 78.55738 \n",
"L 93.271834 78.755922 \n",
"L 93.923797 78.933402 \n",
"L 94.576598 79.089803 \n",
"L 95.230272 79.225104 \n",
"L 95.88485 79.339282 \n",
"L 96.540369 79.432311 \n",
"L 97.196859 79.504163 \n",
"L 97.854357 79.55481 \n",
"L 98.512896 79.584217 \n",
"L 99.17251 79.592354 \n",
"L 99.833233 79.579182 \n",
"L 100.495101 79.544666 \n",
"L 101.158148 79.48876 \n",
"L 101.82241 79.411427 \n",
"L 102.487922 79.312618 \n",
"L 103.154718 79.192288 \n",
"L 103.822835 79.050385 \n",
"L 104.492309 78.88686 \n",
"L 105.163176 78.701658 \n",
"L 105.835473 78.494721 \n",
"L 106.509235 78.265991 \n",
"L 107.184501 78.015409 \n",
"L 107.861306 77.742909 \n",
"L 108.539689 77.448426 \n",
"L 109.219688 77.131892 \n",
"L 109.90134 76.793237 \n",
"L 110.584683 76.432386 \n",
"L 111.269759 76.049264 \n",
"L 111.956602 75.643794 \n",
"L 112.645253 75.215895 \n",
"L 113.335755 74.765482 \n",
"L 114.028142 74.292471 \n",
"L 114.722458 73.796774 \n",
"L 115.418745 73.278298 \n",
"L 116.117039 72.736952 \n",
"L 116.817385 72.172639 \n",
"L 117.519825 71.585258 \n",
"L 118.224399 70.974708 \n",
"L 118.93115 70.340886 \n",
"L 119.640123 69.683683 \n",
"L 120.351357 69.00299 \n",
"L 121.064899 68.298694 \n",
"L 121.780794 67.570676 \n",
"L 122.499083 66.818824 \n",
"L 123.219813 66.043009 \n",
"L 123.943029 65.243109 \n",
"L 124.66878 64.418994 \n",
"L 125.397108 63.570536 \n",
"L 126.128062 62.697599 \n",
"L 126.861691 61.800045 \n",
"L 127.598039 60.877736 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" <path d=\"M 64.952423 49.583939 \n",
"L 65.59229 50.813593 \n",
"L 66.231483 52.02102 \n",
"L 66.870029 53.206298 \n",
"L 67.507962 54.369509 \n",
"L 68.145316 55.510735 \n",
"L 68.782117 56.630043 \n",
"L 69.4184 57.727512 \n",
"L 70.054194 58.803211 \n",
"L 70.689534 59.857215 \n",
"L 71.324447 60.889583 \n",
"L 71.958965 61.900383 \n",
"L 72.59312 62.889681 \n",
"L 73.226942 63.857531 \n",
"L 73.86046 64.803996 \n",
"L 74.49371 65.729131 \n",
"L 75.126716 66.632989 \n",
"L 75.759513 67.515623 \n",
"L 76.392131 68.377085 \n",
"L 77.024599 69.21742 \n",
"L 77.656949 70.036674 \n",
"L 78.289212 70.834894 \n",
"L 78.921416 71.612118 \n",
"L 79.553594 72.368387 \n",
"L 80.185776 73.103741 \n",
"L 80.817991 73.818212 \n",
"L 81.450271 74.511835 \n",
"L 82.082647 75.184644 \n",
"L 82.715148 75.836667 \n",
"L 83.347804 76.467931 \n",
"L 83.980648 77.078462 \n",
"L 84.613709 77.668285 \n",
"L 85.247018 78.237421 \n",
"L 85.880606 78.785889 \n",
"L 86.514502 79.313707 \n",
"L 87.148739 79.820891 \n",
"L 87.783346 80.307455 \n",
"L 88.418355 80.773411 \n",
"L 89.053798 81.218769 \n",
"L 89.689703 81.643536 \n",
"L 90.326103 82.047717 \n",
"L 90.963029 82.431319 \n",
"L 91.600512 82.794341 \n",
"L 92.238583 83.136783 \n",
"L 92.877274 83.458645 \n",
"L 93.516616 83.759922 \n",
"L 94.156642 84.040607 \n",
"L 94.797382 84.300692 \n",
"L 95.438868 84.540167 \n",
"L 96.081133 84.759022 \n",
"L 96.724209 84.957241 \n",
"L 97.368127 85.134807 \n",
"L 98.01292 85.291702 \n",
"L 98.658621 85.427908 \n",
"L 99.305261 85.543402 \n",
"L 99.952875 85.638157 \n",
"L 100.601495 85.712149 \n",
"L 101.251153 85.765349 \n",
"L 101.901885 85.797728 \n",
"L 102.553721 85.809251 \n",
"L 103.206697 85.799883 \n",
"L 103.860847 85.76959 \n",
"L 104.516204 85.71833 \n",
"L 105.172803 85.646064 \n",
"L 105.830677 85.552747 \n",
"L 106.489862 85.438334 \n",
"L 107.150393 85.302779 \n",
"L 107.812303 85.14603 \n",
"L 108.475631 84.968036 \n",
"L 109.14041 84.768742 \n",
"L 109.806675 84.548092 \n",
"L 110.474465 84.306027 \n",
"L 111.143813 84.042487 \n",
"L 111.814759 83.757406 \n",
"L 112.487338 83.45072 \n",
"L 113.161586 83.122362 \n",
"L 113.837542 82.77226 \n",
"L 114.515245 82.400342 \n",
"L 115.19473 82.00653 \n",
"L 115.876036 81.590751 \n",
"L 116.559205 81.152921 \n",
"L 117.244272 80.69296 \n",
"L 117.931277 80.210782 \n",
"L 118.620262 79.706298 \n",
"L 119.311264 79.179421 \n",
"L 120.004325 78.630056 \n",
"L 120.699487 78.058108 \n",
"L 121.396788 77.463478 \n",
"L 122.09627 76.846068 \n",
"L 122.797979 76.205772 \n",
"L 123.501952 75.542486 \n",
"L 124.208234 74.8561 \n",
"L 124.91687 74.146501 \n",
"L 125.6279 73.413579 \n",
"L 126.341369 72.657212 \n",
"L 127.057322 71.877282 \n",
"L 127.775806 71.073663 \n",
"L 128.496862 70.246234 \n",
"L 129.220538 69.394861 \n",
"L 129.946883 68.519414 \n",
"L 130.675939 67.619758 \n",
"\" clip-path=\"url(#p366f7efb7b)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <defs>\n",
" <clipPath id=\"p366f7efb7b\">\n",
" <rect x=\"7.2\" y=\"7.2\" width=\"138.6\" height=\"138.6\"/>\n",
" </clipPath>\n",
" </defs>\n",
"</svg>\n"
],
"text/plain": [
"<Figure size 350x250 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x, y = torch.meshgrid(\n",
" torch.linspace(-1.0, 1.0, 101), torch.linspace(-1.0, 1.0, 101))\n",
"z = x**2 - y**2\n",
"\n",
"ax = d2l.plt.figure().add_subplot(111, projection='3d')\n",
"ax.plot_wireframe(x, y, z, **{'rstride': 10, 'cstride': 10})\n",
"ax.plot([0], [0], [0], 'rx')\n",
"ticks = [-1, 0, 1]\n",
"d2l.plt.xticks(ticks)\n",
"d2l.plt.yticks(ticks)\n",
"ax.set_zticks(ticks)\n",
"d2l.plt.xlabel('x')\n",
"d2l.plt.ylabel('y');"
]
},
{
"cell_type": "markdown",
"id": "254762fc",
"metadata": {
"origin_pos": 15
},
"source": [
"We assume that the input of a function is a $k$-dimensional vector and its\n",
"output is a scalar, so its Hessian matrix will have $k$ eigenvalues.\n",
"The solution of the\n",
"function could be a local minimum, a local maximum, or a saddle point at a\n",
"position where the function gradient is zero:\n",
"\n",
"* When the eigenvalues of the function's Hessian matrix at the zero-gradient position are all positive, we have a local minimum for the function.\n",
"* When the eigenvalues of the function's Hessian matrix at the zero-gradient position are all negative, we have a local maximum for the function.\n",
"* When the eigenvalues of the function's Hessian matrix at the zero-gradient position are negative and positive, we have a saddle point for the function.\n",
"\n",
"For high-dimensional problems the likelihood that at least *some* of the eigenvalues are negative is quite high. This makes saddle points more likely than local minima. We will discuss some exceptions to this situation in the next section when introducing convexity. In short, convex functions are those where the eigenvalues of the Hessian are never negative. Sadly, though, most deep learning problems do not fall into this category. Nonetheless it is a great tool to study optimization algorithms.\n",
"\n",
"### Vanishing Gradients\n",
"\n",
"Probably the most insidious problem to encounter is the vanishing gradient.\n",
"Recall our commonly-used activation functions and their derivatives in :numref:`subsec_activation-functions`.\n",
"For instance, assume that we want to minimize the function $f(x) = \\tanh(x)$ and we happen to get started at $x = 4$. As we can see, the gradient of $f$ is close to nil.\n",
"More specifically, $f'(x) = 1 - \\tanh^2(x)$ and thus $f'(4) = 0.0013$.\n",
"Consequently, optimization will get stuck for a long time before we make progress. This turns out to be one of the reasons that training deep learning models was quite tricky prior to the introduction of the ReLU activation function.\n"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "e609dac1",
"metadata": {
"execution": {
"iopub.execute_input": "2023-08-18T19:42:01.956202Z",
"iopub.status.busy": "2023-08-18T19:42:01.955189Z",
"iopub.status.idle": "2023-08-18T19:42:02.281005Z",
"shell.execute_reply": "2023-08-18T19:42:02.280072Z"
},
"origin_pos": 16,
"tab": [
"pytorch"
]
},
"outputs": [
{
"data": {
"image/svg+xml": [
"<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.314464pt\" height=\"183.35625pt\" viewBox=\"0 0 263.314464 183.35625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
" <metadata>\n",
" <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
" <cc:Work>\n",
" <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
" <dc:date>2023-08-18T19:42:02.235257</dc:date>\n",
" <dc:format>image/svg+xml</dc:format>\n",
" <dc:creator>\n",
" <cc:Agent>\n",
" <dc:title>Matplotlib v3.7.2, https://matplotlib.org/</dc:title>\n",
" </cc:Agent>\n",
" </dc:creator>\n",
" </cc:Work>\n",
" </rdf:RDF>\n",
" </metadata>\n",
" <defs>\n",
" <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
" </defs>\n",
" <g id=\"figure_1\">\n",
" <g id=\"patch_1\">\n",
" <path d=\"M 0 183.35625 \n",
"L 263.314464 183.35625 \n",
"L 263.314464 0 \n",
"L 0 0 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"axes_1\">\n",
" <g id=\"patch_2\">\n",
" <path d=\"M 52.160938 145.8 \n",
"L 247.460938 145.8 \n",
"L 247.460938 7.2 \n",
"L 52.160938 7.2 \n",
"z\n",
"\" style=\"fill: #ffffff\"/>\n",
" </g>\n",
" <g id=\"matplotlib.axis_1\">\n",
" <g id=\"xtick_1\">\n",
" <g id=\"line2d_1\">\n",
" <path d=\"M 61.03821 145.8 \n",
"L 61.03821 7.2 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_2\">\n",
" <defs>\n",
" <path id=\"mfc03ac11c5\" d=\"M 0 0 \n",
"L 0 3.5 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#mfc03ac11c5\" x=\"61.03821\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_1\">\n",
" <!-- −2 -->\n",
" <g transform=\"translate(53.667116 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n",
"L 4684 2272 \n",
"L 4684 1741 \n",
"L 678 1741 \n",
"L 678 2272 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
"L 3431 531 \n",
"L 3431 0 \n",
"L 469 0 \n",
"L 469 531 \n",
"Q 828 903 1448 1529 \n",
"Q 2069 2156 2228 2338 \n",
"Q 2531 2678 2651 2914 \n",
"Q 2772 3150 2772 3378 \n",
"Q 2772 3750 2511 3984 \n",
"Q 2250 4219 1831 4219 \n",
"Q 1534 4219 1204 4116 \n",
"Q 875 4013 500 3803 \n",
"L 500 4441 \n",
"Q 881 4594 1212 4672 \n",
"Q 1544 4750 1819 4750 \n",
"Q 2544 4750 2975 4387 \n",
"Q 3406 4025 3406 3419 \n",
"Q 3406 3131 3298 2873 \n",
"Q 3191 2616 2906 2266 \n",
"Q 2828 2175 2409 1742 \n",
"Q 1991 1309 1228 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_2\">\n",
" <g id=\"line2d_3\">\n",
" <path d=\"M 111.838056 145.8 \n",
"L 111.838056 7.2 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_4\">\n",
" <g>\n",
" <use xlink:href=\"#mfc03ac11c5\" x=\"111.838056\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_2\">\n",
" <!-- 0 -->\n",
" <g transform=\"translate(108.656806 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
"Q 1547 4250 1301 3770 \n",
"Q 1056 3291 1056 2328 \n",
"Q 1056 1369 1301 889 \n",
"Q 1547 409 2034 409 \n",
"Q 2525 409 2770 889 \n",
"Q 3016 1369 3016 2328 \n",
"Q 3016 3291 2770 3770 \n",
"Q 2525 4250 2034 4250 \n",
"z\n",
"M 2034 4750 \n",
"Q 2819 4750 3233 4129 \n",
"Q 3647 3509 3647 2328 \n",
"Q 3647 1150 3233 529 \n",
"Q 2819 -91 2034 -91 \n",
"Q 1250 -91 836 529 \n",
"Q 422 1150 422 2328 \n",
"Q 422 3509 836 4129 \n",
"Q 1250 4750 2034 4750 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_3\">\n",
" <g id=\"line2d_5\">\n",
" <path d=\"M 162.637901 145.8 \n",
"L 162.637901 7.2 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_6\">\n",
" <g>\n",
" <use xlink:href=\"#mfc03ac11c5\" x=\"162.637901\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_3\">\n",
" <!-- 2 -->\n",
" <g transform=\"translate(159.456651 160.398438) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-32\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"xtick_4\">\n",
" <g id=\"line2d_7\">\n",
" <path d=\"M 213.437747 145.8 \n",
"L 213.437747 7.2 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_8\">\n",
" <g>\n",
" <use xlink:href=\"#mfc03ac11c5\" x=\"213.437747\" y=\"145.8\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_4\">\n",
" <!-- 4 -->\n",
" <g transform=\"translate(210.256497 160.398438) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
"L 825 1625 \n",
"L 2419 1625 \n",
"L 2419 4116 \n",
"z\n",
"M 2253 4666 \n",
"L 3047 4666 \n",
"L 3047 1625 \n",
"L 3713 1625 \n",
"L 3713 1100 \n",
"L 3047 1100 \n",
"L 3047 0 \n",
"L 2419 0 \n",
"L 2419 1100 \n",
"L 313 1100 \n",
"L 313 1709 \n",
"L 2253 4666 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-34\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_5\">\n",
" <!-- x -->\n",
" <g transform=\"translate(146.851563 174.076563) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n",
"L 2247 1797 \n",
"L 3578 0 \n",
"L 2900 0 \n",
"L 1881 1375 \n",
"L 863 0 \n",
"L 184 0 \n",
"L 1544 1831 \n",
"L 300 3500 \n",
"L 978 3500 \n",
"L 1906 2253 \n",
"L 2834 3500 \n",
"L 3513 3500 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-78\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"matplotlib.axis_2\">\n",
" <g id=\"ytick_1\">\n",
" <g id=\"line2d_9\">\n",
" <path d=\"M 52.160938 141.807879 \n",
"L 247.460938 141.807879 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_10\">\n",
" <defs>\n",
" <path id=\"m26cee734f3\" d=\"M 0 0 \n",
"L -3.5 0 \n",
"\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </defs>\n",
" <g>\n",
" <use xlink:href=\"#m26cee734f3\" x=\"52.160938\" y=\"141.807879\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_6\">\n",
" <!-- −1.0 -->\n",
" <g transform=\"translate(20.878125 145.607098) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
"L 1825 531 \n",
"L 1825 4091 \n",
"L 703 3866 \n",
"L 703 4441 \n",
"L 1819 4666 \n",
"L 2450 4666 \n",
"L 2450 531 \n",
"L 3481 531 \n",
"L 3481 0 \n",
"L 794 0 \n",
"L 794 531 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
"L 1344 794 \n",
"L 1344 0 \n",
"L 684 0 \n",
"L 684 794 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"179.199219\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_2\">\n",
" <g id=\"line2d_11\">\n",
" <path d=\"M 52.160938 109.729424 \n",
"L 247.460938 109.729424 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_12\">\n",
" <g>\n",
" <use xlink:href=\"#m26cee734f3\" x=\"52.160938\" y=\"109.729424\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_7\">\n",
" <!-- −0.5 -->\n",
" <g transform=\"translate(20.878125 113.528642) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n",
"L 3169 4666 \n",
"L 3169 4134 \n",
"L 1269 4134 \n",
"L 1269 2991 \n",
"Q 1406 3038 1543 3061 \n",
"Q 1681 3084 1819 3084 \n",
"Q 2600 3084 3056 2656 \n",
"Q 3513 2228 3513 1497 \n",
"Q 3513 744 3044 326 \n",
"Q 2575 -91 1722 -91 \n",
"Q 1428 -91 1123 -41 \n",
"Q 819 9 494 109 \n",
"L 494 744 \n",
"Q 775 591 1075 516 \n",
"Q 1375 441 1709 441 \n",
"Q 2250 441 2565 725 \n",
"Q 2881 1009 2881 1497 \n",
"Q 2881 1984 2565 2268 \n",
"Q 2250 2553 1709 2553 \n",
"Q 1456 2553 1204 2497 \n",
"Q 953 2441 691 2322 \n",
"L 691 4666 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-2212\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_3\">\n",
" <g id=\"line2d_13\">\n",
" <path d=\"M 52.160938 77.650968 \n",
"L 247.460938 77.650968 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_14\">\n",
" <g>\n",
" <use xlink:href=\"#m26cee734f3\" x=\"52.160938\" y=\"77.650968\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_8\">\n",
" <!-- 0.0 -->\n",
" <g transform=\"translate(29.257812 81.450187) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_4\">\n",
" <g id=\"line2d_15\">\n",
" <path d=\"M 52.160938 45.572513 \n",
"L 247.460938 45.572513 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_16\">\n",
" <g>\n",
" <use xlink:href=\"#m26cee734f3\" x=\"52.160938\" y=\"45.572513\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_9\">\n",
" <!-- 0.5 -->\n",
" <g transform=\"translate(29.257812 49.371732) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-30\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"ytick_5\">\n",
" <g id=\"line2d_17\">\n",
" <path d=\"M 52.160938 13.494057 \n",
"L 247.460938 13.494057 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"line2d_18\">\n",
" <g>\n",
" <use xlink:href=\"#m26cee734f3\" x=\"52.160938\" y=\"13.494057\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_10\">\n",
" <!-- 1.0 -->\n",
" <g transform=\"translate(29.257812 17.293276) scale(0.1 -0.1)\">\n",
" <use xlink:href=\"#DejaVuSans-31\"/>\n",
" <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n",
" <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"text_11\">\n",
" <!-- f(x) -->\n",
" <g transform=\"translate(14.798438 85.121094) rotate(-90) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
"L 2375 4384 \n",
"L 1825 4384 \n",
"Q 1516 4384 1395 4259 \n",
"Q 1275 4134 1275 3809 \n",
"L 1275 3500 \n",
"L 2222 3500 \n",
"L 2222 3053 \n",
"L 1275 3053 \n",
"L 1275 0 \n",
"L 697 0 \n",
"L 697 3053 \n",
"L 147 3053 \n",
"L 147 3500 \n",
"L 697 3500 \n",
"L 697 3744 \n",
"Q 697 4328 969 4595 \n",
"Q 1241 4863 1831 4863 \n",
"L 2375 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
"Q 1566 4138 1362 3434 \n",
"Q 1159 2731 1159 2009 \n",
"Q 1159 1288 1364 580 \n",
"Q 1569 -128 1984 -844 \n",
"L 1484 -844 \n",
"Q 1016 -109 783 600 \n",
"Q 550 1309 550 2009 \n",
"Q 550 2706 781 3412 \n",
"Q 1013 4119 1484 4856 \n",
"L 1984 4856 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
"L 1013 4856 \n",
"Q 1481 4119 1714 3412 \n",
"Q 1947 2706 1947 2009 \n",
"Q 1947 1309 1714 600 \n",
"Q 1481 -109 1013 -844 \n",
"L 513 -844 \n",
"Q 928 -128 1133 580 \n",
"Q 1338 1288 1338 2009 \n",
"Q 1338 2731 1133 3434 \n",
"Q 928 4138 513 4856 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-66\"/>\n",
" <use xlink:href=\"#DejaVuSans-28\" x=\"35.205078\"/>\n",
" <use xlink:href=\"#DejaVuSans-78\" x=\"74.21875\"/>\n",
" <use xlink:href=\"#DejaVuSans-29\" x=\"133.398438\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <g id=\"line2d_19\">\n",
" <path d=\"M 61.03821 139.5 \n",
"L 64.848201 138.712047 \n",
"L 68.150188 137.821364 \n",
"L 71.198179 136.782374 \n",
"L 73.992171 135.605492 \n",
"L 76.53216 134.312485 \n",
"L 79.072156 132.769871 \n",
"L 81.358147 131.135675 \n",
"L 83.644141 129.23713 \n",
"L 85.930135 127.043226 \n",
"L 88.216127 124.523617 \n",
"L 90.502121 121.65036 \n",
"L 92.788114 118.400162 \n",
"L 95.074106 114.756979 \n",
"L 97.3601 110.714828 \n",
"L 99.900092 105.764482 \n",
"L 102.694083 99.798835 \n",
"L 105.742074 92.759648 \n",
"L 109.806062 82.7726 \n",
"L 118.696034 60.737614 \n",
"L 121.998025 53.274616 \n",
"L 124.792016 47.500736 \n",
"L 127.332009 42.741454 \n",
"L 129.872 38.47183 \n",
"L 132.157994 35.048422 \n",
"L 134.443987 32.010145 \n",
"L 136.729979 29.336587 \n",
"L 139.015972 27.001577 \n",
"L 141.301965 24.975587 \n",
"L 143.587959 23.22771 \n",
"L 146.127952 21.574409 \n",
"L 148.667945 20.18608 \n",
"L 151.461934 18.920419 \n",
"L 154.509925 17.801452 \n",
"L 157.811915 16.841004 \n",
"L 161.367903 16.039842 \n",
"L 165.43189 15.352775 \n",
"L 170.257877 14.771011 \n",
"L 176.09986 14.303146 \n",
"L 183.465836 13.948324 \n",
"L 193.625808 13.698571 \n",
"L 209.881755 13.550986 \n",
"L 238.583665 13.5 \n",
"L 238.583665 13.5 \n",
"\" clip-path=\"url(#pcb712af87f)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_3\">\n",
" <path d=\"M 52.160938 145.8 \n",
"L 52.160938 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_4\">\n",
" <path d=\"M 247.460938 145.8 \n",
"L 247.460938 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_5\">\n",
" <path d=\"M 52.160937 145.8 \n",
"L 247.460938 145.8 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_6\">\n",
" <path d=\"M 52.160937 7.2 \n",
"L 247.460938 7.2 \n",
"\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
" </g>\n",
" <g id=\"patch_7\">\n",
" <path d=\"M 209.960548 66.057911 \n",
"Q 211.633037 40.775352 213.231728 16.608389 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" <path d=\"M 210.972059 20.46765 \n",
"L 213.231728 16.608389 \n",
"L 214.963336 20.73168 \n",
"\" style=\"fill: none; stroke: #000000; stroke-linecap: round\"/>\n",
" </g>\n",
" <g id=\"text_12\">\n",
" <!-- vanishing gradient -->\n",
" <g transform=\"translate(162.637901 77.650968) scale(0.1 -0.1)\">\n",
" <defs>\n",
" <path id=\"DejaVuSans-76\" d=\"M 191 3500 \n",
"L 800 3500 \n",
"L 1894 563 \n",
"L 2988 3500 \n",
"L 3597 3500 \n",
"L 2284 0 \n",
"L 1503 0 \n",
"L 191 3500 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
"Q 1497 1759 1228 1600 \n",
"Q 959 1441 959 1056 \n",
"Q 959 750 1161 570 \n",
"Q 1363 391 1709 391 \n",
"Q 2188 391 2477 730 \n",
"Q 2766 1069 2766 1631 \n",
"L 2766 1759 \n",
"L 2194 1759 \n",
"z\n",
"M 3341 1997 \n",
"L 3341 0 \n",
"L 2766 0 \n",
"L 2766 531 \n",
"Q 2569 213 2275 61 \n",
"Q 1981 -91 1556 -91 \n",
"Q 1019 -91 701 211 \n",
"Q 384 513 384 1019 \n",
"Q 384 1609 779 1909 \n",
"Q 1175 2209 1959 2209 \n",
"L 2766 2209 \n",
"L 2766 2266 \n",
"Q 2766 2663 2505 2880 \n",
"Q 2244 3097 1772 3097 \n",
"Q 1472 3097 1187 3025 \n",
"Q 903 2953 641 2809 \n",
"L 641 3341 \n",
"Q 956 3463 1253 3523 \n",
"Q 1550 3584 1831 3584 \n",
"Q 2591 3584 2966 3190 \n",
"Q 3341 2797 3341 1997 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
"L 3513 0 \n",
"L 2938 0 \n",
"L 2938 2094 \n",
"Q 2938 2591 2744 2837 \n",
"Q 2550 3084 2163 3084 \n",
"Q 1697 3084 1428 2787 \n",
"Q 1159 2491 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1366 3272 1645 3428 \n",
"Q 1925 3584 2291 3584 \n",
"Q 2894 3584 3203 3211 \n",
"Q 3513 2838 3513 2113 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
"L 1178 3500 \n",
"L 1178 0 \n",
"L 603 0 \n",
"L 603 3500 \n",
"z\n",
"M 603 4863 \n",
"L 1178 4863 \n",
"L 1178 4134 \n",
"L 603 4134 \n",
"L 603 4863 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
"L 2834 2853 \n",
"Q 2591 2978 2328 3040 \n",
"Q 2066 3103 1784 3103 \n",
"Q 1356 3103 1142 2972 \n",
"Q 928 2841 928 2578 \n",
"Q 928 2378 1081 2264 \n",
"Q 1234 2150 1697 2047 \n",
"L 1894 2003 \n",
"Q 2506 1872 2764 1633 \n",
"Q 3022 1394 3022 966 \n",
"Q 3022 478 2636 193 \n",
"Q 2250 -91 1575 -91 \n",
"Q 1294 -91 989 -36 \n",
"Q 684 19 347 128 \n",
"L 347 722 \n",
"Q 666 556 975 473 \n",
"Q 1284 391 1588 391 \n",
"Q 1994 391 2212 530 \n",
"Q 2431 669 2431 922 \n",
"Q 2431 1156 2273 1281 \n",
"Q 2116 1406 1581 1522 \n",
"L 1381 1569 \n",
"Q 847 1681 609 1914 \n",
"Q 372 2147 372 2553 \n",
"Q 372 3047 722 3315 \n",
"Q 1072 3584 1716 3584 \n",
"Q 2034 3584 2315 3537 \n",
"Q 2597 3491 2834 3397 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n",
"L 3513 0 \n",
"L 2938 0 \n",
"L 2938 2094 \n",
"Q 2938 2591 2744 2837 \n",
"Q 2550 3084 2163 3084 \n",
"Q 1697 3084 1428 2787 \n",
"Q 1159 2491 1159 1978 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 4863 \n",
"L 1159 4863 \n",
"L 1159 2956 \n",
"Q 1366 3272 1645 3428 \n",
"Q 1925 3584 2291 3584 \n",
"Q 2894 3584 3203 3211 \n",
"Q 3513 2838 3513 2113 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
"Q 2906 2416 2648 2759 \n",
"Q 2391 3103 1925 3103 \n",
"Q 1463 3103 1205 2759 \n",
"Q 947 2416 947 1791 \n",
"Q 947 1169 1205 825 \n",
"Q 1463 481 1925 481 \n",
"Q 2391 481 2648 825 \n",
"Q 2906 1169 2906 1791 \n",
"z\n",
"M 3481 434 \n",
"Q 3481 -459 3084 -895 \n",
"Q 2688 -1331 1869 -1331 \n",
"Q 1566 -1331 1297 -1286 \n",
"Q 1028 -1241 775 -1147 \n",
"L 775 -588 \n",
"Q 1028 -725 1275 -790 \n",
"Q 1522 -856 1778 -856 \n",
"Q 2344 -856 2625 -561 \n",
"Q 2906 -266 2906 331 \n",
"L 2906 616 \n",
"Q 2728 306 2450 153 \n",
"Q 2172 0 1784 0 \n",
"Q 1141 0 747 490 \n",
"Q 353 981 353 1791 \n",
"Q 353 2603 747 3093 \n",
"Q 1141 3584 1784 3584 \n",
"Q 2172 3584 2450 3431 \n",
"Q 2728 3278 2906 2969 \n",
"L 2906 3500 \n",
"L 3481 3500 \n",
"L 3481 434 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
"Q 2534 3019 2420 3045 \n",
"Q 2306 3072 2169 3072 \n",
"Q 1681 3072 1420 2755 \n",
"Q 1159 2438 1159 1844 \n",
"L 1159 0 \n",
"L 581 0 \n",
"L 581 3500 \n",
"L 1159 3500 \n",
"L 1159 2956 \n",
"Q 1341 3275 1631 3429 \n",
"Q 1922 3584 2338 3584 \n",
"Q 2397 3584 2469 3576 \n",
"Q 2541 3569 2628 3553 \n",
"L 2631 2963 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
"L 2906 4863 \n",
"L 3481 4863 \n",
"L 3481 0 \n",
"L 2906 0 \n",
"L 2906 525 \n",
"Q 2725 213 2448 61 \n",
"Q 2172 -91 1784 -91 \n",
"Q 1150 -91 751 415 \n",
"Q 353 922 353 1747 \n",
"Q 353 2572 751 3078 \n",
"Q 1150 3584 1784 3584 \n",
"Q 2172 3584 2448 3432 \n",
"Q 2725 3281 2906 2969 \n",
"z\n",
"M 947 1747 \n",
"Q 947 1113 1208 752 \n",
"Q 1469 391 1925 391 \n",
"Q 2381 391 2643 752 \n",
"Q 2906 1113 2906 1747 \n",
"Q 2906 2381 2643 2742 \n",
"Q 2381 3103 1925 3103 \n",
"Q 1469 3103 1208 2742 \n",
"Q 947 2381 947 1747 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
"L 3597 1613 \n",
"L 953 1613 \n",
"Q 991 1019 1311 708 \n",
"Q 1631 397 2203 397 \n",
"Q 2534 397 2845 478 \n",
"Q 3156 559 3463 722 \n",
"L 3463 178 \n",
"Q 3153 47 2828 -22 \n",
"Q 2503 -91 2169 -91 \n",
"Q 1331 -91 842 396 \n",
"Q 353 884 353 1716 \n",
"Q 353 2575 817 3079 \n",
"Q 1281 3584 2069 3584 \n",
"Q 2775 3584 3186 3129 \n",
"Q 3597 2675 3597 1894 \n",
"z\n",
"M 3022 2063 \n",
"Q 3016 2534 2758 2815 \n",
"Q 2500 3097 2075 3097 \n",
"Q 1594 3097 1305 2825 \n",
"Q 1016 2553 972 2059 \n",
"L 3022 2063 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
"L 1172 3500 \n",
"L 2356 3500 \n",
"L 2356 3053 \n",
"L 1172 3053 \n",
"L 1172 1153 \n",
"Q 1172 725 1289 603 \n",
"Q 1406 481 1766 481 \n",
"L 2356 481 \n",
"L 2356 0 \n",
"L 1766 0 \n",
"Q 1100 0 847 248 \n",
"Q 594 497 594 1153 \n",
"L 594 3053 \n",
"L 172 3053 \n",
"L 172 3500 \n",
"L 594 3500 \n",
"L 594 4494 \n",
"L 1172 4494 \n",
"z\n",
"\" transform=\"scale(0.015625)\"/>\n",
" </defs>\n",
" <use xlink:href=\"#DejaVuSans-76\"/>\n",
" <use xlink:href=\"#DejaVuSans-61\" x=\"59.179688\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"120.458984\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"183.837891\"/>\n",
" <use xlink:href=\"#DejaVuSans-73\" x=\"211.621094\"/>\n",
" <use xlink:href=\"#DejaVuSans-68\" x=\"263.720703\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"327.099609\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"354.882812\"/>\n",
" <use xlink:href=\"#DejaVuSans-67\" x=\"418.261719\"/>\n",
" <use xlink:href=\"#DejaVuSans-20\" x=\"481.738281\"/>\n",
" <use xlink:href=\"#DejaVuSans-67\" x=\"513.525391\"/>\n",
" <use xlink:href=\"#DejaVuSans-72\" x=\"577.001953\"/>\n",
" <use xlink:href=\"#DejaVuSans-61\" x=\"618.115234\"/>\n",
" <use xlink:href=\"#DejaVuSans-64\" x=\"679.394531\"/>\n",
" <use xlink:href=\"#DejaVuSans-69\" x=\"742.871094\"/>\n",
" <use xlink:href=\"#DejaVuSans-65\" x=\"770.654297\"/>\n",
" <use xlink:href=\"#DejaVuSans-6e\" x=\"832.177734\"/>\n",
" <use xlink:href=\"#DejaVuSans-74\" x=\"895.556641\"/>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" </g>\n",
" <defs>\n",
" <clipPath id=\"pcb712af87f\">\n",
" <rect x=\"52.160938\" y=\"7.2\" width=\"195.3\" height=\"138.6\"/>\n",
" </clipPath>\n",
" </defs>\n",
"</svg>\n"
],
"text/plain": [
"<Figure size 350x250 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"x = torch.arange(-2.0, 5.0, 0.01)\n",
"d2l.plot(x, [torch.tanh(x)], 'x', 'f(x)')\n",
"annotate('vanishing gradient', (4, 1), (2, 0.0))"
]
},
{
"cell_type": "markdown",
"id": "1eb9a578",
"metadata": {
"origin_pos": 17
},
"source": [
"As we saw, optimization for deep learning is full of challenges. Fortunately there exists a robust range of algorithms that perform well and that are easy to use even for beginners. Furthermore, it is not really necessary to find *the* best solution. Local optima or even approximate solutions thereof are still very useful.\n",
"\n",
"## Summary\n",
"\n",
"* Minimizing the training error does *not* guarantee that we find the best set of parameters to minimize the generalization error.\n",
"* The optimization problems may have many local minima.\n",
"* The problem may have even more saddle points, as generally the problems are not convex.\n",
"* Vanishing gradients can cause optimization to stall. Often a reparametrization of the problem helps. Good initialization of the parameters can be beneficial, too.\n",
"\n",
"\n",
"## Exercises\n",
"\n",
"1. Consider a simple MLP with a single hidden layer of, say, $d$ dimensions in the hidden layer and a single output. Show that for any local minimum there are at least $d!$ equivalent solutions that behave identically.\n",
"1. Assume that we have a symmetric random matrix $\\mathbf{M}$ where the entries\n",
" $M_{ij} = M_{ji}$ are each drawn from some probability distribution\n",
" $p_{ij}$. Furthermore assume that $p_{ij}(x) = p_{ij}(-x)$, i.e., that the\n",
" distribution is symmetric (see e.g., :citet:`Wigner.1958` for details).\n",
" 1. Prove that the distribution over eigenvalues is also symmetric. That is, for any eigenvector $\\mathbf{v}$ the probability that the associated eigenvalue $\\lambda$ satisfies $P(\\lambda > 0) = P(\\lambda < 0)$.\n",
" 1. Why does the above *not* imply $P(\\lambda > 0) = 0.5$?\n",
"1. What other challenges involved in deep learning optimization can you think of?\n",
"1. Assume that you want to balance a (real) ball on a (real) saddle.\n",
" 1. Why is this hard?\n",
" 1. Can you exploit this effect also for optimization algorithms?\n"
]
},
{
"cell_type": "markdown",
"id": "d883308a",
"metadata": {
"origin_pos": 19,
"tab": [
"pytorch"
]
},
"source": [
"[Discussions](https://discuss.d2l.ai/t/487)\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
},
"required_libs": []
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in new issue View Git Blame Copy Permalink