1- ---
2- source : https://github.com/d2l-ai/d2l-en/blob/master/chapter_preliminaries/autograd.md
3- commit : 9e55a9c
4- ---
5-
61# Automatic Differentiation
72:label : ` sec_autograd `
83
@@ -24,49 +19,42 @@ Automatic differentiation enables the system to subsequently backpropagate gradi
2419Here, * backpropagate* simply means to trace through the computational graph,
2520filling in the partial derivatives with respect to each parameter.
2621
27- ``` {.python .input}
28- from mxnet import autograd, np, npx
29- npx.set_np()
30- ```
31-
32- ``` {.python .input}
33- #@tab pytorch
34- import torch
35- ```
36-
37- ``` {.python .input}
38- #@tab tensorflow
39- import tensorflow as tf
40- ```
4122
4223## A Simple Example
4324
4425As a toy example, say that we are interested
45- in differentiating the function
26+ in ( ** differentiating the function
4627$y = 2\mathbf{x}^{\top}\mathbf{x}$
47- with respect to the column vector $\mathbf{x}$.
28+ with respect to the column vector $\mathbf{x}$.** )
4829To start, let us create the variable ` x ` and assign it an initial value.
4930
5031``` {.python .input}
32+ from mxnet import autograd, np, npx
33+ npx.set_np()
34+
5135x = np.arange(4.0)
5236x
5337```
5438
5539``` {.python .input}
5640#@tab pytorch
41+ import torch
42+
5743x = torch.arange(4.0)
5844x
5945```
6046
6147``` {.python .input}
6248#@tab tensorflow
49+ import tensorflow as tf
50+
6351x = tf.range(4, dtype=tf.float32)
6452x
6553```
6654
67- Before we even calculate the gradient
55+ [ ** Before we even calculate the gradient
6856of $y$ with respect to $\mathbf{x}$,
69- we will need a place to store it.
57+ we will need a place to store it.** ]
7058It is important that we do not allocate new memory
7159every time we take a derivative with respect to a parameter
7260because we will often update the same parameters
@@ -95,7 +83,7 @@ x.grad # The default value is None
9583x = tf.Variable(x)
9684```
9785
98- Now let us calculate $y$.
86+ ( ** Now let us calculate $y$.** )
9987
10088``` {.python .input}
10189# Place our code inside an `autograd.record` scope to build the computational
122110Since ` x ` is a vector of length 4,
123111an inner product of ` x ` and ` x ` is performed,
124112yielding the scalar output that we assign to ` y ` .
125- Next, we can automatically calculate the gradient of ` y `
126- with respect to each component of ` x `
113+ Next, [ ** we can automatically calculate the gradient of ` y `
114+ with respect to each component of ` x ` ** ]
127115by calling the function for backpropagation and printing the gradient.
128116
129117``` {.python .input}
@@ -143,8 +131,8 @@ x_grad = t.gradient(y, x)
143131x_grad
144132```
145133
146- The gradient of the function $y = 2\mathbf{x}^{\top}\mathbf{x}$
147- with respect to $\mathbf{x}$ should be $4\mathbf{x}$.
134+ ( ** The gradient of the function $y = 2\mathbf{x}^{\top}\mathbf{x}$
135+ with respect to $\mathbf{x}$ should be $4\mathbf{x}$.** )
148136Let us quickly verify that our desired gradient was calculated correctly.
149137
150138``` {.python .input}
@@ -161,7 +149,7 @@ x.grad == 4 * x
161149x_grad == 4 * x
162150```
163151
164- Now let us calculate another function of ` x ` .
152+ [ ** Now let us calculate another function of ` x ` .** ]
165153
166154``` {.python .input}
167155with autograd.record():
@@ -172,9 +160,9 @@ x.grad # Overwritten by the newly calculated gradient
172160
173161``` {.python .input}
174162#@tab pytorch
175- # PyTorch accumulates the gradient in default, we need to clear the previous
163+ # PyTorch accumulates the gradient in default, we need to clear the previous
176164# values
177- x.grad.zero_()
165+ x.grad.zero_()
178166y = x.sum()
179167y.backward()
180168x.grad
@@ -196,13 +184,13 @@ For higher-order and higher-dimensional `y` and `x`,
196184the differentiation result could be a high-order tensor.
197185
198186However, while these more exotic objects do show up
199- in advanced machine learning (including in deep learning),
200- more often when we are calling backward on a vector,
187+ in advanced machine learning (including [ ** in deep learning** ] ),
188+ more often ( ** when we are calling backward on a vector,** )
201189we are trying to calculate the derivatives of the loss functions
202190for each constituent of a * batch* of training examples.
203- Here, our intent is not to calculate the differentiation matrix
204- but rather the sum of the partial derivatives
205- computed individually for each example in the batch.
191+ Here, ( ** our intent is** ) not to calculate the differentiation matrix
192+ but rather ( ** the sum of the partial derivatives
193+ computed individually for each example** ) in the batch.
206194
207195``` {.python .input}
208196# When we invoke `backward` on a vector-valued variable `y` (function of `x`),
@@ -236,8 +224,8 @@ t.gradient(y, x) # Same as `y = tf.reduce_sum(x * x)`
236224
237225## Detaching Computation
238226
239- Sometimes, we wish to move some calculations
240- outside of the recorded computational graph.
227+ Sometimes, we wish to [ ** move some calculations
228+ outside of the recorded computational graph.** ]
241229For example, say that ` y ` was calculated as a function of ` x ` ,
242230and that subsequently ` z ` was calculated as a function of both ` y ` and ` x ` .
243231Now, imagine that we wanted to calculate
@@ -309,10 +297,10 @@ t.gradient(y, x) == 2 * x
309297## Computing the Gradient of Python Control Flow
310298
311299One benefit of using automatic differentiation
312- is that even if building the computational graph of a function
313- required passing through a maze of Python control flow
300+ is that [ ** even if** ] building the computational graph of ( ** a function
301+ required passing through a maze of Python control flow** )
314302(e.g., conditionals, loops, and arbitrary function calls),
315- we can still calculate the gradient of the resulting variable.
303+ ( ** we can still calculate the gradient of the resulting variable.** )
316304In the following snippet, note that
317305the number of iterations of the ` while ` loop
318306and the evaluation of the ` if ` statement
0 commit comments