16 KiB
def add(a, b):
return a + b
def fancy_func(a, b, c, d):
e = add(a, b)
f = add(c, d)
g = add(e, f)
return g
print(fancy_func(1, 2, 3, 4))10
Python is an interpreted language. When evaluating the above fancy_func function it performs the operations making up the function's body in sequence. That is, it will evaluate e = add(a, b) and store the results as variable e, thereby changing the program's state. The next two statements f = add(c, d) and g = add(e, f) will be executed similarly, performing additions and storing the results as variables. :numref:fig_compute_graph illustrates the flow of data.
🏷️fig_compute_graph
Although imperative programming is convenient, it may be inefficient. On the one hand, even if the add function is repeatedly called throughout fancy_func, Python will execute the three function calls individually. If these are executed, say, on a GPU (or even on multiple GPUs), the overhead arising from the Python interpreter can become overwhelming. Moreover, it will need to save the variable values of e and f until all the statements in fancy_func have been executed. This is because we do not know whether the variables e and f will be used by other parts of the program after the statements e = add(a, b) and f = add(c, d) are executed.
Symbolic Programming
Consider the alternative, symbolic programming, where computation is usually performed only once the process has been fully defined. This strategy is used by multiple deep learning frameworks, including Theano and TensorFlow (the latter has acquired imperative extensions). It usually involves the following steps:
- Define the operations to be executed.
- Compile the operations into an executable program.
- Provide the required inputs and call the compiled program for execution.
This allows for a significant amount of optimization. First, we can skip the Python interpreter in many cases, thus removing a performance bottleneck that can become significant on multiple fast GPUs paired with a single Python thread on a CPU.
Second, a compiler might optimize and rewrite the above code into print((1 + 2) + (3 + 4)) or even print(10). This is possible since a compiler gets to see the full code before turning it into machine instructions. For instance, it can release memory (or never allocate it) whenever a variable is no longer needed. Or it can transform the code entirely into an equivalent piece.
To get a better idea, consider the following simulation of imperative programming (it is Python after all) below.
def add_():
return '''
def add(a, b):
return a + b
'''
def fancy_func_():
return '''
def fancy_func(a, b, c, d):
e = add(a, b)
f = add(c, d)
g = add(e, f)
return g
'''
def evoke_():
return add_() + fancy_func_() + 'print(fancy_func(1, 2, 3, 4))'
prog = evoke_()
print(prog)
y = compile(prog, '', 'exec')
exec(y)
def add(a, b):
return a + b
def fancy_func(a, b, c, d):
e = add(a, b)
f = add(c, d)
g = add(e, f)
return g
print(fancy_func(1, 2, 3, 4))
10
import torch
from torch import nn
from d2l import torch as d2l
# Factory for networks
def get_net():
net = nn.Sequential(nn.Linear(512, 256),
nn.ReLU(),
nn.Linear(256, 128),
nn.ReLU(),
nn.Linear(128, 2))
return net
x = torch.randn(size=(1, 512))
net = get_net()
net(x)tensor([[-0.1602, 0.0003]], grad_fn=<AddmmBackward0>)
net = torch.jit.script(net)
net(x)tensor([[-0.1602, 0.0003]], grad_fn=<AddmmBackward0>)
#@save
class Benchmark:
"""For measuring running time."""
def __init__(self, description='Done'):
self.description = description
def __enter__(self):
self.timer = d2l.Timer()
return self
def __exit__(self, *args):
print(f'{self.description}: {self.timer.stop():.4f} sec')net = get_net()
with Benchmark('Without torchscript'):
for i in range(1000): net(x)
net = torch.jit.script(net)
with Benchmark('With torchscript'):
for i in range(1000): net(x)Without torchscript: 2.1447 sec
With torchscript: 4.0545 sec
net.save('my_mlp')
!ls -lh my_mlp*-rw-r--r-- 1 ci ci 651K Aug 18 19:32 my_mlp