Statement: This series of post records the personal notes and experiences of learning the BiliBili video tutorial “Pytorch 入门学习”, most of code and pictures are from the courseware PyTorch-Course. All posted content is for personal study only, do not use for other purposes. If there is infringement, please contact e-mail:yangsuoly@qq.com to delete.

1. Introduction to deep learning models

1.1 Definition

Q: What is machine learning?
A: Study of algorithms that:

Improve their performance P
At some task T
With experience E

Conclusion: Modeling, Inference, learning

Modeling: define score function
Inference: solve argmax
Learning: choose w

\text{classify} (x, w) = \mathop{\text{argmax}}\limits_{y} \ \text{score} (x, y, w)

Q: What is deep learning?
A:

Q: What is neural network?
A:

1.2 Activation function

1.2.1 Commonly used activation functions:

$sigmoid$ :

\sigma(x) = \frac{1}{1+e^{-x}}

$tanh$ :

tanh(x) = 2 \sigma(2x) - 1

$ReLU$ :

ReLU(x) = max(0, x)

$Softmax$ :

z_i \rightarrow \frac{e^{z_i}}{\sum_{j=1}^{k} e^{z_j}}

1.2.2 Code implementation

import torch
import torch.nn.functional as F
from torch.autograd import Variable
import matplotlib.pyplot as plt

x = torch.linspace(-5, 5, 200)
x = Variable(x)
x_np = x.data.numpy()

# four activation function
y_relu = F.relu(x).data.numpy()
y_sigmoid = torch.sigmoid(x).data.numpy()
y_tanh = F.tanh(x).data.numpy()
y_softplus = F.softplus(x).data.numpy()

plt.figure(1, figsize=(8, 6))
plt.subplot(221)
plt.plot(x_np, y_relu, c='red', label='relu')
plt.ylim((-1, 5))
plt.legend(loc='best')

plt.subplot(222)
plt.plot(x_np, y_sigmoid, c='red', label='sigmoid')
plt.ylim((-0.2, 1.2))
plt.legend(loc='best')

plt.subplot(223)
plt.plot(x_np, y_tanh, c='red', label='tanh')
plt.ylim((-1.2, 1.2))
plt.legend(loc='best')

plt.subplot(224)
plt.plot(x_np, y_softplus, c='red', label='softplus')
plt.ylim((-0.2, 6))
plt.legend(loc='best')

plt.show()

Result:

1.3 examples of NN

Standard feedforward NN
Convolutional NN
Recurrent NN
Seq2Seq with Attention

Reference: Effective Approaches to Attention-based Neural Machine Translation

2. Introduction to PyTorch

2.1 Framework for deep learning

Difference between PyTorch and Tensorflow:

PyTorch: 动态计算图 Dynamic Computation Graph
Tensorflow: 静态计算图 Static Computation Graph

PyTorch 代码通俗易懂，非常接近 Python 原生代码，不会让人感觉是完全在学习一门新的语言。拥有 Facebook 支持，社区活跃。

Q: What does the PyTorch do?
A: