Statement: This series of post records the personal notes and experiences of learning the BiliBili video tutorial "Pytorch 入门学习", most of code and pictures are from the courseware PyTorch-Course. All posted content is for personal study only, do not use for other purposes. If there is infringement, please contact e-mail:yangsuoly@qq.com to delete.

1. Introduction to deep learning models

1.1 Definition

Q: What is machine learning?
A: Study of algorithms that: - Improve their performance P - At some task T - With experience E

Conclusion: Modeling, Inference, learning

Modeling: define score function
Inference: solve argmax
Learning: choose w

\[ \text{classify} (x, w) = \mathop{\text{argmax}}\limits_{y} \ \text{score} (x, y, w)\]

Q: What is deep learning? A:

Q: What is neural network? A:

1.2 Activation function

1.2.1 Commonly used activation functions:

\(sigmoid\): \[ \sigma(x) = \frac{1}{1+e^{-x}} \]
\(tanh\): \[ tanh(x) = 2 \sigma(2x) - 1 \]
\(ReLU\): \[ ReLU(x) = max(0, x) \]
\(Softmax\): \[ z_i \rightarrow \frac{e^{z_i}}{\sum_{j=1}^{k} e^{z_j}} \]

1.2.2 Code implementation

import torch
import torch.nn.functional as F
from torch.autograd import Variable
import matplotlib.pyplot as plt

x = torch.linspace(-5, 5, 200)
x = Variable(x)
x_np = x.data.numpy()

# four activation function
y_relu = F.relu(x).data.numpy()
y_sigmoid = torch.sigmoid(x).data.numpy()
y_tanh = F.tanh(x).data.numpy()
y_softplus = F.softplus(x).data.numpy()

plt.figure(1, figsize=(8, 6))
plt.subplot(221)
plt.plot(x_np, y_relu, c='red', label='relu')
plt.ylim((-1, 5))
plt.legend(loc='best')

plt.subplot(222)
plt.plot(x_np, y_sigmoid, c='red', label='sigmoid')
plt.ylim((-0.2, 1.2))
plt.legend(loc='best')

plt.subplot(223)
plt.plot(x_np, y_tanh, c='red', label='tanh')
plt.ylim((-1.2, 1.2))
plt.legend(loc='best')

plt.subplot(224)
plt.plot(x_np, y_softplus, c='red', label='softplus')
plt.ylim((-0.2, 6))
plt.legend(loc='best')

plt.show()