DeepHyperX代码理解-HamidaEtAl

代码复现自论文《3-D Deep Learning Approach for Remote Sensing Image Classification》

先对部分基础知识做一些整理：

一、局部连接与参数共享（都减少了参数计算量）

　　局部连接：基于图像局部相关的原理，保留了图像局部结构，同时减少了网络的权值个数，加快了学习速率，同时也在一定程度上减少了过拟合的可能。

　　参数共享：下面用图例解释参数共享与不共享的区别。　　

　　如图是一个3*3大小的卷积核在进行特征提取，channel=1, 在每个位置进行特征提取的时候都是共享一个卷积核，假设有k个channel,则参数

总量为3*3*k,注意不同channel的参数是不能共享的。

　　假设现在不使用参数共享，则卷积核作用于矩阵上的每一个位置时其参数都是不一样的，则卷积核的参数数量就与像素矩阵的大小保持一致了，假
设有k个channel,则参数数量为weight*height*k，这对于尺寸较大的图片来说明显是不可取的。

二、多通道输入输出：

　　首先理解fliter与kernel区别：kernel是fliter的组成成分，一个fliter就会对应生成一个特征图。

　　下图为多通道输入，单通道输出。有一个fliter，一个fliter包括三个kernel，最后生成一个特征图（一个fliter就对应一个特征图）

　　下图为直观展示，输入为8*8*3的RGB三通道图片，设置了4个fliter对应四个输出通道，也就是输出4个特征图。每个fliter有3个kernel。

三、2D卷积代码示例：

 1 import torch as t
 2 import torch.nn as nn
 3
 4 class A(nn.Module):
 5     def __init__(self):
 6         super(A, self).__init__()
 7         #三个2D卷积层结构，每个卷积层的输入和输出通道数均为2，卷积核和大小是(3, 3)
 8         #torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1,
 9         #padding=0, dilation=1, groups=1, bias=True,
10         #padding_mode='zeros', device=None, dtype=None)
11         self.conv1 = nn.Conv2d(2, 2, kernel_size=(3, 3))
12         self.conv2 = nn.Conv2d(2, 2, kernel_size=(3, 3))
13         self.conv3 = nn.Conv2d(2, 2, kernel_size=(3, 3))
14
15 a = A()
16 print(list(a.parameters()))

运行结果：

 1 [Parameter containing:
 2 tensor([[[[ 0.0213, -0.2054,  0.0985],
 3           [ 0.1254,  0.1202, -0.1115],
 4           [-0.0280,  0.0256,  0.1975]],
 5
 6          [[-0.2118, -0.1683,  0.0056],
 7           [-0.0554,  0.2255, -0.1548],
 8           [ 0.1747,  0.0449,  0.1606]]],
 9
10
11         [[[-0.1328, -0.1700, -0.1048],
12           [ 0.0911, -0.2230,  0.0685],
13           [ 0.0886,  0.1765, -0.1879]],
14
15          [[-0.0581,  0.1266, -0.1030],
16           [-0.0170,  0.0387, -0.0641],
17           [-0.0127, -0.2099, -0.2213]]]], requires_grad=True), Parameter containing:
18 tensor([ 0.0877, -0.1969], requires_grad=True), Parameter containing:
19 tensor([[[[-0.0266,  0.1511, -0.0034],
20           [-0.1070, -0.1734, -0.1017],
21           [ 0.0053,  0.1358, -0.0542]],
22
23          [[-0.0436,  0.1587, -0.0375],
24           [ 0.0125, -0.0431, -0.0877],
25           [-0.0766,  0.0405, -0.1306]]],
26
27
28         [[[-0.1449, -0.0315, -0.0236],
29           [ 0.0118,  0.2230, -0.2137],
30           [-0.1108, -0.1178,  0.0027]],
31
32          [[ 0.2184,  0.1964, -0.0959],
33           [-0.0385, -0.0523,  0.2135],
34           [-0.0387,  0.1951, -0.1546]]]], requires_grad=True), Parameter containing:
35 tensor([-0.0942, -0.2029], requires_grad=True), Parameter containing:
36 tensor([[[[ 8.9598e-02, -3.3190e-02,  1.0606e-01],
37           [-2.2397e-02,  2.0944e-01, -6.8180e-02],
38           [-1.7312e-01,  2.2318e-01,  1.9368e-01]],
39
40          [[ 7.0529e-02, -2.0741e-01, -1.2648e-01],
41           [-1.7503e-01,  1.7972e-01, -1.0417e-01],
42           [-1.9124e-01, -4.2022e-02,  1.4635e-01]]],
43
44
45         [[[-8.0857e-02,  8.5098e-03,  7.0629e-02],
46           [ 1.6926e-01, -1.7654e-02, -9.3033e-02],
47           [-2.9836e-02, -2.2935e-01,  1.1450e-01]],
48
49          [[ 1.3848e-01, -5.7713e-02,  1.5293e-04],
50           [-6.3998e-03, -1.0745e-01,  2.8835e-03],
51           [-1.7894e-01,  2.2133e-01,  5.2435e-02]]]], requires_grad=True), Parameter containing:
52 tensor([-0.1026,  0.0207], requires_grad=True)]

　　上面代码是定义的一个2D卷积网络架构，有3个卷积层，每个卷积层的输入通道和输出通道（对应fliter为2）都为2，每个fliter中有两个3*3的kernel（对应两个输入通道）。代码中的两个1*2的向量为bias。

四、3D卷积的示意图和示例代码：

 1 class B(nn.Module):
 2     def __init__(self):
 3         super(B, self).__init__()
 4         self.conv1 = nn.Conv3d(
 5             1,  # 输入图像的channel数,C_in
 6             3,  # 卷积产生的channel数,C_out
 7             kernel_size=2,  # 卷积核的尺寸,这里实际是(2,2,2),第一维表示卷积核处理的帧数
 8             stride=(1,1,1),  # 卷积步长,(D,H,W)
 9             padding=(0,0,0),  # 输入的每一条边补充0的层数,(D,H,W)
10             bias=False)
11         self.conv2 = nn.Conv3d(1, 3, kernel_size=(2, 2, 2), stride=(1, 1, 1), padding=(0, 0, 0), bias=False)
12
13 b = B()
14 print(list(b.parameters()))

运行结果：

 1 [Parameter containing:
 2 tensor([[[[[-0.0895, -0.1651],
 3            [-0.1319,  0.2510]],
 4
 5           [[-0.1616, -0.2614],
 6            [ 0.0383,  0.2656]]]],
 7
 8
 9
10         [[[[-0.0809, -0.1821],
11            [-0.0624, -0.1401]],
12
13           [[-0.3170,  0.1499],
14            [-0.3449,  0.2639]]]],
15
16
17
18         [[[[ 0.2779, -0.0731],
19            [ 0.0439, -0.0353]],
20
21           [[ 0.2853, -0.3177],
22            [ 0.0559, -0.3290]]]]], requires_grad=True), Parameter containing:
23 tensor([[[[[-0.2529, -0.0537],
24            [ 0.0955, -0.1513]],
25
26           [[ 0.0310,  0.2558],
27            [-0.1903, -0.1561]]]],
28
29
30
31         [[[[-0.2919, -0.3142],
32            [ 0.0190, -0.1089]],
33
34           [[-0.0586,  0.1541],
35            [-0.2402, -0.3339]]]],
36
37
38
39         [[[[-0.3260, -0.0500],
40            [ 0.1392,  0.1486]],
41
42           [[ 0.1664,  0.1514],
43            [ 0.0345, -0.1979]]]]], requires_grad=True)]

　　三维数据输入通道为1，输出通道为3，那么fliter数为3，每个fliter中有1个kernel(2,2,2),因为两个3Dconv参数一样，所以每个3Dconv有3个2*2*2的tensor。这里如果对应到高光谱数据集，输入通道也为1。因为整个高光谱数据集就是一个3维的volume，整个数据集图像由长*宽*光谱维组成。

论文复现：

原论文网络结构:

复现网络：

输入维度：Image has dimensions 610x340 and 103 channels

网络结构及参数：

 复现代码：

1     elif name == "hamida":
2         patch_size = kwargs.setdefault("patch_size", 5)
3         center_pixel = True
4         model = HamidaEtAl(n_bands, n_classes, patch_size=patch_size)
5         lr = kwargs.setdefault("learning_rate", 0.01)
6         optimizer = optim.SGD(model.parameters(), lr=lr, weight_decay=0.0005)
7         kwargs.setdefault("batch_size", 100)
8         criterion = nn.CrossEntropyLoss(weight=kwargs["weights"])

 1 class HamidaEtAl(nn.Module):
 2     """
 3     3-D Deep Learning Approach for Remote Sensing Image Classification
 4     Amina Ben Hamida, Alexandre Benoit, Patrick Lambert, Chokri Ben Amar
 5     IEEE TGRS, 2018
 6     https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8344565
 7     """
 8
 9     @staticmethod
10     def weight_init(m):#权重初始化
11         if isinstance(m, nn.Linear) or isinstance(m, nn.Conv3d):
12             init.kaiming_normal_(m.weight) #kaiming_normal_均匀初始化方法
13             init.zeros_(m.bias)
14
15     def __init__(self, input_channels, n_classes, patch_size=5, dilation=1):
16         super(HamidaEtAl, self).__init__()
17         # The first layer is a (3,3,3) kernel sized Conv characterized
18         # by a stride equal to 1 and number of neurons equal to 20
19         self.patch_size = patch_size#每次截取图像patch大小(5,5)
20         self.input_channels = input_channels#输入通道数（波段数）
21         dilation = (dilation, 1, 1)
22
23         if patch_size == 3:
24             self.conv1 = nn.Conv3d(
25                 1, 20, (3, 3, 3), stride=(1, 1, 1), dilation=dilation, padding=1
26             )
27         else:
28             self.conv1 = nn.Conv3d(
29                 1, 20, (3, 3, 3), stride=(1, 1, 1), dilation=dilation, padding=0
30             )
31         # Next pooling is applied using a layer identical to the previous one
32         # with the difference of a 1D kernel size (1,1,3) and a larger stride
33         # equal to 2 in order to reduce the spectral dimension
34         #为什么要用卷积层要替代池化层：
35         # 使用2×2的最大池化，与使用卷积（stride为2）来做down sample性能并没有明显差别，
36         # 而且使用卷积（stride为2）相比卷积（步进为1）+池化，还可以减少卷积运算量和一个池化层。何乐而不为呢。
37         self.pool1 = nn.Conv3d(
38             20, 20, (3, 1, 1), dilation=dilation, stride=(2, 1, 1), padding=(1, 0, 0)
39         )
40         # Then, a duplicate of the first and second layers is created with
41         # 35 hidden neurons per layer.
42         self.conv2 = nn.Conv3d(
43             20, 35, (3, 3, 3), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
44         )
45         self.pool2 = nn.Conv3d(
46             35, 35, (3, 1, 1), dilation=dilation, stride=(2, 1, 1), padding=(1, 0, 0)
47         )
48         # Finally, the 1D spatial dimension is progressively reduced
49         # thanks to the use of two Conv layers, 35 neurons each,
50         # with respective kernel sizes of (1,1,3) and (1,1,2) and strides
51         # respectively equal to (1,1,1) and (1,1,2)
52         self.conv3 = nn.Conv3d(
53             35, 35, (3, 1, 1), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
54         )
55         self.conv4 = nn.Conv3d(
56             35, 35, (2, 1, 1), dilation=dilation, stride=(2, 1, 1), padding=(1, 0, 0)
57         )
58
59         # self.dropout = nn.Dropout(p=0.5)
60
61         self.features_size = self._get_final_flattened_size()
62         # The architecture ends with a fully connected layer where the number
63         # of neurons is equal to the number of input classes.
64         self.fc = nn.Linear(self.features_size, n_classes)
65
66         self.apply(self.weight_init)
67
68     def _get_final_flattened_size(self):#计算fc前的输出tensor大小，用随机的X去测试得到，这样就不用自己去推到了
69         with torch.no_grad():
70             x = torch.zeros(
71                 (1, 1, self.input_channels, self.patch_size, self.patch_size)
72             )
73             x = self.pool1(self.conv1(x))
74             x = self.pool2(self.conv2(x))
75             x = self.conv3(x)
76             x = self.conv4(x)
77             _, t, c, w, h = x.size()
78         return t * c * w * h
79
80     def forward(self, x):
81         x = F.relu(self.conv1(x))
82         x = self.pool1(x)
83         x = F.relu(self.conv2(x))
84         x = self.pool2(x)
85         x = F.relu(self.conv3(x))
86         x = F.relu(self.conv4(x))
87         x = x.view(-1, self.features_size)#对tensor resize相当于做展平（flatten）操作
88         # x = self.dropout(x)
89         x = self.fc(x)
90         return x

关于参数dilation的解释：https://blog.csdn.net/qimo601/article/details/112624091

运行结果：

通过阅读代码发现和原论文中的结构并不完全相同，试着去修改了一层网络参数去接近原网络结构：

self.conv3 = nn.Conv3d(
            35, 35, (3, 1, 1), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
        )

->>

self.conv3 = nn.Conv3d(
            35, 35, (3, 3, 3), dilation=dilation, stride=(1, 1, 1), padding=(1, 0, 0)
        )

但发现最终效果不如上面结果，可能是对原论文提出的结构稍作改进了吧。

参考资料：

https://github.com/nshaud/DeepHyperX

https://blog.csdn.net/weixin_38481963/article/details/109906338

https://blog.csdn.net/abbcdc/article/details/123332063

https://wenku.baidu.com/view/fa899796f221dd36a32d7375a417866fb84ac087.html