论文笔记 Network In Network

这篇论文思路简单、易实现、效果好，是一篇难得的佳作。从实现的角度理解，就是做了以下两个替换：

1. 将线性卷积替换为多层感知机（某种程度上，线性卷积可以认为识一层感知机）。
2. 将全连接层用global average pooling layer替换。

下面我们就来分析引入上述两个替换的妙处。首先分析第一个替换的妙处，替换的效果（图示如下）

论文中提到“The linear convolution is sufficient for abstraction when the instances of the latent concepts are linearly separable.”，然而现实太复杂，the instances of the latent concepts通常不是线性可分的。在这种情况下，通常有两种做法：一是，引入大量的linear convolution（以体量应对复杂现实）；二是，干脆寻找一个能够模拟任意复杂情形的“参数化函数”（以灵活性应对复杂现实）。

可以预见，如果你计算、存储资源充裕的话，你可以采取简单暴力的第一种情形；通常情况下，计算、存储资源受限，因此第二种做法更加接近现实一点（也更容易将算法植入到计算、存储资源有限的移动设备上，如手机）。下面的问题就是寻找所需的“参数化函数”。庆幸的是，多层感知机在某种程度上能够满足我们的需求，此外它能够与BP算法完美兼容（这篇论文选择的就是多层感知机）。这样的Mlpconv layer就可以作为深度网络的几个基本block，用以构建深度网络。

在CNN当中，随着层数的加深，我们得到的特征越来越抽象。这种抽象是以组合较低一层抽象特征得到的。从这个角度理解，如果在较低层就能够比之前对应层更抽象的特征，然后整个网络的输出抽象程度将会变得更高，这样高度抽象的特征对于分类、任务迁移都是有极大帮助的。

下面分析第二个替换的妙处

传统的CNN是将最后一层的卷积输出向量化，然后输入到全连接层，全连接层之后是常用的分类损失函数，如softmax。如果最后一层卷积输出特征维度过高、类别较多，那么这一块引入的参数量是很大的，这会造成网络过拟合（还好，目前有一些防止过拟合的手段，如dropout）。

“The idea is to generate one feature map for each corresponding category of the classification task in the last mlpconv layer. Instead of adding fully connected layers on top od the feature maps, we take the advantage of each feature map, and the resulting vector is fed directly into the softmax layer”，这样做的好处是，直接在类别与feature maps之间建立了联系，“The features maps can be easily interpreted as categories confidence maps”。此外，这里没有引入要学习的参数，也间接起到了防止过拟合的效果。

在Caffe框架下实现上述网络是一个很简单的事情，以在cifar10上的网络结果为例

layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data"
  top: "conv1"
  blobs_lr:
  blobs_lr:
  weight_decay: .
  weight_decay: .
  convolution_param {
    num_output:
    pad:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "relu1"
  type: RELU
  bottom: "conv1"
  top: "conv1"
}
layers {
  name: "cccp1"
  type: CONVOLUTION
  bottom: "conv1"
  top: "cccp1"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp1"
  type: RELU
  bottom: "cccp1"
  top: "cccp1"
}
layers {
  name: "cccp2"
  type: CONVOLUTION
  bottom: "cccp1"
  top: "cccp2"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp2"
  type: RELU
  bottom: "cccp2"
  top: "cccp2"
}

两个kernel_size为1的卷积核实现的就是多层感知机的功能，全部的网络结果代码如下

name: "CIFAR10_full"
layers {
  name: "cifar"
  type: DATA
  top: "data"
  top: "label"
  data_param {
    source: "cifar-train-leveldb"
    batch_size:
  }
  include: { phase: TRAIN }
}
layers {
  name: "cifar"
  type: DATA
  top: "data"
  top: "label"
  data_param {
    source: "cifar-test-leveldb"
    batch_size:
  }
  include: { phase: TEST }
}
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data"
  top: "conv1"
  blobs_lr:
  blobs_lr:
  weight_decay: .
  weight_decay: .
  convolution_param {
    num_output:
    pad:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "relu1"
  type: RELU
  bottom: "conv1"
  top: "conv1"
}
layers {
  name: "cccp1"
  type: CONVOLUTION
  bottom: "conv1"
  top: "cccp1"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp1"
  type: RELU
  bottom: "cccp1"
  top: "cccp1"
}
layers {
  name: "cccp2"
  type: CONVOLUTION
  bottom: "cccp1"
  top: "cccp2"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp2"
  type: RELU
  bottom: "cccp2"
  top: "cccp2"
}
layers {
  name: "pool1"
  type: POOLING
  bottom: "cccp2"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size:
    stride:
  }
}
layers {
  name: "drop3"
  type: DROPOUT
  bottom: "pool1"
  top: "pool1"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "conv2"
  type: CONVOLUTION
  bottom: "pool1"
  top: "conv2"
  blobs_lr:
  blobs_lr:
  weight_decay: .
  weight_decay: .
  convolution_param {
    num_output:
    pad:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "relu2"
  type: RELU
  bottom: "conv2"
  top: "conv2"
}
layers {
  name: "cccp3"
  type: CONVOLUTION
  bottom: "conv2"
  top: "cccp3"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp3"
  type: RELU
  bottom: "cccp3"
  top: "cccp3"
}
layers {
  name: "cccp4"
  type: CONVOLUTION
  bottom: "cccp3"
  top: "cccp4"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp4"
  type: RELU
  bottom: "cccp4"
  top: "cccp4"
}
layers {
  name: "pool2"
  type: POOLING
  bottom: "cccp4"
  top: "pool2"
  pooling_param {
    pool: AVE
    kernel_size:
    stride:
  }
}
layers {
  name: "drop6"
  type: DROPOUT
  bottom: "pool2"
  top: "pool2"
  dropout_param {
    dropout_ratio: 0.5
  }
}
layers {
  name: "conv3"
  type: CONVOLUTION
  bottom: "pool2"
  top: "conv3"
  blobs_lr: .
  blobs_lr: .
  weight_decay: .
  weight_decay: .
  convolution_param {
    num_output:
    pad:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
    }
  }
}
layers {
  name: "relu3"
  type: RELU
  bottom: "conv3"
  top: "conv3"
}
layers {
  name: "cccp5"
  type: CONVOLUTION
  bottom: "conv3"
  top: "cccp5"
  blobs_lr:
  blobs_lr:
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp5"
  type: RELU
  bottom: "cccp5"
  top: "cccp5"
}
layers {
  name: "cccp6"
  type: CONVOLUTION
  bottom: "cccp5"
  top: "cccp6"
  blobs_lr: 0.1
  blobs_lr: 0.1
  weight_decay:
  weight_decay:
  convolution_param {
    num_output:
    group:
    kernel_size:
    weight_filler {
      type: "gaussian"
      std: 0.05
    }
    bias_filler {
      type: "constant"
      value:
    }
  }
}
layers {
  name: "relu_cccp6"
  type: RELU
  bottom: "cccp6"
  top: "cccp6"
}
layers {
  name: "pool3"
  type: POOLING
  bottom: "cccp6"
  top: "pool3"
  pooling_param {
    pool: AVE
    kernel_size:
    stride:
  }
}
layers {
  name: "accuracy"
  type: ACCURACY
  bottom: "pool3"
  bottom: "label"
  top: "accuracy"
  include: { phase: TEST }
}
layers {
  name: "loss"
  type: SOFTMAX_LOSS
  bottom: "pool3"
  bottom: "label"
  top: "loss"
}

总结：这篇文章引入的改进网络结构的方式、global average pooling启发了后续很多算法，以后有时间再慢慢分析。