caffe训练自己的数据集

默认caffe已经编译好了，并且编译好了pycaffe

1 数据准备

首先准备训练和测试数据集，这里准备两类数据，分别放在文件夹0和文件夹1中（之所以使用0和1命名数据类别，是因为方便标注数据类别，直接用文件夹的名字即可）。即训练数据集：/data/train/0、/data/train/1  训练数据集：/data/val/0、/data/val/1。

数据准备好之后，创建记录数据文件和对应标签的txt文件

（1）创建训练数据集的train.txt

 import os
 f =open(r'train.txt',"w")
 path = os.getcwd()+'/data/train/'
 for filename in os.listdir(path) :
     count = 0
     for file in os.listdir(path+filename) :
         count = count + 1
         ff='/'+filename+"/"+file+" "+filename+"\n"
         f.write(ff)
     print '{} class: {}'.format(filename,count)
 f.close()

（2）创建测试数据集val.txt

 import os
 f =open(r'val.txt',"w")
 path = os.getcwd()+'/data/val/'
 for filename in os.listdir(path) :
     count = 0
     for file in os.listdir(path+filename) :
         count = count + 1
         ff='/'+filename+"/"+file+" "+filename+"\n"
         f.write(ff)
     print '{} class: {}'.format(filename,count)
 f.close()

注意，txt中文件的路径为： /类别文件夹名/文件名（空格，不能是制表符）类别

2 创建LMDB数据文件

创建createlmdb.sh使用caffe自带的（bulid/tools下的）convert_imageset创建LMDB数据文件，主要是注意数据文件以及上一步生成的txt文件的位置，注意数据文件的RESIZE，后边在进行训练和测试的时候还要用到，其余就是文件的路径的问题了。

 #!/usr/bin/env sh

 CAFFE_ROOT=/home/caf/object/caffe
 TOOLS=$CAFFE_ROOT/build/tools
 TRAIN_DATA_ROOT=/home/caf/wk/learn/data/train
 VAL_DATA_ROOT=/home/caf/wk/learn/data/val
 DATA=/home/caf/wk/learn/data
 EXAMPLE=/home/caf/wk/learn/data/lmdb
 # Set RESIZE=true to resize the images to  x . Leave as false if images have
 # already been resized using another tool.
 RESIZE=true
 if $RESIZE; then
   RESIZE_HEIGHT=
   RESIZE_WIDTH=
 else
   RESIZE_HEIGHT=
   RESIZE_WIDTH=
 fi

 if [ ! -d "$TRAIN_DATA_ROOT" ]; then
   echo "Error: TRAIN_DATA_ROOT is not a path to a directory: $TRAIN_DATA_ROOT"
   echo "Set the TRAIN_DATA_ROOT variable in create_face_48.sh to the path" \
        "where the face_48 training data is stored."
   exit
 fi

 if [ ! -d "$VAL_DATA_ROOT" ]; then
   echo "Error: VAL_DATA_ROOT is not a path to a directory: $VAL_DATA_ROOT"
   echo "Set the VAL_DATA_ROOT variable in create_face_48.sh to the path" \
        "where the face_48 validation data is stored."
   exit
 fi

 echo "Creating train lmdb..."

 GLOG_logtostderr= $TOOLS/convert_imageset \
     --resize_height=$RESIZE_HEIGHT \
     --resize_width=$RESIZE_WIDTH \
     --shuffle \
     $TRAIN_DATA_ROOT \
     $DATA/train.txt \
     $EXAMPLE/face_train_lmdb

 echo "Creating val lmdb..."

 GLOG_logtostderr= $TOOLS/convert_imageset \
     --resize_height=$RESIZE_HEIGHT \
     --resize_width=$RESIZE_WIDTH \
     --shuffle \
     $VAL_DATA_ROOT \
     $DATA/val.txt \
     $EXAMPLE/face_val_lmdb

 echo "Done."

3 定义网络

caffe接受的网络模型是prototxt文件，对于caffe网络的定义语法有详细的解释，本次实验用的是AlexNet，保存在train_val.prototxt

 name: "AlexNet"
 layer {
   name: "data"
   type: "Data"
   top: "data"
   top: "label"
   include {
     phase: TRAIN
   }
   data_param {
     source: "/home/caf/wk/learn/data/lmdb/face_train_lmdb"
     batch_size:
     backend: LMDB
   }
 }
 layer {
   name: "data"
   type: "Data"
   top: "data"
   top: "label"
   include {
     phase: TEST
   }
   data_param {
     source: "/home/caf/wk/learn/data/lmdb/face_val_lmdb"
     batch_size:
     backend: LMDB
   }
 }
 layer {
   name: "conv1"
   type: "Convolution"
   bottom: "data"
   top: "conv1"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     kernel_size:
     stride:
     weight_filler {
       type: "gaussian"
       std: 0.01
     }
     bias_filler {
       type: "constant"
       value:
     }
   }
 }
 layer {
   name: "relu1"
   type: "ReLU"
   bottom: "conv1"
   top: "conv1"
 }
 layer {
   name: "norm1"
   type: "LRN"
   bottom: "conv1"
   top: "norm1"
   lrn_param {
     local_size:
     alpha: 0.0001
     beta: 0.75
   }
 }
 layer {
   name: "pool1"
   type: "Pooling"
   bottom: "norm1"
   top: "pool1"
   pooling_param {
     pool: MAX
     kernel_size:
     stride:
   }
 }
 layer {
   name: "conv2"
   type: "Convolution"
   bottom: "pool1"
   top: "conv2"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     group:
     weight_filler {
       type: "gaussian"
       std: 0.01
     }
     bias_filler {
       type: "constant"
       value: 0.1
     }
   }
 }
 layer {
   name: "relu2"
   type: "ReLU"
   bottom: "conv2"
   top: "conv2"
 }
 layer {
   name: "norm2"
   type: "LRN"
   bottom: "conv2"
   top: "norm2"
   lrn_param {
     local_size:
     alpha: 0.0001
     beta: 0.75
   }
 }
 layer {
   name: "pool2"
   type: "Pooling"
   bottom: "norm2"
   top: "pool2"
   pooling_param {
     pool: MAX
     kernel_size:
     stride:
   }
 }
 layer {
   name: "conv3"
   type: "Convolution"
   bottom: "pool2"
   top: "conv3"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     weight_filler {
       type: "gaussian"
       std: 0.01
     }
     bias_filler {
       type: "constant"
       value:
     }
   }
 }
 layer {
   name: "relu3"
   type: "ReLU"
   bottom: "conv3"
   top: "conv3"
 }
 layer {
   name: "conv4"
   type: "Convolution"
   bottom: "conv3"
   top: "conv4"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     group:
     weight_filler {
       type: "gaussian"
       std: 0.01
     }
     bias_filler {
       type: "constant"
       value: 0.1
     }
   }
 }
 layer {
   name: "relu4"
   type: "ReLU"
   bottom: "conv4"
   top: "conv4"
 }
 layer {
   name: "conv5"
   type: "Convolution"
   bottom: "conv4"
   top: "conv5"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     group:
     weight_filler {
       type: "gaussian"
       std: 0.01
     }
     bias_filler {
       type: "constant"
       value: 0.1
     }
   }
 }
 layer {
   name: "relu5"
   type: "ReLU"
   bottom: "conv5"
   top: "conv5"
 }
 layer {
   name: "pool5"
   type: "Pooling"
   bottom: "conv5"
   top: "pool5"
   pooling_param {
     pool: MAX
     kernel_size:
     stride:
   }
 }
 layer {
   name: "fc6"
   type: "InnerProduct"
   bottom: "pool5"
   top: "fc6"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   inner_product_param {
     num_output:
     weight_filler {
       type: "gaussian"
       std: 0.005
     }
     bias_filler {
       type: "constant"
       value: 0.1
     }
   }
 }
 layer {
   name: "relu6"
   type: "ReLU"
   bottom: "fc6"
   top: "fc6"
 }
 layer {
   name: "drop6"
   type: "Dropout"
   bottom: "fc6"
   top: "fc6"
   dropout_param {
     dropout_ratio: 0.5
   }
 }
 layer {
   name: "fc7"
   type: "InnerProduct"
   bottom: "fc6"
   top: "fc7"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   inner_product_param {
     num_output:
     weight_filler {
       type: "gaussian"
       std: 0.005
     }
     bias_filler {
       type: "constant"
       value: 0.1
     }
   }
 }
 layer {
   name: "relu7"
   type: "ReLU"
   bottom: "fc7"
   top: "fc7"
 }
 layer {
   name: "drop7"
   type: "Dropout"
   bottom: "fc7"
   top: "fc7"
   dropout_param {
     dropout_ratio: 0.5
   }
 }
 layer {
   name: "fc8"
   type: "InnerProduct"
   bottom: "fc7"
   top: "fc8"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   inner_product_param {
     num_output:
     weight_filler {
       type: "gaussian"
       std: 0.01
     }
     bias_filler {
       type: "constant"
       value:
     }
   }
 }
 layer {
   name: "accuracy"
   type: "Accuracy"
   bottom: "fc8"
   bottom: "label"
   top: "accuracy"
   include {
     phase: TEST
   }
 }
 layer {
   name: "loss"
   type: "SoftmaxWithLoss"
   bottom: "fc8"
   bottom: "label"
   top: "loss"
 }
 layer {
   name: "prob"
   type: "Softmax"
   bottom: "fc8"
   top: "prob"
 }

创建超参数文件slover.prototxt，主要定义训练的参数，包括迭代次数，每迭代多少次保存模型文件，学习率等等，net就是刚才定义的训练网络，这里训练和测试使用同一个网络。

 net: "train_val.prototxt"
 test_iter:
 test_interval:
 base_lr: 0.001
 lr_policy: "step"
 gamma: 0.1
 stepsize:
 display:
 max_iter:
 momentum: 0.9
 weight_decay: 0.005
 solver_mode: GPU
 snapshot:
 snapshot_prefix: "model/"

4 训练模型

创建train.sh使用GPU进行训练，否则太慢！！！

 #!/usr/bin/env sh
 CAFFE_ROOT=/home/caf/object/caffe
 SLOVER_ROOT=/home/caf/wk/learn
 $CAFFE_ROOT/build/tools/caffe train --solver=$SLOVER_ROOT/slover.prototxt --gpu=

在model文件夹下会生成caffemodel文件，使用这些文件用于图像的分类等操作。

4 测试

创建deploy.prototxt进行测试，和训练网络一样，只不过用于实际分类的网络并不需要训练网络那些参数了，因此需要重新定义一个模型文件，测试的图片在该模型中进行。

deploy.prototxt文件和train_val.prototxt文件不同的地方在于：

（1）输入的数据不再是LMDB，也不分为测试集和训练集，输入的类型为Input，定义的维度，和训练集的数据维度保持一致，227*227，否则会报错；

（2）去掉weight_filler和bias_filler，这些参数已经存在于caffemodel中了，由caffemodel进行初始化。

（3）去掉最后的Accuracy层和loss层，换位Softmax层，表示分为某一类的概率。

 name: "AlexNet"
 layer {
   name: "data"
   type: "Input"
   top: "data"
   input_param { shape: { dim:  dim:  dim:  dim:  } }
 }
 layer {
   name: "conv1"
   type: "Convolution"
   bottom: "data"
   top: "conv1"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     kernel_size:
     stride:
   }
 }
 layer {
   name: "relu1"
   type: "ReLU"
   bottom: "conv1"
   top: "conv1"
 }
 layer {
   name: "norm1"
   type: "LRN"
   bottom: "conv1"
   top: "norm1"
   lrn_param {
     local_size:
     alpha: 0.0001
     beta: 0.75
   }
 }
 layer {
   name: "pool1"
   type: "Pooling"
   bottom: "norm1"
   top: "pool1"
   pooling_param {
     pool: MAX
     kernel_size:
     stride:
   }
 }
 layer {
   name: "conv2"
   type: "Convolution"
   bottom: "pool1"
   top: "conv2"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     group:
   }
 }
 layer {
   name: "relu2"
   type: "ReLU"
   bottom: "conv2"
   top: "conv2"
 }
 layer {
   name: "norm2"
   type: "LRN"
   bottom: "conv2"
   top: "norm2"
   lrn_param {
     local_size:
     alpha: 0.0001
     beta: 0.75
   }
 }
 layer {
   name: "pool2"
   type: "Pooling"
   bottom: "norm2"
   top: "pool2"
   pooling_param {
     pool: MAX
     kernel_size:
     stride:
   }
 }
 layer {
   name: "conv3"
   type: "Convolution"
   bottom: "pool2"
   top: "conv3"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
   }
 }
 layer {
   name: "relu3"
   type: "ReLU"
   bottom: "conv3"
   top: "conv3"
 }
 layer {
   name: "conv4"
   type: "Convolution"
   bottom: "conv3"
   top: "conv4"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     group:
   }
 }
 layer {
   name: "relu4"
   type: "ReLU"
   bottom: "conv4"
   top: "conv4"
 }
 layer {
   name: "conv5"
   type: "Convolution"
   bottom: "conv4"
   top: "conv5"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   convolution_param {
     num_output:
     pad:
     kernel_size:
     group:
   }
 }
 layer {
   name: "relu5"
   type: "ReLU"
   bottom: "conv5"
   top: "conv5"
 }
 layer {
   name: "pool5"
   type: "Pooling"
   bottom: "conv5"
   top: "pool5"
   pooling_param {
     pool: MAX
     kernel_size:
     stride:
   }
 }
 layer {
   name: "fc6"
   type: "InnerProduct"
   bottom: "pool5"
   top: "fc6"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   inner_product_param {
     num_output:
   }
 }
 layer {
   name: "relu6"
   type: "ReLU"
   bottom: "fc6"
   top: "fc6"
 }
 layer {
   name: "drop6"
   type: "Dropout"
   bottom: "fc6"
   top: "fc6"
   dropout_param {
     dropout_ratio: 0.5
   }
 }
 layer {
   name: "fc7"
   type: "InnerProduct"
   bottom: "fc6"
   top: "fc7"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   inner_product_param {
     num_output:
   }
 }
 layer {
   name: "relu7"
   type: "ReLU"
   bottom: "fc7"
   top: "fc7"
 }
 layer {
   name: "drop7"
   type: "Dropout"
   bottom: "fc7"
   top: "fc7"
   dropout_param {
     dropout_ratio: 0.5
   }
 }
 layer {
   name: "fc8"
   type: "InnerProduct"
   bottom: "fc7"
   top: "fc8"
   param {
     lr_mult:
     decay_mult:
   }
   param {
     lr_mult:
     decay_mult:
   }
   inner_product_param {
     num_output:
   }
 }
 layer {
   name: "prob"
   type: "Softmax"
   bottom: "fc8"
   top: "prob"
 }

用于训练的python代码，使用caffe中python的接口，主要定义好自己训练好的参数文件，模型文件的位置，以及均值文件的位置。

 import numpy as np
 import matplotlib.pyplot as plt

 import sys
 caffe_root="/home/caf/object/caffe/"
 sys.path.insert(0,caffe_root+'python')
 import caffe
 caffe.set_device(0)
 caffe.set_mode_gpu()
 model_def = 'deploy.prototxt'
 model_weights = 'model/_iter_100.caffemodel'
 net = caffe.Net(model_def,
                 model_weights,
                 caffe.TEST)
 mu = np.load(caffe_root + 'python/caffe/imagenet/ilsvrc_2012_mean.npy')
 mu = mu.mean(1).mean(1)
 #print 'mean-subtracted values:', zip('BGR', mu)
 transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape})
 transformer.set_transpose('data', (2,0,1))
 transformer.set_mean('data', mu)
 transformer.set_raw_scale('data', 255)
 transformer.set_channel_swap('data', (2,1,0))
 net.blobs['data'].reshape(3,227, 227)
 image = caffe.io.load_image('test.jpg')
 transformed_image = transformer.preprocess('data', image)
 #plt.imshow(image)
 #plt.show()
 net.blobs['data'].data[...] = transformed_image
 output = net.forward()
 output_prob = output['prob']
 print output_prob
 print 'predicted class is:', output_prob.argmax()

遇到的问题

（1）标签文件不能用制表符，必须是空格，否则会找不到数据文件

（2）CUDA问题，报一个类似叫CUDASuccess的错误，说明GPU空间不够，需要释放空间，使用  nvidia-smi  命令查看那个程序占用GPU过高，使用   kill -9 PID结束掉即可

（3）由于caffe版本的问题，层的定义 有layer和layers，使用layer定义，type需要加双引号，是字符格式；使用layers定义，type不用加双引号，变为全大写字母