学习笔记（11）- 文本生成RNNLG

https://github.com/shawnwun/RNNLG

数据集

给出了4个行业的语料，餐馆、酒店、电脑、电视，及其组合数据。

数据格式

任务

根据给定格式的命令，生成自然语言。

方法、模型、策略

作者给出了5种模型，2种训练（优化）策略、2种解码方式

* Model
- (knn) kNN generator:
    k-nearest neighbor example-based generator, based on MR similarty.
- (ngram) Class-based Ngram generator [Oh & Rudnicky, 2000]:
    Class-based language model generator by utterance class partitions.
- (hlstm) Heuristic Gated LSTM [Wen et al, 2015a]:
    An MR-conditioned LSTM generator with heuristic gates.
- (sclstm) Semantically Conditioned LSTM [Wen et al, 2015b]:
    An MR-conditioned LSTM generator with learned gates.
- (encdec) Attentive Encoder-Decoder LSTM [Wen et al, 2015c]:
    An encoder-decoder LSTM with slot-value level attention.

* Training Strategy
- (ml) Maximum Likehood Training, using token cross-entropy
- (dt) Discriminative Training (or Expected BLEU training) [Wen et al, 2016]

* Decoding Strategy
- (beam) Beam search
- (sample) Random sampling

快速开始

需要python2环境，依赖：

* Theano 0.8.2 and accompanying packages such as numpy, scipy ...
* NLTK 3.0.0

创建虚机，Python2

virtualenv env
source env/bin/activate
pip install theano==0.8.2
pip install nltk==3.0.0

训练：python main.py -config config/sclstm.cfg -mode train

测试：python main.py -config config/sclstm.cfg -mode test

配置文件和参数

从上面的训练和测试的命令可以看出，参数在config目录下的文件配置，看看config/sclstm.cfg文件的内容

[learn] // parameters for training
lr          = 0.1 : learning rate of SGD.
lr_decay    = 0.5  : learning rate decay.
lr_divide   = 3 : the maximum number of times when validation gets worse.
                  for early stopping.
beta        = 0.0000001  : regularisation parameter.
random_seed = 5 : random seed.
min_impr    = 1.003 : the relative minimal improvement allowed.
debug       = True : debug flag
llogp       = -100000000 : log prob in the last epoch

[train_mode]
mode        = all : training mode, currently only support 'all'
obj         = ml  : training objective, 'ml' or 'dt'
###################################
* Training Strategy
- (ml) Maximum Likehood Training, using token cross-entropy
- (dt) Discriminative Training (or Expected BLEU training) [Wen et al, 2016]
###################################
gamma       = 5.0  : hyperparameter for DT training
batch       = 1 : batch size

[generator] // structure for generator
type        = sclstm : the model type, [hlstm|sclstm|encdec]
hidden      = 80 : hidden layer size

[data] // data and model file
domain      = restaurant  作者给出4种领域：餐馆、酒店、电脑、电视
train       = data/original/restaurant/train.json
valid       = data/original/restaurant/valid.json
test        = data/original/restaurant/test.json
vocab       = resource/vocab  词典
percentage  = 100 : the percentage of train/valid considered
wvec        = vec/vectors-80.txt  : pretrained word vectors 预训练的词向量，有多个维度
model       = model/sclstm-rest.model  : the produced model path 生成的模型文件名称

[gen] // generation parameters, decode='beam' or 'sample'
topk        = 5  : the N-best list returned
overgen     = 20  : number of over-generation
beamwidth   = 10  : the beam width used to decode utterances
detectpairs = resource/detect.pair  :  the mapping file for calculating the slot error rate 见下文
verbose     = 1  : verbose level of the model, not supported yet
decode      = beam  : decoding strategy, 'beam' or 'sample'

Below are knn/ngram specific parameters:
* [ngram]
- ngram         : the N of ngram
- rho           : number of slots considered to partition the dataset

结果

我在自己机器试了一下

inform(name=fresca;phone='4154472668')
Penalty TSER    ASER    Gen
0.0672  0       0       the phone number for fresca is 4154472668
0.1272  0       0       fresca s phone number is 4154472668
0.1694  0       0       the phone number of fresca is 4154472668
0.1781  0       0       the phone number for the fresca is 4154472668
0.2153  0       0       the phone number to fresca is 4154472668

文件resource/detect.pair

{
   "general" : {
       "address"    : "SLOT_ADDRESS",
       "area"       : "SLOT_AREA",
       "count"      : "SLOT_COUNT",
       "food"       : "SLOT_FOOD",
       "goodformeal": "SLOT_GOODFORMEAL",
       "name"       : "SLOT_NAME",
       "near"       : "SLOT_NEAR",
       "phone"      : "SLOT_PHONE",
       "postcode"	 : "SLOT_POSTCODE",
       "price"	     : "SLOT_PRICE",
       "pricerange" : "SLOT_PRICERANGE",
       "battery"    : "SLOT_BATTERY",
       "batteryrating"  : "SLOT_BATTERYRATING",
       "design"     : "SLOT_DESIGN",
       "dimension"  : "SLOT_DIMENSION",
       "drive"      : "SLOT_DRIVE",
       "driverange" : "SLOT_DRIVERANGE",
       "family"     : "SLOT_FAMILY",
       "memory"     : "SLOT_MEMORY",
       "platform"   : "SLOT_PLATFORM",
       "utility"    : "SLOT_UTILITY",
       "warranty"   : "SLOT_WARRANTY",
       "weight"     : "SLOT_WEIGHT",
       "weightrange": "SLOT_WEIGHTRANGE",
       "hdmiport"   : "SLOT_HDMIPORT",
       "ecorating"  : "SLOT_ECORATING",
       "audio"      : "SLOT_AUDIO",
       "accessories": "SLOT_ACCESSORIES",
       "color"      : "SLOT_COLOR",
       "powerconsumption"  : "SLOT_POWERCONSUMPTION",
       "resolution" : "SLOT_RESOLUTION",
       "screensize" : "SLOT_SCREENSIZE",
       "screensizerange" : "SLOT_SCREENSIZERANGE"
   },
   "binary"  : {
       "kidsallowed":["child","kid","kids","children"],
       "dogsallowed":["dog","dogs","puppy"],
       "hasinternet":["internet","wifi"],
       "acceptscreditcards":["card","cards"],
       "isforbusinesscomputing":["business","nonbusiness","home","personal","general"],
       "hasusbport" :["usb"]
   }
}

总结

将结构化的数据，转为非结构化的文本。整个任务的核心就是这个吧