NLP（二十三）使用LSTM进行语言建模以预测最优词

原文链接：http://www.one2know.cn/nlp23/

• N元模型
  
  预测要输入的连续词，比如
  
  
  
  如果抽取两个连续的词汇，则称之为二元模型
• 准备工作
  
  数据集使用 Alice in Wonderland
  
  将初始数据提取N-grams

import nltk
import string

with open('alice_in_wonderland.txt', 'r') as content_file:
    content = content_file.read()
content2 = " ".join("".join([" " if ch in string.punctuation else ch for ch in content]).split())
tokens = nltk.word_tokenize(content2)
tokens = [word.lower() for word in tokens if len(word)>=2]

N = 3
quads = list(nltk.ngrams(tokens,N))
"""
    Return the ngrams generated from a sequence of items, as an iterator.
    For example:
        >>> from nltk.util import ngrams
        >>> list(ngrams([1,2,3,4,5], 3))
        [(1, 2, 3), (2, 3, 4), (3, 4, 5)]
"""
newl_app = []
for ln in quads:
    new1 = ' '.join(ln)
    newl_app.append(new1)
print(newl_app[:3])

输出：

['alice adventures in', 'adventures in wonderland', 'in wonderland alice']

• 如何实现
  
  1.预处理：词转换为词向量
  
  2.创建模型和验证：将输入映射到输出的收敛-发散模型（convergent-divergent）
  
  3.预测：最优词预测
• 代码

from __future__ import print_function

from sklearn.model_selection import train_test_split
import nltk
import numpy as np
import string

with open('alice_in_wonderland.txt', 'r') as content_file:
    content = content_file.read()
content2 = " ".join("".join([" " if ch in string.punctuation else ch for ch in content]).split())
tokens = nltk.word_tokenize(content2)
tokens = [word.lower() for word in tokens if len(word)>=2]

N = 3
quads = list(nltk.ngrams(tokens,N))
"""
    Return the ngrams generated from a sequence of items, as an iterator.
    For example:
        >>> from nltk.util import ngrams
        >>> list(ngrams([1,2,3,4,5], 3))
        [(1, 2, 3), (2, 3, 4), (3, 4, 5)]
"""
newl_app = []
for ln in quads:
    new1 = ' '.join(ln)
    newl_app.append(new1)
# print(newl_app[:3])

# 将单词向量化
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer() # 词=>词向量
"""
    >>> corpus = [
    ...     'This is the first document.',
    ...     'This document is the second document.',
    ...     'And this is the third one.',
    ...     'Is this the first document?',
    ... ]
    >>> vectorizer = CountVectorizer()
    >>> X = vectorizer.fit_transform(corpus)
    >>> print(vectorizer.get_feature_names())
    ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this']
    >>> print(X.toarray())  # doctest: +NORMALIZE_WHITESPACE
    [[0 1 1 1 0 0 1 0 1]
     [0 2 0 1 0 1 1 0 1]
     [1 0 0 1 1 0 1 1 1]
     [0 1 1 1 0 0 1 0 1]]
"""

x_trigm = []
y_trigm = []

for l in newl_app:
    x_str = " ".join(l.split()[0:N-1])
    y_str = l.split()[N-1]
    x_trigm.append(x_str)
    y_trigm.append(y_str)

x_trigm_check = vectorizer.fit_transform(x_trigm).todense()
y_trigm_check = vectorizer.fit_transform(y_trigm).todense()

# Dictionaries from word to integer and integer to word
dictnry = vectorizer.vocabulary_
rev_dictnry = {v:k for k,v in dictnry.items()}

X = np.array(x_trigm_check)
Y = np.array(y_trigm_check)

Xtrain, Xtest, Ytrain, Ytest,xtrain_tg,xtest_tg = train_test_split(X, Y,x_trigm, test_size=0.3,random_state=1)

print("X Train shape",Xtrain.shape, "Y Train shape" , Ytrain.shape)
print("X Test shape",Xtest.shape, "Y Test shape" , Ytest.shape)

# Model Building
from keras.layers import Input,Dense,Dropout
from keras.models import Model

np.random.seed(1)

BATCH_SIZE = 128
NUM_EPOCHS = 20

input_layer = Input(shape = (Xtrain.shape[1],),name="input")
first_layer = Dense(1000,activation='relu',name = "first")(input_layer)
first_dropout = Dropout(0.5,name="firstdout")(first_layer)

second_layer = Dense(800,activation='relu',name="second")(first_dropout)

third_layer = Dense(1000,activation='relu',name="third")(second_layer)
third_dropout = Dropout(0.5,name="thirdout")(third_layer)

fourth_layer = Dense(Ytrain.shape[1],activation='softmax',name = "fourth")(third_dropout)

history = Model(input_layer,fourth_layer)
history.compile(optimizer = "adam",loss="categorical_crossentropy",metrics=["accuracy"])

print (history.summary())

# Model Training
history.fit(Xtrain, Ytrain, batch_size=BATCH_SIZE,epochs=NUM_EPOCHS, verbose=1,validation_split = 0.2)

# Model Prediction
Y_pred = history.predict(Xtest)

# 测试
print ("Prior bigram words","|Actual","|Predicted","\n")
import random
NUM_DISPLAY = 10
for i in random.sample(range(len(xtest_tg)),NUM_DISPLAY):
    print (i,xtest_tg[i],"|",rev_dictnry[np.argmax(Ytest[i])],"|",rev_dictnry[np.argmax(Y_pred[i])])

输出：

X Train shape (17947, 2559) Y Train shape (17947, 2559)
X Test shape (7692, 2559) Y Test shape (7692, 2559)

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input (InputLayer)           (None, 2559)              0
_________________________________________________________________
first (Dense)                (None, 1000)              2560000
_________________________________________________________________
firstdout (Dropout)          (None, 1000)              0
_________________________________________________________________
second (Dense)               (None, 800)               800800
_________________________________________________________________
third (Dense)                (None, 1000)              801000
_________________________________________________________________
thirdout (Dropout)           (None, 1000)              0
_________________________________________________________________
fourth (Dense)               (None, 2559)              2561559
=================================================================
Total params: 6,723,359
Trainable params: 6,723,359
Non-trainable params: 0
_________________________________________________________________
None

Prior bigram words |Actual |Predicted
595 words don | fit | know
3816 in tone | of | of
5792 queen had | only | been
2757 who seemed | to | to
5393 her and | she | she
4197 heard of | one | its
2464 sneeze were | the | of
1590 done with | said | whiting
3039 and most | things | of
4226 the queen | of | said

训练结果不好，因为单词向量维度太大，为2559，相对而言总数据集的单词太少；除了单词预测，还可以字符预测，每碰到一个空格算一个单词