Tutorial on word2vector using GloVe and Word2Vec
Tutorial on word2vector using GloVe and Word2Vec
2018-05-04 10:02:53
Some Important Reference Pages First:
Reference Page: https://github.com/IliaGavrilov/NeuralMachineTranslationBidirectionalLSTM/blob/master/1_Bidirectional_LSTM_Eng_to_French.ipynb
Glove Project Page: https://nlp.stanford.edu/projects/glove/
Word2Vec Project Page: https://code.google.com/archive/p/word2vec/
Pre-trained word2vec model: https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing
Gensim Tutorial: https://radimrehurek.com/gensim/models/word2vec.html
===================================
===== For the Glove
===================================
1. Download one of the pre-trained model from Glove project page and Unzip the files.
Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB download): glove.6B.zip
Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors, 1.75 GB download): glove.42B.300d.zip
Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download): glove.840B.300d.zip
2. Install the needed packages:
pickle, numpy, re, pickle, collections, bcolz
3. Run the following demo to test the results (extract the feature of given single word).
Code:
import pickle
import numpy as np
import re, pickle, collections, bcolz # with open('./glove.840B.300d.txt', 'r', encoding="utf8") as f: with open('./glove.6B.200d.txt', 'r') as f:
lines = [line.split() for line in f] print('==>> begin to load Glove pre-trained models.')
glove_words = [elem[0] for elem in lines]
glove_words_idx = {elem:idx for idx, elem in enumerate(glove_words)} # is elem: idx equal to glove_words_idx[elem]=idx?
glove_vecs = np.stack(np.array(elem[1:], dtype=np.float32) for elem in lines) print('==>> save into .pkl files.')
pickle.dump(glove_words, open('./glove.6B.200d.txt'+'_glove_words.pkl', 'wb'))
pickle.dump(glove_words_idx, open('./glove.6B.200d.txt'+'_glove_words_idx.pkl', 'wb')) ## saving array using specific function.
def save_array(fname, arr):
c=bcolz.carray(arr, rootdir=fname, mode='w')
c.flush() save_array('./glove.6B.200d.txt'+'_glove_vecs'+'.dat', glove_vecs) def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove('./glove.6B.200d.txt')
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape print('==>> shown one demo: "King"')
demo_vector = en_w2v['king']
print(demo_vector)
print("==>> Done !")
Results:
wangxiao@AHU$ python tutorial_Glove_word2vec.py
==>> begin to load Glove pre-trained models.
==>> save into .pkl files.
==>> Loading the glove.6B.200d.txt files.
==>> shown one demo: "King"
[-0.49346 -0.14768 0.32166001 0.056899 0.052572 0.20192 -0.13506 -0.030793 0.15614 -0.23004 -0.66376001 -0.27316001 0.10391 0.57334 -0.032355 -0.32765999 -0.27160001 0.32918999
0.41305 -0.18085 1.51670003 2.16490006 -0.10278 0.098019 -0.018946 0.027292 -0.79479998 0.36631 -0.33151001 0.28839999 0.10436 -0.19166 0.27326 -0.17519 -0.14985999 -0.072333
-0.54370999 -0.29728001 0.081491 -0.42673001 -0.36406001 -0.52034998 0.18455 0.44121 -0.32196 0.39172 0.11952 0.36978999 0.29229 -0.42954001 0.46653 -0.067243 0.31215999 -0.17216
0.48874 0.28029999 -0.17577 -0.35100999 0.020792 0.15974 0.21927001 -0.32499 0.086022 0.38927001 -0.65638 -0.67400998 -0.41896001 1.27090001 0.20857 0.28314999 0.58238 -0.14944001
0.3989 0.52680999 0.35714 -0.39100999 -0.55372 -0.56642002 -0.15762 -0.48004001 0.40448001 0.057518 -1.01569998 0.21754999 0.073296 0.15237001 -0.38361999 -0.75308001 -0.0060254 -0.26232001
-0.54101998 -0.34347001 0.11113 0.47685 -0.73229998 0.77596998 0.015216 -0.66327 -0.21144 -0.42964 -0.72689998 -0.067968 0.50601 0.039817 -0.27584001 -0.34794 -0.0474 0.50734001
-0.30777001 0.11594 -0.19211 0.3107 -0.60075003 0.22044 -0.36265001 -0.59442002 -1.20459998 0.10619 -0.60277998 0.21573 -0.35361999 0.55473 0.58094001 0.077259 1.0776 -0.1867
-1.51680005 0.32418001 0.83332998 0.17366 1.12320006 0.10863 0.55888999 0.30799001 0.084318 -0.43178001 -0.042287 -0.054615 0.054712 -0.80914003 -0.24429999 -0.076909 0.55216002 -0.71895999
0.83319002 0.020735 0.020472 -0.40279001 -0.28874001 0.23758 0.12576 -0.15165 -0.69419998 -0.25174001 0.29591 0.40290001 -1.0618 0.19847 -0.63463002 -0.70842999 0.067943 0.57366002
0.041122 0.17452 0.19430999 -0.28641 -1.13629997 0.45116001 -0.066518 0.82615 -0.45451999 -0.85652 0.18105 -0.24187 0.20152999 0.72298002 0.17415 -0.87327999 0.69814998 0.024706
0.26174 -0.0087155 -0.39348999 0.13801 -0.39298999 -0.23057 -0.22611 -0.14407 0.010511 -0.47389001 -0.15645 0.28601 -0.21772 -0.49535 0.022209 -0.23575 -0.22469001 -0.011578 0.52867001 -0.062309 ]
==>> Done !
4. Extract the feature of the whole sentense.
Given one sentense, such as "I Love Natural Language Processing", we can translate this sentense into a matrix representation. Specifically, we represent each word of the sentense with a vector which lies in a continuous space (as shown above). You can also see this blog: https://blog.csdn.net/fangqingan_java/article/details/50493948 to futher understand this process.
但是有如下的疑问：
1. 给定的句子，长短不一，每一个单词的维度固定，但是总的句子的长度不固定，怎么办？
只好用 padding 的方法了，那么，怎么进行 padding 呢？
For CNN architecture usually the input is (for each sentence) a vector of embedded words [GloVe(w0), GloVe(w1), ..., GloVe(wN)] of fixed length N.
So commonly you preset the maximum sentence length and just pad the tail of the input vector with zero vectors (just as you did). Marking the beginning and the end of the sentence is more relevant for the RNN, where the sentence length is expected to be variable and processing is done sequentially. Having said that, you can add a new dimension to the GloVe vector, setting it to zero for normal words, and arbitrarily to (say) 0.5 for BEGIN, and 1 for END and a random negative value for OOV.
As for unknown words, you should ;) encounter them very rear, otherwise you might consider training the embeddings by yourself. 那么，是什么意思呢？
对于 CNN 的网络来说，由于需要处理 网格状的 data，所以，需要将句子进行填充，以得到完整的 matrix，然后进行处理；
对于 RNN/LSTM 的网络来说，是可以按照时刻，顺序处理的，所以呢？就不需要 padding 了？
2. Here is another tutorial on Word2Vec using deep learning tools --- Keras.
from numpy import array
from keras.preprocessing.text import one_hot
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.embeddings import Embedding
# define documents
docs = ['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!',
'Weak',
'Poor effort!',
'not good',
'poor work',
'Could have done better.']
# define class labels
labels = array([,,,,,,,,,])
# integer encode the documents
vocab_size =
encoded_docs = [one_hot(d, vocab_size) for d in docs]
print("==>> encoded_docs: ")
print(encoded_docs)
# pad documents to a max length of words
max_length =
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')
print("==>> padded_docs: ")
print(padded_docs)
# define the model
model = Sequential()
model.add(Embedding(vocab_size, , input_length=max_length))
model.add(Flatten())
model.add(Dense(, activation='sigmoid'))
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
# summarize the model
print("==>> model.summary()")
print(model.summary())
# fit the model
model.fit(padded_docs, labels, nb_epoch=, verbose=)
# evaluate the model
loss, accuracy = model.evaluate(padded_docs, labels, verbose=)
print("==>> the final Accuracy: ")
print('Accuracy: %f' % (accuracy*))
The output is:
Using TensorFlow backend.
==>> encoded_docs:
[[, ], [, ], [, ], [, ], [], [], [, ], [, ], [, ], [, , , ]]
==>> padded_docs:
[[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]]
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_prod (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
==>> model.summary()
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
embedding_1 (Embedding) (None, , ) embedding_input_1[][]
____________________________________________________________________________________________________
flatten_1 (Flatten) (None, ) embedding_1[][]
____________________________________________________________________________________________________
dense_1 (Dense) (None, ) flatten_1[][]
====================================================================================================
Total params:
Trainable params:
Non-trainable params:
____________________________________________________________________________________________________
None
-- ::59.053198: I tensorflow/core/platform/cpu_feature_guard.cc:] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
==>> the final Accuracy:
Accuracy: 89.999998
3. Maybe translate the sentence into vectors using skip-thought vectors is another goog choice. you can also check this tutorial from: http://www.cnblogs.com/wangxiaocvpr/p/7277025.html
4. But we still want to use GloVe to extract the vector of each word and concatenate them into a long vector.
import os
import numpy as np
from collections import OrderedDict
import re, pickle, collections, bcolz
import pdb seq_home = '/dataset/language-dataset/'
seqlist_path = '/dataset/language-train-video-list.txt'
output_path = 'data/language-train-video-list.pkl' def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) pre_trained_model_path = '/glove/glove.6B.200d.txt' ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove(pre_trained_model_path)
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape with open(seqlist_path,'r') as fp:
seq_list = fp.read().splitlines() data = {}
for i,seq in enumerate(seq_list):
img_list = sorted([p for p in os.listdir(seq_home+seq+'/imgs/') if os.path.splitext(p)[1] == '.jpg']) ## image list
gt = np.loadtxt(seq_home+seq+'/groundtruth_rect.txt', delimiter=',') ## gt files
language_txt = open(seq_home+seq+'/language.txt', "rw+") ## natural language description files line = language_txt.readline()
print("==>> language: %s" % (line)) gloveVector = []
test_txtName = seq_home+seq+'/glove_vector.txt'
f=file(test_txtName, "w") word_list = line.split(' ')
for word_idx in range(len(word_list)):
current_word = word_list[word_idx]
current_GloVe_vector = en_w2v[current_word] ## vector dimension is: 200-D gloveVector = np.concatenate((gloveVector, current_GloVe_vector), axis=0)
f.write(str(current_GloVe_vector))
f.write('\n')
f.close() print(i) assert len(img_list) == len(gt), "Lengths do not match!!" if gt.shape[1]==8:
x_min = np.min(gt[:,[0,2,4,6]],axis=1)[:,None]
y_min = np.min(gt[:,[1,3,5,7]],axis=1)[:,None]
x_max = np.max(gt[:,[0,2,4,6]],axis=1)[:,None]
y_max = np.max(gt[:,[1,3,5,7]],axis=1)[:,None]
gt = np.concatenate((x_min, y_min, x_max-x_min, y_max-y_min),axis=1) data[seq] = {'images':img_list, 'gt':gt, 'gloveVector': gloveVector} with open(output_path, 'wb') as fp:
pickle.dump(data, fp, -1)
The Glove vector can be saved into a txt file for each video, as shown in following screenshots.
[ 0.19495 0.60610002 -0.077356 0.017301 -0.51370001 0.22421999
-0.80773002 0.022378 0.30256 1.06669998 -0.10918 0.57902998
0.23986 0.1691 0.0072384 0.42197999 -0.20458999 0.60271001
0.19188 -0.19616 0.33070001 3.20020008 -0.18104 0.20784
0.49511001 -0.42258999 0.022125 0.24379 0.16714001 -0.20909999
-0.12489 -0.51766998 -0.13569 -0.25979999 -0.17961 -0.47663
-0.89539999 -0.27138999 0.17746 0.45827001 0.21363001 0.22342999
-0.049342 0.34286001 -0.32315001 0.27469999 0.95993 -0.25979
0.21250001 -0.21373001 0.19809 0.15455 -0.48581001 0.38925001
0.33747 -0.27897999 0.19371 -0.45872 -0.054217 -0.24022999
0.59153003 0.12453 -0.21302 0.058223 -0.046671 -0.011614
-0.32025999 0.64120001 -0.28718999 0.035138 0.39287001 -0.030683
0.083529 -0.010964 -0.62427002 -0.13575 -0.38468999 0.11454
-0.61037999 0.12594999 -0.17633 -0.21415 -0.37013999 0.21763
0.055366 -0.25242001 -0.45475999 -0.28105 0.18911 -1.58539999
0.64841002 -0.34621999 0.59254003 -0.39034 -0.44258001 0.20562001
0.44395 0.23019999 -0.35018 -0.36090001 0.62993002 -0.34698999
-0.31964999 -0.17361 0.51714998 0.68493003 -0.15587001 1.42550004
-0.94313997 0.031951 -0.26210001 -0.089544 0.22437 -0.050374
0.035414 -0.070493 0.17037 -0.38598001 0.0095626 0.26363
0.72125 -0.13797 0.70602 -0.50839001 -0.49722001 -0.48706001
0.16254 0.025619 0.33572 -0.64160001 -0.32541999 0.21896
0.05331 0.082011 0.12038 0.088849 -0.04651 -0.085435
0.036835 -0.14695001 -0.25841001 -0.043812 0.053209 -0.48954999
1.73950005 0.99014997 0.09395 -0.20236 -0.050135 0.18337999
0.22713999 0.83146 -0.30974001 0.51995999 0.068264 -0.28237
-0.30096999 -0.031014 0.024615 0.4294 -0.085036 0.051913
0.31251001 -0.34443 -0.085145 0.024975 0.0082017 0.17241
-0.66000998 0.0058962 -0.055387 -0.22315 0.35721999 -0.18962
0.25819999 -0.24685 -0.79571998 -0.09436 -0.10271 0.13713001
1.48660004 -0.57113999 -0.52649999 -0.25181001 -0.40792 -0.18612
0.040009 0.11557 0.017987 0.27149001 -0.14252 -0.087756
0.15196 0.064926 0.01651 -0.25334001 0.27437001 0.24246
0.018494 0.22473 ]
[ 4.32489991e-01 5.34709990e-01 -1.83240008e-02 1.56369999e-01
6.69689998e-02 7.63140023e-02 -7.16499984e-01 2.41520002e-01
1.70919999e-01 8.96220028e-01 -2.00739995e-01 1.88370004e-01
4.53570008e-01 -1.86419994e-01 -3.41060013e-01 -3.10570002e-02
-4.90720011e-02 6.55830026e-01 -7.43409991e-03 1.54770002e-01
1.55849993e-01 3.11129999e+00 -4.12149996e-01 1.89439997e-01
1.86639994e-01 -2.00540006e-01 6.63070008e-02 3.90180014e-02
6.06740005e-02 -3.17880005e-01 1.82290003e-02 -6.80719972e-01
-1.33360000e-02 2.32099995e-01 -8.23459998e-02 -2.68420011e-01
-3.51870000e-01 -2.03860000e-01 6.69749975e-02 3.02689999e-01
1.84330001e-01 1.44429997e-01 1.38980001e-01 1.93629991e-02
-4.09139991e-01 2.24950001e-01 8.02020013e-01 -3.05290014e-01
2.22399995e-01 -6.10979982e-02 3.48910004e-01 1.13349997e-01
-4.14329991e-02 7.31640011e-02 3.17719996e-01 -2.59590000e-01
3.17759991e-01 -1.78080007e-01 2.12380007e-01 1.07529998e-01
4.37519997e-01 -1.81329995e-01 -1.53909996e-01 7.27130026e-02
4.14730012e-01 2.73079991e-01 -1.06940001e-01 7.45599985e-01
3.02549988e-01 1.49999997e-02 6.33790016e-01 1.03399999e-01
2.18640000e-01 -7.43470015e-03 -7.09949970e-01 4.60739993e-02
-8.34990025e-01 1.84519999e-02 -2.75200009e-01 -2.48000007e-02
-1.21320002e-01 3.29720005e-02 1.07320003e-01 2.18070000e-01
2.79079992e-02 -2.99050003e-01 -3.06030005e-01 -3.17970008e-01
2.10580006e-01 -1.16180003e+00 4.57720011e-01 -2.60210007e-01
3.76190007e-01 -2.16749996e-01 -1.68540001e-01 1.77729994e-01
-1.46709993e-01 2.67820001e-01 -5.86109996e-01 -4.98769999e-01
3.83679986e-01 2.51020014e-01 -3.90020013e-01 -6.41409993e-01
5.34929991e-01 2.76479989e-01 -4.32830006e-01 1.61350000e+00
-1.58399999e-01 -2.03970000e-01 -2.18089998e-01 -9.74659983e-04
-7.48440027e-02 3.78959998e-02 -4.03719991e-01 -1.74640000e-01
4.11469996e-01 -2.80400008e-01 -3.42570007e-01 5.72329983e-02
7.18890011e-01 -1.30419999e-01 5.75810015e-01 -3.27690005e-01
7.75609985e-02 -2.66130000e-01 4.82820012e-02 1.24679998e-01
1.29319996e-01 7.65379965e-02 -3.85809988e-01 3.72209996e-01
-1.67280003e-01 1.86330006e-01 -2.20280007e-01 3.27170014e-01
1.74089998e-01 -2.51549989e-01 -5.47109991e-02 -2.48549998e-01
-2.24260002e-01 -3.80129993e-01 5.53080022e-01 -1.52050003e-01
1.12820005e+00 3.70869994e-01 -4.62579988e-02 -3.56009990e-01
-1.72120005e-01 1.23460002e-01 6.58179998e-01 5.05930007e-01
-4.99610007e-01 1.97339997e-01 1.37759998e-01 3.98500008e-04
-1.38300002e-01 -6.67430013e-02 -7.20809996e-02 4.00720000e-01
-3.77620012e-01 -9.86199975e-02 2.04119995e-01 -3.42869997e-01
-2.01509997e-01 -1.18270002e-01 -1.60109997e-01 1.63570002e-01
-3.79029989e-01 -2.45529994e-01 -5.31699993e-02 -4.19999994e-02
1.82099998e-01 2.23959997e-01 4.50800002e-01 -2.03030005e-01
-6.16330028e-01 -2.02739999e-01 2.18419999e-01 3.66939992e-01
7.68440008e-01 -2.92189986e-01 -9.80089977e-02 -2.92939991e-01
-1.93189994e-01 1.45720005e-01 2.45150000e-01 -7.11840019e-02
3.97929996e-01 -3.33019998e-03 -4.01179999e-01 1.39760002e-01
8.27540010e-02 -1.29600003e-01 -3.05500001e-01 8.98869988e-03
2.26559997e-01 3.21410000e-01 -4.29780006e-01 4.74779993e-01]
[ 5.73459983e-01 5.41700006e-01 -2.34770000e-01 -3.62399995e-01
4.03699994e-01 1.13860004e-01 -4.49330002e-01 -3.09909999e-01
-5.34110004e-03 5.84259987e-01 -2.59559993e-02 4.93930012e-01
-3.72090004e-02 -2.84280002e-01 9.76959988e-02 -4.89069998e-01
2.60269996e-02 3.76489997e-01 5.77879995e-02 -4.68070000e-01
8.12880024e-02 3.28250003e+00 -6.36900008e-01 3.79559994e-01
3.81670007e-03 9.36070010e-02 -1.28549993e-01 1.73800007e-01
1.05219997e-01 2.86480010e-01 2.10889995e-01 -4.70759988e-01
2.77330000e-02 -1.98029995e-01 7.63280019e-02 -8.46289992e-01
-7.97079980e-01 -3.87430012e-01 -3.04220002e-02 -2.68489987e-01
4.85850006e-01 1.28950000e-01 3.83540004e-01 3.87219995e-01
-3.85239989e-01 1.90750003e-01 4.89980012e-01 1.32780001e-01
1.07920002e-02 2.67699987e-01 1.78120002e-01 -1.14330001e-01
-3.34939986e-01 8.73059988e-01 7.58750021e-01 -3.03779989e-01
-1.56259999e-01 1.20850001e-03 2.33219996e-01 2.79529989e-01
-1.84939995e-01 -1.41460001e-01 -1.89689994e-01 -3.83859985e-02
3.58740002e-01 6.55129999e-02 6.05649985e-02 6.63389981e-01
-8.32519978e-02 6.51630014e-02 5.17610013e-01 1.61709994e-01
4.60110009e-01 1.63880005e-01 -1.23989999e-01 3.11219990e-01
-1.54119998e-01 -1.09169997e-01 -4.25509989e-01 1.14179999e-01
2.51370013e-01 -5.61579987e-02 -2.59270012e-01 2.81630009e-01
-1.80939995e-02 1.60650000e-01 -4.85060006e-01 -9.89030004e-01
2.50220001e-01 -1.67359993e-01 4.14739996e-01 1.77010000e-01
4.24070001e-01 1.10880002e-01 -1.83599994e-01 -1.24100000e-01
-3.47799987e-01 9.90779996e-02 -2.23810002e-01 -1.12450004e-01
-2.11559996e-01 3.07060010e-03 -2.36070007e-01 2.72610001e-02
3.64300013e-01 3.99219990e-02 -1.83689997e-01 1.22660005e+00
-7.76400030e-01 -6.62249982e-01 1.57239996e-02 -1.49690002e-01
8.46489966e-02 2.68139988e-01 -1.67649999e-01 -3.19420010e-01
2.84940004e-01 -7.00000003e-02 1.20099997e-02 -1.22189999e-01
5.63099980e-01 -3.19999993e-01 5.01089990e-01 -1.02090001e-01
4.65750009e-01 -7.15420008e-01 1.72930002e-01 5.82589984e-01
7.83839971e-02 -3.38440016e-02 -2.51289994e-01 3.65029991e-01
3.15780006e-02 -6.57779992e-01 5.47499992e-02 8.71890008e-01
1.24550000e-01 -4.58770007e-01 -2.69650012e-01 -4.67790008e-01
-2.85780011e-03 1.78100005e-01 6.39689982e-01 1.39950007e-01
9.75960016e-01 1.18359998e-01 -6.39039993e-01 -1.54159993e-01
6.52619973e-02 2.43290007e-01 6.64759994e-01 2.50690013e-01
-1.02519996e-01 -3.28390002e-01 -8.55590031e-02 -1.27739999e-02
-1.94309995e-01 5.61389983e-01 -3.57329994e-01 -2.03439996e-01
-1.24130003e-01 -3.44309986e-01 -2.32960001e-01 -2.11870000e-01
8.53869990e-02 7.00630024e-02 -1.98029995e-01 -2.60230005e-02
-3.90370011e-01 8.00019979e-01 4.05770004e-01 -7.98629969e-02
3.52629989e-01 -3.40429991e-01 3.96759987e-01 2.28619993e-01
-3.50279987e-01 -4.73439991e-01 5.97419977e-01 -1.16570003e-01
1.05519998e+00 -4.15699989e-01 -8.05519968e-02 -5.65709993e-02
-1.66219994e-01 1.92739993e-01 -9.51749980e-02 -2.07810000e-01
1.56200007e-01 5.02309985e-02 -2.79150009e-01 4.37420011e-01
-3.12370002e-01 1.31940007e-01 -3.32780004e-01 1.88769996e-01
-2.34219998e-01 5.44179976e-01 -2.30690002e-01 3.49469990e-01]
[ -1.43299997e-01 -6.54269993e-01 -4.71419990e-01 -2.22780004e-01
-1.74260005e-01 6.43490016e-01 -7.60049999e-01 -8.42769966e-02
2.19850004e-01 -4.24650013e-01 -2.56919991e-02 -3.27909999e-02
-3.98149997e-01 -1.26000000e-02 6.31869972e-01 -6.31640017e-01
-2.70480007e-01 2.09920004e-01 5.72669983e-01 8.88589993e-02
2.24289998e-01 1.78419995e+00 8.73669982e-01 -2.23949999e-01
4.91869986e-01 6.86379969e-02 6.80689991e-01 1.85680002e-01
-2.85780013e-01 -1.06030002e-01 -3.07460010e-01 1.06180003e-02
-1.62290007e-01 -3.62700000e-02 8.02920014e-02 -2.17669994e-01
-7.92110026e-01 -7.06340015e-01 -2.89400011e-01 -1.51869997e-01
3.09009999e-01 -1.41849995e-01 -4.18920010e-01 -1.66299999e-01
-1.68510005e-01 1.04549997e-01 -3.49110007e-01 -1.21399999e+00
3.21929991e-01 1.88219994e-01 -3.50019991e-01 -2.23629996e-01
3.85280013e-01 1.40709996e-01 1.34890005e-01 3.89420003e-01
1.82109997e-01 1.32369995e-01 -2.60610014e-01 5.10110021e-01
1.26430005e-01 -2.94450015e-01 -1.34350002e-01 -1.44600004e-01
1.98740005e-01 -3.41720015e-01 2.40710005e-01 -6.08640015e-02
8.60409975e-01 8.92559998e-04 3.46020013e-01 -2.53780007e-01
2.53749996e-01 1.20920002e-01 -3.82739991e-01 1.55090000e-02
-6.41290024e-02 2.54729986e-01 -1.78489998e-01 2.37949997e-01
3.26880008e-01 2.03940004e-01 -5.65020025e-01 3.87109995e-01
-7.00220019e-02 -4.63999987e-01 -1.98210001e-01 -7.27079988e-01
5.36920011e-01 -9.09250021e-01 -2.36699998e-01 -6.20739982e-02
2.38769993e-01 3.24860007e-01 5.51190019e-01 -4.07079995e-01
1.40870005e-01 2.83569992e-01 -1.33379996e-01 -3.99740010e-01
-1.66620001e-01 4.43010002e-01 -6.85970008e-01 1.54200003e-01
6.58490002e-01 2.95630004e-02 2.61550009e-01 1.10969996e+00
-6.78719997e-01 -1.91960007e-01 7.45159984e-02 -3.53740007e-01
-7.15600014e-01 4.14079994e-01 -3.37280005e-01 2.05780007e-02
2.17340007e-01 6.48889989e-02 -3.51080000e-01 -2.44440004e-01
-1.98780000e-01 3.02210003e-01 -1.32180005e-01 -4.86490011e-01
-3.55599999e-01 2.83119995e-02 7.04949975e-01 7.35610008e-01
5.98779976e-01 -3.94160002e-01 -3.96349996e-01 2.55739987e-01
1.30939996e+00 -6.69730008e-02 4.72090006e-01 3.20030004e-01
-2.39020005e-01 1.87910005e-01 2.14310005e-01 3.10609996e-01
-7.22400010e-01 -2.59559989e-01 7.50709996e-02 5.45759976e-01
1.33249998e+00 5.97169995e-01 -7.56640017e-01 2.30540007e-01
3.45209986e-01 8.61710012e-01 5.65129995e-01 1.33000001e-01
4.27370012e-01 -4.36790008e-03 -5.15600026e-01 3.94329995e-01
-1.58559993e-01 -1.29820004e-01 -5.47240004e-02 -1.01789999e+00
6.39170036e-02 -6.47679985e-01 -7.53360018e-02 1.49210006e-01
5.53300023e-01 -6.24869987e-02 -2.01309994e-01 1.70489997e-01
-3.22829992e-01 2.43039995e-01 2.64449995e-02 -9.52059999e-02
3.80879998e-01 -3.69230002e-01 4.54270005e-01 -3.31169993e-01
-1.93750001e-02 1.59300007e-02 1.85659993e-02 -5.27350008e-01
1.05949998e+00 -4.31519985e-01 2.57169992e-01 -9.44539979e-02
7.53460005e-02 -2.06599995e-01 3.21940005e-01 -1.23549998e-02
4.72169995e-01 2.44020000e-01 -3.63059998e-01 1.71650007e-01
-3.50499988e-01 1.40990004e-01 1.18069999e-01 3.65170002e-01
-1.65889993e-01 -3.93799990e-02 -3.70829999e-01 2.03909993e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 5.24869978e-01 -1.19410001e-01 -2.02419996e-01 -6.23929977e-01
-1.53799996e-01 -6.63369969e-02 -3.68499994e-01 2.86490005e-02
1.37950003e-01 -5.87819993e-01 6.02090001e-01 2.60540005e-02
7.07889974e-01 1.20329998e-01 -1.74310002e-02 4.03360009e-01
-3.19680005e-01 -2.50609994e-01 1.60889998e-01 2.47649997e-01
7.79359996e-01 2.74070001e+00 1.19589999e-01 -2.67529994e-01
-3.82809997e-01 -3.36580008e-01 1.41039997e-01 -4.65480000e-01
-8.92079994e-02 2.22540006e-01 -3.60740013e-02 -7.10140020e-02
6.23199999e-01 3.22770000e-01 4.15650010e-01 -3.68530005e-02
-5.82859993e-01 -6.26510024e-01 -3.26169990e-02 2.74789989e-01
-2.66950011e-01 5.27690016e-02 -1.09500003e+00 -1.99760008e-03
-7.49390006e-01 -1.89999994e-02 -1.87619999e-01 -5.19330025e-01
1.71590000e-01 4.40690011e-01 1.90789998e-01 -4.57339995e-02
-2.43069995e-02 2.32710004e-01 1.02179997e-01 -6.02790006e-02
-2.63680011e-01 -1.49090007e-01 4.33890015e-01 -8.51420015e-02
-6.61419988e-01 -6.23379983e-02 -1.37920007e-01 4.54079986e-01
-1.51400000e-01 1.14929996e-01 5.48650026e-01 2.82370001e-01
-2.55129993e-01 1.11149997e-01 -8.47479999e-02 -9.66319963e-02
4.88200009e-01 7.48070002e-01 -6.74910009e-01 3.21949989e-01
-8.98810029e-01 -3.58159989e-01 8.54640007e-02 -1.67390004e-01
-1.60109997e-01 -1.06349997e-01 -2.96649992e-01 -3.03389996e-01
2.29310006e-01 6.51580021e-02 1.70359999e-01 -2.08039999e-01
-1.12930000e-01 -1.58399999e-01 -1.88710004e-01 5.38689971e-01
-3.58050019e-02 3.26770008e-01 -3.18859994e-01 2.01079994e-02
-1.15209997e-01 -3.67400013e-02 5.97510003e-02 9.09829974e-01
1.55420005e-01 1.63939998e-01 3.57499987e-01 -4.75789994e-01
-3.53519991e-02 -1.30729997e+00 2.40319997e-01 1.24170005e+00
4.63180006e-01 -2.90019996e-02 6.63789988e-01 -2.20129997e-01
6.57410026e-01 -2.34209999e-01 -7.43539989e-01 8.95029977e-02
-1.41269997e-01 3.54860008e-01 -6.87049985e-01 -9.53029990e-02
-1.59720004e-01 -4.36020009e-02 5.39509989e-02 -1.33180007e-01
4.47530001e-01 1.41289994e-01 2.71119997e-02 1.18179999e-01
2.02419996e-01 -5.98179996e-01 -6.83469996e-02 -1.58150002e-01
2.42929995e-01 8.88189971e-02 -1.16219997e-01 -3.82679999e-01
1.30649999e-01 -3.00619990e-01 -8.46930027e-01 1.90159995e-02
6.41009986e-01 -8.42439979e-02 7.83680022e-01 -2.63660014e-01
1.98839998e+00 -7.38060027e-02 7.21970022e-01 4.84739989e-02
1.62070006e-01 3.40840012e-01 -1.85289994e-01 2.85659999e-01
-2.35440001e-01 -2.69939989e-01 -5.27239978e-01 2.74859995e-01
6.56930029e-01 2.20569998e-01 -6.80390000e-01 3.59100014e-01
2.51430005e-01 7.24399984e-02 2.58690000e-01 3.22510004e-01
3.57239991e-02 -3.88200015e-01 1.18890002e-01 -2.90010005e-01
-6.32229984e-01 -1.48519993e-01 -1.21040002e-01 -1.74060002e-01
2.48679996e-01 4.74539995e-01 -2.50710011e-01 -5.23679972e-01
-1.82730004e-01 -8.30529988e-01 6.15109980e-01 -3.84620011e-01
9.74919975e-01 4.49510008e-01 -5.42559981e-01 -3.18340003e-01
-5.07809997e-01 -2.73739994e-02 -1.26379997e-01 5.42479992e-01
4.85489994e-01 8.55289996e-02 -1.42509997e-01 8.77849981e-02
2.47189999e-01 -2.38490000e-01 6.14570007e-02 -4.23129983e-02
3.57470006e-01 9.73920003e-02 -3.58500004e-01 1.70269996e-01]
[ 7.54669979e-02 -2.92360008e-01 -2.60369986e-01 -2.81670004e-01
1.60970002e-01 -1.94720000e-01 -2.82059997e-01 4.91409987e-01
-4.11189999e-03 9.58790034e-02 3.33959997e-01 1.68740004e-02
2.06159994e-01 -1.57429993e-01 -1.61379993e-01 2.15379998e-01
-1.48359999e-01 2.54599992e-02 -4.36809987e-01 -6.36610016e-02
5.52609980e-01 3.08189988e+00 -1.17839999e-01 -4.61309999e-01
-2.76089996e-01 2.09480003e-01 -2.05440000e-01 -5.71550012e-01
3.34479988e-01 1.59130007e-01 2.54360004e-03 1.80040002e-01
1.34719998e-01 -9.74040031e-02 3.55369985e-01 -4.74280000e-01
-7.92569995e-01 -5.44179976e-01 2.42999997e-02 6.35990024e-01
1.23369999e-01 -1.29130006e-01 -2.65650004e-01 -2.49569997e-01
-5.21990001e-01 -4.05229986e-01 4.84030008e-01 1.83729995e-02
2.30389997e-01 6.21379986e-02 -1.92919999e-01 2.95060009e-01
-3.57930005e-01 1.67019993e-01 3.18679988e-01 -3.60540003e-01
-1.09779999e-01 -1.56320006e-01 4.59210008e-01 9.64749977e-02
-3.77000004e-01 -7.76150003e-02 -4.88990009e-01 2.05750003e-01
5.05429983e-01 5.34190014e-02 -2.59779990e-01 5.10420024e-01
9.75209996e-02 3.26059997e-01 1.43539995e-01 2.27149995e-03
4.86149997e-01 4.69379991e-01 -4.11240011e-01 -1.71480000e-01
-3.97439986e-01 -2.89009988e-01 -1.77560002e-01 3.70009989e-02
3.48300010e-01 1.59339994e-01 -7.42810011e-01 1.88970000e-01
4.36850004e-02 5.72080016e-01 -6.70159996e-01 -4.39470001e-02
-2.83360004e-01 -3.19959998e-01 -2.04040006e-01 -8.78980011e-02
-1.57240003e-01 2.18180008e-02 -5.67569971e-01 6.32960021e-01
-1.00970000e-01 -6.55760020e-02 5.82689978e-03 3.30349989e-02
3.97830009e-01 -3.11659992e-01 -6.10889971e-01 2.75590003e-01
1.00079998e-01 -4.19900000e-01 6.35599997e-03 1.87170005e+00
3.14729989e-01 -3.60040009e-01 8.13839972e-01 -2.17099994e-01
-1.84589997e-02 -2.26319999e-01 1.45850003e-01 -1.43500000e-01
-4.14239988e-02 5.59740007e-01 -6.67519987e-01 -2.19589993e-01
1.90109998e-01 3.30150008e-01 6.12900019e-01 4.67709988e-01
4.20260012e-01 -5.28190017e-01 2.31650006e-02 3.29100005e-02
4.73060012e-01 1.40060000e-02 -1.73960000e-01 -4.43619996e-01
4.13769990e-01 -2.06790000e-01 3.92830014e-01 3.02109987e-01
7.31339976e-02 4.21640016e-02 -9.27100003e-01 -4.76139992e-01
2.43100002e-01 -1.33790001e-01 -2.22379997e-01 -4.14570011e-02
1.58500004e+00 3.74810010e-01 2.59939991e-02 -2.42719993e-01
3.05779994e-01 1.46870002e-01 1.16659999e-01 -2.94179991e-02
-7.83390030e-02 -2.25119993e-01 1.33149996e-01 -6.48420006e-02
-2.86870003e-01 -1.05600003e-02 -3.46679986e-01 4.21449989e-02
-6.00409985e-01 8.24810028e-01 3.10220003e-01 1.64890006e-01
-7.29210004e-02 1.93939999e-01 -9.84980017e-02 -2.03830004e-02
-4.09090012e-01 -1.04039997e-01 1.91689998e-01 -1.59689993e-01
3.80259991e-01 6.28019989e-01 2.59499997e-01 -3.33669990e-01
-7.33330011e-01 -4.07429993e-01 6.84230030e-01 -6.63380027e-02
5.04360020e-01 -2.89829999e-01 -3.90859991e-01 -4.59310003e-02
-2.66240001e-01 -1.67699993e-01 -1.50370002e-01 1.48279995e-01
-2.81430006e-01 -1.70870006e-01 -2.55760014e-01 -5.62830009e-02
-1.66500002e-01 3.51060003e-01 4.10320014e-02 2.73110002e-01
3.00200004e-02 1.64649993e-01 -8.41889977e-02 5.75059988e-02]
[ 5.46909988e-01 -7.55890012e-01 -9.20799971e-01 -8.20680022e-01
1.48800001e-01 -1.32200003e-01 2.49499991e-03 5.39979994e-01
-3.12929988e-01 -2.12009996e-02 -2.96559989e-01 1.51110003e-02
-3.76980007e-02 -4.07290012e-01 3.36799994e-02 -2.90179998e-01
-5.64790010e-01 6.47960007e-01 3.96770000e-01 -1.91139996e-01
6.98249996e-01 1.61090004e+00 8.42899978e-01 -1.45980000e-01
-1.13940001e-01 -5.39680004e-01 -7.22540021e-01 -1.47310004e-01
-6.78520024e-01 1.93159997e-01 4.17819992e-03 -3.96059990e-01
8.99320021e-02 3.30909997e-01 -4.09649983e-02 4.59470004e-01
-3.07020009e-01 -2.61469990e-01 5.39489985e-01 6.60130024e-01
-5.68049997e-02 -3.70750010e-01 -1.46880001e-03 -1.48770005e-01
2.13000000e-01 -1.36710003e-01 5.12350023e-01 -5.27249992e-01
2.73449998e-02 3.00159991e-01 4.59829986e-01 3.65709990e-01
2.23989993e-01 1.47589996e-01 -9.97050032e-02 2.26520002e-01
-5.95120013e-01 -4.34839994e-01 -5.26629984e-02 -3.05240005e-01
-1.91280007e-01 -4.08630013e-01 -2.83719987e-01 6.89159989e-01
-7.18529999e-01 2.89909989e-01 -1.31809995e-01 -8.75229985e-02
2.70280004e-01 3.31449993e-02 1.93159997e-01 -2.50629991e-01
1.13150001e-01 4.88560013e-02 -4.41439986e-01 -8.88149977e-01
-3.45840007e-01 -5.47370017e-01 2.13919997e-01 1.46469995e-01
-3.76450002e-01 5.63120008e-01 4.44130003e-01 -6.05080016e-02
5.85870028e-01 4.50850010e-01 -1.27309993e-01 6.86409995e-02
3.78729999e-01 -7.15900004e-01 2.72450000e-01 2.16839999e-01
-5.09050012e-01 2.82240003e-01 5.28559983e-01 2.97140002e-01
-3.33429992e-01 -1.62200004e-01 -3.97210002e-01 3.37130010e-01
6.61469996e-01 -4.09590006e-01 -1.97520003e-01 -8.77860010e-01
-4.36379999e-01 1.79399997e-01 3.39769991e-03 9.73770022e-01
7.32789993e-01 -3.44499983e-02 -4.76709992e-01 -4.99819994e-01
4.97110009e-01 -7.77419984e-01 -3.60000014e-01 -1.58899993e-01
1.19869998e-02 -1.39750004e-01 -5.91250002e-01 -1.58280000e-01
9.19710025e-02 4.62440014e-01 -1.17980000e-02 8.29930007e-01
7.96280026e-01 -5.64369977e-01 -1.02080004e-02 -3.86090010e-01
-6.74889982e-01 -3.42109986e-02 1.99829996e-01 2.01010004e-01
7.43009984e-01 -3.58200014e-01 1.07579999e-01 -7.77289987e-01
3.24770004e-01 -7.43889987e-01 -7.90560007e-01 4.95799989e-01
3.70090008e-01 -1.55349998e-02 1.20260000e+00 -6.90900028e-01
1.49619997e+00 4.31109995e-01 2.64299989e-01 -5.99219978e-01
-4.51359987e-01 7.66879976e-01 -1.43020004e-02 1.43340006e-01
1.19379997e-01 -5.56930006e-01 -9.95339990e-01 8.20089996e-01
-3.02410007e-01 9.95709971e-02 -4.92139995e-01 2.25950003e-01
5.18890023e-01 6.51669979e-01 -7.13440031e-02 3.06109991e-03
-2.95789987e-01 -4.40490007e-01 -1.61009997e-01 2.52220005e-01
-3.50809991e-01 -7.96020031e-02 4.33560014e-01 -1.43889993e-01
-1.26969993e-01 1.01450002e+00 -6.24639988e-02 -5.60909986e-01
-6.52670026e-01 1.86299995e-01 -3.76190007e-01 -2.40750000e-01
2.50979990e-01 -2.06090003e-01 -8.57209980e-01 -5.84480017e-02
-1.23539999e-01 -7.35830009e-01 -4.83509988e-01 2.75110006e-01
-1.10220000e-01 3.17330003e-01 1.87999994e-01 1.05719995e+00
5.85460007e-01 3.13740000e-02 2.11569995e-01 1.13799997e-01
4.65829998e-01 -3.05020005e-01 -4.51409996e-01 2.87349999e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[-0.34636 -0.88984001 -0.50321001 -0.43516001 0.54518998 0.17437001
-0.093541 0.16141 -0.46575999 -0.22 -0.31415001 -0.13484
-0.37617999 -0.67635 0.78820002 -0.33384001 -0.42414001 0.32367
0.50670999 0.21540999 0.43296 1.49049997 0.31795001 -0.15196
0.2579 -0.35666001 -0.63880002 -0.086453 -0.94755 0.19203
0.31161001 -0.74491 -0.59210998 0.4332 -0.064934 -0.48862001
0.35600999 -0.44780999 -0.015773 0.18203001 0.051751 -0.2854
-0.14727999 0.1513 -0.33048001 0.27135 1.16659999 -0.36662
0.090829 0.87301999 -0.13505 0.21204001 0.57270002 0.54259002
-0.50335002 0.16767 -0.82530999 -0.45962 -0.42642 -0.2164
0.088689 -0.15061 -0.16785 -0.31794 -0.69608998 0.40715
-0.29190999 -0.042072 0.90051001 0.35947999 0.030644 -0.028853
0.086821 0.74741 -0.52008998 0.20655 0.44053999 -0.11865
-0.15449999 -0.22457001 -0.15453 0.16101 -0.30825001 -0.28479999
-0.50823998 0.48732999 -0.012029 0.034592 0.48304 -0.56752002
-0.057299 0.22070999 -0.34200001 -0.060634 0.95499998 -0.60952997
0.59577 -0.11553 -0.67475998 0.52658999 0.82163 0.35126001
0.15521 0.12135 0.38191 0.24228001 -0.51485997 1.14810002
0.07281 0.23024 -0.68901998 -0.17606001 -0.24308001 -0.13686
-0.13467 0.059625 -0.68668997 0.15907 0.11983 -0.024954
0.34898001 0.15456 0.047524 0.23616999 0.54784 -1.01380002
0.10351 0.26865 -0.064867 0.23893 0.026141 0.081648
0.74479997 -0.67208999 0.23351 -0.55070001 -0.14296 -0.30303001
-0.40233999 0.012984 0.86865002 -0.14025 1.13900006 -0.093339
1.56060004 0.41804001 0.54067999 -0.43340999 -0.090589 0.56682003
-0.21291 0.45693001 -0.64519 -0.05866 0.21477 0.45563
-0.15218 0.36307001 -0.25441 -0.72013998 0.52191001 0.55509001
-0.073841 0.44994 -0.11501 0.1911 0.077304 0.18629
0.60244 0.028085 0.17228 -0.24455 0.04822 0.51318002
-0.06824 0.35515001 -0.80987 -0.42732999 -0.72728997 0.47817001
0.87611997 -0.18855 0.30390999 -0.14161 0.26699001 -0.81572002
-0.67589998 -0.34687999 0.53188998 0.75443 -0.083874 0.77434999
0.081108 -0.29840001 -0.24409001 -0.14574 -0.1186 0.085964
0.48076999 -0.13097 ]
[ 1.07439995 -0.49325001 -0.23126 -0.197 -0.087485 -0.16806
-0.11092 0.42857999 -0.16893999 0.0094633 -0.50453001 -0.40006
0.31169 0.50295001 -0.48537001 -0.20689 -0.62162 0.38407999
0.22182 0.051087 -0.018028 1.37919998 0.3163 -0.17425001
0.11344 -0.42723 -0.28510001 -0.1246 -0.2024 0.18217
-0.37834001 -0.22421999 0.38877001 0.20065001 -0.29708999 0.77051002
0.13087 -0.25084001 0.54755002 0.38086 0.28174999 -0.15691
-0.71987998 0.24118 0.073913 -0.46965 1.02180004 0.049863
0.036841 0.54706001 -0.15903001 0.53780001 -0.10335 0.51644999
-0.25512001 -0.18553001 -0.51804 -0.24337 0.57081997 -0.39017001
-0.17132001 -0.14939 -0.1724 0.91408002 -0.45778 0.40143001
0.075224 -0.4104 -0.1714 -0.63552999 0.60185999 -0.3193
-0.46136999 0.030652 0.32890001 -0.2472 -0.49831 -0.90982997
-0.057251 0.20213 -0.51845998 0.46320999 0.032707 0.29872999
1.11189997 -0.35784999 0.34929001 -0.51739001 0.25235 -1.0309
0.21052 0.06349 -0.10589 0.43222001 0.20389 0.065589
-0.62914002 0.1096 -0.86363 0.44760999 0.43114999 0.041376
-0.42429999 -0.080897 0.093115 0.22603001 0.31176999 0.83004999
-0.25659001 0.013861 0.38326001 -0.52025998 0.30410001 -0.52507001
-0.78566003 -0.046498 0.41154999 -0.21447 -0.24202999 -0.24732
1.01129997 0.067517 0.18331 -0.17636 0.49777001 -0.21067999
0.0037579 0.22881 -0.15993001 -0.13421001 -0.27379999 -0.20734
0.13407999 -0.57762003 -0.66526997 -0.42083001 0.65882999 -0.53825998
-0.50585997 0.51735002 0.25468999 -0.83724999 0.83806998 -0.42219999
1.0776 0.065962 0.48954999 -0.78614002 -0.19338 0.097524
0.27215999 0.037038 0.61778998 -0.29506999 -0.97285002 0.53106999
-0.32765001 -0.045966 -0.75436997 0.024904 0.64872003 0.023095
0.32062 0.35837999 -0.091125 -0.10866 0.33048001 -0.1162
-0.40981999 0.43928999 -0.16706 0.26047 0.090957 0.92714
0.099946 -0.29513001 -0.35341001 0.33693001 -0.42203999 -0.065625
0.54738998 -0.41751 -0.86232001 -0.65891999 -0.41549 0.067035
-0.41558 0.15092 0.17556 0.94068003 0.22527 0.65908003
0.15809 0.061199 0.63612998 -0.17089 -0.017591 -0.054003
-0.69072002 0.65179998]
[ 1.03529997e-01 7.20499977e-02 -2.93029994e-02 -4.46799994e-01
-8.61259997e-02 7.40030035e-02 -4.65499997e-01 -6.18570000e-02
-5.03650010e-01 1.95480004e-01 -1.03349999e-01 7.75929987e-01
-1.72040001e-01 -4.53520000e-01 2.63040006e-01 1.64340004e-01
2.80279994e-01 2.89330006e-01 3.10360014e-01 1.27750002e-02
6.79520011e-01 2.62770009e+00 3.89640003e-01 -4.49900001e-01
2.17969999e-01 9.16400030e-02 -1.67940006e-01 7.72420019e-02
2.91269988e-01 1.20530002e-01 -5.49659990e-02 9.16600004e-02
1.31300002e-01 -9.33270007e-02 -3.53009999e-01 -5.03880024e-01
-7.29799986e-01 -3.61380011e-01 -2.99659997e-01 2.07839999e-02
-7.03599975e-02 -6.72269985e-02 -3.62650007e-02 -1.46009997e-02
-5.98580018e-02 -1.63020000e-01 3.00660014e-01 -9.28120017e-02
-6.69799969e-02 1.43830001e-01 1.03950001e-01 -1.93039998e-02
-4.07020003e-01 8.86749983e-01 2.67349988e-01 -1.23829998e-01
2.73739994e-02 3.44639987e-01 6.04049981e-01 2.80640006e-01
1.41320005e-01 2.46429995e-01 -4.48760003e-01 5.42909980e-01
1.96759999e-01 -4.94709998e-01 2.68779993e-02 2.69100010e-01
2.53410012e-01 9.88679975e-02 1.77919999e-01 -3.22290003e-01
-1.05930001e-01 1.89520001e-01 -2.57629991e-01 2.43619993e-01
-2.45560005e-01 -1.36539996e-01 -1.08170003e-01 -4.21220005e-01
-1.61640003e-01 -3.41199994e-01 -1.47389993e-01 -2.16409996e-01
-5.62399998e-02 7.15879977e-01 6.40570000e-02 -3.07790011e-01
6.57369971e-01 -6.36910021e-01 2.64039993e-01 2.15059996e-01
1.83850005e-01 3.32610011e-01 -6.30220026e-02 -2.22450003e-01
6.31980002e-02 -4.47950006e-01 -2.52279997e-01 1.97699994e-01
-3.55479985e-01 1.19139999e-01 -2.99149990e-01 1.29390001e-01
-4.30770010e-01 -1.37700006e-01 2.38429993e-01 1.45529997e+00
-1.13710001e-01 -3.79790008e-01 -2.83129990e-01 -4.64819998e-01
-1.92760006e-01 1.98490005e-02 2.70090014e-01 -1.70849994e-01
-5.20099998e-02 -2.03219995e-01 -3.27439994e-01 -5.07570028e-01
2.98289992e-02 1.80350006e-01 3.05330008e-01 2.40700006e-01
4.66120005e-01 -7.12530017e-01 5.96719980e-01 2.13310003e-01
3.48639995e-01 3.80190015e-01 1.38260007e-01 -7.33380020e-02
2.26349995e-01 -4.55599993e-01 8.19810014e-03 3.47319990e-01
-1.36079997e-01 -6.59820020e-01 -2.79430002e-01 4.81799990e-02
-3.97129990e-02 2.86449999e-01 1.41259998e-01 -3.94299999e-02
1.44519997e+00 4.59950000e-01 7.92850032e-02 -3.55580002e-01
3.09360009e-02 -2.50809994e-02 5.64870015e-02 9.00759995e-02
5.20309992e-02 -3.99529994e-01 3.40330005e-01 -4.17400002e-01
-3.82189989e-01 2.22049996e-01 1.09520003e-01 -1.64539993e-01
1.20609999e-01 1.60919994e-01 -3.27459991e-01 2.45800003e-01
-2.28320006e-02 2.74109989e-01 -2.10689995e-02 3.91460001e-01
-5.61020017e-01 6.95510030e-01 -3.55260000e-02 4.71640006e-02
6.71909988e-01 4.81240004e-01 -2.78840009e-02 5.05490005e-01
-5.41859984e-01 -3.52369994e-01 -3.12009990e-01 -1.76020002e-03
9.59439993e-01 -5.03639996e-01 4.39889997e-01 4.71810013e-01
4.25799996e-01 -5.92209995e-01 -3.96219999e-01 1.89040005e-02
3.33819985e-02 -2.90149987e-01 -1.22079998e-01 2.76309997e-02
-2.66250014e-01 -1.97340008e-02 2.31020004e-01 8.76149982e-02
-7.69149978e-03 1.90050006e-02 -4.42119986e-01 -6.81999996e-02]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 0.041052 -0.54705 -0.72193998 -0.31235999 -0.43849 0.10691
-0.50641 -0.45401001 -0.28623 0.018973 0.020495 0.42860001
-0.0057162 -0.21272001 0.71864998 -0.091906 -0.55365002 0.39133
0.15351 -0.27454001 0.56528002 3.04830003 0.30467001 -0.37893
0.37865999 0.13925999 -0.11482 0.48212999 -0.30522999 0.43125001
-0.09667 0.069156 0.31426001 0.26350999 -0.31189999 -0.39881
-0.55656999 -0.35934001 -0.25402001 0.072061 -0.12966999 -0.11247
-0.041192 -0.042619 -0.07848 0.31992 0.41635999 0.26131001
-0.18175 -0.1279 0.21332 -0.41973001 -0.50444001 0.37705001
0.83955002 -0.34571001 -0.43000999 -0.18653999 -0.061082 -0.087612
0.092833 0.52604997 -0.57611001 -0.19328 0.20576 0.24607
0.1631 -0.18138 0.032592 0.19169 0.73565 -0.25718999
0.30072999 0.56699002 -0.21544001 0.18933 -0.12287 -0.65759999
0.021702 0.1041 0.098952 -0.43171999 -0.27517 -0.15448
0.31301001 -0.032041 -0.090526 0.14489999 0.68151999 -0.88242
0.30816999 -0.62702 0.12274 0.014773 -0.16887 0.56159002
0.022004 0.52085 -0.22685 0.09633 0.26956999 0.30489001
0.018463 0.31009001 0.04198 0.32381999 0.13153 0.89722002
0.15475 -0.38806999 -0.52993 -0.35383999 -0.0913 0.57230002
0.48067001 0.24438 0.074138 -0.019957 -0.35030001 -0.034695
-0.12045 0.39998999 -0.37015 -0.53040999 0.10655 -0.44973001
0.43105 -0.44937 0.48675999 0.43836999 0.043421 0.52675003
0.61176002 0.26704001 0.59239 0.23650999 0.12841 -0.10665
-0.46715 -0.039081 -0.24921 0.030486 0.092933 -0.04841
1.83580005 0.077535 -0.11494 -0.13668001 -0.23983 0.31948
0.19205999 0.38894001 0.34755 -0.038804 0.19541 -0.37099999
-0.027576 -0.24127001 -0.16868 0.032815 0.08139 0.054121
-0.42785999 0.26447001 0.054847 -0.21765999 0.015181 0.57656002
0.24033 0.62076002 -0.019055 -0.31509 0.76424998 0.35168999
-0.28512001 0.15175 0.11238 -0.60829997 0.35087001 0.19140001
0.51753998 0.20893 -0.63814002 0.19403 0.24493 0.46606001
-0.32235 0.37286001 -0.19508 0.13237999 -0.35420999 0.22849
0.36032 -0.0050241 -0.051955 -0.37755999 -0.087065 0.3592
0.11564 0.44372001]
[ 4.71520007e-01 -5.87790012e-01 -6.76150024e-01 -4.67000008e-01
1.11709997e-01 3.26370001e-01 -4.65070009e-01 -8.05180013e-01
-1.65340006e-01 -1.13849998e-01 -1.36849999e-01 2.56980002e-01
3.36279988e-01 1.90149993e-01 1.32440001e-01 2.37419993e-01
-1.46239996e+00 8.59059989e-01 5.53650022e-01 1.94330007e-01
2.73930013e-01 1.05669999e+00 6.24970019e-01 -4.30469990e-01
7.14770019e-01 -4.73030001e-01 -8.97960007e-01 2.56910007e-02
-6.42499983e-01 2.15990007e-01 -1.22769997e-01 -5.36949992e-01
5.91489971e-01 6.28649965e-02 1.51260002e-02 -6.57150000e-02
1.61709994e-01 -8.86740014e-02 -7.09370002e-02 6.12349987e-01
1.38689995e-01 -3.67980003e-01 -9.46219981e-01 1.30669996e-01
-2.82139987e-01 -3.02709997e-01 4.05889988e-01 -2.11899996e-01
1.74940005e-01 2.38450006e-01 3.41769993e-01 4.50269997e-01
-7.82140017e-01 1.64210007e-01 7.19319999e-01 -6.80140018e-01
-4.93660003e-01 3.67380008e-02 2.62410015e-01 -8.48299980e-01
-6.59759998e-01 4.04370010e-01 -2.61209998e-02 5.83829999e-01
-3.28000009e-01 6.39530003e-01 1.20350003e-01 7.21519988e-04
8.28130007e-01 -3.83879989e-01 5.35929978e-01 -4.59630013e-01
-5.12839973e-01 1.74339995e-01 2.06220001e-01 -8.01329970e-01
-4.74339992e-01 -4.32810009e-01 -6.11400008e-01 1.71409994e-01
5.54369986e-01 6.11240007e-02 6.43959999e-01 -5.23599982e-01
1.35130000e+00 -1.40279993e-01 3.67210001e-01 -3.68629992e-01
7.95690000e-01 -1.01139998e+00 -1.47060007e-01 4.48889993e-02
4.06240001e-02 5.33150017e-01 3.40680003e-01 2.50710011e-01
-1.26489997e-01 -1.15050003e-01 -1.48660004e-01 7.65860021e-01
9.80419964e-02 6.28759980e-01 -4.09599990e-01 -3.33020017e-02
-3.77389997e-01 -2.54250001e-02 -8.92150030e-02 1.63540006e+00
5.04270017e-01 -3.86139989e-01 -1.25259995e-01 1.65910006e-01
-2.19990000e-01 -6.84350014e-01 -2.99389988e-01 -1.25550002e-01
3.63970011e-01 3.89310002e-01 -9.63360012e-01 -7.42670000e-02
3.48879993e-01 2.36190006e-01 -8.27549994e-01 3.19779992e-01
-3.00359994e-01 -4.01740015e-01 7.04670012e-01 -3.32989991e-01
-1.26369998e-01 -3.72830003e-01 -9.09730017e-01 -1.33279994e-01
-5.32779992e-02 -3.47909987e-01 2.48980001e-01 -4.34430003e-01
-2.42050007e-01 -6.21890008e-01 -1.27769995e+00 -9.66310024e-01
4.20859993e-01 -4.12339985e-01 1.20589995e+00 -2.55379993e-02
1.01170003e+00 -1.11219997e-03 5.92599988e-01 -5.58459997e-01
-3.69489998e-01 5.58549985e-02 -6.60969973e-01 3.91559988e-01
3.09260011e-01 -6.28539994e-02 -1.14900005e+00 4.69440013e-01
-5.05439997e-01 -1.75500005e-01 -3.80259991e-01 -1.97349995e-01
7.31180012e-01 -1.77149996e-01 4.18940008e-01 4.23940003e-01
-5.16149998e-02 -1.20180003e-01 3.96189988e-01 -2.36780003e-01
1.19029999e-01 2.93020010e-01 -1.81150004e-01 4.59089994e-01
1.93489999e-01 4.75919992e-01 -3.67170006e-01 -8.76540027e-04
-6.15379997e-02 -4.12099995e-02 -1.95240006e-01 -6.47260016e-03
4.30709988e-01 6.84989989e-02 -4.27579999e-01 -3.15990001e-01
-2.59119987e-01 -6.58209980e-01 -1.92790002e-01 5.57280004e-01
-2.24089995e-01 2.21340001e-01 4.68760014e-01 6.37290001e-01
6.58949971e-01 8.79120007e-02 -3.06369990e-01 -3.22079986e-01
6.47899985e-01 1.75490007e-01 -5.78580022e-01 8.94800007e-01]
===================================
===== For the word2vec
===================================
1. Download and unzip the pre-trained model of GoogleNews-vectors-negative300.bin.gz.
2. Install the gensim tools:
sudo pip install --upgrade gensim
3. Code for vector extraction from given sentence.
import gensim
print("==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin")
dictFileName = './GoogleNews-vectors-negative300.bin'
wv = gensim.models.KeyedVectors.load_word2vec_format(dictFileName, binary=True)
The Output is:
==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loading projection weights from ./GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loaded (3000000, 300) matrix from ./GoogleNews-vectors-negative300.bin
INFO:root:Data statistic
INFO:root:train_labels:19337
INFO:root:test_labels:20632
INFO:root:train_sentences:19337
INFO:root:dev_sentences:2000
INFO:root:test_sentences:20632
INFO:root:dev_labels:2000
embed_size:300
vocab_size:3000000
vocab_path = ''data/bilstm.vocab''
index_to_word = [key for key in wv.vocab]
word_to_index = {}
for index, word in enumerate(index_to_word):
word_to_index[word] = index
with open(vocab_path, "w") as f:
f.write(json.dumps(word_to_index))
Let's take a deep understanding on the bidirectional-LSTM-for-text-classification-master
class BiLSTM(nn.Module):
def __init__(self, embedding_matrix, hidden_size=150, num_layer=2, embedding_freeze=False):
super(BiLSTM,self).__init__() # embedding layer
vocab_size = embedding_matrix.shape[0]
embed_size = embedding_matrix.shape[1]
self.hidden_size = hidden_size
self.num_layer = num_layer
self.embed = nn.Embedding(vocab_size, embed_size)
self.embed.weight = nn.Parameter(torch.from_numpy(embedding_matrix).type(torch.FloatTensor), requires_grad=not embedding_freeze)
self.embed_dropout = nn.Dropout(p=0.3)
self.custom_params = []
if embedding_freeze == False:
self.custom_params.append(self.embed.weight) # The first LSTM layer
self.lstm1 = nn.LSTM(embed_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm1.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.lstm1_dropout = nn.Dropout(p=0.3) # The second LSTM layer
self.lstm2 = nn.LSTM(2*self.hidden_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm2.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param)
self.lstm2_dropout = nn.Dropout(p=0.3) # Attention
self.attention = nn.Linear(2*self.hidden_size,1)
self.attention_dropout = nn.Dropout(p=0.5) # Fully-connected layer
self.fc = weight_norm(nn.Linear(2*self.hidden_size,3))
for param in self.fc.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.hidden1=self.init_hidden()
self.hidden2=self.init_hidden() def init_hidden(self, batch_size=3):
if torch.cuda.is_available():
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda(),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda())
else:
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size))) def forward(self, sentences): print("==>> sentences: ", sentences) # get embedding vectors of input
padded_sentences, lengths = torch.nn.utils.rnn.pad_packed_sequence(sentences, padding_value=int(0), batch_first=True)
print("==>> padded_sentences: ", padded_sentences) embeds = self.embed(padded_sentences)
print("==>> embeds: ", embeds) # pdb.set_trace() noise = Variable(torch.zeros(embeds.shape).cuda())
noise.data.normal_(std=0.3)
embeds += noise
embeds = self.embed_dropout(embeds)
# add noise packed_embeds = torch.nn.utils.rnn.pack_padded_sequence(embeds, lengths, batch_first=True) print("==>> packed_embeds: ", packed_embeds) # First LSTM layer
# self.hidden = num_layers*num_directions batch_size hidden_size
packed_out_lstm1, self.hidden1 = self.lstm1(packed_embeds, self.hidden1)
padded_out_lstm1, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm1, padding_value=int(0))
padded_out_lstm1 = self.lstm1_dropout(padded_out_lstm1)
packed_out_lstm1 = torch.nn.utils.rnn.pack_padded_sequence(padded_out_lstm1, lengths) pdb.set_trace() # Second LSTM layer
packed_out_lstm2, self.hidden2 = self.lstm2(packed_out_lstm1, self.hidden2)
padded_out_lstm2, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm2, padding_value=int(0), batch_first=True)
padded_out_lstm2 = self.lstm2_dropout(padded_out_lstm2) # attention
unnormalize_weight = F.tanh(torch.squeeze(self.attention(padded_out_lstm2), 2))
unnormalize_weight = F.softmax(unnormalize_weight, dim=1)
unnormalize_weight = torch.nn.utils.rnn.pack_padded_sequence(unnormalize_weight, lengths, batch_first=True)
unnormalize_weight, lengths = torch.nn.utils.rnn.pad_packed_sequence(unnormalize_weight, padding_value=0.0, batch_first=True)
logging.debug("unnormalize_weight size: %s" % (str(unnormalize_weight.size())))
normalize_weight = torch.nn.functional.normalize(unnormalize_weight, p=1, dim=1)
normalize_weight = normalize_weight.view(normalize_weight.size(0), 1, -1)
weighted_sum = torch.squeeze(normalize_weight.bmm(padded_out_lstm2), 1) # fully connected layer
output = self.fc(self.attention_dropout(weighted_sum))
return output
==>> Some Testing:
(a). len(wv.vocab) = 300,0000
(b). wv.vocab is what ? Something like this:
{ ... , u'fivemonth': <gensim.models.keyedvectors.Vocab object at 0x7f90945bd810>,
u'retractable_roofs_Indians': <gensim.models.keyedvectors.Vocab object at 0x7f90785f5690>,
u'Dac_Lac_province': <gensim.models.keyedvectors.Vocab object at 0x7f908d8eda10>,
u'Kenneth_Klinge': <gensim.models.keyedvectors.Vocab object at 0x7f9081563410>}
(c). index_to_word: count the words from the pre-trained model.
{ ... , u"Lina'la_Sin_Casino", u'fivemonth', u'retractable_roofs_Indians', u'Dac_Lac_province', u'Kenneth_Klinge'}
(d). word_to_index: give each word a index as following
{ ... , u'fivemonth': 2999996, u'Pidduck': 2999978, u'Dac_Lac_province': 2999998, u'Kenneth_Klinge': 2999999 }
(e). dataset is:
[ ... , [1837614, 1569052, 1837614, 1288695, 2221039, 2323218, 1837614, 1837614, 2029395, 1612781, 311032, 1524921, 1837614, 2973515, 2033866, 882731, 2462275, 2809106, 1479961, 826019, 73590, 953550, 1837614],
[1524921, 1113778, 1837614, 318169, 1837614, 1954969, 196613, 943118, 1837614, 2687790, 291413, 2774825, 2366038, 296869, 1468080, 856987, 1802099, 724308, 1207907, 2264894, 2206446, 812434],
[564298, 477983, 1837614, 1449153, 1837614, 211925, 2206446, 481834, 488597, 280760, 2072822, 1344872, 1791678, 2458776, 2965810, 2168205, 387112, 2656471, 1391, 1837614, 1801696, 2093846, 1210651],
[2493381, 133883, 2441902, 1014220, 1837614, 2597880, 1756105, 2651537, 1391, 2641114, 2517536, 1109601, 122269, 1782479, 2965835, 488597, 1767716, 753333, 564298, 2380935, 228060, 1837614, 371618],
[1837614, 1344872, 2458776, 2965810, 1837614, 2015408, 1837614, 528014, 1991322, 1837614, 908982, 1484130, 2349526, 988689, 753336, 1837614, 364492, 2183116, 826019, 73590, 953550, 1837614],
[1837614, 2673785, 1990947, 1219831, 2635341, 1247040, 1837614, 799543, 1990947, 1219831, 2722301, 1837614, 1427513, 969099, 2157673, 1430111, 1837614]]
(f). the variable during the training process:
# ('==>> sentences: ', PackedSequence(data=tensor(
# [ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# ('==>> padded_sentences: ', tensor(
# [[ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06]], device='cuda:0'))
# length: 23
# ('==>> embeds: ', tensor([[
# [-0.0684, 0.1826, -0.1777, ..., 0.1904, -0.1021, 0.1729],
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.1982, -0.0693, 0.1230, ..., -0.1357, -0.0306, 0.1104],
# ...,
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.0518, -0.0299, 0.0415, ..., 0.0776, -0.1660, 0.1602],
# [-0.0532, -0.0004, 0.0337, ..., -0.2373, -0.1709, 0.0233]]], device='cuda:0'))
# ('==>> packed_embeds: ', PackedSequence(data=tensor([
# [-0.3647, 0.2966, -0.2359, ..., -0.0000, 0.2657, -0.4302],
# [ 1.1699, 0.0000, 0.3312, ..., 0.5714, 0.1930, -0.2267],
# [-0.0627, -1.0548, 0.4966, ..., -0.5135, -0.0150, -0.0000],
# ...,
# [-0.6065, -0.7562, 0.3320, ..., -0.5854, -0.2089, -0.5737],
# [-0.0000, 0.4390, 0.0000, ..., 0.6891, 0.0250, -0.0000],
# [ 0.6909, -0.0000, -0.0000, ..., 0.1867, 0.0594, -0.2385]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# packed_out_lstm1
# PackedSequence(data=tensor([
# [-0.0000, -0.0000, 0.1574, ..., 0.1864, -0.0901, -0.3205],
# [-0.0000, -0.3490, 0.1774, ..., 0.1677, -0.0000, -0.3688],
# [-0.3055, -0.0000, 0.2240, ..., 0.0000, -0.0927, -0.0000],
# ...,
# [-0.3188, -0.4134, 0.1339, ..., 0.3161, -0.0000, -0.3846],
# [-0.3355, -0.4365, 0.1575, ..., 0.2775, -0.0886, -0.4015],
# [-0.0000, -0.0000, 0.2452, ..., 0.1763, -0.0000, -0.2748]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]))
## self.hidden1
# self.hidden1
# (tensor([
# [[-0.0550, -0.4266, 0.0498, 0.0750, 0.3122, 0.0499, -0.2899,
# -0.0980, -0.1776, 0.4376, 0.1555, 0.7167, -0.0162, 0.0081,
# -0.3512, 0.0250, 0.0948, -0.3596, 0.0670, -0.6623, 0.0026,
# -0.0262, 0.1934, 0.3206, -0.1941, 0.3229, -0.6488, -0.3858,
# 0.3169, -0.0216, -0.1136, 0.1411, -0.0296, -0.2860, -0.1277,
# -0.0154, 0.2620, -0.3553, 0.0637, 0.8245, 0.6886, -0.0048,
# -0.1717, 0.2495, -0.8636, -0.0217, -0.2365, 0.1324, 0.1790,
# -0.1515, 0.3530, -0.1644, 0.0073, 0.6709, 0.1577, -0.0190,
# 0.0384, 0.4871, -0.7375, -0.1804, 0.3034, -0.3516, -0.2870,
# 0.6387, 0.0414, 0.6983, -0.3211, 0.0449, 0.2127, -0.0421,
# -0.3454, -0.8145, -0.3629, -0.0828, -0.1558, 0.4048, -0.4971,
# -0.7495, 0.0622, -0.3318, 0.3913, -0.0322, -0.0678, 0.0307,
# -0.0153, -0.1535, -0.2719, 0.0128, -0.1521, -0.2924, 0.7109,
# -0.8551, 0.0330, 0.0482, 0.2410, -0.0655, -0.2496, 0.1816,
# 0.4963, -0.7593, -0.0022, -0.1122, 0.6857, -0.5693, 0.5805,
# 0.7660, 0.4430, 0.0243, -0.0313, 0.0780, -0.2419, 0.0745,
# -0.0119, 0.5761, -0.0285, -0.1085, -0.1783, -0.0706, 0.1243,
# 0.6333, 0.0296, 0.5557, 0.2717, -0.0071, 0.0503, -0.0405,
# -0.4542, 0.8905, 0.4492, 0.8809, 0.7021, 0.8616, 0.2825,
# -0.2114, 0.3026, -0.1384, 0.1252, 0.4989, 0.2236, -0.5374,
# -0.1352, -0.0561, 0.0378, -0.5291, -0.1004, -0.3723, 0.0836,
# 0.3500, 0.0542, 0.2013]], # [[-0.0041, -0.0040, -0.1428, 0.2783, -0.1378, -0.4242, 0.1000,
# 0.1641, -0.0175, -0.0896, 0.3241, 0.3513, -0.4675, -0.1250,
# 0.0546, 0.2400, -0.0997, -0.5614, 0.2026, -0.1505, 0.0833,
# -0.3128, -0.4646, -0.0778, 0.2204, 0.5597, -0.7004, 0.0419,
# -0.3699, -0.5748, 0.7741, -0.7220, 0.0494, 0.3430, 0.1389,
# 0.0178, 0.0136, 0.0273, 0.1559, 0.4333, 0.2411, 0.0804,
# -0.0202, 0.6204, -0.4104, 0.5382, 0.1804, 0.0179, 0.1118,
# 0.3084, 0.1894, 0.0092, -0.2182, 0.0022, 0.4377, -0.0575,
# 0.0906, -0.0531, 0.0936, -0.1203, -0.0836, -0.1718, 0.2059,
# 0.7192, 0.0833, 0.3712, 0.2354, -0.1141, -0.0993, -0.0433,
# -0.1474, -0.0078, 0.0834, 0.0864, 0.9305, -0.3387, 0.0818,
# 0.3775, 0.1609, -0.0246, 0.2563, -0.1253, -0.0897, 0.0322,
# -0.0848, 0.0673, -0.1051, 0.0205, 0.0183, -0.0007, 0.1229,
# -0.3388, -0.0948, 0.0335, 0.0450, -0.2747, 0.2763, -0.2691,
# 0.6240, -0.0018, 0.2048, 0.3943, 0.2015, -0.5962, -0.0069,
# -0.3460, -0.7910, 0.3002, -0.4653, -0.0611, 0.6912, -0.8154,
# -0.0443, -0.0189, -0.1265, -0.1202, 0.0013, -0.5983, 0.0879,
# 0.0752, 0.8593, 0.0357, 0.0953, -0.0525, 0.2069, -0.6292,
# -0.0456, -0.7646, 0.0166, 0.0584, 0.0142, 0.0575, -0.1658,
# -0.4304, 0.3228, 0.4094, 0.0149, 0.1478, 0.1447, 0.4192,
# -0.0783, -0.0683, 0.0259, 0.0665, 0.6224, 0.3775, 0.0247,
# 0.1710, -0.3622, 0.0931]], # [[-0.2315, -0.2179, 0.1716, -0.6143, 0.4329, 0.2288, 0.1208,
# -0.3435, 0.6314, 0.0106, 0.1470, -0.0915, -0.3019, 0.1302,
# 0.2325, 0.1794, 0.0145, -0.6598, 0.0062, -0.0743, 0.0232,
# 0.9310, -0.8155, -0.0449, -0.5504, 0.5746, 0.3607, 0.4053,
# -0.1887, -0.5448, 0.1522, -0.0189, -0.4852, 0.6322, 0.0011,
# -0.1590, -0.0054, 0.2972, -0.0270, 0.0047, 0.2944, -0.0629,
# -0.1138, 0.1349, 0.5449, 0.8018, -0.0218, 0.0523, 0.3262,
# -0.0506, 0.2821, -0.0661, -0.7165, -0.3653, 0.3321, -0.0255,
# -0.0551, -0.1826, 0.6027, -0.1995, 0.0598, 0.0205, -0.1769,
# -0.0789, 0.4500, -0.1641, 0.5002, -0.0716, -0.3708, -0.0020,
# -0.5195, -0.0896, 0.1421, 0.1149, 0.3407, 0.2649, 0.0858,
# 0.2778, -0.3768, 0.6176, -0.2148, 0.5444, 0.3009, 0.4848,
# -0.1174, 0.0019, 0.6213, 0.2524, -0.0816, 0.4639, 0.4747,
# -0.7812, 0.2435, -0.0867, 0.1617, 0.2194, 0.0426, 0.1393,
# -0.0448, 0.0506, -0.5524, 0.0707, 0.2226, -0.0337, 0.7445,
# -0.4516, 0.1107, -0.2617, -0.1914, 0.7238, -0.2689, 0.0110,
# -0.3139, -0.0027, -0.5964, -0.9012, -0.4319, 0.0112, -0.0306,
# 0.4002, -0.1117, -0.1021, 0.1652, -0.2872, 0.3640, 0.2162,
# -0.3843, -0.0869, -0.1623, 0.0297, -0.0048, -0.0735, -0.0886,
# -0.4138, 0.2325, -0.4248, 0.3354, 0.0712, -0.4079, 0.0821,
# 0.1413, 0.2241, -0.1938, -0.0807, 0.3551, -0.0814, 0.1438,
# -0.6870, -0.3647, 0.0276]], # [[ 0.0258, 0.3281, -0.8145, -0.0476, -0.2886, -0.8013, 0.2135,
# 0.1541, 0.2069, 0.1345, -0.0171, -0.0228, -0.5237, 0.4917,
# 0.5187, 0.1402, 0.0928, -0.0373, 0.2698, 0.1259, 0.0021,
# -0.1624, -0.4100, 0.5377, -0.1013, -0.5658, -0.1015, 0.5609,
# 0.1661, 0.5731, 0.0012, -0.1766, 0.0743, -0.3630, 0.1082,
# 0.4643, 0.0175, -0.0260, 0.3810, -0.6425, -0.5515, 0.8800,
# -0.1158, -0.5741, 0.0463, 0.4033, 0.0803, 0.0403, 0.1159,
# 0.4471, 0.0294, 0.2899, 0.0248, -0.1772, 0.6600, -0.2252,
# -0.4896, -0.1285, -0.2377, -0.4179, -0.4056, -0.3224, -0.6855,
# -0.2703, 0.2971, 0.1259, -0.0456, -0.2495, 0.8141, 0.4453,
# 0.7480, -0.0578, 0.8023, -0.3586, -0.5229, 0.2299, 0.9668,
# -0.0717, 0.5355, -0.0743, 0.5246, 0.1604, -0.1464, -0.0757,
# 0.0414, -0.0861, 0.2245, 0.1247, 0.0676, -0.2053, 0.0113,
# 0.7875, -0.0308, 0.2025, 0.1289, -0.0020, -0.3099, 0.5317,
# -0.0117, 0.0928, -0.4100, -0.6184, 0.1171, 0.0216, -0.1266,
# 0.1640, 0.0821, -0.4097, -0.0691, 0.5805, 0.1692, -0.2021,
# 0.5971, 0.1172, -0.6535, -0.0579, 0.1177, 0.1123, -0.1943,
# 0.0488, -0.1305, -0.4859, -0.2758, -0.2972, -0.0605, -0.0029,
# -0.1508, 0.0375, -0.5942, -0.2139, -0.0335, -0.2320, -0.1152,
# -0.2054, -0.2643, -0.1770, 0.1245, 0.6334, -0.0363, 0.0264,
# -0.3348, -0.0434, -0.3794, -0.0913, 0.1293, -0.6537, 0.6490,
# 0.1305, -0.0631, -0.2243]]], device='cuda:0'),
# tensor([[[ -0.0775, -4.6346, 7.9077, 0.1164, 0.8626, 0.4240,
# -0.9286, -0.1612, -0.6049, 0.6771, 0.7443, 1.7457,
# -0.3930, 0.0112, -13.5393, 0.0317, 0.1236, -0.8475,
# 0.1212, -1.3623, 0.0117, -0.3297, 0.9009, 0.3840,
# -0.5885, 0.7411, -8.9613, -1.1402, 0.4511, -0.0753,
# -0.3107, 1.6518, -0.0870, -3.1360, -12.0200, -0.0464,
# 0.2756, -0.6695, 0.1604, 4.8299, 6.4623, -0.9555,
# -0.6904, 0.4469, -11.3343, -0.1669, -0.2747, 0.1590,
# 0.5829, -0.3345, 2.1731, -0.5636, 0.0207, 0.9874,
# 0.6291, -1.2261, 0.1946, 1.1287, -1.5759, -0.1875,
# 0.5550, -1.7350, -0.8235, 1.5122, 0.2019, 5.5143,
# -3.8153, 0.6771, 0.3011, -0.2994, -0.7320, -1.5857,
# -0.4785, -0.5584, -0.3226, 1.1932, -4.3901, -1.6923,
# 0.3526, -1.0625, 0.9279, -0.1843, -0.4376, 2.2389,
# -0.1558, -0.3959, -1.2987, 0.0279, -0.4938, -0.3364,
# 3.2596, -2.2647, 0.1448, 0.0726, 0.3968, -0.1885,
# -0.3960, 0.2141, 0.6785, -2.1622, -0.0043, -0.7516,
# 0.9367, -0.7092, 4.1853, 1.3348, 0.6993, 0.2043,
# -0.0916, 0.1392, -1.5672, 0.0867, -0.0346, 0.9226,
# -0.0470, -0.6870, -0.3002, -0.1131, 0.7785, 1.0582,
# 0.0914, 2.8785, 0.8164, -0.1048, 0.0573, -0.0499,
# -0.5990, 3.1714, 0.7925, 1.7461, 1.3243, 2.9236,
# 0.8966, -0.4455, 0.8763, -0.3036, 0.3302, 2.6581,
# 0.4608, -0.7280, -2.9457, -0.1973, 0.0585, -0.6555,
# -0.6621, -0.4549, 0.5812, 0.4495, 0.1350, 1.8521]], # [[ -0.0053, -0.0843, -0.1763, 0.3929, -0.1668, -0.6609,
# 0.1269, 0.2214, -0.0208, -0.3571, 0.7532, 0.7496,
# -4.9288, -0.5457, 0.3557, 0.4795, -0.2318, -0.9659,
# 0.6826, -1.6542, 0.4917, -0.3956, -1.5164, -0.2274,
# 0.6779, 1.1201, -3.1397, 0.0434, -0.4993, -0.8809,
# 6.1257, -5.6283, 0.4273, 1.5070, 0.6624, 0.1289,
# 0.2180, 0.9920, 0.1646, 0.8828, 0.5732, 0.3255,
# -0.0679, 0.9843, -1.8408, 1.0547, 0.1959, 0.0748,
# 0.1907, 0.4751, 0.3174, 0.0747, -0.6487, 0.0377,
# 0.5554, -0.4095, 0.2593, -0.0568, 0.3751, -0.3646,
# -0.2031, -0.3284, 0.4058, 1.2788, 0.1348, 1.8184,
# 0.8482, -0.7494, -0.2395, -0.4352, -0.1584, -0.0105,
# 0.2676, 0.3763, 2.1413, -1.3001, 0.3923, 1.6432,
# 0.2987, -1.2708, 5.5667, -0.1727, -0.5106, 0.5180,
# -6.7258, 0.5001, -0.3052, 0.0843, 0.0474, -0.2306,
# 0.1908, -2.2523, -1.5432, 0.2809, 0.7099, -0.4145,
# 0.7393, -0.6529, 0.7825, -0.0019, 1.3337, 2.2042,
# 9.2887, -0.8515, -0.0610, -0.6146, -1.5616, 0.3592,
# -1.3585, -0.2641, 1.4763, -3.2525, -0.5447, -0.0453,
# -1.0416, -2.4657, 0.0556, -0.7654, 0.2062, 0.0855,
# 3.0740, 0.0952, 0.4923, -0.1772, 0.9173, -4.2004,
# -0.1298, -2.4266, 0.0181, 0.5039, 0.0399, 7.9909,
# -0.5778, -2.9112, 0.4854, 1.2364, 0.0686, 0.6365,
# 0.1869, 0.6050, -0.1246, -0.1848, 0.5406, 0.2110,
# 1.2367, 1.9466, 0.0302, 0.2002, -0.5902, 0.1069]], # [[ -0.3559, -0.2859, 0.2699, -1.4359, 0.9814, 0.2811,
# 0.8539, -2.6654, 2.5455, 0.0434, 0.5947, -0.5325,
# -0.4638, 0.5487, 1.4376, 0.9863, 0.7429, -1.8308,
# 0.0402, -0.2282, 0.0366, 12.7877, -1.2491, -0.1437,
# -1.4960, 0.7364, 0.8599, 1.8343, -3.5117, -1.2758,
# 0.2930, -0.0472, -0.7527, 0.9555, 0.0446, -0.3389,
# -0.1985, 1.7953, -0.5702, 0.0141, 0.6166, -0.0924,
# -0.5182, 0.5146, 2.0801, 2.7460, -0.2606, 0.2090,
# 0.9266, -0.4758, 0.9961, -0.1723, -1.2069, -1.1735,
# 0.3683, -0.4933, -0.0604, -0.2354, 0.8239, -5.4226,
# 0.0854, 0.1185, -0.2656, -0.2689, 0.6047, -0.6246,
# 1.0131, -0.1673, -0.4990, -0.0690, -0.6092, -0.5205,
# 0.1808, 0.3061, 0.3924, 0.5868, 0.1452, 2.8930,
# -0.6085, 1.6086, -0.4763, 5.0389, 1.1569, 3.4060,
# -0.7565, 0.0247, 0.8477, 0.3714, -0.1043, 1.5607,
# 4.0700, -1.8363, 0.4370, -0.3571, 0.7268, 0.3435,
# 0.0972, 7.1477, -0.1486, 0.3342, -0.9733, 0.2311,
# 0.6104, -0.4988, 2.8838, -1.3387, 1.4291, -0.4121,
# -0.6722, 2.6834, -0.5188, 0.0428, -0.3452, -0.0131,
# -0.9004, -3.0346, -0.9254, 0.0150, -0.0386, 1.0639,
# -0.2444, -0.4562, 4.1626, -1.9304, 1.0662, 2.0683,
# -0.9553, -0.6434, -1.6777, 0.0702, -0.0113, -0.5503,
# -0.1873, -0.6916, 1.0729, -0.8234, 0.6421, 0.3022,
# -0.6065, 0.1016, 0.7792, 0.2533, -0.2670, -0.1314,
# 0.7515, -0.9859, 0.5050, -1.0552, -0.5632, 1.0697]], # [[ 0.3612, 2.4727, -4.6103, -1.6459, -1.7761, -1.4302,
# 0.2737, 0.3302, 4.0617, 0.7206, -0.0749, -0.8146,
# -1.0134, 0.8741, 1.9300, 0.5426, 0.1386, -1.3920,
# 0.4602, 1.3387, 0.0068, -0.3648, -7.6665, 0.9011,
# -0.3286, -1.5220, -0.2155, 0.7959, 4.0746, 0.9382,
# 0.0023, -0.2666, 0.4571, -1.9530, 0.3216, 2.1178,
# 0.4043, -0.0309, 2.5116, -1.2250, -0.9842, 5.0822,
# -4.1296, -5.3579, 0.9115, 0.4843, 0.1365, 0.0491,
# 0.1446, 0.6523, 0.0765, 0.3761, 0.0310, -0.4825,
# 7.1485, -0.4211, -3.7914, -0.2492, -0.3775, -0.4745,
# -1.3320, -1.8203, -1.0266, -0.4446, 2.2385, 0.6003,
# -0.1759, -1.9601, 2.3865, 1.3325, 4.8762, -0.2398,
# 7.5251, -0.4380, -2.3422, 0.6013, 13.8362, -0.4112,
# 2.3579, -0.1720, 1.0265, 0.6521, -0.7363, -0.7864,
# 0.2986, -0.1298, 0.5078, 0.1386, 1.4856, -0.3133,
# 0.9933, 1.5144, -0.0433, 1.0841, 0.3962, -0.0024,
# -0.3937, 2.2719, -0.0198, 1.6771, -1.2469, -0.8017,
# 0.1607, 0.0244, -0.1429, 0.9912, 0.1635, -1.2396,
# -0.1615, 1.0921, 0.8146, -0.3309, 0.8553, 0.4243,
# -3.5547, -0.1382, 0.1513, 0.4036, -0.2505, 0.2295,
# -0.6219, -0.7644, -0.7568, -0.4494, -0.0775, -0.0178,
# -0.2550, 0.2258, -2.7895, -0.3362, -0.2364, -0.9864,
# -1.3459, -1.8118, -0.4397, -0.8312, 0.3526, 1.3541,
# -0.0467, 1.6161, -0.4478, -0.5202, -0.4164, -0.8265,
# 0.1626, -4.2044, 3.2649, 0.2940, -0.8260, -0.4956]]], device='cuda:0'))
######## Padding functions used in pytorch. #########
1. torch.nn.utils.rnn.PackedSequence(*args)
Holds the data and list of batch_sizes of a packed sequence.
All RNN modules accept packed sequences as inputs.
2. torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False)
Packs a Tensor containing padded sequences of variable length.
Input can be of size T*B** where T is the length of the longest sequence, B is the batch size, and the * is any number of dimensions.
If batch_first is True B*T** inputs are expected. The sequences should be sorted by length in a decreasing order.
3. torch.nn.utils.rnn.pad_packed_sequence(sequence, batch_first=False, padding_values=0.0, total_length=None)
Pads a packed batch of variable length sequences.
It is an inverse operation to pack_padded_sequence().
Batch elements will be ordered decreasingly by their length.
Note: total_length is useful to implement the pack sequence -> rnn -> unpack sequence .
Return: Tuple of tensor containing the padded sequence, and a tensor containing the list of lengths of each sequence in the batch.
4. torch.nn.utils.rnn.pad_sequence(sequence, batch_first=False, padding_value=0)
Pad a list of variable length Tensors with zero.
5. torch.nn.utils.rnn.pack_sequence(sequences)
Packs a list of variable length Tensors.
6. Tutorials on these functions.
(1). https://zhuanlan.zhihu.com/p/34418001
(2). https://zhuanlan.zhihu.com/p/28472545
总结起来就是：在利用 recurrent neural network 处理变长的句子序列时，我们可以配套的使用：
torch.nn.utils.rnn.pack_padded_sequence () 来对一个 mini-batch 中的句子进行 padding；
torch.nn.utils.rnn.pad_packed_sequence () 来避免 padding 对句子表示的影响。
===
Tutorial on word2vector using GloVe and Word2Vec的更多相关文章
- 词向量(one-hot/SVD/NNLM/Word2Vec/GloVe)
目录 词向量简介 1. 基于one-hot编码的词向量方法 2. 统计语言模型 3. 从分布式表征到SVD分解 3.1 分布式表征(Distribution) 3.2 奇异值分解(SVD) 3.3 基 ...
- word2vec 和 glove 模型的区别
2019-09-09 15:36:13 问题描述:word2vec 和 glove 这两个生成 word embedding 的算法有什么区别. 问题求解: GloVe (global vectors ...
- [白话解析] 带你一起梳理Word2vec相关概念
[白话解析] 带你一起梳理Word2vec相关概念 0x00 摘要 本文将尽量使用易懂的方式,尽可能不涉及数学公式,而是从整体的思路上来说,运用感性直觉的思考来帮大家梳理Word2vec相关概念. 0 ...
- NLP︱高级词向量表达（一）——GloVe（理论、相关测评结果、R&python实现、相关应用）
有很多改进版的word2vec,但是目前还是word2vec最流行,但是Glove也有很多在提及,笔者在自己实验的时候,发现Glove也还是有很多优点以及可以深入研究对比的地方的,所以对其进行了一定的 ...
- Getting Started with Word2Vec
Getting Started with Word2Vec 1. Source by Google Project with Code: https://code.google.com/archive ...
- 四步理解GloVe！(附代码实现)
1. 说说GloVe 正如GloVe论文的标题而言,GloVe的全称叫Global Vectors for Word Representation,它是一个基于全局词频统计(count-based & ...
- 词嵌入之GloVe
什么是GloVe GloVe(Global Vectors for Word Representation)是一个基于全局词频统计(count-based & overall statisti ...
- RNN 与 LSTM 的应用
之前已经介绍过关于 Recurrent Neural Nnetwork 与 Long Short-Trem Memory 的网络结构与参数求解算法( 递归神经网络(Recurrent Neural N ...
- 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史（转载）
转载 https://zhuanlan.zhihu.com/p/49271699 首发于深度学习前沿笔记 写文章 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史 张 ...
随机推荐
- MVC中的Ajax与增删改查(二)
上一篇记录的是前台操作,下面写一下后台 ,本来自认为是没有必要做补充,毕竟思路啥的都有,实际上在做删除操作的时候,折腾了一天,还是自己太嫩,逻辑不够严谨,这里作下记录. 关于表结构这里再作下说明: ① ...
- react修改app.js添加中文内容后中文部分乱码解决
[问题]:配置完react后修改app.js内容时添加中文出现如下乱码的中文. [A解决]文档——文本编码——转换文本编码,在弹出窗口修改,确定,搞定 [B解决]首先在EditPlus内:工具——首选 ...
- python 内置函数 sorted()
sorted() 函数对所有可迭代的对象进行排序操作. sort 与 sorted 区别: sort 是应用在 list 上的方法,sorted 可以对所有可迭代的对象进行排序操作. list 的 s ...
- linux下怎么删除名称带空格的文件
linux下怎么删除名称带空格的文件-rm 'mysql bin.000005' 用引号把文件名括起来 某些情况下会出现名称带空格的文件, 如果想要删除的话,直接用rm mysql bin.00000 ...
- Linux服务器---流量监控ntop
Ntop Ntop 是一款类似于sniffer的流量监控工具,它显示出的流量信息比mrtg更加详细. 1 .安装一些依赖软件 [root@localhost bandwidthd]# yum ins ...
- 转：专题三线程池中的I/O线程
上一篇文章主要介绍了如何利用线程池中的工作者线程来实现多线程,使多个线程可以并发地工作,从而高效率地使用系统资源.在这篇文章中将介绍如何用线程池中的I/O线程来执行I/O操作,希望对大家有所帮助. 目 ...
- 洛谷 P2799 国王的魔镜
把项链当做字符串输进去,可以用gets #include<iostream>#include<cstdio>#include<cmath>#include<c ...
- 重复打印相同内容(Doc档)的时候自动生成打印编号
昨天突然接到一个好久未联系的朋友电话,说是江湖救急,要打印一份单据,单据上有个号码要自动生成,如下图,最土的办法是打印完一张,手工改下号码,但这种方法估计碰到成百上千张时估计会疯掉 网上找了实现方法, ...
- Java笔记 #03# HtmlUnit爬虫
存档留用 (= 存档留着备用) 爬的是一个开放的自动回复机器人 API 网站 http://i.itpk.cn/. 结构 大致如下: 我做的事情就是[输入文字,点击按钮,爬取内容],如上图所示. pa ...
- K8S学习笔记之二进制的方式创建一个Kubernetes集群
0x00 单节点搭建和简述 minikube Minikube是一个工具,可以在本地快速运行一个单点的Kubernetes,尝试Kubernetes或日常开发的用户使用.不能用于生产环境. 官方地址: ...