Tutorial on word2vector using GloVe and Word2Vec
Tutorial on word2vector using GloVe and Word2Vec
2018-05-04 10:02:53
Some Important Reference Pages First:
Reference Page: https://github.com/IliaGavrilov/NeuralMachineTranslationBidirectionalLSTM/blob/master/1_Bidirectional_LSTM_Eng_to_French.ipynb
Glove Project Page: https://nlp.stanford.edu/projects/glove/
Word2Vec Project Page: https://code.google.com/archive/p/word2vec/
Pre-trained word2vec model: https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing
Gensim Tutorial: https://radimrehurek.com/gensim/models/word2vec.html
===================================
===== For the Glove
===================================
1. Download one of the pre-trained model from Glove project page and Unzip the files.
Wikipedia 2014 + Gigaword 5 (6B tokens, 400K vocab, uncased, 50d, 100d, 200d, & 300d vectors, 822 MB download): glove.6B.zip
Common Crawl (42B tokens, 1.9M vocab, uncased, 300d vectors, 1.75 GB download): glove.42B.300d.zip
Common Crawl (840B tokens, 2.2M vocab, cased, 300d vectors, 2.03 GB download): glove.840B.300d.zip
2. Install the needed packages:
pickle, numpy, re, pickle, collections, bcolz
3. Run the following demo to test the results (extract the feature of given single word).
Code:
import pickle
import numpy as np
import re, pickle, collections, bcolz # with open('./glove.840B.300d.txt', 'r', encoding="utf8") as f: with open('./glove.6B.200d.txt', 'r') as f:
lines = [line.split() for line in f] print('==>> begin to load Glove pre-trained models.')
glove_words = [elem[0] for elem in lines]
glove_words_idx = {elem:idx for idx, elem in enumerate(glove_words)} # is elem: idx equal to glove_words_idx[elem]=idx?
glove_vecs = np.stack(np.array(elem[1:], dtype=np.float32) for elem in lines) print('==>> save into .pkl files.')
pickle.dump(glove_words, open('./glove.6B.200d.txt'+'_glove_words.pkl', 'wb'))
pickle.dump(glove_words_idx, open('./glove.6B.200d.txt'+'_glove_words_idx.pkl', 'wb')) ## saving array using specific function.
def save_array(fname, arr):
c=bcolz.carray(arr, rootdir=fname, mode='w')
c.flush() save_array('./glove.6B.200d.txt'+'_glove_vecs'+'.dat', glove_vecs) def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove('./glove.6B.200d.txt')
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape print('==>> shown one demo: "King"')
demo_vector = en_w2v['king']
print(demo_vector)
print("==>> Done !")
Results:
wangxiao@AHU$ python tutorial_Glove_word2vec.py
==>> begin to load Glove pre-trained models.
==>> save into .pkl files.
==>> Loading the glove.6B.200d.txt files.
==>> shown one demo: "King"
[-0.49346 -0.14768 0.32166001 0.056899 0.052572 0.20192 -0.13506 -0.030793 0.15614 -0.23004 -0.66376001 -0.27316001 0.10391 0.57334 -0.032355 -0.32765999 -0.27160001 0.32918999
0.41305 -0.18085 1.51670003 2.16490006 -0.10278 0.098019 -0.018946 0.027292 -0.79479998 0.36631 -0.33151001 0.28839999 0.10436 -0.19166 0.27326 -0.17519 -0.14985999 -0.072333
-0.54370999 -0.29728001 0.081491 -0.42673001 -0.36406001 -0.52034998 0.18455 0.44121 -0.32196 0.39172 0.11952 0.36978999 0.29229 -0.42954001 0.46653 -0.067243 0.31215999 -0.17216
0.48874 0.28029999 -0.17577 -0.35100999 0.020792 0.15974 0.21927001 -0.32499 0.086022 0.38927001 -0.65638 -0.67400998 -0.41896001 1.27090001 0.20857 0.28314999 0.58238 -0.14944001
0.3989 0.52680999 0.35714 -0.39100999 -0.55372 -0.56642002 -0.15762 -0.48004001 0.40448001 0.057518 -1.01569998 0.21754999 0.073296 0.15237001 -0.38361999 -0.75308001 -0.0060254 -0.26232001
-0.54101998 -0.34347001 0.11113 0.47685 -0.73229998 0.77596998 0.015216 -0.66327 -0.21144 -0.42964 -0.72689998 -0.067968 0.50601 0.039817 -0.27584001 -0.34794 -0.0474 0.50734001
-0.30777001 0.11594 -0.19211 0.3107 -0.60075003 0.22044 -0.36265001 -0.59442002 -1.20459998 0.10619 -0.60277998 0.21573 -0.35361999 0.55473 0.58094001 0.077259 1.0776 -0.1867
-1.51680005 0.32418001 0.83332998 0.17366 1.12320006 0.10863 0.55888999 0.30799001 0.084318 -0.43178001 -0.042287 -0.054615 0.054712 -0.80914003 -0.24429999 -0.076909 0.55216002 -0.71895999
0.83319002 0.020735 0.020472 -0.40279001 -0.28874001 0.23758 0.12576 -0.15165 -0.69419998 -0.25174001 0.29591 0.40290001 -1.0618 0.19847 -0.63463002 -0.70842999 0.067943 0.57366002
0.041122 0.17452 0.19430999 -0.28641 -1.13629997 0.45116001 -0.066518 0.82615 -0.45451999 -0.85652 0.18105 -0.24187 0.20152999 0.72298002 0.17415 -0.87327999 0.69814998 0.024706
0.26174 -0.0087155 -0.39348999 0.13801 -0.39298999 -0.23057 -0.22611 -0.14407 0.010511 -0.47389001 -0.15645 0.28601 -0.21772 -0.49535 0.022209 -0.23575 -0.22469001 -0.011578 0.52867001 -0.062309 ]
==>> Done !
4. Extract the feature of the whole sentense.
Given one sentense, such as "I Love Natural Language Processing", we can translate this sentense into a matrix representation. Specifically, we represent each word of the sentense with a vector which lies in a continuous space (as shown above). You can also see this blog: https://blog.csdn.net/fangqingan_java/article/details/50493948 to futher understand this process.
但是有如下的疑问：
1. 给定的句子，长短不一，每一个单词的维度固定，但是总的句子的长度不固定，怎么办？
只好用 padding 的方法了，那么，怎么进行 padding 呢？
For CNN architecture usually the input is (for each sentence) a vector of embedded words [GloVe(w0), GloVe(w1), ..., GloVe(wN)] of fixed length N.
So commonly you preset the maximum sentence length and just pad the tail of the input vector with zero vectors (just as you did). Marking the beginning and the end of the sentence is more relevant for the RNN, where the sentence length is expected to be variable and processing is done sequentially. Having said that, you can add a new dimension to the GloVe vector, setting it to zero for normal words, and arbitrarily to (say) 0.5 for BEGIN, and 1 for END and a random negative value for OOV.
As for unknown words, you should ;) encounter them very rear, otherwise you might consider training the embeddings by yourself. 那么，是什么意思呢？
对于 CNN 的网络来说，由于需要处理 网格状的 data，所以，需要将句子进行填充，以得到完整的 matrix，然后进行处理；
对于 RNN/LSTM 的网络来说，是可以按照时刻，顺序处理的，所以呢？就不需要 padding 了？
2. Here is another tutorial on Word2Vec using deep learning tools --- Keras.
from numpy import array
from keras.preprocessing.text import one_hot
from keras.preprocessing.sequence import pad_sequences
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Flatten
from keras.layers.embeddings import Embedding
# define documents
docs = ['Well done!',
'Good work',
'Great effort',
'nice work',
'Excellent!',
'Weak',
'Poor effort!',
'not good',
'poor work',
'Could have done better.']
# define class labels
labels = array([,,,,,,,,,])
# integer encode the documents
vocab_size =
encoded_docs = [one_hot(d, vocab_size) for d in docs]
print("==>> encoded_docs: ")
print(encoded_docs)
# pad documents to a max length of words
max_length =
padded_docs = pad_sequences(encoded_docs, maxlen=max_length, padding='post')
print("==>> padded_docs: ")
print(padded_docs)
# define the model
model = Sequential()
model.add(Embedding(vocab_size, , input_length=max_length))
model.add(Flatten())
model.add(Dense(, activation='sigmoid'))
# compile the model
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['acc'])
# summarize the model
print("==>> model.summary()")
print(model.summary())
# fit the model
model.fit(padded_docs, labels, nb_epoch=, verbose=)
# evaluate the model
loss, accuracy = model.evaluate(padded_docs, labels, verbose=)
print("==>> the final Accuracy: ")
print('Accuracy: %f' % (accuracy*))
The output is:
Using TensorFlow backend.
==>> encoded_docs:
[[, ], [, ], [, ], [, ], [], [], [, ], [, ], [, ], [, , , ]]
==>> padded_docs:
[[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]]
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_prod (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python2./dist-packages/keras/backend/tensorflow_backend.py:: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
==>> model.summary()
____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
embedding_1 (Embedding) (None, , ) embedding_input_1[][]
____________________________________________________________________________________________________
flatten_1 (Flatten) (None, ) embedding_1[][]
____________________________________________________________________________________________________
dense_1 (Dense) (None, ) flatten_1[][]
====================================================================================================
Total params:
Trainable params:
Non-trainable params:
____________________________________________________________________________________________________
None
-- ::59.053198: I tensorflow/core/platform/cpu_feature_guard.cc:] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
==>> the final Accuracy:
Accuracy: 89.999998
3. Maybe translate the sentence into vectors using skip-thought vectors is another goog choice. you can also check this tutorial from: http://www.cnblogs.com/wangxiaocvpr/p/7277025.html
4. But we still want to use GloVe to extract the vector of each word and concatenate them into a long vector.
import os
import numpy as np
from collections import OrderedDict
import re, pickle, collections, bcolz
import pdb seq_home = '/dataset/language-dataset/'
seqlist_path = '/dataset/language-train-video-list.txt'
output_path = 'data/language-train-video-list.pkl' def load_glove(loc):
return (bcolz.open(loc+'_glove_vecs.dat')[:],
pickle.load(open(loc+'_glove_words.pkl', 'rb')),
pickle.load(open(loc+'_glove_words_idx.pkl', 'rb'))) pre_trained_model_path = '/glove/glove.6B.200d.txt' ###############################################
print('==>> Loading the glove.6B.200d.txt files.')
en_vecs, en_wv_word, en_wv_idx = load_glove(pre_trained_model_path)
en_w2v = {w: en_vecs[en_wv_idx[w]] for w in en_wv_word}
n_en_vec, dim_en_vec = en_vecs.shape with open(seqlist_path,'r') as fp:
seq_list = fp.read().splitlines() data = {}
for i,seq in enumerate(seq_list):
img_list = sorted([p for p in os.listdir(seq_home+seq+'/imgs/') if os.path.splitext(p)[1] == '.jpg']) ## image list
gt = np.loadtxt(seq_home+seq+'/groundtruth_rect.txt', delimiter=',') ## gt files
language_txt = open(seq_home+seq+'/language.txt', "rw+") ## natural language description files line = language_txt.readline()
print("==>> language: %s" % (line)) gloveVector = []
test_txtName = seq_home+seq+'/glove_vector.txt'
f=file(test_txtName, "w") word_list = line.split(' ')
for word_idx in range(len(word_list)):
current_word = word_list[word_idx]
current_GloVe_vector = en_w2v[current_word] ## vector dimension is: 200-D gloveVector = np.concatenate((gloveVector, current_GloVe_vector), axis=0)
f.write(str(current_GloVe_vector))
f.write('\n')
f.close() print(i) assert len(img_list) == len(gt), "Lengths do not match!!" if gt.shape[1]==8:
x_min = np.min(gt[:,[0,2,4,6]],axis=1)[:,None]
y_min = np.min(gt[:,[1,3,5,7]],axis=1)[:,None]
x_max = np.max(gt[:,[0,2,4,6]],axis=1)[:,None]
y_max = np.max(gt[:,[1,3,5,7]],axis=1)[:,None]
gt = np.concatenate((x_min, y_min, x_max-x_min, y_max-y_min),axis=1) data[seq] = {'images':img_list, 'gt':gt, 'gloveVector': gloveVector} with open(output_path, 'wb') as fp:
pickle.dump(data, fp, -1)
The Glove vector can be saved into a txt file for each video, as shown in following screenshots.
[ 0.19495 0.60610002 -0.077356 0.017301 -0.51370001 0.22421999
-0.80773002 0.022378 0.30256 1.06669998 -0.10918 0.57902998
0.23986 0.1691 0.0072384 0.42197999 -0.20458999 0.60271001
0.19188 -0.19616 0.33070001 3.20020008 -0.18104 0.20784
0.49511001 -0.42258999 0.022125 0.24379 0.16714001 -0.20909999
-0.12489 -0.51766998 -0.13569 -0.25979999 -0.17961 -0.47663
-0.89539999 -0.27138999 0.17746 0.45827001 0.21363001 0.22342999
-0.049342 0.34286001 -0.32315001 0.27469999 0.95993 -0.25979
0.21250001 -0.21373001 0.19809 0.15455 -0.48581001 0.38925001
0.33747 -0.27897999 0.19371 -0.45872 -0.054217 -0.24022999
0.59153003 0.12453 -0.21302 0.058223 -0.046671 -0.011614
-0.32025999 0.64120001 -0.28718999 0.035138 0.39287001 -0.030683
0.083529 -0.010964 -0.62427002 -0.13575 -0.38468999 0.11454
-0.61037999 0.12594999 -0.17633 -0.21415 -0.37013999 0.21763
0.055366 -0.25242001 -0.45475999 -0.28105 0.18911 -1.58539999
0.64841002 -0.34621999 0.59254003 -0.39034 -0.44258001 0.20562001
0.44395 0.23019999 -0.35018 -0.36090001 0.62993002 -0.34698999
-0.31964999 -0.17361 0.51714998 0.68493003 -0.15587001 1.42550004
-0.94313997 0.031951 -0.26210001 -0.089544 0.22437 -0.050374
0.035414 -0.070493 0.17037 -0.38598001 0.0095626 0.26363
0.72125 -0.13797 0.70602 -0.50839001 -0.49722001 -0.48706001
0.16254 0.025619 0.33572 -0.64160001 -0.32541999 0.21896
0.05331 0.082011 0.12038 0.088849 -0.04651 -0.085435
0.036835 -0.14695001 -0.25841001 -0.043812 0.053209 -0.48954999
1.73950005 0.99014997 0.09395 -0.20236 -0.050135 0.18337999
0.22713999 0.83146 -0.30974001 0.51995999 0.068264 -0.28237
-0.30096999 -0.031014 0.024615 0.4294 -0.085036 0.051913
0.31251001 -0.34443 -0.085145 0.024975 0.0082017 0.17241
-0.66000998 0.0058962 -0.055387 -0.22315 0.35721999 -0.18962
0.25819999 -0.24685 -0.79571998 -0.09436 -0.10271 0.13713001
1.48660004 -0.57113999 -0.52649999 -0.25181001 -0.40792 -0.18612
0.040009 0.11557 0.017987 0.27149001 -0.14252 -0.087756
0.15196 0.064926 0.01651 -0.25334001 0.27437001 0.24246
0.018494 0.22473 ]
[ 4.32489991e-01 5.34709990e-01 -1.83240008e-02 1.56369999e-01
6.69689998e-02 7.63140023e-02 -7.16499984e-01 2.41520002e-01
1.70919999e-01 8.96220028e-01 -2.00739995e-01 1.88370004e-01
4.53570008e-01 -1.86419994e-01 -3.41060013e-01 -3.10570002e-02
-4.90720011e-02 6.55830026e-01 -7.43409991e-03 1.54770002e-01
1.55849993e-01 3.11129999e+00 -4.12149996e-01 1.89439997e-01
1.86639994e-01 -2.00540006e-01 6.63070008e-02 3.90180014e-02
6.06740005e-02 -3.17880005e-01 1.82290003e-02 -6.80719972e-01
-1.33360000e-02 2.32099995e-01 -8.23459998e-02 -2.68420011e-01
-3.51870000e-01 -2.03860000e-01 6.69749975e-02 3.02689999e-01
1.84330001e-01 1.44429997e-01 1.38980001e-01 1.93629991e-02
-4.09139991e-01 2.24950001e-01 8.02020013e-01 -3.05290014e-01
2.22399995e-01 -6.10979982e-02 3.48910004e-01 1.13349997e-01
-4.14329991e-02 7.31640011e-02 3.17719996e-01 -2.59590000e-01
3.17759991e-01 -1.78080007e-01 2.12380007e-01 1.07529998e-01
4.37519997e-01 -1.81329995e-01 -1.53909996e-01 7.27130026e-02
4.14730012e-01 2.73079991e-01 -1.06940001e-01 7.45599985e-01
3.02549988e-01 1.49999997e-02 6.33790016e-01 1.03399999e-01
2.18640000e-01 -7.43470015e-03 -7.09949970e-01 4.60739993e-02
-8.34990025e-01 1.84519999e-02 -2.75200009e-01 -2.48000007e-02
-1.21320002e-01 3.29720005e-02 1.07320003e-01 2.18070000e-01
2.79079992e-02 -2.99050003e-01 -3.06030005e-01 -3.17970008e-01
2.10580006e-01 -1.16180003e+00 4.57720011e-01 -2.60210007e-01
3.76190007e-01 -2.16749996e-01 -1.68540001e-01 1.77729994e-01
-1.46709993e-01 2.67820001e-01 -5.86109996e-01 -4.98769999e-01
3.83679986e-01 2.51020014e-01 -3.90020013e-01 -6.41409993e-01
5.34929991e-01 2.76479989e-01 -4.32830006e-01 1.61350000e+00
-1.58399999e-01 -2.03970000e-01 -2.18089998e-01 -9.74659983e-04
-7.48440027e-02 3.78959998e-02 -4.03719991e-01 -1.74640000e-01
4.11469996e-01 -2.80400008e-01 -3.42570007e-01 5.72329983e-02
7.18890011e-01 -1.30419999e-01 5.75810015e-01 -3.27690005e-01
7.75609985e-02 -2.66130000e-01 4.82820012e-02 1.24679998e-01
1.29319996e-01 7.65379965e-02 -3.85809988e-01 3.72209996e-01
-1.67280003e-01 1.86330006e-01 -2.20280007e-01 3.27170014e-01
1.74089998e-01 -2.51549989e-01 -5.47109991e-02 -2.48549998e-01
-2.24260002e-01 -3.80129993e-01 5.53080022e-01 -1.52050003e-01
1.12820005e+00 3.70869994e-01 -4.62579988e-02 -3.56009990e-01
-1.72120005e-01 1.23460002e-01 6.58179998e-01 5.05930007e-01
-4.99610007e-01 1.97339997e-01 1.37759998e-01 3.98500008e-04
-1.38300002e-01 -6.67430013e-02 -7.20809996e-02 4.00720000e-01
-3.77620012e-01 -9.86199975e-02 2.04119995e-01 -3.42869997e-01
-2.01509997e-01 -1.18270002e-01 -1.60109997e-01 1.63570002e-01
-3.79029989e-01 -2.45529994e-01 -5.31699993e-02 -4.19999994e-02
1.82099998e-01 2.23959997e-01 4.50800002e-01 -2.03030005e-01
-6.16330028e-01 -2.02739999e-01 2.18419999e-01 3.66939992e-01
7.68440008e-01 -2.92189986e-01 -9.80089977e-02 -2.92939991e-01
-1.93189994e-01 1.45720005e-01 2.45150000e-01 -7.11840019e-02
3.97929996e-01 -3.33019998e-03 -4.01179999e-01 1.39760002e-01
8.27540010e-02 -1.29600003e-01 -3.05500001e-01 8.98869988e-03
2.26559997e-01 3.21410000e-01 -4.29780006e-01 4.74779993e-01]
[ 5.73459983e-01 5.41700006e-01 -2.34770000e-01 -3.62399995e-01
4.03699994e-01 1.13860004e-01 -4.49330002e-01 -3.09909999e-01
-5.34110004e-03 5.84259987e-01 -2.59559993e-02 4.93930012e-01
-3.72090004e-02 -2.84280002e-01 9.76959988e-02 -4.89069998e-01
2.60269996e-02 3.76489997e-01 5.77879995e-02 -4.68070000e-01
8.12880024e-02 3.28250003e+00 -6.36900008e-01 3.79559994e-01
3.81670007e-03 9.36070010e-02 -1.28549993e-01 1.73800007e-01
1.05219997e-01 2.86480010e-01 2.10889995e-01 -4.70759988e-01
2.77330000e-02 -1.98029995e-01 7.63280019e-02 -8.46289992e-01
-7.97079980e-01 -3.87430012e-01 -3.04220002e-02 -2.68489987e-01
4.85850006e-01 1.28950000e-01 3.83540004e-01 3.87219995e-01
-3.85239989e-01 1.90750003e-01 4.89980012e-01 1.32780001e-01
1.07920002e-02 2.67699987e-01 1.78120002e-01 -1.14330001e-01
-3.34939986e-01 8.73059988e-01 7.58750021e-01 -3.03779989e-01
-1.56259999e-01 1.20850001e-03 2.33219996e-01 2.79529989e-01
-1.84939995e-01 -1.41460001e-01 -1.89689994e-01 -3.83859985e-02
3.58740002e-01 6.55129999e-02 6.05649985e-02 6.63389981e-01
-8.32519978e-02 6.51630014e-02 5.17610013e-01 1.61709994e-01
4.60110009e-01 1.63880005e-01 -1.23989999e-01 3.11219990e-01
-1.54119998e-01 -1.09169997e-01 -4.25509989e-01 1.14179999e-01
2.51370013e-01 -5.61579987e-02 -2.59270012e-01 2.81630009e-01
-1.80939995e-02 1.60650000e-01 -4.85060006e-01 -9.89030004e-01
2.50220001e-01 -1.67359993e-01 4.14739996e-01 1.77010000e-01
4.24070001e-01 1.10880002e-01 -1.83599994e-01 -1.24100000e-01
-3.47799987e-01 9.90779996e-02 -2.23810002e-01 -1.12450004e-01
-2.11559996e-01 3.07060010e-03 -2.36070007e-01 2.72610001e-02
3.64300013e-01 3.99219990e-02 -1.83689997e-01 1.22660005e+00
-7.76400030e-01 -6.62249982e-01 1.57239996e-02 -1.49690002e-01
8.46489966e-02 2.68139988e-01 -1.67649999e-01 -3.19420010e-01
2.84940004e-01 -7.00000003e-02 1.20099997e-02 -1.22189999e-01
5.63099980e-01 -3.19999993e-01 5.01089990e-01 -1.02090001e-01
4.65750009e-01 -7.15420008e-01 1.72930002e-01 5.82589984e-01
7.83839971e-02 -3.38440016e-02 -2.51289994e-01 3.65029991e-01
3.15780006e-02 -6.57779992e-01 5.47499992e-02 8.71890008e-01
1.24550000e-01 -4.58770007e-01 -2.69650012e-01 -4.67790008e-01
-2.85780011e-03 1.78100005e-01 6.39689982e-01 1.39950007e-01
9.75960016e-01 1.18359998e-01 -6.39039993e-01 -1.54159993e-01
6.52619973e-02 2.43290007e-01 6.64759994e-01 2.50690013e-01
-1.02519996e-01 -3.28390002e-01 -8.55590031e-02 -1.27739999e-02
-1.94309995e-01 5.61389983e-01 -3.57329994e-01 -2.03439996e-01
-1.24130003e-01 -3.44309986e-01 -2.32960001e-01 -2.11870000e-01
8.53869990e-02 7.00630024e-02 -1.98029995e-01 -2.60230005e-02
-3.90370011e-01 8.00019979e-01 4.05770004e-01 -7.98629969e-02
3.52629989e-01 -3.40429991e-01 3.96759987e-01 2.28619993e-01
-3.50279987e-01 -4.73439991e-01 5.97419977e-01 -1.16570003e-01
1.05519998e+00 -4.15699989e-01 -8.05519968e-02 -5.65709993e-02
-1.66219994e-01 1.92739993e-01 -9.51749980e-02 -2.07810000e-01
1.56200007e-01 5.02309985e-02 -2.79150009e-01 4.37420011e-01
-3.12370002e-01 1.31940007e-01 -3.32780004e-01 1.88769996e-01
-2.34219998e-01 5.44179976e-01 -2.30690002e-01 3.49469990e-01]
[ -1.43299997e-01 -6.54269993e-01 -4.71419990e-01 -2.22780004e-01
-1.74260005e-01 6.43490016e-01 -7.60049999e-01 -8.42769966e-02
2.19850004e-01 -4.24650013e-01 -2.56919991e-02 -3.27909999e-02
-3.98149997e-01 -1.26000000e-02 6.31869972e-01 -6.31640017e-01
-2.70480007e-01 2.09920004e-01 5.72669983e-01 8.88589993e-02
2.24289998e-01 1.78419995e+00 8.73669982e-01 -2.23949999e-01
4.91869986e-01 6.86379969e-02 6.80689991e-01 1.85680002e-01
-2.85780013e-01 -1.06030002e-01 -3.07460010e-01 1.06180003e-02
-1.62290007e-01 -3.62700000e-02 8.02920014e-02 -2.17669994e-01
-7.92110026e-01 -7.06340015e-01 -2.89400011e-01 -1.51869997e-01
3.09009999e-01 -1.41849995e-01 -4.18920010e-01 -1.66299999e-01
-1.68510005e-01 1.04549997e-01 -3.49110007e-01 -1.21399999e+00
3.21929991e-01 1.88219994e-01 -3.50019991e-01 -2.23629996e-01
3.85280013e-01 1.40709996e-01 1.34890005e-01 3.89420003e-01
1.82109997e-01 1.32369995e-01 -2.60610014e-01 5.10110021e-01
1.26430005e-01 -2.94450015e-01 -1.34350002e-01 -1.44600004e-01
1.98740005e-01 -3.41720015e-01 2.40710005e-01 -6.08640015e-02
8.60409975e-01 8.92559998e-04 3.46020013e-01 -2.53780007e-01
2.53749996e-01 1.20920002e-01 -3.82739991e-01 1.55090000e-02
-6.41290024e-02 2.54729986e-01 -1.78489998e-01 2.37949997e-01
3.26880008e-01 2.03940004e-01 -5.65020025e-01 3.87109995e-01
-7.00220019e-02 -4.63999987e-01 -1.98210001e-01 -7.27079988e-01
5.36920011e-01 -9.09250021e-01 -2.36699998e-01 -6.20739982e-02
2.38769993e-01 3.24860007e-01 5.51190019e-01 -4.07079995e-01
1.40870005e-01 2.83569992e-01 -1.33379996e-01 -3.99740010e-01
-1.66620001e-01 4.43010002e-01 -6.85970008e-01 1.54200003e-01
6.58490002e-01 2.95630004e-02 2.61550009e-01 1.10969996e+00
-6.78719997e-01 -1.91960007e-01 7.45159984e-02 -3.53740007e-01
-7.15600014e-01 4.14079994e-01 -3.37280005e-01 2.05780007e-02
2.17340007e-01 6.48889989e-02 -3.51080000e-01 -2.44440004e-01
-1.98780000e-01 3.02210003e-01 -1.32180005e-01 -4.86490011e-01
-3.55599999e-01 2.83119995e-02 7.04949975e-01 7.35610008e-01
5.98779976e-01 -3.94160002e-01 -3.96349996e-01 2.55739987e-01
1.30939996e+00 -6.69730008e-02 4.72090006e-01 3.20030004e-01
-2.39020005e-01 1.87910005e-01 2.14310005e-01 3.10609996e-01
-7.22400010e-01 -2.59559989e-01 7.50709996e-02 5.45759976e-01
1.33249998e+00 5.97169995e-01 -7.56640017e-01 2.30540007e-01
3.45209986e-01 8.61710012e-01 5.65129995e-01 1.33000001e-01
4.27370012e-01 -4.36790008e-03 -5.15600026e-01 3.94329995e-01
-1.58559993e-01 -1.29820004e-01 -5.47240004e-02 -1.01789999e+00
6.39170036e-02 -6.47679985e-01 -7.53360018e-02 1.49210006e-01
5.53300023e-01 -6.24869987e-02 -2.01309994e-01 1.70489997e-01
-3.22829992e-01 2.43039995e-01 2.64449995e-02 -9.52059999e-02
3.80879998e-01 -3.69230002e-01 4.54270005e-01 -3.31169993e-01
-1.93750001e-02 1.59300007e-02 1.85659993e-02 -5.27350008e-01
1.05949998e+00 -4.31519985e-01 2.57169992e-01 -9.44539979e-02
7.53460005e-02 -2.06599995e-01 3.21940005e-01 -1.23549998e-02
4.72169995e-01 2.44020000e-01 -3.63059998e-01 1.71650007e-01
-3.50499988e-01 1.40990004e-01 1.18069999e-01 3.65170002e-01
-1.65889993e-01 -3.93799990e-02 -3.70829999e-01 2.03909993e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 5.24869978e-01 -1.19410001e-01 -2.02419996e-01 -6.23929977e-01
-1.53799996e-01 -6.63369969e-02 -3.68499994e-01 2.86490005e-02
1.37950003e-01 -5.87819993e-01 6.02090001e-01 2.60540005e-02
7.07889974e-01 1.20329998e-01 -1.74310002e-02 4.03360009e-01
-3.19680005e-01 -2.50609994e-01 1.60889998e-01 2.47649997e-01
7.79359996e-01 2.74070001e+00 1.19589999e-01 -2.67529994e-01
-3.82809997e-01 -3.36580008e-01 1.41039997e-01 -4.65480000e-01
-8.92079994e-02 2.22540006e-01 -3.60740013e-02 -7.10140020e-02
6.23199999e-01 3.22770000e-01 4.15650010e-01 -3.68530005e-02
-5.82859993e-01 -6.26510024e-01 -3.26169990e-02 2.74789989e-01
-2.66950011e-01 5.27690016e-02 -1.09500003e+00 -1.99760008e-03
-7.49390006e-01 -1.89999994e-02 -1.87619999e-01 -5.19330025e-01
1.71590000e-01 4.40690011e-01 1.90789998e-01 -4.57339995e-02
-2.43069995e-02 2.32710004e-01 1.02179997e-01 -6.02790006e-02
-2.63680011e-01 -1.49090007e-01 4.33890015e-01 -8.51420015e-02
-6.61419988e-01 -6.23379983e-02 -1.37920007e-01 4.54079986e-01
-1.51400000e-01 1.14929996e-01 5.48650026e-01 2.82370001e-01
-2.55129993e-01 1.11149997e-01 -8.47479999e-02 -9.66319963e-02
4.88200009e-01 7.48070002e-01 -6.74910009e-01 3.21949989e-01
-8.98810029e-01 -3.58159989e-01 8.54640007e-02 -1.67390004e-01
-1.60109997e-01 -1.06349997e-01 -2.96649992e-01 -3.03389996e-01
2.29310006e-01 6.51580021e-02 1.70359999e-01 -2.08039999e-01
-1.12930000e-01 -1.58399999e-01 -1.88710004e-01 5.38689971e-01
-3.58050019e-02 3.26770008e-01 -3.18859994e-01 2.01079994e-02
-1.15209997e-01 -3.67400013e-02 5.97510003e-02 9.09829974e-01
1.55420005e-01 1.63939998e-01 3.57499987e-01 -4.75789994e-01
-3.53519991e-02 -1.30729997e+00 2.40319997e-01 1.24170005e+00
4.63180006e-01 -2.90019996e-02 6.63789988e-01 -2.20129997e-01
6.57410026e-01 -2.34209999e-01 -7.43539989e-01 8.95029977e-02
-1.41269997e-01 3.54860008e-01 -6.87049985e-01 -9.53029990e-02
-1.59720004e-01 -4.36020009e-02 5.39509989e-02 -1.33180007e-01
4.47530001e-01 1.41289994e-01 2.71119997e-02 1.18179999e-01
2.02419996e-01 -5.98179996e-01 -6.83469996e-02 -1.58150002e-01
2.42929995e-01 8.88189971e-02 -1.16219997e-01 -3.82679999e-01
1.30649999e-01 -3.00619990e-01 -8.46930027e-01 1.90159995e-02
6.41009986e-01 -8.42439979e-02 7.83680022e-01 -2.63660014e-01
1.98839998e+00 -7.38060027e-02 7.21970022e-01 4.84739989e-02
1.62070006e-01 3.40840012e-01 -1.85289994e-01 2.85659999e-01
-2.35440001e-01 -2.69939989e-01 -5.27239978e-01 2.74859995e-01
6.56930029e-01 2.20569998e-01 -6.80390000e-01 3.59100014e-01
2.51430005e-01 7.24399984e-02 2.58690000e-01 3.22510004e-01
3.57239991e-02 -3.88200015e-01 1.18890002e-01 -2.90010005e-01
-6.32229984e-01 -1.48519993e-01 -1.21040002e-01 -1.74060002e-01
2.48679996e-01 4.74539995e-01 -2.50710011e-01 -5.23679972e-01
-1.82730004e-01 -8.30529988e-01 6.15109980e-01 -3.84620011e-01
9.74919975e-01 4.49510008e-01 -5.42559981e-01 -3.18340003e-01
-5.07809997e-01 -2.73739994e-02 -1.26379997e-01 5.42479992e-01
4.85489994e-01 8.55289996e-02 -1.42509997e-01 8.77849981e-02
2.47189999e-01 -2.38490000e-01 6.14570007e-02 -4.23129983e-02
3.57470006e-01 9.73920003e-02 -3.58500004e-01 1.70269996e-01]
[ 7.54669979e-02 -2.92360008e-01 -2.60369986e-01 -2.81670004e-01
1.60970002e-01 -1.94720000e-01 -2.82059997e-01 4.91409987e-01
-4.11189999e-03 9.58790034e-02 3.33959997e-01 1.68740004e-02
2.06159994e-01 -1.57429993e-01 -1.61379993e-01 2.15379998e-01
-1.48359999e-01 2.54599992e-02 -4.36809987e-01 -6.36610016e-02
5.52609980e-01 3.08189988e+00 -1.17839999e-01 -4.61309999e-01
-2.76089996e-01 2.09480003e-01 -2.05440000e-01 -5.71550012e-01
3.34479988e-01 1.59130007e-01 2.54360004e-03 1.80040002e-01
1.34719998e-01 -9.74040031e-02 3.55369985e-01 -4.74280000e-01
-7.92569995e-01 -5.44179976e-01 2.42999997e-02 6.35990024e-01
1.23369999e-01 -1.29130006e-01 -2.65650004e-01 -2.49569997e-01
-5.21990001e-01 -4.05229986e-01 4.84030008e-01 1.83729995e-02
2.30389997e-01 6.21379986e-02 -1.92919999e-01 2.95060009e-01
-3.57930005e-01 1.67019993e-01 3.18679988e-01 -3.60540003e-01
-1.09779999e-01 -1.56320006e-01 4.59210008e-01 9.64749977e-02
-3.77000004e-01 -7.76150003e-02 -4.88990009e-01 2.05750003e-01
5.05429983e-01 5.34190014e-02 -2.59779990e-01 5.10420024e-01
9.75209996e-02 3.26059997e-01 1.43539995e-01 2.27149995e-03
4.86149997e-01 4.69379991e-01 -4.11240011e-01 -1.71480000e-01
-3.97439986e-01 -2.89009988e-01 -1.77560002e-01 3.70009989e-02
3.48300010e-01 1.59339994e-01 -7.42810011e-01 1.88970000e-01
4.36850004e-02 5.72080016e-01 -6.70159996e-01 -4.39470001e-02
-2.83360004e-01 -3.19959998e-01 -2.04040006e-01 -8.78980011e-02
-1.57240003e-01 2.18180008e-02 -5.67569971e-01 6.32960021e-01
-1.00970000e-01 -6.55760020e-02 5.82689978e-03 3.30349989e-02
3.97830009e-01 -3.11659992e-01 -6.10889971e-01 2.75590003e-01
1.00079998e-01 -4.19900000e-01 6.35599997e-03 1.87170005e+00
3.14729989e-01 -3.60040009e-01 8.13839972e-01 -2.17099994e-01
-1.84589997e-02 -2.26319999e-01 1.45850003e-01 -1.43500000e-01
-4.14239988e-02 5.59740007e-01 -6.67519987e-01 -2.19589993e-01
1.90109998e-01 3.30150008e-01 6.12900019e-01 4.67709988e-01
4.20260012e-01 -5.28190017e-01 2.31650006e-02 3.29100005e-02
4.73060012e-01 1.40060000e-02 -1.73960000e-01 -4.43619996e-01
4.13769990e-01 -2.06790000e-01 3.92830014e-01 3.02109987e-01
7.31339976e-02 4.21640016e-02 -9.27100003e-01 -4.76139992e-01
2.43100002e-01 -1.33790001e-01 -2.22379997e-01 -4.14570011e-02
1.58500004e+00 3.74810010e-01 2.59939991e-02 -2.42719993e-01
3.05779994e-01 1.46870002e-01 1.16659999e-01 -2.94179991e-02
-7.83390030e-02 -2.25119993e-01 1.33149996e-01 -6.48420006e-02
-2.86870003e-01 -1.05600003e-02 -3.46679986e-01 4.21449989e-02
-6.00409985e-01 8.24810028e-01 3.10220003e-01 1.64890006e-01
-7.29210004e-02 1.93939999e-01 -9.84980017e-02 -2.03830004e-02
-4.09090012e-01 -1.04039997e-01 1.91689998e-01 -1.59689993e-01
3.80259991e-01 6.28019989e-01 2.59499997e-01 -3.33669990e-01
-7.33330011e-01 -4.07429993e-01 6.84230030e-01 -6.63380027e-02
5.04360020e-01 -2.89829999e-01 -3.90859991e-01 -4.59310003e-02
-2.66240001e-01 -1.67699993e-01 -1.50370002e-01 1.48279995e-01
-2.81430006e-01 -1.70870006e-01 -2.55760014e-01 -5.62830009e-02
-1.66500002e-01 3.51060003e-01 4.10320014e-02 2.73110002e-01
3.00200004e-02 1.64649993e-01 -8.41889977e-02 5.75059988e-02]
[ 5.46909988e-01 -7.55890012e-01 -9.20799971e-01 -8.20680022e-01
1.48800001e-01 -1.32200003e-01 2.49499991e-03 5.39979994e-01
-3.12929988e-01 -2.12009996e-02 -2.96559989e-01 1.51110003e-02
-3.76980007e-02 -4.07290012e-01 3.36799994e-02 -2.90179998e-01
-5.64790010e-01 6.47960007e-01 3.96770000e-01 -1.91139996e-01
6.98249996e-01 1.61090004e+00 8.42899978e-01 -1.45980000e-01
-1.13940001e-01 -5.39680004e-01 -7.22540021e-01 -1.47310004e-01
-6.78520024e-01 1.93159997e-01 4.17819992e-03 -3.96059990e-01
8.99320021e-02 3.30909997e-01 -4.09649983e-02 4.59470004e-01
-3.07020009e-01 -2.61469990e-01 5.39489985e-01 6.60130024e-01
-5.68049997e-02 -3.70750010e-01 -1.46880001e-03 -1.48770005e-01
2.13000000e-01 -1.36710003e-01 5.12350023e-01 -5.27249992e-01
2.73449998e-02 3.00159991e-01 4.59829986e-01 3.65709990e-01
2.23989993e-01 1.47589996e-01 -9.97050032e-02 2.26520002e-01
-5.95120013e-01 -4.34839994e-01 -5.26629984e-02 -3.05240005e-01
-1.91280007e-01 -4.08630013e-01 -2.83719987e-01 6.89159989e-01
-7.18529999e-01 2.89909989e-01 -1.31809995e-01 -8.75229985e-02
2.70280004e-01 3.31449993e-02 1.93159997e-01 -2.50629991e-01
1.13150001e-01 4.88560013e-02 -4.41439986e-01 -8.88149977e-01
-3.45840007e-01 -5.47370017e-01 2.13919997e-01 1.46469995e-01
-3.76450002e-01 5.63120008e-01 4.44130003e-01 -6.05080016e-02
5.85870028e-01 4.50850010e-01 -1.27309993e-01 6.86409995e-02
3.78729999e-01 -7.15900004e-01 2.72450000e-01 2.16839999e-01
-5.09050012e-01 2.82240003e-01 5.28559983e-01 2.97140002e-01
-3.33429992e-01 -1.62200004e-01 -3.97210002e-01 3.37130010e-01
6.61469996e-01 -4.09590006e-01 -1.97520003e-01 -8.77860010e-01
-4.36379999e-01 1.79399997e-01 3.39769991e-03 9.73770022e-01
7.32789993e-01 -3.44499983e-02 -4.76709992e-01 -4.99819994e-01
4.97110009e-01 -7.77419984e-01 -3.60000014e-01 -1.58899993e-01
1.19869998e-02 -1.39750004e-01 -5.91250002e-01 -1.58280000e-01
9.19710025e-02 4.62440014e-01 -1.17980000e-02 8.29930007e-01
7.96280026e-01 -5.64369977e-01 -1.02080004e-02 -3.86090010e-01
-6.74889982e-01 -3.42109986e-02 1.99829996e-01 2.01010004e-01
7.43009984e-01 -3.58200014e-01 1.07579999e-01 -7.77289987e-01
3.24770004e-01 -7.43889987e-01 -7.90560007e-01 4.95799989e-01
3.70090008e-01 -1.55349998e-02 1.20260000e+00 -6.90900028e-01
1.49619997e+00 4.31109995e-01 2.64299989e-01 -5.99219978e-01
-4.51359987e-01 7.66879976e-01 -1.43020004e-02 1.43340006e-01
1.19379997e-01 -5.56930006e-01 -9.95339990e-01 8.20089996e-01
-3.02410007e-01 9.95709971e-02 -4.92139995e-01 2.25950003e-01
5.18890023e-01 6.51669979e-01 -7.13440031e-02 3.06109991e-03
-2.95789987e-01 -4.40490007e-01 -1.61009997e-01 2.52220005e-01
-3.50809991e-01 -7.96020031e-02 4.33560014e-01 -1.43889993e-01
-1.26969993e-01 1.01450002e+00 -6.24639988e-02 -5.60909986e-01
-6.52670026e-01 1.86299995e-01 -3.76190007e-01 -2.40750000e-01
2.50979990e-01 -2.06090003e-01 -8.57209980e-01 -5.84480017e-02
-1.23539999e-01 -7.35830009e-01 -4.83509988e-01 2.75110006e-01
-1.10220000e-01 3.17330003e-01 1.87999994e-01 1.05719995e+00
5.85460007e-01 3.13740000e-02 2.11569995e-01 1.13799997e-01
4.65829998e-01 -3.05020005e-01 -4.51409996e-01 2.87349999e-01]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[-0.34636 -0.88984001 -0.50321001 -0.43516001 0.54518998 0.17437001
-0.093541 0.16141 -0.46575999 -0.22 -0.31415001 -0.13484
-0.37617999 -0.67635 0.78820002 -0.33384001 -0.42414001 0.32367
0.50670999 0.21540999 0.43296 1.49049997 0.31795001 -0.15196
0.2579 -0.35666001 -0.63880002 -0.086453 -0.94755 0.19203
0.31161001 -0.74491 -0.59210998 0.4332 -0.064934 -0.48862001
0.35600999 -0.44780999 -0.015773 0.18203001 0.051751 -0.2854
-0.14727999 0.1513 -0.33048001 0.27135 1.16659999 -0.36662
0.090829 0.87301999 -0.13505 0.21204001 0.57270002 0.54259002
-0.50335002 0.16767 -0.82530999 -0.45962 -0.42642 -0.2164
0.088689 -0.15061 -0.16785 -0.31794 -0.69608998 0.40715
-0.29190999 -0.042072 0.90051001 0.35947999 0.030644 -0.028853
0.086821 0.74741 -0.52008998 0.20655 0.44053999 -0.11865
-0.15449999 -0.22457001 -0.15453 0.16101 -0.30825001 -0.28479999
-0.50823998 0.48732999 -0.012029 0.034592 0.48304 -0.56752002
-0.057299 0.22070999 -0.34200001 -0.060634 0.95499998 -0.60952997
0.59577 -0.11553 -0.67475998 0.52658999 0.82163 0.35126001
0.15521 0.12135 0.38191 0.24228001 -0.51485997 1.14810002
0.07281 0.23024 -0.68901998 -0.17606001 -0.24308001 -0.13686
-0.13467 0.059625 -0.68668997 0.15907 0.11983 -0.024954
0.34898001 0.15456 0.047524 0.23616999 0.54784 -1.01380002
0.10351 0.26865 -0.064867 0.23893 0.026141 0.081648
0.74479997 -0.67208999 0.23351 -0.55070001 -0.14296 -0.30303001
-0.40233999 0.012984 0.86865002 -0.14025 1.13900006 -0.093339
1.56060004 0.41804001 0.54067999 -0.43340999 -0.090589 0.56682003
-0.21291 0.45693001 -0.64519 -0.05866 0.21477 0.45563
-0.15218 0.36307001 -0.25441 -0.72013998 0.52191001 0.55509001
-0.073841 0.44994 -0.11501 0.1911 0.077304 0.18629
0.60244 0.028085 0.17228 -0.24455 0.04822 0.51318002
-0.06824 0.35515001 -0.80987 -0.42732999 -0.72728997 0.47817001
0.87611997 -0.18855 0.30390999 -0.14161 0.26699001 -0.81572002
-0.67589998 -0.34687999 0.53188998 0.75443 -0.083874 0.77434999
0.081108 -0.29840001 -0.24409001 -0.14574 -0.1186 0.085964
0.48076999 -0.13097 ]
[ 1.07439995 -0.49325001 -0.23126 -0.197 -0.087485 -0.16806
-0.11092 0.42857999 -0.16893999 0.0094633 -0.50453001 -0.40006
0.31169 0.50295001 -0.48537001 -0.20689 -0.62162 0.38407999
0.22182 0.051087 -0.018028 1.37919998 0.3163 -0.17425001
0.11344 -0.42723 -0.28510001 -0.1246 -0.2024 0.18217
-0.37834001 -0.22421999 0.38877001 0.20065001 -0.29708999 0.77051002
0.13087 -0.25084001 0.54755002 0.38086 0.28174999 -0.15691
-0.71987998 0.24118 0.073913 -0.46965 1.02180004 0.049863
0.036841 0.54706001 -0.15903001 0.53780001 -0.10335 0.51644999
-0.25512001 -0.18553001 -0.51804 -0.24337 0.57081997 -0.39017001
-0.17132001 -0.14939 -0.1724 0.91408002 -0.45778 0.40143001
0.075224 -0.4104 -0.1714 -0.63552999 0.60185999 -0.3193
-0.46136999 0.030652 0.32890001 -0.2472 -0.49831 -0.90982997
-0.057251 0.20213 -0.51845998 0.46320999 0.032707 0.29872999
1.11189997 -0.35784999 0.34929001 -0.51739001 0.25235 -1.0309
0.21052 0.06349 -0.10589 0.43222001 0.20389 0.065589
-0.62914002 0.1096 -0.86363 0.44760999 0.43114999 0.041376
-0.42429999 -0.080897 0.093115 0.22603001 0.31176999 0.83004999
-0.25659001 0.013861 0.38326001 -0.52025998 0.30410001 -0.52507001
-0.78566003 -0.046498 0.41154999 -0.21447 -0.24202999 -0.24732
1.01129997 0.067517 0.18331 -0.17636 0.49777001 -0.21067999
0.0037579 0.22881 -0.15993001 -0.13421001 -0.27379999 -0.20734
0.13407999 -0.57762003 -0.66526997 -0.42083001 0.65882999 -0.53825998
-0.50585997 0.51735002 0.25468999 -0.83724999 0.83806998 -0.42219999
1.0776 0.065962 0.48954999 -0.78614002 -0.19338 0.097524
0.27215999 0.037038 0.61778998 -0.29506999 -0.97285002 0.53106999
-0.32765001 -0.045966 -0.75436997 0.024904 0.64872003 0.023095
0.32062 0.35837999 -0.091125 -0.10866 0.33048001 -0.1162
-0.40981999 0.43928999 -0.16706 0.26047 0.090957 0.92714
0.099946 -0.29513001 -0.35341001 0.33693001 -0.42203999 -0.065625
0.54738998 -0.41751 -0.86232001 -0.65891999 -0.41549 0.067035
-0.41558 0.15092 0.17556 0.94068003 0.22527 0.65908003
0.15809 0.061199 0.63612998 -0.17089 -0.017591 -0.054003
-0.69072002 0.65179998]
[ 1.03529997e-01 7.20499977e-02 -2.93029994e-02 -4.46799994e-01
-8.61259997e-02 7.40030035e-02 -4.65499997e-01 -6.18570000e-02
-5.03650010e-01 1.95480004e-01 -1.03349999e-01 7.75929987e-01
-1.72040001e-01 -4.53520000e-01 2.63040006e-01 1.64340004e-01
2.80279994e-01 2.89330006e-01 3.10360014e-01 1.27750002e-02
6.79520011e-01 2.62770009e+00 3.89640003e-01 -4.49900001e-01
2.17969999e-01 9.16400030e-02 -1.67940006e-01 7.72420019e-02
2.91269988e-01 1.20530002e-01 -5.49659990e-02 9.16600004e-02
1.31300002e-01 -9.33270007e-02 -3.53009999e-01 -5.03880024e-01
-7.29799986e-01 -3.61380011e-01 -2.99659997e-01 2.07839999e-02
-7.03599975e-02 -6.72269985e-02 -3.62650007e-02 -1.46009997e-02
-5.98580018e-02 -1.63020000e-01 3.00660014e-01 -9.28120017e-02
-6.69799969e-02 1.43830001e-01 1.03950001e-01 -1.93039998e-02
-4.07020003e-01 8.86749983e-01 2.67349988e-01 -1.23829998e-01
2.73739994e-02 3.44639987e-01 6.04049981e-01 2.80640006e-01
1.41320005e-01 2.46429995e-01 -4.48760003e-01 5.42909980e-01
1.96759999e-01 -4.94709998e-01 2.68779993e-02 2.69100010e-01
2.53410012e-01 9.88679975e-02 1.77919999e-01 -3.22290003e-01
-1.05930001e-01 1.89520001e-01 -2.57629991e-01 2.43619993e-01
-2.45560005e-01 -1.36539996e-01 -1.08170003e-01 -4.21220005e-01
-1.61640003e-01 -3.41199994e-01 -1.47389993e-01 -2.16409996e-01
-5.62399998e-02 7.15879977e-01 6.40570000e-02 -3.07790011e-01
6.57369971e-01 -6.36910021e-01 2.64039993e-01 2.15059996e-01
1.83850005e-01 3.32610011e-01 -6.30220026e-02 -2.22450003e-01
6.31980002e-02 -4.47950006e-01 -2.52279997e-01 1.97699994e-01
-3.55479985e-01 1.19139999e-01 -2.99149990e-01 1.29390001e-01
-4.30770010e-01 -1.37700006e-01 2.38429993e-01 1.45529997e+00
-1.13710001e-01 -3.79790008e-01 -2.83129990e-01 -4.64819998e-01
-1.92760006e-01 1.98490005e-02 2.70090014e-01 -1.70849994e-01
-5.20099998e-02 -2.03219995e-01 -3.27439994e-01 -5.07570028e-01
2.98289992e-02 1.80350006e-01 3.05330008e-01 2.40700006e-01
4.66120005e-01 -7.12530017e-01 5.96719980e-01 2.13310003e-01
3.48639995e-01 3.80190015e-01 1.38260007e-01 -7.33380020e-02
2.26349995e-01 -4.55599993e-01 8.19810014e-03 3.47319990e-01
-1.36079997e-01 -6.59820020e-01 -2.79430002e-01 4.81799990e-02
-3.97129990e-02 2.86449999e-01 1.41259998e-01 -3.94299999e-02
1.44519997e+00 4.59950000e-01 7.92850032e-02 -3.55580002e-01
3.09360009e-02 -2.50809994e-02 5.64870015e-02 9.00759995e-02
5.20309992e-02 -3.99529994e-01 3.40330005e-01 -4.17400002e-01
-3.82189989e-01 2.22049996e-01 1.09520003e-01 -1.64539993e-01
1.20609999e-01 1.60919994e-01 -3.27459991e-01 2.45800003e-01
-2.28320006e-02 2.74109989e-01 -2.10689995e-02 3.91460001e-01
-5.61020017e-01 6.95510030e-01 -3.55260000e-02 4.71640006e-02
6.71909988e-01 4.81240004e-01 -2.78840009e-02 5.05490005e-01
-5.41859984e-01 -3.52369994e-01 -3.12009990e-01 -1.76020002e-03
9.59439993e-01 -5.03639996e-01 4.39889997e-01 4.71810013e-01
4.25799996e-01 -5.92209995e-01 -3.96219999e-01 1.89040005e-02
3.33819985e-02 -2.90149987e-01 -1.22079998e-01 2.76309997e-02
-2.66250014e-01 -1.97340008e-02 2.31020004e-01 8.76149982e-02
-7.69149978e-03 1.90050006e-02 -4.42119986e-01 -6.81999996e-02]
[ 2.41689995e-01 -3.45340014e-01 -2.23069996e-01 -1.29069996e+00
2.52849996e-01 -5.51280022e-01 -8.03359970e-02 -8.17670021e-03
3.11360002e-01 -4.51009989e-01 2.46610001e-01 3.64410013e-01
9.43359971e-01 -3.54200006e-02 7.80480027e-01 -3.97650003e-01
3.11250001e-01 -1.77430004e-01 -4.19889987e-01 -3.78149986e-01
6.72299981e-01 3.17160010e+00 3.24960016e-02 -3.16400006e-02
5.80680013e-01 -4.44579989e-01 -5.56120016e-02 1.80519998e-01
2.85719991e-01 9.58700031e-02 2.14369997e-01 4.97310013e-02
1.87199995e-01 1.19139999e-01 2.74080001e-02 -8.06079984e-01
-3.08349997e-01 -8.97369981e-01 -1.97720006e-01 2.67409999e-02
-3.87650013e-01 1.16590001e-01 -2.01100007e-01 2.01010004e-01
-7.91329965e-02 -5.09539992e-02 6.01889985e-03 3.34699988e-01
-2.11180001e-01 7.40419999e-02 -2.81410009e-01 -5.96150011e-02
-3.52959991e-01 6.47480011e-01 5.39080016e-02 -3.13760012e-01
-3.66219997e-01 -2.77550012e-01 2.26760004e-02 4.88109998e-02
1.43120006e-01 -1.85800001e-01 -5.69639981e-01 -5.41190028e-01
1.86159998e-01 1.88539997e-01 2.75209993e-01 -1.78350002e-01
-3.74379992e-01 1.21090002e-01 1.86099997e-03 -9.21269972e-03
1.01860002e-01 9.80810001e-02 -3.72449994e-01 6.64150000e-01
5.73659986e-02 -4.38450009e-01 -4.05250013e-01 -5.59589982e-01
-1.13430001e-01 -5.49870014e-01 -2.63209999e-01 -2.84709990e-01
1.44109994e-01 1.03600003e-01 -3.21980000e-01 -2.15299994e-01
9.86679971e-01 -4.19369996e-01 3.11899990e-01 3.33799988e-01
1.60180002e-01 3.31369996e-01 -2.54939999e-02 -3.78799997e-02
-1.20480001e-01 -1.21749997e-01 9.47659984e-02 2.61579990e-01
2.99309995e-02 -2.96249986e-01 4.34009999e-01 -3.65360007e-02
-4.28519994e-01 -3.96380007e-01 -2.49730006e-01 1.10179996e+00
-2.28119999e-01 2.43239999e-01 8.38569999e-02 -4.88810003e-01
-2.13159993e-01 4.02490012e-02 -4.05160010e-01 -1.24109998e-01
-1.97280005e-01 -7.36959994e-01 -4.45820004e-01 -4.49919999e-01
-7.37069994e-02 -1.64539993e-01 1.60109997e-01 -4.19519991e-01
4.14169997e-01 -5.72770000e-01 5.13670027e-01 8.11180025e-02
2.92879995e-02 3.53089988e-01 -1.08029999e-01 1.37020007e-01
3.88280004e-01 -2.88129985e-01 6.82030022e-01 1.74119994e-01
-8.85400027e-02 -2.97850013e-01 -2.84550011e-01 1.35700002e-01
1.52319998e-01 2.20369995e-01 6.88120008e-01 -1.30999997e-01
1.81970000e+00 -4.51530010e-01 3.95529985e-01 -6.08169973e-01
3.36979985e-01 1.10900000e-01 -2.84759998e-01 3.04610014e-01
-2.16059998e-01 -5.81110008e-02 4.41720009e-01 -2.42310002e-01
-1.27580002e-01 -4.87929992e-02 -1.54949993e-01 -8.54910016e-01
9.29120034e-02 -6.08090013e-02 2.90730000e-02 -3.87349993e-01
-7.08530024e-02 -6.59749985e-01 -3.81570011e-01 5.01699984e-01
-7.35599995e-01 4.15210009e-01 2.13280007e-01 -3.37790012e-01
6.69019997e-01 4.24860001e-01 -1.21480003e-01 -1.06260004e-02
1.27450004e-01 -1.35609999e-01 2.34229997e-01 3.51099998e-01
1.28410006e+00 1.29820004e-01 2.13569999e-01 3.28570008e-01
1.65669993e-01 -2.14579999e-01 -4.42750007e-01 3.28500003e-01
1.80010006e-01 6.48650005e-02 -3.58799994e-01 -1.42259998e-02
3.11250001e-01 -2.20489994e-01 3.28290015e-02 3.85250002e-01
-1.05120003e-01 2.78010011e-01 -1.01709999e-01 -7.15209991e-02]
[ 0.041052 -0.54705 -0.72193998 -0.31235999 -0.43849 0.10691
-0.50641 -0.45401001 -0.28623 0.018973 0.020495 0.42860001
-0.0057162 -0.21272001 0.71864998 -0.091906 -0.55365002 0.39133
0.15351 -0.27454001 0.56528002 3.04830003 0.30467001 -0.37893
0.37865999 0.13925999 -0.11482 0.48212999 -0.30522999 0.43125001
-0.09667 0.069156 0.31426001 0.26350999 -0.31189999 -0.39881
-0.55656999 -0.35934001 -0.25402001 0.072061 -0.12966999 -0.11247
-0.041192 -0.042619 -0.07848 0.31992 0.41635999 0.26131001
-0.18175 -0.1279 0.21332 -0.41973001 -0.50444001 0.37705001
0.83955002 -0.34571001 -0.43000999 -0.18653999 -0.061082 -0.087612
0.092833 0.52604997 -0.57611001 -0.19328 0.20576 0.24607
0.1631 -0.18138 0.032592 0.19169 0.73565 -0.25718999
0.30072999 0.56699002 -0.21544001 0.18933 -0.12287 -0.65759999
0.021702 0.1041 0.098952 -0.43171999 -0.27517 -0.15448
0.31301001 -0.032041 -0.090526 0.14489999 0.68151999 -0.88242
0.30816999 -0.62702 0.12274 0.014773 -0.16887 0.56159002
0.022004 0.52085 -0.22685 0.09633 0.26956999 0.30489001
0.018463 0.31009001 0.04198 0.32381999 0.13153 0.89722002
0.15475 -0.38806999 -0.52993 -0.35383999 -0.0913 0.57230002
0.48067001 0.24438 0.074138 -0.019957 -0.35030001 -0.034695
-0.12045 0.39998999 -0.37015 -0.53040999 0.10655 -0.44973001
0.43105 -0.44937 0.48675999 0.43836999 0.043421 0.52675003
0.61176002 0.26704001 0.59239 0.23650999 0.12841 -0.10665
-0.46715 -0.039081 -0.24921 0.030486 0.092933 -0.04841
1.83580005 0.077535 -0.11494 -0.13668001 -0.23983 0.31948
0.19205999 0.38894001 0.34755 -0.038804 0.19541 -0.37099999
-0.027576 -0.24127001 -0.16868 0.032815 0.08139 0.054121
-0.42785999 0.26447001 0.054847 -0.21765999 0.015181 0.57656002
0.24033 0.62076002 -0.019055 -0.31509 0.76424998 0.35168999
-0.28512001 0.15175 0.11238 -0.60829997 0.35087001 0.19140001
0.51753998 0.20893 -0.63814002 0.19403 0.24493 0.46606001
-0.32235 0.37286001 -0.19508 0.13237999 -0.35420999 0.22849
0.36032 -0.0050241 -0.051955 -0.37755999 -0.087065 0.3592
0.11564 0.44372001]
[ 4.71520007e-01 -5.87790012e-01 -6.76150024e-01 -4.67000008e-01
1.11709997e-01 3.26370001e-01 -4.65070009e-01 -8.05180013e-01
-1.65340006e-01 -1.13849998e-01 -1.36849999e-01 2.56980002e-01
3.36279988e-01 1.90149993e-01 1.32440001e-01 2.37419993e-01
-1.46239996e+00 8.59059989e-01 5.53650022e-01 1.94330007e-01
2.73930013e-01 1.05669999e+00 6.24970019e-01 -4.30469990e-01
7.14770019e-01 -4.73030001e-01 -8.97960007e-01 2.56910007e-02
-6.42499983e-01 2.15990007e-01 -1.22769997e-01 -5.36949992e-01
5.91489971e-01 6.28649965e-02 1.51260002e-02 -6.57150000e-02
1.61709994e-01 -8.86740014e-02 -7.09370002e-02 6.12349987e-01
1.38689995e-01 -3.67980003e-01 -9.46219981e-01 1.30669996e-01
-2.82139987e-01 -3.02709997e-01 4.05889988e-01 -2.11899996e-01
1.74940005e-01 2.38450006e-01 3.41769993e-01 4.50269997e-01
-7.82140017e-01 1.64210007e-01 7.19319999e-01 -6.80140018e-01
-4.93660003e-01 3.67380008e-02 2.62410015e-01 -8.48299980e-01
-6.59759998e-01 4.04370010e-01 -2.61209998e-02 5.83829999e-01
-3.28000009e-01 6.39530003e-01 1.20350003e-01 7.21519988e-04
8.28130007e-01 -3.83879989e-01 5.35929978e-01 -4.59630013e-01
-5.12839973e-01 1.74339995e-01 2.06220001e-01 -8.01329970e-01
-4.74339992e-01 -4.32810009e-01 -6.11400008e-01 1.71409994e-01
5.54369986e-01 6.11240007e-02 6.43959999e-01 -5.23599982e-01
1.35130000e+00 -1.40279993e-01 3.67210001e-01 -3.68629992e-01
7.95690000e-01 -1.01139998e+00 -1.47060007e-01 4.48889993e-02
4.06240001e-02 5.33150017e-01 3.40680003e-01 2.50710011e-01
-1.26489997e-01 -1.15050003e-01 -1.48660004e-01 7.65860021e-01
9.80419964e-02 6.28759980e-01 -4.09599990e-01 -3.33020017e-02
-3.77389997e-01 -2.54250001e-02 -8.92150030e-02 1.63540006e+00
5.04270017e-01 -3.86139989e-01 -1.25259995e-01 1.65910006e-01
-2.19990000e-01 -6.84350014e-01 -2.99389988e-01 -1.25550002e-01
3.63970011e-01 3.89310002e-01 -9.63360012e-01 -7.42670000e-02
3.48879993e-01 2.36190006e-01 -8.27549994e-01 3.19779992e-01
-3.00359994e-01 -4.01740015e-01 7.04670012e-01 -3.32989991e-01
-1.26369998e-01 -3.72830003e-01 -9.09730017e-01 -1.33279994e-01
-5.32779992e-02 -3.47909987e-01 2.48980001e-01 -4.34430003e-01
-2.42050007e-01 -6.21890008e-01 -1.27769995e+00 -9.66310024e-01
4.20859993e-01 -4.12339985e-01 1.20589995e+00 -2.55379993e-02
1.01170003e+00 -1.11219997e-03 5.92599988e-01 -5.58459997e-01
-3.69489998e-01 5.58549985e-02 -6.60969973e-01 3.91559988e-01
3.09260011e-01 -6.28539994e-02 -1.14900005e+00 4.69440013e-01
-5.05439997e-01 -1.75500005e-01 -3.80259991e-01 -1.97349995e-01
7.31180012e-01 -1.77149996e-01 4.18940008e-01 4.23940003e-01
-5.16149998e-02 -1.20180003e-01 3.96189988e-01 -2.36780003e-01
1.19029999e-01 2.93020010e-01 -1.81150004e-01 4.59089994e-01
1.93489999e-01 4.75919992e-01 -3.67170006e-01 -8.76540027e-04
-6.15379997e-02 -4.12099995e-02 -1.95240006e-01 -6.47260016e-03
4.30709988e-01 6.84989989e-02 -4.27579999e-01 -3.15990001e-01
-2.59119987e-01 -6.58209980e-01 -1.92790002e-01 5.57280004e-01
-2.24089995e-01 2.21340001e-01 4.68760014e-01 6.37290001e-01
6.58949971e-01 8.79120007e-02 -3.06369990e-01 -3.22079986e-01
6.47899985e-01 1.75490007e-01 -5.78580022e-01 8.94800007e-01]
===================================
===== For the word2vec
===================================
1. Download and unzip the pre-trained model of GoogleNews-vectors-negative300.bin.gz.
2. Install the gensim tools:
sudo pip install --upgrade gensim
3. Code for vector extraction from given sentence.
import gensim
print("==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin")
dictFileName = './GoogleNews-vectors-negative300.bin'
wv = gensim.models.KeyedVectors.load_word2vec_format(dictFileName, binary=True)
The Output is:
==>> loading the pre-trained word2vec model: GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loading projection weights from ./GoogleNews-vectors-negative300.bin
INFO:gensim.models.utils_any2vec:loaded (3000000, 300) matrix from ./GoogleNews-vectors-negative300.bin
INFO:root:Data statistic
INFO:root:train_labels:19337
INFO:root:test_labels:20632
INFO:root:train_sentences:19337
INFO:root:dev_sentences:2000
INFO:root:test_sentences:20632
INFO:root:dev_labels:2000
embed_size:300
vocab_size:3000000
vocab_path = ''data/bilstm.vocab''
index_to_word = [key for key in wv.vocab]
word_to_index = {}
for index, word in enumerate(index_to_word):
word_to_index[word] = index
with open(vocab_path, "w") as f:
f.write(json.dumps(word_to_index))
Let's take a deep understanding on the bidirectional-LSTM-for-text-classification-master
class BiLSTM(nn.Module):
def __init__(self, embedding_matrix, hidden_size=150, num_layer=2, embedding_freeze=False):
super(BiLSTM,self).__init__() # embedding layer
vocab_size = embedding_matrix.shape[0]
embed_size = embedding_matrix.shape[1]
self.hidden_size = hidden_size
self.num_layer = num_layer
self.embed = nn.Embedding(vocab_size, embed_size)
self.embed.weight = nn.Parameter(torch.from_numpy(embedding_matrix).type(torch.FloatTensor), requires_grad=not embedding_freeze)
self.embed_dropout = nn.Dropout(p=0.3)
self.custom_params = []
if embedding_freeze == False:
self.custom_params.append(self.embed.weight) # The first LSTM layer
self.lstm1 = nn.LSTM(embed_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm1.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.lstm1_dropout = nn.Dropout(p=0.3) # The second LSTM layer
self.lstm2 = nn.LSTM(2*self.hidden_size, self.hidden_size, num_layer, dropout=0.3, bidirectional=True)
for param in self.lstm2.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param)
self.lstm2_dropout = nn.Dropout(p=0.3) # Attention
self.attention = nn.Linear(2*self.hidden_size,1)
self.attention_dropout = nn.Dropout(p=0.5) # Fully-connected layer
self.fc = weight_norm(nn.Linear(2*self.hidden_size,3))
for param in self.fc.parameters():
self.custom_params.append(param)
if param.data.dim() > 1:
nn.init.orthogonal(param)
else:
nn.init.normal(param) self.hidden1=self.init_hidden()
self.hidden2=self.init_hidden() def init_hidden(self, batch_size=3):
if torch.cuda.is_available():
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda(),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)).cuda())
else:
return (Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size)),
Variable(torch.zeros(self.num_layer*2, batch_size, self.hidden_size))) def forward(self, sentences): print("==>> sentences: ", sentences) # get embedding vectors of input
padded_sentences, lengths = torch.nn.utils.rnn.pad_packed_sequence(sentences, padding_value=int(0), batch_first=True)
print("==>> padded_sentences: ", padded_sentences) embeds = self.embed(padded_sentences)
print("==>> embeds: ", embeds) # pdb.set_trace() noise = Variable(torch.zeros(embeds.shape).cuda())
noise.data.normal_(std=0.3)
embeds += noise
embeds = self.embed_dropout(embeds)
# add noise packed_embeds = torch.nn.utils.rnn.pack_padded_sequence(embeds, lengths, batch_first=True) print("==>> packed_embeds: ", packed_embeds) # First LSTM layer
# self.hidden = num_layers*num_directions batch_size hidden_size
packed_out_lstm1, self.hidden1 = self.lstm1(packed_embeds, self.hidden1)
padded_out_lstm1, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm1, padding_value=int(0))
padded_out_lstm1 = self.lstm1_dropout(padded_out_lstm1)
packed_out_lstm1 = torch.nn.utils.rnn.pack_padded_sequence(padded_out_lstm1, lengths) pdb.set_trace() # Second LSTM layer
packed_out_lstm2, self.hidden2 = self.lstm2(packed_out_lstm1, self.hidden2)
padded_out_lstm2, lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out_lstm2, padding_value=int(0), batch_first=True)
padded_out_lstm2 = self.lstm2_dropout(padded_out_lstm2) # attention
unnormalize_weight = F.tanh(torch.squeeze(self.attention(padded_out_lstm2), 2))
unnormalize_weight = F.softmax(unnormalize_weight, dim=1)
unnormalize_weight = torch.nn.utils.rnn.pack_padded_sequence(unnormalize_weight, lengths, batch_first=True)
unnormalize_weight, lengths = torch.nn.utils.rnn.pad_packed_sequence(unnormalize_weight, padding_value=0.0, batch_first=True)
logging.debug("unnormalize_weight size: %s" % (str(unnormalize_weight.size())))
normalize_weight = torch.nn.functional.normalize(unnormalize_weight, p=1, dim=1)
normalize_weight = normalize_weight.view(normalize_weight.size(0), 1, -1)
weighted_sum = torch.squeeze(normalize_weight.bmm(padded_out_lstm2), 1) # fully connected layer
output = self.fc(self.attention_dropout(weighted_sum))
return output
==>> Some Testing:
(a). len(wv.vocab) = 300,0000
(b). wv.vocab is what ? Something like this:
{ ... , u'fivemonth': <gensim.models.keyedvectors.Vocab object at 0x7f90945bd810>,
u'retractable_roofs_Indians': <gensim.models.keyedvectors.Vocab object at 0x7f90785f5690>,
u'Dac_Lac_province': <gensim.models.keyedvectors.Vocab object at 0x7f908d8eda10>,
u'Kenneth_Klinge': <gensim.models.keyedvectors.Vocab object at 0x7f9081563410>}
(c). index_to_word: count the words from the pre-trained model.
{ ... , u"Lina'la_Sin_Casino", u'fivemonth', u'retractable_roofs_Indians', u'Dac_Lac_province', u'Kenneth_Klinge'}
(d). word_to_index: give each word a index as following
{ ... , u'fivemonth': 2999996, u'Pidduck': 2999978, u'Dac_Lac_province': 2999998, u'Kenneth_Klinge': 2999999 }
(e). dataset is:
[ ... , [1837614, 1569052, 1837614, 1288695, 2221039, 2323218, 1837614, 1837614, 2029395, 1612781, 311032, 1524921, 1837614, 2973515, 2033866, 882731, 2462275, 2809106, 1479961, 826019, 73590, 953550, 1837614],
[1524921, 1113778, 1837614, 318169, 1837614, 1954969, 196613, 943118, 1837614, 2687790, 291413, 2774825, 2366038, 296869, 1468080, 856987, 1802099, 724308, 1207907, 2264894, 2206446, 812434],
[564298, 477983, 1837614, 1449153, 1837614, 211925, 2206446, 481834, 488597, 280760, 2072822, 1344872, 1791678, 2458776, 2965810, 2168205, 387112, 2656471, 1391, 1837614, 1801696, 2093846, 1210651],
[2493381, 133883, 2441902, 1014220, 1837614, 2597880, 1756105, 2651537, 1391, 2641114, 2517536, 1109601, 122269, 1782479, 2965835, 488597, 1767716, 753333, 564298, 2380935, 228060, 1837614, 371618],
[1837614, 1344872, 2458776, 2965810, 1837614, 2015408, 1837614, 528014, 1991322, 1837614, 908982, 1484130, 2349526, 988689, 753336, 1837614, 364492, 2183116, 826019, 73590, 953550, 1837614],
[1837614, 2673785, 1990947, 1219831, 2635341, 1247040, 1837614, 799543, 1990947, 1219831, 2722301, 1837614, 1427513, 969099, 2157673, 1430111, 1837614]]
(f). the variable during the training process:
# ('==>> sentences: ', PackedSequence(data=tensor(
# [ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# ('==>> padded_sentences: ', tensor(
# [[ 4.0230e+05, 1.8376e+06, 2.0185e+06, 1.8376e+06, 2.8157e+06,
# 1.8376e+06, 1.8376e+06, 1.0394e+06, 1.8376e+06, 2.9841e+06,
# 4.4713e+05, 1.1352e+06, 2.3532e+06, 1.8376e+06, 1.8376e+06,
# 1.8376e+06, 1.9550e+06, 3.8429e+04, 6.2537e+05, 2.3764e+05,
# 1.8376e+06, 1.5428e+06, 1.4214e+06]], device='cuda:0'))
# length: 23
# ('==>> embeds: ', tensor([[
# [-0.0684, 0.1826, -0.1777, ..., 0.1904, -0.1021, 0.1729],
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.1982, -0.0693, 0.1230, ..., -0.1357, -0.0306, 0.1104],
# ...,
# [ 0.0801, 0.1050, 0.0498, ..., 0.0037, 0.0476, -0.0688],
# [-0.0518, -0.0299, 0.0415, ..., 0.0776, -0.1660, 0.1602],
# [-0.0532, -0.0004, 0.0337, ..., -0.2373, -0.1709, 0.0233]]], device='cuda:0'))
# ('==>> packed_embeds: ', PackedSequence(data=tensor([
# [-0.3647, 0.2966, -0.2359, ..., -0.0000, 0.2657, -0.4302],
# [ 1.1699, 0.0000, 0.3312, ..., 0.5714, 0.1930, -0.2267],
# [-0.0627, -1.0548, 0.4966, ..., -0.5135, -0.0150, -0.0000],
# ...,
# [-0.6065, -0.7562, 0.3320, ..., -0.5854, -0.2089, -0.5737],
# [-0.0000, 0.4390, 0.0000, ..., 0.6891, 0.0250, -0.0000],
# [ 0.6909, -0.0000, -0.0000, ..., 0.1867, 0.0594, -0.2385]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])))
# packed_out_lstm1
# PackedSequence(data=tensor([
# [-0.0000, -0.0000, 0.1574, ..., 0.1864, -0.0901, -0.3205],
# [-0.0000, -0.3490, 0.1774, ..., 0.1677, -0.0000, -0.3688],
# [-0.3055, -0.0000, 0.2240, ..., 0.0000, -0.0927, -0.0000],
# ...,
# [-0.3188, -0.4134, 0.1339, ..., 0.3161, -0.0000, -0.3846],
# [-0.3355, -0.4365, 0.1575, ..., 0.2775, -0.0886, -0.4015],
# [-0.0000, -0.0000, 0.2452, ..., 0.1763, -0.0000, -0.2748]], device='cuda:0'),
# batch_sizes=tensor([ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]))
## self.hidden1
# self.hidden1
# (tensor([
# [[-0.0550, -0.4266, 0.0498, 0.0750, 0.3122, 0.0499, -0.2899,
# -0.0980, -0.1776, 0.4376, 0.1555, 0.7167, -0.0162, 0.0081,
# -0.3512, 0.0250, 0.0948, -0.3596, 0.0670, -0.6623, 0.0026,
# -0.0262, 0.1934, 0.3206, -0.1941, 0.3229, -0.6488, -0.3858,
# 0.3169, -0.0216, -0.1136, 0.1411, -0.0296, -0.2860, -0.1277,
# -0.0154, 0.2620, -0.3553, 0.0637, 0.8245, 0.6886, -0.0048,
# -0.1717, 0.2495, -0.8636, -0.0217, -0.2365, 0.1324, 0.1790,
# -0.1515, 0.3530, -0.1644, 0.0073, 0.6709, 0.1577, -0.0190,
# 0.0384, 0.4871, -0.7375, -0.1804, 0.3034, -0.3516, -0.2870,
# 0.6387, 0.0414, 0.6983, -0.3211, 0.0449, 0.2127, -0.0421,
# -0.3454, -0.8145, -0.3629, -0.0828, -0.1558, 0.4048, -0.4971,
# -0.7495, 0.0622, -0.3318, 0.3913, -0.0322, -0.0678, 0.0307,
# -0.0153, -0.1535, -0.2719, 0.0128, -0.1521, -0.2924, 0.7109,
# -0.8551, 0.0330, 0.0482, 0.2410, -0.0655, -0.2496, 0.1816,
# 0.4963, -0.7593, -0.0022, -0.1122, 0.6857, -0.5693, 0.5805,
# 0.7660, 0.4430, 0.0243, -0.0313, 0.0780, -0.2419, 0.0745,
# -0.0119, 0.5761, -0.0285, -0.1085, -0.1783, -0.0706, 0.1243,
# 0.6333, 0.0296, 0.5557, 0.2717, -0.0071, 0.0503, -0.0405,
# -0.4542, 0.8905, 0.4492, 0.8809, 0.7021, 0.8616, 0.2825,
# -0.2114, 0.3026, -0.1384, 0.1252, 0.4989, 0.2236, -0.5374,
# -0.1352, -0.0561, 0.0378, -0.5291, -0.1004, -0.3723, 0.0836,
# 0.3500, 0.0542, 0.2013]], # [[-0.0041, -0.0040, -0.1428, 0.2783, -0.1378, -0.4242, 0.1000,
# 0.1641, -0.0175, -0.0896, 0.3241, 0.3513, -0.4675, -0.1250,
# 0.0546, 0.2400, -0.0997, -0.5614, 0.2026, -0.1505, 0.0833,
# -0.3128, -0.4646, -0.0778, 0.2204, 0.5597, -0.7004, 0.0419,
# -0.3699, -0.5748, 0.7741, -0.7220, 0.0494, 0.3430, 0.1389,
# 0.0178, 0.0136, 0.0273, 0.1559, 0.4333, 0.2411, 0.0804,
# -0.0202, 0.6204, -0.4104, 0.5382, 0.1804, 0.0179, 0.1118,
# 0.3084, 0.1894, 0.0092, -0.2182, 0.0022, 0.4377, -0.0575,
# 0.0906, -0.0531, 0.0936, -0.1203, -0.0836, -0.1718, 0.2059,
# 0.7192, 0.0833, 0.3712, 0.2354, -0.1141, -0.0993, -0.0433,
# -0.1474, -0.0078, 0.0834, 0.0864, 0.9305, -0.3387, 0.0818,
# 0.3775, 0.1609, -0.0246, 0.2563, -0.1253, -0.0897, 0.0322,
# -0.0848, 0.0673, -0.1051, 0.0205, 0.0183, -0.0007, 0.1229,
# -0.3388, -0.0948, 0.0335, 0.0450, -0.2747, 0.2763, -0.2691,
# 0.6240, -0.0018, 0.2048, 0.3943, 0.2015, -0.5962, -0.0069,
# -0.3460, -0.7910, 0.3002, -0.4653, -0.0611, 0.6912, -0.8154,
# -0.0443, -0.0189, -0.1265, -0.1202, 0.0013, -0.5983, 0.0879,
# 0.0752, 0.8593, 0.0357, 0.0953, -0.0525, 0.2069, -0.6292,
# -0.0456, -0.7646, 0.0166, 0.0584, 0.0142, 0.0575, -0.1658,
# -0.4304, 0.3228, 0.4094, 0.0149, 0.1478, 0.1447, 0.4192,
# -0.0783, -0.0683, 0.0259, 0.0665, 0.6224, 0.3775, 0.0247,
# 0.1710, -0.3622, 0.0931]], # [[-0.2315, -0.2179, 0.1716, -0.6143, 0.4329, 0.2288, 0.1208,
# -0.3435, 0.6314, 0.0106, 0.1470, -0.0915, -0.3019, 0.1302,
# 0.2325, 0.1794, 0.0145, -0.6598, 0.0062, -0.0743, 0.0232,
# 0.9310, -0.8155, -0.0449, -0.5504, 0.5746, 0.3607, 0.4053,
# -0.1887, -0.5448, 0.1522, -0.0189, -0.4852, 0.6322, 0.0011,
# -0.1590, -0.0054, 0.2972, -0.0270, 0.0047, 0.2944, -0.0629,
# -0.1138, 0.1349, 0.5449, 0.8018, -0.0218, 0.0523, 0.3262,
# -0.0506, 0.2821, -0.0661, -0.7165, -0.3653, 0.3321, -0.0255,
# -0.0551, -0.1826, 0.6027, -0.1995, 0.0598, 0.0205, -0.1769,
# -0.0789, 0.4500, -0.1641, 0.5002, -0.0716, -0.3708, -0.0020,
# -0.5195, -0.0896, 0.1421, 0.1149, 0.3407, 0.2649, 0.0858,
# 0.2778, -0.3768, 0.6176, -0.2148, 0.5444, 0.3009, 0.4848,
# -0.1174, 0.0019, 0.6213, 0.2524, -0.0816, 0.4639, 0.4747,
# -0.7812, 0.2435, -0.0867, 0.1617, 0.2194, 0.0426, 0.1393,
# -0.0448, 0.0506, -0.5524, 0.0707, 0.2226, -0.0337, 0.7445,
# -0.4516, 0.1107, -0.2617, -0.1914, 0.7238, -0.2689, 0.0110,
# -0.3139, -0.0027, -0.5964, -0.9012, -0.4319, 0.0112, -0.0306,
# 0.4002, -0.1117, -0.1021, 0.1652, -0.2872, 0.3640, 0.2162,
# -0.3843, -0.0869, -0.1623, 0.0297, -0.0048, -0.0735, -0.0886,
# -0.4138, 0.2325, -0.4248, 0.3354, 0.0712, -0.4079, 0.0821,
# 0.1413, 0.2241, -0.1938, -0.0807, 0.3551, -0.0814, 0.1438,
# -0.6870, -0.3647, 0.0276]], # [[ 0.0258, 0.3281, -0.8145, -0.0476, -0.2886, -0.8013, 0.2135,
# 0.1541, 0.2069, 0.1345, -0.0171, -0.0228, -0.5237, 0.4917,
# 0.5187, 0.1402, 0.0928, -0.0373, 0.2698, 0.1259, 0.0021,
# -0.1624, -0.4100, 0.5377, -0.1013, -0.5658, -0.1015, 0.5609,
# 0.1661, 0.5731, 0.0012, -0.1766, 0.0743, -0.3630, 0.1082,
# 0.4643, 0.0175, -0.0260, 0.3810, -0.6425, -0.5515, 0.8800,
# -0.1158, -0.5741, 0.0463, 0.4033, 0.0803, 0.0403, 0.1159,
# 0.4471, 0.0294, 0.2899, 0.0248, -0.1772, 0.6600, -0.2252,
# -0.4896, -0.1285, -0.2377, -0.4179, -0.4056, -0.3224, -0.6855,
# -0.2703, 0.2971, 0.1259, -0.0456, -0.2495, 0.8141, 0.4453,
# 0.7480, -0.0578, 0.8023, -0.3586, -0.5229, 0.2299, 0.9668,
# -0.0717, 0.5355, -0.0743, 0.5246, 0.1604, -0.1464, -0.0757,
# 0.0414, -0.0861, 0.2245, 0.1247, 0.0676, -0.2053, 0.0113,
# 0.7875, -0.0308, 0.2025, 0.1289, -0.0020, -0.3099, 0.5317,
# -0.0117, 0.0928, -0.4100, -0.6184, 0.1171, 0.0216, -0.1266,
# 0.1640, 0.0821, -0.4097, -0.0691, 0.5805, 0.1692, -0.2021,
# 0.5971, 0.1172, -0.6535, -0.0579, 0.1177, 0.1123, -0.1943,
# 0.0488, -0.1305, -0.4859, -0.2758, -0.2972, -0.0605, -0.0029,
# -0.1508, 0.0375, -0.5942, -0.2139, -0.0335, -0.2320, -0.1152,
# -0.2054, -0.2643, -0.1770, 0.1245, 0.6334, -0.0363, 0.0264,
# -0.3348, -0.0434, -0.3794, -0.0913, 0.1293, -0.6537, 0.6490,
# 0.1305, -0.0631, -0.2243]]], device='cuda:0'),
# tensor([[[ -0.0775, -4.6346, 7.9077, 0.1164, 0.8626, 0.4240,
# -0.9286, -0.1612, -0.6049, 0.6771, 0.7443, 1.7457,
# -0.3930, 0.0112, -13.5393, 0.0317, 0.1236, -0.8475,
# 0.1212, -1.3623, 0.0117, -0.3297, 0.9009, 0.3840,
# -0.5885, 0.7411, -8.9613, -1.1402, 0.4511, -0.0753,
# -0.3107, 1.6518, -0.0870, -3.1360, -12.0200, -0.0464,
# 0.2756, -0.6695, 0.1604, 4.8299, 6.4623, -0.9555,
# -0.6904, 0.4469, -11.3343, -0.1669, -0.2747, 0.1590,
# 0.5829, -0.3345, 2.1731, -0.5636, 0.0207, 0.9874,
# 0.6291, -1.2261, 0.1946, 1.1287, -1.5759, -0.1875,
# 0.5550, -1.7350, -0.8235, 1.5122, 0.2019, 5.5143,
# -3.8153, 0.6771, 0.3011, -0.2994, -0.7320, -1.5857,
# -0.4785, -0.5584, -0.3226, 1.1932, -4.3901, -1.6923,
# 0.3526, -1.0625, 0.9279, -0.1843, -0.4376, 2.2389,
# -0.1558, -0.3959, -1.2987, 0.0279, -0.4938, -0.3364,
# 3.2596, -2.2647, 0.1448, 0.0726, 0.3968, -0.1885,
# -0.3960, 0.2141, 0.6785, -2.1622, -0.0043, -0.7516,
# 0.9367, -0.7092, 4.1853, 1.3348, 0.6993, 0.2043,
# -0.0916, 0.1392, -1.5672, 0.0867, -0.0346, 0.9226,
# -0.0470, -0.6870, -0.3002, -0.1131, 0.7785, 1.0582,
# 0.0914, 2.8785, 0.8164, -0.1048, 0.0573, -0.0499,
# -0.5990, 3.1714, 0.7925, 1.7461, 1.3243, 2.9236,
# 0.8966, -0.4455, 0.8763, -0.3036, 0.3302, 2.6581,
# 0.4608, -0.7280, -2.9457, -0.1973, 0.0585, -0.6555,
# -0.6621, -0.4549, 0.5812, 0.4495, 0.1350, 1.8521]], # [[ -0.0053, -0.0843, -0.1763, 0.3929, -0.1668, -0.6609,
# 0.1269, 0.2214, -0.0208, -0.3571, 0.7532, 0.7496,
# -4.9288, -0.5457, 0.3557, 0.4795, -0.2318, -0.9659,
# 0.6826, -1.6542, 0.4917, -0.3956, -1.5164, -0.2274,
# 0.6779, 1.1201, -3.1397, 0.0434, -0.4993, -0.8809,
# 6.1257, -5.6283, 0.4273, 1.5070, 0.6624, 0.1289,
# 0.2180, 0.9920, 0.1646, 0.8828, 0.5732, 0.3255,
# -0.0679, 0.9843, -1.8408, 1.0547, 0.1959, 0.0748,
# 0.1907, 0.4751, 0.3174, 0.0747, -0.6487, 0.0377,
# 0.5554, -0.4095, 0.2593, -0.0568, 0.3751, -0.3646,
# -0.2031, -0.3284, 0.4058, 1.2788, 0.1348, 1.8184,
# 0.8482, -0.7494, -0.2395, -0.4352, -0.1584, -0.0105,
# 0.2676, 0.3763, 2.1413, -1.3001, 0.3923, 1.6432,
# 0.2987, -1.2708, 5.5667, -0.1727, -0.5106, 0.5180,
# -6.7258, 0.5001, -0.3052, 0.0843, 0.0474, -0.2306,
# 0.1908, -2.2523, -1.5432, 0.2809, 0.7099, -0.4145,
# 0.7393, -0.6529, 0.7825, -0.0019, 1.3337, 2.2042,
# 9.2887, -0.8515, -0.0610, -0.6146, -1.5616, 0.3592,
# -1.3585, -0.2641, 1.4763, -3.2525, -0.5447, -0.0453,
# -1.0416, -2.4657, 0.0556, -0.7654, 0.2062, 0.0855,
# 3.0740, 0.0952, 0.4923, -0.1772, 0.9173, -4.2004,
# -0.1298, -2.4266, 0.0181, 0.5039, 0.0399, 7.9909,
# -0.5778, -2.9112, 0.4854, 1.2364, 0.0686, 0.6365,
# 0.1869, 0.6050, -0.1246, -0.1848, 0.5406, 0.2110,
# 1.2367, 1.9466, 0.0302, 0.2002, -0.5902, 0.1069]], # [[ -0.3559, -0.2859, 0.2699, -1.4359, 0.9814, 0.2811,
# 0.8539, -2.6654, 2.5455, 0.0434, 0.5947, -0.5325,
# -0.4638, 0.5487, 1.4376, 0.9863, 0.7429, -1.8308,
# 0.0402, -0.2282, 0.0366, 12.7877, -1.2491, -0.1437,
# -1.4960, 0.7364, 0.8599, 1.8343, -3.5117, -1.2758,
# 0.2930, -0.0472, -0.7527, 0.9555, 0.0446, -0.3389,
# -0.1985, 1.7953, -0.5702, 0.0141, 0.6166, -0.0924,
# -0.5182, 0.5146, 2.0801, 2.7460, -0.2606, 0.2090,
# 0.9266, -0.4758, 0.9961, -0.1723, -1.2069, -1.1735,
# 0.3683, -0.4933, -0.0604, -0.2354, 0.8239, -5.4226,
# 0.0854, 0.1185, -0.2656, -0.2689, 0.6047, -0.6246,
# 1.0131, -0.1673, -0.4990, -0.0690, -0.6092, -0.5205,
# 0.1808, 0.3061, 0.3924, 0.5868, 0.1452, 2.8930,
# -0.6085, 1.6086, -0.4763, 5.0389, 1.1569, 3.4060,
# -0.7565, 0.0247, 0.8477, 0.3714, -0.1043, 1.5607,
# 4.0700, -1.8363, 0.4370, -0.3571, 0.7268, 0.3435,
# 0.0972, 7.1477, -0.1486, 0.3342, -0.9733, 0.2311,
# 0.6104, -0.4988, 2.8838, -1.3387, 1.4291, -0.4121,
# -0.6722, 2.6834, -0.5188, 0.0428, -0.3452, -0.0131,
# -0.9004, -3.0346, -0.9254, 0.0150, -0.0386, 1.0639,
# -0.2444, -0.4562, 4.1626, -1.9304, 1.0662, 2.0683,
# -0.9553, -0.6434, -1.6777, 0.0702, -0.0113, -0.5503,
# -0.1873, -0.6916, 1.0729, -0.8234, 0.6421, 0.3022,
# -0.6065, 0.1016, 0.7792, 0.2533, -0.2670, -0.1314,
# 0.7515, -0.9859, 0.5050, -1.0552, -0.5632, 1.0697]], # [[ 0.3612, 2.4727, -4.6103, -1.6459, -1.7761, -1.4302,
# 0.2737, 0.3302, 4.0617, 0.7206, -0.0749, -0.8146,
# -1.0134, 0.8741, 1.9300, 0.5426, 0.1386, -1.3920,
# 0.4602, 1.3387, 0.0068, -0.3648, -7.6665, 0.9011,
# -0.3286, -1.5220, -0.2155, 0.7959, 4.0746, 0.9382,
# 0.0023, -0.2666, 0.4571, -1.9530, 0.3216, 2.1178,
# 0.4043, -0.0309, 2.5116, -1.2250, -0.9842, 5.0822,
# -4.1296, -5.3579, 0.9115, 0.4843, 0.1365, 0.0491,
# 0.1446, 0.6523, 0.0765, 0.3761, 0.0310, -0.4825,
# 7.1485, -0.4211, -3.7914, -0.2492, -0.3775, -0.4745,
# -1.3320, -1.8203, -1.0266, -0.4446, 2.2385, 0.6003,
# -0.1759, -1.9601, 2.3865, 1.3325, 4.8762, -0.2398,
# 7.5251, -0.4380, -2.3422, 0.6013, 13.8362, -0.4112,
# 2.3579, -0.1720, 1.0265, 0.6521, -0.7363, -0.7864,
# 0.2986, -0.1298, 0.5078, 0.1386, 1.4856, -0.3133,
# 0.9933, 1.5144, -0.0433, 1.0841, 0.3962, -0.0024,
# -0.3937, 2.2719, -0.0198, 1.6771, -1.2469, -0.8017,
# 0.1607, 0.0244, -0.1429, 0.9912, 0.1635, -1.2396,
# -0.1615, 1.0921, 0.8146, -0.3309, 0.8553, 0.4243,
# -3.5547, -0.1382, 0.1513, 0.4036, -0.2505, 0.2295,
# -0.6219, -0.7644, -0.7568, -0.4494, -0.0775, -0.0178,
# -0.2550, 0.2258, -2.7895, -0.3362, -0.2364, -0.9864,
# -1.3459, -1.8118, -0.4397, -0.8312, 0.3526, 1.3541,
# -0.0467, 1.6161, -0.4478, -0.5202, -0.4164, -0.8265,
# 0.1626, -4.2044, 3.2649, 0.2940, -0.8260, -0.4956]]], device='cuda:0'))
######## Padding functions used in pytorch. #########
1. torch.nn.utils.rnn.PackedSequence(*args)
Holds the data and list of batch_sizes of a packed sequence.
All RNN modules accept packed sequences as inputs.
2. torch.nn.utils.rnn.pack_padded_sequence(input, lengths, batch_first=False)
Packs a Tensor containing padded sequences of variable length.
Input can be of size T*B** where T is the length of the longest sequence, B is the batch size, and the * is any number of dimensions.
If batch_first is True B*T** inputs are expected. The sequences should be sorted by length in a decreasing order.
3. torch.nn.utils.rnn.pad_packed_sequence(sequence, batch_first=False, padding_values=0.0, total_length=None)
Pads a packed batch of variable length sequences.
It is an inverse operation to pack_padded_sequence().
Batch elements will be ordered decreasingly by their length.
Note: total_length is useful to implement the pack sequence -> rnn -> unpack sequence .
Return: Tuple of tensor containing the padded sequence, and a tensor containing the list of lengths of each sequence in the batch.
4. torch.nn.utils.rnn.pad_sequence(sequence, batch_first=False, padding_value=0)
Pad a list of variable length Tensors with zero.
5. torch.nn.utils.rnn.pack_sequence(sequences)
Packs a list of variable length Tensors.
6. Tutorials on these functions.
(1). https://zhuanlan.zhihu.com/p/34418001
(2). https://zhuanlan.zhihu.com/p/28472545
总结起来就是：在利用 recurrent neural network 处理变长的句子序列时，我们可以配套的使用：
torch.nn.utils.rnn.pack_padded_sequence () 来对一个 mini-batch 中的句子进行 padding；
torch.nn.utils.rnn.pad_packed_sequence () 来避免 padding 对句子表示的影响。
===
Tutorial on word2vector using GloVe and Word2Vec的更多相关文章
- 词向量(one-hot/SVD/NNLM/Word2Vec/GloVe)
目录 词向量简介 1. 基于one-hot编码的词向量方法 2. 统计语言模型 3. 从分布式表征到SVD分解 3.1 分布式表征(Distribution) 3.2 奇异值分解(SVD) 3.3 基 ...
- word2vec 和 glove 模型的区别
2019-09-09 15:36:13 问题描述:word2vec 和 glove 这两个生成 word embedding 的算法有什么区别. 问题求解: GloVe (global vectors ...
- [白话解析] 带你一起梳理Word2vec相关概念
[白话解析] 带你一起梳理Word2vec相关概念 0x00 摘要 本文将尽量使用易懂的方式,尽可能不涉及数学公式,而是从整体的思路上来说,运用感性直觉的思考来帮大家梳理Word2vec相关概念. 0 ...
- NLP︱高级词向量表达（一）——GloVe（理论、相关测评结果、R&python实现、相关应用）
有很多改进版的word2vec,但是目前还是word2vec最流行,但是Glove也有很多在提及,笔者在自己实验的时候,发现Glove也还是有很多优点以及可以深入研究对比的地方的,所以对其进行了一定的 ...
- Getting Started with Word2Vec
Getting Started with Word2Vec 1. Source by Google Project with Code: https://code.google.com/archive ...
- 四步理解GloVe！(附代码实现)
1. 说说GloVe 正如GloVe论文的标题而言,GloVe的全称叫Global Vectors for Word Representation,它是一个基于全局词频统计(count-based & ...
- 词嵌入之GloVe
什么是GloVe GloVe(Global Vectors for Word Representation)是一个基于全局词频统计(count-based & overall statisti ...
- RNN 与 LSTM 的应用
之前已经介绍过关于 Recurrent Neural Nnetwork 与 Long Short-Trem Memory 的网络结构与参数求解算法( 递归神经网络(Recurrent Neural N ...
- 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史（转载）
转载 https://zhuanlan.zhihu.com/p/49271699 首发于深度学习前沿笔记 写文章 从Word Embedding到Bert模型—自然语言处理中的预训练技术发展史 张 ...
随机推荐
- quick player no exit
QuickXDev插件自动升级后player no exist 昨晚上QuickXDev插件运行还ok,今天打开电脑启动sublime text2后,右键run with player提示player ...
- Mysql Federated For Windows
[1]windows环境下打开federated (1)关闭.命令:mysql> net stop mysql (2)添加federated字段.在my.ini文件中添加一个字段,注意位于[my ...
- QDialog 使用Demo
[1].pro QT += core gui greaterThan(QT_MAJOR_VERSION, ): QT += widgets TARGET = TestDialog TEMPLATE = ...
- redis的数据类型命令
存储sortedset: 存储:zadd key score menber1 score menber2 ... 升序排列:zrange key start end [withscores] 降序排列 ...
- rabbitmq和redis用作消息队列的区别
将redis发布订阅模式用做消息队列和rabbitmq的区别: 可靠性redis :没有相应的机制保证消息的可靠消费,如果发布者发布一条消息,而没有对应的订阅者的话,这条消息将丢失,不会存在内存中:r ...
- Linux基础命令---显示登录用户logname
logname 显示当前登录的用户名称.此命令的适用范围:RedHat.RHEL.Ubuntu.CentOS.SUSE.openSUSE.Fedora. 1.语法 logname 2 ...
- GUI常用对话框4
%普通对话框 dialog %单击时会关闭当前窗口 %自定义 关于对话框 点击确定 关闭 h = dialog( ]); uicontrol( ], ... 'string','确定','Callba ...
- What’s WOYO PDR-007 Paintless Dent Repair Heat Induction?
when you car body with dent, which tools can fixing this problem? is there a device which with the c ...
- 介绍python中运算符优先级
下面这个表给出Python的运算符优先级,从最低的优先级(最松散地结合)到最高的优先级(最紧密地结合).这意味着在一个表达式中,Python会首先计算表中较下面的运算符,然后在计算列在表上部的运算符. ...
- ogg 12.3 for sqlserver 2016 CDC模式配置
本文主要讲述ogg 12.3 通过CDC抽取sqlserver 2016 enterprise的过程,投递配置相对简单,所以不在此阐述. 配置步骤概述 1. 解压ogg 12.3 for sqlser ...