本章介绍tf基础知识，主要包括cookbook的第一、二章节。

方针：先会用，后定制

Ref: TensorFlow 如何入门？

Ref: 如何高效的学习 TensorFlow 代码?

顺便推荐该领域三件装备：

How TensorFlow Works?

Steps

  1. Import or generate datasets
  2. Transform and normalize data
  3. Partition datasets into train, test, and validation sets
  4. Set algorithm parameters (hyperparameters)
  5. Initialize variables and placeholders
  6. Define the model structure
  7. Declare the loss functions
  8. Initialize and train the model
  9. Evaluate the model
  10. Tune hyperparameters
  11. Deploy/predict new outcomes

Declaring Variables and Tensors

Tensor 张量 【相当于常数的设置】

import tensorflow as tf

# Fixed tensors 四种创建

zero_tsr = tf.zeros([2, 3])

zero_tsr

ones_tsr = tf.ones([2, 3])

ones_tsr

filled_tsr = tf.fill([2, 3], 42)

filled_tsr

constant_tsr = tf.constant([1,2,3])

constant_tsr

# Tensors of similar shape 拷贝创建

zeros_similar = tf.zeros_like(constant_tsr)

zeros_similar

ones_similar = tf.ones_like(constant_tsr)

ones_similar


# 对比，变量的拷贝式创建
w2 = tf.Variable(w1.initialized_value())　　

# Sequence tensors 线性创建
linear_tsr = tf.linspace(start=0.0, stop=1.0, num=3)

linear_tsr

integer_seq_tsr = tf.range(start=6, limit=15, delta=3)

integer_seq_tsr

# Random tensors

randunif_tsr = tf.random_uniform([2, 3], minval=0, maxval=1)

randunif_tsr

randnorm_tsr = tf.random_normal([2, 3], mean=0.0, stddev=1.0)

randnorm_tsr

runcnorm_tsr = tf.truncated_normal([2, 3], mean=0.0, stddev=1.0)

runcnorm_tsr


# 拷贝创建

input_tensor = runcnorm_tsr

shuffled_output = tf.random_shuffle(input_tensor)

shuffled_output

crop_size = 10

cropped_output = tf.random_crop(input_tensor, crop_size)

cropped_output

#cropped_image = tf.random_crop(my_image, [height/2, width/2, 3])

my_var = tf.Variable(tf.zeros([2, 3]))

my_var

#We can convert any numpy array to a Python list, or

#constant to a tensor using the function convert_to_tensor().

#convert_to_tensor()

打印/显示 tensor 的内容：通过sess将tensor动（运算）起来，形成流，之后便能打印出内容。

# How to print detail in tensor
# <Sol 1>
sess = tf.Session()

print(sess.run(zero_tsr))
close(sess)

# <Sol 2> 推荐!

with tf.Session():

    print(zero_tsr.eval())

Computational graph and Placeholder 

import tensorflow as tf

#定义‘符号’变量，也称为占位符

a = tf.placeholder("float")

b = tf.placeholder("float")

y = tf.mul(a, b) #构造一个op节点



sess = tf.Session()#建立会话

#运行会话，输入数据，并计算节点，同时打印结果

print(sess.run(y, feed_dict={a:3, b:3}))

# 任务完成, 关闭会话.

sess.close()

通俗易懂：http://blog.csdn.net/fei13971414170/article/details/73309106

在 TensorFlow 中，数据不是以整数，浮点数或者字符串形式存在的，而是被封装在一个叫做 tensor 的对象中。【tensor是个Object】

Tensor是张量的意思，张量包含了0到任意维度的量，其中：

    • 零维的叫作常数；
    • 一维的叫作向量；
    • 二维的叫作矩阵；
    • 多维度的就直接叫作张量。
# tensor1 是一个零维的 int32 tensor

tensor1 = tf.constant(1234) 

# tensor2 是一个一维的 int32 tensor

tensor2 = tf.constant([123,456,789]) 

 # tensor3 是一个二维的 int32 tensor

tensor3 = tf.constant([ [123,456,789], [222,333,444] ])

Placeholder占位符

使用feed_dict设置tensor的时候，需要你给出的值类型与占位符定义的类型相同。

x = tf.placeholder(tf.string)

y = tf.placeholder(tf.int32)

z = tf.placeholder(tf.float32)

with tf.Session() as sess:

    output = sess.run(x, feed_dict={x: 'Test String', y: 123, z: 45.67})

Using Placeholders and Variables

What's the difference between tf.placeholder and tf.Variable

In short, you use tf.Variable for trainable variables such as weights (W) and biases (B) for your model. 【权重和偏移，要训练的数据】

weights = tf.Variable(tf.truncated_normal([IMAGE_PIXELS, hidden1_units], stddev=1.0 / math.sqrt(float(IMAGE_PIXELS))), name='weights')

biases  = tf.Variable(tf.zeros([hidden1_units]), name='biases')

tf.placeholder is used to feed actual training examples. 【用于得到传递进来的真实的训练样本】

images_placeholder = tf.placeholder(tf.float32, shape=(batch_size, IMAGE_PIXELS))

labels_placeholder = tf.placeholder(tf.int32,   shape=(batch_size))

This is how you feed the training examples during the training:

for step in xrange(FLAGS.max_steps):

    feed_dict = {　　# 这么写，看上去友好些

       images_placeholder: images_feed,

       labels_placeholder: labels_feed,

     }

    _, loss_value = sess.run([train_op, loss], feed_dict=feed_dict)

Your tf.variables will be trained (modified) as the result of this training.

The difference is that with tf.Variable you have to provide an initial value when you declare it.

With tf.placeholder you don't have to provide an initial value and you can specify it at run time with the feed_dict argument inside Session.run .

Working with Matrices

矩阵计算

# Matrices and Matrix Operations

#----------------------------------

#

# This function introduces various ways to create

# matrices and how to use them in Tensorflow

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Declaring matrices

sess = tf.Session()

# Declaring matrices

# Identity matrix

identity_matrix = tf.diag([1.0,1.0,1.0])

print(sess.run(identity_matrix))

#[[ 1.  0.  0.]

# [ 0.  1.  0.]

# [ 0.  0.  1.]]

# 2x3 random norm matrix

A = tf.truncated_normal([2,3])

print(sess.run(A))

#[[  4.79249517e-04  -6.00280046e-01   1.36713833e-01]

# [ -1.25442386e+00   8.82814229e-02  -2.52978474e-01]]

# 2x3 constant matrix

B = tf.fill([2,3], 5.0)

print(sess.run(B))

#[[ 5.  5.  5.]

# [ 5.  5.  5.]]

# 3x2 random uniform matrix

C = tf.random_uniform([3,2])

print(sess.run(C))

#[[ 0.07532465  0.23328125]

# [ 0.21761775  0.35856724]

# [ 0.88200712  0.27035964]]

print(sess.run(C)) # Note that we are reinitializing, hence the new random variabels

# Create matrix from np array [python lib]

D = tf.convert_to_tensor(np.array([[1., 2., 3.], [-3., -7., -1.], [0., 5., -2.]]))

print(sess.run(D))

#[[ 1.  2.  3.]

# [-3. -7. -1.]

# [ 0.  5. -2.]]

# Matrix addition/subtraction

print(sess.run(A+B))

#[[ 6.08604956  4.14716625  4.46166086]

# [ 3.24823093  4.94008398  5.14025211]]

print(sess.run(B-B))

#[[ 0.  0.  0.]

# [ 0.  0.  0.]]

# Matrix Multiplication

print(sess.run(tf.matmul(B, identity_matrix)))

#[[ 5.  5.  5.]

# [ 5.  5.  5.]]

# Matrix Transpose 因为C是随即变量，故sess一次，就会重新随机，故，这里选择D做实验会好些

print(sess.run(tf.transpose(C))) # Again, new random variables

# Matrix Determinant 方阵的行列式

print(sess.run(tf.matrix_determinant(D)))

# Matrix Inverse

print(sess.run(tf.matrix_inverse(D)))

# Cholesky Decomposition　-->

print(sess.run(tf.cholesky(identity_matrix)))

# Eigenvalues and Eigenvectors 特征值 and 特征向量

print(sess.run(tf.self_adjoint_eig(D)))
#( array([-10.65907521, -0.22750691, 2.88658212]), 
#  array([[ 0.21749542, 0.63250104, -0.74339638],
#         [ 0.84526515, 0.2587998 ,  0.46749277],
#         [-0.4880805 , 0.73004459,  0.47834331]]) )

补充：Cholesky Decomposition

将一个正定Hermite矩阵分解成为一个下三角阵和它的共轭转置阵的乘积。
如果矩阵A是正定Hermite阵，那么矩阵A可以做如下分解：

其中L是一个下三角矩阵且主对角线元素严格正定，L*是L的共轭转置矩阵。

Declaring Operations

矩阵计算

# Operations

#----------------------------------

#

# This function introduces various operations

# in Tensorflow

# Declaring Operations

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Open graph session

sess = tf.Session()

# div() vs truediv() vs floordiv()

print(sess.run(tf.div(3,4)))

#0

print(sess.run(tf.truediv(3,4)))

#0.75

print(sess.run(tf.floordiv(3.0,4.0)))

#0.0

# Mod function

print(sess.run(tf.mod(22.0,5.0)))

#2.0

# Cross Product

print(sess.run(tf.cross([1.,0.,0.],[0.,1.,0.])))

#[ 0.  0.  1.]


# Trig functions

print(sess.run(tf.sin(3.1416)))

#-7.23998e-06

print(sess.run(tf.cos(3.1416)))

#-1.0

# Tangemt

print(sess.run(tf.div(tf.sin(3.1416/4.), tf.cos(3.1416/4.))))

#1.0

# Custom operation

test_nums = range(15)

#from tensorflow.python.ops import math_ops

#print(sess.run(tf.equal(test_num, 3)))

def custom_polynomial(x_val):

    # Return 3x^2 - x + 10

    return(tf.sub(3 * tf.square(x_val), x_val) + 10)

print(sess.run(custom_polynomial(11)))

# What should we get with list comprehension

expected_output = [3*x*x-x+10 for x in test_nums]

print(expected_output)

# Tensorflow custom function output

for num in test_nums:

    print(sess.run(custom_polynomial(num)))

这为之后激活函数的实现打下了基础。

Implementing Activation Functions

# Activation Functions

#----------------------------------

#

# This function introduces activation

# functions in Tensorflow

# Implementing Activation Functions

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Open graph session

sess = tf.Session()

# X range

x_vals = np.linspace(start=-10., stop=10., num=100)

x_vals

# ReLU activation

print(sess.run(tf.nn.relu([-3., 3., 10.])))

y_relu = sess.run(tf.nn.relu(x_vals))

#[  0.   3.  10.]

# ReLU-6 activation

# This is defined as min(max(0,x),6).

# This will come in handy when we

# discuss deeper neural networks in Chapters 8, Convolutional Neural Networks and

# Chapter 9, Recurrent Neural Networks.

print(sess.run(tf.nn.relu6([-3., 3., 10.])))

y_relu6 = sess.run(tf.nn.relu6(x_vals))

# Sigmoid activation

print(sess.run(tf.nn.sigmoid([-1., 0., 1.])))

y_sigmoid = sess.run(tf.nn.sigmoid(x_vals))

# Hyper Tangent activation

print(sess.run(tf.nn.tanh([-1., 0., 1.])))

y_tanh = sess.run(tf.nn.tanh(x_vals))

# Softsign activation

print(sess.run(tf.nn.softsign([-1., 0., 1.])))

y_softsign = sess.run(tf.nn.softsign(x_vals))

# Softplus activation

print(sess.run(tf.nn.softplus([-1., 0., 1.])))

y_softplus = sess.run(tf.nn.softplus(x_vals))

# Exponential linear activation

print(sess.run(tf.nn.elu([-1., 0., 1.])))

y_elu = sess.run(tf.nn.elu(x_vals))

# Plot the different functions

plt.plot(x_vals, y_softplus, 'r--', label='Softplus', linewidth=2)

plt.plot(x_vals, y_relu,     'b:',  label='ReLU',     linewidth=2)

plt.plot(x_vals, y_relu6,    'g-.', label='ReLU6',    linewidth=2)

plt.plot(x_vals, y_elu,      'k-',  label='ExpLU',    linewidth=0.5)

plt.ylim([-1.5,7])

plt.legend(loc='top left')

plt.show()

plt.plot(x_vals, y_sigmoid,  'r--', label='Sigmoid',  linewidth=2)

plt.plot(x_vals, y_tanh,     'b:',  label='Tanh',     linewidth=2)

plt.plot(x_vals, y_softsign, 'g-.', label='Softsign', linewidth=2)

plt.ylim([-2,2])

plt.legend(loc='top left')

plt.show()

Result:  

Working with Data Sources

# Data gathering

#----------------------------------

#

# This function gives us the ways to access

# the various data sets we will need

# Data Gathering

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Iris Data

from sklearn import datasets

iris = datasets.load_iris()

print(len(iris.data))

print(len(iris.target))

print(iris.data[0])

print(set(iris.target))

# Low Birthrate Data

import requests

birthdata_url = 'https://www.umass.edu/statdata/statdata/data/lowbwt.dat'

birth_file = requests.get(birthdata_url)

birth_data = birth_file.text.split('\r\n')[5:]

birth_header = [x for x in birth_data[0].split(' ') if len(x)>=1]

birth_data = [[float(x) for x in y.split(' ') if len(x)>=1] for y in birth_data[1:] if len(y)>=1]

print(len(birth_data))

print(len(birth_data[0]))

# Housing Price Data

import requests

housing_url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data'

housing_header = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV']

housing_file = requests.get(housing_url)

housing_data = [[float(x) for x in y.split(' ') if len(x)>=1] for y in housing_file.text.split('\n') if len(y)>=1]

print(len(housing_data))

print(len(housing_data[0]))

# MNIST Handwriting Data

from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)　　# 本地加载

print(len(mnist.train.images))

print(len(mnist.test.images))

print(len(mnist.validation.images))

print(mnist.train.labels[1,:])

# Ham/Spam Text Data

import requests

import io

from zipfile import ZipFile

# Get/read zip file

zip_url = 'http://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip'

r = requests.get(zip_url)

z = ZipFile(io.BytesIO(r.content))

file = z.read('SMSSpamCollection')

# Format Data

text_data = file.decode()

text_data = text_data.encode('ascii',errors='ignore')

text_data = text_data.decode().split('\n')

text_data = [x.split('\t') for x in text_data if len(x)>=1]

[text_data_target, text_data_train] = [list(x) for x in zip(*text_data)]

print(len(text_data_train))

print(set(text_data_target))

print(text_data_train[1])

# Movie Review Data

import requests

import io

import tarfile

movie_data_url = 'http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz'

r = requests.get(movie_data_url)

# Stream data into temp object

stream_data = io.BytesIO(r.content)

tmp = io.BytesIO()

while True:

    s = stream_data.read(16384)

    if not s:

        break

    tmp.write(s)

stream_data.close()

tmp.seek(0)

# Extract tar file

tar_file = tarfile.open(fileobj=tmp, mode="r:gz")

pos = tar_file.extractfile('rt-polaritydata/rt-polarity.pos')

neg = tar_file.extractfile('rt-polaritydata/rt-polarity.neg')

# Save pos/neg reviews

pos_data = []

for line in pos:

    pos_data.append(line.decode('ISO-8859-1').encode('ascii',errors='ignore').decode())

neg_data = []

for line in neg:

    neg_data.append(line.decode('ISO-8859-1').encode('ascii',errors='ignore').decode())

tar_file.close()

print(len(pos_data))

print(len(neg_data))

print(neg_data[0])

# The Works of Shakespeare Data

import requests

shakespeare_url = 'http://www.gutenberg.org/cache/epub/100/pg100.txt'

# Get Shakespeare text

response = requests.get(shakespeare_url)

shakespeare_file = response.content

# Decode binary into string

shakespeare_text = shakespeare_file.decode('utf-8')

# Drop first few descriptive paragraphs.

shakespeare_text = shakespeare_text[7675:]

print(len(shakespeare_text))

# English-German Sentence Translation Data

import requests

import io

from zipfile import ZipFile

sentence_url = 'http://www.manythings.org/anki/deu-eng.zip'

r = requests.get(sentence_url)

z = ZipFile(io.BytesIO(r.content))

file = z.read('deu.txt')

# Format Data

eng_ger_data = file.decode()

eng_ger_data = eng_ger_data.encode('ascii',errors='ignore')

eng_ger_data = eng_ger_data.decode().split('\n')

eng_ger_data = [x.split('\t') for x in eng_ger_data if len(x)>=1]

[english_sentence, german_sentence] = [list(x) for x in zip(*eng_ger_data)]

print(len(english_sentence))

print(len(german_sentence))

print(eng_ger_data[10])

Outlines

  • Operations in a Computational Graph
  • Layering Nested Operations
  • Working with Multiple Layers
  • Implementing Loss Functions
  • Implementing Back Propagation
  • Working with Batch and Stochastic Training
  • Combining Everything Together
  • Evaluating Models

这里通过简单的线性回归来做一些基本的算法练习。

基本的"计算图"构建

Operations in a Computational Graph

# Operations on a Computational Graph

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Create graph

sess = tf.Session()

# Create tensors

# Create data to feed in

x_vals = np.array([1., 3., 5., 7., 9.])

# 计算图的构造

x_data = tf.placeholder(tf.float32)　　# 变量：输入不能确定

m      = tf.constant(3.)　　           # 常量

prod   = tf.mul(x_data, m)　　         # Operation


# 喂数据

for x_val in x_vals:

    print(sess.run(prod, feed_dict={x_data: x_val}))　　# 往 x_data（变量）里面喂x_val（输入数据）

merged = tf.merge_all_summaries()

my_writer = tf.train.SummaryWriter('/home/unsw/Programmer/1-python/Tensorflow/TF/TensreFlowMachineLearningCookbook_Code/myTest', sess.graph)

所有可用的 summary 操作详细信息，可以查看summary_operation文档。

在TensorFlow中，所有的操作只有当你执行，或者另一个操作依赖于它的输出时才会运行。我们刚才创建的这些节点（summary nodes）都围绕着你的图像：没有任何操作依赖于它们的结果。因此，为了生成汇总信息，我们需要运行所有这些节点。这样的手动工作是很乏味的，因此可以使用tf.merge_all_summaries来将他们合并为一个操作。

然后你可以执行合并命令，它会依据特点步骤将所有数据生成一个序列化的Summary protobuf对象。最后，为了将汇总数据写入磁盘，需要将汇总的protobuf对象传递给tf.train.Summarywriter。

SummaryWriter 的构造函数中包含了参数 logdir。这个 logdir 非常重要，所有事件都会写到它所指的目录下。此外，SummaryWriter 中还包含了一个可选择的参数 GraphDef。如果输入了该参数，那么 TensorBoard 也会显示你的图像。

Layering Nested Operations

# Layering Nested Operations

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Create graph

sess = tf.Session()

# Create tensors

# Create data to feed in

my_array = np.array([[1., 3., 5., 7., 9.],

                    [-2., 0., 2., 4., 6.],

                    [-6., -3., 0., 3., 6.]])

x_vals = np.array([my_array, my_array + 1])

x_data = tf.placeholder(tf.float32, shape=(3, 5))

m1 = tf.constant([[1.],[0.],[-1.],[2.],[4.]])

sess.run(m1)

m2 = tf.constant([[2.]])

sess.run(m2)

a1 = tf.constant([[10.]])

sess.run(a1)

# 1st Operation Layer = Multiplication

prod1 = tf.matmul(x_data, m1)

# 2nd Operation Layer = Multiplication

prod2 = tf.matmul(prod1, m2)

# 3rd Operation Layer = Addition

add1 = tf.add(prod2, a1)

for x_val in x_vals:

    print(sess.run(add1, feed_dict={x_data: x_val}))

merged = tf.merge_all_summaries()

my_writer = tf.train.SummaryWriter('/home/nick/OneDrive/Documents/tensor_flow_book/Code/2_Tensorflow_Way', sess.graph)

Working with Multiple Layers

Figure, 简单的卷积网络

# Layering Nested Operations

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Create graph

sess = tf.Session()

# Create tensors

# Create a small random 'image' of size 4x4

x_shape = [1, 4, 4, 1]

x_val   = np.random.uniform(size=x_shape)

x_data  = tf.placeholder(tf.float32, shape=x_shape)

# 【x_data <--feed-- x_val】

# 卷积

my_filter    = tf.constant(0.25, shape=[2, 2, 1, 1])

my_strides   = [1, 2, 2, 1]

mov_avg_layer= tf.nn.conv2d(x_data, my_filter, my_strides, padding='SAME', name='Moving_Avg_Window')

# Define a custom layer which will be sigmoid(Ax+b) where

# x is a 2x2 matrix and A and b are 2x2 matrices

def custom_layer(input_matrix):

    input_matrix_sqeezed = tf.squeeze(input_matrix)

    A     = tf.constant([[1., 2.], [-1., 3.]])    # Const

    b     = tf.constant(1., shape=[2, 2])         # Const_1
    # 以上：节点输入；以下，节点计算
    temp1 = tf.matmul(A, input_matrix_sqeezed)    # [MatMul]

    temp  = tf.add(temp1, b) # Ax + b             # [Add]

    return(tf.sigmoid(temp))                      # [Sigmoid]

# Add custom layer to graph

with tf.name_scope('Custom_Layer') as scope:

    custom_layer1 = custom_layer(mov_avg_layer)   # <-- 卷积的结果给于自定义层来处理

# The output should be an array that is 2x2, but size (1,2,2,1)

#print(sess.run(mov_avg_layer, feed_dict={x_data: x_val}))

# After custom operation, size is now 2x2 (squeezed out size 1 dims)

print(sess.run(custom_layer1, feed_dict={x_data: x_val}))

merged = tf.merge_all_summaries()

my_writer = tf.train.SummaryWriter('/home/unsw/Programmer/1-python/Tensorflow/TF/TensreFlowMachineLearningCookbook_Code/myTest', sess.graph)

tf.squeeze()

给定张量输入，此操作返回相同类型的张量，并删除所有尺寸为1的尺寸。

如果不想删除所有大小是1的维度，可以通过squeeze_dims指定。

# 't' is a tensor of shape [1, 2, 1, 3, 1, 1]

shape(squeeze(t))  # => [2, 3]

# 't' is a tensor of shape [1, 2, 1, 3, 1, 1]

shape(squeeze(t, [2, 4]))  # => [1, 2, 3, 1]　　指定了删除的‘1’.

损失函数

Implementing Loss Functions 

看样子就是scikit一套的接口【之前的激活函数和这里的损失函数需单独总结数学原理】

# Loss Functions

#----------------------------------

#

#  This python script illustrates the different

#  loss functions for regression and classification.

import matplotlib.pyplot as plt

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Create graph

sess = tf.Session()

###### Numerical Predictions ######

x_vals = tf.linspace(-1., 1., 500)

target = tf.constant(0.)

# L2 loss

# L = (pred - actual)^2

l2_y_vals = tf.square(target - x_vals)

l2_y_out  = sess.run(l2_y_vals)

# L1 loss

# L = abs(pred - actual)

l1_y_vals = tf.abs(target - x_vals)

l1_y_out  = sess.run(l1_y_vals)

# Pseudo-Huber loss

# L = delta^2 * (sqrt(1 + ((pred - actual)/delta)^2) - 1)

delta1 = tf.constant(0.25)

phuber1_y_vals = tf.mul(tf.square(delta1), tf.sqrt(1. + tf.square((target - x_vals)/delta1)) - 1.)

phuber1_y_out  = sess.run(phuber1_y_vals)

delta2 = tf.constant(5.)

phuber2_y_vals = tf.mul(tf.square(delta2), tf.sqrt(1. + tf.square((target - x_vals)/delta2)) - 1.)

phuber2_y_out = sess.run(phuber2_y_vals)

# Plot the output:

x_array = sess.run(x_vals)　　# <-- 横轴代表x_vals的取值

plt.plot(x_array, l2_y_out,      'b-',  label='L2 Loss')

plt.plot(x_array, l1_y_out,      'r--', label='L1 Loss')

plt.plot(x_array, phuber1_y_out, 'k-.', label='P-Huber Loss (0.25)')

plt.plot(x_array, phuber2_y_out, 'g:',  label='P-Huber Loss (5.0)')

plt.ylim(-0.2, 0.4)

plt.legend(loc='lower right', prop={'size': 11})

plt.show()


###############################################################################################

###### Categorical Predictions ######

x_vals  = tf.linspace(-3., 5., 500)

target  = tf.constant(1.)

targets = tf.fill([500,], 1.)

# Hinge loss

# Use for predicting binary (-1, 1) classes

# L = max(0, 1 - (pred * actual))

hinge_y_vals = tf.maximum(0., 1. - tf.mul(target, x_vals))

hinge_y_out  = sess.run(hinge_y_vals)

# Cross entropy loss

# L = -actual * (log(pred)) - (1-actual)(log(1-pred))

xentropy_y_vals = - tf.mul(target, tf.log(x_vals)) - tf.mul((1. - target), tf.log(1. - x_vals))

xentropy_y_out  = sess.run(xentropy_y_vals)

# Sigmoid entropy loss

# L = -actual * (log(sigmoid(pred))) - (1-actual)(log(1-sigmoid(pred)))

# or

# L = max(actual, 0) - actual * pred + log(1 + exp(-abs(actual)))

xentropy_sigmoid_y_vals = tf.nn.sigmoid_cross_entropy_with_logits(x_vals, targets)

xentropy_sigmoid_y_out  = sess.run(xentropy_sigmoid_y_vals)

# Weighted (softmax) cross entropy loss

# L = -actual * (log(pred)) * weights - (1-actual)(log(1-pred))

# or

# L = (1 - pred) * actual + (1 + (weights - 1) * pred) * log(1 + exp(-actual))

weight = tf.constant(0.5)

xentropy_weighted_y_vals = tf.nn.weighted_cross_entropy_with_logits(x_vals, targets, weight)

xentropy_weighted_y_out = sess.run(xentropy_weighted_y_vals)

# Plot the output

x_array = sess.run(x_vals)

plt.plot(x_array, hinge_y_out, 'b-', label='Hinge Loss')

plt.plot(x_array, xentropy_y_out, 'r--', label='Cross Entropy Loss')

plt.plot(x_array, xentropy_sigmoid_y_out, 'k-.', label='Cross Entropy Sigmoid Loss')

plt.plot(x_array, xentropy_weighted_y_out, 'g:', label='Weighted Cross Entropy Loss (x0.5)')

plt.ylim(-1.5, 3)

#plt.xlim(-1, 3)

plt.legend(loc='lower right', prop={'size': 11})

plt.show()

# Softmax entropy loss

# L = -actual * (log(softmax(pred))) - (1-actual)(log(1-softmax(pred)))

unscaled_logits = tf.constant([[1., -3., 10.]])

target_dist = tf.constant([[0.1, 0.02, 0.88]])

softmax_xentropy = tf.nn.softmax_cross_entropy_with_logits(unscaled_logits, target_dist)

print(sess.run(softmax_xentropy))

# Sparse entropy loss

# Use when classes and targets have to be mutually exclusive

# L = sum( -actual * log(pred) )

unscaled_logits = tf.constant([[1., -3., 10.]])

sparse_target_dist = tf.constant([2])

sparse_xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(unscaled_logits, sparse_target_dist)

print(sess.run(sparse_xentropy)) 

Result: 

Loss function Use Benefits Disadvantages
L2 Regression More stable  Less robust
L1 Regression More robust Less stable
Psuedo-Huber Regression More robust and stable One more parameter
Hinge Classification Creates a max margin for use in SVM Unbounded loss affected by outliers
Cross-entropy Classification More stable Unbounded loss, less robust

Implementing Back Propagation

  • Online training - 非batch

# Back Propagation

#----------------------------------

#

# This python function shows how to implement back propagation

# in regression and classification models.

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Create graph

sess = tf.Session()

# Regression Example:

# We will create sample data as follows:

# x-data: 100 random samples from a normal ~ N(1, 0.1)

# target: 100 values of the value 10.

# We will fit the model:

# x-data * A = target

# Theoretically, A = 10.

# Create data

x_vals   = np.random.normal(1, 0.1, 100)

y_vals   = np.repeat(10., 100)

x_data   = tf.placeholder(shape=[1], dtype=tf.float32)

y_target = tf.placeholder(shape=[1], dtype=tf.float32)

# Create variable (one model parameter = A)

A = tf.Variable(tf.random_normal(shape=[1]))

# Add operation to graph

my_output = tf.mul(x_data, A)

# Add L2 loss operation to graph

loss = tf.square(my_output - y_target)

# Initialize variables

init = tf.initialize_all_variables()

sess.run(init)

# Create Optimizer (solver)

my_opt = tf.train.GradientDescentOptimizer(0.02)

train_step = my_opt.minimize(loss)

# Run Loop: 100 times

for i in range(100):

    rand_index = np.random.choice(100)

    rand_x = [ x_vals[rand_index] ]

    rand_y = [ y_vals[rand_index] ]

    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})

    if (i+1)%25==0:

        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))

        print('Loss = ' + str(sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})))　　<-- loss查看

Result:

Step #200 A = [ 3.09365487]

Loss = [[ 0.00431256]]

Step #400 A = [ 0.26573253]

Loss = [[ 0.0705369]]

Step #600 A = [-0.67981237]

Loss = [[ 0.03407416]]

Step #800 A = [-0.81504095]

Loss = [[ 0.17166217]]

Step #1000 A = [-1.02729309]

Loss = [[ 0.25573865]]

Step #1200 A = [-0.96181494]

Loss = [[ 0.04576259]]

Step #1400 A = [-1.02739477]

Loss = [[ 0.05697485]]


Ending Accuracy = 0.98

  • batch training的batch index

以下示例，主要演示了增加一个维度在数据，专门用于作为batch idx。【expand_dims(...)】

至于batch training的具体内容，将在下一个环节讲解。

# Classification Example

# We will create sample data as follows:

# x-data: sample 50 random values from a normal = N(-1, 1)

#         + sample 50 random values from a normal = N(1, 1)

# target: 50 values of 0 + 50 values of 1.

#         These are essentially 100 values of the corresponding output index

# We will fit the binary classification model:

# If sigmoid(x+A) < 0.5 -> 0 else 1

# Theoretically, A should be -(mean1 + mean2)/2

ops.reset_default_graph()

# Create graph

sess = tf.Session()

# Create data

x_vals   = np.concatenate((np.random.normal(-1, 1, 50), np.random.normal(3, 1, 50)))

y_vals   = np.concatenate((np.repeat(0., 50), np.repeat(1., 50)))

x_data   = tf.placeholder(shape=[1], dtype=tf.float32)

y_target = tf.placeholder(shape=[1], dtype=tf.float32)

# Create variable (one model parameter = A)

A = tf.Variable(tf.random_normal(mean=10, shape=[1]))

# Add operation to graph

# Want to create the operstion sigmoid(x + A)

# Note, the sigmoid() part is in the loss function

my_output = tf.add(x_data, A)

【这里就一个简单的加法运算】

# Now we have to add another dimension to each (batch size of 1)

my_output_expanded = tf.expand_dims(my_output, 0)

y_target_expanded  = tf.expand_dims(y_target,  0)

# Initialize variables

init = tf.initialize_all_variables()

sess.run(init)

# Add classification loss (cross entropy)

xentropy = tf.nn.sigmoid_cross_entropy_with_logits(my_output_expanded, y_target_expanded)

# Create Optimizer

my_opt = tf.train.GradientDescentOptimizer(0.05)

train_step = my_opt.minimize(xentropy)

# Run loop

for i in range(1400):

    rand_index = np.random.choice(100)

    rand_x = [x_vals[rand_index]]

    rand_y = [y_vals[rand_index]]

    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})

    if (i+1)%200==0:

        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))

        print('Loss = ' + str(sess.run(xentropy, feed_dict={x_data: rand_x, y_target: rand_y})))

# Evaluate Predictions

predictions = []

for i in range(len(x_vals)):

    x_val = [x_vals[i]]

    prediction = sess.run(tf.round(tf.sigmoid(my_output)), feed_dict={x_data: x_val})　　# my_output是实际输出值，sigmoid化后再四舍五入

    predictions.append(prediction[0])

accuracy = sum(x==y for x,y in zip(predictions, y_vals))/100.　　# 实际输出值 与 预计值 比较，求正确率

print('Ending Accuracy = ' + str(np.round(accuracy, 2)))

Working with Batch and Stochastic Training

根据 np.random.choice(100)，本来就是Stochastic training or Online training.

【这个例子是不错的呦】

# Batch and Stochastic Training

#----------------------------------

#

#  This python function illustrates two different training methods:

#  batch and stochastic training.  For each model, we will use

#  a regression model that predicts one model variable.

import matplotlib.pyplot as plt

import numpy as np

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# We will implement a regression example in stochastic and batch training

# Stochastic Training:

# Create graph

sess = tf.Session()

# Create data

x_vals   = np.random.normal(1, 0.1, 100)

y_vals   = np.repeat(10., 100)

x_data   = tf.placeholder(shape=[1], dtype=tf.float32)

y_target = tf.placeholder(shape=[1], dtype=tf.float32)

# Create variable (one model parameter = A)

A = tf.Variable(tf.random_normal(shape=[1]))

# Add operation to graph

my_output = tf.mul(x_data, A)

# Add L2 loss operation to graph

loss = tf.square(my_output - y_target)

# Initialize variables

init = tf.initialize_all_variables()

sess.run(init)

# Create Optimizer

my_opt = tf.train.GradientDescentOptimizer(0.02)

train_step = my_opt.minimize(loss)

loss_stochastic = []

# Run Loop

for i in range(100):

    rand_index = np.random.choice(100)

    rand_x = [x_vals[rand_index]]

    rand_y = [y_vals[rand_index]]

    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})

    if (i+1)%5==0:

        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))

        temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})

        print('Loss = ' + str(temp_loss))

        loss_stochastic.append(temp_loss)
# Batch Training:

# Re-initialize graph

ops.reset_default_graph()

sess = tf.Session()

# Declare batch size

batch_size = 20

# Create data

x_vals   = np.random.normal(1, 0.1, 100)

y_vals   = np.repeat(10., 100)

x_data   = tf.placeholder(shape=[None, 1], dtype=tf.float32)

y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)

# Create variable (one model parameter = A)

A = tf.Variable(tf.random_normal(shape=[1,1]))

# Add operation to graph

my_output = tf.matmul(x_data, A)

# Add L2 loss operation to graph 因为是回归问题

loss = tf.reduce_mean(tf.square(my_output - y_target))　　# 因为是batch training

# Initialize variables

init = tf.initialize_all_variables()

sess.run(init)

# Create Optimizer

my_opt = tf.train.GradientDescentOptimizer(0.02)

train_step = my_opt.minimize(loss)

loss_batch = []

# Run Loop

for i in range(100):　　# NB： 这是不是标准的minnbatch and epoch方式

    rand_index = np.random.choice(100, size=batch_size)

    rand_x = np.transpose([x_vals[rand_index]])

    rand_y = np.transpose([y_vals[rand_index]])

    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})

    if (i+1)%5==0:

        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)))

        temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})　　<-- loss查看print('Loss = ' + str(temp_loss))

        loss_batch.append(temp_loss)

plt.plot(range(0, 100, 5), loss_stochastic, 'b-', label='Stochastic Loss')

plt.plot(range(0, 100, 5), loss_batch, 'r--', label='Batch Loss, size=20')

plt.legend(loc='upper right', prop={'size': 11})

plt.show() 

Result: 

Binary classifier for Iris Dataset

# Combining Everything Together

#----------------------------------

# This file will perform binary classification on the

# class if iris dataset. We will only predict if a flower is

# I.setosa or not.

#

# We will create a simple binary classifier by creating a line

# and running everything through a sigmoid to get a binary predictor.

# The two features we will use are pedal length and pedal width.

#

# We will use batch training, but this can be easily

# adapted to stochastic training.

import matplotlib.pyplot as plt

import numpy as np

from sklearn import datasets

import tensorflow as tf

from tensorflow.python.framework import ops

ops.reset_default_graph()

# Load the iris data

# iris.target = {0, 1, 2}, where '0' is setosa

# iris.data ~ [sepal.width, sepal.length, pedal.width, pedal.length]

iris = datasets.load_iris()

binary_target = np.array([1. if x==0 else 0. for x in iris.target])

iris_2d = np.array([[x[2], x[3]] for x in iris.data])   # Jeff: only consider two features.

# Declare batch size

batch_size = 20

# Create graph

sess = tf.Session()

Init

Construct graph:

# Declare placeholders 三个feed_dict的入口

x1_data  = tf.placeholder(shape=[None, 1], dtype=tf.float32)

x2_data  = tf.placeholder(shape=[None, 1], dtype=tf.float32)

y_target = tf.placeholder(shape=[None, 1], dtype=tf.float32)

# Create variables A and b (0 = x1 - A*x2 + b)

A = tf.Variable(tf.random_normal(shape=[1, 1]))

b = tf.Variable(tf.random_normal(shape=[1, 1]))

###################################

# Add model to graph:

# x1 - A*x2 + b

###################################

my_mult   = tf.matmul(x2_data, A)

my_add    = tf.add(my_mult, b)

my_output = tf.sub(x1_data, my_add)

#my_output = tf.sub(x_data[0], tf.add(tf.matmul(x_data[1], A), b))

# Add classification loss (cross entropy)

xentropy = tf.nn.sigmoid_cross_entropy_with_logits(my_output, y_target)

# Create Optimizer

my_opt     = tf.train.GradientDescentOptimizer(0.05)

train_step = my_opt.minimize(xentropy)

# Initialize variables

init = tf.initialize_all_variables()

sess.run(init)

Training loop: 

# Run Loop

for i in range(1000):

    rand_index = np.random.choice(len(iris_2d), size=batch_size)

    #rand_x = np.transpose([iris_2d[rand_index]])

    rand_x  = iris_2d[rand_index]
    # 获取一个样本数据

    rand_x1 = np.array([[x[0]] for x in rand_x])

    rand_x2 = np.array([[x[1]] for x in rand_x])

    #rand_y = np.transpose([binary_target[rand_index]])

    rand_y  = np.array([[y] for y in binary_target[rand_index]])

    sess.run(train_step, feed_dict={x1_data: rand_x1, x2_data: rand_x2, y_target: rand_y})

    if (i+1)%200==0:

        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)) + ', b = ' + str(sess.run(b))) 
# Visualize Results

# Pull out slope/intercept

[[slope]] = sess.run(A)

[[intercept]] = sess.run(b)

# Create fitted line

x = np.linspace(0, 3, num=50)

ablineValues = []

for i in x:

  ablineValues.append(slope*i+intercept)

# Plot the fitted line over the data

setosa_x = [a[1] for i,a in enumerate(iris_2d) if binary_target[i]==1]

setosa_y = [a[0] for i,a in enumerate(iris_2d) if binary_target[i]==1]

non_setosa_x = [a[1] for i,a in enumerate(iris_2d) if binary_target[i]==0]

non_setosa_y = [a[0] for i,a in enumerate(iris_2d) if binary_target[i]==0]

plt.plot(setosa_x, setosa_y, 'rx', ms=10, mew=2, label='setosa')

plt.plot(non_setosa_x, non_setosa_y, 'ro', label='Non-setosa')

plt.plot(x, ablineValues, 'b-')

plt.xlim([0.0, 2.7])

plt.ylim([0.0, 7.1])

plt.suptitle('Linear Separator For I.setosa', fontsize=20)

plt.xlabel('Petal Length')

plt.ylabel('Petal Width')

plt.legend(loc='lower right')

plt.show()

Visualize Results

Result: 

Evaluating Models

（略，建议专题讲解）

[Tensorflow] Cookbook - The Tensorflow Way的更多相关文章

  1. [Tensorflow] Practice - The Tensorflow Way

    该系列主要是<Tensorflow 实战Google深度学习框架 >阅读笔记:有了Cookbook的热身后,以这本书作为基础形成个人知识体系. Ref: [Tensorflow] Cook ...

  2. 移动端目标识别(３)——使用TensorFlow Lite将tensorflow模型部署到移动端(ssd)之Running on mobile with TensorFlow Lite　（写的很乱，回头更新一个简洁的版本）

    承接移动端目标识别(2) 使用TensorFlow Lite在移动设备上运行         在本节中,我们将向您展示如何使用TensorFlow Lite获得更小的模型,并允许您利用针对移动设备优化 ...

  3. 移动端目标识别(1)——使用TensorFlow Lite将tensorflow模型部署到移动端(ssd)之TensorFlow Lite简介

    平时工作就是做深度学习,但是深度学习没有落地就是比较虚,目前在移动端或嵌入式端应用的比较实际,也了解到目前主要有 caffe2,腾讯ncnn,tensorflow,因为工作用tensorflow比较多 ...

  4. 移动端目标识别(2)——使用TENSORFLOW LITE将TENSORFLOW模型部署到移动端(SSD)之TF Lite Developer Guide

    TF Lite开发人员指南 目录: 1 选择一个模型 使用一个预训练模型 使用自己的数据集重新训练inception-V3,MovileNet 训练自己的模型 2 转换模型格式 转换tf.GraphD ...

  5. TensorFlow+Keras 03 TensorFlow 与 Keras 介绍

    1 TensorFlow 架构图 1.1 处理器 TensorFlow 可以在CPU.GPU.TPU中执行 1.2 平台 TensorFlow 具备跨平台能力,Windows .Linux.Andro ...

  6. [Tensorflow] **Android Meets TensorFlow

    TensorFlow Dev Summit 2017 From: Android Meets TensorFlow: How to Accelerate Your App with AI (Googl ...

  7. 使用TensorFlow Serving优化TensorFlow模型

    使用TensorFlow Serving优化TensorFlow模型 https://www.tensorflowers.cn/t/7464 https://mp.weixin.qq.com/s/qO ...

  8. TensorFlow应用实战 | TensorFlow基础知识

    挺长的~超出估计值了~预计阅读时间20分钟. 从helloworld开始 mkdir 1.helloworld cd 1.helloworldvim helloworld.py 代码: # -*- c ...

  9. tensorflow兼容处理 tensorflow.compat.v1

    https://www.wandouip.com/t5i183316/ 引言 原来引用过一个段子,这里还要再引用一次.是关于苹果的.大意是,苹果发布了新的开发语言Swift,有非常多优秀的特征,于是很 ...

随机推荐

  1. java.net.SocketException四大异常解决方案

    java.net.SocketException如何才能更好的使用呢?这个就需要我们先要了解有关这个语言的相关问题.希望大家有所帮助.那么我们就来看看有关java.net.SocketExceptio ...

  2. eclipse中出现的一些错误

    selection dose not contain a main type解决办法:main忘记加static

  3. Saltstack 命令

    命令格式 salt '<操作目标>' <方法>[参数] 查看被控主机内存使用情况 [root@node1 ~]# salt '*' cmd.run 'free -h' node ...

  4. tcp keepalive选项

    之前一直对tcp keepalive选项理解有误, 以为通过setsockopt函数设置SO_KEEPALIVE和相关参数后该socket则使用设置的keepalive相关参数 否则使用系统默认的:k ...

  5. Hbase多版本的读写（Shell&Java API版）

    Hbase是基于HDFS的NOsql数据库,它很多地方跟数据库差不多,也有很多不同的地方.这里就不一一列举了,不过Hbase有个版本控制的特性,这个特性在很多场景下都会发挥很大的作用.本篇就介绍下基于 ...

  6. linux中apt-get和yum和wget的区别

    1.RedHat系列:Redhat.Centos.Fedora等   yum 2.Debian系列:Debian.Ubuntu等   apt-get wget类似迅雷

  7. Makefile 中的.PHONY

    PHONY 目标并非实际的文件名:只是在显式请求时执行命令的名字.有两种理由需要使用PHONY 目标:避免和同名文件冲突,改善性能. 所谓的PHONY这个单词就是伪造的意思,makefile中将.PH ...

  8. [Java web]Spring+Struts2+Hibernate整合过程

    摘要 最近一直在折腾java web相关内容,这里就把最近学习的spring+struts2+hibernate进行一个整合,也就是大家经常说的ssh. 环境 工具IDE :Idea 2018 数据库 ...

  9. mysql5.7报err 1055错误 sql_mode=only_full_group_by

    vim /etc/my.cnf 末尾增加 sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_B ...

  10. python3简单使用requests 用户代理，cookie池

    官方文档:http://docs.python-requests.org/en/master/ 参考文档:http://www.cnblogs.com/zhaof/p/6915127.html#und ...