MNIST pros

运行TensorFlow的InteractiveSession/交互式会话

sess = tf.InteractiveSession()用来创建交互式的会话,其余使用同Session

1
2
import numpy as np
import tensorflow as tf
D:\Anaconda3\Anaconda3_py36\lib\site-packages\h5py\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters

准备工作

和上一节类似

1
2
3
4
5
6
7
# http://yann.lecun.com/exdb/mnist/  to download mnist dataset
# https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/input_data.py
# to read dataset
from load_mnist import *
import tensorflow.examples.tutorials.mnist.input_data as input_data
minst_dir = "MNIST_data/"
mnist = input_data.read_data_sets(minst_dir, one_hot=True)
1
2
x = tf.placeholder("float", shape=[None, 784]) 
y_ = tf.placeholder("float", shape=[None, 10]) # true label

构建一个多层卷积网络

权重初始化

1
2
3
4
5
6
7
8
def weight_variable(shape):
initial = tf.truncated_normal(shape, stddev=0.1) # The generated values follow a normal distribution with specified mean and
# standard deviation.从截断的正态分布中输出随机值,中如果x的取值在区间(μ-2σ,μ+2σ)之外则重新进行选择
return tf.Variable(initial)

def bias_variable(shape):
initial = tf.constant(0.1, shape=shape) # Constant 2-D tensor populated with scalar value 0.1. 初始值为0.1,广播到shape形状的二维数组
return tf.Variable(initial)

卷积和汇聚(池化

由于Pooling翻译成池化让人不明所以,个人习惯依据其作用,而称之为汇聚。

简单起见,卷积使用1步长(stride size),0边距(padding size)的模板,保证输出和输入是同一个大小。我们的池化用简单传统的2x2大小的模板做max pooling.

1
2
3
4
5
def conv2d(x, W):
return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
return tf.nn.max_pool(x, ksize=[1,2,2,1], strides=[1,2,2,1], padding='SAME')

第一层卷积

它由一个卷积接一个max pooling完成。卷积在每个5x5的patch中算出32个特征。卷积的权重张量形状是[5, 5, 1, 32],前两个维度是patch的大小,接着是输入的通道数目,最后是输出的通道数目。 而对于每一个输出通道都有一个对应的偏置量。

1
2
3
4
5
6
7
8
9
10
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])

# convert x to 4d vector
x_image = tf.reshape(x, [-1, 28,28, 1])# num x height x width x channel

# We then convolve x_image with the weight tensor, add the bias, apply the ReLU function, and finally max pool.
# 我们把x_image和权值向量进行卷积,加上偏置项,然后应用ReLU激活函数,最后进行max pooling。
h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1)+b_conv1)
h_pool1 = max_pool_2x2(h_conv1)

第二层卷积

为了构建一个更深的网络,我们会把几个类似的层堆叠起来。第二层中,每个5x5的patch会得到64个特征。

1
2
3
4
5
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])

h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2)+b_conv2)
h_pool2 = max_pool_2x2(h_conv2)

密集连接(全连接)层

经过两层卷积(激活)汇聚组合拳,原本的28x28的图像,特征图的尺寸减小到7x7(主要得益于汇聚)。
并且一张图片有64个特征图(得益于卷积)。
引入一个有1024个神经元的全连接层,来处理整个图片。

1
2
3
4
5
W_fc1 = weight_variable([7*7*64, 1024])
b_fc1 = bias_variable([1024])

h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1)+b_fc1)

Dropout

好像还没有官方中文翻译,那就叫它断电吧。使用TensorFlow构建计算图和使用vhdl构建电路有很多相似之处

减少过拟合。用一个placeholder来代表一个神经元的输出在dropout中保持不变的概率。这样我们可以在训练过程中启用dropout,在测试过程中关闭dropout。 TensorFlow的tf.nn.dropout操作除了可以屏蔽神经元的输出外,还会自动处理神经元输出值的scale。所以用dropout的时候可以不用考虑scale。

1
2
keep_prob = tf.placeholder("float")
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)

输出层

最后添加一个softmax层,如同前一节的softmax回归一样。

1
2
3
4
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])

y_conv = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2)+b_fc2) # 输出层不需要激活函数

训练和评估

我们会用更加复杂的ADAM优化器来做梯度最速下降,在feed_dict中加入额外的参数keep_prob来控制dropout比例。然后每100次迭代输出一次日志。

1
import time
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
cross_entropy = -tf.reduce_sum(y_*tf.log(y_conv))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy) # 优化器只是按照规则(参数更新算法)来更新参数的
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float")) # acc rate

sess = tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

print('Training...')
start_t = time.clock()
for step in range(20001):
batch = mnist.train.next_batch(50)
if step%100==0:
train_accuracy = accuracy.eval(feed_dict={x:batch[0], y_:batch[1], keep_prob:1.0})
print("Step %d, training accuracy %f"%(step, train_accuracy))
train_step.run(feed_dict={x:batch[0], y_:batch[1], keep_prob:0.5})
end_t = time.clock()
print("Train Time Cost: ", end_t-start_t)

print('Testing...')
print("Test accuracy %f"%accuracy.eval(feed_dict={x:mnist.test.images,
y_:mnist.test.labels,
keep_prob:1.0}))

sess.close()
Training...
Step 0, training accuracy 0.120000
Step 100, training accuracy 0.780000
Step 200, training accuracy 0.860000
Step 300, training accuracy 0.860000
Step 400, training accuracy 0.980000
Step 500, training accuracy 0.860000
Step 600, training accuracy 0.940000
Step 700, training accuracy 0.940000
Step 800, training accuracy 0.960000
Step 900, training accuracy 0.940000
Step 1000, training accuracy 0.960000
Step 1100, training accuracy 1.000000
Step 1200, training accuracy 0.960000
Step 1300, training accuracy 0.980000
Step 1400, training accuracy 1.000000
Step 1500, training accuracy 0.960000
Step 1600, training accuracy 0.940000
Step 1700, training accuracy 0.940000
Step 1800, training accuracy 0.940000
Step 1900, training accuracy 0.980000
Step 2000, training accuracy 0.980000
Step 2100, training accuracy 0.960000
Step 2200, training accuracy 0.940000
Step 2300, training accuracy 0.980000
Step 2400, training accuracy 0.980000
Step 2500, training accuracy 0.960000
Step 2600, training accuracy 1.000000
Step 2700, training accuracy 1.000000
Step 2800, training accuracy 0.980000
Step 2900, training accuracy 1.000000
Step 3000, training accuracy 0.980000
Step 3100, training accuracy 1.000000
Step 3200, training accuracy 0.980000
Step 3300, training accuracy 1.000000
Step 3400, training accuracy 1.000000
Step 3500, training accuracy 0.960000
Step 3600, training accuracy 1.000000
Step 3700, training accuracy 0.980000
Step 3800, training accuracy 1.000000
Step 3900, training accuracy 0.980000
Step 4000, training accuracy 0.980000
Step 4100, training accuracy 1.000000
Step 4200, training accuracy 0.980000
Step 4300, training accuracy 0.960000
Step 4400, training accuracy 1.000000
Step 4500, training accuracy 1.000000
Step 4600, training accuracy 1.000000
Step 4700, training accuracy 1.000000
Step 4800, training accuracy 1.000000
Step 4900, training accuracy 0.980000
Step 5000, training accuracy 0.980000
Step 5100, training accuracy 0.980000
Step 5200, training accuracy 1.000000
Step 5300, training accuracy 1.000000
Step 5400, training accuracy 1.000000
Step 5500, training accuracy 1.000000
Step 5600, training accuracy 1.000000
Step 5700, training accuracy 0.980000
Step 5800, training accuracy 1.000000
Step 5900, training accuracy 0.980000
Step 6000, training accuracy 0.980000
Step 6100, training accuracy 0.980000
Step 6200, training accuracy 1.000000
Step 6300, training accuracy 0.980000
Step 6400, training accuracy 0.980000
Step 6500, training accuracy 1.000000
Step 6600, training accuracy 0.980000
Step 6700, training accuracy 1.000000
Step 6800, training accuracy 1.000000
Step 6900, training accuracy 1.000000
Step 7000, training accuracy 1.000000
Step 7100, training accuracy 0.980000
Step 7200, training accuracy 1.000000
Step 7300, training accuracy 1.000000
Step 7400, training accuracy 0.980000
Step 7500, training accuracy 1.000000
Step 7600, training accuracy 0.980000
Step 7700, training accuracy 0.980000
Step 7800, training accuracy 1.000000
Step 7900, training accuracy 0.980000
Step 8000, training accuracy 1.000000
Step 8100, training accuracy 1.000000
Step 8200, training accuracy 0.980000
Step 8300, training accuracy 0.980000
Step 8400, training accuracy 1.000000
Step 8500, training accuracy 0.980000
Step 8600, training accuracy 1.000000
Step 8700, training accuracy 0.980000
Step 8800, training accuracy 0.980000
Step 8900, training accuracy 1.000000
Step 9000, training accuracy 1.000000
Step 9100, training accuracy 1.000000
Step 9200, training accuracy 1.000000
Step 9300, training accuracy 0.980000
Step 9400, training accuracy 1.000000
Step 9500, training accuracy 0.960000
Step 9600, training accuracy 0.980000
Step 9700, training accuracy 1.000000
Step 9800, training accuracy 1.000000
Step 9900, training accuracy 1.000000
Step 10000, training accuracy 1.000000
Step 10100, training accuracy 1.000000
Step 10200, training accuracy 1.000000
Step 10300, training accuracy 1.000000
Step 10400, training accuracy 1.000000
Step 10500, training accuracy 1.000000
Step 10600, training accuracy 1.000000
Step 10700, training accuracy 1.000000
Step 10800, training accuracy 0.980000
Step 10900, training accuracy 1.000000
Step 11000, training accuracy 1.000000
Step 11100, training accuracy 1.000000
Step 11200, training accuracy 0.980000
Step 11300, training accuracy 1.000000
Step 11400, training accuracy 1.000000
Step 11500, training accuracy 0.980000
Step 11600, training accuracy 0.980000
Step 11700, training accuracy 0.960000
Step 11800, training accuracy 1.000000
Step 11900, training accuracy 1.000000
Step 12000, training accuracy 0.980000
Step 12100, training accuracy 1.000000
Step 12200, training accuracy 0.980000
Step 12300, training accuracy 1.000000
Step 12400, training accuracy 1.000000
Step 12500, training accuracy 1.000000
Step 12600, training accuracy 1.000000
Step 12700, training accuracy 1.000000
Step 12800, training accuracy 1.000000
Step 12900, training accuracy 1.000000
Step 13000, training accuracy 1.000000
Step 13100, training accuracy 1.000000
Step 13200, training accuracy 1.000000
Step 13300, training accuracy 1.000000
Step 13400, training accuracy 1.000000
Step 13500, training accuracy 1.000000
Step 13600, training accuracy 1.000000
Step 13700, training accuracy 1.000000
Step 13800, training accuracy 1.000000
Step 13900, training accuracy 1.000000
Step 14000, training accuracy 1.000000
Step 14100, training accuracy 1.000000
Step 14200, training accuracy 1.000000
Step 14300, training accuracy 1.000000
Step 14400, training accuracy 1.000000
Step 14500, training accuracy 1.000000
Step 14600, training accuracy 0.980000
Step 14700, training accuracy 1.000000
Step 14800, training accuracy 1.000000
Step 14900, training accuracy 1.000000
Step 15000, training accuracy 0.980000
Step 15100, training accuracy 1.000000
Step 15200, training accuracy 1.000000
Step 15300, training accuracy 1.000000
Step 15400, training accuracy 1.000000
Step 15500, training accuracy 1.000000
Step 15600, training accuracy 1.000000
Step 15700, training accuracy 1.000000
Step 15800, training accuracy 1.000000
Step 15900, training accuracy 1.000000
Step 16000, training accuracy 1.000000
Step 16100, training accuracy 1.000000
Step 16200, training accuracy 1.000000
Step 16300, training accuracy 1.000000
Step 16400, training accuracy 1.000000
Step 16500, training accuracy 1.000000
Step 16600, training accuracy 1.000000
Step 16700, training accuracy 0.960000
Step 16800, training accuracy 1.000000
Step 16900, training accuracy 1.000000
Step 17000, training accuracy 1.000000
Step 17100, training accuracy 1.000000
Step 17200, training accuracy 0.980000
Step 17300, training accuracy 1.000000
Step 17400, training accuracy 1.000000
Step 17500, training accuracy 1.000000
Step 17600, training accuracy 1.000000
Step 17700, training accuracy 1.000000
Step 17800, training accuracy 1.000000
Step 17900, training accuracy 1.000000
Step 18000, training accuracy 1.000000
Step 18100, training accuracy 1.000000
Step 18200, training accuracy 1.000000
Step 18300, training accuracy 1.000000
Step 18400, training accuracy 1.000000
Step 18500, training accuracy 1.000000
Step 18600, training accuracy 1.000000
Step 18700, training accuracy 1.000000
Step 18800, training accuracy 1.000000
Step 18900, training accuracy 1.000000
Step 19000, training accuracy 1.000000
Step 19100, training accuracy 1.000000
Step 19200, training accuracy 1.000000
Step 19300, training accuracy 1.000000
Step 19400, training accuracy 1.000000
Step 19500, training accuracy 1.000000
Step 19600, training accuracy 1.000000
Step 19700, training accuracy 1.000000
Step 19800, training accuracy 1.000000
Step 19900, training accuracy 0.980000
Step 20000, training accuracy 1.000000
Train Time Cost:  205.04848637568546
Testing...
Test accuracy 0.992300

总结

本节对mnist数据集,使用TensorFlow构建了卷积池化API,并构建了两层卷积池化层,一层全连接层的神经网络,使用dropout来减少过拟合,最后输出到softmax回归层,从而构建了完整的卷积神经网络模型。优化器使用Adam,经过2万次迭代的训练,在测试集上精确度达到了(99%).