StarMirror/data-science-ipython-notebooks
1
0
Fork 0
mirror of https://github.com/donnemartin/data-science-ipython-notebooks.git synced 2024-03-22 13:30:56 +08:00
Code Issues Projects Releases Wiki Activity
62f06d25d7
data-science-ipython-notebooks/deep-learning/tensor-flow-exercises/2_fullyconnected.ipynb
Donne Martin 4c9ffb83e7 Add TensorFlow fullyconnected notebook.
2015-12-27 07:22:58 -05:00

612 lines
19 KiB
Python
Raw Blame History

{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "kR-4eNdK6lYS"
},
"source": [
"Deep Learning with TensorFlow\n",
"=============\n",
"\n",
"Credits: Forked from [TensorFlow](https://github.com/tensorflow/tensorflow) by Google\n",
"\n",
"Setup\n",
"------------\n",
"\n",
"Refer to the [setup instructions](https://github.com/donnemartin/data-science-ipython-notebooks/tree/feature/deep-learning/deep-learning/tensor-flow-exercises/README.md).\n",
"\n",
"Exercise 2\n",
"------------\n",
"\n",
"Previously in `1_notmnist.ipynb`, we created a pickle with formatted datasets for training, development and testing on the [notMNIST dataset](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html).\n",
"\n",
"The goal of this exercise is to progressively train deeper and more accurate models using TensorFlow."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "JLpLa8Jt7Vu4"
},
"outputs": [],
"source": [
"# These are all the modules we'll be using later. Make sure you can import them\n",
"# before proceeding further.\n",
"import cPickle as pickle\n",
"import numpy as np\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1HrCK6e17WzV"
},
"source": [
"First reload the data we generated in `1_notmist.ipynb`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 19456,
"status": "ok",
"timestamp": 1449847956073,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "y3-cj1bpmuxc",
"outputId": "0ddb1607-1fc4-4ddb-de28-6c7ab7fb0c33"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 28, 28) (200000,)\n",
"Validation set (10000, 28, 28) (10000,)\n",
"Test set (18724, 28, 28) (18724,)\n"
]
}
],
"source": [
"pickle_file = 'notMNIST.pickle'\n",
"\n",
"with open(pickle_file, 'rb') as f:\n",
" save = pickle.load(f)\n",
" train_dataset = save['train_dataset']\n",
" train_labels = save['train_labels']\n",
" valid_dataset = save['valid_dataset']\n",
" valid_labels = save['valid_labels']\n",
" test_dataset = save['test_dataset']\n",
" test_labels = save['test_labels']\n",
" del save # hint to help gc free up memory\n",
" print 'Training set', train_dataset.shape, train_labels.shape\n",
" print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
" print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "L7aHrm6nGDMB"
},
"source": [
"Reformat into a shape that's more adapted to the models we're going to train:\n",
"- data as a flat matrix,\n",
"- labels as float 1-hot encodings."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 19723,
"status": "ok",
"timestamp": 1449847956364,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "IRSyYiIIGIzS",
"outputId": "2ba0fc75-1487-4ace-a562-cf81cae82793"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 784) (200000, 10)\n",
"Validation set (10000, 784) (10000, 10)\n",
"Test set (18724, 784) (18724, 10)\n"
]
}
],
"source": [
"image_size = 28\n",
"num_labels = 10\n",
"\n",
"def reformat(dataset, labels):\n",
" dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)\n",
" # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]\n",
" labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
" return dataset, labels\n",
"train_dataset, train_labels = reformat(train_dataset, train_labels)\n",
"valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)\n",
"test_dataset, test_labels = reformat(test_dataset, test_labels)\n",
"print 'Training set', train_dataset.shape, train_labels.shape\n",
"print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
"print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "nCLVqyQ5vPPH"
},
"source": [
"We're first going to train a multinomial logistic regression using simple gradient descent.\n",
"\n",
"TensorFlow works like this:\n",
"* First you describe the computation that you want to see performed: what the inputs, the variables, and the operations look like. These get created as nodes over a computation graph. This description is all contained within the block below:\n",
"\n",
" with graph.as_default():\n",
" ...\n",
"\n",
"* Then you can run the operations on this graph as many times as you want by calling `session.run()`, providing it outputs to fetch from the graph that get returned. This runtime operation is all contained in the block below:\n",
"\n",
" with tf.Session(graph=graph) as session:\n",
" ...\n",
"\n",
"Let's load all the data into TensorFlow and build the computation graph corresponding to our training:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "Nfv39qvtvOl_"
},
"outputs": [],
"source": [
"# With gradient descent training, even this much data is prohibitive.\n",
"# Subset the training data for faster turnaround.\n",
"train_subset = 10000\n",
"\n",
"graph = tf.Graph()\n",
"with graph.as_default():\n",
"\n",
" # Input data.\n",
" # Load the training, validation and test data into constants that are\n",
" # attached to the graph.\n",
" tf_train_dataset = tf.constant(train_dataset[:train_subset, :])\n",
" tf_train_labels = tf.constant(train_labels[:train_subset])\n",
" tf_valid_dataset = tf.constant(valid_dataset)\n",
" tf_test_dataset = tf.constant(test_dataset)\n",
" \n",
" # Variables.\n",
" # These are the parameters that we are going to be training. The weight\n",
" # matrix will be initialized using random valued following a (truncated)\n",
" # normal distribution. The biases get initialized to zero.\n",
" weights = tf.Variable(\n",
" tf.truncated_normal([image_size * image_size, num_labels]))\n",
" biases = tf.Variable(tf.zeros([num_labels]))\n",
" \n",
" # Training computation.\n",
" # We multiply the inputs with the weight matrix, and add biases. We compute\n",
" # the softmax and cross-entropy (it's one operation in TensorFlow, because\n",
" # it's very common, and it can be optimized). We take the average of this\n",
" # cross-entropy across all training examples: that's our loss.\n",
" logits = tf.matmul(tf_train_dataset, weights) + biases\n",
" loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))\n",
" \n",
" # Optimizer.\n",
" # We are going to find the minimum of this loss using gradient descent.\n",
" optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
" \n",
" # Predictions for the training, validation, and test data.\n",
" # These are not part of training, but merely here so that we can report\n",
" # accuracy figures as we train.\n",
" train_prediction = tf.nn.softmax(logits)\n",
" valid_prediction = tf.nn.softmax(\n",
" tf.matmul(tf_valid_dataset, weights) + biases)\n",
" test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "KQcL4uqISHjP"
},
"source": [
"Let's run this computation and iterate:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 9
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 57454,
"status": "ok",
"timestamp": 1449847994134,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "z2cjdenH869W",
"outputId": "4c037ba1-b526-4d8e-e632-91e2a0333267"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized\n",
"Loss at step 0 : 17.2939\n",
"Training accuracy: 10.8%\n",
"Validation accuracy: 13.8%\n",
"Loss at step 100 : 2.26903\n",
"Training accuracy: 72.3%\n",
"Validation accuracy: 71.6%\n",
"Loss at step 200 : 1.84895\n",
"Training accuracy: 74.9%\n",
"Validation accuracy: 73.9%\n",
"Loss at step 300 : 1.60701\n",
"Training accuracy: 76.0%\n",
"Validation accuracy: 74.5%\n",
"Loss at step 400 : 1.43912\n",
"Training accuracy: 76.8%\n",
"Validation accuracy: 74.8%\n",
"Loss at step 500 : 1.31349\n",
"Training accuracy: 77.5%\n",
"Validation accuracy: 75.0%\n",
"Loss at step 600 : 1.21501\n",
"Training accuracy: 78.1%\n",
"Validation accuracy: 75.4%\n",
"Loss at step 700 : 1.13515\n",
"Training accuracy: 78.6%\n",
"Validation accuracy: 75.4%\n",
"Loss at step 800 : 1.0687\n",
"Training accuracy: 79.2%\n",
"Validation accuracy: 75.6%\n",
"Test accuracy: 82.9%\n"
]
}
],
"source": [
"num_steps = 801\n",
"\n",
"def accuracy(predictions, labels):\n",
" return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))\n",
" / predictions.shape[0])\n",
"\n",
"with tf.Session(graph=graph) as session:\n",
" # This is a one-time operation which ensures the parameters get initialized as\n",
" # we described in the graph: random weights for the matrix, zeros for the\n",
" # biases. \n",
" tf.initialize_all_variables().run()\n",
" print 'Initialized'\n",
" for step in xrange(num_steps):\n",
" # Run the computations. We tell .run() that we want to run the optimizer,\n",
" # and get the loss value and the training predictions returned as numpy\n",
" # arrays.\n",
" _, l, predictions = session.run([optimizer, loss, train_prediction])\n",
" if (step % 100 == 0):\n",
" print 'Loss at step', step, ':', l\n",
" print 'Training accuracy: %.1f%%' % accuracy(\n",
" predictions, train_labels[:train_subset, :])\n",
" # Calling .eval() on valid_prediction is basically like calling run(), but\n",
" # just to get that one numpy array. Note that it recomputes all its graph\n",
" # dependencies.\n",
" print 'Validation accuracy: %.1f%%' % accuracy(\n",
" valid_prediction.eval(), valid_labels)\n",
" print 'Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "x68f-hxRGm3H"
},
"source": [
"Let's now switch to stochastic gradient descent training instead, which is much faster.\n",
"\n",
"The graph will be similar, except that instead of holding all the training data into a constant node, we create a `Placeholder` node which will be fed actual data at every call of `sesion.run()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "qhPMzWYRGrzM"
},
"outputs": [],
"source": [
"batch_size = 128\n",
"\n",
"graph = tf.Graph()\n",
"with graph.as_default():\n",
"\n",
" # Input data. For the training data, we use a placeholder that will be fed\n",
" # at run time with a training minibatch.\n",
" tf_train_dataset = tf.placeholder(tf.float32,\n",
" shape=(batch_size, image_size * image_size))\n",
" tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))\n",
" tf_valid_dataset = tf.constant(valid_dataset)\n",
" tf_test_dataset = tf.constant(test_dataset)\n",
" \n",
" # Variables.\n",
" weights = tf.Variable(\n",
" tf.truncated_normal([image_size * image_size, num_labels]))\n",
" biases = tf.Variable(tf.zeros([num_labels]))\n",
" \n",
" # Training computation.\n",
" logits = tf.matmul(tf_train_dataset, weights) + biases\n",
" loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))\n",
" \n",
" # Optimizer.\n",
" optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
" \n",
" # Predictions for the training, validation, and test data.\n",
" train_prediction = tf.nn.softmax(logits)\n",
" valid_prediction = tf.nn.softmax(\n",
" tf.matmul(tf_valid_dataset, weights) + biases)\n",
" test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "XmVZESmtG4JH"
},
"source": [
"Let's run it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 6
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 66292,
"status": "ok",
"timestamp": 1449848003013,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "FoF91pknG_YW",
"outputId": "d255c80e-954d-4183-ca1c-c7333ce91d0a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized\n",
"Minibatch loss at step 0 : 16.8091\n",
"Minibatch accuracy: 12.5%\n",
"Validation accuracy: 14.0%\n",
"Minibatch loss at step 500 : 1.75256\n",
"Minibatch accuracy: 77.3%\n",
"Validation accuracy: 75.0%\n",
"Minibatch loss at step 1000 : 1.32283\n",
"Minibatch accuracy: 77.3%\n",
"Validation accuracy: 76.6%\n",
"Minibatch loss at step 1500 : 0.944533\n",
"Minibatch accuracy: 83.6%\n",
"Validation accuracy: 76.5%\n",
"Minibatch loss at step 2000 : 1.03795\n",
"Minibatch accuracy: 78.9%\n",
"Validation accuracy: 77.8%\n",
"Minibatch loss at step 2500 : 1.10219\n",
"Minibatch accuracy: 80.5%\n",
"Validation accuracy: 78.0%\n",
"Minibatch loss at step 3000 : 0.758874\n",
"Minibatch accuracy: 82.8%\n",
"Validation accuracy: 78.8%\n",
"Test accuracy: 86.1%\n"
]
}
],
"source": [
"num_steps = 3001\n",
"\n",
"with tf.Session(graph=graph) as session:\n",
" tf.initialize_all_variables().run()\n",
" print \"Initialized\"\n",
" for step in xrange(num_steps):\n",
" # Pick an offset within the training data, which has been randomized.\n",
" # Note: we could use better randomization across epochs.\n",
" offset = (step * batch_size) % (train_labels.shape[0] - batch_size)\n",
" # Generate a minibatch.\n",
" batch_data = train_dataset[offset:(offset + batch_size), :]\n",
" batch_labels = train_labels[offset:(offset + batch_size), :]\n",
" # Prepare a dictionary telling the session where to feed the minibatch.\n",
" # The key of the dictionary is the placeholder node of the graph to be fed,\n",
" # and the value is the numpy array to feed to it.\n",
" feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}\n",
" _, l, predictions = session.run(\n",
" [optimizer, loss, train_prediction], feed_dict=feed_dict)\n",
" if (step % 500 == 0):\n",
" print \"Minibatch loss at step\", step, \":\", l\n",
" print \"Minibatch accuracy: %.1f%%\" % accuracy(predictions, batch_labels)\n",
" print \"Validation accuracy: %.1f%%\" % accuracy(\n",
" valid_prediction.eval(), valid_labels)\n",
" print \"Test accuracy: %.1f%%\" % accuracy(test_prediction.eval(), test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "7omWxtvLLxik"
},
"source": [
"---\n",
"Problem\n",
"-------\n",
"\n",
"Turn the logistic regression example with SGD into a 1-hidden layer neural network with rectified linear units (nn.relu()) and 1024 hidden nodes. This model should improve your validation / test accuracy.\n",
"\n",
"---"
]
}
],
"metadata": {
"colabVersion": "0.3.2",
"colab_default_view": {},
"colab_views": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Reference in New Issue View Git Blame Copy Permalink