StarMirror/data-science-ipython-notebooks
1
0
Fork 0
mirror of https://github.com/donnemartin/data-science-ipython-notebooks.git synced 2024-03-22 13:30:56 +08:00
Code Issues Projects Releases Wiki Activity

Merge pull request #18 from donnemartin/feature/deep-learning

Browse Source 
Add TensorFlow and DeepDream deep learning notebooks
This commit is contained in:
Donne Martin 2015-12-27 07:28:10 -05:00
commit 5ecc14908b
12 changed files with 4692 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

22
README.md
View File

@ -5,7 +5,7 @@
# data-science-ipython-notebooks
This repo is a collection of IPython Notebooks I reference while working with data. Although I developed and maintain most notebooks, some notebooks I reference were created by other authors, who are credited within their notebook(s) by providing their names and/or a link to their source.
This repo is a collection of IPython Notebooks I reference while working with data. Although I developed and maintain many notebooks, other notebooks I reference were created by various authors, who are credited within their notebook(s) by providing their names and/or a link to their source.
For detailed instructions, scripts, and tools to more optimally set up your development environment for data analysis, check out the [dev-setup](https://github.com/donnemartin/dev-setup) repo.
@ -21,6 +21,7 @@ For detailed instructions, scripts, and tools to more optimally set up your deve
* [amazon web services](#aws)
* [kaggle-and-business-analyses](#kaggle-and-business-analyses)
* [scikit-learn](#scikit-learn)
* [deep-learning](#deep-learning)
* [statistical-inference-scipy](#statistical-inference-scipy)
* [pandas](#pandas)
* [matplotlib](#matplotlib)
@ -120,6 +121,25 @@ IPython Notebook(s) demonstrating scikit-learn functionality.
| [gmm](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/scikit-learn/scikit-learn-gmm.ipynb) | Gaussian mixture models. |
| [validation](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/scikit-learn/scikit-learn-validation.ipynb) | Validation and model selection. |
<br/>
<p align="center">
<img src="http://i.imgur.com/ZhKXrKZ.png">
</p>
## deep-learning
IPython Notebook(s) demonstrating deep learning functionality.
| Notebook | Description |
|--------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [deep dream](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/deep-dream/dream.ipynb) | Caffe-based computer vision program which uses a convolutional neural network to find and enhance patterns in images. |
| [ts-not-mnist](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/1_notmnist.ipynb) | Learn simple data curation by creating a pickle with formatted datasets for training, development and testing in TensorFlow. |
| [ts-fully-connected](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/2_fullyconnected.ipynb) | Progressively train deeper and more accurate models using logistic regression and neural networks in TensorFlow. |
| [ts-regularization](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/3_regularization.ipynb) | Explore regularization techniques by training fully connected networks to classify notMNIST characters in TensorFlow. |
| [ts-convolutions](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/4_convolutions.ipynb) | Create convolutional neural networks in TensorFlow. |
| [ts-word2vec](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/5_word2vec.ipynb) | Train a skip-gram model over Text8 data in TensorFlow. |
| [ts-lstm](http://nbviewer.ipython.org/github/donnemartin/data-science-ipython-notebooks/blob/master/deep-learning/tensor-flow-exercises/6_lstm.ipynb) | Train a LSTM character model over Text8 data in TensorFlow. |
<br/>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/data-science-ipython-notebooks/master/images/scipy.png">

597
deep-learning/deep-dream/dream.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

BIN
deep-learning/deep-dream/flowers.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 30 KiB

BIN
deep-learning/deep-dream/sky1024px.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 64 KiB

680
deep-learning/tensor-flow-exercises/1_notmnist.ipynb Normal file
View File

@ -0,0 +1,680 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5hIbr52I7Z7U"
},
"source": [
"Deep Learning with TensorFlow\n",
"=============\n",
"\n",
"Credits: Forked from [TensorFlow](https://github.com/tensorflow/tensorflow) by Google\n",
"\n",
"Setup\n",
"------------\n",
"\n",
"Refer to the [setup instructions](https://github.com/donnemartin/data-science-ipython-notebooks/tree/feature/deep-learning/deep-learning/tensor-flow-exercises/README.md).\n",
"\n",
"Exercise 1\n",
"------------\n",
"\n",
"The objective of this exercise is to learn about simple data curation practices, and familiarize you with some of the data we'll be reusing later.\n",
"\n",
"This notebook uses the [notMNIST](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html) dataset to be used with python experiments. This dataset is designed to look like the classic [MNIST](http://yann.lecun.com/exdb/mnist/) dataset, while looking a little more like real data: it's a harder task, and the data is a lot less 'clean' than MNIST."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "apJbCsBHl-2A"
},
"outputs": [],
"source": [
"# These are all the modules we'll be using later. Make sure you can import them\n",
"# before proceeding further.\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import os\n",
"import tarfile\n",
"import urllib\n",
"from IPython.display import display, Image\n",
"from scipy import ndimage\n",
"from sklearn.linear_model import LogisticRegression\n",
"import cPickle as pickle"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "jNWGtZaXn-5j"
},
"source": [
"First, we'll download the dataset to our local machine. The data consists of characters rendered in a variety of fonts on a 28x28 image. The labels are limited to 'A' through 'J' (10 classes). The training set has about 500k and the testset 19000 labelled examples. Given these sizes, it should be possible to train models quickly on any machine."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 186058,
"status": "ok",
"timestamp": 1444485672507,
"user": {
"color": "#1FA15D",
"displayName": "Vincent Vanhoucke",
"isAnonymous": false,
"isMe": true,
"permissionId": "05076109866853157986",
"photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
"sessionId": "2a0a5e044bb03b66",
"userId": "102167687554210253930"
},
"user_tz": 420
},
"id": "EYRJ4ICW6-da",
"outputId": "0d0f85df-155f-4a89-8e7e-ee32df36ec8d"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Found and verified notMNIST_large.tar.gz\n",
"Found and verified notMNIST_small.tar.gz\n"
]
}
],
"source": [
"url = 'http://yaroslavvb.com/upload/notMNIST/'\n",
"\n",
"def maybe_download(filename, expected_bytes):\n",
" \"\"\"Download a file if not present, and make sure it's the right size.\"\"\"\n",
" if not os.path.exists(filename):\n",
" filename, _ = urllib.urlretrieve(url + filename, filename)\n",
" statinfo = os.stat(filename)\n",
" if statinfo.st_size == expected_bytes:\n",
" print 'Found and verified', filename\n",
" else:\n",
" raise Exception(\n",
" 'Failed to verify' + filename + '. Can you get to it with a browser?')\n",
" return filename\n",
"\n",
"train_filename = maybe_download('notMNIST_large.tar.gz', 247336696)\n",
"test_filename = maybe_download('notMNIST_small.tar.gz', 8458043)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "cC3p0oEyF8QT"
},
"source": [
"Extract the dataset from the compressed .tar.gz file.\n",
"This should give you a set of directories, labelled A through J."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 186055,
"status": "ok",
"timestamp": 1444485672525,
"user": {
"color": "#1FA15D",
"displayName": "Vincent Vanhoucke",
"isAnonymous": false,
"isMe": true,
"permissionId": "05076109866853157986",
"photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
"sessionId": "2a0a5e044bb03b66",
"userId": "102167687554210253930"
},
"user_tz": 420
},
"id": "H8CBE-WZ8nmj",
"outputId": "ef6c790c-2513-4b09-962e-27c79390c762"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['notMNIST_large/A', 'notMNIST_large/B', 'notMNIST_large/C', 'notMNIST_large/D', 'notMNIST_large/E', 'notMNIST_large/F', 'notMNIST_large/G', 'notMNIST_large/H', 'notMNIST_large/I', 'notMNIST_large/J']\n",
"['notMNIST_small/A', 'notMNIST_small/B', 'notMNIST_small/C', 'notMNIST_small/D', 'notMNIST_small/E', 'notMNIST_small/F', 'notMNIST_small/G', 'notMNIST_small/H', 'notMNIST_small/I', 'notMNIST_small/J']\n"
]
}
],
"source": [
"num_classes = 10\n",
"\n",
"def extract(filename):\n",
" tar = tarfile.open(filename)\n",
" tar.extractall()\n",
" tar.close()\n",
" root = os.path.splitext(os.path.splitext(filename)[0])[0] # remove .tar.gz\n",
" data_folders = [os.path.join(root, d) for d in sorted(os.listdir(root))]\n",
" if len(data_folders) != num_classes:\n",
" raise Exception(\n",
" 'Expected %d folders, one per class. Found %d instead.' % (\n",
" num_folders, len(data_folders)))\n",
" print data_folders\n",
" return data_folders\n",
" \n",
"train_folders = extract(train_filename)\n",
"test_folders = extract(test_filename)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "4riXK3IoHgx6"
},
"source": [
"---\n",
"Problem 1\n",
"---------\n",
"\n",
"Let's take a peek at some of the data to make sure it looks sensible. Each exemplar should be an image of a character A through J rendered in a different font. Display a sample of the images that we just downloaded. Hint: you can use the package IPython.display.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "PBdkjESPK8tw"
},
"source": [
"Now let's load the data in a more manageable format.\n",
"\n",
"We'll convert the entire dataset into a 3D array (image index, x, y) of floating point values, normalized to have approximately zero mean and standard deviation ~0.5 to make training easier down the road. The labels will be stored into a separate array of integers 0 through 9.\n",
"\n",
"A few images might not be readable, we'll just skip them."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 30
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 399874,
"status": "ok",
"timestamp": 1444485886378,
"user": {
"color": "#1FA15D",
"displayName": "Vincent Vanhoucke",
"isAnonymous": false,
"isMe": true,
"permissionId": "05076109866853157986",
"photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
"sessionId": "2a0a5e044bb03b66",
"userId": "102167687554210253930"
},
"user_tz": 420
},
"id": "h7q0XhG3MJdf",
"outputId": "92c391bb-86ff-431d-9ada-315568a19e59"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"notMNIST_large/A\n",
"Could not read: notMNIST_large/A/SG90IE11c3RhcmQgQlROIFBvc3Rlci50dGY=.png : cannot identify image file - it's ok, skipping.\n",
"Could not read: notMNIST_large/A/RnJlaWdodERpc3BCb29rSXRhbGljLnR0Zg==.png : cannot identify image file - it's ok, skipping.\n",
"Could not read: notMNIST_large/A/Um9tYW5hIEJvbGQucGZi.png : cannot identify image file - it's ok, skipping.\n",
"notMNIST_large/B\n",
"Could not read: notMNIST_large/B/TmlraXNFRi1TZW1pQm9sZEl0YWxpYy5vdGY=.png : cannot identify image file - it's ok, skipping.\n",
"notMNIST_large/C\n",
"notMNIST_large/D\n",
"Could not read: notMNIST_large/D/VHJhbnNpdCBCb2xkLnR0Zg==.png : cannot identify image file - it's ok, skipping.\n",
"notMNIST_large/E\n",
"notMNIST_large/F\n",
"notMNIST_large/G\n",
"notMNIST_large/H\n",
"notMNIST_large/I\n",
"notMNIST_large/J\n",
"Full dataset tensor: (529114, 28, 28)\n",
"Mean: -0.0816593\n",
"Standard deviation: 0.454232\n",
"Labels: (529114,)\n",
"notMNIST_small/A\n",
"Could not read: notMNIST_small/A/RGVtb2NyYXRpY2FCb2xkT2xkc3R5bGUgQm9sZC50dGY=.png : cannot identify image file - it's ok, skipping.\n",
"notMNIST_small/B\n",
"notMNIST_small/C\n",
"notMNIST_small/D\n",
"notMNIST_small/E\n",
"notMNIST_small/F\n",
"Could not read: notMNIST_small/F/Q3Jvc3NvdmVyIEJvbGRPYmxpcXVlLnR0Zg==.png : cannot identify image file - it's ok, skipping.\n",
"notMNIST_small/G\n",
"notMNIST_small/H\n",
"notMNIST_small/I\n",
"notMNIST_small/J\n",
"Full dataset tensor: (18724, 28, 28)\n",
"Mean: -0.0746364\n",
"Standard deviation: 0.458622\n",
"Labels: (18724,)\n"
]
}
],
"source": [
"image_size = 28 # Pixel width and height.\n",
"pixel_depth = 255.0 # Number of levels per pixel.\n",
"\n",
"def load(data_folders, min_num_images, max_num_images):\n",
" dataset = np.ndarray(\n",
" shape=(max_num_images, image_size, image_size), dtype=np.float32)\n",
" labels = np.ndarray(shape=(max_num_images), dtype=np.int32)\n",
" label_index = 0\n",
" image_index = 0\n",
" for folder in data_folders:\n",
" print folder\n",
" for image in os.listdir(folder):\n",
" if image_index >= max_num_images:\n",
" raise Exception('More images than expected: %d >= %d' % (\n",
" num_images, max_num_images))\n",
" image_file = os.path.join(folder, image)\n",
" try:\n",
" image_data = (ndimage.imread(image_file).astype(float) -\n",
" pixel_depth / 2) / pixel_depth\n",
" if image_data.shape != (image_size, image_size):\n",
" raise Exception('Unexpected image shape: %s' % str(image_data.shape))\n",
" dataset[image_index, :, :] = image_data\n",
" labels[image_index] = label_index\n",
" image_index += 1\n",
" except IOError as e:\n",
" print 'Could not read:', image_file, ':', e, '- it\\'s ok, skipping.'\n",
" label_index += 1\n",
" num_images = image_index\n",
" dataset = dataset[0:num_images, :, :]\n",
" labels = labels[0:num_images]\n",
" if num_images < min_num_images:\n",
" raise Exception('Many fewer images than expected: %d < %d' % (\n",
" num_images, min_num_images))\n",
" print 'Full dataset tensor:', dataset.shape\n",
" print 'Mean:', np.mean(dataset)\n",
" print 'Standard deviation:', np.std(dataset)\n",
" print 'Labels:', labels.shape\n",
" return dataset, labels\n",
"train_dataset, train_labels = load(train_folders, 450000, 550000)\n",
"test_dataset, test_labels = load(test_folders, 18000, 20000)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "vUdbskYE2d87"
},
"source": [
"---\n",
"Problem 2\n",
"---------\n",
"\n",
"Let's verify that the data still looks good. Displaying a sample of the labels and images from the ndarray. Hint: you can use matplotlib.pyplot.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "GPTCnjIcyuKN"
},
"source": [
"Next, we'll randomize the data. It's important to have the labels well shuffled for the training and test distributions to match."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "6WZ2l2tN2zOL"
},
"outputs": [],
"source": [
"np.random.seed(133)\n",
"def randomize(dataset, labels):\n",
" permutation = np.random.permutation(labels.shape[0])\n",
" shuffled_dataset = dataset[permutation,:,:]\n",
" shuffled_labels = labels[permutation]\n",
" return shuffled_dataset, shuffled_labels\n",
"train_dataset, train_labels = randomize(train_dataset, train_labels)\n",
"test_dataset, test_labels = randomize(test_dataset, test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "puDUTe6t6USl"
},
"source": [
"---\n",
"Problem 3\n",
"---------\n",
"Convince yourself that the data is still good after shuffling!\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "cYznx5jUwzoO"
},
"source": [
"---\n",
"Problem 4\n",
"---------\n",
"Another check: we expect the data to be balanced across classes. Verify that.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "LA7M7K22ynCt"
},
"source": [
"Prune the training data as needed. Depending on your computer setup, you might not be able to fit it all in memory, and you can tune train_size as needed.\n",
"\n",
"Also create a validation dataset for hyperparameter tuning."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 411281,
"status": "ok",
"timestamp": 1444485897869,
"user": {
"color": "#1FA15D",
"displayName": "Vincent Vanhoucke",
"isAnonymous": false,
"isMe": true,
"permissionId": "05076109866853157986",
"photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
"sessionId": "2a0a5e044bb03b66",
"userId": "102167687554210253930"
},
"user_tz": 420
},
"id": "s3mWgZLpyuzq",
"outputId": "8af66da6-902d-4719-bedc-7c9fb7ae7948"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training (200000, 28, 28) (200000,)\n",
"Validation (10000, 28, 28) (10000,)\n"
]
}
],
"source": [
"train_size = 200000\n",
"valid_size = 10000\n",
"\n",
"valid_dataset = train_dataset[:valid_size,:,:]\n",
"valid_labels = train_labels[:valid_size]\n",
"train_dataset = train_dataset[valid_size:valid_size+train_size,:,:]\n",
"train_labels = train_labels[valid_size:valid_size+train_size]\n",
"print 'Training', train_dataset.shape, train_labels.shape\n",
"print 'Validation', valid_dataset.shape, valid_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "tIQJaJuwg5Hw"
},
"source": [
"Finally, let's save the data for later reuse:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "QiR_rETzem6C"
},
"outputs": [],
"source": [
"pickle_file = 'notMNIST.pickle'\n",
"\n",
"try:\n",
" f = open(pickle_file, 'wb')\n",
" save = {\n",
" 'train_dataset': train_dataset,\n",
" 'train_labels': train_labels,\n",
" 'valid_dataset': valid_dataset,\n",
" 'valid_labels': valid_labels,\n",
" 'test_dataset': test_dataset,\n",
" 'test_labels': test_labels,\n",
" }\n",
" pickle.dump(save, f, pickle.HIGHEST_PROTOCOL)\n",
" f.close()\n",
"except Exception as e:\n",
" print 'Unable to save data to', pickle_file, ':', e\n",
" raise"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 413065,
"status": "ok",
"timestamp": 1444485899688,
"user": {
"color": "#1FA15D",
"displayName": "Vincent Vanhoucke",
"isAnonymous": false,
"isMe": true,
"permissionId": "05076109866853157986",
"photoUrl": "//lh6.googleusercontent.com/-cCJa7dTDcgQ/AAAAAAAAAAI/AAAAAAAACgw/r2EZ_8oYer4/s50-c-k-no/photo.jpg",
"sessionId": "2a0a5e044bb03b66",
"userId": "102167687554210253930"
},
"user_tz": 420
},
"id": "hQbLjrW_iT39",
"outputId": "b440efc6-5ee1-4cbc-d02d-93db44ebd956"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Compressed pickle size: 718193801\n"
]
}
],
"source": [
"statinfo = os.stat(pickle_file)\n",
"print 'Compressed pickle size:', statinfo.st_size"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "gE_cRAQB33lk"
},
"source": [
"---\n",
"Problem 5\n",
"---------\n",
"\n",
"By construction, this dataset might contain a lot of overlapping samples, including training data that's also contained in the validation and test set! Overlap between training and test can skew the results if you expect to use your model in an environment where there is never an overlap, but are actually ok if you expect to see training samples recur when you use it.\n",
"Measure how much overlap there is between training, validation and test samples.\n",
"Optional questions:\n",
"- What about near duplicates between datasets? (images that are almost identical)\n",
"- Create a sanitized validation and test set, and compare your accuracy on those in subsequent exercises.\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "L8oww1s4JMQx"
},
"source": [
"---\n",
"Problem 6\n",
"---------\n",
"\n",
"Let's get an idea of what an off-the-shelf classifier can give you on this data. It's always good to check that there is something to learn, and that it's a problem that is not so trivial that a canned solution solves it.\n",
"\n",
"Train a simple model on this data using 50, 100, 1000 and 5000 training samples. Hint: you can use the LogisticRegression model from sklearn.linear_model.\n",
"\n",
"Optional question: train an off-the-shelf model on all the data!\n",
"\n",
"---"
]
}
],
"metadata": {
"colabVersion": "0.3.2",
"colab_default_view": {},
"colab_views": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

611
deep-learning/tensor-flow-exercises/2_fullyconnected.ipynb Normal file
View File

@ -0,0 +1,611 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "kR-4eNdK6lYS"
},
"source": [
"Deep Learning with TensorFlow\n",
"=============\n",
"\n",
"Credits: Forked from [TensorFlow](https://github.com/tensorflow/tensorflow) by Google\n",
"\n",
"Setup\n",
"------------\n",
"\n",
"Refer to the [setup instructions](https://github.com/donnemartin/data-science-ipython-notebooks/tree/feature/deep-learning/deep-learning/tensor-flow-exercises/README.md).\n",
"\n",
"Exercise 2\n",
"------------\n",
"\n",
"Previously in `1_notmnist.ipynb`, we created a pickle with formatted datasets for training, development and testing on the [notMNIST dataset](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html).\n",
"\n",
"The goal of this exercise is to progressively train deeper and more accurate models using TensorFlow."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "JLpLa8Jt7Vu4"
},
"outputs": [],
"source": [
"# These are all the modules we'll be using later. Make sure you can import them\n",
"# before proceeding further.\n",
"import cPickle as pickle\n",
"import numpy as np\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1HrCK6e17WzV"
},
"source": [
"First reload the data we generated in `1_notmist.ipynb`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 19456,
"status": "ok",
"timestamp": 1449847956073,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "y3-cj1bpmuxc",
"outputId": "0ddb1607-1fc4-4ddb-de28-6c7ab7fb0c33"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 28, 28) (200000,)\n",
"Validation set (10000, 28, 28) (10000,)\n",
"Test set (18724, 28, 28) (18724,)\n"
]
}
],
"source": [
"pickle_file = 'notMNIST.pickle'\n",
"\n",
"with open(pickle_file, 'rb') as f:\n",
" save = pickle.load(f)\n",
" train_dataset = save['train_dataset']\n",
" train_labels = save['train_labels']\n",
" valid_dataset = save['valid_dataset']\n",
" valid_labels = save['valid_labels']\n",
" test_dataset = save['test_dataset']\n",
" test_labels = save['test_labels']\n",
" del save # hint to help gc free up memory\n",
" print 'Training set', train_dataset.shape, train_labels.shape\n",
" print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
" print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "L7aHrm6nGDMB"
},
"source": [
"Reformat into a shape that's more adapted to the models we're going to train:\n",
"- data as a flat matrix,\n",
"- labels as float 1-hot encodings."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 19723,
"status": "ok",
"timestamp": 1449847956364,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "IRSyYiIIGIzS",
"outputId": "2ba0fc75-1487-4ace-a562-cf81cae82793"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 784) (200000, 10)\n",
"Validation set (10000, 784) (10000, 10)\n",
"Test set (18724, 784) (18724, 10)\n"
]
}
],
"source": [
"image_size = 28\n",
"num_labels = 10\n",
"\n",
"def reformat(dataset, labels):\n",
" dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)\n",
" # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]\n",
" labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
" return dataset, labels\n",
"train_dataset, train_labels = reformat(train_dataset, train_labels)\n",
"valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)\n",
"test_dataset, test_labels = reformat(test_dataset, test_labels)\n",
"print 'Training set', train_dataset.shape, train_labels.shape\n",
"print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
"print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "nCLVqyQ5vPPH"
},
"source": [
"We're first going to train a multinomial logistic regression using simple gradient descent.\n",
"\n",
"TensorFlow works like this:\n",
"* First you describe the computation that you want to see performed: what the inputs, the variables, and the operations look like. These get created as nodes over a computation graph. This description is all contained within the block below:\n",
"\n",
" with graph.as_default():\n",
" ...\n",
"\n",
"* Then you can run the operations on this graph as many times as you want by calling `session.run()`, providing it outputs to fetch from the graph that get returned. This runtime operation is all contained in the block below:\n",
"\n",
" with tf.Session(graph=graph) as session:\n",
" ...\n",
"\n",
"Let's load all the data into TensorFlow and build the computation graph corresponding to our training:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "Nfv39qvtvOl_"
},
"outputs": [],
"source": [
"# With gradient descent training, even this much data is prohibitive.\n",
"# Subset the training data for faster turnaround.\n",
"train_subset = 10000\n",
"\n",
"graph = tf.Graph()\n",
"with graph.as_default():\n",
"\n",
" # Input data.\n",
" # Load the training, validation and test data into constants that are\n",
" # attached to the graph.\n",
" tf_train_dataset = tf.constant(train_dataset[:train_subset, :])\n",
" tf_train_labels = tf.constant(train_labels[:train_subset])\n",
" tf_valid_dataset = tf.constant(valid_dataset)\n",
" tf_test_dataset = tf.constant(test_dataset)\n",
" \n",
" # Variables.\n",
" # These are the parameters that we are going to be training. The weight\n",
" # matrix will be initialized using random valued following a (truncated)\n",
" # normal distribution. The biases get initialized to zero.\n",
" weights = tf.Variable(\n",
" tf.truncated_normal([image_size * image_size, num_labels]))\n",
" biases = tf.Variable(tf.zeros([num_labels]))\n",
" \n",
" # Training computation.\n",
" # We multiply the inputs with the weight matrix, and add biases. We compute\n",
" # the softmax and cross-entropy (it's one operation in TensorFlow, because\n",
" # it's very common, and it can be optimized). We take the average of this\n",
" # cross-entropy across all training examples: that's our loss.\n",
" logits = tf.matmul(tf_train_dataset, weights) + biases\n",
" loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))\n",
" \n",
" # Optimizer.\n",
" # We are going to find the minimum of this loss using gradient descent.\n",
" optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
" \n",
" # Predictions for the training, validation, and test data.\n",
" # These are not part of training, but merely here so that we can report\n",
" # accuracy figures as we train.\n",
" train_prediction = tf.nn.softmax(logits)\n",
" valid_prediction = tf.nn.softmax(\n",
" tf.matmul(tf_valid_dataset, weights) + biases)\n",
" test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "KQcL4uqISHjP"
},
"source": [
"Let's run this computation and iterate:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 9
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 57454,
"status": "ok",
"timestamp": 1449847994134,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "z2cjdenH869W",
"outputId": "4c037ba1-b526-4d8e-e632-91e2a0333267"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized\n",
"Loss at step 0 : 17.2939\n",
"Training accuracy: 10.8%\n",
"Validation accuracy: 13.8%\n",
"Loss at step 100 : 2.26903\n",
"Training accuracy: 72.3%\n",
"Validation accuracy: 71.6%\n",
"Loss at step 200 : 1.84895\n",
"Training accuracy: 74.9%\n",
"Validation accuracy: 73.9%\n",
"Loss at step 300 : 1.60701\n",
"Training accuracy: 76.0%\n",
"Validation accuracy: 74.5%\n",
"Loss at step 400 : 1.43912\n",
"Training accuracy: 76.8%\n",
"Validation accuracy: 74.8%\n",
"Loss at step 500 : 1.31349\n",
"Training accuracy: 77.5%\n",
"Validation accuracy: 75.0%\n",
"Loss at step 600 : 1.21501\n",
"Training accuracy: 78.1%\n",
"Validation accuracy: 75.4%\n",
"Loss at step 700 : 1.13515\n",
"Training accuracy: 78.6%\n",
"Validation accuracy: 75.4%\n",
"Loss at step 800 : 1.0687\n",
"Training accuracy: 79.2%\n",
"Validation accuracy: 75.6%\n",
"Test accuracy: 82.9%\n"
]
}
],
"source": [
"num_steps = 801\n",
"\n",
"def accuracy(predictions, labels):\n",
" return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))\n",
" / predictions.shape[0])\n",
"\n",
"with tf.Session(graph=graph) as session:\n",
" # This is a one-time operation which ensures the parameters get initialized as\n",
" # we described in the graph: random weights for the matrix, zeros for the\n",
" # biases. \n",
" tf.initialize_all_variables().run()\n",
" print 'Initialized'\n",
" for step in xrange(num_steps):\n",
" # Run the computations. We tell .run() that we want to run the optimizer,\n",
" # and get the loss value and the training predictions returned as numpy\n",
" # arrays.\n",
" _, l, predictions = session.run([optimizer, loss, train_prediction])\n",
" if (step % 100 == 0):\n",
" print 'Loss at step', step, ':', l\n",
" print 'Training accuracy: %.1f%%' % accuracy(\n",
" predictions, train_labels[:train_subset, :])\n",
" # Calling .eval() on valid_prediction is basically like calling run(), but\n",
" # just to get that one numpy array. Note that it recomputes all its graph\n",
" # dependencies.\n",
" print 'Validation accuracy: %.1f%%' % accuracy(\n",
" valid_prediction.eval(), valid_labels)\n",
" print 'Test accuracy: %.1f%%' % accuracy(test_prediction.eval(), test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "x68f-hxRGm3H"
},
"source": [
"Let's now switch to stochastic gradient descent training instead, which is much faster.\n",
"\n",
"The graph will be similar, except that instead of holding all the training data into a constant node, we create a `Placeholder` node which will be fed actual data at every call of `sesion.run()`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "qhPMzWYRGrzM"
},
"outputs": [],
"source": [
"batch_size = 128\n",
"\n",
"graph = tf.Graph()\n",
"with graph.as_default():\n",
"\n",
" # Input data. For the training data, we use a placeholder that will be fed\n",
" # at run time with a training minibatch.\n",
" tf_train_dataset = tf.placeholder(tf.float32,\n",
" shape=(batch_size, image_size * image_size))\n",
" tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))\n",
" tf_valid_dataset = tf.constant(valid_dataset)\n",
" tf_test_dataset = tf.constant(test_dataset)\n",
" \n",
" # Variables.\n",
" weights = tf.Variable(\n",
" tf.truncated_normal([image_size * image_size, num_labels]))\n",
" biases = tf.Variable(tf.zeros([num_labels]))\n",
" \n",
" # Training computation.\n",
" logits = tf.matmul(tf_train_dataset, weights) + biases\n",
" loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))\n",
" \n",
" # Optimizer.\n",
" optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)\n",
" \n",
" # Predictions for the training, validation, and test data.\n",
" train_prediction = tf.nn.softmax(logits)\n",
" valid_prediction = tf.nn.softmax(\n",
" tf.matmul(tf_valid_dataset, weights) + biases)\n",
" test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "XmVZESmtG4JH"
},
"source": [
"Let's run it:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 6
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 66292,
"status": "ok",
"timestamp": 1449848003013,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "FoF91pknG_YW",
"outputId": "d255c80e-954d-4183-ca1c-c7333ce91d0a"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized\n",
"Minibatch loss at step 0 : 16.8091\n",
"Minibatch accuracy: 12.5%\n",
"Validation accuracy: 14.0%\n",
"Minibatch loss at step 500 : 1.75256\n",
"Minibatch accuracy: 77.3%\n",
"Validation accuracy: 75.0%\n",
"Minibatch loss at step 1000 : 1.32283\n",
"Minibatch accuracy: 77.3%\n",
"Validation accuracy: 76.6%\n",
"Minibatch loss at step 1500 : 0.944533\n",
"Minibatch accuracy: 83.6%\n",
"Validation accuracy: 76.5%\n",
"Minibatch loss at step 2000 : 1.03795\n",
"Minibatch accuracy: 78.9%\n",
"Validation accuracy: 77.8%\n",
"Minibatch loss at step 2500 : 1.10219\n",
"Minibatch accuracy: 80.5%\n",
"Validation accuracy: 78.0%\n",
"Minibatch loss at step 3000 : 0.758874\n",
"Minibatch accuracy: 82.8%\n",
"Validation accuracy: 78.8%\n",
"Test accuracy: 86.1%\n"
]
}
],
"source": [
"num_steps = 3001\n",
"\n",
"with tf.Session(graph=graph) as session:\n",
" tf.initialize_all_variables().run()\n",
" print \"Initialized\"\n",
" for step in xrange(num_steps):\n",
" # Pick an offset within the training data, which has been randomized.\n",
" # Note: we could use better randomization across epochs.\n",
" offset = (step * batch_size) % (train_labels.shape[0] - batch_size)\n",
" # Generate a minibatch.\n",
" batch_data = train_dataset[offset:(offset + batch_size), :]\n",
" batch_labels = train_labels[offset:(offset + batch_size), :]\n",
" # Prepare a dictionary telling the session where to feed the minibatch.\n",
" # The key of the dictionary is the placeholder node of the graph to be fed,\n",
" # and the value is the numpy array to feed to it.\n",
" feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}\n",
" _, l, predictions = session.run(\n",
" [optimizer, loss, train_prediction], feed_dict=feed_dict)\n",
" if (step % 500 == 0):\n",
" print \"Minibatch loss at step\", step, \":\", l\n",
" print \"Minibatch accuracy: %.1f%%\" % accuracy(predictions, batch_labels)\n",
" print \"Validation accuracy: %.1f%%\" % accuracy(\n",
" valid_prediction.eval(), valid_labels)\n",
" print \"Test accuracy: %.1f%%\" % accuracy(test_prediction.eval(), test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "7omWxtvLLxik"
},
"source": [
"---\n",
"Problem\n",
"-------\n",
"\n",
"Turn the logistic regression example with SGD into a 1-hidden layer neural network with rectified linear units (nn.relu()) and 1024 hidden nodes. This model should improve your validation / test accuracy.\n",
"\n",
"---"
]
}
],
"metadata": {
"colabVersion": "0.3.2",
"colab_default_view": {},
"colab_views": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

323
deep-learning/tensor-flow-exercises/3_regularization.ipynb Normal file
View File

@ -0,0 +1,323 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "kR-4eNdK6lYS"
},
"source": [
"Deep Learning with TensorFlow\n",
"=============\n",
"\n",
"Credits: Forked from [TensorFlow](https://github.com/tensorflow/tensorflow) by Google\n",
"\n",
"Setup\n",
"------------\n",
"\n",
"Refer to the [setup instructions](https://github.com/donnemartin/data-science-ipython-notebooks/tree/feature/deep-learning/deep-learning/tensor-flow-exercises/README.md).\n",
"\n",
"Exercise 3\n",
"------------\n",
"\n",
"Previously in `2_fullyconnected.ipynb`, you trained a logistic regression and a neural network model.\n",
"\n",
"The goal of this exercise is to explore regularization techniques."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "JLpLa8Jt7Vu4"
},
"outputs": [],
"source": [
"# These are all the modules we'll be using later. Make sure you can import them\n",
"# before proceeding further.\n",
"import cPickle as pickle\n",
"import numpy as np\n",
"import tensorflow as tf"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "1HrCK6e17WzV"
},
"source": [
"First reload the data we generated in _notmist.ipynb_."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 11777,
"status": "ok",
"timestamp": 1449849322348,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "y3-cj1bpmuxc",
"outputId": "e03576f1-ebbe-4838-c388-f1777bcc9873"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 28, 28) (200000,)\n",
"Validation set (10000, 28, 28) (10000,)\n",
"Test set (18724, 28, 28) (18724,)\n"
]
}
],
"source": [
"pickle_file = 'notMNIST.pickle'\n",
"\n",
"with open(pickle_file, 'rb') as f:\n",
" save = pickle.load(f)\n",
" train_dataset = save['train_dataset']\n",
" train_labels = save['train_labels']\n",
" valid_dataset = save['valid_dataset']\n",
" valid_labels = save['valid_labels']\n",
" test_dataset = save['test_dataset']\n",
" test_labels = save['test_labels']\n",
" del save # hint to help gc free up memory\n",
" print 'Training set', train_dataset.shape, train_labels.shape\n",
" print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
" print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "L7aHrm6nGDMB"
},
"source": [
"Reformat into a shape that's more adapted to the models we're going to train:\n",
"- data as a flat matrix,\n",
"- labels as float 1-hot encodings."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 11728,
"status": "ok",
"timestamp": 1449849322356,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "IRSyYiIIGIzS",
"outputId": "3f8996ee-3574-4f44-c953-5c8a04636582"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 784) (200000, 10)\n",
"Validation set (10000, 784) (10000, 10)\n",
"Test set (18724, 784) (18724, 10)\n"
]
}
],
"source": [
"image_size = 28\n",
"num_labels = 10\n",
"\n",
"def reformat(dataset, labels):\n",
" dataset = dataset.reshape((-1, image_size * image_size)).astype(np.float32)\n",
" # Map 2 to [0.0, 1.0, 0.0 ...], 3 to [0.0, 0.0, 1.0 ...]\n",
" labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
" return dataset, labels\n",
"train_dataset, train_labels = reformat(train_dataset, train_labels)\n",
"valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)\n",
"test_dataset, test_labels = reformat(test_dataset, test_labels)\n",
"print 'Training set', train_dataset.shape, train_labels.shape\n",
"print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
"print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "RajPLaL_ZW6w"
},
"outputs": [],
"source": [
"def accuracy(predictions, labels):\n",
" return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))\n",
" / predictions.shape[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "sgLbUAQ1CW-1"
},
"source": [
"---\n",
"Problem 1\n",
"---------\n",
"\n",
"Introduce and tune L2 regularization for both logistic and neural network models. Remember that L2 amounts to adding a penalty on the norm of the weights to the loss. In TensorFlow, you can compue the L2 loss for a tensor `t` using `nn.l2_loss(t)`. The right amount of regularization should improve your validation / test accuracy.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "na8xX2yHZzNF"
},
"source": [
"---\n",
"Problem 2\n",
"---------\n",
"Let's demonstrate an extreme case of overfitting. Restrict your training data to just a few batches. What happens?\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "ww3SCBUdlkRc"
},
"source": [
"---\n",
"Problem 3\n",
"---------\n",
"Introduce Dropout on the hidden layer of the neural network. Remember: Dropout should only be introduced during training, not evaluation, otherwise your evaluation results would be stochastic as well. TensorFlow provides `nn.dropout()` for that, but you have to make sure it's only inserted during training.\n",
"\n",
"What happens to our extreme overfitting case?\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "-b1hTz3VWZjw"
},
"source": [
"---\n",
"Problem 4\n",
"---------\n",
"\n",
"Try to get the best performance you can using a multi-layer model! The best reported test accuracy using a deep network is [97.1%](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html?showComment=1391023266211#c8758720086795711595).\n",
"\n",
"One avenue you can explore is to add multiple layers.\n",
"\n",
"Another one is to use learning rate decay:\n",
"\n",
" global_step = tf.Variable(0) # count the number of steps taken.\n",
" learning_rate = tf.train.exponential_decay(0.5, step, ...)\n",
" optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss, global_step=global_step)\n",
" \n",
" ---\n"
]
}
],
"metadata": {
"colabVersion": "0.3.2",
"colab_default_view": {},
"colab_views": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

489
deep-learning/tensor-flow-exercises/4_convolutions.ipynb Normal file
View File

@ -0,0 +1,489 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "4embtkV0pNxM"
},
"source": [
"Deep Learning with TensorFlow\n",
"=============\n",
"\n",
"Credits: Forked from [TensorFlow](https://github.com/tensorflow/tensorflow) by Google\n",
"\n",
"Setup\n",
"------------\n",
"\n",
"Refer to the [setup instructions](https://github.com/donnemartin/data-science-ipython-notebooks/tree/feature/deep-learning/deep-learning/tensor-flow-exercises/README.md).\n",
"\n",
"Exercise 4\n",
"------------\n",
"\n",
"Previously in `2_fullyconnected.ipynb` and `3_regularization.ipynb`, we trained fully connected networks to classify [notMNIST](http://yaroslavvb.blogspot.com/2011/09/notmnist-dataset.html) characters.\n",
"\n",
"The goal of this exercise is make the neural network convolutional."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "tm2CQN_Cpwj0"
},
"outputs": [],
"source": [
"# These are all the modules we'll be using later. Make sure you can import them\n",
"# before proceeding further.\n",
"import cPickle as pickle\n",
"import numpy as np\n",
"import tensorflow as tf"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 11948,
"status": "ok",
"timestamp": 1446658914837,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "y3-cj1bpmuxc",
"outputId": "016b1a51-0290-4b08-efdb-8c95ffc3cd01"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 28, 28) (200000,)\n",
"Validation set (10000, 28, 28) (10000,)\n",
"Test set (18724, 28, 28) (18724,)\n"
]
}
],
"source": [
"pickle_file = 'notMNIST.pickle'\n",
"\n",
"with open(pickle_file, 'rb') as f:\n",
" save = pickle.load(f)\n",
" train_dataset = save['train_dataset']\n",
" train_labels = save['train_labels']\n",
" valid_dataset = save['valid_dataset']\n",
" valid_labels = save['valid_labels']\n",
" test_dataset = save['test_dataset']\n",
" test_labels = save['test_labels']\n",
" del save # hint to help gc free up memory\n",
" print 'Training set', train_dataset.shape, train_labels.shape\n",
" print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
" print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "L7aHrm6nGDMB"
},
"source": [
"Reformat into a TensorFlow-friendly shape:\n",
"- convolutions need the image data formatted as a cube (width by height by #channels)\n",
"- labels as float 1-hot encodings."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 1
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 11952,
"status": "ok",
"timestamp": 1446658914857,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "IRSyYiIIGIzS",
"outputId": "650a208c-8359-4852-f4f5-8bf10e80ef6c"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training set (200000, 28, 28, 1) (200000, 10)\n",
"Validation set (10000, 28, 28, 1) (10000, 10)\n",
"Test set (18724, 28, 28, 1) (18724, 10)\n"
]
}
],
"source": [
"image_size = 28\n",
"num_labels = 10\n",
"num_channels = 1 # grayscale\n",
"\n",
"import numpy as np\n",
"\n",
"def reformat(dataset, labels):\n",
" dataset = dataset.reshape(\n",
" (-1, image_size, image_size, num_channels)).astype(np.float32)\n",
" labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)\n",
" return dataset, labels\n",
"train_dataset, train_labels = reformat(train_dataset, train_labels)\n",
"valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)\n",
"test_dataset, test_labels = reformat(test_dataset, test_labels)\n",
"print 'Training set', train_dataset.shape, train_labels.shape\n",
"print 'Validation set', valid_dataset.shape, valid_labels.shape\n",
"print 'Test set', test_dataset.shape, test_labels.shape"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "AgQDIREv02p1"
},
"outputs": [],
"source": [
"def accuracy(predictions, labels):\n",
" return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))\n",
" / predictions.shape[0])"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "5rhgjmROXu2O"
},
"source": [
"Let's build a small network with two convolutional layers, followed by one fully connected layer. Convolutional networks are more expensive computationally, so we'll limit its depth and number of fully connected nodes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
}
},
"colab_type": "code",
"collapsed": true,
"id": "IZYv70SvvOan"
},
"outputs": [],
"source": [
"batch_size = 16\n",
"patch_size = 5\n",
"depth = 16\n",
"num_hidden = 64\n",
"\n",
"graph = tf.Graph()\n",
"\n",
"with graph.as_default():\n",
"\n",
" # Input data.\n",
" tf_train_dataset = tf.placeholder(\n",
" tf.float32, shape=(batch_size, image_size, image_size, num_channels))\n",
" tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))\n",
" tf_valid_dataset = tf.constant(valid_dataset)\n",
" tf_test_dataset = tf.constant(test_dataset)\n",
" \n",
" # Variables.\n",
" layer1_weights = tf.Variable(tf.truncated_normal(\n",
" [patch_size, patch_size, num_channels, depth], stddev=0.1))\n",
" layer1_biases = tf.Variable(tf.zeros([depth]))\n",
" layer2_weights = tf.Variable(tf.truncated_normal(\n",
" [patch_size, patch_size, depth, depth], stddev=0.1))\n",
" layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))\n",
" layer3_weights = tf.Variable(tf.truncated_normal(\n",
" [image_size / 4 * image_size / 4 * depth, num_hidden], stddev=0.1))\n",
" layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))\n",
" layer4_weights = tf.Variable(tf.truncated_normal(\n",
" [num_hidden, num_labels], stddev=0.1))\n",
" layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))\n",
" \n",
" # Model.\n",
" def model(data):\n",
" conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME')\n",
" hidden = tf.nn.relu(conv + layer1_biases)\n",
" conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME')\n",
" hidden = tf.nn.relu(conv + layer2_biases)\n",
" shape = hidden.get_shape().as_list()\n",
" reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])\n",
" hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)\n",
" return tf.matmul(hidden, layer4_weights) + layer4_biases\n",
" \n",
" # Training computation.\n",
" logits = model(tf_train_dataset)\n",
" loss = tf.reduce_mean(\n",
" tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))\n",
" \n",
" # Optimizer.\n",
" optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)\n",
" \n",
" # Predictions for the training, validation, and test data.\n",
" train_prediction = tf.nn.softmax(logits)\n",
" valid_prediction = tf.nn.softmax(model(tf_valid_dataset))\n",
" test_prediction = tf.nn.softmax(model(tf_test_dataset))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "both",
"colab": {
"autoexec": {
"startup": false,
"wait_interval": 0
},
"output_extras": [
{
"item_id": 37
}
]
},
"colab_type": "code",
"collapsed": false,
"executionInfo": {
"elapsed": 63292,
"status": "ok",
"timestamp": 1446658966251,
"user": {
"color": "",
"displayName": "",
"isAnonymous": false,
"isMe": true,
"permissionId": "",
"photoUrl": "",
"sessionId": "0",
"userId": ""
},
"user_tz": 480
},
"id": "noKFb2UovVFR",
"outputId": "28941338-2ef9-4088-8bd1-44295661e628"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Initialized\n",
"Minibatch loss at step 0 : 3.51275\n",
"Minibatch accuracy: 6.2%\n",
"Validation accuracy: 12.8%\n",
"Minibatch loss at step 50 : 1.48703\n",
"Minibatch accuracy: 43.8%\n",
"Validation accuracy: 50.4%\n",
"Minibatch loss at step 100 : 1.04377\n",
"Minibatch accuracy: 68.8%\n",
"Validation accuracy: 67.4%\n",
"Minibatch loss at step 150 : 0.601682\n",
"Minibatch accuracy: 68.8%\n",
"Validation accuracy: 73.0%\n",
"Minibatch loss at step 200 : 0.898649\n",
"Minibatch accuracy: 75.0%\n",
"Validation accuracy: 77.8%\n",
"Minibatch loss at step 250 : 1.3637\n",
"Minibatch accuracy: 56.2%\n",
"Validation accuracy: 75.4%\n",
"Minibatch loss at step 300 : 1.41968\n",
"Minibatch accuracy: 62.5%\n",
"Validation accuracy: 76.0%\n",
"Minibatch loss at step 350 : 0.300648\n",
"Minibatch accuracy: 81.2%\n",
"Validation accuracy: 80.2%\n",
"Minibatch loss at step 400 : 1.32092\n",
"Minibatch accuracy: 56.2%\n",
"Validation accuracy: 80.4%\n",
"Minibatch loss at step 450 : 0.556701\n",
"Minibatch accuracy: 81.2%\n",
"Validation accuracy: 79.4%\n",
"Minibatch loss at step 500 : 1.65595\n",
"Minibatch accuracy: 43.8%\n",
"Validation accuracy: 79.6%\n",
"Minibatch loss at step 550 : 1.06995\n",
"Minibatch accuracy: 75.0%\n",
"Validation accuracy: 81.2%\n",
"Minibatch loss at step 600 : 0.223684\n",
"Minibatch accuracy: 100.0%\n",
"Validation accuracy: 82.3%\n",
"Minibatch loss at step 650 : 0.619602\n",
"Minibatch accuracy: 87.5%\n",
"Validation accuracy: 81.8%\n",
"Minibatch loss at step 700 : 0.812091\n",
"Minibatch accuracy: 75.0%\n",
"Validation accuracy: 82.4%\n",
"Minibatch loss at step 750 : 0.276302\n",
"Minibatch accuracy: 87.5%\n",
"Validation accuracy: 82.3%\n",
"Minibatch loss at step 800 : 0.450241\n",
"Minibatch accuracy: 81.2%\n",
"Validation accuracy: 82.3%\n",
"Minibatch loss at step 850 : 0.137139\n",
"Minibatch accuracy: 93.8%\n",
"Validation accuracy: 82.3%\n",
"Minibatch loss at step 900 : 0.52664\n",
"Minibatch accuracy: 75.0%\n",
"Validation accuracy: 82.2%\n",
"Minibatch loss at step 950 : 0.623835\n",
"Minibatch accuracy: 87.5%\n",
"Validation accuracy: 82.1%\n",
"Minibatch loss at step 1000 : 0.243114\n",
"Minibatch accuracy: 93.8%\n",
"Validation accuracy: 82.9%\n",
"Test accuracy: 90.0%\n"
]
}
],
"source": [
"num_steps = 1001\n",
"\n",
"with tf.Session(graph=graph) as session:\n",
" tf.initialize_all_variables().run()\n",
" print \"Initialized\"\n",
" for step in xrange(num_steps):\n",
" offset = (step * batch_size) % (train_labels.shape[0] - batch_size)\n",
" batch_data = train_dataset[offset:(offset + batch_size), :, :, :]\n",
" batch_labels = train_labels[offset:(offset + batch_size), :]\n",
" feed_dict = {tf_train_dataset : batch_data, tf_train_labels : batch_labels}\n",
" _, l, predictions = session.run(\n",
" [optimizer, loss, train_prediction], feed_dict=feed_dict)\n",
" if (step % 50 == 0):\n",
" print \"Minibatch loss at step\", step, \":\", l\n",
" print \"Minibatch accuracy: %.1f%%\" % accuracy(predictions, batch_labels)\n",
" print \"Validation accuracy: %.1f%%\" % accuracy(\n",
" valid_prediction.eval(), valid_labels)\n",
" print \"Test accuracy: %.1f%%\" % accuracy(test_prediction.eval(), test_labels)"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "KedKkn4EutIK"
},
"source": [
"---\n",
"Problem 1\n",
"---------\n",
"\n",
"The convolutional model above uses convolutions with stride 2 to reduce the dimensionality. Replace the strides a max pooling operation (`nn.max_pool()`) of stride 2 and kernel size 2.\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"id": "klf21gpbAgb-"
},
"source": [
"---\n",
"Problem 2\n",
"---------\n",
"\n",
"Try to get the best performance you can using a convolutional net. Look for example at the classic [LeNet5](http://yann.lecun.com/exdb/lenet/) architecture, adding Dropout, and/or adding learning rate decay.\n",
"\n",
"---"
]
}
],
"metadata": {
"colabVersion": "0.3.2",
"colab_default_view": {},
"colab_views": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}

889
deep-learning/tensor-flow-exercises/5_word2vec.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

1064
deep-learning/tensor-flow-exercises/6_lstm.ipynb Normal file
View File

File diff suppressed because it is too large Load Diff

6
deep-learning/tensor-flow-exercises/Dockerfile Normal file
View File

@ -0,0 +1,6 @@
FROM b.gcr.io/tensorflow/tensorflow:latest
MAINTAINER Vincent Vanhoucke <vanhoucke@google.com>
RUN pip install scikit-learn
ADD *.ipynb /notebooks/
WORKDIR /notebooks
CMD ["/run_jupyter.sh"]

12
deep-learning/tensor-flow-exercises/README.md Normal file
View File

@ -0,0 +1,12 @@
Exercises
===========================================================
Building the Docker container
-----------------------------
docker build -t $USER/exercises .
Running the container
---------------------
docker run -p 8888:8888 -it --rm $USER/exercises