data-science-ipython-notebooks/deep-learning/keras-tutorial/2.2.2 Supervised Learning - ConvNet HandsOn Part II.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Credits: Forked from [deep-learning-keras-tensorflow](https://github.com/leriomaggio/deep-learning-keras-tensorflow) by Valerio Maggiohttps://github.com/donnemartin/system-design-primer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolution Nets for MNIST"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Deep Learning models can take quite a bit of time to run, particularly if GPU isn't used. \n",
    "\n",
    "In the interest of time, you could sample a subset of observations (e.g. $1000$) that are a particular number of your choice (e.g. $6$) and $1000$ observations that aren't that particular number (i.e. $\\neq 6$). \n",
    "\n",
    "We will build a model using that and see how it performs on the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using Theano backend.\n",
      "Using gpu device 0: GeForce GTX 760 (CNMeM is enabled with initial size: 90.0% of memory, cuDNN 4007)\n"
     ]
    }
   ],
   "source": [
    "#Import the required libraries\n",
    "import numpy as np\n",
    "np.random.seed(1338)\n",
    "\n",
    "from keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.layers.core import Dense, Dropout, Activation, Flatten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.layers.convolutional import Convolution2D\n",
    "from keras.layers.pooling import MaxPooling2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "from keras.optimizers import SGD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Loading Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "path_to_dataset = \"euroscipy_2016_dl-keras/data/mnist.pkl.gz\"\n",
    "\n",
    "#Load the training and testing data\n",
    "(X_train, y_train), (X_test, y_test) = mnist.load_data(path_to_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "X_test_orig = X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Data Preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)\n",
    "X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)\n",
    "\n",
    "X_train = X_train.astype('float32')\n",
    "X_test = X_test.astype('float32')\n",
    "\n",
    "X_train /= 255\n",
    "X_test /= 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Seed for reproducibilty\n",
    "np.random.seed(1338)\n",
    "\n",
    "# Test data\n",
    "X_test = X_test.copy()\n",
    "Y = y_test.copy()\n",
    "\n",
    "# Converting the output to binary classification(Six=1,Not Six=0)\n",
    "Y_test = Y == 6\n",
    "Y_test = Y_test.astype(int)\n",
    "\n",
    "# Selecting the 5918 examples where the output is 6\n",
    "X_six = X_train[y_train == 6].copy()\n",
    "Y_six = y_train[y_train == 6].copy()\n",
    "\n",
    "# Selecting the examples where the output is not 6\n",
    "X_not_six = X_train[y_train != 6].copy()\n",
    "Y_not_six = y_train[y_train != 6].copy()\n",
    "\n",
    "# Selecting 6000 random examples from the data that \n",
    "# only contains the data where the output is not 6\n",
    "random_rows = np.random.randint(0,X_six.shape[0],6000)\n",
    "X_not_six = X_not_six[random_rows]\n",
    "Y_not_six = Y_not_six[random_rows]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Appending the data with output as 6 and data with output as <> 6\n",
    "X_train = np.append(X_six,X_not_six)\n",
    "\n",
    "# Reshaping the appended data to appropraite form\n",
    "X_train = X_train.reshape(X_six.shape[0] + X_not_six.shape[0], \n",
    "                          1, img_rows, img_cols)\n",
    "\n",
    "# Appending the labels and converting the labels to \n",
    "# binary classification(Six=1,Not Six=0)\n",
    "Y_labels = np.append(Y_six,Y_not_six)\n",
    "Y_train = Y_labels == 6 \n",
    "Y_train = Y_train.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(11918, 1, 28, 28) (11918,) (10000, 1, 28, 28) (10000, 2)\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape, Y_labels.shape, X_test.shape, Y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Converting the classes to its binary categorical form\n",
    "nb_classes = 2\n",
    "Y_train = np_utils.to_categorical(Y_train, nb_classes)\n",
    "Y_test = np_utils.to_categorical(Y_test, nb_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A simple CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "#Initializing the values for the convolution neural network\n",
    "nb_epoch = 2\n",
    "batch_size = 128\n",
    "# number of convolutional filters to use\n",
    "nb_filters = 32\n",
    "# size of pooling area for max pooling\n",
    "nb_pool = 2\n",
    "# convolution kernel size\n",
    "nb_conv = 3\n",
    "\n",
    "sgd = SGD(lr=0.1, decay=1e-6, momentum=0.9, nesterov=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 1: Model Definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 2: Compile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 3: Fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.2890 - acc: 0.9326 - val_loss: 0.1251 - val_acc: 0.9722\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.1341 - acc: 0.9612 - val_loss: 0.1298 - val_acc: 0.9599\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6ccb68f630>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Step 4: Evaluate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.129807630396\n",
      "Test accuracy: 0.9599\n"
     ]
    }
   ],
   "source": [
    "# Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Let's plot our model Predictions!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA50AAABVCAYAAADKf4AOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXm4VmW5/z+PgMIJA4UABYckPZoYaJSWeEmWph3MMicQ\nTpAoTimCx711b3NhbGWTECAKKhYdUBQnNAtSK0xIJcejRqeccjghDj9NTAXx+f1x32tP7I17eOf3\n+7mufT3rfdaz1nvfe71reu4pxBgRQgghhBBCCCGywTb5FkAIIYQQQgghROmil04hhBBCCCGEEFkj\nKy+d3bp1WxdCiG3969at27psyJNtpK/0lc7St5iQvtJXOkvfYkL6St9S0rnc9E0J2YjpDCHE9uw3\nhECMMWRcoCwjfVu9XVnp69uWlc7StziQvq3erqz09W3LSmfpWxxI31ZvV1b6+rZFp3O56Zsi91oh\nhBBCCCGEEFlDL51CCCGEEEIIIbJGzl46V6xYwd57781ee+1FbW1trr42b5xyyin07duXL3zhC/kW\nJWeU2zGWvqVNuekL5aez9C1tpG9po+es0j/G0rfEiDFm/M92W8/mzZvjwIED44svvhg3btwYBw8e\nHNeuXRub4ttlRaZs/jXVN8YYH3jggfj444/H/fbbb4t1pahva45xuelbSjpL39LWt7U6S9/i+NM1\nS/qWur4x6jlL16zi1bnc9E3/cmLpXLNmDXvuuSe77bYbXbp04aSTTuLOO+/MxVfnjWHDhrHDDjvk\nW4ycUW7HWPpK31Kj3HSWvtK3lCg3fUHPWaV+jKVv6embk5fOV199lV122aXu84ABA3j11Vdz8dUi\nR5TbMZa+0rfUKDedpa/0LSXKTd9ypNyOsfQtPX1z8tLppuRGhFCU2X5FC5TbMZa+0rfUKDedpa/0\nLSXKTd9ypNyOsfQtPX1z8tI5YMAAXnrppbrPr7zyCjvvvHMuvlrkiHI7xtJX+pYa5aaz9JW+pUS5\n6VuOlNsxlr4lqG8uAmQ/+uijuuDYDz/8MA4ePDj++c9/Lpng2Kb6przwwgtx0KBBza4rNX1bc4zL\nTd9S0ln6lra+rdVZ+hbHn65Z0rfU9U3Rc5auWcWoc7npm/51zsqbbBM6derE3LlzOeKII/j44485\n5ZRT2GeffZodG0KypX25QIgxabWde9SoUaxcuZI333yTXXfdlSlTpjBu3LgtxpWKvq09xuWmL5SG\nztK3tPUFncPSt55S0Fn6lra+oOcsXbPqKQWdy0Hf4G/cGSWEENuzX/NdviTj8mSKlv6R0rd1lJu+\nvi3lpLP0LSykryF9W0+56Sx9Cwvpa0jf1lOMOpebvik5iekUQgghhBBCCFGe6KVTCCGEEEIIIUTW\nyElMp2g7J8TPArALLwNwxcsXA5DsWj/mG/EAAI56bwUAG7pflUMJhRBCCCGKld0BiBMsDrT2Guut\neBzC/oXrvihEsSJLpxBCCCGEEEKIrCFLZ4ERX5sCQG2TMNxab7s16FsdHgPg3aF9AAikQclJ1uQr\nCPolAExdZ/+ke+NyAO4PD+VLogxhlusH4pS6Y1txrq0Js8vk2AohSoyjYdgXm1+16mEAkni7tc/4\nne50YFWSdclEC5yfWHvFTGv3nsTotdcBsGj2aQCMOffaRpss/q9TfZskBwJ2lN0BeCBeCcBMf97q\n4mvPH/Jj4KOcSyVEtqiOG+kULgcgmW994fTcW/OzYuns2rXrayEE2vq33Xa9siFO1pG+0lc6S99i\nQvpKX+ksfYsJ6St9S0nnctM3JSslUxp9QQHXkmkrram10xF942tTqO3b/LqKI6y95TcjOP6JuwGo\n3b/xmL/HGQDMC/9srwiN5cmyvu3mWZudmfE5mzOZ/JCLcFDSod22tpZS1nTunQAw+416Md739s74\nWwAeDH/I6Ffm9Bgfmdh3PmNfGV5uxyzb/MSsIEB7rL4F+5u+OwHgkhEm3j11x/sJoP3nc270HW/f\nNXcXAGafZZaQicEDpNptnf+GNXcPs3bEPO9/rcUtCvb4ZomC1PfWBIBHv/d5AA44aC21Dzc/tOJQ\na2++39oXG6yrbKEkQEHqnEVyo+9w+65VXwPgNj/l0gqBrwJrPmEPe3v7BWCvJ1ycIUmbJcmFviOj\nJccYGk4BYNJS3++vfbcLk47svk3o97wl2dP3aGummefF9RWjAHgpLAHqPQkrXoOf9RkJwCnhp96b\n3n/aTiEc4zh/CsnpjfumpF+ZJJn9LpVMEUIIIYQQQgiRDxTTWQh8JwFgXt8pdV0V37U23GEzEZX3\nLPeOh4FvAxDHmkWhdqGt+ip/tP0wKKvi5psrB1rsSJ2WHbRw5p/vAxB/aJNDc0o0ad5py2cD8GYH\nrjq/nfBVvnb8gwBs06sU/lEnALDxZDv2Nd57RPg6AA9yEx2xdGafo/krZuGcd7b1nHu8xXpNZKd2\n7tMsnHHQIQCsHGG9X+vts7JvJO3cb7YYDsDKWAHAobPXECZe7etatsoWPd0TAJa+a9aD7YJ54Dx7\nnN3H7vVh9zbdrgG192dJNtEmHowTAahpYp/4i7dVR8Dh3/YPQ5ts/IE1NcPrt/n9ENvR1w7yc/ah\nJHPCZoCjuQuAf/jnMX7N4oQkL/KIbNKXU+JmAOb1tB/vlZW25m1vUwtnGtM7sy+AWT+fi78HYGCY\nkH1Rs0hTKydA9SUXATA12TZncsjSKYQQQgghhBAiaxSWpfOJBID4vM2S/f27n+Ef7AzAV857wsYs\n9LFvJzkVLatYSBSbljW0cL7jK5Mthp8R3wVgYZNZyTE1t7a4TUkwNAFg12CKd/Y6pbQqErPwOCza\nrNtvF+wBwJxTWx77x/Vm+RqeZuq98EhbMS3JmnyZY0cArpluqXjf7MCeTuIm1o/fzT/58eexDuwx\nz6yyuLeaYY27KzdYO6X7szkWqLWcDED8+V7UWok7DoimRHArbXtZ6Bkl0wze18anbSEkHdpvpuny\nxiQANu7fA6iX13Joe7AiS3MtVu4w4xjPu4WzLaQ5Cvh5y2Mq+7ddpJzRM4Hv2OK1Px8DwKnnL7aO\n7azpdekrALzVr38BWucbMCihV5jSqKvKEuITzjRLZXUS4Z4VvrZpgK7ZiaqjPbPsEC5npa+J/+3x\n+3s94j2/zJzcHeDz/BmAzf558VHpzTfJhzj54xsJALE6UDvcutI8Eg3jGwHCv7vV+u27YJmbvb9z\ns49am1UxO8Kw+Dk+H+yCc2ULYyosMTMzm3kGWxbWZUewAmBqyJ2FM0WWTiGEEEIIIYQQWaOgLJ3x\neI9r+lva87r/wVQ3Zx3sa4b/qJ1f4m7Z4SyftVmWtHNHGWREAsC5RM6943+8c2aLw68+YTJQX7uz\nbFhgzVNDrK2++FFfkeRDmg5zRLBYqDmtGDvHsxofy1EArPSA1jAi1mU+LVjmngPAzLPN0tkpWixy\neyzUX2U1M+9IP+3nbTFaOncHIC6yf0LSZG23ZQVel3X+ngB1Vk6AI8J/+9Iv2r/fQQmvuCdD1V+t\nqzL8X/v3lxUslviZ3jZnm8bhNryZvr9hXwC6dX/Se27PjWgZxyzatdGuUhW/uBLGJrbKDQCeY5gD\nzKGBx97yz+7SMGnHy5h11YX24Wy7uVfe45aR/kVy7rqXzZo/2TWnd5jCjQtt1Xpva5psMumyAYD5\neZzhSTMfumswAF8JZ/moV7MhbdsYCkvdmSD9DX/1Ncue3TrvArONpVaTWfE0+gaLkazZy4fc6nVa\njysES+cFrHDLboUftDFVSf7EySn+IDHrDAA2XmLX2pnD6+MZUxrHN8Ir0cp09F/4FrVu5U/etpP8\ng54FaOkclACwJIQW/U0ejgsBCOFfAFwZ7ZlyY7g+y8KVLwX10hkm2IPWMZMtgPfOX4zkmO/b8s/4\nAQA79LKo9ZpLbZuqnaGmhWcSvwfy/U9Z+9P3AN9uUTwOgDGhkJLuJJ84YmTclcVNHtYr/AW8svq2\nzItUQMSfu+KftaZ66l3
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f6cab246898>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "slice = 15\n",
    "predicted = model.predict(X_test[:slice]).argmax(-1)\n",
    "\n",
    "plt.figure(figsize=(16,8))\n",
    "for i in range(slice):\n",
    "    plt.subplot(1, slice, i+1)\n",
    "    plt.imshow(X_test_orig[i], interpolation='nearest')\n",
    "    plt.text(0, 0, predicted[i], color='black', \n",
    "             bbox=dict(facecolor='white', alpha=1))\n",
    "    plt.axis('off')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Dense Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.3044 - acc: 0.9379 - val_loss: 0.1469 - val_acc: 0.9625\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.1189 - acc: 0.9640 - val_loss: 0.1058 - val_acc: 0.9655\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6cf59f7358>"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.105762729073\n",
      "Test accuracy: 0.9655\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding Dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.3128 - acc: 0.9097 - val_loss: 0.1438 - val_acc: 0.9624\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 0s - loss: 0.1362 - acc: 0.9580 - val_loss: 0.1145 - val_acc: 0.9628\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6ccb180208>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.11448907243\n",
      "Test accuracy: 0.9628\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Adding more Convolution Layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n",
    "model.add(Activation('relu'))\n",
    "model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "model.add(Dropout(0.25))\n",
    "    \n",
    "model.add(Flatten())\n",
    "model.add(Dense(128))\n",
    "model.add(Activation('relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(nb_classes))\n",
    "model.add(Activation('softmax'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.4707 - acc: 0.8288 - val_loss: 0.2307 - val_acc: 0.9399\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.1882 - acc: 0.9383 - val_loss: 0.1195 - val_acc: 0.9621\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f6cc97b8748>"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "          nb_epoch=nb_epoch,verbose=1,\n",
    "          validation_data=(X_test, Y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test score: 0.11954063682\n",
      "Test accuracy: 0.9621\n"
     ]
    }
   ],
   "source": [
    "#Evaluating the model on the test data    \n",
    "score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "print('Test score:', score)\n",
    "print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise\n",
    "\n",
    "The above code has been written as a function. \n",
    "\n",
    "Change some of the **hyperparameters** and see what happens. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "# Function for constructing the convolution neural network\n",
    "# Feel free to add parameters, if you want\n",
    "\n",
    "def build_model():\n",
    "    \"\"\"\"\"\"\n",
    "    model = Sequential()\n",
    "    model.add(Convolution2D(nb_filters, nb_conv, nb_conv,\n",
    "                        border_mode='valid',\n",
    "                        input_shape=(1, img_rows, img_cols)))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Convolution2D(nb_filters, nb_conv, nb_conv))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
    "    model.add(Dropout(0.25))\n",
    "    \n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(128))\n",
    "    model.add(Activation('relu'))\n",
    "    model.add(Dropout(0.5))\n",
    "    model.add(Dense(nb_classes))\n",
    "    model.add(Activation('softmax'))\n",
    "    \n",
    "    model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='sgd',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "    model.fit(X_train, Y_train, batch_size=batch_size, \n",
    "              nb_epoch=nb_epoch,verbose=1,\n",
    "              validation_data=(X_test, Y_test))\n",
    "          \n",
    "\n",
    "    #Evaluating the model on the test data    \n",
    "    score, accuracy = model.evaluate(X_test, Y_test, verbose=0)\n",
    "    print('Test score:', score)\n",
    "    print('Test accuracy:', accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 11918 samples, validate on 10000 samples\n",
      "Epoch 1/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.5634 - acc: 0.7860 - val_loss: 0.3574 - val_acc: 0.9363\n",
      "Epoch 2/2\n",
      "11918/11918 [==============================] - 1s - loss: 0.2372 - acc: 0.9292 - val_loss: 0.2253 - val_acc: 0.9190\n",
      "Test score: 0.225333989978\n",
      "Test accuracy: 0.919\n",
      "1 loop, best of 1: 5.45 s per loop\n"
     ]
    }
   ],
   "source": [
    "#Timing how long it takes to build the model and test it.\n",
    "%timeit -n1 -r1 build_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Batch Normalisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize the activations of the previous layer at each batch, i.e. applies a transformation that maintains the mean activation close to 0 and the activation standard deviation close to 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## How to BatchNorm in Keras\n",
    "\n",
    "```python\n",
    "from keras.layers.normalization import BatchNormalization\n",
    "\n",
    "BatchNormalization(epsilon=1e-06, mode=0, \n",
    "                   axis=-1, momentum=0.99, \n",
    "                   weights=None, beta_init='zero', \n",
    "                   gamma_init='one')\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Try to add a new BatchNormalization layer to the Model \n",
    "# (after the Dropout layer)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}