data-science-ipython-notebooks/numpy/02.01-Understanding-Data-Types.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--BOOK_INFORMATION-->\n",
    "<img align=\"left\" style=\"padding-right:10px;\" src=\"figures/PDSH-cover-small.png\">\n",
    "*This notebook contains an excerpt from the [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by Jake VanderPlas; the content is available [on GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).*\n",
    "\n",
    "*The text is released under the [CC-BY-NC-ND license](https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode), and code is released under the [MIT license](https://opensource.org/licenses/MIT). If you find this content useful, please consider supporting the work by [buying the book](http://shop.oreilly.com/product/0636920034919.do)!*\n",
    "\n",
    "*No changes were made to the contents of this notebook from the original.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "< [Introduction to NumPy](02.00-Introduction-to-NumPy.ipynb) | [Contents](Index.ipynb) | [The Basics of NumPy Arrays](02.02-The-Basics-Of-NumPy-Arrays.ipynb) >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding Data Types in Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Effective data-driven science and computation requires understanding how data is stored and manipulated.\n",
    "This section outlines and contrasts how arrays of data are handled in the Python language itself, and how NumPy improves on this.\n",
    "Understanding this difference is fundamental to understanding much of the material throughout the rest of the book.\n",
    "\n",
    "Users of Python are often drawn-in by its ease of use, one piece of which is dynamic typing.\n",
    "While a statically-typed language like C or Java requires each variable to be explicitly declared, a dynamically-typed language like Python skips this specification. For example, in C you might specify a particular operation as follows:\n",
    "\n",
    "```C\n",
    "/* C code */\n",
    "int result = 0;\n",
    "for(int i=0; i<100; i++){\n",
    "    result += i;\n",
    "}\n",
    "```\n",
    "\n",
    "While in Python the equivalent operation could be written this way:\n",
    "\n",
    "```python\n",
    "# Python code\n",
    "result = 0\n",
    "for i in range(100):\n",
    "    result += i\n",
    "```\n",
    "\n",
    "Notice the main difference: in C, the data types of each variable are explicitly declared, while in Python the types are dynamically inferred. This means, for example, that we can assign any kind of data to any variable:\n",
    "\n",
    "```python\n",
    "# Python code\n",
    "x = 4\n",
    "x = \"four\"\n",
    "```\n",
    "\n",
    "Here we've switched the contents of ``x`` from an integer to a string. The same thing in C would lead (depending on compiler settings) to a compilation error or other unintented consequences:\n",
    "\n",
    "```C\n",
    "/* C code */\n",
    "int x = 4;\n",
    "x = \"four\";  // FAILS\n",
    "```\n",
    "\n",
    "This sort of flexibility is one piece that makes Python and other dynamically-typed languages convenient and easy to use.\n",
    "Understanding *how* this works is an important piece of learning to analyze data efficiently and effectively with Python.\n",
    "But what this type-flexibility also points to is the fact that Python variables are more than just their value; they also contain extra information about the type of the value. We'll explore this more in the sections that follow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Python Integer Is More Than Just an Integer\n",
    "\n",
    "The standard Python implementation is written in C.\n",
    "This means that every Python object is simply a cleverly-disguised C structure, which contains not only its value, but other information as well. For example, when we define an integer in Python, such as ``x = 10000``, ``x`` is not just a \"raw\" integer. It's actually a pointer to a compound C structure, which contains several values.\n",
    "Looking through the Python 3.4 source code, we find that the integer (long) type definition effectively looks like this (once the C macros are expanded):\n",
    "\n",
    "```C\n",
    "struct _longobject {\n",
    "    long ob_refcnt;\n",
    "    PyTypeObject *ob_type;\n",
    "    size_t ob_size;\n",
    "    long ob_digit[1];\n",
    "};\n",
    "```\n",
    "\n",
    "A single integer in Python 3.4 actually contains four pieces:\n",
    "\n",
    "- ``ob_refcnt``, a reference count that helps Python silently handle memory allocation and deallocation\n",
    "- ``ob_type``, which encodes the type of the variable\n",
    "- ``ob_size``, which specifies the size of the following data members\n",
    "- ``ob_digit``, which contains the actual integer value that we expect the Python variable to represent.\n",
    "\n",
    "This means that there is some overhead in storing an integer in Python as compared to an integer in a compiled language like C, as illustrated in the following figure:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Integer Memory Layout](figures/cint_vs_pyint.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here ``PyObject_HEAD`` is the part of the structure containing the reference count, type code, and other pieces mentioned before.\n",
    "\n",
    "Notice the difference here: a C integer is essentially a label for a position in memory whose bytes encode an integer value.\n",
    "A Python integer is a pointer to a position in memory containing all the Python object information, including the bytes that contain the integer value.\n",
    "This extra information in the Python integer structure is what allows Python to be coded so freely and dynamically.\n",
    "All this additional information in Python types comes at a cost, however, which becomes especially apparent in structures that combine many of these objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Python List Is More Than Just a List\n",
    "\n",
    "Let's consider now what happens when we use a Python data structure that holds many Python objects.\n",
    "The standard mutable multi-element container in Python is the list.\n",
    "We can create a list of integers as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L = list(range(10))\n",
    "L"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "int"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(L[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, similarly, a list of strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L2 = [str(c) for c in L]\n",
    "L2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(L2[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Because of Python's dynamic typing, we can even create heterogeneous lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[bool, str, float, int]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L3 = [True, \"2\", 3.0, 4]\n",
    "[type(item) for item in L3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But this flexibility comes at a cost: to allow these flexible types, each item in the list must contain its own type info, reference count, and other information–that is, each item is a complete Python object.\n",
    "In the special case that all variables are of the same type, much of this information is redundant: it can be much more efficient to store data in a fixed-type array.\n",
    "The difference between a dynamic-type list and a fixed-type (NumPy-style) array is illustrated in the following figure:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Array Memory Layout](figures/array_vs_list.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the implementation level, the array essentially contains a single pointer to one contiguous block of data.\n",
    "The Python list, on the other hand, contains a pointer to a block of pointers, each of which in turn points to a full Python object like the Python integer we saw earlier.\n",
    "Again, the advantage of the list is flexibility: because each list element is a full structure containing both data and type information, the list can be filled with data of any desired type.\n",
    "Fixed-type NumPy-style arrays lack this flexibility, but are much more efficient for storing and manipulating data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fixed-Type Arrays in Python\n",
    "\n",
    "Python offers several different options for storing data in efficient, fixed-type data buffers.\n",
    "The built-in ``array`` module (available since Python 3.3) can be used to create dense arrays of a uniform type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array('i', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import array\n",
    "L = list(range(10))\n",
    "A = array.array('i', L)\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here ``'i'`` is a type code indicating the contents are integers.\n",
    "\n",
    "Much more useful, however, is the ``ndarray`` object of the NumPy package.\n",
    "While Python's ``array`` object provides efficient storage of array-based data, NumPy adds to this efficient *operations* on that data.\n",
    "We will explore these operations in later sections; here we'll demonstrate several ways of creating a NumPy array.\n",
    "\n",
    "We'll start with the standard NumPy import, under the alias ``np``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Arrays from Python Lists\n",
    "\n",
    "First, we can use ``np.array`` to create arrays from Python lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 4, 2, 5, 3])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# integer array:\n",
    "np.array([1, 4, 2, 5, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that unlike Python lists, NumPy is constrained to arrays that all contain the same type.\n",
    "If types do not match, NumPy will upcast if possible (here, integers are up-cast to floating point):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 3.14,  4.  ,  2.  ,  3.  ])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([3.14, 4, 2, 3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to explicitly set the data type of the resulting array, we can use the ``dtype`` keyword:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.,  2.,  3.,  4.], dtype=float32)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array([1, 2, 3, 4], dtype='float32')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, unlike Python lists, NumPy arrays can explicitly be multi-dimensional; here's one way of initializing a multidimensional array using a list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2, 3, 4],\n",
       "       [4, 5, 6],\n",
       "       [6, 7, 8]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# nested lists result in multi-dimensional arrays\n",
    "np.array([range(i, i + 3) for i in [2, 4, 6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The inner lists are treated as rows of the resulting two-dimensional array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Arrays from Scratch\n",
    "\n",
    "Especially for larger arrays, it is more efficient to create arrays from scratch using routines built into NumPy.\n",
    "Here are several examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a length-10 integer array filled with zeros\n",
    "np.zeros(10, dtype=int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  1.,  1.,  1.,  1.],\n",
       "       [ 1.,  1.,  1.,  1.,  1.],\n",
       "       [ 1.,  1.,  1.,  1.,  1.]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 3x5 floating-point array filled with ones\n",
    "np.ones((3, 5), dtype=float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3.14,  3.14,  3.14,  3.14,  3.14],\n",
       "       [ 3.14,  3.14,  3.14,  3.14,  3.14],\n",
       "       [ 3.14,  3.14,  3.14,  3.14,  3.14]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 3x5 array filled with 3.14\n",
    "np.full((3, 5), 3.14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an array filled with a linear sequence\n",
    "# Starting at 0, ending at 20, stepping by 2\n",
    "# (this is similar to the built-in range() function)\n",
    "np.arange(0, 20, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.  ,  0.25,  0.5 ,  0.75,  1.  ])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an array of five values evenly spaced between 0 and 1\n",
    "np.linspace(0, 1, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.99844933,  0.52183819,  0.22421193],\n",
       "       [ 0.08007488,  0.45429293,  0.20941444],\n",
       "       [ 0.14360941,  0.96910973,  0.946117  ]])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 3x3 array of uniformly distributed\n",
    "# random values between 0 and 1\n",
    "np.random.random((3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.51772646,  0.39614948, -0.10634696],\n",
       "       [ 0.25671348,  0.00732722,  0.37783601],\n",
       "       [ 0.68446945,  0.15926039, -0.70744073]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 3x3 array of normally distributed random values\n",
    "# with mean 0 and standard deviation 1\n",
    "np.random.normal(0, 1, (3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2, 3, 4],\n",
       "       [5, 7, 8],\n",
       "       [0, 5, 0]])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 3x3 array of random integers in the interval [0, 10)\n",
    "np.random.randint(0, 10, (3, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0.,  0.],\n",
       "       [ 0.,  1.,  0.],\n",
       "       [ 0.,  0.,  1.]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a 3x3 identity matrix\n",
    "np.eye(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.,  1.,  1.])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create an uninitialized array of three integers\n",
    "# The values will be whatever happens to already exist at that memory location\n",
    "np.empty(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NumPy Standard Data Types\n",
    "\n",
    "NumPy arrays contain values of a single type, so it is important to have detailed knowledge of those types and their limitations.\n",
    "Because NumPy is built in C, the types will be familiar to users of C, Fortran, and other related languages.\n",
    "\n",
    "The standard NumPy data types are listed in the following table.\n",
    "Note that when constructing an array, they can be specified using a string:\n",
    "\n",
    "```python\n",
    "np.zeros(10, dtype='int16')\n",
    "```\n",
    "\n",
    "Or using the associated NumPy object:\n",
    "\n",
    "```python\n",
    "np.zeros(10, dtype=np.int16)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Data type\t    | Description |\n",
    "|---------------|-------------|\n",
    "| ``bool_``     | Boolean (True or False) stored as a byte |\n",
    "| ``int_``      | Default integer type (same as C ``long``; normally either ``int64`` or ``int32``)| \n",
    "| ``intc``      | Identical to C ``int`` (normally ``int32`` or ``int64``)| \n",
    "| ``intp``      | Integer used for indexing (same as C ``ssize_t``; normally either ``int32`` or ``int64``)| \n",
    "| ``int8``      | Byte (-128 to 127)| \n",
    "| ``int16``     | Integer (-32768 to 32767)|\n",
    "| ``int32``     | Integer (-2147483648 to 2147483647)|\n",
    "| ``int64``     | Integer (-9223372036854775808 to 9223372036854775807)| \n",
    "| ``uint8``     | Unsigned integer (0 to 255)| \n",
    "| ``uint16``    | Unsigned integer (0 to 65535)| \n",
    "| ``uint32``    | Unsigned integer (0 to 4294967295)| \n",
    "| ``uint64``    | Unsigned integer (0 to 18446744073709551615)| \n",
    "| ``float_``    | Shorthand for ``float64``.| \n",
    "| ``float16``   | Half precision float: sign bit, 5 bits exponent, 10 bits mantissa| \n",
    "| ``float32``   | Single precision float: sign bit, 8 bits exponent, 23 bits mantissa| \n",
    "| ``float64``   | Double precision float: sign bit, 11 bits exponent, 52 bits mantissa| \n",
    "| ``complex_``  | Shorthand for ``complex128``.| \n",
    "| ``complex64`` | Complex number, represented by two 32-bit floats| \n",
    "| ``complex128``| Complex number, represented by two 64-bit floats| "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More advanced type specification is possible, such as specifying big or little endian numbers; for more information, refer to the [NumPy documentation](http://numpy.org/).\n",
    "NumPy also supports compound data types, which will be covered in [Structured Data: NumPy's Structured Arrays](02.09-Structured-Data-NumPy.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--NAVIGATION-->\n",
    "< [Introduction to NumPy](02.00-Introduction-to-NumPy.ipynb) | [Contents](Index.ipynb) | [The Basics of NumPy Arrays](02.02-The-Basics-Of-NumPy-Arrays.ipynb) >"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}