ipython-data-notebooks

Continually updated IPython Data Science Notebooks geared towards processing big data (AWS, Spark, Hadoop MapReduce, HDFS, Linux command line, Python, NumPy, pandas, matplotlib, SciPy, scikit-learn, Kaggle).

kaggle

IPython Notebook(s) used in kaggle competitions.

Notebook	Description
titanic	Predicts survival on the Titanic. Demonstrates data cleaning, exploratory data analysis, and machine learning.

spark

IPython Notebook(s) demonstrating spark and HDFS functionality.

Notebook	Description
spark	In-memory cluster computing framework, up to 100 times faster for certain applications and is well suited for machine learning algorithms.
hdfs	Reliably stores very large files across machines in a large cluster.

aws

IPython Notebook(s) demonstrating Amazon Web Services functionality.

Notebook	Description
mrjob	Supports MapReduce jobs in Python 2.5+ and runs them locally or on Hadoop clusters.
s3distcp	Combines smaller files and aggregates them together by taking in a pattern and target file. S3DistCp can also be used to transfer large volumes of data from S3 to your Hadoop cluster.
s3cmd	Interacts with S3 through the command line.
s3-parallel-put	Uploads multiple files to S3 in parallel.
redshift	Acts as a fast data warehouse built on top of technology from massive parallel processing (MPP).
kinesis	Streams data in real time with the ability to process thousands of data streams per second.
lambda	Runs code in response to events, automatically managing compute resources.

python-core

IPython Notebook(s) demonstrating core Python functionality geared towards data analysis.

Notebook	Description
data structures	Tuples, lists, dicts, sets.
data structure utilities	Slice, range, xrange, bisect, sort, sorted, reversed, enumerate, zip, list comprehensions.
functions	Functions as objects, lambda functions, closures, *args, **kwargs currying, generators, generator expressions, itertools.
datetime	Datetime, strftime, strptime, timedelta.
unit tests	Nose unit tests.

pandas

IPython Notebook(s) demonstrating pandas functionality.

Notebook	Description
pandas	Software library written for data manipulation and analysis in Python. Offers data structures and operations for manipulating numerical tables and time series.
pandas io	Input and output operations.
pandas cleaning	Data wrangling operations.

matplotlib

IPython Notebook(s) demonstrating matplotlib functionality.

Notebook	Description
matplotlib	Python 2D plotting library which produces publication quality figures in a variety of hardcopy formats and interactive environments across platforms.

numpy

IPython Notebook(s) demonstrating NumPy functionality.

Notebook	Description
numpy	Adds Python support for large, multi-dimensional arrays and matrices, along with a large library of high-level mathematical functions to operate on these arrays.

commands

IPython Notebook(s) demonstrating various command lines for Linux, Git, etc.

Notebook	Description
linux	Unix-like and mostly POSIX-compliant computer operating system. Disk usage, splitting files, grep, sed, curl, viewing running processes, terminal syntax highlighting, and Vim.
anaconda	Distribution of the Python programming language for large-scale data processing, predictive analytics, and scientific computing, that aims to simplify package management and deployment.
ipython notebook	Web-based interactive computational environment where you can combine code execution, text, mathematics, plots and rich media into a single document.
git	Distributed revision control system with an emphasis on speed, data integrity, and support for distributed, non-linear workflows.
ruby	Used to interact with the AWS command line and for Jekyll, a blog framework that can be hosted on GitHub Pages.
jekyll	Simple, blog-aware, static site generator for personal, project, or organization sites. Renders Markdown or Textile and Liquid templates, and produces a complete, static website ready to be served by Apache HTTP Server, Nginx or another web server.

scikit-learn

[Coming Soon] IPython Notebook(s) demonstrating scikit-learn functionality.

scipy

[Coming Soon] IPython Notebook(s) demonstrating SciPy functionality.

References

• Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython
• Building Machine Learning Systems with Python
• sklearn_pycon2015
• parallel_ml_tutorial
• Think Stats

License

Copyright 2014 Donne Martin

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.