2015-06-07 09:47:50 +08:00
{
" cells " : [
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" ##Customer Churn## \n " ,
" \n " ,
2015-06-14 18:34:33 +08:00
" Credits: Forked from [growth-workshop](https://github.com/aprial/growth-workshop) by [aprial](https://github.com/aprial), as featured on the [yhat blog](http://blog.yhathq.com/posts/predicting-customer-churn-with-sklearn.html) "
2015-06-07 09:47:50 +08:00
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" \n " ,
" \" Churn Rate \" is a business term describing the rate at which customers leave or cease paying for a product or service. It ' s a critical figure in many businesses, as it ' s often the case that acquiring new customers is a lot more costly than retaining existing ones (in some cases, 5 to 20 times more expensive). \n " ,
" \n " ,
" Understanding what keeps customers engaged, therefore, is incredibly valuable, as it is a logical foundation from which to develop retention strategies and roll out operational practices aimed to keep customers from walking out the door. Consequently, there ' s growing interest among companies to develop better churn-detection techniques, leading many to look to data mining and machine learning for new and creative approaches. \n " ,
" \n " ,
" Predicting churn is particularly important for businesses w/ subscription models such as cell phone, cable, or merchant credit card processing plans. But modeling churn has wide reaching applications in many domains. For example, casinos have used predictive models to predict ideal room conditions for keeping patrons at the blackjack table and when to reward unlucky gamblers with front row seats to Celine Dion. Similarly, airlines may offer first class upgrades to complaining customers. The list goes on. "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" Wait, don ' t go! \n " ,
" \n " ,
" So what are some of ops strategies that companies employ to prevent churn? Well, reducing churn, it turns out, often requires non-trivial resources. Specialized retention teams are common in many industries and exist expressly to call down lists of at-risk customers to plead for their continued business. \n " ,
" \n " ,
"  \n " ,
" \n " ,
" Organizing and running such teams is tough. From an ops perspective, cross-geographic teams must be well organized and trained to respond to a huge spectrum of customer complaints. Customers must be accurately targeted based on churn-risk, and retention treatments must be well-conceived and correspond reasonably to match expected customer value to ensure the economics make sense. Spending $1,000 on someone who wasn ' t about to leave can get expensive pretty quickly. \n " ,
" \n " ,
" Within this frame of mind, efficiently dealing with turnover is an exercise of distinguishing who is likely to churn from who is not using the data at our disposal. The remainder of this post will explore a simple case study to show how Python and its scientific libraries can be used to predict churn and how you might deploy such a solution within operations to guide a retention team. \n " ,
" \n " ,
" ##The Dataset## \n " ,
" \n " ,
" The data set we ' ll be using is a longstanding telecom customer data set. \n " ,
" \n " ,
" The data is straightforward. Each row represents a subscribing telephone customer. Each column contains customer attributes such as phone number, call minutes used during different times of day, charges incurred for services, lifetime account duration, and whether or not the customer is still a customer. "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 1 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" from __future__ import division \n " ,
" import pandas as pd \n " ,
" import numpy as np \n " ,
" import matplotlib.pyplot as plt \n " ,
" import json \n " ,
" \n " ,
" from sklearn.cross_validation import KFold \n " ,
" from sklearn.preprocessing import StandardScaler \n " ,
" from sklearn.cross_validation import train_test_split \n " ,
" from sklearn.svm import SVC \n " ,
" from sklearn.ensemble import RandomForestClassifier as RF \n " ,
" % matplotlib inline \n "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 2 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Column names: \n " ,
" [ ' State ' , ' Account Length ' , ' Area Code ' , ' Phone ' , \" Int ' l Plan \" , ' VMail Plan ' , ' VMail Message ' , ' Day Mins ' , ' Day Calls ' , ' Day Charge ' , ' Eve Mins ' , ' Eve Calls ' , ' Eve Charge ' , ' Night Mins ' , ' Night Calls ' , ' Night Charge ' , ' Intl Mins ' , ' Intl Calls ' , ' Intl Charge ' , ' CustServ Calls ' , ' Churn? ' ] \n " ,
" \n " ,
" Sample data: \n "
]
} ,
{
" data " : {
" text/html " : [
" <div style= \" max-height:1000px;max-width:1500px;overflow:auto; \" > \n " ,
" <table border= \" 1 \" class= \" dataframe \" > \n " ,
" <thead> \n " ,
" <tr style= \" text-align: right; \" > \n " ,
" <th></th> \n " ,
" <th>State</th> \n " ,
" <th>Account Length</th> \n " ,
" <th>Area Code</th> \n " ,
" <th>Phone</th> \n " ,
" <th>Int ' l Plan</th> \n " ,
" <th>VMail Plan</th> \n " ,
" <th>Night Charge</th> \n " ,
" <th>Intl Mins</th> \n " ,
" <th>Intl Calls</th> \n " ,
" <th>Intl Charge</th> \n " ,
" <th>CustServ Calls</th> \n " ,
" <th>Churn?</th> \n " ,
" </tr> \n " ,
" </thead> \n " ,
" <tbody> \n " ,
" <tr> \n " ,
" <th>0</th> \n " ,
" <td> KS</td> \n " ,
" <td> 128</td> \n " ,
" <td> 415</td> \n " ,
" <td> 382-4657</td> \n " ,
" <td> no</td> \n " ,
" <td> yes</td> \n " ,
" <td> 11.01</td> \n " ,
" <td> 10.0</td> \n " ,
" <td> 3</td> \n " ,
" <td> 2.70</td> \n " ,
" <td> 1</td> \n " ,
" <td> False.</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>1</th> \n " ,
" <td> OH</td> \n " ,
" <td> 107</td> \n " ,
" <td> 415</td> \n " ,
" <td> 371-7191</td> \n " ,
" <td> no</td> \n " ,
" <td> yes</td> \n " ,
" <td> 11.45</td> \n " ,
" <td> 13.7</td> \n " ,
" <td> 3</td> \n " ,
" <td> 3.70</td> \n " ,
" <td> 1</td> \n " ,
" <td> False.</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>2</th> \n " ,
" <td> NJ</td> \n " ,
" <td> 137</td> \n " ,
" <td> 415</td> \n " ,
" <td> 358-1921</td> \n " ,
" <td> no</td> \n " ,
" <td> no</td> \n " ,
" <td> 7.32</td> \n " ,
" <td> 12.2</td> \n " ,
" <td> 5</td> \n " ,
" <td> 3.29</td> \n " ,
" <td> 0</td> \n " ,
" <td> False.</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>3</th> \n " ,
" <td> OH</td> \n " ,
" <td> 84</td> \n " ,
" <td> 408</td> \n " ,
" <td> 375-9999</td> \n " ,
" <td> yes</td> \n " ,
" <td> no</td> \n " ,
" <td> 8.86</td> \n " ,
" <td> 6.6</td> \n " ,
" <td> 7</td> \n " ,
" <td> 1.78</td> \n " ,
" <td> 2</td> \n " ,
" <td> False.</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>4</th> \n " ,
" <td> OK</td> \n " ,
" <td> 75</td> \n " ,
" <td> 415</td> \n " ,
" <td> 330-6626</td> \n " ,
" <td> yes</td> \n " ,
" <td> no</td> \n " ,
" <td> 8.41</td> \n " ,
" <td> 10.1</td> \n " ,
" <td> 3</td> \n " ,
" <td> 2.73</td> \n " ,
" <td> 3</td> \n " ,
" <td> False.</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>5</th> \n " ,
" <td> AL</td> \n " ,
" <td> 118</td> \n " ,
" <td> 510</td> \n " ,
" <td> 391-8027</td> \n " ,
" <td> yes</td> \n " ,
" <td> no</td> \n " ,
" <td> 9.18</td> \n " ,
" <td> 6.3</td> \n " ,
" <td> 6</td> \n " ,
" <td> 1.70</td> \n " ,
" <td> 0</td> \n " ,
" <td> False.</td> \n " ,
" </tr> \n " ,
" </tbody> \n " ,
" </table> \n " ,
" </div> "
] ,
" text/plain " : [
" State Account Length Area Code Phone Int ' l Plan VMail Plan \\ \n " ,
" 0 KS 128 415 382-4657 no yes \n " ,
" 1 OH 107 415 371-7191 no yes \n " ,
" 2 NJ 137 415 358-1921 no no \n " ,
" 3 OH 84 408 375-9999 yes no \n " ,
" 4 OK 75 415 330-6626 yes no \n " ,
" 5 AL 118 510 391-8027 yes no \n " ,
" \n " ,
" Night Charge Intl Mins Intl Calls Intl Charge CustServ Calls Churn? \n " ,
" 0 11.01 10.0 3 2.70 1 False. \n " ,
" 1 11.45 13.7 3 3.70 1 False. \n " ,
" 2 7.32 12.2 5 3.29 0 False. \n " ,
" 3 8.86 6.6 7 1.78 2 False. \n " ,
" 4 8.41 10.1 3 2.73 3 False. \n " ,
" 5 9.18 6.3 6 1.70 0 False. "
]
} ,
" execution_count " : 2 ,
" metadata " : { } ,
" output_type " : " execute_result "
}
] ,
" source " : [
" churn_df = pd.read_csv( ' ../data/churn.csv ' ) \n " ,
" col_names = churn_df.columns.tolist() \n " ,
" \n " ,
" print \" Column names: \" \n " ,
" print col_names \n " ,
" \n " ,
" to_show = col_names[:6] + col_names[-6:] \n " ,
" \n " ,
" print \" \\ nSample data: \" \n " ,
" churn_df[to_show].head(6) \n " ,
" "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" We ' ll be keeping the statistical model pretty simple for this example so the feature space is almost unchanged from what you see above. The following code simply drops irrelevant columns and converts strings to boolean values (since models don ' t handle \" yes \" and \" no \" very well). The rest of the numeric columns are left untouched. "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 3 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" # Isolate target data \n " ,
" churn_result = churn_df[ ' Churn? ' ] \n " ,
" y = np.where(churn_result == ' True. ' ,1,0) \n "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 4 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" # We don ' t need these columns \n " ,
" to_drop = [ ' State ' , ' Area Code ' , ' Phone ' , ' Churn? ' ] \n " ,
" churn_feat_space = churn_df.drop(to_drop,axis=1) "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 5 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" # ' yes ' / ' no ' has to be converted to boolean values \n " ,
" # NumPy converts these from boolean to 1. and 0. later \n " ,
" yes_no_cols = [ \" Int ' l Plan \" , \" VMail Plan \" ] \n " ,
" churn_feat_space[yes_no_cols] = churn_feat_space[yes_no_cols] == ' yes ' "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 6 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Index([u ' Account Length ' , u ' Int ' l Plan ' , u ' VMail Plan ' , u ' VMail Message ' , u ' Day Mins ' , u ' Day Calls ' , u ' Day Charge ' , u ' Eve Mins ' , u ' Eve Calls ' , u ' Eve Charge ' , u ' Night Mins ' , u ' Night Calls ' , u ' Night Charge ' , u ' Intl Mins ' , u ' Intl Calls ' , u ' Intl Charge ' , u ' CustServ Calls ' ], dtype= ' object ' ) \n "
]
}
] ,
" source " : [
" # Pull out features for future use \n " ,
" features = churn_feat_space.columns \n " ,
" print features "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 7 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Feature space holds 3333 observations and 17 features \n " ,
" Unique target labels: [0 1] \n "
]
}
] ,
" source " : [
" X = churn_feat_space.as_matrix().astype(np.float) \n " ,
" \n " ,
" # This is important \n " ,
" scaler = StandardScaler() \n " ,
" X = scaler.fit_transform(X) \n " ,
" print \" Feature space holds %d observations and %d features \" % X .shape \n " ,
" print \" Unique target labels: \" , np.unique(y) \n " ,
" "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" One slight side note. Many predictors care about the relative size of different features even though those scales might be arbitrary. For instance: the number of points a basketball team scores per game will naturally be a couple orders of magnitude larger than their win percentage. But this doesn ' t mean that the latter is 100 times less signifigant. `StandardScaler` fixes this by normalizing each feature to a range of around 1.0 to -1.0 thereby preventing models from misbehaving. Well, at least for that reason. \n " ,
" \n " ,
" Great, I now have a feature space `X` and a set of target values `y`. On to the predictions! \n " ,
" \n " ,
" ##How good is your model?## \n " ,
" \n " ,
" Express, test, cycle. A machine learning pipeline should be anything but static. There are always new features to design, new data to use, new classifiers to consider each with unique parameters to tune. And for every change it ' s critical to be able to ask, \" Is the new version better than the last? \" So how do I do that? \n " ,
" \n " ,
" As a good start, cross validation will be used throught this example. Cross validation attempts to avoid overfitting (training on and predicting the same datapoint) while still producing a prediction for each observation dataset. This is accomplished by systematically hiding different subsets of the data while training a set of models. After training, each model predicts on the subset that had been hidden to it, emulating multiple train-test splits. When done correctly, every observation will have a ' fair ' corresponding prediction. \n " ,
" \n " ,
" Here ' s what that looks like using `scikit-learn` libraries. "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 8 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" from sklearn.cross_validation import KFold \n " ,
" \n " ,
" def run_cv(X,y,clf_class,**kwargs): \n " ,
" # Construct a kfolds object \n " ,
" kf = KFold(len(y),n_folds=3,shuffle=True) \n " ,
" y_pred = y.copy() \n " ,
" \n " ,
" # Iterate through folds \n " ,
" for train_index, test_index in kf: \n " ,
" X_train, X_test = X[train_index], X[test_index] \n " ,
" y_train = y[train_index] \n " ,
" # Initialize a classifier with key word arguments \n " ,
" clf = clf_class(**kwargs) \n " ,
" clf.fit(X_train,y_train) \n " ,
" y_pred[test_index] = clf.predict(X_test) \n " ,
" return y_pred "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" Let ' s compare three fairly unique algorithms support vector machines, random forest, and k-nearest-neighbors. Nothing fancy here, just passing each to cross validation and determining how often the classifier predicted the correct class. "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 9 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Logistic Regression: \n " ,
" 0.862 \n " ,
" Gradient Boosting Classifier \n " ,
" 0.950 \n " ,
" Support vector machines: \n " ,
" 0.912 \n " ,
" Random forest: \n " ,
" 0.935 \n " ,
" K-nearest-neighbors: \n " ,
" 0.893 \n "
]
}
] ,
" source " : [
" from sklearn.svm import SVC \n " ,
" from sklearn.ensemble import RandomForestClassifier as RF \n " ,
" from sklearn.neighbors import KNeighborsClassifier as KNN \n " ,
" from sklearn.linear_model import LogisticRegression as LR \n " ,
" from sklearn.ensemble import GradientBoostingClassifier as GBC \n " ,
" from sklearn.metrics import average_precision_score \n " ,
" \n " ,
" def accuracy(y_true,y_pred): \n " ,
" # NumPy interpretes True and False as 1. and 0. \n " ,
" return np.mean(y_true == y_pred) \n " ,
" \n " ,
" print \" Logistic Regression: \" \n " ,
" print \" %.3f \" % a ccuracy(y, run_cv(X,y,LR)) \n " ,
" print \" Gradient Boosting Classifier \" \n " ,
" print \" %.3f \" % a ccuracy(y, run_cv(X,y,GBC)) \n " ,
" print \" Support vector machines: \" \n " ,
" print \" %.3f \" % a ccuracy(y, run_cv(X,y,SVC)) \n " ,
" print \" Random forest: \" \n " ,
" print \" %.3f \" % a ccuracy(y, run_cv(X,y,RF)) \n " ,
" print \" K-nearest-neighbors: \" \n " ,
" print \" %.3f \" % a ccuracy(y, run_cv(X,y,KNN)) "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" \n " ,
" Random forest won, right? \n " ,
" \n " ,
" ##Precision and recall## \n " ,
" \n " ,
" Measurements aren ' t golden formulas which always spit out high numbers for good models and low numbers for bad ones. Inherently they convey something sentiment about a model ' s performance, and it ' s the job of the human designer to determine each number ' s validity. The problem with accuracy is that outcomes aren ' t necessarily equal. If my classifier predicted a customer would churn and they didn ' t, that ' s not the best but it ' s forgivable. However, if my classifier predicted a customer would return, I didn ' t act, and then they churned... that ' s really bad. \n " ,
" \n " ,
" We ' ll be using another built in `scikit-learn` function to construction a confusion matrix. A confusion matrix is a way of visualizing predictions made by a classifier and is just a table showing the distribution of predictions for a specific class. The x-axis indicates the true class of each observation (if a customer churned or not) while the y-axis corresponds to the class predicted by the model (if my classifier said a customer would churned or not). \n " ,
" \n " ,
" ## Confusion matrix and confusion tables: \n " ,
" The columns represent the actual class and the rows represent the predicted class. Lets evaluate performance: \n " ,
" \n " ,
" | | condition True | condition false| \n " ,
" |------|----------------|---------------| \n " ,
" |prediction true|True Positive|False positive| \n " ,
" |Prediction False|False Negative|True Negative| \n " ,
" \n " ,
" Sensitivity, Recall or True Positive Rate quantify the models ability to predict our positive classes. \n " ,
" \n " ,
" $$TPR = \\ frac { TP} { TP + FN}$$ \n " ,
" \n " ,
" Specificity or True Negative Rate quantify the models ability to predict our Negative classes. \n " ,
" \n " ,
" $$TNR = \\ frac { TN} { FP + TN}$$ \n " ,
" \n " ,
" ### Example: \n " ,
" \n " ,
" | | Spam | Ham| \n " ,
" |------|----------------|---------------| \n " ,
" |prediction Spam|100|50| \n " ,
" |Prediction Ham|75|900| \n " ,
" \n " ,
" $$TPR = \\ frac {100} { 100 + 75} = 57.14 \\ % Sensitive $$ \n " ,
" \n " ,
" $$TNR = \\ frac { 900} { 50 + 900} = 94.73 \\ % Specific $$ \n " ,
" \n "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 10 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" [[2821 29] \n " ,
" [ 244 239]] \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAUAAAAEWCAYAAAAXR05AAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAHjhJREFUeJzt3Xm4HFW57/HvLxOjTCIKYhIUooCoBE2OcqMR0YvDFTmK \n iMoRZy8qXsUJjwNxwqPn4CznyqCgggOCgoLA4RANqAQhUYYwSpgTAYEwCBl4zx9rdVLpdO/u3bv2 \n 7q5dv8/z1LO7q1avWruq++1Vq6rrVURgZlZHE/rdADOzfnEANLPacgA0s9pyADSz2nIANLPacgA0 \n s9rqawCUtImksyTdJ+knI6jnTZLOLbNt/SJpjqRrenzt0yUtlrRC0vvKbpuND5KWSnpJm2U9v/+q \n qKsAKOmNkv4k6QFJd0g6W9LeJaz/dcB2wDYRcVCvlUTEjyLif5fQnlEl6TFJTx2qTEQsiIhn9LiK \n jwIXRMQWEfGtHutYS9JWkk6UdGcOqtdK+thI6x0N+UO9T5tlT5a0qtW2l3SGpK+MYL0d9+kw65ue \n 67y8af62klZKuqmE1USeNlwwsvdf5XQMgJI+BHwV+DwpWD0F+Dbw6hLWPw24LiIeK6GuqlDbBdKk \n EdY9Dbi6lxdKmthi9leBTYFnRMQWpH1+Q+/NK19hmwVttm1E3A5cABzS9NptgJcD3x9pM3p60dD7 \n exNJuxeevxH4K20Cl/UoItpOwJbAA8BrhyizEfA14PY8fRWYkpfNBW4DPgQsB+4ADs3L5gGPAivz \n Ot4GHAX8oFD3dOAxYEJ+fihwI7CC9GZ4Y2H+gsLrXgBcCtwHLASeX1g2H/gscFGu51zg8W3+t0b7 \n PwL8Lbf/NcArgOuAe4CPF8rPAv4A3JvLfhOYnJf9Lv8vD+b/98BC/R8F7gROyvNuza95Wl7Hnvn5 \n DsBdwAtbtPW/gdXAP/L/tXPefyfnti8F/hVQYZtdDBwD3A18tkWdVwD7t9k26+2bwrZ9e1P938z7 \n YQmwT1PZo4FLgPuBXwBbF5a/Grgqb8sLSUG4sWxp3mZ/Bh4BTgHWAA/nbfvhFu09GLihad5hwGWF \n bfvzvK3+Cry/UG4C8AlS8F9Bem/t2Gqf5vLvBK7P++6XwPaFuh7L670euHGI7foJ4MuF+ZfmeTcV \n 5n280KargNc01fVO0hdiY/lz8vybgCPy9rsP+DGwUeE9f2vTtm5ZNi9/FbA476eLgT0Kyz5Gen+v \n AK4p7v9BmToFwP2AVRTe5C3KfBb4PbBtni4mf5jyxlxFCmwTSd+2DwFb5uWfAU4u1PUZ2gRAYDPS \n B2WXvOyJwG7NARDYJu+MN+XXvQH4O/nDRfrgXU8KEBuTPlxHDxEAVwGfzO1/BylY/Ci3ZzfSh25a \n Lj+TFAQnsK439oGmN/9TW9R/NDA5t6f5DfiO/ObdhBSsvzzEvrgQeFvh+cnAGbmt04BrG8vzNlsF \n vDe3d+MW9R0HXJnL7tLmgzqh1foL9X8gb7vXkz5AWxX2w215G24KnNbY98AMUlB5SX7tR/I+m1T4 \n UF4OPJl1H9ybGOIDlrfffcDehXl/AA4n9eAuy/t5ErAT6Yv2ZbncR4C/sO699yzSsE2rfboP6Uvq \n OcAU4BvAb5veA+cCW1EIJC226zTglty23UhfIC9h/QD4OuBJ+fHr8zZ7Yn5+YN6+e+XnTwOmFrbf \n H4EnAVuT3qfvbhMAbxqi7J6kjs3zcjv/JZefDDw9t7/RvqnF7TQoU6cA+Cbgzg5lbgD2Kzx/WWMn \n 5Y35MOt/SJYDs/Ljo1g/4DU/b7wZGgHwXuCfgU2a2nAo6wLgIcAfm5b/HnhL4UP6icKy/wuc0+Z/ \n a7S/0Wt6XG7P8wpl/kT7XtL/A05vevM3B8BHyT3mVm/APO+XpN7YYnKPss36LmRdD2xirrvYc3oX \n cGFhm93cYd9uDByZ/8eVpCC0X/O+aVp/MQDe3lTfJcCbC2W/WFi2a27vBOBTwI8Ly0T6ML+w8KE8 \n tKnuIQNgLnMc8P/z413y+rYFZjdvi/x/n5gfXwv8nzZ1Nu/TE4AvFZ5vlrfd1EL5uUO0sbFdJwLn \n kz5PX8rtWS8AtnjtokY7SUH2/W3K3UQ+esrP/w04ttX7r0PZY2k6ciD19F5ICrjLc5vbvmf7PXUa \n A7wH2FbSUOV2AG4uPL8lz1tbR6w/xvcwsHmH9W4gIh4CDgLeA9wh6VeSnt6mPbc0zbu5qU3LCo// \n 0aE990Tes7kspB1bfP1mAJJm5HbdKel+4AvA44eoG+CuiFjZoczxwO7ANyNiVYeyjbZuS/ombt43 \n Ty48v3XIiiIeiYijI+K5pP/jp8DPJG3VoQ0Ntzc9vxnYvs36b8nt3TaXWbsP8/a/dThtb+Mk4EBJ \n G5G+KH8TEXeTels7SLq3MZECznb5dTuSeoTd2J7CNs/v23t6aHuQevBvJR3F/ICmsUZJ/yJpUaHN \n zyRtv27aPJzPQLuy04AjmrbbjqRD/htJHYCjgOWSTpVU3PcDoVMA/APpW/KAIcrcQfrWapia5/Xi \n QdLhUMOTigsj4ryIeFmefw3pG73Z7aQdUzSNDT+Mo+FY0iHCzhGxJWnMrdM2jqEWStqcNMZ6PDBP \n 0tZdtuVu0iHo9MK8qaSeVFfrXq+REQ+QDtU3Ix0iPpQXtd1frP+hh7Qfiu+NqU2PV5EOH++gsA8l \n iXTyrbgPm9vezf9yMWk4ZH/S0c1Jef6tpJ7V1oVpi4h4VWH5zl3UD02fB0mbkb48hmp7O6eTxptv \n jIjifkPSNOC7pCGMbSJia9JwRSNIDqfNw9Vo/y3AF5q22+YR8ROAiDg1IuaQ9mWQeo8DZcgPZ0Tc \n D3wa+Lak/SVtKmmypJdLavwzpwKfzKfpt83lf9BjexYDL5T0FElbkr6FAZC0XW7DZqQPykOkge9m \n 5wAzJB0saZKkg4BnAL8qlOnprF0XNicNhj8s6Rmkw+ui5aRDg+H4OrAwIt4F/Br4zw7lBRARa0g9 \n ti9I2jx/YD4I/LDbFUv6lKTnSpoiaWPSeN69wLURcRfpQ32IpImS3tbif9tO0uH5PXMgaT+cXWjn \n myXtKmlT0ljyz3Jv72fAKyXtI2kyaRD+EdJQRjsdt22u+2Tgy6QTRGflRZcAD0j6aL42daKkZ0p6 \n bl5+PPA5STsreVY+g9xqvacCb5X07NzT/CJpSKb5qKSj3Ht8MWkcuNlmpKByNzBB0ltJPcCG44EP \n S5qZ27yzpKkt6ulF4/NzHPAeSbPyOjaT9Mr8fpuR999GpE7UI7T+vPZVx8tgIuIY0lncT5LOkN1C \n Oot1Ri7yedIY0V/y9Kc8b20VQ1VfXB4R/wX8JNdzKekN2lg+gfQBvp10SDGHdQFmbT0RcQ/pzNQR \n pDfHh4FXRcTf27Qp6NzGoZ4XfZh0ucIK0rfzj5vKHwWclA8XXjfEutPAl7Q/aQyo8X9+CJgp6eAu \n 2/t+0hfFX4EFpJM33yuU69QTeSyXbwS7lwCvjIiH8/J3kk4Q3E0aqL+46fWXkMba7gI+R7qa4N7C \n +n9AugTlTtIJg8MBIuJa4M2kM8h3Aa8kjW2tHqKtR5O+iO/Nl261czKpN/mTxnBCHqJ5FenExV/z \n Or8LbJFfcwzpy+Q80om440jjo9C0TyPiAtIY5s9JvcGdSIewDd30/oqficsj4qbmZRFxNfAfpKO0 \n ZaTgd1HhdaeRhmBOIb0fTyedxGi3vubPxFBta7ThMtJ74FuknvX1pBMhkK4OOZq0Le8kHZof2VxZ \n vzUG981KJelQ0gmZOW2WX0g64XXimDbMrMC/BbZ+Gq2hCLOuOACOMkn7SbpG0vUa0J+RjZJuDrEH \n +vBD6WeAyyVd0e+22OjwIfAoyj8vuxbYlzSGdilwcEQs6WvDrCuS5pCuTDg5Ivbod3usfO4Bjq5Z \n pJ9fLc0D7j8mXYJhFRARC0hnvW2ccgAcXU9m/Yteb2PDa+PMrE8cAEeXxxfMBpgD4Oi6nXTNWcNT \n WP+XGGbWRw6Ao+tPwC7
" text/plain " : [
" <matplotlib.figure.Figure at 0x10c33edd0> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" [[2814 36] \n " ,
" [ 147 336]] \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAT0AAAEWCAYAAAAU8EBoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAG4lJREFUeJzt3Xm4HHWd7/H3JyFh3xclCARHGBYZJVyTUS4aHHCAxyuD \n 4iB4GVG8OBdRRp1xwPEq4SKOoxdFcfQZNgEFYRAUFAZQORJBDGAiW5BFwp4gEBL2LHzvH/XrnErT \n fU6f6j7pqq7P63nqoZZfV337VOXLr5auryICM7O6mNDvAMzM1iQnPTOrFSc9M6sVJz0zqxUnPTOr \n FSc9M6uV2iQ9SetKukLSM5Iu6mI9H5R0dS9j6xdJe0u6u+Bn/1zSPElLJR3b69jGk6Spkl6RVJvj \n 34aVbqdLOlzSLZKelfSYpCsl7dWDVR8CbAVsFhGHFl1JRPwgIv66B/GMq/SP+vUjtYmI2RGxc8FN \n fBb4RURsFBGnF1zHKpJOlLQ87fdnJN0kae9u19tvkhZIeiF9r2fT/yReuwa3PyTpqDW1vSooVdKT \n 9Gng68DJZAlqW+DbwHt6sPrtgXsi4pUerKsq1HaBtFaX694euKvIByVNbDE7gAsjYkNgc+DnwCXF \n wyuNAN4dERumYaOIWDiWFXS5r/zrg2YRUYoB2Bh4FnjfCG3WBr4BPJqGrwOT07KZwCPAp4FFwGPA \n kWnZLOBlYFnaxkeAE4Hzc+ueCrwCTEjTRwL3A0uBPwKH5+bPzn3ubcDNwDPAHOCtuWVDwEnAr9N6 \n rgY2b/PdGvH/E/BEiv9vgAOBe4CngONz7acDvwEWp7bfAialZden7/Jc+r7vz63/s8DjwLlp3sPp \n M3+WtrFHmp4C/Al4e4tYfwmsAF5M3+sNaf+dl2JfAPwLoNzf7AbgVOBJ4KQW62zeH7um77DFaN83 \n LX8F+Fj6Wy0GTs8tmwB8LX2f+4GPN+3rKcDl6fvfC3y0Ka7/BM5P3/U2YEfgBLLj7EFgvxGO2QeA \n dxY8lvP7SsDxwH3pb3gRsGlqvw7w/TR/MdlxuBXwpdx+ehb4Zr//nZdh6HsAuYNgf2B540Bs0+Yk \n 4EZgizTc0PgHlA6U5ekgnQgcADwPbJyWfxE4L7euL9Im6QHrA0uAHdOy1wC7pvEjSUkP2CwdZB9M \n n/sA8HTuYBxK/4jekA7M64Avt/lujfg/n+L/aDqIf5Di2RV4Adg+tZ9GlggmMNzrOi63vleA17dY \n /5eBSSmemaSkl9p8FLgTWJcsQf/bCPviOuAjuenzgMtSrNsDf2gsT3+z5WTJZgKwTov1ndjYH8Bk \n 4F+Be3PLO/m+lwMbkZ0hPAH8dVr298B8YBtg0xT7SoaT3vXA6Wm7b0qf3ScX14vAfmm/nEuW1E/I \n 7ac/jvB3egD4q4LHcn5fHZfaT0nzvgtckNp/LH33dciS4x7Ahq32k4dyJb0PAo+P0uY+YP/c9LuA \n B3IHygvkkibZ/4mn5w7efJJrnp7K6klvMfBeYN2mGI5kOOkdAdzUtPxG4EO5A+5zuWX/G7iqzXdr \n xN/oHW2Y4nlLrs0twEFtPv8PwKW56VZJ72VSbyI37+Gm9fwEuB2YR64n1WJ71wFHpfGJad0755Yf \n DVyX+5s9OMq+PTGtYzFZ7+RJ0v90xvB935abvgj4bBr/JXB0btl+uX29bdre+rnlpwDn5OK6Orfs \n f5D1mpr300Zt4lyQ2i9Ow6Vp/v2MfCw376u7yPUYga3JzlwmAh8mS5q7j7SfPGRDma7pPQVsMcod \n tSlkpxMND6V5q9YRq1+zewHYYKyBRMTzwKFkPYTHJP1U0p+3ieehpnkPNsWUv37z4ijxPBXpSE1t \n IUvc+c+vDyBppxTX45KWkJ3KbD7CugH+FBHLRmlzJrAb8K2IWD5K20asW5D1Ppr3zTa56YdHWRfA \n RRGxKVnP+g7gE40FHX7f/N86v++3btp+fp9NAZ5O+7xd7E/kxl8Enmyxn9rt1yD7H9WmaXhvLqaR \n juXmfTUVuEzSYkmLyZLgCrLT2PPJeuY/lPSopK80XQf0db2cMiW935D93+3gEdo8RrbzG7ZL84p4 \n DlgvN73aHbWIuCYi3pXm3w2c0WIdj5KdauVtn+aPt++QHfhviIiNya6hjbY/Rzz4JW1Adp3pTGCW \n pE07jOVJstOxqbl525Fdl+po22m5ACLiKbKe4tGSdkjLi3zfhsdTPPnYGh4DNkvfvV3s42G0Y7n5 \n 7/UQWc9w09ywXkQ8HhErIuKkiNiN7Brzu4G/a7Oe2itN0ouIJcAXgG9LOkjSepImSTpA0ldSswuB \n z0vaQtIWqf35BTc5D3i7pG0lbUx2jQYASVulGNYn+8f8PNk1oGZXATtJOkzSWpIOBXYGfppr0/YO \n apc2IDttekHSzmSnznmLyG5OjMVpwJyIOBr4Gdl1o5E0ktRK4GLgS5I2kLQ98Cmyi+udWu3vFBH3 \n AFeQXcyH0b9vq/U11nkx8ElJ26REfnxuOw+TXZL4sqS1Jf0F2Y2uscRexFiP5e8Cp0jaDkDSlpLe \n k8ZnSto93RV/luyYbRyvRY6DgVaapAcQEaeS3X39PNkpxUPAMWQXyCF7lOUWsjtot6Xxk/OrGGn1 \n +eUR8XOy6z63kd19vSK3fALZP9pHyU6792b4H9mq9aQeybuBz5D1dv6R7PGEp9vEFIwe40jTef8I \n HE52R/E/gB82tT8RODedDh0ywrYDQNJBZNeVGt/z08A0SYd1GO8nyP7n8EdgNtkNmHNy7Trp6TW3 \n +Srwd5K2YvTv2+pv15h3Btnp3+/JjpkfNbU/jKzX9RhwKfCFiPjlCHGNZT+1M9Zj+TSymxXXSFpK \n dmY0PS17Ldkd5iVkveEhhhPoacAhkp6W9I0CcQ4cDV+aMDMbfKXq6ZmZjTcnvXEgaX9Jd0u6V9I/ \n 9zse65yksyUtknR7v2Ox8eGk12PpYvLpZA9b7wocJmmX/kZlY3AO2b6zAeWk13vTgfsiYkF6zu2H \n wEF9jsk6FBGzyR4itgHlpNd727D6g7CPsPqDrmbWR056vefb4WYl5qTXe4+S/Z6zYVvG/+l+M+uQ \n k17v3QLsmN7OO5nsN7yX9zkmM0uc9HosIlYAx5L9AuAush/Rz+9vVNYpSReS/SxtJ0kPS/pwv2Oy \n 3vIvMsysVtzTM7NacdIzs1px0jOzWnHSM7NacdIzs1rptvZpVyT51rFZn0REV2/1Huu/32631yt9 \n TXqQ1WEcVENkZa0G1ayB3nsw2HtwVk/WcvLoTYDsVehl0fekZ2bVNanfARTgpGdmhVUxgVQx5sqY \n 2u8ArEtT+x1A6a3b7wAK8N3bcTS13wFYl6b2O4DSm9Th0CyVXr1O0p2S7pD0yTT/Iklz0/CApLm5 \n z5yQSjDcLeldufl7Sro9LTtttJjd0zOzwrpIIMuBT0XEvFRo/VZJ10bEoY0Gkr4GPJPGdyV7Y9Gu \n ZC/l/bmkHSN7ecB3gKMiYo6kKyXtHxH/1W7D7umZWWFFe3oRsTAi5qXx54D5wJTGckkC/pasKDpk \n JRcujIjlEbEAuA+YIWlrYMOImJPanQf8zUgxu6dnZoX1IoFImgrsAfw2N3tvYFFE3J+mpwA35ZY3 \n yjAsZ/WX9D7KKOUZnPTMrLB2j6zcAdzZwefTqe0lwHGpx9dwGHBBl+G15KRnZoW1S3p7pKHh4hZt \n JE0CfgR8PyJ+nJu/FnAwMC3XvLkMw+vIeniPpvH8/EdHitnX9MyssHU7HJqla3ZnAXdFxDeaFu8L \n zI+Ix3LzLgc+IGmypB2AHYE5EbEQWCppRlrnEcCPGYF7emZWWBcJZC/gfwK35R5LOSHddT2U4RsY \n AETEXZIuJivBsAI4JoZf+34M8D2y/HrlSHduu4zZzOqu6M/QIuLXtDnTjIiWdUki4hTglBbzbwV2 \n 73TbTnpmVlgVE0gVYzazkvALB8ysVqqYQKoYs5mVhHt6ZlYrVXzLipOemRXmnp6Z1UoVE0gVYzaz \n kpjUaQZZMa5hjImTnpk
" text/plain " : [
" <matplotlib.figure.Figure at 0x10c60b1d0> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" [[2813 37] \n " ,
" [ 317 166]] \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAT0AAAEWCAYAAAAU8EBoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAHg5JREFUeJzt3Xm4HFW57/HvL4EAAoZZASFBARmciNdE5aDRgwper8hR \n BFRExYPnII6co4BeJQ54nXAenqOAgAJ6UDAoKIhsCahEJNEgQcRjgACJypAwCJne+8danVQ63b17 \n V/dOd3X/Ps9Tz+6qWlW1uqv63atqVderiMDMbFhM6HUFzMw2Jgc9MxsqDnpmNlQc9MxsqDjomdlQ \n cdAzs6FSmaAnaQtJl0p6QNJ3O1jP6yX9tJt16xVJB0m6peSyT5U0X9JySSd2u27WPkkPSpraZtk1 \n kp7cZN6bJM3pZt0GUdeDnqTXSboh78i7JV0m6cAurPo1wE7AdhFxZNmVRMR3IuJlXajPuGp1cNdE \n xJyI2KfkJt4HXBURj4+IL5dcx1qSTpN0XmF8V0m3SPp8g7Lfyu/vOYVpe0pa02k9uqmdICJpZn4v \n X6mbfq2kY9vZTkRsHRGLOqiqjUFXg56k9wKfAz5GClC7AV8BXtmF1U8Bbo2IvvpijDM1nSFt0uG6 \n pwA3l1lQ0sQGk6MwfwpwDXBJRLy7yWruIx0n46oLn1M7HgbekN93TVD4TKqmyT4eDBHRlQGYDDwI \n vLpFmc2AzwN35eFzwKQ8byawGHgvsBS4G3hTnjcLeAxYkbfxFuA04LzCuqcCa4AJefxNwJ+B5cD/ \n AK8rTJ9TWO75wG+AB4C5wPMK80aAjwDX5vX8FNi+yXur1f8/gb/m+r8KeDlwK3AvcHKh/HTgV8D9 \n ueyXgE3zvGvye3kov98jCut/H3APcE6edmde5il5Gwfk8V2AvwEvaFDXnwOrgH/k97Vn3n/n5rov \n Aj4AqPCZXQecAfwd+EiDdZ4GnJfrcTtwWovj4Gzgs/l9vCBP2xNYU3c8nZk/m8XARwv79in5Pfw9 \n v8dvA5MLyy7Kn9Pv83ucADwX+GX+vOcDLyyU3+BYAfYBHs2f04PAfS32+53AF4CzCtPnAG8sjL+F \n 9E/mPuAnwO6FeWuAJ+fX2wOXAstIx+PHWP94XQO8jXRM3Q98ue59XEs6lh4AFgIvLszfBZidj5M/ \n AW+t238X5X24LNd3OnBDHl8CfLZb8aKXQzeD3iHAytqB2aTMR/KBt0Merqt9gfLBszJ/+BOBQ0n/ \n QSfn+R8Gzi2s68M0CXrAlnlH7ZXnPQHYr3BgzMmvt8sHzuvzckflg3LbPH8kHxx7ApsDVwOfaHHw \n rwQ+mOv/VtKX8ju5PvsBjwBTcvlp+aCawLpW17safRHq1v8JYNNcn5nkoJfLvBX4A7AFKUB/qsW+ \n uBp4S2H8XODiXNcpwB9r8/NnthJ4e67v5g3W9+G8PxdTCO5Ntn02KYi9o7Av6oPexcDX8nvZEbge \n OD7Pewrwz/lz2AH4BfC5wrKLgBuBXUn/aHfN++KQPP/gPL79KMfKsRQCTov9fmdebhmwd56+NugB \n h+Xj6Kn58/sAcF2jfQ1cCJyf9+++wB3ANXVlZwOPJ51J/RV4Wd1+ehfpGHwtKfhtk+dfA3wZmAQ8 \n My/7ojzvNFKj4pV5fHPSP+XX5/HHATN6HbC6Equ6tqIUOO4ZpcxttQMvj78U+Evh4HmEQtAktfim \n F3ZKMcjVj09l/aB3P/AvwBZ1dXgT675oxwC/rpv/S+DY/Ppq4NTCvH8HLm9x8D/CutbR1rk+zymU \n uQE4rMny7wZ+0OiLUFj/Y+SWcfELV7eeHwILSK2ZTVvsi6uB4/LriXnd+xTmHw9cXfjMbh9l355G \n +tLfV6x3k7Jnk/4BTiK1Cg+hEPRIAeRRCsEVOBr4eZP1vQq4sTD+F/JZQh5/P4V/mHnaT4A3kr7M \n ox4rLd7L2n0AfBK4ML8uBr3LWf8fzATSP/Tdivs674cV5ACc532UDVt6zy+Mfxd4f6G+d9XV73rg \n DaQAuQrYsjDvdODswv4bqVv2F3n6Dq0+g6oN3bymdy+wg6RW69yFdJDX3JGnrV1HrH/N7hFgq7FW \n JCIeBo4E/g24W9KPJD21SX3uqJt2e12dlhRe/2OU+twb+WjJZSEF7uLyWwJI2jvX6x5Jy4CPk1oe \n rfwtIlaMUuabwP7AlyJi5Shla3XdgdRqqt83uxbG72xjXbNJAe3nknYHkHRq7tR6UNJX11sgvZeP \n 5iEKs6bk+twj6X5J9wNfJ7X4kPQESRdKWpw/u/PY8LO7s259R9TWldd3IPDEiHiE9o4VJO1eeC/L \n GxT5FPAySc+omz4F+EJh2/fm6bvWldsR2KSu7osbbKd4TD5CPqayu+rK3g7snIf78nejpn4f12/r \n OGBvYKGkuZL+d4O6VE43g96vSK2Fw1uUuZvUIqvZPU8r4yHSf+maJxZnRsQVEfHSPP0W4BsN1nEX \n 6YAsmsKGB854+BrplHbPiJhMOuUZbX9Eq5mStiJdM/0mMEvStm3W5e+k06KphWm7s/6XoOW2yZ0u \n EXES8CNS4NslIk6P1Du5dUScUF8e+BawDfDqwrw7ScfS9hGxbR4mR8TT8/zTgdXA0/JndwwbfnbF \n +t5BOivYtjBsHRGfynVudqys954j4o7Ce3l8/QcQEfeSPv/6Dpo7SKfmxe1vGRG/riv3N1JrbLfC \n tN0Ym/pAOoX0Hbsb2C4fIzUt93FE3BYRr4uIHUmt2IskbTHG+vSdrgW9iFgGfAj4iqTDJD1O0qaS \n DpX0yVzsAuCDknaQtEMuf16zdY5iPvACSbtJmgycUpshaadchy1JX+aHSV+SepcDe0s6WtImko4k \n XcD+UaFM0x7UDm1FukD+iKR9SKfORUtJ167G4gvA3Ig4HvgxqXXUSi1QrQa+B3xc0la5F/I9pA6C \n MYuIE0mnz1dJ2qnZdnPZVaTrge8vTLsHuAI4Q9LWkiZIeoqkF+QiW5H26XJJu5I6j1r5NvB/JL1U \n 0kRJm+dbTXYd5VhZCjxJ0qZjePtnAM8jXY+r+TpwqqT9ACRNlnRE/YJ5P/wAOC3fl7oPKaC3+ocj \n 1j9Gd5L0zvzdO4J0PF8WEYtJl24+IWmz3Bp9Cy32saQ3SNoxjy7L9aj83RNdvWUlIs4g9b5+kHSR \n 9A7gBNJFaUj/AW8g9ar9Pr8u/ldstXOjOD8ifka6nvF7Uu/rpYX5E0hf2rtIpxIHsS6orF1P/s/8 \n CuAkUmvnP4BXRMR9TeoUjF7HVuNF/0HqJVwO/BfpAnax/GnAOfmU6DUtth0Akg4jXSOtvc/3AtMk \n Hd1mfd9B+sL/D+l61HdIp6q1cqO19OrLHE/qfbxS0najlL2A1BIpTnsj6Zpfrcfzv1nXmp9F6gha \n Rtrv329Vv/yFPww4lXXH5UmkYNHqWLmK1DG0RNJfR3nvtW09SDrN3bYw7RLy9b58Or4AeFmj5YET \n ST3XS0g99BeQrvM1Klsbj8LrXwN7kVqNHyXdTXF/nn80qTV/Nym4figift5gPTUvA26S9CDpTouj \n IuKxZh9CVdQuuptZH8pnSTtFxJt7XZdBUZmfoZkNA6WfBz5DyXTSKejFoy1n7XPQGweSDsk/wfqT \n pPePvoT1C0lnSVoqaUGPqrA16XT9IdIlj89ExOwe1WUg+fS2y/LPd/5IugH2LtL1xqMjYmFPK2Zt \n kXQQKeCcW+gttgHill73TQdui4hF+T65C0kX0a0CImIO6WZlG1AOet23KxveXFp/75SZ9YiDXvf5 \n eoFZH3PQ67672PCO+kY/JTKzHnDQ674bgL0kTZU0ifS7Tve+mfUJB70uyz+rOpH0aKebge+657Y6 \n JF1A+rnW3pLulOSbggeMb1kxs6Hilp6ZDRUHPTMbKg56ZjZUHPTMbKg46JnZUNkYOUGbkuSuY7Me \n iYiOngo+1u9vp9vrlp4GPUjPCR9UI6RUWYNq1kDvPRjsPTirK2tpN1v7B7uyte7oedAzs+oaS/KQ \n fuGgZ2alVTGAuCNjHE3
" text/plain " : [
" <matplotlib.figure.Figure at 0x10c7c6e10> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" [[2820 30] \n " ,
" [ 124 359]] \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAUkAAAEWCAYAAADrUmWKAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAHrBJREFUeJzt3Xm8HGWd7/HPNyGBsBh2JSoJKhkiw1VQiYrRMIOKXB0G \n xUFQFMWLc3G7gsOIG8RxGUbFXcaFKKCAiKKgIDBKJKgY0ERZgoAaCJBElhB2svC7fzxP59TpdFf3 \n 6dOnT9c53/frVa9TXetTS//OU09V108RgZmZNTZhtAtgZtbPHCTNzEo4SJqZlXCQNDMr4SBpZlbC \n QdLMrMSoBklJUyRdJOl+Sd8bxnLeKOnSbpZttEiaI+mmDuf9O0lLJD0g6V3dLttIkXSypLNy/66S \n HpSk0S5XrwznmHdh3UdJWjiCy79Y0pGFzx+XdLekuyQ9vQrHuq0gKekISdfmDborb/h+XVj/ocDO \n wPYRcVinC4mI70bEK7tQnhEl6QlJzyibJiIWRsQeHa7iBODnEfGkiPhyh8sYRNLuks6V9DdJayTd \n LOmLkp7ajeVnGx/WjYjbI2Kb6MIDvJIWSDq6ZPyMfEwezN1KSV+RtNlw192iXIPOg2Ee83bW90pJ \n V+Z/nn/L++U1I7W+oog4KCI2/gMEjgP2iIhpEbG8W8d6JLUMkpKOAz4HfJwU0J4OfAX4py6sfzpw \n c0Q80YVlVUXT/5pd+HJOB27sZEZJExsMexbwW+AO4LkRMRXYD/gz8JImyxnRADNE7X75pkbENsBe \n wIuAd45ckTbqSe1J0qHAecC3gadGxM7AR4GeBMk6uwL3RsS9w11Qo/N1xERE0w6YCjwIvK5kms2B \n zwN35u5zwOQ8bi7pC3YcsAq4Czgqj5sHPA6szet4G3AycFZh2TOAJ4AJ+fNRpC/oA8BfgCMKwxcW \n 5nsxcA1wP7AIeFFh3ALgY8BVeTmXAjs02bZa+f8N+Fsu/z8DBwE3A/cCHyhMvy/wG2B1nvZLwKQ8 \n 7sq8LQ/l7X19YfknACuAM/Kw5XmeZ+Z17J0/TwPuBl7aoKy/ANYDj+btelY+fmfmsi8DPgSosM9+ \n BZwK3AN8rMEyvwP8uMU50mgbtgV+ktd7H3AR6Qtam2c34Je5nJfl/XRWk2M+FTg97887gP+oOx+u \n Aj6d1/MX4MA87hOF/fEg8MUGZR+0rjzsFOBrhc+z8jmzGrgeeE3d96PZ/n1W3sb78zE7p8V5sLyw \n 3GXA8cAf8vznApsXxp9Q2B9vz8t7RoPtE3A7cHzJ8TuKwd+dL+R51gDXAi+pO7+vzeNWAp/Nw7fI \n 58o9eT8tAnYqfN+OBv4ReATYkLd7fgfHuvR8HamuVZA8EFhXPIkaTPMx4NfAjrn7VW0D8sFfRwp+ \n E4FXAQ+T/nMDnAScWVjWSTQJksBW+eDsnsc9GXh2/YEGts8H6o15vjeQvkDbFQ7aLaSTeAvgCuBT \n JQFgHfDhXP635wP03VyeZ+cDPz1Pv08+kSYwUKt7b2F5g07mwvI/BUzK5ZnL4C/M24EbgCmkgP5f \n JcfiCuBthc9nAhfksk4H/lQbn/fZOlKtaQKwRYPlrQDe3EaQrN+G7YFDcv/WpJrMBYV5fgN8Jk8/ \n hxQsz2wSJC8ATsvbvxOpZntMYRvWkr6EAv4VuLPZ/igJkhPz52nAEgb+kU8CbgU+AGwG7J/LOrON \n /XsOcGLunwy8uMV5UDzmfwWuBp4CbJfPo3cUvpMrSMF7Cik4baBxkNwjr2v6EILkG/M6J5AqNysY \n qPT8Bnhj7t8S2Df3vwO4MB9vAXsD29QfA+Bldds51GNder6OVNcqSL4RWNFimlvJ/73z51cAfy0c \n /EcY/J96VWHnnszgoFj/eeNOzCfiauC1wJRmBxo4Eri6bvyvgbcUDtoHC+P+L3BJSQB4hIHawTa5 \n PC8oTHMtcHCT+f8f8MMWX47Haydhoy9MHvZj4DrSF3hSybG4Ajg690/My96jMP4Y4IrCPrutxbFd \n B7yi8Pld+Rg8CHy92TY0WM5zgfty/655uVMK479Lg5ok6R/hY8UvBHA48IvCNtxSGLdlnnfn+v3R \n pFy1da3O3ROkmmntCz6HuvMfOJv0z7zV/j0D+BqFGnSL86A+SB5R+HwKcFrunw98ojDumfXLK4zb \n L48rOzZHUQiSDcbfB+yV+39J+o7uWDfNW0mVo72anJNva7KdQz3WpefrSHWt2iTvBXaUVDbdNOC2 \n wufb87CNy4jBbY6PkGoXQxIRDwOHkWoLd0n6iaS/a1Ke2+uG3VZXppWF/kdblOfeyEcpTwsp0Bfn \n 3wpA0sxcrhWS1pAu+XYoWTbA3RGxtsU03wT2BL4UEetaTFsr646kmlD9sSnecFneYln3UthvEfHl \n iNiO1LxSbHsctA2StpT0NUnL8n74JTA138WcBqyOiEcL8xfLWDQ9b8MKSaslrQb+m1TLqNl4LCPi \n kdxbPJ5Bazvk7dqS9A+19qTENDbdR7VzaQfK9+8JpFrVIknXS3prG+Uoqj9Ht8r9u9SV6Y6SZdTa \n /nZpd6WS3i/pxvzEyWrSJfCOefTRwExgqaRFkv53Hn4WaZ+dK+lOSad00DbdzrFudb6OiFZB8jek \n /5aHlExzF+k/Qs2ueVgnHiKdqDVPKY6MiMsi4hV5+E3ANxos407SDi+anoePtNNIl0bPinST40O0 \n 3selX2JJW5OC0jeBeZK2a7Ms95BqbDMKw3Zl8JeqVQD5OanmvkmxGHzjoX45x5O+TPvm/fCywjwr \n gO0kFY/z9CZlWU46/3aIiO1yNzUi9mpR7mblKp844jFSDfCFkrYnncdPr3tEpXYule7fiFgVEcdE \n xFNJl6NfbfVkQ5tWkG6e1jy92YSky//lpKdIWpI0h9T+/vqI2Db/41hDPtYRcWtEHBERO5Fqt+dL \n mhIR6yPiYxGxJ+l+wKuBNw9xu9o51kM6nt1S+gWOiDWkO2FfkXRwriFMkvQqSafkyc4BPixpR0k7 \n 5unP6rA8S4CX5uenpgIn1kZI2jmXYSvSyfkwqS2m3iXATEmHS9pM0mGktpmfFKYZqTuLW5MuRR+R \n tAfpUr5oFenyaCi+ACyKiGOAn5L+u5apndAbSG2Bn5C0taTpwPtIbVjtOhmYI+mzkqYB5GM8i/IT \n dmtS7WdNDjYn1UZExG2kJop5+Vx6CelLtYmIWEG6sXOqpG0kTZD0TEkvbbP87e5vAUjanNRcsyIi \n 7iO1iT0CnJDLOjeX9dx8ddR0/0p6vaSn5eXfT9pftSuqTs6D2jl7HvBWSXvkfzQfaTZDvgI6DvhI \n fh7ySXkfvkTS1xrMsg3pZtc9kiZL+ijwpI0FkN4kqVazW1PbJkn7S9or33F+kPT9bPTdbKoLx3rE \n tHwEKCJOJe3oD5Pu4t0OHEtqZIX0aNC1wB9zd20etnERZYsvjo+I/wG+l5dzDemuaG38BNJJeCfp \n MmIOA0Fo43IiPV7walJt5h7g/cCr80nfqExB6zKWfS56P3AEqXH/66S7ksXpTwbOyJcTh5asOwAk \n HUxq461t53HAPpIOb7O87yb9M/kLsJDU9vetwnSl/5kj4hZgNvA04A+SHiC12d3B4C9n/XI+T2p8 \n v4d0+XpJ3TRH5OXeR/qnekbJNryZdOPjxjz99xm4wmi0DcXPXwAOlXSfpM+XbOr9kh4kXeLOJj/e \n lpsQXkO64Xg38GXgyIi4Oc+3yf6NiPl53POBq/Nyfwy8JyKW5XEn0955UNym2vn9M+CLpLa+m0lX \n e5BqYZvOGPEDUjPV20jfnZWkm60/ql828LPc3Uy6w/4og5uuXglcn7fpc8AbIuJxUnvi90mB80bS \n zdFmFaWy4zXUY90TtRsSZlZBkmaRbupNjvH1vHHP+LfbZhUj6RBJm+f26VOACx0gR46DZJ+QdKCk \n myTdIunfR7s81j5J8yWtknRdj1Z5DKld81ZS+19927d1kS+3+0Bu8P4TcACp3ega4PCIWDqqBbO2 \n 5LvCD5EeiG/3zrtVhGu
" text/plain " : [
" <matplotlib.figure.Figure at 0x10d212790> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" [[2769 81] \n " ,
" [ 381 102]] \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAT0AAAEWCAYAAAAU8EBoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAHS9JREFUeJzt3Xm8JHV57/HPd1Z2EBEFBAZlUdCrLM6oXHBU4AVGJbhj \n QkDxYkSEq0QF48IYl4i5GGPEXGURVFCDu0KAKEdBhVFkZBEckG3YRvZBBmZ98sfv13NqerrP6VPd \n Pd116vt+vep1uquqf/V0V52nf7V0PYoIzMzqYsqgAzAzW5+c9MysVpz0zKxWnPTMrFac9MysVpz0 \n zKxWKpf0JG0o6UeSHpH0rS7a+RtJF/cytkGRtJ+km0q+djdJCyQtkXRcr2PrhU7fn6STJX1ljOmT \n Zp1PhKTHJM0adBzDQv26Tk/SW4H3AbsBjwELgE9GxC+7bPcI4DjgJRGxuutAh5yk1cDOEXFrn9o/ \n E3gkIk7sUXunAM+OiCN60V4XccwCbgWmdbudSPoqcDiwHFgBXAMcHxE3dBelDUJfenqS3gd8DvgE \n sDWwPfBF4LU9aH5HYGEdEl6B2k6QpnXZ9o7AH8q8UNLUFqOH7Wr3tp/dBATwmYjYFNgWuBM4uwft \n rqUH69I6ERE9HYDNST27148xz0zgX4G78/A5YEaeNhe4i9RLXAzcAxyVp80DlpG+cR8D3g6cAnyt \n 0PYsYDUwJT8/CvgTsIT0zf/WwvjLC697KfAb4BFgPqkn2Zg2AnwcuCK3czHw1DbvrRH/+4E/5/j/ \n GngVsBB4EDipMP9s4NfAw3neLwDT87Rf5Pfyl/x+31ho/wPAvcA5edyi/Jpn52XsmZ9vC9wP7N8i \n 1p8BK4En8vvaOa+/c3PstwP/yOgewVHAL4HTgAeAj7doc6310TTttcAN+b1eBjynMG0vUg9qCfBt \n 4FvAPxU+00WFeT+YP4MlwE3AK5qXTUpMq/PntgR4cYt1vgdwaf687gNObhP32cX3mtfl44Xn2wLf \n yZ/ZrcB7CtM2zOvoIdKXywea3svtedy1eT1MybH+Kn9OC4CXFeY/itbb887Az0nb7/3ANwuvWQ08 \n q/D/Odb6vQL4bI73VuDgXueIQQ/9SHoHk3YBpowxz8fzSt0qD79sbFR5A1+RN+CpwCHA48DmefrH \n gHMLbX2MNkkP2Bh4FNglT3s6sHthBV+eH2+ZN7C/ya97S17pT8nTR4Cb84a1Aekf9tNt3lsj/g/n \n +N9BShDfyPHsDiwFdiz8s8/Oy230uk5otcE2tf9pYHqOZy5r/yO9g5RcNiQl6FPHWBeXAW8vPD8X \n +F6OdUfgj43p+TNbAbw7x7tBi/ZOoUXSA3YlJe9X5s/l/fkznQbMAO4A3pOnHUb6cituE42kvhsp \n oT0jP9+B0X/oNdtCjn3Nl1+Ldb4p6UvjvXn5mwCz23xGZzOagDcGvgb8LD+fAlyd1/c0YCdSUjoo \n T//n/BlvDmxHSm53Ftq+HfhdnjYz/32AnGyAA/LzpzL29nw+OWnn9/PSVttQB+t3OXA0qYf898Dd \n g05SPc9RPW8wJY57x5nnFgrfIMBBwG2FDXxp08a6uLFBsm7Prvn5LNZOeg8DrwM2bIqh+A9wBHBl \n 0/RfAUfmx5cBHypMexdwUZv31oi/8e25aY7nRYV5fgsc2ub1/xf4bqsNttD+MnLPuDBuUVM7PwCu \n I/UUpo+xLi4Djs6Pp+a2iz2wY4DLCp/ZHeOs27XWR2H8R1i79yFSb+1lwP7AXU3zX07rpLdz3h5e \n 2fy+WLunt2Y7aLPODweu7nCb/iqpF/YwsIqU1LbK0+Y0fybAycBZ+fGfgAML045m7S+o28h7Mvn5 \n Byl8qedx/wX8HbAR7bfnc4D/D2zXIv7VwLM6XL83F6ZtlF+7dSefU1WGfhzTexDYStJYbW9L+mZv \n uDOPW9NGrH3Mbinpm3hCIuJx4M2kb6x7JP1Y0m5t4rmzadwdTTHdV3j8xDjxPBh5q8nzQvpHLb5+ \n YwBJu+a47pX0KPBJ0rf6WO6PiOXjzHMGafftCxGxYpx5G7FuReo9Nq+b7QrPF43TVjvbUPiM8+ez \n KLe9DekwR1HL5UTELaQvhlOAxZLOl7RNiXi2J+2+dSKAz0bEU0jJdBkpCUHqLW0r6eHGQEp6W+fp \n 2za9l7tatF+cviPwxqb29iX1bJfSfnv+AOmLZL6k6yW9rcVyOlm/a7bzvDwo8b83zPqR9H5N2igO \n G2Oee0gbT8MOeVwZfyF9IzU8ozgxIi6JiIPy+JuAVpc03E3a2Ip2ZN1/xH74EmmXdueI2Jx0jGW8 \n 9RJjTZS0CemY6RnAPElP6TCWB0i7r7MK43Zg7X/UMZc9xvR7KHzGkkRKPHeRdjO3a5p/h7YLiDg/ \n IvbL7QXwmQnE0XAnqffTKeVlLwKOBz4iaTNSwrotIp5SGDaLiFfn191Lep8N27OuYqx3knqrxfY2 \n jYhT8/Jbbs8RsTgijomI7YB3AqdLan5/nazfSa/nSS8iHgU+CnxR0qGSNpI0XdIhkhob5/nAhyVt \n JWmrPP/XSi5yAbC/pO0lbU76lgVA0tY5ho1JK/tx0u5Js4uAXSUdLmmapDcDzwF+XJinF2cBW9mE \n dLB9qaTnkHadixaTTk5MxOeB+RFxDPAT4D/Gmb/xD72KdBLhk5I2kbQj6ZjX1yewbAFTJM2UtEEe \n ZuZ2/0rSKyRNB04EniQdRrgSWCXpuPz5Hwq8qGXjqWf8itzmstxGq3V6P2nXrN1n9xNgG0kn5Fg3 \n lTR7jPe0RkT8N+kQzbuAq4DHJH0gX0M6VdLzJO2TZ/82cLKkLSRtR7rcaqyE/HXgNZIOym1tIGmu \n pO3G2p4lvVHSM3Mbj+RlrHWFQ4/Wb+X15ZKViDiNdPb1w6SzRHcCx5IOoEK6lOW3pIO61+bHnyg2 \n MVbzxel5A/xWbuc3wI8K06eQVurdpN3u/RhNKmvaiYgHgVeT/hEfAP4BeHVEPNQmpmD8GMd6XvQP \n wFtJZ+O+DHyzaf5TgHPyrs4bxlh2OlCWEsZBjL7P9wF7STq8w3jfQ/pnupV0XO0bjF6eMd77bsxz \n OGkXfmkebo6IhcDfks5O3w/8FfCaiFiZd9VfRzre1Tih9GPSQfXmGGeSTuLcT+pFbcXoF11xnS4l \n HSr4paSHJM1pmv4YcCDwmtzOQtKxw3bvqfl9f5bU45tK2nZeSPrM7ietx83yfB8n9aRuAy4B/rPp \n fa29oIi7gEOBDzH6v3Mi+cuE9tvzPsCVkh4jHc89PiJuL8TfMNH1O976rpy+XZxs1g1JVwGnR8Q5 \n g46llyS9C3hTRLx80LHUVeV+hmaTk6T9JT0j794eCTyPdNay0vJ72lfSlHzS4X2M7vHYADjp9YGk \n gyXdJOlmSR8cdDwVsRvp+OzDpF24N0TE4rFf0nuSzpK0WNJ1PWpyBumY6hLgp8D3gdN71LaV4N3b \n Hss/zfoj6aLSu0nHGQ+PiBsHGph1RNJ+pCsCzo2I5w86Hus99/R6bzZwS0Tcnq+P+ybpwLRVQERc \n Tupt2iTlpNd727HuxajN16CZ2YA46fWejxeYDTEnvd67m3WvwK/VFe9mw8xJr/d+C+wiaZakGaTf \n Sv5wwDGZWeak12MRsZL0U6OLSb+p/ZbP3FaHpPNJP43bVdKiNj/ctwrzJStmVivu6ZlZrTjpmVmt \n OOmZWa046ZlZrTjpmVmtDLTOpiSfOjYbkIjo6m7gE/3/7XZ5vTLw4sIfG3QAfTRC+1vxTgbzJvXa \n g8m9Buf1pJVPjD8LkG6hXiRpe1I5yq1JP938ckT8m6RvkcqFAmwBPBIRe+bXnEyqdb2KdGfoS/L4 \n vUkV6zYALoyIE8aKZeBJz8yqa3r5l64A3hsRC3Ihq6slXRoRb27MIOlfSPU+kLQ76ddNu5Nu4PHf \n knbJVfW+RCpjOl/ShZIOjoi2N6D1MT0zK21ah0OziLgvIhbkx38BbqRQcjVXy3sTqYgYpNuznR8R \n K3Ltj1uAObn856YRMT/
" text/plain " : [
" <matplotlib.figure.Figure at 0x10d38d7d0> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
}
] ,
" source " : [
" from sklearn.metrics import confusion_matrix \n " ,
" from sklearn.metrics import precision_score \n " ,
" from sklearn.metrics import recall_score \n " ,
" \n " ,
" def draw_confusion_matrices(confusion_matricies,class_names): \n " ,
" class_names = class_names.tolist() \n " ,
" for cm in confusion_matrices: \n " ,
" classifier, cm = cm[0], cm[1] \n " ,
" print(cm) \n " ,
" \n " ,
" fig = plt.figure() \n " ,
" ax = fig.add_subplot(111) \n " ,
" cax = ax.matshow(cm) \n " ,
" plt.title( ' Confusion matrix for %s ' % c lassifier) \n " ,
" fig.colorbar(cax) \n " ,
" ax.set_xticklabels([ ' ' ] + class_names) \n " ,
" ax.set_yticklabels([ ' ' ] + class_names) \n " ,
" plt.xlabel( ' Predicted ' ) \n " ,
" plt.ylabel( ' True ' ) \n " ,
" plt.show() \n " ,
" \n " ,
" y = np.array(y) \n " ,
" class_names = np.unique(y) \n " ,
" \n " ,
" confusion_matrices = [ \n " ,
" ( \" Support Vector Machines \" , confusion_matrix(y,run_cv(X,y,SVC)) ), \n " ,
" ( \" Random Forest \" , confusion_matrix(y,run_cv(X,y,RF)) ), \n " ,
" ( \" K-Nearest-Neighbors \" , confusion_matrix(y,run_cv(X,y,KNN)) ), \n " ,
" ( \" Gradient Boosting Classifier \" , confusion_matrix(y,run_cv(X,y,GBC)) ), \n " ,
" ( \" Logisitic Regression \" , confusion_matrix(y,run_cv(X,y,LR)) ) \n " ,
" ] \n " ,
" \n " ,
" # Pyplot code not included to reduce clutter \n " ,
" # from churn_display import draw_confusion_matrices \n " ,
" % matplotlib inline \n " ,
" \n " ,
" draw_confusion_matrices(confusion_matrices,class_names) "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" An important question to ask might be, When an individual churns, how often does my classifier predict that correctly? This measurement is called \" recall \" and a quick look at these diagrams can demonstrate that random forest is clearly best for this criteria. Out of all the churn cases (outcome \" 1 \" ) random forest correctly retrieved 330 out of 482. This translates to a churn \" recall \" of about 68 % (330/482≈2/3), far better than support vector machines (≈50 % ) or k-nearest-neighbors (≈35 % ). \n " ,
" \n " ,
" Another question of importance is \" precision \" or, When a classifier predicts an individual will churn, how often does that individual actually churn? The differences in sematic are small from the previous question, but it makes quite a different. Random forest again out preforms the other two at about 93 % precision (330 out of 356) with support vector machines a little behind at about 87 % (235 out of 269). K-nearest-neighbors lags at about 80 % . \n " ,
" \n " ,
" While, just like accuracy, precision and recall still rank random forest above SVC and KNN, this won ' t always be true. When different measurements do return a different pecking order, understanding the values and tradeoffs of each rating should effect how you proceed. "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" ## ROC Plots & AUC \n " ,
" \n " ,
" Another important metric to consider is ROC plots. We ' ll cover the majority of these concepts in lecture, but if you ' re itching for more, one of the best resources out there is this [academic paper](https://cours.etsmtl.ca/sys828/REFS/A1/Fawcett_PRL2006.pdf). \n " ,
" \n " ,
" Simply put, the area under the curve (AUC) of a receiver operating characteristic (ROC) curve is a way to reduce ROC performance to a single value representing expected performance. \n " ,
" To explain with a little more detail, a ROC curve plots the true positives (sensitivity) vs. false positives (1 − "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 11 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Support vector machines: \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAX4AAAEZCAYAAACQK04eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAIABJREFUeJzsnXeYJFX1v9/P5gyzIGHJGZYcJCihF1QWJJgQUUEMNIqg \n P1EkKLArSRQRBcEtQRAUlChBEBWmSV+CZEQQlrTksIFdWDbO+f1xbs/W9HT39Mx0mOm57/PUM91V \n t6pOdfecunXuuecjMyMSiUQiA4dBjTYgEolEIvUlOv5IJBIZYETHH4lEIgOM6PgjkUhkgBEdfyQS \n iQwwouOPRCKRAUZ0/JGKkfQfSbs22o5GI+kCST+u8zkvkXRKPc9ZKyR9SdKtPdw3/gargGIef/9E \n 0ovASsBS4H3gn8C3zWxuI+1qNiQdCnzdzHZpsB0XAy+b2UkNtmMKsJ6ZHVyHc12CX/OJtT7XQCP2 \n +PsvBuxjZmOBLYHNgbr2QquBpCED8dyNRNLggXjuyDKi428CzOxN4B/Apvl1knaU9H+SZkt6VNJu \n qW3jJV0s6VVJsyRdl9q2T2g/W9I9kjZPbXtR0u6SJkiaL6kltW1rSW/n/7ElfU3Sf8Px/y5pzVTb \n NklHSHoW+F+xa5K0n6Qngx2tkjYusOO4sH2WpN9LGt6Na/ihpMeBeZIGh2NNlzQ3HPNToe0mwAXA \n TpLmSZoV1reHXSRlJL0i6WhJb0p6LTwl5M+3gqQbJb0r6QFJp0q6q9R3KWnn1Pc2Q9Ihqc3jJd0U \n 7LxP0rqp/X4V2r8r6UFJO6e2TZF0taTLJL0LfEXShyXdG87zmqRzJQ1N7bOppH9KminpDUnHS9oT \n OB44MHwej4S2y0m6KBznFUmnSBoUth0avoOzJb0DTAnr7grbJemX4bN7V9Lj4dxZ4IvAD8O5rk99 \n f3uE14MlnZD67h6UtHqpzzaSwszi0g8X4AVgj/B6deBx4KTwfjXgHWByeP+x8H6F8P5vwBXAcsAQ \n YJewfmvgTeDDgIBDwnmGps65e3h9G/CNlD0/B84Pr/cHngU2wjsXPwLuSbVtA24FlgeGF7m2DYH3 \n gD2AwcAx4XhDwvYXw/WuBrQAdwOnVHgNLwIPh32Hh3WfA1YJrz8fzr1yeP8V4K4C+y4GfhJeZ4DF \n wJRg61546G25sP3PwOXACGATYAZwZ4nvdC1gLnBgONZ4YMuw7ZLwHW4Xtv0RuCK175fCZzEIOBp4 \n HRgWtk0BFgH7hfcjgG2A7UP7tYD/At8N28eG/b8HDAPGANuHbScDlxbYfR1+gxwJfAi4H8iGbYeG \n z+fb4Vwjwrq7wvY9gQeBceH9Rqnvov1zLvjd53+Dx4TfwQbh/ebA+Eb/b/aHpeEGxKWHX5w7sHnB \n UbSFf75BYduxRf45/447wVXxcYHlihzzgiL/aE+z7MaQ/qf7OnBbeK3g0HYO728BvpY6xiDcGa4R \n 3rcBmTLXdiLw59R7Aa8Au6bsyKa27wVM78Y1HNrFZ/tIykm2O6nU9otZdqPJAPPzn31Y9ybuVAfj \n DneD1LZTCo+X2nY8cE2JbRcDScE1P1XmGmYBm4fXU4BcF9f8/4Brw+uDgIdKtJsCXJZ6vzKwABiR \n WncQcHvq83up4BjtnymwO/7Ut0P6Myz8nFPr0r/B/wH7Nup/sD8vMdTTfzFgfzMbhzuf3fHeIHgP \n 7oDwGD9b0mzgo8AqwBrALDN7t8gx1wK+X7Df6sCEIm2vxUMgqwC7Am1mdnfqOL9KHWNmWL9aav+X \n y1zbqviNxC/U/8tfLrP/jJSNlVxDh3NLOkTSI6n2mwErlLGvkJlm1pZ6Px/vJX8If6JKn++VMsdZ \n HXi+zPY3U68/COcAQNIP5KG1OeEalgNWLHVeSRuGsNHrIfxzGsuueY0u7EizFjAUeD31+f0Wv/Y8 \n Jb9rM7sdOA/4DfCmpGmSxlZ47tWB5ypsG0kRHX8TYGZ3AucCZ4ZVM/BeWUtqGWtmP8P/CcdLWq7I \n oWYApxXsN8bM/lLknLPxcYUD8VjsFQXHyRYcZ7SZ3Zc+RJlLeg13KIDHgXFn9GqqzZoFr/PbKrmG \n 9nNLWgtI8FDEeDNrAf6DP2WUs7OSdLi3gSXB9jxrlGgL/t2sV8FxOyBpFzzscYCZLR+u4V2WXQN0 \n tvcCPLyzvpkth4fj8v5gBrAuxWkreP8ysBAPI+Y/7+XMbPNUm7KflZmda2bbARPxMN8xlewXzr1+ \n F20iRYiOv3k4B9he0g54/HdfSZ8IA2AjwiDkamb2Oh6KOV/S8pKGalle9O+Ab0raPgy6jZb0SUlj \n SpzzcjwG/tnwOs9vgRMkTYT2wb8DunEtVwKflA8kDwW+j4cT/i9sF3CEpNUkjcedVt6xd/caRuMO \n 5h1gkKSv4j3+PG8Cq6cHPsP50061KGa2FH8ymiJppHyA+mBKO7Q/AR+TdICkIfKB4S1T5yzFWPwG \n 846kYZJOAsZ1Yd4YPFQ4P9j1rdS2vwGrSvqupOGSxkraPmx7E1g73IwJv6d/AGeHdoMkracKc+0l \n bSdph/D5zse/56Wpc5W6AQFcCJwiaf3wXW8Rfg+RLoiOv0kws3eAPwDHmtkr+ADrCcBbeA/u+yz7 \n vg/GB9yexv+5vhOO8RBwGP7oPQsfUD2E0o7qBrzH9bqZPZGy5a/408efQxjhCXwQr71JF9fyDPBl \n /CnmbeCTeCx3SWr/y3GH81yw89SeXIOZ/Rf4BXAv8Abu9O9ONbkNeBJ4Q9JbqfOnj1fueo7Ewy5v \n 4N/PFXjcv5gtLwN749/VTHysYYsS50yf9+9heQYf+/mAVKisxL4/wJ/U5uJPPH/OtzGzecDHgX3x \n Qd5n8HAiwFXh70xJD4bXh+CDwP/FP/Or8LBiObvz68aF888Ktr+DJwoAXARMDCGka+nM2Xgn4R/4 \n E87v8MHjSBfECVyRfoekF/BJVbc32pbuIulMYCUz+2qjbYkMXGKPPxKpIZI2CiEIhXDJ1/AMrEik \n YQzImYuRSB0Zi4d3JuBhtbPM7IbGmhQZ6MRQTyQSiQwwYqgnEolEBhj9ItQjKT6WRCKRSA8ws06p \n wP3C8UNx4ytB0hQzm1Jlc/o08ZoHBvGaBwa9ueZSneYY6olEIpEBRnT8kUgkMsAYCI4/12gDGkCu \n 0QY0gFyjDWgAuUYb0AByjTagAeSqfcCapnNK+j0+3f6tgqJN6Ta/xkvMzsfL5T5SpI31NMYfiUQi \n A5VSvrPWPf6LgcmlNkraG68OuAGQxSsGRiKRSKSG1NTxm9ldwOwyTfbDC1dhZvcDy0tauZY2RSKR \n yECn0TH+1egsUhE1MyORyIAjX+q6HvSFPP7Ci42TtSKRSNMSHPwGwEdw4ZmNGDJkU9ZddxXlco/l \n 260/f/6YFZYsGXb/uHHbWSazsJo2NNrxv0pHRaLV6aiy1I6kKam3OTPL1c6sSCTS7EjMwgXqO3Ls \n eBhZLkJdwFL42RmwdEnnTUfjQgVpngQ2LWy4ZAkjn3+dv02yXRkMM/eeybu7vstKf1mJsQ+yAOta \n +AdAUoZl2gml29W6SJuktYEbi2X1hMHdI81sb0k7AueY2Y5F2sWsnkgkUlUkzIo4VE2VMYUd8N54 \n flkPl/hcw8wWdTyONALaFhQ/zVgze6+wPfDoruy6xZ3LPXYWe+65CltuOZzNN19j2uuvHwFcggvp \n HJ7NZl/r3TUW95017fFLugLYDVhR0svAybgwM2Y2zcxulrS3pOnA+0AUp4hEIj2iZA++E+/CZ1aD \n Hd9HJ2EM7tRgNnATHQXjAfgfvEWBXrUBx8KCn8Gv6RyqXlx4jFZaZxrWIjR70l95EPgJcNPn3n77 \n JeBWXB3tsmw2W7Neeb8oyxx7/JFIpCukhQYjOvWwfZvOAD4KrIMnleT9yc5mdk+R9n8ARjJ+/Ous \n uOIrrLrqS2y22atPX3fd3Rv/6U8b9sbO1kk8M6mV/DE+CexkmcyBSZKMA8b0tpefppTvjI4/EonU \n nXzv/HruZhxFguNd8Az
" text/plain " : [
" <matplotlib.figure.Figure at 0x10c334610> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Random forests: \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAX4AAAEZCAYAAACQK04eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAIABJREFUeJzsnXl4JFXVuN/TW5bJMpl9X5iVYR1AcGEJoAIK4wLI5wIi \n Yo8K6geICj+RGQH9UFT8UGFaEZUdBD9AUFSYZt+RVQZm3zNLMplM9qT7/P64t5NKpzvpZLrTyeS+ \n z1NPd1XdunWql1Onzj33HFFVHA6HwzF88OVbAIfD4XAMLE7xOxwOxzDDKX6Hw+EYZjjF73A4HMMM \n p/gdDodjmOEUv8PhcAwznOJ3ZIyIvCUix+ZbjnwjIjeKyPcH+Jx/EJGrBvKcuUJEPi8ij/bzWPcb \n zALi4viHJiKyDhgHxIAG4J/ABapal0+59jVE5Fzgy6p6TJ7luAXYqKo/yLMcS4BZqnr2AJzrD5hr \n viLX5xpuOIt/6KLAqapaChwCHAQMqBWaDUQkMBzPnU9ExD8cz+3oxCn+fQBV3Qb8AzggsU1E3i8i \n z4rILhF5TUSO8+wbJSK3iMhmEakRkb949p1q2+8SkWdE5CDPvnUicoKITBKRRhGp8OxbKCI7En9s \n ETlPRP5j+/+7iEzztI2LyNdFZCXwbqprEpFFIvK2lWO5iMxPkuN7dn+NiPxeRAr6cA3fEZE3gD0i \n 4rd9rRKROtvnJ23b/YEbgQ+IyB4RqbHbO9wuIlIpIptE5GIR2SYiW+xTQuJ8o0XkIRHZLSIvisjV \n IvJUuu9SRI72fG8bROQcz+5RIvJXK+fzIrKf57hf2va7ReRlETnas2+JiPxZRG4Vkd3AF0XkfSLy \n nD3PFhG5QUSCnmMOEJF/iki1iFSJyGUichJwGXCW/Tz+bduWi8jNtp9NInKViPjsvnPtd/BzEdkJ \n LLHbnrL7RUR+YT+73SLyhj13GPgc8B17rgc839+J9r1fRC73fHcvi8iUdJ+tw4OqumUILsBa4ET7 \n fgrwBvADuz4Z2AmcbNc/bNdH2/WHgTuBciAAHGO3LwS2Ae8DBDjHnifoOecJ9v1jwPkeeX4K/Ma+ \n /wSwEpiHMS7+H/CMp20ceBQYCRSkuLa5QD1wIuAHLrX9Bez+dfZ6JwMVwNPAVRlewzrgVXtsgd12 \n BjDBvv+MPfd4u/5F4Kkk+W4BfmjfVwJtwBIr6ykY11u53X8XcAdQCOwPbACeTPOdTgfqgLNsX6OA \n Q+y+P9jv8Ai77zbgTs+xn7efhQ+4GNgKhOy+JUArsMiuFwKHAUfa9tOB/wDfsvtL7fEXASGgBDjS \n 7rsS+FOS3H/B3CCLgLHAC0DY7jvXfj4X2HMV2m1P2f0nAS8DZXZ9nue76Pick373id/gpfZ3MMeu \n HwSMyvd/cygseRfALf384owC22MVRdz++Xx233dT/Dn/jlGCEzHjAuUp+rwxxR9tBZ03Bu+f7svA \n Y/a9WIV2tF3/G3Cepw8fRhlOtetxoLKHa7sCuMuzLsAm4FiPHGHP/lOAVX24hnN7+Wz/7VGSHUrK \n s/8WOm80lUBj4rO327ZhlKofo3DnePZdldyfZ99lwH1p9t0CRJKu+Z0erqEGOMi+XwJEe7nm/wbu \n t+8/C7ySpt0S4FbP+nigGSj0bPss8Ljn81uf1EfHZwqcgHnqO8r7GSZ/zp5t3t/gu8Bp+foPDuXF \n uXqGLgp8QlXLMMrnBIw1CMaCO9M+xu8SkV3Ah4AJwFSgRlV3p+hzOnBJ0nFTgEkp2t6PcYFMAI4F \n 4qr6tKefX3r6qLbbJ3uO39jDtU3E3EjMhZp/+cYejt/gkTGTa+hybhE5R0T+7Wl/IDC6B/mSqVbV \n uGe9EWMlj8U8UXnPt6mHfqYAa3rYv83zvsmeAwAR+bYY11qtvYZyYEy684rIXOs22mrdP9fQec1T \n e5HDy3QgCGz1fH43Ya49QdrvWlUfB34F/BrYJiLLRKQ0w3NPAVZn2NbhwSn+fQBVfRK4AbjWbtqA \n scoqPEupqv4E8yccJSLlKbraAFyTdFyJqt6d4py7MOMKZ2F8sXcm9RNO6meEqj7v7aKHS9qCUSiA \n 8QNjlNFmT5tpSe8T+zK5ho5zi8h0IIJxRYxS1QrgLcxTRk9yZhIOtwNot7InmJqmLZjvZlYG/XZB \n RI7BuD3OVNWR9hp203kN0F3eGzHundmqWo5xxyX0wQZgP1ITT1rfCLRg3IiJz7tcVQ/ytOnxs1LV \n G1T1CGABxs13aSbH2XPP7qWNIwVO8e87XA8cKSJHYfy/p4nIR+0AWKEdhJysqlsxrpjfiMhIEQlK \n Z1z0b4GvisiRdtBthIh8XERK0pzzDowP/HT7PsFNwOUisgA6Bv/O7MO13AN8XMxAchC4BONOeNbu \n F+DrIjJZREZhlFZCsff1GkZgFMxOwCciX8JY/Am2AVO8A5/2/F6lmhJVjWGejJaISJGYAeqzSa/Q \n bgc+LCJnikhAzMDwIZ5zpqMUc4PZKSIhEfkBUNaLeCUYV2Gjletrnn0PAxNF5FsiUiAipSJypN23 \n DZhhb8bY39M/gJ/bdj4RmSUZxtqLyBEicpT9fBsx33PMc650NyCA3wFXichs+10fbH8Pjl5win8f \n QVV3An8EvquqmzADrJcD2zEW3CV0ft9nYwbcVmD+XN+0fbwCfAXz6F2DGVA9h/SK6kGMxbVVVd/0 \n yPJ/mKePu6wb4U3MIF5Hk16u5T3gC5inmB3AxzG+3HbP8XdgFM5qK+fV/bkGVf0P8DPgOaAKo/Sf \n 9jR5DHgbqBKR7Z7ze/vr6XouxLhdqjDfz50Yv38qWTYCH8N8V9WYsYaD05zTe96/2+U9zNhPEx5X \n WZpjv415UqvDPPHclWijqnuAjwCnYQZ538O4EwHuta/VIvKyfX8OZhD4P5jP/F6MW7EnuRPbyuz5 \n a6zsOzGBAgA3AwusC+l+uvNzjJHwD8wTzm8xg8eOXnATuBxDDhFZi5lU9Xi+ZekrInItME5Vv5Rv \n WRzDF2fxOxw5RETmWReEWHfJeZgILIcjbwzLmYsOxwBSinHvTMK41a5T1QfzK5JjuONcPQ6HwzHM \n cK4eh8PhGGYMCVePiLjHEofD4egHqtotFHhIKH5ILXwmiMgSVV2SZXEGNe6ahwfumocHe3PN6Yxm \n 5+pxOByOYYZT/A6HwzHMGA6KP5pvAfJANN8C5IFovgXIA9F8C5AHovkWIA9Es91hTsM5ReT3mOn2 \n 25OSNnnb/C8mxWwjJl3uv1O00f76+B0Oh2O4kk535trivwU4Od1OEfkYJjvgHCCMyRjocDgcjhyS \n U8Wvqk8Bu3posgiTuApVfQEYKSLjcymTw+FwDHfy7eOfTPciFa5mpsPhGNbYNN7pUonvNYMhjj/Z \n /+Qmazkcjn2WqERlNM+WFbBjP4jPuJk3Pvw0W46pDcQm7qaptDZeFwLkCwd9RX938p8l2vra+Nse \n v3p7rx33gXwr/s10rUg0ha5VljoQkSWe1aiqRnMnlsPhcPRMVKISpLa4nDen+2maCToVZPIOmqa9 \n S82MKhrGbaexYjuNJTuoLzhMDpWP+xf5kGLZ7juS5qJ26kqV1bFmnt0eNaV0AL/fz6LTFnHECQey \n 8e0NK1gtbZnKJCKVdNZOSN8u10naRGQG8FCqqB47uHuhqn5MRN4PXK+q70/RzkX1OByOnBCVqADF \n RWycUMymmT5apgFTFN9EJTAuTmCMEqxoJ1S+m1jJFhqKqmgIjWCs/0DmEWCP+miMx3ytsT1FEr/L \n 94T/j3tuCyafp+TIE7XsvK+37A5saWvwrwrQuslH8551vLNmFU+vb6Sl7cVTDz61+pQTTvm23+9f \n IyKLw+Hwlr25tnS6M6cWv4jcCRwHjBGRjcCVmMLMqOoyVX1ERD4mIquABsAVp3A4HP0iocCF1tEl \n rJ4eZM80ITYNmKj4xlsFPipOqDxOQUmMgqIYxYXCiIAQo40gTUxoD9DY6qOpSWltbiPW8i/eDvyK \n R8dXsyfYSmvHuGhwxoc
" text/plain " : [
" <matplotlib.figure.Figure at 0x10d933b50> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" K-nearest-neighbors: \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAX4AAAEZCAYAAACQK04eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAIABJREFUeJzsnXd4HNW1wH9n1WV12XLv3bhhXMC0pYUSSoBgAwk1ZCEk \n BEINPAgmQPIghAeBkHgJvdfQSwho6c0G3HuTZUuyLMnqbbXn/XFH9lpWWdtarcr9fd98uzNz5865 \n M7tn7px77jmiqlgsFoul5+CKtAAWi8Vi6Vis4rdYLJYehlX8FovF0sOwit9isVh6GFbxWywWSw/D \n Kn6LxWLpYVjFbwkZEVkqIkdEWo5IIyL/EJGbO/icj4vI7R15znAhIj8Tkff38Vj7G2wHxPrxd01E \n ZCOQBTQAlcAHwK9VtSyScnU3RORC4BeqeniE5XgM2Kyqf4iwHPOAkap6Xgec63FMm28J97l6GrbH \n 33VR4GRVTQamAJOADu2FtgciEt0Tzx1JRCSqJ57bsgur+LsBqloA/Ac4oHGbiBwsIl+ISImI/CAi \n RwbtyxCRx0Rki4gUi8i/g/ad7JQvEZHPRWRS0L6NInK0iAwQkSoRSQ/ad6CIFDb+sUXkYhFZ7tT/ \n nogMCSobEJHLRWQNsKq5NonIqSKyzJEjW0TGNZHj987+YhF5VETi9qIN14vIYqBcRKKcutaKSJlT \n 50+csuOBfwCHiEi5iBQ723eaXUTELSK5InK1iBSIyFbnLaHxfJki8qaIlIrINyJyh4h82tK9FJHD \n gu5bjoicH7Q7Q0TecuT8SkRGBB13v1O+VEQWiMhhQfvmicjLIvKUiJQCF4jIDBH50jnPVhF5QERi \n go45QEQ+EJEiEckXkRtF5HjgRmCucz2+d8qmisgjTj25InK7iLicfRc69+BeEdkOzHO2fersFxH5 \n P+falYrIYufcHuBc4HrnXK8H3b9jnO9RInJT0L1bICKDWrq2liBU1S5dcAE2AMc43wcBi4E/OOsD \n ge3ACc76sc56prP+NvAckApEA4c72w8ECoAZgADnO+eJCTrn0c73D4FLguT5C/CQ8/00YA0wFtO5 \n +B/g86CyAeB9IA2Ia6ZtY4AK4BggCrjOqS/a2b/Rae9AIB34DLg9xDZsBL5zjo1ztv0U6Od8n+Oc \n u6+zfgHwaRP5HgP+6Hx3A/XAPEfWEzGmt1Rn//PAs0A8MB7IAT5p4Z4OBcqAuU5dGcAUZ9/jzj2c \n 7ux7Gngu6NifOdfCBVwN5AGxzr55QB1wqrMeD0wDZjrlhwLLgSud/cnO8b8DYoEkYKaz71bgySZy \n /xvzgEwA+gBfAx5n34XO9fm1c654Z9unzv7jgQVAirM+Nuhe7LzOTX73jb/B65zfwWhnfRKQEen/ \n ZldYIi6AXfbxxhkFVu4oioDz53M5+25o5s/5HkYJ9seMC6Q2U+c/mvmjrWTXgyH4T/cL4EPnuzgK \n 7TBn/V3g4qA6XBhlONhZDwDuVtp2C/B80LoAucARQXJ4gvafCKzdizZc2Ma1/T5ISe5UUkH7H2PX \n g8YNVDVee2dbAUapRmEU7uigfbc3rS9o343AKy3sewzwNmnzilbaUAxMcr7PA3xttPkq4FXn+znA \n whbKzQOeClrvC9QA8UHbzgE+Crp+m5rUsfOaAkdj3vpmBV/Dptc5aFvwb3AVcEqk/oNdebGmnq6L \n AqepagpG+RyN6Q2C6cGd5bzGl4hICXAo0A8YDBSramkzdQ4Frmly3CBgQDNlX8WYQPoBRwABVf0s \n qJ77g+oocrYPDDp+cytt6495kJiGmn/55laOzwmSMZQ27HZuETlfRL4PKj8RyGxFvqYUqWogaL0K \n 00vug3mjCj5fbiv1DALWt7K/IOh7tXMOAETkWjGmtR1OG1KB3i2dV0TGOGajPMf8cye72jy4DTmC \n GQrEAHlB1++fmLY30uK9VtWPgAeBvwMFIjJfRJJDPPcgYF2IZS1BWMXfDVDVT4AHgLucTTmYXll6 \n 0JKsqndj/oQZIpLaTFU5wJ1NjktS1ReaOWcJZlxhLsYW+1yTejxN6umlql8FV9FKk7ZiFApg7MAY \n ZbQlqMyQJt8b94XShp3nFpGhgBdjishQ1XRgKeYtozU5Q3GHKwT8juyNDG6hLJh7MzKEendDRA7H \n mD3OUtU0pw2l7GoD7CnvPzDmnVGqmooxxzXqgxxgBM0TaLK+GajFmBEbr3eqqk4KKtPqtVLVB1R1 \n OjABY+a7LpTjnHOPaqOMpRms4u8+3AfMFJFZGPvvKSLyI2cALN4ZhByoqnkYU8xDIpImIjGyyy/6 \n YeAyEZnpDLr1EpEfi0hSC+d8FmMDP9P53sg/gZtEZALsHPw7ay/a8iLwYzEDyTHANRhzwhfOfgEu \n F5GBIpKBUVqNin1v29ALo2C2Ay4RuQjT42+kABgUPPDpnD9YqTaLqjZg3ozmiUiCmAHq82hZoT0D \n HCsiZ4lItJiB4SlB52yJZMwDZruIxIrIH4CUNsRLwpgKqxy5fhW0722gv4hcKSJxIpIsIjOdfQXA \n MOdhjPN7+g9wr1POJSIjJURfexGZLiKznOtbhbnPDUHnaukBBPAv4HYRGeXc68nO78HSBlbxdxNU \n dTvwBHCDquZiBlhvArZhenDXsOt+n4cZcFuJ+XP91qljIfBLzKt3MWZA9XxaVlRvYHpceaq6JEiW \n 1zBvH887ZoQlmEG8nUXaaMtq4OeYt5hC4McYW64/6PhnMQpnnSPnHfvSBlVdDvwV+BLIxyj9z4KK \n fAgsA/JFZFvQ+YPra609v8GYXfIx9+c5jN2/OVk2Aydh7lURZqxhcgvnDD7ve86yGjP2U02QqayF \n Y6/FvKmVYd54nm8so6rlwHHAKZhB3tUYcyLAS85nkYgscL6fjxkEXo655i9hzIqtyd24LcU5f7Ej \n +3aMowDAI8AEx4T0KntyL6aT8B/MG87DmMFjSxvYCVyWLoeIbMBMqvoo0rLsLSJyF5ClqhdFWhZL \n z8X2+C2WMCIiYx0ThDjmkosxHlgWS8TokTMXLZYOJBlj3hmAMavdo6pvRFYkS0/HmnosFoulh2FN \n PRaLxdLD6BKmHhGxryUWi8WyD6jqHq7AXULxQ/PCh4KIzFPVee0sTqfGtrlnYNvcM9ifNrfUabam \n HovFYulhWMVvsVgsPYyeoPh9kRYgAvgiLUAE8EVagAjgi7QAEcAXaQEigK+9KwyrO6eIPIqZbr+t \n SdCm4DJ/w4SYrcKEy/2+mTK6rzZ+i8Vi6am0pDvD3eN/DDihpZ0ichImOuBowIOJGGixWCyWMBJW \n xa+qnwIlrRQ5FRO4ClX9GkgTkb7hlMlisVh6OpG28Q9kzyQVNmemxWKxhJHO4Mff1P5kJ2tZLJYe \n gYjEYUKbj3PFJE9N6dX71CRX7NDzJ17tyohP7JUs8a5vK3/o+/Cnd2xrq669IdKKfwu7ZyQaxO5Z \n lnYiIvOCVn2q6gufWBaLxdL+yG/XxA4sLjtubPWG8/qVbZstyGB1+rqB+nJ27CinMiqGrLEEYidW \n ypbFhavq18Y1m7+h2fpF3OzKndByuXAHaRORYcCbzXn1OIO7v1HVk0TkYOA+VT24mXLWq8disXR6 \n RGQ4rujxZI49OsEV746uKx1TX1mYdMUsb2BAaWpU//xY0kugJLOaitQdevW6X0mUy+VPjIkt9NdX \n +EYeNGXN3LPP/llsbOxKl8vl8Xg8W/dTnmZ1Z1h7/CLyHHAk0FtENgO3YhIzo6rzVfUdETlJRNYC \n lYBNTmGxWDo1IpLIlCm1/Oj5Yb0qOGZExY7j+9ftmDaksr5fUmxGfEVdMRQuozromIQt2XUJ8YEN \n DUkr302J+u7JH+VVLCNPGy6Rn0U3Zpbzer2nYLKOXQs85fF4wtYr7xJhmW2P32KxRAIZetAJxPc5 \n I7a+fmZsVdHwhvK8pOqqQtevJ/2TWSWj6JcP5SnVBBK3ozF5gb9ue9hfUl9UnUjNygZ/bfaGAB9X \n mJSUuaraNFH9bni93hQ
" text/plain " : [
" <matplotlib.figure.Figure at 0x10d93b510> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
} ,
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Gradient Boosting Classifier: \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAX4AAAEZCAYAAACQK04eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAIABJREFUeJzsnXmYHFX1v99PNkJWEvYlLLIjEED2dRJQWQIIKAgKgkIj \n CPqVXX5iEhUURERBIAXIvsgqILuQYRMi+yaRnYQtkD1kAZI5vz/O7aSmp7unZ6Z7eqb7vs9Tz3RX \n 3ao61d1z6ta5556PzIxIJBKJ1A89qm1AJBKJRDqX6PgjkUikzoiOPxKJROqM6PgjkUikzoiOPxKJ \n ROqM6PgjkUikzoiOP1Iykl6RtHO17ag2ki6W9MtOPueVkn7TmeesFJK+J+n+du4bf4NlQDGPv3si \n 6V1gBWARMBd4EPiJmc2upl21hqTDgR+Z2U5VtuMKYLKZ/arKdowB1jazQzvhXFfi13xGpc9Vb8Qe \n f/fFgFFmNhAYDmwCdGovtBxI6lWP564mknrW47kjS4iOvwYwsynAA8BXs+skbSvp35JmSHpB0i6p \n bUMlXSHpA0nTJd2e2jYqtJ8h6QlJm6S2vStppKRVJM2TNCS1bXNJn2b/sSX9UNJ/w/Hvk7R6qm2T \n pGMlvQH8L981SdpH0qvBjvGSNsix47Swfbqkv0laqg3XcIqkl4A5knqGY70paXY45rdC2w2Bi4Ht \n JM2RND2sXxx2kdQg6X1JJ0iaIunD8JSQPd+yku6SNEvSfyT9VtJjhb5LSTumvrdJkg5LbR4q6Z/B \n zqckfSW1359D+1mSnpG0Y2rbGEm3SLpG0izgB5K2kvRkOM+Hki6Q1Du1z1clPShpmqSPJf1C0jeB \n XwAHhc/j+dB2sKTLw3Hel/QbST3CtsPDd3CepKnAmLDusbBdkv4UPrtZkl4K584AhwCnhHPdkfr+ \n dg2ve0o6PfXdPSNptUKfbSSFmcWlGy7AO8Cu4fVqwEvAr8L7VYGpwO7h/W7h/bLh/d3ADcBgoBew \n U1i/OTAF2AoQcFg4T+/UOUeG1w8BR6bs+QNwUXi9L/AGsD7eufh/wBOptk3A/cAywFJ5rm094DNg \n V6AncHI4Xq+w/d1wvasCQ4DHgd+UeA3vAs+FfZcK674NrBReHxjOvWJ4/wPgsRz7rgB+HV43AF8C \n Y4Kte+Cht8Fh+43A9UBfYENgEvBoge90DWA2cFA41lBgeNh2ZfgOtwzbrgVuSO37vfBZ9ABOAD4C \n +oRtY4AvgH3C+77AFsDWof0awH+Bn4XtA8P+Pwf6AAOArcO20cDVOXbfjt8glwaWByYAmbDt8PD5 \n /CScq29Y91jY/k3gGWBQeL9+6rtY/Dnn/O6zv8GTw+9g3fB+E2Botf83u8NSdQPi0s4vzh3YnOAo \n msI/X4+w7dQ8/5z34U5wZXxcYHCeY16c5x9tIktuDOl/uh8BD4XXCg5tx/D+XuCHqWP0wJ3hsPC+ \n CWgocm1nADem3gt4H9g5ZUcmtX0P4M02XMPhrXy2z6ec5GInldp+BUtuNA3AvOxnH9ZNwZ1qT9zh \n rpva9pvc46W2/QK4tcC2K4Ak55pfK3IN04FNwusxQGMr1/x/wG3h9cHAswXajQGuSb1fEVgA9E2t \n Oxh4OPX5vZdzjMWfKTASf+rbJv0Z5n7OqXXp3+D/gL2r9T/YnZcY6um+GLCvmQ3Cnc9IvDcI3oP7 \n TniMnyFpBrADsBIwDJhuZrPyHHMN4MSc/VYDVsnT9jY8BLISsDPQZGaPp47z59QxpoX1q6b2n1zk \n 2lbGbyR+of5fPrnI/pNSNpZyDc3OLekwSc+n2m8MLFvEvlymmVlT6v08vJe8PP5ElT7f+0WOsxrw \n dpHtU1Kv54dzACDpJHlobWa4hsHAcoXOK2m9EDb6KIR/zmTJNQ9rxY40awC9gY9Sn98l+LVnKfhd \n m9nDwIXAX4EpksZJGljiuVcD3iqxbSRFdPw1gJk9ClwAnB1WTcJ7ZUNSy0AzOwf/JxwqaXCeQ00C \n zszZb4CZ/T3POWfg4woH4bHYG3KOk8k5Tn8zeyp9iCKX9CHuUACPA+PO6INUm9VzXme3lXINi88t \n aQ0gwUMRQ81sCPAK/pRRzM5S0uE+BRYG27MMK9AW/LtZu4TjNkPSTnjY4ztmtky4hlksuQZoae/F \n eHhnHTMbjIfjsv5gEvAV8tOU834y8DkeRsx+3oPNbJNUm6KflZldYGZbAhvhYb6TS9kvnHudVtpE \n 8hAdf+1wPrC1pG3w+O/ekr4RBsD6hkHIVc3sIzwUc5GkZST11pK86EuBH0vaOgy69Ze0l6QBBc55 \n PR4DPyC8znIJcLqkjWDx4N932nAtNwF7yQeSewMn4uGEf4ftAo6VtKqkobjTyjr2tl5Df9zBTAV6 \n SDoC7/FnmQKslh74DOdPO9W8mNki/MlojKSl5QPUh1LYoV0H7CbpO5J6yQeGh6fOWYiB+A1mqqQ+ \n kn4FDGrFvAF4qHBesOuY1La7gZUl/UzSUpIGSto6bJsCrBluxoTf0wPAeaFdD0lrq8Rce0lbStom \n fL7z8O95UepchW5AAJcBv5G0TviuNw2/h0grRMdfI5jZVOAq4FQzex8fYD0d+ATvwZ3Iku/7UHzA \n bSL+z/XTcIxngaPwR+/p+IDqYRR2VHfiPa6PzOzllC3/wJ8+bgxhhJfxQbzFTVq5lteB7+NPMZ8C \n e+Gx3IWp/a/HHc5bwc7ftucazOy/wB+BJ4GPcaf/eKrJQ8CrwMeSPkmdP328YtdzHB52+Rj/fm7A \n 4/75bJkM7Il/V9PwsYZNC5wzfd77wvI6PvYzn1SorMC+J+FParPxJ54bs23MbA7wdWBvfJD3dTyc \n CHBz+DtN0jPh9WH4IPB/8c/8ZjysWMzu7LpB4fzTg+1T8UQBgMuBjUII6TZach7eSXgAf8K5FB88 \n jrRCnMAV6XZIegefVPVwtW1pK5LOBlYwsyOqbUukfok9/kikgkhaP4QgFMIlP8QzsCKRqlGXMxcj \n kU5kIB7eWQUPq51rZndW16RIvRNDPZFIJFJnxFBPJBKJ1BndItQjKT6WRCKRSDswsxapwN3C8UN+ \n 40tB0hgzG1Nmc7o08Zrrg3jN9UFHrrlQpzmGeiKRSKTOiI4/EolE6ox6cPyN1TagCjRW24Aq0Fht \n A6pAY7UNqAKN1TagCjSW+4AVTeeU9Dd8uv0nOUWb0m3+gpeYnYeXy30+Txtrb4w/EolE6pVCvrPS \n Pf4rgN0LbZS0J14dcF0gg1cMjEQikUgFqajjN7PHgBlFmuyDF67CzCYAy0hasZI2RSKRSL1T7Rj/ \n qrQUqYiamZFIJFJBukIef278KU7WikQiJaPGxp8B+1fbjmL0+eKL3l99770NZaZZL73U68tZs3os \n WrCABR9/3HPBRx/1/GLGjB5fOeqouf2/8pWsFgHLQc8BoK8++eTKv77iijnltKfajv8DmisSrUZz \n laXFSBqTettoZo2VMysSiXQjdgTGh6XzOGH4XTRpIO8dCXP+1XL76gkM+kYw8CH24t9cxQ+Y8tal \n fPbZMy2a/+TsVQdswnpYT2PantOYtfMsVvj7Cgx85gezuaJ14R8ASQ0s0U4o3K7SRdokrQnclS+r \n JwzuHmdme0raFjjfzLbN0y5m9UQi3QA1No4Ezi37cZuatPL06SvIrEV4etqgQUN+ec01//h/1103 \n Md++U6DPyzDQcqILa8K8dT2bsBljYKN/wbpfhI6xgT6BZX4Cj57iojSh3ejRYxg7djh89yVYP/c4 \n Z8GNv3BBeIBdgdsxO0/SicBXcbWx14HXruXa+77P95ceN27chsCVwKRTppwyauYZMzvk9wr5zkqn \n c94A7IKLPk8BRuPCzJjZuNDmQjzzZy5whJk9V6rxkUika6HGxr/hql23lvO4P7r77jUzd911+aQV \n V7wpd9u0+fP7fPDcc/23WrRo6t6uONeMr8Me/4Ktc9ePYAQ701Ih8n7u5ymearH+MA7jCFrq58xn \n PgtZ2GJ9X/rSm94t1ud
" text/plain " : [
" <matplotlib.figure.Figure at 0x10dcfa1d0> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
}
] ,
" source " : [
" from sklearn.metrics import roc_curve, auc \n " ,
" from scipy import interp \n " ,
" \n " ,
" def plot_roc(X, y, clf_class, **kwargs): \n " ,
" kf = KFold(len(y), n_folds=5, shuffle=True) \n " ,
" y_prob = np.zeros((len(y),2)) \n " ,
" mean_tpr = 0.0 \n " ,
" mean_fpr = np.linspace(0, 1, 100) \n " ,
" all_tpr = [] \n " ,
" for i, (train_index, test_index) in enumerate(kf): \n " ,
" X_train, X_test = X[train_index], X[test_index] \n " ,
" y_train = y[train_index] \n " ,
" clf = clf_class(**kwargs) \n " ,
" clf.fit(X_train,y_train) \n " ,
" # Predict probabilities, not classes \n " ,
" y_prob[test_index] = clf.predict_proba(X_test) \n " ,
" fpr, tpr, thresholds = roc_curve(y[test_index], y_prob[test_index, 1]) \n " ,
" mean_tpr += interp(mean_fpr, fpr, tpr) \n " ,
" mean_tpr[0] = 0.0 \n " ,
" roc_auc = auc(fpr, tpr) \n " ,
" plt.plot(fpr, tpr, lw=1, label= ' ROC fold %d (area = %0.2f ) ' % (i, roc_auc)) \n " ,
" mean_tpr /= len(kf) \n " ,
" mean_tpr[-1] = 1.0 \n " ,
" mean_auc = auc(mean_fpr, mean_tpr) \n " ,
" plt.plot(mean_fpr, mean_tpr, ' k-- ' ,label= ' Mean ROC (area = %0.2f ) ' % mean_auc, lw=2) \n " ,
" \n " ,
" plt.plot([0, 1], [0, 1], ' -- ' , color=(0.6, 0.6, 0.6), label= ' Random ' ) \n " ,
" plt.xlim([-0.05, 1.05]) \n " ,
" plt.ylim([-0.05, 1.05]) \n " ,
" plt.xlabel( ' False Positive Rate ' ) \n " ,
" plt.ylabel( ' True Positive Rate ' ) \n " ,
" plt.title( ' Receiver operating characteristic ' ) \n " ,
" plt.legend(loc= \" lower right \" ) \n " ,
" plt.show() \n " ,
" \n " ,
" \n " ,
" print \" Support vector machines: \" \n " ,
" plot_roc(X,y,SVC,probability=True) \n " ,
" \n " ,
" print \" Random forests: \" \n " ,
" plot_roc(X,y,RF,n_estimators=18) \n " ,
" \n " ,
" print \" K-nearest-neighbors: \" \n " ,
" plot_roc(X,y,KNN) \n " ,
" \n " ,
" print \" Gradient Boosting Classifier: \" \n " ,
" plot_roc(X,y,GBC) "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" ## Feature Importance \n " ,
" \n " ,
" Now that we understand the accuracy of each individual model for our particular dataset, let ' s dive a little deeper to get a better understanding of what features or behaviours are causing our customers to churn. In the next section, we will be using a `RandomForestClassifer` to build an ensemble of decision trees to predict whether a customer will churn or not churn. One of the first steps in building a decision tree to calculating the _information gain_ associated with splitting on a particular feature. (More on this later.) \n " ,
" \n " ,
" Let ' s look at the Top 10 features in our dataset that contribute to customer churn: "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 12 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Feature ranking: \n " ,
" 1. Account Length (0.188157) \n " ,
" 2. Int ' l Plan (0.102871) \n " ,
" 3. VMail Plan (0.096911) \n " ,
" 4. VMail Message (0.071246) \n " ,
" 5. Day Mins (0.062060) \n " ,
" 6. Day Calls (0.032456) \n " ,
" 7. Day Charge (0.030478) \n " ,
" 8. Eve Mins (0.026043) \n " ,
" 9. Eve Calls (0.025411) \n " ,
" 10. Eve Charge (0.021707) \n "
]
} ,
{
" data " : {
" image/png " : " iVBORw0KGgoAAAANSUhEUgAAAXYAAAEKCAYAAAAGvn7fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz \n AAALEgAACxIB0t1+/AAAGKFJREFUeJzt3X20XXV95/H3JzeE5wIqDxqCcVFadVWnOE5Mi8BVU1ag \n amy7OshoLeJYZjqpTH1YDF1dEuta49K1HLF16qQQuyxiM5WCjWuwBIrXsnwIyRQES8IQJEOejOEp \n IJQxkM/8sXfI5nLvebgP59zzO5/XWmfds8/+7fPd++Tmc3/3t397X9kmIiLKMa/fOxARETMrwR4R \n UZgEe0REYRLsERGFSbBHRBQmwR4RUZgEewwVSVdIurrf+xExm5R57NEpSduAk4Dn6pcM/ILtH0/z \n PS+xfdu0d3DASFoFnG77d/q9L1GW+f3egRgoBt4+wyFsQFPdWNKI7efat5xbJOX/XsyaDMXEtEk6 \n TtIaSbsk7ZD0SUnz6nWnS7pN0sOS9kr6iqTj6nXXAqcB35D0pKSPShqVtH3c+2+T9Nb6+SpJ10u6 \n VtI+4Hdb1Z9gX1fVdZG0WNIBSRdLekjSI5L+g6R/I+luSY9J+rPGthdL+o6kP5P0uKTNB/erXv8K \n Sevq97lf0r8fV7e535cCVwAX1sd+Z93u/ZLulfSEpAck/V7jPUbr4/uwpD318V7cWH+kpM/Wn9fj \n km6XdES9bqmk79bHdJekc8cd1wN1zR9J+nddfgvEXGM7jzw6egAPAm+b4PUbgS8CRwInAhuA36vX \n nQ68DTgMeBnwbeBz497zrY3lUWD7BHXfWj9fBfwMeGe9fESr+hPs65XAtfXzxcAB4M+BBcCvAf+v \n fr+XAa8A9gDn1O0vBvYDlwEjwL8FHgeOr9f/I/CF+r3+FfAT4C0t9vtK4K/G7d8FwKvq5+cATwFn \n Nj6b/fV7jQDn1+uPq9f/d+A24OVUnbal9b4sBB4GltftltXLLwWOBvYBZ9TrTgZe2+/vtTym90iP \n Pboh4Ot1r+8xSTdIOpkqYP7Q9r/Y3gtcBbwbwPYDtv/B9n7bDwOfA86dtEJnvmt7Xf38uFb1JzmG \n 8T5p+2e2bwGeBL5q+2Hbu4DbgTMbbX9i+/O2n7P9N8B9wNslLQJ+Fbi8fq8fANcA75tov20/U+/L \n C/bH9k22H6yf/yOwHji70WQ/8Cd1/W8CPwV+sf4N5f3AZbZ32z5g+/u2fwa8F7jJ9t/X73srsAn4 \n daqhsAPA6yQdaXuP7Xsn+exiQGScL7phYIUbY+ySllD1xndLz2fUPOChev3JwOeBNwPH1useneZ+ \n 7Gg8f2Wr+h3a03j+LxMsH91Y3jlu2/9L1UN+OfCo7aca6x4C3jjJfk9I0vlUPfkzqI7jKODuRpNH \n bB9oLD8NHEP1G8YRwAMTvO0rgd+W9I7Ga/OB22w/LelC4KPAGknfAT5i+752+xpzV3rsMV3bqYYv \n Xmr7hPpxnO3X1ev/K9Usml+yfRzwO7zw+278tKynqMIMqE6OUg2vNDW3aVd/vOlOA1s4bvmVwK76 \n 8RJJxzTWncYLw3x87WZAI+lw4G+BzwAn2T4BuInOTi4/DDwD/PwE6x6iGn46ofE41vZnAGyvt30e \n cAqwBch00AGXYI9psb2barjgv0k6VtK8+oTpOXWTY6jC+glJC4GPjXuLPVTj8Af9H+AISRdIOgz4 \n Y+DwadQfbyozcJrbnCTpQ5IOk/TbwKuphjl2AN8FPiXpcEmvBy4BvtLiffcAi3XoV40F9eNh4EDd \n ez+vkx2se/FfovocXi5pRNKvSFpQ78M7JJ1Xv35EfSJ2oaSTJK2QdDTVMM9THJrOGgMqwR4z4X1U \n gXQv1TDL16h6fwCfAN5AdYLuG1Q90mbP9VPAH9dj9h+2vQ/4farx6R1UY8jNWTLmxT3fVvXHG799 \n Jz34ZpsNVMMke4FPAr9l+7F63UVUJ2R3ATcAH28MW02031+rvz4iaZPtJ4EPAX9TH8dFwN+12Jfx \n PgrcA2wEHqH6bOfVP3RWAH9EdUL3IeAjVD+w5gF/SDXE9AjVeP5/bFEjBkDbC5QkLac6GTUCXGP7 \n 0+PWrwD+hOrXygPAxw5+M7fbNmKQ1FMLP2D77HZtI/qp5cnTenzzC1TTo3YCGyWts7250exW239X \n t38d1VSxn+9w24iImGHthmKWAFttb7O9H1hL9Svd88bNAjiGanywo20jBsxEwykRc067YF/IC8c3 \n d/DiWQFIepekzcA3qcYIO942YlDY/rLtyU7KRswZ7YK9o96J7a/bfg3wDuDaxln+iIjosXYXKO0E \n FjWWF9HiIgvbt6u6udFL6nZtt5WUX20jIqbA9sSd6Fb3G6AK/geopnAtAO4CXjOuzekcml3zBuCB \n Tret27nX91Fo1F41THWH8ZjzWQ9H7SE9Zk+2rmWP3fazklYCN1NNWVxje7OkS+v1q4HfAt4naT/V \n nON3t9q2Vb2IiJi+tveKcXWjoW+Oe2114/lnqC6B7mjbiIiYXcN+5enYkNXtZ+1hq9vP2v2q28/a \n /arb79oT6vufxpNkT3YCICIiJtQqO4e9xx4RUZwEe0REYRLsERGFSbBHRBQmwR4RUZgEe0REYRLs \n ERGFSbBHRBQmwR4RUZgEe0REYRLsERGFSbBHRBQmwR4RUZgEe0REYRLsERGFSbBHRBQmwR4RUZgE \n e0REYRLsERGFSbBHRBQmwR4RUZgEe0REYRLsERGFSbBHRBQmwR4RUZgEe0REYdoGu6TlkrZIul/S \n 5ROsf4+kH0i6W9J3JL2+sW5b/fqdku6Y6Z2PiIgXm99qpaQR4AvAMmAnsFHSOtubG81+BJxje5+k \n 5cBfAEvrdQZGbT8687seERETaddjXwJstb3N9n5gLbCi2cD292zvqxc3AKeOew/NyJ5GRERHWvbY \n gYXA9sbyDuBNLdp/ALipsWzgVknPAattXz2lvZxBEqPAaL04CozVz8fs559HRAysdsHuTt9I0luA \n S4CzGi+fZXu3pBOBWyRtsX37FPZzxtThPQYgYfv5kI+IKEK7YN8JLGosL6Lqtb9AfcL0amC57ccO \n vm57d/11r6QbqYZ2XhTsklY1Fsdsj3W4/xERQ0HSKHTWEZU9eadc0nzgPuBtwC7gDuCi5slTSacB \n twHvtf39xutHASO2n5R0NLAe+ITt9eNq2HZfxuHrHnvOAUTEwGmVnS177LaflbQSuBkYAdbY3izp \n 0nr9auDjwAnAFyUB7Le9BDgFuKF+bT5w3fhQj4iImdeyx96THUiPPSKia62yM1eeRkQUJsEeEVGY \n BHtERGES7BERhUmwR0QUJsEeEVGYBHtERGES7BERhUmwR0QUJsEeEVGYBHtERGES7BERhUmwR0QU \n JsEeEVGYBHtERGES7BERhUmwR0QUJsEeEVGYBHtERGES7BERhUmwR0QUJsEeEVGYBHtERGES7BER \n hUmwR0QUJsEeEVGYBHtERGES7BERhUmwR0QUpm2wS1ouaYuk+yVdPsH690j6gaS7JX1H0us73TYi \n ImaebE++UhoB7gOWATuBjcBFtjc32vwKcK/tfZKWA6tsL+1k23p729YMH1dHJGzTl9oREdPRKjvb \n 9diXAFttb7O9H1gLrGg2sP092/vqxQ3AqZ1uGxERM69dsC8EtjeWd9SvTeYDwE1T3DYiImbA/Dbr \n Jx+nGUfSW4BLgLOmsO2qxuKY7bFOt42IGAaSRoHRTtq2C/adwKLG8iKqnvf4gq8HrgaW236sm20B \n bK/qZGcjIoZV3eEdO7gs6crJ2rYbitkEnCFpsaQFwIXAumYDSacBNwDvtb21m20jImLmteyx235W \n 0krgZmAEWGN7s6RL6/WrgY8DJwBflASw3/aSybadxWOJiAjaTHfsyQ5kumNERNemM90xIiIGTII9 \n IqIwCfaIiMIk2CMiCtNuHvvAkDSFs8Duert+neiNiOhUMcEOXVzqWlOX2yTRI2IQZCgmIqIwCfaI \n iMIk2CMiCpNgj4goTII
" text/plain " : [
" <matplotlib.figure.Figure at 0x10d5ceed0> "
]
} ,
" metadata " : { } ,
" output_type " : " display_data "
}
] ,
" source " : [
" train_index,test_index = train_test_split(churn_df.index) \n " ,
" \n " ,
" forest = RF() \n " ,
" forest_fit = forest.fit(X[train_index], y[train_index]) \n " ,
" forest_predictions = forest_fit.predict(X[test_index]) \n " ,
" \n " ,
" importances = forest_fit.feature_importances_[:10] \n " ,
" std = np.std([tree.feature_importances_ for tree in forest.estimators_], \n " ,
" axis=0) \n " ,
" indices = np.argsort(importances)[::-1] \n " ,
" \n " ,
" # Print the feature ranking \n " ,
" print( \" Feature ranking: \" ) \n " ,
" \n " ,
" for f in range(10): \n " ,
" print( \" %d . %s ( %f ) \" % (f + 1, features[f], importances[indices[f]])) \n " ,
" \n " ,
" # Plot the feature importances of the forest \n " ,
" #import pylab as pl \n " ,
" plt.figure() \n " ,
" plt.title( \" Feature importances \" ) \n " ,
" plt.bar(range(10), importances[indices], yerr=std[indices], color= \" r \" , align= \" center \" ) \n " ,
" plt.xticks(range(10), indices) \n " ,
" plt.xlim([-1, 10]) \n " ,
" plt.show() "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" ## Thinking in Probabilities \n " ,
" \n " ,
" Decision making often favors probability over simple classifications. There ' s plainly more information in statements like \" there ' s a 20 % c hance of rain tomorrow \" and \" about 55 % o f test takers pass the California bar exam \" than just saying \" it shouldn ' t rain tomorrow \" or \" you ' ll probably pass. \" Probability predictions for churn also allow us to gauge a customers expected value, and their expected loss. Who do you want to reach out to first, the client with a 80 % c hurn risk who pays 20,000 annually, or the client who ' s worth 100,000 a year with a 40 % r isk? How much should you spend on each client? \n " ,
" \n " ,
" While I ' m moving a bit away from my expertise, being able to ask that question requires producing predictions a little differently. However, `scikit-learn` makes moving to probabilities easy; my three models have `predict_proba()` built right into their class objects. This is the same cross validation code with only a few lines changed. "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 13 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" def run_prob_cv(X, y, clf_class, roc=False, **kwargs): \n " ,
" kf = KFold(len(y), n_folds=5, shuffle=True) \n " ,
" y_prob = np.zeros((len(y),2)) \n " ,
" for train_index, test_index in kf: \n " ,
" X_train, X_test = X[train_index], X[test_index] \n " ,
" y_train = y[train_index] \n " ,
" clf = clf_class(**kwargs) \n " ,
" clf.fit(X_train,y_train) \n " ,
" # Predict probabilities, not classes \n " ,
" y_prob[test_index] = clf.predict_proba(X_test) \n " ,
" return y_prob "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" ## How good is good? \n " ,
" \n " ,
" Determining how good a predictor which gives probabilities rather than classes is a bit more difficult. If I predict there ' s a 20 % li kelihood of rain tomorrow I don ' t get to live out all the possible outcomes of the universe. It either rains or it doesn ' t. \n " ,
" \n " ,
" What helps is that the predictors aren ' t making one prediction, they ' re making 3000+. So for every time I predict an event to occur 20 % o f the time I can see how often those events actually happen. Here ' s we ' ll use `pandas` to help me compare the predictions made by random forest against the actual outcomes. "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 14 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" data " : {
" text/plain " : [
" 0.0 1796 \n " ,
" 0.1 681 \n " ,
" 0.2 258 \n " ,
" 0.3 112 \n " ,
" 0.8 94 \n " ,
" 0.9 89 \n " ,
" 0.7 79 \n " ,
" 0.4 74 \n " ,
" 0.5 53 \n " ,
" 0.6 50 \n " ,
" 1.0 47 \n " ,
" dtype: int64 "
]
} ,
" execution_count " : 14 ,
" metadata " : { } ,
" output_type " : " execute_result "
}
] ,
" source " : [
" import warnings \n " ,
" warnings.filterwarnings( ' ignore ' ) \n " ,
" \n " ,
" # Use 10 estimators so predictions are all multiples of 0.1 \n " ,
" pred_prob = run_prob_cv(X, y, RF, n_estimators=10) \n " ,
" pred_churn = pred_prob[:,1] \n " ,
" is_churn = y == 1 \n " ,
" \n " ,
" # Number of times a predicted probability is assigned to an observation \n " ,
" counts = pd.value_counts(pred_churn) \n " ,
" counts[:] "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 15 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" data " : {
" text/html " : [
" <div style= \" max-height:1000px;max-width:1500px;overflow:auto; \" > \n " ,
" <table border= \" 1 \" class= \" dataframe \" > \n " ,
" <thead> \n " ,
" <tr style= \" text-align: right; \" > \n " ,
" <th></th> \n " ,
" <th>pred_prob</th> \n " ,
" <th>count</th> \n " ,
" <th>true_prob</th> \n " ,
" </tr> \n " ,
" </thead> \n " ,
" <tbody> \n " ,
" <tr> \n " ,
" <th>0 </th> \n " ,
" <td> 0.0</td> \n " ,
" <td> 1796</td> \n " ,
" <td> 0.027840</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>1 </th> \n " ,
" <td> 0.1</td> \n " ,
" <td> 681</td> \n " ,
" <td> 0.020558</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>2 </th> \n " ,
" <td> 0.2</td> \n " ,
" <td> 258</td> \n " ,
" <td> 0.081395</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>3 </th> \n " ,
" <td> 0.3</td> \n " ,
" <td> 112</td> \n " ,
" <td> 0.089286</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>4 </th> \n " ,
" <td> 0.4</td> \n " ,
" <td> 74</td> \n " ,
" <td> 0.270270</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>5 </th> \n " ,
" <td> 0.5</td> \n " ,
" <td> 53</td> \n " ,
" <td> 0.622642</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>6 </th> \n " ,
" <td> 0.6</td> \n " ,
" <td> 50</td> \n " ,
" <td> 0.800000</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>7 </th> \n " ,
" <td> 0.7</td> \n " ,
" <td> 79</td> \n " ,
" <td> 0.898734</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>8 </th> \n " ,
" <td> 0.8</td> \n " ,
" <td> 94</td> \n " ,
" <td> 0.957447</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>9 </th> \n " ,
" <td> 0.9</td> \n " ,
" <td> 89</td> \n " ,
" <td> 0.977528</td> \n " ,
" </tr> \n " ,
" <tr> \n " ,
" <th>10</th> \n " ,
" <td> 1.0</td> \n " ,
" <td> 47</td> \n " ,
" <td> 1.000000</td> \n " ,
" </tr> \n " ,
" </tbody> \n " ,
" </table> \n " ,
" </div> "
] ,
" text/plain " : [
" pred_prob count true_prob \n " ,
" 0 0.0 1796 0.027840 \n " ,
" 1 0.1 681 0.020558 \n " ,
" 2 0.2 258 0.081395 \n " ,
" 3 0.3 112 0.089286 \n " ,
" 4 0.4 74 0.270270 \n " ,
" 5 0.5 53 0.622642 \n " ,
" 6 0.6 50 0.800000 \n " ,
" 7 0.7 79 0.898734 \n " ,
" 8 0.8 94 0.957447 \n " ,
" 9 0.9 89 0.977528 \n " ,
" 10 1.0 47 1.000000 "
]
} ,
" execution_count " : 15 ,
" metadata " : { } ,
" output_type " : " execute_result "
}
] ,
" source " : [
" from collections import defaultdict \n " ,
" true_prob = defaultdict(float) \n " ,
" \n " ,
" # calculate true probabilities \n " ,
" for prob in counts.index: \n " ,
" true_prob[prob] = np.mean(is_churn[pred_churn == prob]) \n " ,
" true_prob = pd.Series(true_prob) \n " ,
" \n " ,
" # pandas-fu \n " ,
" counts = pd.concat([counts,true_prob], axis=1).reset_index() \n " ,
" counts.columns = [ ' pred_prob ' , ' count ' , ' true_prob ' ] \n " ,
" counts "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" We can see that random forests predicted that 75 individuals would have a 0.9 proability of churn and in actuality that group had a ~0.97 rate. \n " ,
" \n " ,
" ## Calibration and Descrimination \n " ,
" \n " ,
" Using the `DataFrame` above we can draw a pretty simple graph to help visualize probability measurements. The x axis represents the churn probabilities which random forest assigned to a group of individuals. The y axis is the actual rate of churn within that group, and each point is scaled relative to the size of the group. \n " ,
" \n " ,
"  \n " ,
" \n " ,
" Calibration is a relatively simple measurement and can be summed up as so: Events predicted to happen 60 % o f the time should happen 60 % o f the time. For all individuals I predict to have a churn risk of between 30 and 40 % , the true churn rate for that group should be about 35 % . For the graph above think of it as, How close are my predictions to the red line? \n " ,
" \n " ,
" Discrimination measures _How far are my predictions away from the green line?_ Why is that important? \n " ,
" \n " ,
" Well, if we assign a churn probability of 15 % to every individual we ' ll have near perfect calibration due to averages, but I ' ll be lacking any real insight. Discrimination gives a model a better score if it ' s able to isolate groups which are further from the base set. \n " ,
" \n " ,
" Equations are replicated from [Yang, Yates, and Smith (1991)](https://www.google.com/search?q=Measures+of+Discrimination+Skill+in+Probabilistic+Judgment&oq=Measures+of+Discrimination+Skill+in+Probabilistic+Judgment) and the code Yhat wrote can be found on GitHub [here](https://github.com/EricChiang/churn/blob/master/churn_measurements.py). "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 17 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [ ] ,
" source " : [
" from churn_measurements import calibration, discrimination \n " ,
" from sklearn.metrics import roc_curve, auc \n " ,
" from scipy import interp \n " ,
" from __future__ import division \n " ,
" from operator import idiv \n " ,
" \n " ,
" \n " ,
" \n " ,
" def print_measurements(pred_prob): \n " ,
" churn_prob, is_churn = pred_prob[:,1], y == 1 \n " ,
" print \" %-20s %.4f \" % ( \" Calibration Error \" , calibration(churn_prob, is_churn)) \n " ,
" print \" %-20s %.4f \" % ( \" Discrimination \" , discrimination(churn_prob,is_churn)) \n " ,
" \n " ,
" print \" Note -- Lower calibration is better, higher discrimination is better \" "
]
} ,
{
" cell_type " : " code " ,
" execution_count " : 18 ,
" metadata " : {
" collapsed " : false
} ,
" outputs " : [
{
" name " : " stdout " ,
" output_type " : " stream " ,
" text " : [
" Support vector machines: \n " ,
" Calibration Error 0.0016 \n " ,
" Discrimination 0.0678 \n " ,
" Note -- Lower calibration is better, higher discrimination is better \n " ,
" Random forests: \n " ,
" Calibration Error 0.0072 \n " ,
" Discrimination 0.0845 \n " ,
" Note -- Lower calibration is better, higher discrimination is better \n " ,
" K-nearest-neighbors: \n " ,
" Calibration Error 0.0023 \n " ,
" Discrimination 0.0443 \n " ,
" Note -- Lower calibration is better, higher discrimination is better \n " ,
" Gradient Boosting Classifier: \n " ,
" Calibration Error 0.0017 \n " ,
" Discrimination 0.0859 \n " ,
" Note -- Lower calibration is better, higher discrimination is better \n " ,
" Random Forest: \n " ,
" Calibration Error 0.0062 \n " ,
" Discrimination 0.0782 \n " ,
" Note -- Lower calibration is better, higher discrimination is better \n "
]
}
] ,
" source " : [
" print \" Support vector machines: \" \n " ,
" print_measurements(run_prob_cv(X,y,SVC,probability=True)) \n " ,
" \n " ,
" print \" Random forests: \" \n " ,
" print_measurements(run_prob_cv(X,y,RF,n_estimators=18)) \n " ,
" \n " ,
" print \" K-nearest-neighbors: \" \n " ,
" print_measurements(run_prob_cv(X,y,KNN)) \n " ,
" \n " ,
" print \" Gradient Boosting Classifier: \" \n " ,
" print_measurements(run_prob_cv(X,y,GBC)) \n " ,
" \n " ,
" print \" Random Forest: \" \n " ,
" print_measurements(run_prob_cv(X,y,RF)) "
]
} ,
{
" cell_type " : " markdown " ,
" metadata " : { } ,
" source " : [
" Unlike the classification comparisons earlier, random forest isn ' t as clearly the front-runner here. While it ' s good at differentiating between high and low probability churn events, it has trouble assigning an accurate probability estimate to those events. For example the group which random forest predicts to have a 30 % c hurn rate actually had a true churn rate of 14 % . Clearly there ' s more work to be done, but I leave that to you as a challenge. "
]
}
] ,
" metadata " : {
" kernelspec " : {
" display_name " : " Python 2 " ,
" language " : " python " ,
" name " : " python2 "
} ,
" language_info " : {
" codemirror_mode " : {
" name " : " ipython " ,
" version " : 2
} ,
" file_extension " : " .py " ,
" mimetype " : " text/x-python " ,
" name " : " python " ,
" nbconvert_exporter " : " python " ,
" pygments_lexer " : " ipython2 " ,
" version " : " 2.7.10 "
}
} ,
" nbformat " : 4 ,
" nbformat_minor " : 0
}
