You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
396 lines
12 KiB
396 lines
12 KiB
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<!--\n",
|
|
"If you're creating new material, please share w the community! We'll review it and perform any maintenance. Simply:\n",
|
|
"1. Fork your new repo over to `data-staging`\n",
|
|
"2. Create an issue and tag `jeffboykin`\n",
|
|
"3. *Optional*: Comment on what you made, where you think it should go, or how it should be used!\n",
|
|
"\n",
|
|
"If you have questions or input on existing materials, get in touch: \n",
|
|
"1. Log an issue to this repo to alert us of a problem.\n",
|
|
"2. Suggest an edit yourself by forking this repo, making edits, and submitting a pull request with your changes back to our master branch.\n",
|
|
"3. Reach out to the data team on Slack and share your thoughts!\n",
|
|
"-->\n",
|
|
"\n",
|
|
"<img src=\"http://imgur.com/1ZcRyrc.png\" style=\"float: left; margin: 20px; height: 55px\">\n",
|
|
"\n",
|
|
"# Model Evaluation & Unbalanced Classes Lab\n",
|
|
"\n",
|
|
"_Authors: Matt Speck (DC), Matt Brems (DC)_\n",
|
|
"\n",
|
|
"## Overview\n",
|
|
"\n",
|
|
"In this lab, we will be exploring various metrics for evaluating classification models, writing a function to generate a ROC curve by hand, and exploring methods of dealing with unbalanced classes. The AUC ROC is one of the most important metrics for evaluating a classification model, and while Sklearn does have modules for calculating this, one of the best ways to understand the ROC curve is to write it yourself. \n",
|
|
"\n",
|
|
"Since the AUC ROC--in addition to other metrics--is a metric to evaluate a model, we'll have to create a model to evaluate first. We'll be using a logistic regression to predict whether or not a credit card client will default on their loan. The dataset and data dictionary are provided in the repository.\n",
|
|
"\n",
|
|
"## Required Objectives\n",
|
|
"\n",
|
|
"1. [Train a logistic regression on the `defaultcc.csv` dataset](#train-logit)\n",
|
|
"1. [Create a confusion matrix for your logistic regression](#confusion)\n",
|
|
"1. [Write a function that will plot an ROC curve](#roc-func)\n",
|
|
"1. [Write a function that takes in either 'sensitivity' or 'specificity' and a value of sensitivity/specificity, and returns the sensitivity, specificity, and threshold that generates the input value of sensitivity/specificity.](#sens-spec)\n",
|
|
"1. [Exacerbate unbalanced classes by artificially dropping 70% of values in underrepresented class.](#very-unbalanced)\n",
|
|
"1. [Explain which is worse in our case: false positives or false negatives](#fp-vs-fn)\n",
|
|
"1. [Build a new logistic regression model based on the unbalanced classes. Generate a confusion matrix based on this new model and evaluate.](#second-logit)\n",
|
|
"1. [Compare logistic regression models](#compare-logits)\n",
|
|
"1. [Try two methods of accounting for unbalanced classes: undersampling and changing the threshold](#two-methods)\n",
|
|
"\n",
|
|
"## Bonus Objectives\n",
|
|
"\n",
|
|
"2. [BONUS: Build out your function to approximate the area under the ROC curve.](#bonus-1)\n",
|
|
"2. [BONUS: Try accounting for unbalanced classes through oversampling until you get results that are 50% positive and 50% negative. Generate a confusion matrix and briefly summarize your results.](#bonus-2)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2017-12-18T21:47:48.528301Z",
|
|
"start_time": "2017-12-18T21:47:42.418211Z"
|
|
},
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"%matplotlib inline\n",
|
|
"%config InlineBackend.figure_format = 'retina'\n",
|
|
"\n",
|
|
"from sklearn.model_selection import train_test_split\n",
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|
"from sklearn.metrics import confusion_matrix, auc, roc_curve\n",
|
|
"from random import seed"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='read-data'></a>\n",
|
|
"\n",
|
|
"## Read in the defaultcc data from the repo. You may want to examine page 3 of the .pdf for a data dictionary."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2017-11-12T22:29:12.838392Z",
|
|
"start_time": "2017-11-12T22:29:12.663172Z"
|
|
},
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"df = pd.read_csv('defaultcc.csv', header=1) # The row at index 1 has the column names for this csv file"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='train-logit'></a>\n",
|
|
"\n",
|
|
"## Fit a logistic regression model predicting whether or not someone will default on their credit card. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='confusion'></a>\n",
|
|
"\n",
|
|
"## Generate a confusion matrix. Write a few sentences summarizing your results."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='roc-func'></a>\n",
|
|
"\n",
|
|
"## Write a function that will create an ROC curve for you. Here's a strategy you might consider:\n",
|
|
"\n",
|
|
"0. In order to even begin, you'll need some fit model. Build a logistic regression model with X and y as defined above.\n",
|
|
"\n",
|
|
"1. We want to look at all values of your \"threshold\" - that is, anything where .predict() gives you above your threshold falls in the \"positive class,\" and anything that is below your threshold falls in the \"negative class.\" Start the threshold at 0.\n",
|
|
"\n",
|
|
"2. At this value of your threshold, calculate the sensitivity and specificity. Store these values.\n",
|
|
"\n",
|
|
"3. Increment your threshold by some \"step.\" Maybe set your step to be 0.01, or even smaller.\n",
|
|
"\n",
|
|
"4. At this value of your threshold, calculate the sensitivity and specificity. Store these values.\n",
|
|
"\n",
|
|
"5. Repeat steps 3 and 4 until you get to the threshold of 1.\n",
|
|
"\n",
|
|
"6. Plot the values of sensitivity and 1 - specificity."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2017-11-12T22:38:36.611549Z",
|
|
"start_time": "2017-11-12T22:38:36.599353Z"
|
|
},
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='sens-spec'></a>\n",
|
|
"\n",
|
|
"## Either build out your function from above or create a new function that uses the above function to take in the string \"sensitivity\" or \"specificity\" and a value of sensitivity/specificity, and returns the sensitivity, specificity, and threshold that generates the input value of sensitivity/specificity.\n",
|
|
"\n",
|
|
"<a id='sens-spec-example'></a>\n",
|
|
"### For example, function(\"sensitivity\", 0.95) might return \"Sensitivity: 95%, Specificity: 90%, Threshold: 50%.\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2017-11-09T19:28:04.313994Z",
|
|
"start_time": "2017-11-09T19:28:04.237001Z"
|
|
},
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='very-unbalanced'></a>\n",
|
|
"\n",
|
|
"## Note that the defaultcc data has approximately 20% of observations in class 1 and 80% in class 2. Set a seed of 48 and artificially drop 70% of the values marked 1. This will ensure we have very unbalanced classes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='fp-vs-fn'></a>\n",
|
|
"\n",
|
|
"## Which is worse in our particular use-case - false positives or false negatives? Why? (If you feel there's not one clear answer, defend your conclusion.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='second-logit'></a>\n",
|
|
"\n",
|
|
"## Build the same logistic regression model based on the unbalanced classes. Generate a confusion matrix based on this new model. What do you notice?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='compare-logits'></a>\n",
|
|
"\n",
|
|
"## Using your function, plot the ROC of both models. How do they compare? Summarize your results."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='two-methods'></a>\n",
|
|
"\n",
|
|
"## Try two methods of accounting for unbalanced classes. For each, generate a confusion matrix and briefly summarize your results.\n",
|
|
"\n",
|
|
"1. Undersample the 0s until 50% of your observations have a \"positive outcome\" and 50% of your observations have a \"negative outcome.\"\n",
|
|
"2. Change your threshold for classifying observations as positive or negative to 10%.\n",
|
|
"3. Do 2. again, but for 90%."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Method 1: Undersampling"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Method 2: Changing Threshold"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='bonus-1'></a>\n",
|
|
"\n",
|
|
"## BONUS: Build out your function to approximate the area under the ROC curve. I recommend using [step size] * [height of the curve in the middle of that step size] to create rectangles and then summing the areas of those rectangles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"<a id='bonus-2'></a>\n",
|
|
"\n",
|
|
"## BONUS: Try accounting for unbalanced classes through oversampling until you get results that are 50% positive and 50% negative. Generate a confusion matrix and briefly summarize your results."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {
|
|
"collapsed": true
|
|
},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 2",
|
|
"language": "python",
|
|
"name": "python2"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 2
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython2",
|
|
"version": "2.7.13"
|
|
},
|
|
"varInspector": {
|
|
"cols": {
|
|
"lenName": 16,
|
|
"lenType": 16,
|
|
"lenVar": 40
|
|
},
|
|
"kernels_config": {
|
|
"python": {
|
|
"delete_cmd_postfix": "",
|
|
"delete_cmd_prefix": "del ",
|
|
"library": "var_list.py",
|
|
"varRefreshCmd": "print(var_dic_list())"
|
|
},
|
|
"r": {
|
|
"delete_cmd_postfix": ") ",
|
|
"delete_cmd_prefix": "rm(",
|
|
"library": "var_list.r",
|
|
"varRefreshCmd": "cat(var_dic_list()) "
|
|
}
|
|
},
|
|
"types_to_exclude": [
|
|
"module",
|
|
"function",
|
|
"builtin_function_or_method",
|
|
"instance",
|
|
"_Feature"
|
|
],
|
|
"window_display": false
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|