You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
1110 lines
31 KiB
1110 lines
31 KiB
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"#  Pandas for Exploratory Data Analysis I \n",
|
|
"by [@josephofiowa](https://twitter.com/josephofiowa)\n",
|
|
"\n",
|
|
"Pandas is the most prominent Python library for exploratory data analysis (EDA). The functions Pandas supports are integral to understanding, formatting, and preparing our data. Formally, we use Pandas to investigate, wrangle, munge, and clean our data. Pandas is the Swiss Army Knife of data manipulation!\n",
|
|
"\n",
|
|
"\n",
|
|
"We'll have two coding-heavy sessions on Pandas. In this one, we'll use Pandas to:\n",
|
|
" - Read in a dataset\n",
|
|
" - Investigate a dataset's integrity\n",
|
|
" - Filter, sort, and manipulate a DataFrame's series"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## About the Dataset: Adventureworks Cycles\n",
|
|
"\n",
|
|
"<img align=\"right\" src=\"http://lh6.ggpht.com/_XjcDyZkJqHg/TPaaRcaysbI/AAAAAAAAAFo/b1U3q-qbTjY/AdventureWorks%20Logo%5B5%5D.png?imgmax=800\">\n",
|
|
"\n",
|
|
"For today's Pandas exercises, we will be using a dataset developed by Microsoft for training purposes in SQL server, known the [Adventureworks Cycles 2014OLTP Database](https://github.com/Microsoft/sql-server-samples/releases/tag/adventureworks). It is based on a fictitious company called Adventure Works Cycles (AWC), a multinational manufacturer and seller of bicycles and accessories. The company is based in Bothell, Washington, USA and has regional sales offices in several countries. We will be looking at a single table from this database, the Production.Product table, which outlines some of the products this company sells. \n",
|
|
"\n",
|
|
"A full data dictionary can be viewed [here](https://www.sqldatadictionary.com/AdventureWorks2014/).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's take a closer look at the Production.Product table [data dictionary](https://www.sqldatadictionary.com/AdventureWorks2014/Production.Product.html), which is a description of the fields (columns) in the table (the .csv file we will import below):\n",
|
|
"- **ProductID** - Primary key for Product records.\n",
|
|
"- **Name** - Name of the product.\n",
|
|
"- **ProductNumber** - Unique product identification number.\n",
|
|
"- **MakeFlag** - 0 = Product is purchased, 1 = Product is manufactured in-house.\n",
|
|
"- **FinishedGoodsFlag** - 0 = Product is not a salable item. 1 = Product is salable.\n",
|
|
"- **Color** - Product color.\n",
|
|
"- **SafetyStockLevel** - Minimum inventory quantity.\n",
|
|
"- **ReorderPoint** - Inventory level that triggers a purchase order or work order.\n",
|
|
"- **StandardCost** - Standard cost of the product.\n",
|
|
"- **ListPrice** - Selling price.\n",
|
|
"- **Size** - Product size.\n",
|
|
"- **SizeUnitMeasureCode** - Unit of measure for the Size column.\n",
|
|
"- **WeightUnitMeasureCode** - Unit of measure for the Weight column.\n",
|
|
"- **DaysToManufacture** - Number of days required to manufacture the product.\n",
|
|
"- **ProductLine** - R = Road, M = Mountain, T = Touring, S = Standard\n",
|
|
"- **Class** - H = High, M = Medium, L = Low\n",
|
|
"- **Style** - W = Womens, M = Mens, U = Universal\n",
|
|
"- **ProductSubcategoryID** - Product is a member of this product subcategory. Foreign key to ProductSubCategory.ProductSubCategoryID.\n",
|
|
"- **ProductModelID** - Product is a member of this product model. Foreign key to ProductModel.ProductModelID.\n",
|
|
"- **SellStartDate** - Date the product was available for sale.\n",
|
|
"- **SellEndDate** - Date the product was no longer available for sale.\n",
|
|
"- **DiscontinuedDate** - Date the product was discontinued.\n",
|
|
"- **rowguid** - ROWGUIDCOL number uniquely identifying the record. Used to support a merge replication sample.\n",
|
|
"- **ModifiedDate** - Date and time the record was last updated.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Importing Pandas\n",
|
|
"\n",
|
|
"To [import a library](https://docs.python.org/3/reference/import.html), we write `import` and the library name. For Pandas, is it common to name the library `pd` so that when we reference a function from the Pandas library, we only write `pd` to reference the aliased [namespace](https://docs.python.org/3/tutorial/classes.html#python-scopes-and-namespaces) -- not `pandas`."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"%matplotlib inline"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# we can see the details about the imported package by referencing its private class propertys:\n",
|
|
"print(f'I am using {pd.__name__} \\\n",
|
|
"Version: {pd.__version__}.\\n\\\n",
|
|
"It is installed at: {pd.__path__}')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Reading in Data\n",
|
|
"\n",
|
|
"Pandas dramatically simplifies the process of reading in data. When we say \"reading in data,\" we mean loading a file into our machine's memory.\n",
|
|
"\n",
|
|
"When you have a CSV, for example, and then you double-click to open it in Microsoft Excel, the open file is \"read into memory.\" You can now manipulate the CSV.\n",
|
|
"\n",
|
|
"When we read data into memory in Python, we are creating an object. We will soon explore this object. _(And, as an aside, when we have a file that is greater than the size of our computer's memory, this is approaching \"Big Data.\")_"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Because we are working with a CSV, we will use the [read CSV](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) method.<br>A [delimiter](https://en.wikipedia.org/wiki/Delimiter-separated_values) is a character that separates fields (columns) in the imported file. Just because a file says `.csv` does not necessarily mean that a comma is used as the delimiter. In this case, we have a tab character as the delimiter for our columns, so we will be using `sep='\\t'` to tell pandas to 'cut' the columns every time it sees a [tab character in the file](http://vim.wikia.com/wiki/Showing_the_ASCII_value_of_the_current_character)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# read the dataset as a DataFrame into a variable named 'prod'\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# show the head of this dataset\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"*Documentation Pause*\n",
|
|
"\n",
|
|
"How did we know how to use `pd.read_csv`? Let's take a look at the [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html). Note the first argument required (`filepath`).\n",
|
|
"> Take a moment to dissect other arguments and options when reading in data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We have just created a data structure called a `DataFrame`. See?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# show the type of this 'prod' dataset\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Inspecting our DataFrame: The basics\n",
|
|
"\n",
|
|
"We'll now perform basic operations on the DataFrame, denoted with comments."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# print the first and last 3 rows\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Notice that `.head()` is a method (denoted by parantheses), so it takes arguments."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Class Question:** \n",
|
|
"- What do you think changes if we pass a different number `head()` argument?\n",
|
|
"- How would we print the last 5 rows?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# identify the shape (rows by columns)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here we have 504 rows, and 25 columns. This is a tuple, so we can extract the parts using indexing:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# print the number of rows\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Using the index\n",
|
|
"An [index](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Index.html) is an immutable ndarray implementing an ordered, sliceable set. The basic object storing axis labels for all pandas objects. Think of it as a 'row address' for your data frame (table). It is best practice to explicitly set the index of your dataframe, as these are commonly used as [primary keys](https://en.wikipedia.org/wiki/Primary_key) which can be used to [join](https://www.w3schools.com/sql/sql_join.asp) your dataframe to other dataframes.\n",
|
|
"\n",
|
|
"The dataframe can store different types (int, string, datetime), and when importing data will automatically assign a number to each row, starting at zero and counting up. You can overwrite this, which is what we are going to do."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# display the index as it sits (auto-generated upon import)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# note that our auto-generated index has no name\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Here we are looking at three columns;\n",
|
|
"# the one on the left is the index (automatically generated upon import by pandas)\n",
|
|
"# 'ProductID' is our PK (primary key) from our imported table. 'Name' is a data column.\n",
|
|
"# Notice that the generated index starts at zero and our PK starts at 1.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting the index overwrites the automatically generated index\n",
|
|
"# with our PK, which resided in the 'ProductID' column.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Note how our index property has changed as a result\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# And our index has also inherited the name of our 'ProductID' column\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Column headers and datatypes"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# print the columns\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# examine the datatypes of the columns\n",
|
|
"# note that these were automatically inferred by pandas upon import!\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Class Question:** Why do datatypes matter? What operations could we perform on some datatypes that we could not on others? Note the importance of this in checking dataset integrity."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Selecting a Column\n",
|
|
"\n",
|
|
"We can select columns in two ways. Either we treat the column as an attribute of the DataFrame or we index the DataFrame for a specific element (in this case, the element is a column name)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# show the difference in data types by using single and double square brackets\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# select the Name column only, returned as a Series object\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# select the Name column only, returned as a DataFrame object\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# selecting > 1 column (must use double brackets!)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Class Question:** What if we wanted to select a column that has a space in it? Which method from the above two would we use? Why?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## loc and iloc\n",
|
|
"\n",
|
|
"`loc` and `iloc` are ways to select multiple rows and columns _at the same time_. \n",
|
|
"- `loc` uses label-based selection (the index values and column names)\n",
|
|
"- `iloc` used position-based selection (the position of the row/column within the df)\n",
|
|
"- For each of them, you specify the rows you want first, followed by the columns. The row values are required, the columns are not."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# use loc to select columns bewteen 'Name' and `Color, and rows with index value 1-3\n",
|
|
"# Note that both endpoints are included for the row and column ranges; this is slightly different than the Python range() function or list slicing\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# use iloc to get the first 5 rows and the last two columns\n",
|
|
"# The 0 at the start of the row range is optional\n",
|
|
"# iloc *does* work like list slicing in that the endpoint of the range is excluded\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Renaming Columns\n",
|
|
"\n",
|
|
"Perhaps we want to rename our columns. There are a few options for doing this.\n",
|
|
"\n",
|
|
"Renaming **specific** columns by using a dictionary:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# rename one or more columns with a dictionary. Note: inplace=False will return a new dataframe,\n",
|
|
"# but leave the original dataframe untouched. Change this to True to modify the original dataframe.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Renaming **ALL** columns with a new list of column names.\n",
|
|
"\n",
|
|
"Note that the `pd.DataFrame.columns` property can be cast to a `list` type. Originally, it's a `pd.core.indexes.base.Index` object:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print('My columns look like:\\n', prod.columns)\n",
|
|
"print('\\nAnd the type is:\\n', type(prod.columns))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can explicitly cast these to a list object as such, by using the built-in `list()` function:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"print('Now my columns look like:\\n', list(prod.columns))\n",
|
|
"print('\\nAnd are of type:\\n', type(list(prod.columns)))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can place these columns into a variable, `cols`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# declare a list of strings - these strings will become the new column names\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can use list indexing to mutate the columns we want:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# overwrite one of the column names using list indexing\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Finally, we can set the `pd.DataFrame.columns` property (this is a settable class property), to the new `cols` list, overwriting the existing columns header names:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Note that our first column name has changed from 'Name' to 'NewName'\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# display the final result of the 'NewName' column\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Common Column Operations\n",
|
|
"\n",
|
|
"While this is non-comprehensive, these are a few key column-specific data checks.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Descriptive statistics:** the minimum, first quartile, median, third quartile, and maximum.\n",
|
|
"\n",
|
|
"(And more! The mean too.)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Five Number Summary (all assumes numeric data):\n",
|
|
"- **Min:** The smallest value in the column\n",
|
|
"- **Max:** The largest value in the column\n",
|
|
"- **Quartile:** A quartile is one fourth of our data\n",
|
|
" - **First quartile:** This is the bottom most 25 percent\n",
|
|
" - **Median:** The middle value. (Line all values biggest to smallest - median is the middle!) Also the 50th percentile\n",
|
|
" - **Third quartile:** This the the top 75 percentile of our data\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# note - .describe() *default* only checks numeric datatypes\n",
|
|
"# show .describe() for the 'MakeFlag', 'SafetyStockLevel', 'StandardCost' fields.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Value Counts:** `pd.Series.value_counts()` count the occurrence of each value within our series."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# show the most popular product colors (aggregated by count, descending by default)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Unique values:** Determine the number of distinct values within a given series."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# What are the unique colors for the products?\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# how many distinct colors are there?\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# We can also include nulls with .nunique() as such:\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Filtering on a Single Condition\n",
|
|
"\n",
|
|
"Filtering and sorting are key processes that allow us to drill into the 'nitty gritty' and cross sections of our dataset.\n",
|
|
"\n",
|
|
"To filter, we use a process called **Boolean Filtering**, wherein we define a Boolean condition, and use that Boolean condition to filer on our DataFrame."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Recall: our given dataset has a column `Color`. Let's see if we can find all products that are `Black`. Let's take a look at the first 10 rows of the dataframe to see how it looks as-is:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# show the first 10 rows of the 'Color' column\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"By applying a `boolean mask` to this dataframe, `== 'Black'`, we can get the following:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# set the previous output to == Black\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we can use that 'mask' from above, and apply it to our full dataframe. Every time we have a `True` in a row, we return the row. If we have a `False` in that row, we do not return it. The result is a dataframe that only has rows where `Color` is `Black`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# show the full prod DataFrame where Color is Black\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Let's calculate the **average ListPrice** for the **salable products**."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"> Think: What are the component parts of this problem?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# First, we need to get salable items. \n",
|
|
"# Use your data dictionary from the beginning of this lesson.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now, we need to find average list price of those above items. Let's just get the 'ListPrice' column for starters."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"To get the average of that column, just take `.mean()`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can take a shortcut and just use `.describe()` here:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Sneak peek**: Another handy trick is to use `.hist()` to get a distribution of a continuous variable - in this case, `ListPrice`. We'll cover this more in future lessons:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Filtering on Multiple Conditions\n",
|
|
"\n",
|
|
"Here, we will filter on _multiple conditions_. Before, we filtered on rows where Color was Black. We also filtered where FinishedGoodsFlag was equal to 1. Let's see what happens when we filter on *both* simultaneously. \n",
|
|
"\n",
|
|
"The format for multiple conditions is:\n",
|
|
"\n",
|
|
"`df[ (df['col1'] == value1) & (df['col2'] == value2) ]`\n",
|
|
"\n",
|
|
"Or, more simply:\n",
|
|
"\n",
|
|
"`df[ (CONDITION 1) & (CONDITION 2) ]`\n",
|
|
"\n",
|
|
"Which eventually may evaluate to something like:\n",
|
|
"\n",
|
|
"`df[ True & False ]`\n",
|
|
"\n",
|
|
"...on a row-by-row basis. If the end result is `False`, the row is omitted.\n",
|
|
"\n",
|
|
"_Don't forget parentheses in your conditions!_ This is a common mistake."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Let's look at a table where Color is Black, AND FinishedGoodsFlag is 1\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Here we have an example of a list price of greater than 50, \n",
|
|
"# OR a product size that is not equal to 'XL'\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Sorting\n",
|
|
"\n",
|
|
"We can sort one column of our DataFrame as well."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# let's sort by standard cost, descending\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"This one is a little more advanced, but it demonstrates a few things:\n",
|
|
"- Conversion of a `numpy.ndarray` object (return type of `pd.Series.unique()`) into a `pd.Series` object\n",
|
|
"- `pd.Series.sort_values` with the `by=` kwarg omitted (if only one column is the operand, `by=` doesn't need specified\n",
|
|
"- Alphabetical sort of a string field, `ascending=True` means A->Z\n",
|
|
"- Inclusion of nulls, `NaN` in a string field (versus omission with a float/int as prior example)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#alternate method\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Independent Exercises\n",
|
|
"\n",
|
|
"Do your best to complete the following prompts. Don't hesitate to look at code we wrote together!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Print the first 4 rows of the whole DataFrame."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"How many rows are in the dataframe? Return the answer as an int."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"How many columns? Retrun the answer as an int."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"How many different product lines are there?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"What are the values of these product lines?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Do the number of values for the product lines match the number you have using `.nunique()`? Why or why not?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Take the output from your previous answer (using `.unique()`). Select the label corresponding to the `Road` product line using list indexing notation. How many characters are in this string?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Do you notice anything odd about this?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"How many products are there for the `Road` product line? Don't forget what you just worked on above! Return your answer as an int."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"How many products are there in the `Women's` `Mountain` category? Return your answer as an int. _Hint: Use the data dictionary above!_"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"**Challenge:** What are the top 3 _most expensive list price_ product that are either in the `Women's` `Mountain` category, _OR_ `Silver` in `Color`? Return your answer as a DataFrame object, with the `ProductID` index, `NewName` relabeled as `Name`, and `ListPrice` columns. Perform the statement in one execution, and do not mutate the source DataFrame."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# your answer here\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Recap\n",
|
|
"\n",
|
|
"We covered a lot of ground! It's ok if this takes a while to gel.\n",
|
|
"\n",
|
|
"```python\n",
|
|
"\n",
|
|
"# basic DataFrame operations\n",
|
|
"df.head()\n",
|
|
"df.tail()\n",
|
|
"df.shape\n",
|
|
"df.columns\n",
|
|
"df.Index\n",
|
|
"\n",
|
|
"# selecting columns\n",
|
|
"df.column_name\n",
|
|
"df['column_name']\n",
|
|
"\n",
|
|
"# renaming columns\n",
|
|
"df.rename({'old_name':'new_name'}, inplace=True)\n",
|
|
"df.columns = ['new_column_a', 'new_column_b']\n",
|
|
"\n",
|
|
"# notable columns operations\n",
|
|
"df.describe() # five number summary\n",
|
|
"df['col1'].nunique() # number of unique values\n",
|
|
"df['col1'].value_counts() # number of occurrences of each value in column\n",
|
|
"\n",
|
|
"# filtering\n",
|
|
"df[ df['col1'] < 50 ] # filter column to be less than 50\n",
|
|
"df[ (df['col1'] == value1) & (df['col2'] > value2) ] # filter column where col1 is equal to value1 AND col2 is greater to value 2\n",
|
|
"\n",
|
|
"# sorting\n",
|
|
"df.sort_values(by='column_name', ascending = False) # sort biggest to smallest\n",
|
|
"\n",
|
|
"```\n",
|
|
"\n",
|
|
"\n",
|
|
"It's common to refer back to your own code *all the time.* Don't hesistate to reference this guide! 🐼\n",
|
|
"\n",
|
|
"\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.7.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|