This week’s tutorial will cover grouping and summarising data with Pandas and Plotly, and handling missing data with Pandas.

It is based on:

• swcarpentry.github.io/python-novice-gapminder/08-data-frames.html
• swcarpentry.github.io/python-novice-gapminder/09-plotting.html

Setup¶

1. Make a new notebook for this week
2. What’s the first thing to do? RENAME IT!
3. Name it week5.ipynb

Load Data into Pandas¶

As we did last week, let’s load the Pandas and Plotly libraries, and load our AirBnB listings data into Pandas:

We can see how big the DataFrame is using .shape:

We can also see what columns we have available to work with:

Grouping Data in Plotly¶

When analysing data, we often want to compare different groups.

One simple way is to colour data points by a categorical column when plotting. For example, colouring listings by their room type:

We can also specify a categorical column as a facet_row or facet_col to generate a separate graph for data points having each “facet” value.

In the plot below, note that we:

• Sort the data by the accommodates, so that the facet plots are sorted by value.
• Increase the height of the output to better view the large number of plots.
• Set histnorm='percent' so that groups with few data points still have a “full size” plot.

It can be useful to compute our own categorical columns, such as to compare one category against all others:

Grouping Data in Pandas¶

We can also use Pandas’ groupby() to split up a DataFrame according to a categorical attribute:

groupby() can also be used to produce a DataFrame of aggregate statistics for each group. In the following, see how we:

• Specify the columns from which to create a group for each combinations of values (['accommodates', 'room_type'])
• Use indexing to select a list of columns to calculate the statistic for (['price_nzd', 'review_scores_rating'])
• Choose a statistic to calculate: mean()
  • You can also compute sum(), std(), and more.

Note that groupby() turns the group columns into an index. We can turn it back into a regular column with reset_index():

Such DataFrames produced with groupby() can be very useful for producing plots of statistics, like bar charts:

We can also export our new DataFrame to a CSV file:

Handling Missing Data in Pandas¶

We can also use .count() to count the number of rows in each group, but notice that the count for the rating column is less than the price column:

This is because there are missing (aka null, aka NA) values in the rating column that are ignored by count().

We can use isna() to get a mask of rows where a value is missing:

We can then invert the mask with ~ and filter out rows with missing values:

Alternatively, we can use fillna() to replace missing values with a fixed value. For example, we make want to consider un-reviewed listings with a mid-point rating:

Finally, dropna() on a DataFrame can be used to simply remove any rows where any column contains a missing value: