---
jupyter:
  jupyterbook:
    show_stderr: true
  jupytext:
    formats: ''
    notebook_metadata_filter: all,-language_info
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.2'
      jupytext_version: 1.14.1
  kernelspec:
    display_name: Python 3 (ipykernel)
    language: python
    name: python3
---

# Selecting columns from DataFrames

```{python}
# Load the Pandas data science library, call it 'pd'
import pandas as pd
# Turn on a setting to use Pandas more safely.
# We will discuss this setting later.
pd.set_option('mode.copy_on_write', True)

# Load the library for plotting, name it 'plt'
import matplotlib.pyplot as plt
# Make plots look a little more fancy
plt.style.use('fivethirtyeight')
```

Let us return to the DataFrame we had at the end of the previous notebook [The
data frame index](df_index.Rmd).

```{python}
# Original data frame before dropping missing values.
gender_data = pd.read_csv('data/gender_stats_min.csv')
gender_data_no_na = gender_data.dropna()
labeled_gdata = gender_data_no_na.set_index('country_code')
# Show the result
labeled_gdata
```

## Selecting columns

You have seen array indexing (in [Selecting with
arrays](array_indexing).  You remember that array indexing uses
square brackets.  *Indexing* is the technical term for getting data from a
value using square brackets.  Data frames also allow indexing.  For example, we
often want to get all the data for a single column of the data frame.  To do
this, we use the same square bracket notation as we use for array indexing,
with the name of the column inside the square brackets.

```{python}
gdp = labeled_gdata['gdp_us_billion']
```

Call this *column label indexing*.

We all also call this *direct indexing with column labels* or DICL.

By *direct indexing* we mean indexing a DataFrame using square brackets after
the DataFrame value — the kind of indexing you will be familiar with from
arrays. Later we will come across another form of indexing, called *indirect*
indexing, but that is for another time.

What `type` of value came back from this *direct indexing with column labels*?

```{python}
type(gdp)
```

Pandas sent back a Series value from our selection of the column.  The Series
is a Pandas' way of representing a column of data.  Here is the default display
for the `gdp` Series:

```{python}
gdp
```

The display shows the *values* from the column, that you saw previously in the
DataFrame `gdp_us_billion` column. You see the values printed on the right.
Notice too that the Series has kept the *row labels* from the DataFrame, so
each value now has it's corresponding row label.  The index values are on the
left.

Notice that, if your string specifying the column name does not match a column
name exactly, you will get a long error.   This gives you some practice in
reading long error messages - skip to the end first, you will often see the
most helpful information there.

```{python tags=c("raises-exception")}
# The correct column name is in lower case.
labeled_gdata['GDP_US_BILLION']
```


## GDP and Maternal Mortality Rate


There are two ways of getting plots from data in data frames.   Here we use
the most basic method, that you have already seen.   Soon, we will get onto [a
more elegant plotting method](df_plotting).

The `gdp` variable is a sequence of values, so we can do a histogram on these
values, as we have done histograms on arrays.

```{python}
plt.hist(gdp)
plt.title('Gross Domestic Product (US billion)');
```

Now we have had a look at the GDP values, we will look at the values for the
Maternal Mortality Rate (MMR).  The MMR is the number of women who die in
childbirth for every 100,000 births.  These are the numbers in the
`mat_mort_ratio` column.

```{python}
mmr = labeled_gdata['mat_mort_ratio']
mmr
```

Notice the row labels on the left, and the values from the `mat_mort_ratio`
column on the right.

```{python}
plt.hist(mmr)
plt.title('Maternal Mortality Rate');
```

## Selecting more than one column

Let us say that we would like to avoid distraction by restricting our data
frame to the columns we are interested in.

There are our columns labels:

```{python}
col_labels = list(labeled_gdata)
col_labels
```

Let us say that, for now, we only want the first three columns.  We can do that
with *direct indexing with column labels* (DICL) too.  To do that, we put a
*list* of strings between the square brackets, instead of the string with the
column name.

Here are the column labels we want:

```{python}
# Select all labels but the last.
col_labels_we_want = col_labels[:-1]
col_labels_we_want
```

We can select only these columns using direct indexing:

```{python}
# Direct indexing with a list of column labels
thinner_gdata = labeled_gdata[col_labels_we_want]
thinner_gdata
```

## Dropping columns

We have just used DICL to select the columns we want.  In the case above, we
selected three out of the four column labels, and our DICL operation had the
effect of dropping the last column.

Another way of dropping a column is to use the DataFrame `.drop` method.

In fact the `.drop` method allows us to drop *columns and rows*.  To tell `.drop` we want to drop *columns* rather than rows, we use the `column=` keyword argument to `.drop`, like this:

```{python}
# Drop the population column.
thinner_again = labeled_gdata.drop(columns=['population'])
thinner_again
```


## Back to GDP and MMR

We are interested in the relationship of `gpp` and `mmr`.  Maybe richer
countries have better health care, and fewer maternal deaths.

Here is a plot, using the standard Matplotlib `scatter`
function.

```{python}
plt.scatter(gdp, mmr);
plt.title('Maternal Mortality Rate as a function of GDP');
```

The scatterplot is not convincing in showing a relationship between GDP and the
MMR.  That is puzzling, given the assumption that higher GDP tends to be
associated with better healthcare.

We need to explore some more.  For example, could it be that, beyond a certain
level of GDP, healthcare is good enough to prevent most maternal deaths?  In
that case, we might still expect a relationship between GDP and MMR in
lower-income countries, but a much weaker relationship for lower-income
countries.

To go further, we will see what we can do by sorting the DataFrame, and looking
at the first and last rows.
