---
jupyter:
  jupytext:
    notebook_metadata_filter: all,-language_info
    split_at_heading: true
    text_representation:
      extension: .Rmd
      format_name: rmarkdown
      format_version: '1.2'
      jupytext_version: 1.8.2
  kernelspec:
    display_name: Python 3
    language: python
    name: python3
---

# Cross-tabulation

```{python tags=c("hide-cell")}
import numpy as np
import pandas as pd
# Safe setting for Pandas.  Needs Pandas version >= 1.5.
pd.set_option('mode.copy_on_write', True)
```

It is often useful to make tables that count rows in categories defined by
columns.

Here we will use a dataset with data about passengers on the [RMS
Titanic](https://en.wikipedia.org/wiki/Titanic).

See [the Titanic dataset page](data/titanic) for some more information about
this dataset, and a link to download the data.

```{python}
# Load the dataset
df = pd.read_csv('data/titanic_clean.csv')
df.head()
```

As you can see, we have various bits of information about the passengers on
the Titanic.  Several columns give labels - categories.  For example, the
`gender` column categorizes the passenger as `male` or `female`, and the
`survived` column categorizes the passenger by whether they survived the
sinking - `yes` or `no`.


## A one-way table


As you know, we can use the `value_counts` method of the data frame to see the
unique values for any column, with the counts of the number of times each
value occurs.

```{python}
# Counts of each unique value in the 'survived' column.
df['survived'].value_counts()
```

Often we want to see *proportions* instead of counts.   We can use the `normalize` keyword argument for this:

```{python}
# Proportions of each unique value in the 'survived' column.
df['survived'].value_counts(normalize=True)
```

## Two-way tables


We often want to break down the rows by more than one category.  Remember that
the Titanic passengers and crew tended to give preference to women and
children, when loading the lifeboats.  So, we may want to see the counts of
passengers who survived, broken down by `gender`.

This is the role of the Pandas `crosstab` function.   It is a Pandas function
because it is function inside the Pandas module; we can get this function with
`pd.crosstab` (assuming we have done the usual `import pandas as pd`).

The first argument to `pd.crosstab` is the category we want to see in the rows; the second argument is the category we want to see in the columns.

Here is a cross-tabulation of `gender` (in the rows) by `survived` (in the columns):

```{python}
# Cross-tabulation of counts for 'gender' (rows) by 'survived' (columns).
pd.crosstab(df['gender'], df['survived'])
```

We will often want to see these values as proportions rather than counts.  For
example, we may be interested in the *proportion* of women and men that
survived.   As for `value_counts` above, we use the `normalize` keyword to ask
for proportions.  This time we have to specify the direction that Pandas
should use for the proportion.  We could be interested in the proportion
across the column (proportions of `male` and `female` passengers within the
`yes` "survived" category, likewise for the `no` category).  More likely, in
this case, we will be interested in proportions across the row (proportion who
survived within `male` category, proportion who survived within `female`
category).   We give Pandas this information with the value for the
`normalize` keyword argument. Pandas uses the term `index` to refer to the
rows.  Remember, Pandas also uses the term index for the row labels.


```{python}
# Cross-tabulation of proportions for 'gender' (rows) by 'survived' (columns).
# Proportions are over the row.
pd.crosstab(df['gender'], df['survived'], normalize='index')
```

We could also ask for the proportions over the columns.

```{python}
# Cross-tabulation of proportions for 'gender' (rows) by 'survived' (columns).
# Proportions are over the column.
pd.crosstab(df['gender'], df['survived'], normalize='columns')
```

Of course, the arguments we are passing to `pd.crosstab` are Pandas Series.  We can pass any Pandas series as our row or column specification.

For example, we might want to make a new Series that categorizes passengers as
`male`, `female` or `child`.  We can use that Series as an argument to
`pd.crosstab`.

```{python}
mwc = df['gender']
mwc[df['age'] <= 16] = 'child'
pd.crosstab(mwc, df['survived'], normalize='index')
```

## Three-way tables (or four ...)


Another factor that we know was in play was passenger class.  For various
reasons, first class passengers, in particular, got better access to the
lifeboats than passengers in other classes.   This adds a further factor that
might explain the numbers and proportions that survived.

Now we have two factors (categories) that we think may predict survival - `class` and `gender`.

`pd.crosstab` takes this in its stride.  We can just pass a list of Series as our row or column specification.

```{python}
# Counts for passengers who survived, broken down by class and male, female,
# child.
categories = [df['class'], mwc]
pd.crosstab(categories, df['survived'])
```

`normalize` over the rows will give proportions for each sub-category:

```{python}
# Proportions surviving in sub-categories of class, male/female/child.
pd.crosstab(categories, df['survived'], normalize='index')
```

You can specify multiple Series for the columns of the table.  Here we just run the equivalent table but reversing the rows and columns:

```{python}
# Proportions surviving in sub-categories of class, male/female/child,
# with 'survived' in the rows, and the subcategories in the columns.
pd.crosstab(df['survived'], categories, normalize='columns')
```

You can specify more than two columns in the row or column arguments, or both.
Try experimenting with different arguments.
