Week 5 Application Exercise

This is the starting notebook for the Week 5 application exercise. It is intended to demonstrate several things:

• The use of simulation as a tool for understanding statistical methods
• Performing hypothesis tests
• The difference between paired and two-sample tests

Save the notebook file and .py file into the same folder to start work on the assignment.

Please treat the experiment as a black box and infer its behavior using the statistical techniques we have learned in class. After class, I invite you to look at its source code and see how it works.

The core idea of this is to identify whether “fabulators” under condition A have higher (or lower) nonsense production than under condition B. The code will describe an experimental design, and allow you to “run” the experiment to draw samples.

Software Requirements

This exercise requires an additional Python package that is not included in a default Anaconda install - the seedbank library. You can install this with Pip:

pip install seedbank

It's also available in Conda-Forge:

conda install -c conda-forge seedbank

Because it only has a few dependencies, and they are all included in almost all base Conda environments, the Pip installation works fine, and doesn't mix packages between Conda repositories.

Setup

This project requires an additional

We need to do our usual imports:

import pandas as pd
import numpy as np
import scipy.stats as stats
import statsmodels.api as sm
import seaborn as sns
import matplotlib.pyplot as plt

Because we are using random number generation, we want to seed the random number generator. If we initialize it with the a fixed seed, re-running the notebook repeatedly will produce the same results. This is useful for debugging and reproducibility. We'll often want to re-run with a different seed before submitting, just to make sure that our results aren't accidentally the result of a pathological choice of random seeds.

The Seedbank library initializes and seeds a wide range of Python random number generators. The basic usage is to directly seed with a call to initialize (for teaching notebooks, I often use the current date as my random seed):

import seedbank
seedbank.initialize(20210923)

However, if we all ran that code, we would all get the same results, but for this exercise I would like different teams to get different results. Therefore, we will take advantage of another Seedbank feature that allows us to specify additional string keys that get incorporated into the random seed. In the following cell, edit it to use your team name:

TEAM_NAME = 'DemoTeam'
import seedbank
seedbank.initialize(20220922, TEAM_NAME)

The last piece is to import our custom module to get a 'world' from which we can sample:

from cs533_w5_world import Experiment

And then create our experiment:

exp = Experiment(TEAM_NAME)

Getting Data

We first need to know where our data is coming from. The experiment describes itself:

print(exp.describe())

We can run an instance of this experiment with size 50:

SAMPLE_SIZE = 50
data = exp.run_experiment(SAMPLE_SIZE)
data

Comparing Conditions/Groups

Review the experiment description. You need to compare A and B with a t-test, but the precise details will depend on your experiment structure.

✅ Do you need to use a paired t-test or an independent two-sample t-test for this analysis?

✅ What is the null hypothesis for the test with this data?

✅ Compute the means of both groups or conditions:

✅ Compute the difference in means. How much more nonsense is produced in A vs. B?

This is also called the effect size (or specifically, the unstandardized effect size).

✅ Run the appropriate t-test to test if this difference is statistically significant and obtain a p-value:

✅ What does this result mean?

Bootstrap

✅ Bootstrap a confidence interval for the effect size (note that the bootstrap procedure will differ between paired and unpaired analyses):

Sampling Distribution

✅ Compute the effect size of 100 runs of your experiment. Describe the distribution of these effect sizes numerically and graphically.

⚠ While the confidence interval above will likely be close percentiles of the effect size distribution, they are not the same thing. Why is that?

The Answers

The experiment can tell you the answers (do not run this until you have completed the rest):

exp.answers()

Other Analysis

If you have time, create a second experiment with the opposite configuration of your initial. The experiment class takes a paired option that you can use to force a paired or unpaired design by passing True or False:

exp2 = Experiment(paired=True)

If you needed a paired analysis above, create an unpaired experiment (paired=False); if you used an independent analysis above, create a paired analysis. Repeat as much of your analysis as you can with the new experimental design.