{
"cells": [
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"import otter\n",
"grader = otter.Notebook()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Homework 9: Bootstrap, Resampling, CLT"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Reading**: \n",
"* [Estimation](https://www.inferentialthinking.com/chapters/13/Estimation.html)\n",
"* [Why the mean matters](https://www.inferentialthinking.com/chapters/14/Why_the_Mean_Matters.html)\n",
"\n",
"Directly sharing answers is not okay, but discussing problems with the course staff or with other students is encouraged. Refer to the policies page to learn more about how to learn cooperatively.\n",
"\n",
"For all problems that you must write our explanations and sentences for, you **must** provide your answer in the designated space. Moreover, throughout this homework and all future ones, please be sure to not re-assign variables throughout the notebook! For example, if you use `max_temperature` in your answer to one question, do not reassign it later on.\n",
"\n",
"As usual, **run the cell below** to import modules and autograder tests."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'imports complete'"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Run this cell to set up the notebook, but please don't change it.\n",
"\n",
"# These lines import the Numpy and Datascience modules.\n",
"import numpy as np\n",
"from datascience import *\n",
"\n",
"# These lines do some fancy plotting magic.\n",
"import matplotlib\n",
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"plt.style.use('fivethirtyeight')\n",
"import warnings\n",
"warnings.simplefilter('ignore', FutureWarning)\n",
"\n",
"# These lines load the tests.\n",
"import otter\n",
"grader = otter.Notebook()\n",
"'imports complete'\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Preliminaries\n",
"The British Royal Air Force wanted to know how many warplanes the Germans had (some number `N`, which is a *parameter*), and they needed to estimate that quantity knowing only a random sample of the planes' serial numbers (from 1 to `N`). We know that the German's warplanes are labeled consecutively from 1 to `N`, so `N` would be the total number of warplanes they have. \n",
"\n",
"We normally investigate the random variation among our estimates by simulating a sampling procedure from the population many times and computing estimates from each sample that we generate. In real life, if the British Royal Air Force (RAF) had known what the population looked like, they would have known `N` and would not have had any reason to think about random sampling. However, they didn't know what the population looked like, so they couldn't have run the simulations that we normally do. \n",
"\n",
"Simulating a sampling procedure many times was a useful exercise in *understanding random variation* for an estimate, but it's not as useful as a tool for practical data analysis.\n",
"\n",
"Let's flip that sampling idea on its head to make it practical. **Given *just* a random sample of serial numbers, we'll estimate `N`, and then we'll use simulation to find out how accurate our estimate probably is, without ever looking at the whole population.** This is an example of *statistical inference*.\n",
"\n",
"We (the RAF in World War II) want to know the number of warplanes fielded by the Germans. That number is `N`. The warplanes have serial numbers from 1 to `N`, so `N` is also equal to the largest serial number on any of the warplanes.\n",
"\n",
"We only see a small number of serial numbers (assumed to be a random sample with replacement from among all the serial numbers), so we have to use estimation."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 1.1\n",
"Is `N` a population parameter or a statistic? If we use our random sample to compute a number that is an estimate of `N`, is that a population parameter or a statistic?\n",
"\n",
"Set `N` and `N_estimate` to either the string `\"parameter\"` or `\"statistic\"` to indicate whether each value is a parameter or a statistic.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q1_1\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"N = ...\n",
"N_estimate = ..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q1_1\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To make the situation realistic, we're going to hide the true number of warplanes from you. You'll have access only to this random sample:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [],
"source": [
"observations = Table.read_table(\"serial_numbers.csv\")\n",
"num_observations = observations.num_rows\n",
"observations"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 1.2\n",
"The average of the sample is about half of `N`. So one way to estimate `N` is to take twice the mean of the serial numbers we see. Write a function that computes that statistic. It should take as its argument an array of serial numbers and return twice their mean. Call the function `mean_based_estimator`. \n",
"\n",
"After that, use the function and the `observations` table to compute an estimate of `N` called `mean_based_estimate`.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q1_2\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"def mean_based_estimator(nums):\n",
" ...\n",
"\n",
"mean_based_estimate = ...\n",
"mean_based_estimate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q1_2\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 1.3\n",
"We can also estimate `N` by using the biggest serial number in the sample. Compute this value and give it the name `max_estimate`.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q1_3\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"max_estimate = ...\n",
"max_estimate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q1_3\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 1.4\n",
"Let's take a look at the values of `max_estimate` and `mean_based_estimate` that we got for our dataset. Which of these values is closer to the true population maximum `N`? Based off of our estimators, can we give a lower bound for what `N` must be? In other words, is there a value that `N` must be greater than or equal to?\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q1_4\n",
"manual: true\n",
"-->\n",
"<!-- EXPORT TO PDF -->"
]
},
{
"cell_type": "markdown",
"metadata": {
"export_pdf": true,
"manual_problem_id": "max_estimate"
},
"source": [
"*Write your answer here, replacing this text.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can't just confidently proclaim that `max_estimate` or `mean_based_estimate` is equal to `N`. What if we're really far off? We want to get a sense of the accuracy of our estimates."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Resampling\n",
"To do this, we'll use resampling. That is, we won't exactly simulate the observations the RAF would have really seen. Rather we sample from our current sample, or \"resample.\"\n",
"\n",
"Why does that make any sense?\n",
"\n",
"When we try to find the value of a population parameter, we ideally would like to use the whole population. However, we often only have access to one sample and we must use that to estimate the parameter instead.\n",
"\n",
"Here, we would like to use the population of serial numbers to draw more samples and run a simulation about estimates of `N`. But we still only have our sample. So, we **use our sample in place of the population** to run the simulation. We resample from our original sample with replacement as many times as there are elements in the original sample. This resampling technique is called *bootstrapping*. \n",
"\n",
"Note that in order for bootstrapping to work well, you must start with a large, random sample. Then the Law of Large Numbers says that with high probability, your sample is representative of the population."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 2.1\n",
"Write a function called `simulate_resample`. The function should take one argument `tbl`, which is a table like `observations`. The function should generate a resample from the observed serial numbers in `tbl`.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q2_1\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
"def simulate_resample(tbl):\n",
" ...\n",
"\n",
"simulate_resample(observations) # Don't delete this line"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q2_1\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll use many resamples at once to see what estimates typically look like. However, we don't often pay attention to single resamples, so it's easy to misunderstand them. Let's first answer some questions about our resample."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 2.2\n",
"Which of the following statements are true?\n",
"\n",
"1. The original sample can contain serial numbers that are not in the resample.\n",
"2. Because the sample size is small, the histogram of the resample might look very different from the histogram of the original sample.\n",
"3. The resample can contain serial numbers that are not in the original sample.\n",
"4. The original sample has exactly one copy of each serial number for every German plane.\n",
"5. The resample has either zero, one, or more than one copy of each serial number.\n",
"6. The resample has exactly the same sample size as the original sample.\n",
"\n",
"Assign `true_statements` to an array of the number(s) corresponding to correct statements.\n",
"\n",
"*Note:* The \"original sample\" refers to `observations`, and the \"resample\" refers the output of one call of `simulate_resample()`. \n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q2_2\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"true_statements = ..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q2_2\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"Now let's write a function to do many resamples at once.\n",
"\n",
"#### Question 2.3\n",
"Write a function called `sample_estimates`. It should take 3 arguments:\n",
"1. `serial_num_tbl`: A table from which the data should be sampled. The table will look like `observations`. \n",
"2. `statistic`: A *function* that takes in an array of serial numbers as its argument and computes a statistic from the array (i.e. returns a calculated number). \n",
"3. `num_replications`: The number of simulations to perform.\n",
"\n",
"*Hint: You should use the function `simulate_resample` which you defined in Question 2.1*\n",
"\n",
"The function should simulate many samples **with replacement** from the given table. For each of those samples, it should compute the statistic on that sample. Then it should **return an array** containing each of those statistics. The code below provides an example use of your function and describes how you can verify that you've written it correctly.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q2_3\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"for_assignment_type": "solution"
},
"outputs": [],
"source": [
"def sample_estimates(serial_num_tbl, statistic, num_replications):\n",
" ...\n",
"\n",
"# DON'T CHANGE THE CODE BELOW THIS COMMENT! (If you do, you will fail the hidden test)\n",
"# This is just an example to test your function.\n",
"# This should generate an empirical histogram of twice-mean-based estimates\n",
"# of N from samples of size 50 if N is 1000. This should be a bell-shaped\n",
"# curve centered at roughly 900 with most of its mass in [800, 1200]. To verify your\n",
"# answer, make sure that's what you see!\n",
"population = Table().with_column(\"serial number\", np.arange(1, 1000+1))\n",
"one_sample = Table.read_table(\"one_sample.csv\") #This is a sample from the population table\n",
"example_estimates = sample_estimates(\n",
" one_sample,\n",
" mean_based_estimator,\n",
" 10000)\n",
"Table().with_column(\"mean-based estimate\", example_estimates).hist(bins=np.arange(0, 1500, 25))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q2_3\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can go back to the sample we actually observed (the table `observations`) and estimate how much our mean-based estimate of `N` would have varied from sample to sample."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 2.4\n",
"Using the bootstrap and the sample `observations`, simulate the approximate distribution of *mean-based estimates* of `N`. Use 7,500 replications and save the estimates in an array called `bootstrap_mean_based_estimates`. \n",
"\n",
"We have provided code that plots a histogram, allowing you to visualize the simulated estimates.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q2_4\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [],
"source": [
"bootstrap_mean_based_estimates = ...\n",
"\n",
"# Don't change the code below! This plots bootstrap_mean_based_estimates.\n",
"Table().with_column(\"mean-based estimate\", bootstrap_mean_based_estimates).hist(bins=np.arange(0, 200, 4)) "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q2_4\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 2.5\n",
"Using the bootstrap and the sample `observations`, simulate the approximate distribution of *max estimates* of `N`. Use 7,500 replications and save the estimates in an array called `bootstrap_max_estimates`.\n",
"\n",
"We have provided code that plots a histogram, allowing you to visualize the simulated estimates.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q2_5\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"bootstrap_max_estimates = ...\n",
"\n",
"# Don't change the code below! This plots bootstrap_max_estimates.\n",
"Table().with_column(\"max estimate\", bootstrap_max_estimates).hist(bins=np.arange(0, 200, 4)) "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q2_5\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 2.6\n",
"`N` was actually 150! Compare the histograms of estimates you generated in 2.4 and 2.5 and answer the following questions:\n",
"\n",
"1. How does the distribution of values for the mean-based estimates differ from the max estimates? Do both distributions contain the true max value?\n",
"2. Which estimator is more dependent on the original random sample? Why so?\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q2_6\n",
"manual: true\n",
"-->\n",
"<!-- EXPORT TO PDF -->"
]
},
{
"cell_type": "markdown",
"metadata": {
"export_pdf": true
},
"source": [
"*Write your answer here, replacing this text.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Computing intervals"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 3.1\n",
"Compute an interval that covers the middle 95% of the mean-based bootstrap estimates. Assign your values to `left_end_1` and `right_end_1`. \n",
"\n",
"*Hint:* Use the `percentile` function! Read up on its documentation [here](http://data8.org/sp19/python-reference.html).\n",
"\n",
"Verify that your interval looks like it covers 95% of the area in the histogram. The red dot on the histogram is the value of the parameter (150).\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q3_1\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [],
"source": [
"left_end_1 = ...\n",
"right_end_1 = ...\n",
"print(\"Middle 95% of bootstrap estimates: [{:f}, {:f}]\".format(left_end_1, right_end_1))\n",
"\n",
"# Don't change the code below! It draws your interval and N on the histogram of mean-based estimates.\n",
"Table().with_column(\"mean-based estimate\", bootstrap_mean_based_estimates).hist(bins=np.arange(0, 200, 4)) \n",
"plt.plot(make_array(left_end_1, right_end_1), make_array(0, 0), color='yellow', lw=3, zorder=1)\n",
"plt.scatter(150, 0, color='red', s=30, zorder=2);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q3_1\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 3.2\n",
"Write code that simulates the sampling and bootstrapping process again, as follows:\n",
"\n",
"1. Generate a new set of random observations the RAF might have seen by sampling from the `population` table we have created for you below. Use the sample size `num_observations`.\n",
"2. Compute an estimate of `N` from these new observations, using `mean_based_estimator`.\n",
"3. Using only the new observations, compute 10,000 bootstrap estimates of `N`.\n",
"4. Plot these bootstrap estimates and compute an interval covering the middle 95%.\n",
"\n",
"*Note:* Traditionally, when we bootstrap using a sample from the population, that sample is usually a simple random sample (i.e., sampled uniformly at random from the population without replacement). However, if the population size is big enough, the difference between sampling with replacement and without replacement is negligible. Think about why that's the case! This is why when we define `new_observations`, we sample with replacement.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q3_2\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"population = Table().with_column(\"serial number\", np.arange(1, 150+1))\n",
"\n",
"new_observations = ...\n",
"new_mean_based_estimate = ...\n",
"new_bootstrap_estimates = ...\n",
"Table().with_column(\"mean-based estimate\", new_bootstrap_estimates).hist(bins=np.arange(0, 252, 4))\n",
"new_left_end = ...\n",
"new_right_end = ...\n",
"\n",
"# Don't change code below this line!\n",
"print(\"New mean-based estimate: {:f}\".format(new_mean_based_estimate))\n",
"print(\"Middle 95% of bootstrap estimates: [{:f}, {:f}]\".format(new_left_end, new_right_end))\n",
"\n",
"plt.plot(make_array(new_left_end, new_right_end), make_array(0, 0), color='yellow', lw=3, zorder=1)\n",
"plt.scatter(150, 0, color='red', s=30, zorder=2);"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q3_2\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 3.3\n",
"Does the interval covering the middle 95% of the new bootstrap estimates include `N`? If you ran that cell 100 times and generated 100 intervals, how many of those intervals would you expect to include `N`?\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q3_3\n",
"manual: true\n",
"-->\n",
"<!-- EXPORT TO PDF -->"
]
},
{
"cell_type": "markdown",
"metadata": {
"export_pdf": true
},
"source": [
"*Write your answer here, replacing this text.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's look at what happens when we use a small number of resamples:\n",
"\n",
"<img src=\"smallrephist.png\" width=\"525\"/>\n",
"\n",
"This histogram and confidence interval was generated using 10 resamples of `new_observations`."
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"#### Question 3.4\n",
"In the cell below, explain why this histogram and confidence interval look different from the ones you generated previously in Question 3.2 where the number of resamples was 10,000.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q3_4\n",
"manual: true\n",
"-->\n",
"<!-- EXPORT TO PDF -->"
]
},
{
"cell_type": "markdown",
"metadata": {
"export_pdf": true
},
"source": [
"*Write your answer here, replacing this text.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## 4. The CLT and Book Reviews\n",
"\n",
"Your friend has recommended you a book, so you look for it on an online marketplace. You decide to look at reviews for the book just to be sure that it's worth buying. Let's say that on Amazon, the book only has 80% positive reviews. On GoodReads, it has 95% positive reviews. You decide to investigate a bit further by looking at the percentage of positive reviews for the book on 5 different websites that you know of, and you collect these positive review percentages in a table called `reviews.csv`."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, we've loaded in the table for you."
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"reviews = Table.read_table(\"reviews.csv\") \n",
"reviews"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"**Question 4.1**. Calculate the average percentage of positive reviews from your sample and assign it to `initial_sample_mean`.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_1\n",
"manual: false\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {},
"outputs": [],
"source": [
"initial_sample_mean = ...\n",
"initial_sample_mean"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q4_1\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"You've calculated the average percentage of positive reviews from your sample, so now you want to do some inference using this information. \n",
"\n",
"**Question 4.2**. First, simulate 5000 bootstrap resamples of the positive review percentages. For each bootstrap resample, calculate the resample mean and store the resampled means in an array called `resample_positive_percentages`. Then, plot a histogram of the resampled means.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_2\n",
"manual: false\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 72,
"metadata": {},
"outputs": [],
"source": [
"resample_positive_percentages = make_array()\n",
"\n",
"for i in np.arange(5000):\n",
" resample = ...\n",
" resample_avg_positive = ...\n",
" resample_positive_percentages = ...\n",
" \n",
"# Do NOT change these lines.\n",
"(Table().with_column(\"Average % of Positive Reviews in Resamples\", \n",
" resample_positive_percentages).hist(\"Average % of Positive Reviews in Resamples\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q4_2\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"**Question 4.3**. What is the the shape of the empirical distribution of the average percentage of positive reviews based on our original sample? What value is the distribution centered at? Assign your answer to the variable `initial_sample_mean_distribution`--your answer should be either `1`, `2`, `3`, or `4` corresponding to the following choices:\n",
"\n",
"*Hint: Look at the histogram you made in Question 2. Run the cell that generated the histogram a few times to check your intuition.*\n",
"\n",
"1. The distribution is approximately normal because of the Central Limit Theorem, and it is centered at the original sample mean.\n",
"2. The distribution is not necessarily normal because the Central Limit Theorem may not apply, and it is centered at the original sample mean.\n",
"3. The distribution is approximately normal because of the Central Limit Theorem, but it is not centered at the original sample mean.\n",
"4. The distribution is not necessarily normal because the Central Limit Theorem may not apply, and it is not centered at the original sample mean.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_3\n",
"manual: false\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 75,
"metadata": {},
"outputs": [],
"source": [
"initial_sample_mean_distribution = ..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q4_3\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"According to the Central Limit Theorem, the probability distribution of the sum or average of a *large random sample* drawn with replacement will be roughly normal, regardless of the distribution of the population from which the sample is drawn. \n",
"\n",
"**Question 4.4**. Note the statement about the sample being large and random. Is this sample large and random? Give a brief explanation.\n",
"\n",
"*Note: The setup at the beginning of this exercise explains how the sample was gathered.*\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_4\n",
"manual: true\n",
"-->\n",
"<!-- EXPORT TO PDF -->"
]
},
{
"cell_type": "markdown",
"metadata": {
"export_pdf": true
},
"source": [
"*Write your answer here, replacing this text.*"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"Though you have an estimate of the true percentage of positive reviews (the sample mean), you want to measure how variable this estimate is. \n",
"\n",
"**Question 4.5**. Find the standard deviation of your resampled average positive review percentages, which you stored in `resample_positive_percentages`, and assign the result to the variable `resampled_means_variability`.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_5\n",
"manual: false\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 79,
"metadata": {},
"outputs": [],
"source": [
"resampled_means_variability = ...\n",
"resampled_means_variability"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q4_5\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This estimate is pretty variable! To make the estimate less variable, let's say you found a way to randomly sample reputable marketplaces from across the web which sell this book. Let's say that there are up to 150 of these marketplaces. The percentages of positive reviews are loaded into the table `more_reviews`."
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"# Just run this cell\n",
"more_reviews = Table.read_table(\"more_reviews.csv\")\n",
"more_reviews"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"In the next few questions, we'll test an important result of the Central Limit Theorem. According to the CLT, the standard deviation of all possible sample means can be calculated using the following formula:\n",
"\n",
"$$\n",
"\\text{SD of all possible sample means} = \\dfrac{\\text{Population SD}}{\\sqrt{\\text{sample size}}}\n",
"$$\n",
"\n",
"This formula gives us another way to approximate the SD of the sample means other than calculating it empirically. We can test how well this formula works by calculating the SD of sample means for different sample sizes.\n",
"\n",
"The following code calculates the SD of sample means using the CLT and empirically for a range of sample sizes. Then, it plots a scatter plot comparing the SD of the sample means calculated with both methods. Each point corresponds to a different sample size. "
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"# Just run this cell. It's not necessary for you to read this code, but you can do 99% of this on your own!\n",
"# Note: this cell might take a bit to run.\n",
"\n",
"def empirical_sample_mean_sd(n):\n",
" sample_means = make_array()\n",
" for i in np.arange(500):\n",
" sample = more_reviews.sample(n).column('Positive Review Percentage') \n",
" sample_mean = np.mean(sample) \n",
" sample_means = np.append(sample_means, sample_mean) \n",
" return np.std(sample_means)\n",
"\n",
"def predict_sample_mean_sd(n):\n",
" return np.std(more_reviews.column(0)) / (n**0.5)\n",
"\n",
"sd_table = Table().with_column('Sample Size', np.arange(1,151))\n",
"predicted = sd_table.apply(predict_sample_mean_sd, 'Sample Size')\n",
"empirical = sd_table.apply(empirical_sample_mean_sd, 'Sample Size')\n",
"sd_table = sd_table.with_columns('Predicted SD', predicted, 'Empirical SD', empirical)\n",
"sd_table.scatter('Sample Size')\n",
"plt.ylabel(\"SD of Sample Mean\");"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"**Question 4.6**. Assign the numbers corresponding to all true statements to an array called `sample_mean_sd_statements`. \n",
"\n",
"1. The law of large numbers tells us that the distribution of a large random sample should resemble the distribution from which it is drawn.\n",
"2. The SD of the sample means is proportional to the square root of the sample size. \n",
"3. The SD of the sample means is proportional to 1 divided by the square root of the sample size. \n",
"4. The law of large numbers guarantees that empirical and predicted sample mean SDs will be exactly equal to each other when the sample size is large.\n",
"5. The law of large numbers guarantees that empirical and predicted sample mean SDs will be approximately equal to each other when the sample size is large.\n",
"6. The plot above shows that as our sample size increases, our estimate for the true percentage of positive reviews becomes more accurate.\n",
"7. The plot above shows that the size of the population affects the SD of the sample means.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_6\n",
"manual: false\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [],
"source": [
"sample_mean_sd_statements = ..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q4_6\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false,
"editable": false
},
"source": [
"Often times, when conducting statistical inference, you'll want your estimate of a population parameter to have a certain accuracy. It is common to measure accuracy of an estimate using the SD of the estimate--as the SD goes down, your estimate becomes less variable. As a result, the width of the confidence interval for your estimate decreases (think about why this is true). We know from the Central Limit Theorem that when we estimate a sample mean, the SD of the sample mean decreases as the sample size increases (again, think about why this is true). \n",
"\n",
"**Question 4.7**. Imagine you are asked to estimate the true average percentage of positive reviews for this book and you have not yet taken a sample of review websites. Which of these is the best way to decide how large your sample should be to achieve a certain level of accuracy for your estimate of the true average percentage of positive reviews? Assign `sample_size_calculation` to either `1`, `2`, or `3` corresponding to the statements below.\n",
"\n",
"*Note: Assume you know the population SD or can estimate it with reasonable accuracy.*\n",
"1. Take many random samples of different sizes, then calculate empirical confidence intervals using the bootstrap until you reach your desired accuracy. \n",
"2. Use the Central Limit Theorem to calculate what sample size you need in advance. \n",
"3. Randomly pick a sample size and hope for the best.\n",
"\n",
"<!--\n",
"BEGIN QUESTION\n",
"name: q4_7\n",
"manual: false\n",
"-->"
]
},
{
"cell_type": "code",
"execution_count": 92,
"metadata": {},
"outputs": [],
"source": [
"sample_size_calculation = ..."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false,
"editable": false
},
"outputs": [],
"source": [
"grader.check(\"q4_7\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Congratulations, you're done with Homework 9! Submit your assignment."
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.0"
}
},
"nbformat": 4,
"nbformat_minor": 1
}