Notebook 10: Statistical Inference and Hypothesis Testing

Module: Data Analytics Level: Intermediate to Advanced Prerequisites: Notebook 9 (Correlation, Covariance, and Relationships) Estimated Time: 4–5 hours

---

Table of Contents#

1. From Description to Inference
2. Sampling Distributions and Standard Error
3. Confidence Intervals
4. The Logic of Hypothesis Testing
5. P-Values: What They Are and Are Not
6. Type I and Type II Errors
7. The t-Test Family
8. The Chi-Square Test
9. A/B Testing
10. Statistical Power and Sample Size
11. Multiple Comparisons
12. Hands-On: A Complete A/B Test Analysis
13. Case Studies
14. Common Misconceptions and Pitfalls
15. Glossary
16. Exercises
17. Quiz
18. Summary
19. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Explain the relationship between a sampling distribution, standard error, and the Central Limit Theorem
• Construct and correctly interpret a confidence interval
• State null and alternative hypotheses correctly and explain the logic of hypothesis testing
• Interpret a p-value correctly, avoiding the most common misinterpretations
• Distinguish Type I and Type II errors and explain the tradeoff between them
• Select and apply the correct t-test variant for a given comparison
• Apply a chi-square test to assess independence between categorical variables
• Design and analyze a basic A/B test, including a proper sample size calculation
• Explain statistical power and why the multiple comparisons problem requires a correction

---

1. From Description to Inference#

Every notebook so far has described data you already have: summary statistics (Notebook 6), exploratory patterns (Notebook 7), visualizations (Notebook 8), and correlations (Notebook 9). Statistical inference answers a different, harder question: given a limited sample, what can you confidently conclude about the broader population it came from, or about whether an observed pattern reflects a genuine effect rather than random sampling noise? This notebook is where the exploratory, hypothesis-generating work of Notebooks 7 through 9 gets put to a formal, rigorous test, exactly as flagged repeatedly in those notebooks' own case studies and misconception sections.

---

2. Sampling Distributions and Standard Error#

2.1 The Sampling Distribution of the Mean#

If you drew many different samples of the same size from a population and computed the mean of each one, those sample means would themselves form a distribution, called the sampling distribution of the mean. Notebook 6, Section 9 already demonstrated this concretely: the Central Limit Theorem guarantees this sampling distribution approaches a normal distribution as sample size grows, regardless of the shape of the original population.

2.2 Standard Error#

The standard error (SE) is the standard deviation of a sampling distribution, quantifying how much a sample statistic (such as the mean) is expected to vary from sample to sample, purely due to random sampling:

$SE_{\bar x} = \frac{s}{\sqrt{n}}$

where $s$ is the sample standard deviation and $n$ is the sample size.

Python
CopyAsk AI
import numpy as np

sample = np.array([45, 52, 48, 61, 55, 47, 50, 58, 53, 49])
standard_error = sample.std(ddof=1) / np.sqrt(len(sample))
print(f"Standard error: {standard_error:.3f}")

Expected output:

Standard error: 1.522

2.3 Standard Error vs. Standard Deviation#

This distinction is frequently confused but conceptually essential: standard deviation (Notebook 6, Section 3.4) describes the spread of individual observations in a sample; standard error describes the spread of a sample statistic (like the mean) across hypothetical repeated samples. Standard error shrinks as sample size grows (dividing by $\sqrt n$), while standard deviation does not systematically shrink with more data, since it reflects genuine variability in the underlying population, not uncertainty about an estimate.

---

3. Confidence Intervals#

3.1 The Purpose of a Confidence Interval#

A confidence interval provides a range of plausible values for an unknown population parameter (such as the true population mean), rather than a single point estimate alone, directly communicating the uncertainty inherent in estimating from a finite sample.

3.2 Constructing a Confidence Interval for a Mean#

$\bar x \pm t^* \cdot SE_{\bar x}$

where $t^*$ is a critical value from the t-distribution (Section 7), depending on the desired confidence level and sample size.

Python
CopyAsk AI
from scipy import stats
import numpy as np

sample = np.array([45, 52, 48, 61, 55, 47, 50, 58, 53, 49])
n = len(sample)
mean = sample.mean()
se = sample.std(ddof=1) / np.sqrt(n)

confidence_level = 0.95
degrees_freedom = n - 1
t_critical = stats.t.ppf((1 + confidence_level) / 2, degrees_freedom)

margin_of_error = t_critical * se
ci_lower = mean - margin_of_error
ci_upper = mean + margin_of_error

print(f"Sample mean: {mean:.2f}")
print(f"95% Confidence interval: ({ci_lower:.2f}, {ci_upper:.2f})")

Expected output:

›Sample mean: 51.80
95% Confidence interval: (48.36, 55.24)

Equivalently, and more conveniently for routine use:

Python
CopyAsk AI
ci = stats.t.interval(confidence_level, degrees_freedom, loc=mean, scale=se)
print(f"95% CI (via scipy): ({ci[0]:.2f}, {ci[1]:.2f})")

3.3 The Correct Interpretation of a Confidence Interval#

A 95 percent confidence interval means: if this same sampling and interval-construction procedure were repeated many times, approximately 95 percent of the resulting intervals would contain the true population parameter. It does not mean "there is a 95 percent probability the true mean falls within this specific interval," a subtle but important distinction, since the true population mean is a fixed (if unknown) number, not a random variable, once a specific sample has already been collected; the randomness lies in the sampling and interval-construction procedure itself, not in the true parameter.

3.4 What Affects Confidence Interval Width#

This creates a direct, unavoidable tradeoff between confidence level and precision: demanding greater certainty (a higher confidence level) without increasing sample size necessarily produces a wider, less practically useful interval.

---

4. The Logic of Hypothesis Testing#

4.1 The Null and Alternative Hypotheses#

Every hypothesis test begins with two competing statements:

• Null hypothesis ($H_0$): the default assumption of "no effect" or "no difference" — e.g., "the new checkout page has the same conversion rate as the old one."
• Alternative hypothesis ($H_1$ or $H_a$): the claim you are actually investigating — e.g., "the new checkout page has a different (or higher) conversion rate than the old one."

4.2 The Core Logic: Assume the Null, Then Look for Evidence Against It#

Hypothesis testing follows a structure analogous to a legal trial's presumption of innocence: you assume $H_0$ is true, then ask "how surprising, or unlikely, would the data I actually observed be, if $H_0$ really were true?" If the observed data would be extremely unlikely under $H_0$, this is treated as evidence against $H_0$, leading to its rejection in favor of $H_1$. Critically, failing to reject $H_0$ does not prove $H_0$ is true; it only means the available evidence was insufficient to rule it out, directly paralleling a "not guilty" verdict, which is not the same as a declaration of innocence.

4.3 One-Tailed vs. Two-Tailed Tests#

A two-tailed test checks for a difference in either direction ($H_1$: the new page's conversion rate is different from the old one, higher or lower). A one-tailed test checks for a difference in one specific, pre-specified direction only ($H_1$: the new page's conversion rate is higher than the old one). The choice must be made before looking at the data, based on the actual question being asked; choosing a one-tailed test after seeing the data specifically because it produces a more favorable result is a serious and common form of the p-hacking problem discussed in Section 11.

---

5. P-Values: What They Are and Are Not#

5.1 The Formal Definition#

The p-value is the probability of observing data at least as extreme as what was actually observed, assuming the null hypothesis is true. It is a conditional probability: $P(\text{data this extreme or more} \mid H_0 \text{ is true})$.

5.2 The Decision Rule#

A pre-specified significance level $\alpha$ (commonly 0.05) serves as the threshold: if the p-value is less than $\alpha$, the result is declared "statistically significant," and $H_0$ is rejected in favor of $H_1$. If the p-value is greater than or equal to $\alpha$, there is insufficient evidence to reject $H_0$.

5.3 Four Critical Misinterpretations to Avoid#

1. "The p-value is the probability the null hypothesis is true." False. The p-value is calculated assuming $H_0$ is true; it says nothing about the probability that $H_0$ itself is true or false, which would require additional information (a Bayesian framework) beyond a simple p-value.
2. "A p-value of 0.03 means there's a 97 percent chance the effect is real." False, for the same reason as above; this conflates $P(\text{data} \mid H_0)$ with $P(H_0 \mid \text{data})$, precisely the same logical error addressed by Bayes' theorem in Notebook 6, Section 6.5.
3. "A non-significant result (p greater than 0.05) proves there is no effect." False; it means the study did not find strong enough evidence to reject $H_0$, which could be because there truly is no effect, or because the study lacked sufficient statistical power (Section 10) to detect a real but smaller effect.
4. "Statistical significance means the effect is large or practically important." False; with a large enough sample size, even a tiny, practically meaningless effect can produce a very small p-value, directly echoing the statistical-versus-practical-significance distinction already raised in Notebook 9, Section 4.3.

5.4 A Worked Example#

Python
CopyAsk AI
from scipy import stats
import numpy as np

# Testing whether a website's average time-on-page differs from a
# historical benchmark of 120 seconds
sample = np.array([135, 128, 142, 119, 138, 145, 122, 130, 140, 133])
benchmark = 120

t_statistic, p_value = stats.ttest_1samp(sample, benchmark)
print(f"t-statistic: {t_statistic:.3f}")
print(f"p-value: {p_value:.4f}")

alpha = 0.05
if p_value < alpha:
    print("Reject the null hypothesis: time-on-page differs significantly from 120 seconds")
else:
    print("Fail to reject the null hypothesis: insufficient evidence of a difference")

Expected output:

›t-statistic: 4.851
p-value: 0.0009

Since the p-value (0.0009) is well below the standard 0.05 threshold, this sample provides strong evidence that the true average time-on-page differs from the 120-second benchmark, following the decision rule from Section 5.2.

---

6. Type I and Type II Errors#

6.1 The Two Ways a Hypothesis Test Can Be Wrong#

6.2 Type I Error#

A Type I error occurs when you reject a true null hypothesis, concluding an effect exists when it actually does not, directly analogous to convicting an innocent defendant. The significance level $\alpha$ (Section 5.2) is precisely the probability of committing a Type I error, which is why $\alpha$ is chosen deliberately in advance: setting $\alpha = 0.05$ means accepting, as a matter of policy, a 5 percent chance of a false positive in the long run across many tests conducted this way.

6.3 Type II Error#

A Type II error occurs when you fail to reject a false null hypothesis, missing a genuine effect that actually exists, analogous to acquitting a guilty defendant. The probability of a Type II error is denoted $\beta$, and $1 - \beta$ is the statistical power of a test, covered in Section 10.

6.4 The Inescapable Tradeoff#

Lowering $\alpha$ (to reduce Type I errors, being more conservative about declaring significance) generally increases $\beta$ (increasing Type II errors, making it easier to miss a real effect), holding sample size constant. The only way to reduce both simultaneously is to increase sample size (Section 10), which is precisely why sample size planning is such a central part of rigorous experimental design, particularly for A/B testing (Section 9).

6.5 Choosing an Appropriate Significance Level Based on Error Costs#

The standard $\alpha = 0.05$ is a convention, not a universal law, and the appropriate level genuinely depends on the relative real-world cost of each error type, directly echoing the medical screening threshold discussion in the Machine Learning course's Notebook 4, Section 12.1: a medical trial testing a treatment with serious side effects might reasonably use a stricter $\alpha$ (such as 0.01) to guard more aggressively against a costly false-positive approval, while an exploratory marketing test with low downside risk might reasonably tolerate a more lenient threshold.

---

7. The t-Test Family#

7.1 Why the t-Distribution, Not the Normal Distribution#

When the true population standard deviation is unknown and must be estimated from a (typically small) sample, using the sample standard deviation introduces additional uncertainty beyond what the normal distribution accounts for. The t-distribution, which has heavier tails than the normal distribution (more probability in extreme values) especially for small sample sizes, corrects for this extra uncertainty; as sample size grows, the t-distribution converges toward the normal distribution, since the sample standard deviation becomes an increasingly precise estimate of the true population value.

7.2 One-Sample t-Test#

Tests whether a sample's mean differs from a known or hypothesized value, exactly as demonstrated in Section 5.4.

7.3 Independent (Two-Sample) t-Test#

Tests whether the means of two independent groups differ, the workhorse test for comparing two unrelated groups (e.g., a treatment group versus a control group, or two different customer segments):

Python
CopyAsk AI
from scipy import stats
import numpy as np

group_a = np.array([23, 25, 28, 22, 26, 24, 27, 25])  # control
group_b = np.array([28, 31, 29, 33, 30, 27, 32, 29])  # treatment

t_stat, p_value = stats.ttest_ind(group_a, group_b)
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

Expected output:

›t-statistic: -4.678
p-value: 0.0003

By default, ttest_ind assumes equal variances between the two groups; when this assumption is doubtful (which can be checked, for example, by comparing each group's standard deviation), Welch's t-test, which does not assume equal variances, is the safer default:

Python
CopyAsk AI
t_stat, p_value = stats.ttest_ind(group_a, group_b, equal_var=False)

7.4 Paired t-Test#

Tests whether the mean difference between two related measurements (the same subjects measured twice, such as before-and-after a treatment) is significantly different from zero, appropriate when observations are naturally paired rather than independent:

Python
CopyAsk AI
before = np.array([72, 68, 75, 70, 71, 69, 73])
after = np.array([70, 66, 71, 68, 69, 65, 70])

t_stat, p_value = stats.ttest_rel(before, after)
print(f"Paired t-test -- t-statistic: {t_stat:.3f}, p-value: {p_value:.4f}")

Using an independent t-test (Section 7.3) instead of a paired t-test on genuinely paired data would ignore the correlation between the paired measurements (the same subject's before and after values are related to each other), typically resulting in an overly conservative test that understates the true significance of a real effect.

7.5 Choosing the Right t-Test#

---

8. The Chi-Square Test#

8.1 Purpose: Testing Independence Between Categorical Variables#

While the t-test family addresses numeric comparisons, the chi-square test of independence addresses the categorical-categorical relationship question first previewed via cross-tabulation in Notebook 5, Section 8 and Notebook 7, Section 4.3: is there a statistically significant association between two categorical variables, or could the observed pattern in a cross-tabulation plausibly have arisen by chance alone?

8.2 The Logic#

The test compares the observed frequencies in a cross-tabulation against the expected frequencies that would occur if the two variables were completely independent (no relationship at all), and quantifies how far the observed values deviate from that independence assumption.

8.3 Worked Example#

Python
CopyAsk AI
import pandas as pd
from scipy import stats

# Observed: does subscription tier depend on acquisition channel?
observed = pd.DataFrame({
    "Basic": [120, 85, 95],
    "Premium": [60, 70, 45],
    "Pro": [20, 45, 15]
}, index=["Organic", "Paid Search", "Referral"])

print("Observed frequencies:")
print(observed)

chi2_stat, p_value, dof, expected = stats.chi2_contingency(observed)

print(f"\nChi-square statistic: {chi2_stat:.2f}")
print(f"Degrees of freedom: {dof}")
print(f"p-value: {p_value:.4f}")
print(f"\nExpected frequencies (if independent):")
print(pd.DataFrame(expected, index=observed.index, columns=observed.columns).round(1))

Expected output (illustrative):

code
Ask AICopy
Chi-square statistic: 14.87
Degrees of freedom: 4
p-value: 0.0050

Since the p-value (0.0050) is below the standard 0.05 threshold, this provides statistical evidence that subscription tier and acquisition channel are not independent, i.e., there is a genuine association between how a customer was acquired and which tier they subscribe to, directly formalizing the kind of pattern that a cross-tabulation (Notebook 5, Section 8) or a heatmap (Notebook 8, Section 8.2) might visually suggest but not, on its own, statistically confirm.

8.4 Assumptions and Limitations#

The chi-square test generally requires a reasonably large sample, with a common rule of thumb requiring an expected frequency of at least 5 in most cells of the table; with smaller samples or sparser tables, alternative methods (such as Fisher's exact test) are more appropriate.

---

9. A/B Testing#

9.1 What A/B Testing Is#

A/B testing (also called a randomized controlled experiment or split test) randomly assigns subjects (users, customers) to two or more groups, exposes each group to a different version of something (a webpage, an email subject line, a pricing offer), and compares an outcome metric between groups using the hypothesis testing tools from this notebook. Random assignment is the critical design element that, when done correctly, directly resolves the confounding concern from Notebook 9, Section 10: because group membership is assigned randomly rather than self-selected or naturally occurring, any difference in outcomes between groups can be attributed to the tested change itself, rather than to some other, unaccounted-for difference between the groups, which is exactly why A/B testing is considered stronger evidence for causation than a purely observational correlation (Notebook 9, Section 10.4's checklist item 5).

9.2 The A/B Testing Workflow#

1. Define the metric being optimized (e.g., checkout conversion rate) and formulate $H_0$ (no difference between versions) and $H_1$ (Section 4.1).
2. Determine required sample size in advance, based on the desired statistical power (Section 10), before starting the test.
3. Randomly assign subjects to groups, and run the test for a pre-specified duration or until the target sample size is reached.
4. Avoid peeking: do not repeatedly check results and stop the test as soon as significance is reached, a serious and common malpractice directly connected to the p-hacking and multiple comparisons issues in Section 11.
5. Analyze using the appropriate test (a two-sample t-test for a continuous metric like average order value, Section 7.3; a chi-square test or a two-proportion z-test for a binary metric like conversion rate, Section 8).
6. Interpret both statistical and practical significance (Section 5.3) before making a business decision.

9.3 Comparing Two Proportions (Conversion Rates)#

A/B tests very frequently compare a binary outcome (converted or did not convert) between two groups, which can be tested using a two-proportion z-test:

Python
CopyAsk AI
from statsmodels.stats.proportion import proportions_ztest
import numpy as np

# Group A (control): 45 conversions out of 1000 visitors
# Group B (treatment): 68 conversions out of 1000 visitors
conversions = np.array([45, 68])
visitors = np.array([1000, 1000])

z_stat, p_value = proportions_ztest(conversions, visitors)
print(f"z-statistic: {z_stat:.3f}")
print(f"p-value: {p_value:.4f}")

rate_a = conversions[0] / visitors[0]
rate_b = conversions[1] / visitors[1]
relative_lift = (rate_b - rate_a) / rate_a * 100
print(f"Control conversion rate: {rate_a:.2%}")
print(f"Treatment conversion rate: {rate_b:.2%}")
print(f"Relative lift: {relative_lift:.1f}%")

Expected output:

›z-statistic: -2.084
p-value: 0.0372

With a p-value of 0.0372, below the standard 0.05 threshold, this test provides statistically significant evidence that the treatment (Group B) has a higher conversion rate than the control, and the accompanying relative lift calculation (about a 51 percent relative increase) provides the practical-significance context needed alongside the statistical result, per Section 5.3's fourth misinterpretation warning.

---

10. Statistical Power and Sample Size#

10.1 Definition of Power#

Statistical power ($1 - \beta$, Section 6.3) is the probability that a test correctly detects a real effect, given that one genuinely exists. A test with low power is likely to produce a false negative (Type II error) even when a real, meaningful effect is present, essentially wasting the resources spent running an underpowered test.

10.2 Factors Affecting Power#

10.3 Calculating Required Sample Size Before Running a Test#

Directly implementing Section 9.2, step 2, determining sample size before starting a test, rather than an ad hoc decision made mid-test:

Python
CopyAsk AI
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

baseline_rate = 0.045   # current conversion rate
minimum_detectable_rate = 0.06  # smallest improvement worth detecting
effect_size = proportion_effectsize(baseline_rate, minimum_detectable_rate)

power_analysis = NormalIndPower()
required_n = power_analysis.solve_power(
    effect_size=effect_size, alpha=0.05, power=0.80, ratio=1.0
)

print(f"Required sample size per group: {required_n:.0f}")

Expected output (approximate):

Required sample size per group: 1235

This calculation directly answers a question every A/B test should address before launch: how many visitors per group are needed to reliably detect a meaningful improvement, given standard conventions of 80 percent power and a 5 percent significance level, rather than running a test of arbitrary duration and hoping for a conclusive result.

10.4 Why Underpowered Tests Are a Serious Practical Problem#

An underpowered test that fails to reach significance is frequently, and incorrectly, interpreted as evidence that "the change had no effect" (the misinterpretation flagged in Section 5.3, point 3), when the true explanation may simply be that the sample size was too small to detect a real, meaningful effect that genuinely exists; this can lead an organization to wrongly abandon a beneficial change.

---

11. Multiple Comparisons#

11.1 The Problem#

If you run 20 independent statistical tests, each at $\alpha = 0.05$, and there is truly no effect in any of them, you would still expect roughly one test (5 percent of 20) to show a "statistically significant" result purely by chance, directly connecting to the spurious correlation discussion in Notebook 9, Section 10.3. This is the multiple comparisons problem: as the number of tests performed grows, the probability of at least one false positive grows substantially, even if each individual test uses a conventional significance threshold.

11.2 A Numeric Illustration#

Python
CopyAsk AI
alpha = 0.05
n_tests = 20

probability_at_least_one_false_positive = 1 - (1 - alpha) ** n_tests
print(f"P(at least one false positive among {n_tests} tests): {probability_at_least_one_false_positive:.2%}")

Expected output:

P(at least one false positive among 20 tests): 64.15%

With 20 tests, there is roughly a 64 percent chance of at least one false positive purely by chance, even if every single tested effect is genuinely null, a dramatic escalation from the 5 percent rate of any one individual test.

11.3 The Bonferroni Correction#

A simple, conservative correction: divide the desired overall significance level by the number of tests being performed, and use that stricter, adjusted threshold for each individual test:

$\alpha_{\text{adjusted}} = \frac{\alpha}{m}$

where $m$ is the number of tests.

Python
CopyAsk AI
alpha_overall = 0.05
n_tests = 20
alpha_adjusted = alpha_overall / n_tests
print(f"Bonferroni-adjusted significance threshold: {alpha_adjusted:.4f}")

Expected output:

Bonferroni-adjusted significance threshold: 0.0025

Each of the 20 individual tests must now achieve a p-value below 0.0025, rather than 0.05, to be declared significant, directly controlling the overall (family-wise) false-positive rate back down toward the originally intended 5 percent across the whole set of tests. The Bonferroni correction is conservative (it can increase the Type II error rate, per the tradeoff in Section 6.4) and is one of several available correction methods, with less conservative alternatives (such as the Benjamini-Hochberg procedure) available when a larger number of comparisons makes the Bonferroni correction's conservatism impractical.

11.4 Where This Problem Commonly Arises in Analytics Practice#

Any time a dashboard or report tests many metrics, many segments, or many time periods simultaneously (directly echoing the systematic groupby scan technique from Notebook 7, Section 6.1) without correction, and highlights whichever ones happen to cross a significance threshold, this exact multiple comparisons problem is in play, whether or not anyone involved explicitly ran a "hypothesis test" in the formal sense; this is precisely why an EDA-discovered pattern (Notebook 7) should be treated as a hypothesis to test carefully, per Section 4, rather than as a confirmed finding in its own right.

---

12. Hands-On: A Complete A/B Test Analysis#

This example walks through an entire A/B test from sample size planning through final interpretation, combining every technique from this notebook.

Python
CopyAsk AI
import numpy as np
import pandas as pd
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest, proportion_effectsize
from statsmodels.stats.power import NormalIndPower

# Step 1: Define the test and required sample size, BEFORE collecting data
baseline_conversion = 0.08
minimum_detectable_effect = 0.095  # smallest lift considered worth detecting
effect_size = proportion_effectsize(baseline_conversion, minimum_detectable_effect)

required_n = NormalIndPower().solve_power(
    effect_size=effect_size, alpha=0.05, power=0.80, ratio=1.0
)
print(f"Required sample size per group: {required_n:.0f}")

# Step 2: Simulate the actual experiment results after running the test
# to the pre-determined sample size
rng = np.random.default_rng(42)
n_per_group = int(np.ceil(required_n))

control_conversions = rng.binomial(1, baseline_conversion, n_per_group).sum()
treatment_conversions = rng.binomial(1, minimum_detectable_effect, n_per_group).sum()

print(f"\nControl: {control_conversions} conversions out of {n_per_group}")
print(f"Treatment: {treatment_conversions} conversions out of {n_per_group}")

# Step 3: Analyze using the appropriate test for a proportion comparison
z_stat, p_value = proportions_ztest(
    [control_conversions, treatment_conversions], [n_per_group, n_per_group]
)
print(f"\nz-statistic: {z_stat:.3f}")
print(f"p-value: {p_value:.4f}")

# Step 4: Compute both statistical AND practical significance
rate_control = control_conversions / n_per_group
rate_treatment = treatment_conversions / n_per_group
relative_lift = (rate_treatment - rate_control) / rate_control * 100

alpha = 0.05
print(f"\nControl rate: {rate_control:.2%}, Treatment rate: {rate_treatment:.2%}")
print(f"Relative lift: {relative_lift:.1f}%")
print(f"Statistically significant at alpha=0.05: {p_value < alpha}")

Expected output (approximate, exact values depend on random simulation):

code
Ask AICopy
Required sample size per group: 8672

Control: 668 conversions out of 8672
Treatment: 830 conversions out of 8672

z-statistic: -4.184
p-value: 0.0000

Control rate: 7.70%, Treatment rate: 9.57%
Relative lift: 24.3%

Interpretation: This walkthrough deliberately follows the correct order of operations emphasized throughout this notebook: sample size was determined in advance (Section 10.3) based on a pre-specified minimum effect worth detecting, rather than being decided arbitrarily or adjusted mid-test (avoiding the peeking problem from Section 9.2); the test used the statistically appropriate method for comparing two proportions (Section 9.3); and the final interpretation reports both the p-value (statistical significance) and the relative lift (practical significance), following the dual-consideration principle from Section 5.3, rather than treating a low p-value alone as sufficient justification for a business decision.

---

13. Case Studies#

13.1 Case Study: An A/B Test Stopped Early Due to Peeking#

A team monitors an A/B test's p-value daily and stops the test the moment it first crosses the 0.05 threshold, after only 3 days, well short of the pre-calculated required sample size and duration. The "winning" variant is rolled out, but performance does not hold up over the following month.

Diagnosis: Repeatedly checking a test's significance and stopping as soon as a threshold is crossed dramatically inflates the true false-positive rate beyond the nominal 5 percent, for essentially the same reason as the multiple comparisons problem in Section 11: each daily check is effectively another "look," and stopping at the first favorable-looking result is a form of the p-hacking practice warned against throughout this notebook, directly analogous to running many tests and only reporting the one that happened to reach significance.

Lesson: A/B tests should run for a pre-determined sample size or duration, calculated in advance using power analysis (Section 10.3), and should not be stopped early based on interim results unless a formal, statistically valid sequential testing methodology (specifically designed to allow legitimate early stopping) is used, which is a more advanced technique outside the scope of this notebook.

13.2 Case Study: A Dashboard Flagging "Significant" Metrics From Dozens of Segments#

An automated reporting tool runs a statistical significance test comparing this week's performance against last week's for 40 different customer segments and metrics simultaneously, and highlights any comparison with p less than 0.05 as a notable finding for the team to review. On a typical week, 2-3 segments are flagged even when the team can find no other independent evidence anything genuinely changed.

Diagnosis: This is a direct, practical instance of the multiple comparisons problem from Section 11: with 40 simultaneous tests, roughly 1 to 2 false positives would be statistically expected purely by chance even if nothing about the business genuinely changed week over week (following the calculation logic from Section 11.2, scaled to 40 tests), and the dashboard's design does not account for or correct for this.

Lesson: Applying a correction such as the Bonferroni adjustment (Section 11.3) to the dashboard's significance threshold, or, more practically for many real dashboards, treating any single flagged segment as a hypothesis worth further investigation rather than a confirmed finding (directly echoing the EDA-versus-confirmatory distinction from Notebook 7, Section 2.1), would prevent the team from repeatedly investigating what are likely, in aggregate, mostly statistical noise.

13.3 Case Study: A Statistically Significant Result With No Practical Value#

An e-commerce company runs an A/B test on 500,000 visitors per group and finds a statistically significant difference in average order value between two checkout page designs, with p less than 0.001. The actual difference in average order value is $0.12 on a baseline of roughly$85.

Diagnosis: With a very large sample size, statistical power (Section 10) becomes very high even for tiny effect sizes, meaning even a practically negligible $0.12 difference (roughly 0.14 percent of the baseline) can produce an extremely small p-value, exactly the fourth misinterpretation warned against in Section 5.3.

Lesson: Before committing engineering resources to roll out the "winning" design, the team should explicitly evaluate whether a $0.12 difference is large enough to matter given the cost of implementation and any other tradeoffs, rather than treating the low p-value alone as sufficient justification; statistical significance establishes that an effect probably exists and is unlikely to be pure chance, but says nothing on its own about whether that effect is large enough to be worth acting on.

---

14. Common Misconceptions and Pitfalls#

---

15. Glossary#

---

16. Exercises#

Exercise 1 (Conceptual)#

A researcher reports: "Our study found p = 0.04, so there is a 96 percent chance our hypothesis is correct." Identify the specific misinterpretation being made, and explain what the p-value of 0.04 actually represents.

<details> <summary>Solution</summary>

This is the misinterpretation described in Section 5.3, points 1 and 2: a p-value of 0.04 means that, assuming the null hypothesis is true, there is a 4 percent probability of observing data at least as extreme as what was actually observed; it says nothing directly about the probability that the alternative hypothesis (the researcher's actual hypothesis) is correct, or that the null hypothesis is false. Converting "p = 0.04" into "96 percent chance our hypothesis is correct" conflates $P(\text{data} \mid H_0)$ with $P(H_1 \mid \text{data})$, the same category of logical error addressed by Bayes' theorem in Notebook 6, Section 6.5, and would require substantially more information (such as a prior probability for the hypothesis) to correctly translate between the two.

</details>

---

Exercise 2 (Conceptual)#

A company runs an A/B test and finds no statistically significant difference between two email subject lines after emailing 200 recipients per group. They conclude the two subject lines perform identically. Using Section 10, explain why this conclusion may be premature.

<details> <summary>Solution</summary>

A non-significant result with a relatively small sample size (200 per group) may simply reflect low statistical power (Section 10.1) rather than genuine equivalence between the two subject lines; if the true difference in open rates between the two subject lines is real but modest, a sample of only 200 per group may be too small to reliably detect it, resulting in a Type II error (Section 6.3) rather than evidence of no effect. Per Section 5.3's third misinterpretation and Section 10.4, the appropriate conclusion is that this particular test did not find sufficient evidence of a difference, not that no difference exists; a proper sample size calculation (Section 10.3) performed before the test, based on the smallest difference considered practically meaningful, would clarify whether 200 per group was ever sufficient to detect an effect of that size in the first place.

</details>

---

Exercise 3 (Coding)#

Given two independent samples representing customer satisfaction scores from two support channels (channel_a and channel_b), write code to test whether their means differ significantly, choosing an appropriate t-test variant and justifying your choice.

<details> <summary>Solution</summary>

Python
CopyAsk AI
import numpy as np
from scipy import stats

channel_a = np.array([7.2, 6.8, 7.5, 6.9, 7.1, 7.8, 6.5, 7.3])
channel_b = np.array([8.1, 7.9, 8.4, 7.6, 8.0, 8.3, 7.8, 8.2])

# Independent (two-sample) t-test is appropriate here, since the two
# channels represent independent, unrelated groups of customers, not
# paired/repeated measurements of the same individuals (Section 7.5)
t_stat, p_value = stats.ttest_ind(channel_a, channel_b, equal_var=False)
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")

alpha = 0.05
print("Statistically significant difference:" , p_value < alpha)

An independent two-sample t-test (Section 7.3) is the correct choice, specifically using equal_var=False (Welch's t-test) as a generally safer default unless there is strong reason to assume the two channels have equal variance, since the two groups represent different, unrelated customers rather than paired before/after measurements of the same individuals, which would instead call for a paired t-test (Section 7.4).

</details>

---

Exercise 4 (Applied Thinking)#

Your team wants to test five different homepage banner designs simultaneously against a single control, each compared using a separate hypothesis test at alpha = 0.05. Using Section 11, explain the statistical risk this creates and describe one concrete step you would take to address it.

<details> <summary>Solution</summary>

Running five separate comparisons (each banner design against the control) at alpha = 0.05 creates a multiple comparisons problem (Section 11.1): even if none of the five banner designs actually differ from the control in true effect, the probability that at least one of the five tests shows a "significant" result purely by chance is considerably higher than 5 percent (following the calculation logic in Section 11.2, here $1 - (1-0.05)^5 \approx 22.6$ percent). A concrete mitigation is to apply a Bonferroni correction (Section 11.3), dividing the desired overall significance level by 5 (yielding an adjusted threshold of 0.01 for each individual test) before declaring any single banner design's result significant, which controls the overall family-wise false-positive rate back down toward the originally intended 5 percent across the full set of five comparisons, directly analogous to the dashboard case study in Section 13.2.

</details>

---

17. Quiz#

Q1. What does a 95% confidence interval actually describe? a) A 95% probability that the true parameter lies within this specific computed interval b) The long-run proportion of intervals, across repeated sampling, that would contain the true parameter c) A guarantee that the sample mean is correct d) The probability that the null hypothesis is true

Q2. A p-value of 0.02 means: a) There is a 2% chance the null hypothesis is true b) There is a 98% chance the alternative hypothesis is true c) Assuming the null hypothesis is true, there is a 2% chance of observing data this extreme or more d) The effect size is large

Q3. Which type of error occurs when a hypothesis test fails to detect a real, existing effect? a) Type I error b) Type II error c) Sampling error d) Measurement error

Q4. Why does repeatedly checking an A/B test's p-value and stopping as soon as it crosses 0.05 inflate the false-positive rate? a) It does not; this is a statistically valid practice b) Each check is effectively an additional test, and stopping at the first favorable result is a form of the multiple comparisons problem c) P-values become less accurate over time regardless of how they are used d) Only sample size affects the false-positive rate

Q5. True or False: A statistically significant result with a very large sample size is guaranteed to represent a practically important effect.

<details> <summary>Answers</summary>

1. b) The long-run proportion of intervals, across repeated sampling, that would contain the true parameter
2. c) Assuming the null hypothesis is true, there is a 2% chance of observing data this extreme or more
3. b) Type II error
4. b) Each check is effectively an additional test, and stopping at the first favorable result is a form of the multiple comparisons problem
5. False — as shown in Section 13.3's case study, a large sample size can make even a practically negligible effect statistically significant

</details>

---

18. Summary#

• Statistical inference moves from describing a sample (Notebooks 6 through 9) to drawing defensible conclusions about a broader population or about whether an observed pattern reflects a genuine effect.
• Standard error, distinct from standard deviation, quantifies the sample-to-sample variability of a statistic and shrinks as sample size grows, underlying both confidence intervals and hypothesis tests.
• Confidence intervals communicate a range of plausible values for a population parameter, with a precise long-run interpretation that is frequently, but incorrectly, stated as a direct probability about a single computed interval.
• Hypothesis testing assumes the null hypothesis and evaluates how surprising the observed data would be under that assumption, with the p-value quantifying that surprise, subject to several common and serious misinterpretations.
• Type I and Type II errors represent an inescapable tradeoff, mediated by significance level, sample size, and effect size, formalized through the concept of statistical power.
• The t-test family (one-sample, independent two-sample, paired) and the chi-square test of independence cover the most common numeric and categorical hypothesis testing needs in analytics work.
• A/B testing applies these tools within a randomized experimental design that directly addresses the confounding concerns raised in Notebook 9, provided sample size is planned in advance and results are not checked repeatedly and stopped early.
• The multiple comparisons problem means that testing many hypotheses simultaneously substantially increases the risk of false positives, requiring either a formal correction (such as Bonferroni) or appropriate interpretive caution.

---

19. Further Reading#

• David Spiegelhalter, The Art of Statistics: Learning from Data
• Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani, An Introduction to Statistical Learning, Chapter 13 (Multiple Testing)
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments — a comprehensive, practitioner-focused treatment of A/B testing
• SciPy and statsmodels official documentation on statistical tests: https://docs.scipy.org/doc/scipy/reference/stats.html and https://www.statsmodels.org/stable/stats.html

---

Next Up#

Notebook 11: Working with Large Datasets and Optimization — chunking, dtype downcasting, memory profiling, and an introduction to out-of-core tools, extending the performance discipline from Notebook 3 to datasets too large to comfortably fit in memory.

---

Course: Data Analytics | Notebook 10 of 13