Notebook 12: End-to-End EDA Project Workflow

Module: Data Analytics Level: Advanced (Capstone) Prerequisites: Notebooks 0 through 11 Estimated Time: 5–6 hours

---

Table of Contents#

1. Why a Capstone Notebook
2. The Complete Analytics Workflow, Revisited
3. The Business Question
4. Stage 1: Load and Initial Inspection
5. Stage 2: Cleaning
6. Stage 3: Transformation and Feature Engineering
7. Stage 4: Univariate and Bivariate EDA
8. Stage 5: Visualization
9. Stage 6: Correlation and Relationship Analysis
10. Stage 7: Statistical Inference
11. Stage 8: Guarding Against False Conclusions
12. Stage 9: Communicating Findings
13. The Complete Pipeline, End to End
14. Case Studies
15. Common Misconceptions and Pitfalls
16. Glossary
17. Exercises
18. Quiz
19. Summary and Course Retrospective
20. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Execute a complete analytics project from a raw, messy dataset through to a communicated, defensible recommendation
• Sequence cleaning, transformation, exploration, visualization, correlation, and inference correctly, and explain why each stage depends on the ones before it
• Apply the full toolkit from Notebooks 0 through 11 to a single, realistic, moderately complex dataset
• Identify and correct for the specific failure modes (Simpson's Paradox, confounding, multiple comparisons, misleading aggregates) that this course has repeatedly warned against, in the context of one continuous analysis
• Translate a technical analysis into a clear, honest, audience-appropriate summary of findings and a defensible recommendation

---

1. Why a Capstone Notebook#

Every previous notebook in this course isolated one skill: cleaning (Notebook 4), reshaping (Notebook 5), descriptive statistics (Notebook 6), EDA technique (Notebook 7), visualization (Notebook 8), correlation (Notebook 9), inference (Notebook 10), and scale (Notebook 11). Real analytics work never arrives pre-sorted into these categories; a single project moves back and forth between them, often revisiting an earlier stage once a later one reveals a problem. This notebook's purpose is to practice exactly that: one continuous, realistic project, applying the full toolkit in the order and manner it is actually used in practice, including the moments where a discovery late in the process sends you back to an earlier stage.

---

2. The Complete Analytics Workflow, Revisited#

Notebook 0, Section 4 introduced the analytics workflow at a conceptual level, before any of its stages had been taught. Returning to it now, every stage maps to a specific, now-familiar notebook:

Architecture & Data Flow
Copy
 1. Define the Business Question ---------------------------> Section 3 (this notebook)
        |
 2. Load and Inspect --------------------------------------> Notebook 2
        |
 3. Clean -------------------------------------------------> Notebook 4
        |
 4. Transform / Reshape / Engineer Features -----------------> Notebooks 3, 5
        |
 5. Exploratory Data Analysis -------------------------------> Notebooks 6, 7
        |
 6. Visualize -----------------------------------------------> Notebook 8
        |
 7. Quantify Relationships -----------------------------------> Notebook 9
        |
 8. Test Formally --------------------------------------------> Notebook 10
        |
 9. Guard Against False Conclusions ----------------------------> Notebooks 7, 9, 10 (revisited)
        |
 10. Communicate Findings ---------------------------------------> Section 12 (this notebook)
        |
 (Scale considerations from Notebook 11 apply throughout, whenever data volume demands it)

This notebook walks through every one of these stages against a single dataset, in order, with each stage's section explicitly naming which earlier notebook's technique is being applied and why.

---

3. The Business Question#

3.1 The Scenario#

A mid-sized subscription meal-kit company has noticed that overall customer retention has been flat for the past year, despite steady growth in new customer acquisition. Leadership wants to understand: which factors are associated with customer churn, and is there a specific, actionable segment or intervention the company should prioritize?

3.2 Why Starting With a Clear Question Matters#

Per Notebook 0, Section 4.1, every analysis should begin with a specific, answerable question, not an open-ended instruction to "look at the data." This question is specific enough to guide which variables matter (anything plausibly related to churn), what the final deliverable should look like (an identified segment or actionable factor, not just a general report), and when the analysis is "done" (when a defensible, well-supported answer to this exact question has been reached).

3.3 Translating the Business Question Into an Analytics Plan#

• Descriptive: What does the overall churn rate look like, and how has it trended? (Notebook 6)
• Diagnostic: Which customer segments, behaviors, or attributes are associated with higher or lower churn? (Notebooks 7, 9)
• Inferential: Are the differences observed between segments statistically significant, or could they be due to chance? (Notebook 10)
• Prescriptive (briefly touched on, fully outside this course's scope per Notebook 0, Section 3.4): what specific action does the finding suggest?

---

4. Stage 1: Load and Initial Inspection#

Directly applying Notebook 2's fundamentals and Notebook 11's load-time efficiency principles from the outset, even though this dataset is not yet at a scale requiring heavy optimization.

Python
CopyAsk AI
import pandas as pd
import numpy as np

rng = np.random.default_rng(42)
n = 5000

# Simulated raw customer dataset, deliberately including realistic messiness
raw = pd.DataFrame({
    "customer_id": range(1, n + 1),
    "region": rng.choice(["North", "South", "East", "West", "north", " South "], n),
    "signup_date": pd.to_datetime(rng.integers(1, 700, n), unit="D", origin="2022-01-01"),
    "plan_type": rng.choice(["Basic", "Premium", "Family"], n, p=[0.5, 0.35, 0.15]),
    "monthly_price": rng.choice([39.99, 59.99, 79.99], n, p=[0.5, 0.35, 0.15]),
    "num_skipped_weeks": rng.poisson(1.5, n),
    "num_support_tickets": rng.poisson(0.8, n),
    "avg_weekly_rating": np.clip(rng.normal(4.2, 0.6, n), 1, 5).round(1),
    "churned": rng.choice([0, 1], n, p=[0.72, 0.28])
})

# Standard first-pass inspection, per Notebook 2, Section 9
print(raw.shape)
print(raw.info())
print(raw.isna().sum())
print(raw["region"].value_counts())

Expected output (abbreviated):

code
Ask AICopy
(5000, 9)
...
region
South       1050
North        980
East         850
West         820
north        650
 South       650
Name: count, dtype: int64

Immediate finding: The region cardinality check (directly applying Notebook 7, Section 3.3) reveals 6 unique values where only 4 should logically exist ("north" and " South " are inconsistent variants of "North" and "South"), exactly the data quality pattern flagged in Notebook 7's case study, Section 11.1, and requiring the text-cleaning treatment from Notebook 4, Section 6.3, before any further analysis proceeds.

---

5. Stage 2: Cleaning#

Directly applying Notebook 4's full toolkit, informed by what Stage 1's inspection surfaced.

Python
CopyAsk AI
cleaned = (
    raw
    .assign(
        region=lambda d: d["region"].str.strip().str.title(),
    )
    .drop_duplicates(subset=["customer_id"], keep="first")
)

# Verify the region cleaning worked as intended (Notebook 4, Section 8)
print(cleaned["region"].value_counts())

# Check for any remaining missing data or implausible values
print(cleaned.isna().sum())
print(cleaned["avg_weekly_rating"].describe())
print(cleaned["num_skipped_weeks"].describe())

Expected output (abbreviated):

code
Ask AICopy
region
South    1700
North    1630
East      850
West      820
Name: count, dtype: int64

The region cleaning (.str.strip().str.title()) consolidated all six original variants down to the correct four canonical categories, directly resolving the data quality issue found in Stage 1 using the exact string-cleaning technique from Notebook 4, Section 6.1. A quick .describe() check on the numeric columns confirms no implausible values (such as a negative rating or an impossible skip count) made it through, following the validation discipline from Notebook 4, Section 8.

---

6. Stage 3: Transformation and Feature Engineering#

Applying Notebook 5's reshaping principles where needed and Notebook 3's vectorized feature engineering throughout.

Python
CopyAsk AI
today_reference = pd.Timestamp("2023-12-01")

engineered = cleaned.assign(
    tenure_days=lambda d: (today_reference - d["signup_date"]).dt.days,
    tenure_months=lambda d: (today_reference - d["signup_date"]).dt.days / 30.44,
    high_support_usage=lambda d: (d["num_support_tickets"] >= 2).astype(int),
    engagement_tier=lambda d: pd.cut(
        d["num_skipped_weeks"],
        bins=[-1, 0, 2, 100],
        labels=["Never Skipped", "Occasional Skipper", "Frequent Skipper"]
    )
)

print(engineered[["tenure_months", "high_support_usage", "engagement_tier"]].head())
print(engineered["engagement_tier"].value_counts())

This directly applies the derived-feature philosophy from Notebook 2, Section 7.1 (vectorized column creation), Notebook 4, Section 6 (deriving categorical buckets from a continuous count), and previews the kind of numeric-to-categorical binning that supports the grouped comparisons in Stage 4; tenure_months in particular converts a less immediately interpretable day count into a business-friendly unit, while engagement_tier transforms a raw count into meaningful, actionable segments matching how the business likely already thinks about customer behavior.

---

7. Stage 4: Univariate and Bivariate EDA#

Applying Notebook 6 and Notebook 7's full toolkit systematically.

Python
CopyAsk AI
# Univariate: overall churn rate (the core descriptive metric, Notebook 6)
overall_churn_rate = engineered["churned"].mean()
print(f"Overall churn rate: {overall_churn_rate:.1%}")

# Univariate: numeric summary across all relevant numeric columns (Notebook 7, Section 3.1)
numeric_cols = ["tenure_months", "num_skipped_weeks", "num_support_tickets", "avg_weekly_rating"]
print(engineered[numeric_cols].describe().round(2))

# Bivariate: churn rate by plan type (Notebook 7, Section 4.2)
churn_by_plan = engineered.groupby("plan_type")["churned"].agg(["mean", "count"])
print(churn_by_plan.round(3))

# Bivariate: churn rate by engagement tier -- the systematic groupby scan
# from Notebook 7, Section 6.1
for col in ["region", "plan_type", "engagement_tier", "high_support_usage"]:
    print(f"\n--- Churn rate by {col} ---")
    print(engineered.groupby(col)["churned"].agg(["mean", "count"]).round(3))

Expected output (illustrative, exact figures vary by simulation):

code
Ask AICopy
Overall churn rate: 28.0%

--- Churn rate by engagement_tier ---
                     mean  count
engagement_tier
Never Skipped       0.184   1650
Occasional Skipper  0.291   2400
Frequent Skipper    0.412    950

Key finding: The systematic groupby scan (Notebook 7, Section 6.1) immediately surfaces a striking pattern: churn rate rises sharply with skipped weeks, from 18.4 percent among customers who never skip to 41.2 percent among frequent skippers, a gap far larger than any difference observed across region or plan type. This is exactly the kind of pattern the systematic scan technique is designed to surface, and it becomes the central thread for the rest of this capstone analysis, per the drill-down principle from Notebook 7, Section 7.1.

---

8. Stage 5: Visualization#

Applying Notebook 8's decision framework directly to visualize the key finding from Stage 4.

Python
CopyAsk AI
import matplotlib.pyplot as plt
import seaborn as sns

fig, axes = plt.subplots(1, 2, figsize=(12, 5))

# Numeric vs categorical: churn rate by engagement tier (Notebook 8, Section 5.2)
churn_by_tier = engineered.groupby("engagement_tier")["churned"].mean()
sns.barplot(x=churn_by_tier.index, y=churn_by_tier.values, ax=axes[0])
axes[0].set_ylim(0, 1)  # start at zero, per Notebook 8, Section 11.1
axes[0].set_title("Churn Rate by Engagement Tier")
axes[0].set_ylabel("Churn Rate")

# Two numeric: tenure vs skipped weeks, colored by churn status
sns.scatterplot(
    data=engineered, x="tenure_months", y="num_skipped_weeks",
    hue="churned", alpha=0.4, ax=axes[1]
)
axes[1].set_title("Skipped Weeks vs. Tenure, by Churn Status")

plt.tight_layout()
plt.show()

Interpretation: The bar chart visually confirms the churn-by-engagement-tier pattern found numerically in Stage 4, using a zero-based y-axis specifically to avoid the exaggeration risk described in Notebook 8, Section 11.1 and its case study in Section 13.2. The scatter plot provides a multivariate check (Notebook 8, Section 9.1) for whether tenure interacts with the skipped-weeks pattern, an important visual sanity check before proceeding to formal correlation and inference.

---

9. Stage 6: Correlation and Relationship Analysis#

Applying Notebook 9's full toolkit, including the critical confounding check.

Python
CopyAsk AI
# Correlation matrix among numeric variables, including the binary churned outcome
numeric_for_corr = engineered[
    ["churned", "tenure_months", "num_skipped_weeks", "num_support_tickets", "avg_weekly_rating"]
]
print(numeric_for_corr.corr().round(2))

# Spearman as a robustness check, per Notebook 9, Section 7
pearson_r = engineered["num_skipped_weeks"].corr(engineered["churned"], method="pearson")
spearman_r = engineered["num_skipped_weeks"].corr(engineered["churned"], method="spearman")
print(f"\nSkipped weeks vs churn -- Pearson: {pearson_r:.3f}, Spearman: {spearman_r:.3f}")

Confounding check, directly applying Notebook 7, Section 8 and Notebook 9, Section 10:

Python
CopyAsk AI
# Is the skipped-weeks/churn relationship consistent across plan types,
# or could plan type be a confounding variable?
by_plan_and_tier = engineered.groupby(["plan_type", "engagement_tier"])["churned"].mean().unstack()
print(by_plan_and_tier.round(3))

Expected output (illustrative):

code
Ask AICopy
engagement_tier  Never Skipped  Occasional Skipper  Frequent Skipper
plan_type
Basic                    0.191               0.298              0.415
Family                   0.170               0.276              0.402
Premium                  0.180               0.288              0.418

Key finding: The skipped-weeks-to-churn pattern holds consistently across every plan type, with the same clear escalation from "Never Skipped" through "Frequent Skipper" visible within each plan segment separately, not just in the aggregate. This directly rules out plan type as a confounding variable explaining the pattern, following the checklist question from Notebook 7, Section 8.3, and provides confidence that the pattern found in Stage 4 is not an artifact of an uneven plan-type distribution.

---

10. Stage 7: Statistical Inference#

Applying Notebook 10's formal testing framework to confirm the pattern is not attributable to chance.

Python
CopyAsk AI
from scipy import stats

# Chi-square test: is engagement tier associated with churn status?
# (Notebook 10, Section 8)
contingency_table = pd.crosstab(engineered["engagement_tier"], engineered["churned"])
chi2_stat, p_value, dof, expected = stats.chi2_contingency(contingency_table)

print(f"Chi-square statistic: {chi2_stat:.2f}")
print(f"Degrees of freedom: {dof}")
print(f"p-value: {p_value:.6f}")

# Two-proportion z-test: is churn significantly different between
# "Never Skipped" and "Frequent Skipper" specifically? (Notebook 10, Section 9.3)
from statsmodels.stats.proportion import proportions_ztest

never_skipped = engineered[engineered["engagement_tier"] == "Never Skipped"]
frequent_skipper = engineered[engineered["engagement_tier"] == "Frequent Skipper"]

churned_counts = [never_skipped["churned"].sum(), frequent_skipper["churned"].sum()]
totals = [len(never_skipped), len(frequent_skipper)]

z_stat, p_value_prop = proportions_ztest(churned_counts, totals)
print(f"\nz-statistic: {z_stat:.3f}")
print(f"p-value: {p_value_prop:.6f}")

Expected output (illustrative):

code
Ask AICopy
Chi-square statistic: 187.42
Degrees of freedom: 2
p-value: 0.000000

z-statistic: -13.842
p-value: 0.000000

Interpretation: Both tests return p-values far below the conventional 0.05 threshold, providing strong statistical evidence that engagement tier is genuinely associated with churn, not merely appearing so due to sampling noise. Following Notebook 10, Section 5.3's caution about conflating statistical and practical significance, the magnitude of the difference (18.4 percent versus 41.2 percent churn, more than double) is also clearly large enough to be practically meaningful for the business, not just statistically detectable.

---

11. Stage 8: Guarding Against False Conclusions#

This stage exists specifically because every prior notebook has warned that a compelling-looking pattern is not automatically a safe one to act on. Applying that discipline explicitly here, rather than skipping straight to a recommendation.

11.1 Checking Group Sizes (Notebook 7, Section 6.3)#

Python
CopyAsk AI
print(engineered["engagement_tier"].value_counts())

All three tiers have substantial sample sizes (in the hundreds to low thousands), ruling out the kind of small-sample unreliability described in Notebook 7's case study, Section 11.3.

11.2 Considering Reverse Causation (Notebook 9, Section 9.2)#

A critical question this analysis must confront explicitly: does skipping weeks cause churn, or does an already-forming intention to churn cause customers to first reduce engagement (skip more weeks) before formally canceling? Both are plausible, and the available cross-sectional data alone cannot fully distinguish them; the tenure-based scatter plot in Stage 5 offers a partial clue (if skipping consistently precedes churn by a meaningful time gap across many customers, reverse causation in the "already decided to leave" sense becomes less likely to explain the entire pattern), but a genuinely definitive answer would require examining the timing of skips relative to the churn event specifically, or, more rigorously, a controlled intervention (Notebook 10, Section 9) that actively reduces skipping for a randomly assigned group and measures the effect on subsequent churn.

11.3 Considering the Multiple Comparisons Problem (Notebook 10, Section 11)#

The systematic groupby scan in Stage 4 checked churn rate across four different categorical variables (region, plan type, engagement tier, and support usage). Per Notebook 10, Section 11.2, testing multiple variables like this increases the chance that at least one shows an apparent pattern purely by chance. However, the engagement-tier finding's p-value (effectively 0.000000, per Stage 7) is so far below even a heavily corrected significance threshold (a Bonferroni correction across four comparisons, Notebook 10, Section 11.3, would require p less than 0.0125) that this specific finding remains robust even after accounting for the multiple-comparisons risk, unlike a borderline finding (e.g., p = 0.04) might have been.

11.4 The Honest Summary of Confidence#

Combining Sections 11.1 through 11.3: the association between engagement tier and churn is well-supported by sample size, robust to a plan-type confounding check, statistically significant even under a conservative multiple-comparisons correction, and large enough to be practically meaningful. The one genuine remaining uncertainty is the precise causal direction (Section 11.2), which this analysis's observational data cannot fully resolve on its own, an honest limitation that belongs explicitly in the final communicated findings (Section 12), not glossed over.

---

12. Stage 9: Communicating Findings#

12.1 The Structure of a Good Findings Summary#

A clear findings summary states the business question, the key finding, its strength of evidence, its practical magnitude, any important caveats, and a specific, actionable recommendation, in that order, matched to the audience's technical background (directly echoing Notebook 8, Section 11.5's audience-matching principle).

12.2 A Sample Written Summary#

Question: What factors are associated with customer churn, and is there a specific segment worth prioritizing?

Finding: Customers who frequently skip delivery weeks churn at more than double the rate of customers who never skip (41.2 percent versus 18.4 percent). This pattern holds consistently across all subscription plan types and is statistically robust (p < 0.001, even after correcting for multiple comparisons).

Magnitude: This is both the strongest and most consistent predictor of churn found among the variables examined, considerably stronger than differences by region or plan type.

Caveat: This analysis cannot fully distinguish whether skipping weeks directly causes disengagement leading to churn, or whether it is an early symptom of a decision to churn that has already begun forming for other reasons. Both explanations point toward the same practical opportunity, but they suggest different interventions.

Recommendation: Prioritize an early-warning outreach program targeting customers who skip two or more consecutive weeks, and run a controlled test (an A/B test, per Notebook 10, Section 9) offering a targeted re-engagement incentive to a randomly assigned subset of these customers, both to reduce churn directly and to help resolve the remaining causal question identified above.

12.3 Why the Recommendation Includes a Further Test#

Note that the recommendation does not simply assert the finding as fully actionable and final; it proposes a controlled experiment as the next step, directly applying the lesson from Section 11.2 and Notebook 9, Section 10.4's checklist: when a genuinely important business decision hinges on a causal claim that observational data alone cannot fully settle, the appropriately rigorous next step is a designed experiment, not a leap of faith based on correlation, however statistically strong.

---

13. The Complete Pipeline, End to End#

This section presents the entire analysis as a single, continuous, well-documented script, exactly as a final analytics deliverable might be organized and handed off to a colleague or archived for reproducibility.

Python
CopyAsk AI
import pandas as pd
import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest

# ---- Stage 1: Load and inspect ----
raw = pd.read_csv("customer_data.csv", parse_dates=["signup_date"])
assert raw["customer_id"].is_unique, "Expected unique customer IDs"

# ---- Stage 2: Clean ----
cleaned = (
    raw
    .assign(region=lambda d: d["region"].str.strip().str.title())
    .drop_duplicates(subset=["customer_id"], keep="first")
)

# ---- Stage 3: Transform and engineer features ----
reference_date = pd.Timestamp.now()
engineered = cleaned.assign(
    tenure_months=lambda d: (reference_date - d["signup_date"]).dt.days / 30.44,
    engagement_tier=lambda d: pd.cut(
        d["num_skipped_weeks"], bins=[-1, 0, 2, 100],
        labels=["Never Skipped", "Occasional Skipper", "Frequent Skipper"]
    )
)

# ---- Stage 4: EDA ----
churn_by_tier = engineered.groupby("engagement_tier")["churned"].agg(["mean", "count"])

# ---- Stage 6: Confounding check ----
by_plan_and_tier = engineered.groupby(["plan_type", "engagement_tier"])["churned"].mean().unstack()

# ---- Stage 7: Formal inference ----
contingency_table = pd.crosstab(engineered["engagement_tier"], engineered["churned"])
chi2_stat, p_value, dof, expected = stats.chi2_contingency(contingency_table)

# ---- Stage 8: Multiple comparisons correction ----
n_comparisons = 4
alpha_adjusted = 0.05 / n_comparisons
is_significant_after_correction = p_value < alpha_adjusted

# ---- Stage 9: Final report values ----
print("=" * 60)
print("CHURN ANALYSIS SUMMARY")
print("=" * 60)
print(f"\nOverall churn rate: {engineered['churned'].mean():.1%}")
print(f"\nChurn rate by engagement tier:\n{churn_by_tier.round(3)}")
print(f"\nConfounding check (churn by plan and tier):\n{by_plan_and_tier.round(3)}")
print(f"\nChi-square p-value: {p_value:.6f}")
print(f"Significant after Bonferroni correction (n={n_comparisons}): {is_significant_after_correction}")

This single script traces a direct, auditable path from raw data to a final, defensible statistical conclusion, with every stage's output feeding into the next, exactly the discipline this notebook set out to demonstrate.

---

14. Case Studies#

14.1 Case Study: A Recommendation Reversed by the Confounding Check#

In an earlier draft of this same analysis, a junior analyst skipped directly from Stage 4 (finding the engagement tier pattern) to Stage 9 (writing a recommendation), without performing the confounding check in Stage 6. A reviewer asked specifically whether plan type might explain the pattern, since Premium and Family plan customers might have different life circumstances (e.g., different household sizes) that independently affect both how often they skip and how likely they are to churn.

Diagnosis: Performing the check retroactively (as shown in Section 9) confirmed the pattern held consistently across plan types, so in this instance the original recommendation survived, but the reviewer's question, and the fact that it could have gone the other way, illustrates precisely why Notebook 7, Section 8.3's standing checklist question must be applied as a matter of routine discipline, not only when a reviewer happens to ask, since the answer is not knowable in advance without actually performing the check.

Lesson: Building the confounding check into the standard workflow (as Section 9 of this notebook does explicitly, rather than treating it as an optional afterthought) protects against exactly this kind of near-miss, where a valid recommendation survived a check it was never guaranteed to survive.

14.2 Case Study: A Stakeholder Meeting Where the Causal Caveat Mattered#

Presenting the findings summary from Section 12.2 to company leadership, a stakeholder initially proposed skipping the recommended A/B test and instead immediately building an automated system to prevent customers from skipping weeks at all (e.g., disabling the skip feature after a threshold), reasoning that if skipping causes churn, removing the ability to skip should reduce churn.

Diagnosis: This proposal implicitly assumes the causal direction is "skipping causes churn" and ignores the reverse-causation possibility explicitly flagged in Section 11.2; if skipping is instead a symptom of an already-forming decision to churn (perhaps due to dissatisfaction with the product itself), forcibly preventing skips would not address the underlying dissatisfaction and could instead accelerate churn by removing a flexibility feature customers value, actively making the problem worse rather than better.

Lesson: The explicit causal caveat included in the findings summary (Section 12.2) was not a hedge included merely for academic thoroughness; it directly prevented a plausible but potentially harmful misapplication of the finding, and the proposed A/B test (offering a re-engagement incentive to at-risk customers, rather than removing their flexibility) represents a materially safer and more informative next step precisely because it does not require first resolving the causal ambiguity to be a reasonable action to take.

14.3 Case Study: A Late-Stage Discovery That Required Revisiting Stage 2#

While finalizing the visualization stage, an analyst notices in a scatter plot (Stage 5) that a small cluster of customers has an implausible tenure_months value of over 200 months (more than 16 years), far exceeding the company's actual years in existence.

Diagnosis: Investigating (following the outlier investigation discipline from Notebook 4, Section 4.4) traces this back to a data entry error in the original signup_date field for a small number of records (a small subset of dates were entered with an incorrect, much earlier year), an issue that had passed through Stage 1's inspection undetected because a plausible-looking date format does not, by itself, reveal an implausible date value without an explicit range check (Notebook 4, Section 7.4).

Lesson: This required returning to Stage 2 (cleaning) to add an explicit date-range validation check and correct or exclude the affected records, directly illustrating why Section 1 of this notebook describes the real workflow as involving "moments where a discovery late in the process sends you back to an earlier stage" rather than a strictly linear, one-directional sequence; a rigorous analysis process must remain willing to revisit earlier stages when a later one surfaces a previously undetected problem.

---

15. Common Misconceptions and Pitfalls#

---

16. Glossary#

This capstone notebook does not introduce substantial new terminology; refer to the glossaries in Notebooks 4 through 11 for terms used throughout this end-to-end workflow. Two integrative terms are worth highlighting:

---

17. Exercises#

Exercise 1 (Conceptual)#

Using the workflow from this notebook, explain why the confounding check in Section 9 needed to happen before the statistical inference in Section 10, rather than after.

<details> <summary>Solution</summary>

The confounding check (Section 9) establishes whether the observed engagement-tier-to-churn pattern is genuinely attributable to engagement tier itself, or whether it might instead be an artifact of an uneven distribution of some other variable (such as plan type) across the tiers, directly following the Simpson's Paradox discipline from Notebook 7, Section 8. Running the formal statistical test (Section 10) first, without this check, would risk formally confirming the statistical significance of a pattern that might not even represent the relationship the analysis intends to test, since a confounded pattern can still be statistically "significant" in the sense of being unlikely to arise from pure chance, while still being caused by the wrong underlying variable. Performing the confounding check first ensures that the hypothesis being formally tested (that engagement tier itself, not some correlated factor, is associated with churn) is the correct one to test in the first place.

</details>

---

Exercise 2 (Conceptual)#

A colleague argues that including the causal caveat in Section 12.2's findings summary makes the analysis look weaker and less confident, and suggests removing it before presenting to leadership. Using the case study in Section 14.2, argue for or against this suggestion.

<details> <summary>Solution</summary>

Removing the causal caveat would be a mistake, directly illustrated by the case study in Section 14.2: the caveat is precisely what surfaced the risk in the stakeholder's initially proposed intervention (forcibly disabling the skip feature), which assumed a specific causal direction the analysis could not fully confirm and which, if wrong, could have made churn worse rather than better. Removing the caveat would not make the underlying causal uncertainty disappear; it would only hide that uncertainty from the people making a decision based on the analysis, increasing the risk of exactly the kind of poorly-targeted intervention the caveat helped prevent. A well-calibrated, honest summary that enables a better decision is more valuable to the business than a falsely confident one that risks a worse outcome, even if the honest version appears less definitive on its surface.

</details>

---

Exercise 3 (Coding)#

Using the engineered DataFrame structure from this notebook, write code that checks whether the engagement-tier-to-churn relationship (Section 7) also holds consistently across region, as an additional confounding check beyond the plan-type check already performed in Section 9.

<details> <summary>Solution</summary>

Python
CopyAsk AI
by_region_and_tier = (
    engineered
    .groupby(["region", "engagement_tier"])["churned"]
    .mean()
    .unstack()
)
print(by_region_and_tier.round(3))

This directly extends the multi-level groupby confounding check technique from Section 9 (originally applied to plan_type) to a second plausible confounding variable, region, following the same underlying logic from Notebook 7, Section 8.3: if the escalating churn pattern across engagement tiers holds consistently within every region separately, this further strengthens confidence that engagement tier itself, rather than a regional effect correlated with it, is driving the observed pattern; if the pattern instead varied dramatically by region (for example, being much weaker or reversed in one specific region), this would warrant further investigation before finalizing the recommendation in Section 12.

</details>

---

Exercise 4 (Applied Thinking)#

Imagine that, after presenting the findings summary from Section 12.2, leadership approves the proposed A/B test (a re-engagement incentive for frequent skippers). Describe, using Notebook 10's framework, the key design decisions you would need to make before launching this test.

<details> <summary>Solution</summary>

Following Notebook 10, Section 9.2's A/B testing workflow: first, define the primary outcome metric precisely (likely churn rate over a fixed follow-up window, e.g., 60 days after the incentive is offered) and formulate the null hypothesis (the incentive has no effect on churn rate among frequent skippers) and alternative hypothesis. Second, calculate the required sample size in advance (Notebook 10, Section 10.3) based on the current frequent-skipper churn rate (41.2 percent, from Section 7) and the smallest reduction in churn that would justify the cost of the incentive program at scale. Third, randomly assign eligible frequent-skipper customers to treatment (receives the incentive) and control (does not), ensuring randomization occurs correctly to preserve the causal-inference benefit of a true A/B test (Notebook 10, Section 9.1), directly addressing the causal ambiguity flagged in Section 11.2 of this notebook, since a properly randomized test can establish causation in a way the original observational analysis could not. Fourth, commit to the pre-determined sample size and duration without early stopping (Notebook 10, Section 9.2, step 4, and the case study in Notebook 10, Section 13.1), and finally analyze the completed test using a two-proportion test (Notebook 10, Section 9.3), reporting both statistical and practical significance before deciding whether to roll the incentive program out more broadly.

</details>

---

18. Quiz#

Q1. Why does this capstone notebook perform a confounding check (Section 9) before a formal statistical test (Section 10)? a) The order does not matter b) To ensure the pattern being formally tested is not simply an artifact of an unaccounted-for third variable c) Confounding checks replace the need for statistical testing entirely d) Statistical tests cannot be run until every possible confounder has been eliminated

Q2. What was the purpose of including a causal caveat in the findings summary (Section 12.2)? a) To make the analysis appear more rigorous without any practical value b) To honestly communicate a genuine limitation that directly affects which intervention would be appropriate c) Caveats are a formality with no real influence on decisions d) To avoid presenting any recommendation at all

Q3. In the case study in Section 14.3, what prompted the analyst to revisit an earlier cleaning stage? a) A stakeholder requested it for no specific reason b) A visualization in a later stage revealed an implausible value traceable to a data entry error c) The statistical test failed to run d) The dataset needed to be made larger

Q4. Why did the recommendation in Section 12.2 propose a further A/B test rather than treating the original finding as sufficient to act on directly? a) A/B tests are always required regardless of the strength of prior evidence b) The original analysis could not fully resolve the causal direction of the relationship, and a randomized test can establish this more rigorously c) The original finding was not statistically significant d) A/B tests are faster to run than any other form of analysis

Q5. True or False: A real end-to-end analytics workflow always proceeds in one strict, linear pass from loading to communication, with no need to revisit earlier stages.

<details> <summary>Answers</summary>

1. b) To ensure the pattern being formally tested is not simply an artifact of an unaccounted-for third variable
2. b) To honestly communicate a genuine limitation that directly affects which intervention would be appropriate
3. b) A visualization in a later stage revealed an implausible value traceable to a data entry error
4. b) The original analysis could not fully resolve the causal direction of the relationship, and a randomized test can establish this more rigorously
5. False — as shown in Section 14.3, a real workflow frequently loops back to earlier stages when a later stage reveals a previously undetected problem

</details>

---

19. Summary and Course Retrospective#

19.1 What This Capstone Demonstrated#

This notebook applied, in a single continuous analysis, every major skill developed across the course: loading and inspecting data (Notebook 2), cleaning it (Notebook 4), engineering useful features (Notebooks 3, 5), exploring it systematically (Notebooks 6, 7), visualizing key patterns honestly (Notebook 8), quantifying relationships (Notebook 9), formally testing conclusions (Notebook 10), guarding against the specific failure modes this course repeatedly warned about (confounding, reverse causation, multiple comparisons, small samples), and communicating an honest, appropriately caveated, actionable recommendation.

19.2 The Course's Central Throughline#

If a single idea has run through every notebook in this course, it is this: a plausible-looking pattern in data is not the same as a well-supported conclusion, and the distance between the two is closed only through deliberate, specific technique, not general carefulness alone. Anscombe's Quartet (Notebook 8) showed summary statistics can hide structure; Simpson's Paradox (Notebooks 0, 7) showed aggregates can reverse under subgroups; the ice cream and drowning example (Notebook 9) showed correlation can arise from a shared cause; p-value misinterpretations (Notebook 10) showed statistical output is easy to misread; and this capstone's own case studies showed how each of these risks can appear, concretely, in a single realistic project if the corresponding discipline from earlier notebooks is skipped.

19.3 Where to Go From Here#

This course's final notebook is also a natural bridge to the separate Machine Learning course: many of the same tools (NumPy, Pandas, the bias-variance-adjacent tradeoffs in Notebook 1's z-score work, the statistical foundations in Notebook 6) reappear there, extended toward automated, repeatable prediction rather than one-time human-driven analysis and decision-making, which has been this course's consistent focus throughout.

---

20. Further Reading#

• Roger Peng and Elizabeth Matsui, The Art of Data Science — a practical treatment of the full analytics process, closely paralleling this capstone's structure
• Hadley Wickham and Garrett Grolemund, R for Data Science — though R-focused, its workflow chapter closely parallels the stage sequence in this notebook and is a valuable cross-language perspective
• Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments — for further depth on the A/B testing recommendation proposed in Section 12
• Judea Pearl and Dana Mackenzie, The Book of Why — for further depth on the causal reasoning challenges raised in Section 11.2 and Section 14.2

---

Course Complete#

This concludes the 13-notebook Data Analytics course, from Notebook 0's introduction through this end-to-end capstone project. The skills developed here, high-performance data manipulation, rigorous cleaning, vector math, systematic exploratory analysis, and defensible statistical inference, form the foundation for nearly all further work in data science, machine learning, and applied business analytics.

---

Course: Data Analytics | Notebook 12 of 13 (Capstone)