Notebook 9: Correlation, Covariance, and Relationships

Module: Data Analytics Level: Intermediate Prerequisites: Notebook 8 (Data Visualization for EDA) Estimated Time: 3–4 hours

---

Table of Contents#

1. From Visual Relationships to Quantified Ones
2. Covariance
3. Pearson Correlation Coefficient
4. Interpreting Correlation Strength
5. When Pearson Correlation Fails: Nonlinearity and Outliers
6. Spearman Rank Correlation
7. Pearson vs. Spearman: Choosing the Right One
8. Correlation Matrices
9. Correlation Is Not Causation
10. Confounding Variables and Spurious Correlation
11. Hands-On: A Complete Correlation Analysis
12. Case Studies
13. Common Misconceptions and Pitfalls
14. Glossary
15. Exercises
16. Quiz
17. Summary
18. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Compute and interpret covariance, and explain why it is hard to interpret in isolation
• Compute, interpret, and correctly apply the Pearson correlation coefficient
• Recognize when Pearson correlation is misleading due to nonlinearity or outliers, connecting directly to Anscombe's Quartet
• Compute and interpret Spearman rank correlation, and choose between Pearson and Spearman appropriately
• Build and read a correlation matrix and its heatmap representation
• Explain precisely why correlation does not imply causation, using the three classical explanations
• Identify confounding variables and recognize spurious correlations in real or proposed analyses

---

1. From Visual Relationships to Quantified Ones#

Notebook 8 established that a scatter plot can reveal a relationship's shape, strength, and irregularities in ways numeric summaries alone cannot. This notebook provides the precise, quantifiable statistical measures that complement those visuals: numbers that let you rank, compare, and formally communicate the strength of a relationship between two variables, while remaining acutely aware, per the discussion in Section 9, of what these numbers can and cannot tell you about the real world.

---

2. Covariance#

2.1 Definition#

Covariance measures how two variables change together. For a sample of paired observations $(x_i, y_i)$:

$\text{Cov}(X, Y) = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar x)(y_i - \bar y)$

Note the direct structural parallel to sample variance from Notebook 6, Section 3.2 (which is simply the special case $\text{Cov}(X, X) = \text{Var}(X)$), including the same $n-1$ Bessel's correction and the same underlying justification (Notebook 6, Section 3.3).

2.2 Interpreting the Sign#

• Positive covariance: when $X$ is above its mean, $Y$ tends to also be above its mean (and vice versa) — the variables tend to move in the same direction.
• Negative covariance: when $X$ is above its mean, $Y$ tends to be below its mean — the variables tend to move in opposite directions.
• Covariance near zero: no consistent linear tendency for the variables to move together in either direction.

Python
CopyAsk AI
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "ad_spend": [1000, 1500, 2000, 2500, 3000],
    "sales": [12000, 15000, 19000, 21000, 26000]
})

covariance = df["ad_spend"].cov(df["sales"])
print(f"Covariance: {covariance:.2f}")

Expected output:

Covariance: 2062500.00

2.3 The Problem With Covariance: Scale Dependence#

The magnitude of covariance is not directly interpretable in isolation, because it depends on the units and scale of both variables. A covariance of 2,062,500 between ad spend (in dollars) and sales (in dollars) cannot be directly compared to a covariance of 45 between two variables measured in different units; simply changing ad spend from dollars to thousands of dollars would change the covariance value by a factor of 1,000 without the underlying relationship's real strength changing at all.

Python
CopyAsk AI
df["ad_spend_thousands"] = df["ad_spend"] / 1000
covariance_rescaled = df["ad_spend_thousands"].cov(df["sales"])
print(f"Covariance after rescaling ad_spend to thousands: {covariance_rescaled:.2f}")

Expected output:

Covariance after rescaling ad_spend to thousands: 2062.50

This exact scale-dependence problem directly motivates the Pearson correlation coefficient (Section 3), which normalizes covariance to remove this dependence on arbitrary units.

---

3. Pearson Correlation Coefficient#

3.1 Definition#

The Pearson correlation coefficient $r$ normalizes covariance by the standard deviations of both variables, producing a scale-independent measure of linear association:

$r = \frac{\text{Cov}(X, Y)}{s_X \cdot s_Y} = \frac{\sum_{i=1}^{n}(x_i - \bar x)(y_i - \bar y)}{\sqrt{\sum_{i=1}^{n}(x_i - \bar x)^2}\sqrt{\sum_{i=1}^{n}(y_i - \bar y)^2}}$

3.2 Key Properties#

• $r$ always ranges from $-1$ to $+1$, regardless of the original variables' units or scale, directly resolving the scale-dependence problem identified in Section 2.3.
• $r = +1$: a perfect positive linear relationship (every point falls exactly on an upward-sloping line).
• $r = -1$: a perfect negative linear relationship (every point falls exactly on a downward-sloping line).
• $r = 0$: no linear relationship (though, as Section 5 will show, this does not necessarily mean no relationship at all).

3.3 Computing Pearson Correlation#

Python
CopyAsk AI
r = df["ad_spend"].corr(df["sales"])  # defaults to Pearson
print(f"Pearson correlation: {r:.3f}")

r_rescaled = df["ad_spend_thousands"].corr(df["sales"])
print(f"Pearson correlation after rescaling: {r_rescaled:.3f}")

Expected output:

›Pearson correlation: 0.999
Pearson correlation after rescaling: 0.999

Confirming Section 3.2's key property directly: unlike covariance (Section 2.3), the correlation coefficient is completely unaffected by rescaling one of the variables, since the standard deviation term in the denominator rescales proportionally along with the covariance in the numerator.

---

4. Interpreting Correlation Strength#

4.1 A Common (But Context-Dependent) Rule of Thumb#

These thresholds are conventional guidelines, not universal statistical laws; what counts as a "strong" correlation varies substantially by field. A correlation of 0.3 might be considered remarkably strong and highly actionable in social science or behavioral research (where human behavior is influenced by countless factors), while the same 0.3 might be considered weak and practically uninteresting in physics or engineering, where near-deterministic relationships are often expected.

4.2 Correlation Squared: The Coefficient of Determination#

$r^2$ (the square of the Pearson correlation) has a direct, useful interpretation: the proportion of variance in one variable that is linearly explainable by the other. A correlation of $r = 0.7$ means $r^2 = 0.49$, or approximately 49 percent of the variance in one variable is linearly associated with the other, leaving 51 percent unexplained by that linear relationship alone. This exact quantity, $r^2$, reappears directly as a standard model evaluation metric in the Machine Learning course's regression evaluation content.

4.3 Statistical Significance vs. Practical Significance#

A correlation coefficient computed from a sample is itself an estimate, and, per Notebook 10's hypothesis testing framework, can be tested for statistical significance (whether it is meaningfully different from zero, given the sample size). With a very large sample, even a very small, practically meaningless correlation (such as $r = 0.02$) can be statistically significant; conversely, with a small sample, even a large correlation might not be statistically distinguishable from zero. Both the magnitude (practical significance, Section 4.1) and the statistical significance (formally addressed in Notebook 10) should be considered together, not conflated.

---

5. When Pearson Correlation Fails: Nonlinearity and Outliers#

5.1 The Nonlinearity Problem, Revisited#

Notebook 8, Section 1.1 and its case study in Section 13.1 already demonstrated this concretely: Pearson correlation specifically measures the strength of a linear relationship, and can be low or near zero even when a strong, genuine, but nonlinear (e.g., quadratic, logarithmic, exponential) relationship exists.

Python
CopyAsk AI
import numpy as np
import pandas as pd

x = np.linspace(-10, 10, 100)
y = x ** 2  # a perfect, deterministic, but strongly NONLINEAR relationship

r = pd.Series(x).corr(pd.Series(y))
print(f"Pearson correlation for y = x^2: {r:.3f}")

Expected output:

Pearson correlation for y = x^2: 0.000

Despite $y$ being perfectly, deterministically determined by $x$ (a relationship as strong as a relationship can possibly be), the Pearson correlation is essentially zero, because the relationship is symmetric and curved rather than linear: as $x$ increases from $-10$ to $0$, $y$ decreases, and as $x$ increases from $0$ to $10$, $y$ increases, and these two opposing linear tendencies cancel out entirely in the correlation calculation.

5.2 The Outlier Sensitivity Problem#

Because Pearson correlation, like the mean and standard deviation it is built from (Notebook 6, Sections 2.3 and 3.2), is not a robust statistic, a single extreme outlier can dramatically inflate or deflate the coefficient, exactly the mechanism behind Anscombe's Dataset III and IV (Notebook 8, Section 1.1):

Python
CopyAsk AI
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y_no_outlier = np.array([2, 4, 5, 4, 5, 7, 8, 7, 9, 10])
y_with_outlier = y_no_outlier.copy()
y_with_outlier[0] = 100  # inject one extreme outlier

r_no_outlier = np.corrcoef(x, y_no_outlier)[0, 1]
r_with_outlier = np.corrcoef(x, y_with_outlier)[0, 1]

print(f"Correlation without outlier: {r_no_outlier:.3f}")
print(f"Correlation with one outlier: {r_with_outlier:.3f}")

Expected output (approximate):

›Correlation without outlier: 0.978
Correlation with one outlier: 0.406

A single altered data point dramatically changed the apparent strength of an otherwise strong, consistent relationship, directly illustrating why the outlier detection and investigation discipline from Notebook 4, Section 4 should always precede correlation analysis, not follow it.

---

6. Spearman Rank Correlation#

6.1 The Core Idea#

Spearman correlation addresses both problems from Section 5 by first converting each variable's values to their ranks (1st smallest, 2nd smallest, and so on) and then computing the Pearson correlation formula (Section 3.1) on those ranks instead of the raw values:

$\rho = 1 - \frac{6\sum_{i=1}^n d_i^2}{n(n^2-1)}$

where $d_i$ is the difference between the ranks of $x_i$ and $y_i$ (this simplified formula assumes no tied ranks; software handles ties via a more general rank-based Pearson computation).

6.2 Why Rank-Based Correlation Is More Robust#

Because ranking converts every value to its relative position rather than its raw magnitude, a single extreme outlier can only ever shift by one rank position at most (from, say, "largest" to "largest," regardless of exactly how extreme it is), rather than distorting the calculation in proportion to its actual numeric magnitude, as happened in Section 5.2.

Python
CopyAsk AI
import pandas as pd

x = pd.Series([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
y_with_outlier = pd.Series([2, 4, 5, 4, 5, 7, 8, 7, 9, 100])

pearson_r = x.corr(y_with_outlier, method="pearson")
spearman_r = x.corr(y_with_outlier, method="spearman")

print(f"Pearson: {pearson_r:.3f}")
print(f"Spearman: {spearman_r:.3f}")

Expected output (approximate):

›Pearson: 0.406
Spearman: 0.964

Spearman correctly recognizes that, aside from the one extreme value, the underlying rank-order relationship between $x$ and $y$ is still very strong and consistent, since the outlier is still simply the largest value in its position, exactly as it would be even if it were only moderately large rather than extreme.

6.3 Spearman and Monotonic (Not Just Linear) Relationships#

A further advantage: Spearman correlation captures any monotonic relationship (one variable consistently increases as the other increases, or consistently decreases, but not necessarily at a constant linear rate), not just strictly linear ones, making it more robust to the kind of curved-but-consistently-directional relationship that would understate Pearson correlation without being as extreme as the perfectly symmetric $y=x^2$ case from Section 5.1.

Python
CopyAsk AI
x = np.linspace(1, 10, 50)
y = np.log(x)  # monotonically increasing but NOT linear

pearson_r = np.corrcoef(x, y)[0, 1]
spearman_r = pd.Series(x).corr(pd.Series(y), method="spearman")

print(f"Pearson (linear-only): {pearson_r:.3f}")
print(f"Spearman (monotonic): {spearman_r:.3f}")

Expected output (approximate):

›Pearson (linear-only): 0.966
Spearman (monotonic): 1.000

Spearman correctly identifies a perfect monotonic relationship ($\rho = 1.0$), since $y$ increases every single time $x$ increases with no exceptions, while Pearson, though still high here, slightly understates the relationship's true consistency because the rate of increase is not constant (a logarithmic curve, not a straight line).

---

7. Pearson vs. Spearman: Choosing the Right One#

A practical, low-cost habit: compute both Pearson and Spearman correlation for any important relationship, exactly as done throughout Section 6. A large discrepancy between the two (as in Section 6.2's outlier example) is itself a valuable diagnostic signal, immediately suggesting either an outlier problem or a genuinely nonlinear-but-monotonic relationship worth visualizing (Notebook 8, Section 6.1) before drawing conclusions.

---

8. Correlation Matrices#

8.1 Computing a Full Correlation Matrix#

Directly extending the preview from Notebook 7, Section 5.3 and Notebook 8, Section 9.4:

Python
CopyAsk AI
df = pd.DataFrame({
    "ad_spend": [1000, 1500, 2000, 2500, 3000, 1200, 1800],
    "sales": [12000, 15000, 19000, 21000, 26000, 13500, 17200],
    "website_visits": [5000, 6200, 7800, 8500, 10200, 5400, 6900],
    "customer_satisfaction": [7.2, 7.0, 6.8, 7.5, 6.5, 7.8, 7.1]
})

correlation_matrix = df.corr()
print(correlation_matrix.round(2))

Expected output (illustrative):

code
Ask AICopy
                        ad_spend  sales  website_visits  customer_satisfaction
ad_spend                    1.00   0.99            0.99                  -0.31
sales                       0.99   1.00            0.99                  -0.28
website_visits              0.99   0.99            1.00                  -0.25
customer_satisfaction      -0.31  -0.28           -0.25                   1.00

8.2 Reading a Correlation Matrix#

The diagonal is always exactly 1.0 (any variable is perfectly correlated with itself), and the matrix is symmetric (the correlation between $A$ and $B$ is identical to the correlation between $B$ and $A$), so only the upper or lower triangle needs to be visually scanned for genuinely new information.

Python
CopyAsk AI
import numpy as np

# Mask the upper triangle and diagonal to avoid redundant visual inspection
mask = np.triu(np.ones_like(correlation_matrix, dtype=bool))
print(correlation_matrix.mask(mask).round(2))

8.3 Multicollinearity Detection#

A correlation matrix is also the primary tool for detecting multicollinearity, the condition (already introduced in the Machine Learning course's Notebook 3, Section 5 and Section 6.2) where two or more features are highly correlated with each other, which can cause serious problems for linear regression coefficient stability. Spotting a very high correlation (commonly above 0.8 or 0.9 as a rough threshold) between two candidate predictor variables during EDA, before any modeling begins, allows this issue to be addressed proactively.

8.4 Visualizing at Scale#

For datasets with many numeric variables, a heatmap (Notebook 8, Section 9.4) is essential for making a large correlation matrix visually scannable, since a table of dozens of rows and columns of numbers becomes very difficult for a human to parse directly.

---

9. Correlation Is Not Causation#

9.1 Why This Warning Is So Persistent#

This is one of the most repeated warnings in all of statistics precisely because correlation is easy to compute and visually compelling, while establishing genuine causation requires much stronger evidence: a controlled experiment (like an A/B test, previewed in Notebook 0, Section 9.2's Simpson's Paradox discussion and formalized in Notebook 10), a well-designed natural experiment, or a rigorously argued causal model, none of which a correlation coefficient alone provides.

9.2 Three Classical Alternative Explanations for an Observed Correlation#

When two variables $X$ and $Y$ are correlated, there are at least three possible explanations beyond "$X$ causes $Y$":

1. Reverse causation: $Y$ actually causes $X$, not the other way around. (Example: a correlation between "having more support tickets" and "being a longer-tenured customer" might actually reflect that longer-tenured customers use more features and therefore encounter more edge cases, rather than support tickets somehow causing tenure.)
2. Confounding (a common cause): a third variable $Z$ causes both $X$ and $Y$, creating an association between them even though neither directly causes the other. (Covered in depth in Section 10.)
3. Coincidence (spurious correlation): with enough variables and enough data, some pairs will show a statistically notable correlation purely by chance, with no underlying causal or even indirect connection at all. (Also covered in Section 10.)

9.3 A Worked Example Distinguishing the Explanations#

Suppose ice cream sales and drowning incidents are strongly positively correlated across months.

• Not: ice cream sales cause drowning (direct causation)
• Not: drowning incidents cause ice cream sales (reverse causation)
• Actually: hot weather (the confounding variable $Z$) independently increases both ice cream sales (people buy more ice cream when it's hot) and swimming activity, and therefore drowning risk (more people swim when it's hot), creating a correlation between ice cream sales and drownings with no direct causal link between the two at all.

This is the single most commonly cited illustrative example of confounding in introductory statistics, precisely because the true explanation (shared weather-driven cause) is so immediately intuitive once stated.

---

10. Confounding Variables and Spurious Correlation#

10.1 Formal Definition of Confounding#

A confounding variable $Z$ is a variable that is associated with both the presumed cause $X$ and the presumed effect $Y$, such that some or all of the observed association between $X$ and $Y$ is actually attributable to $Z$, rather than to any direct relationship between $X$ and $Y$ themselves. This is precisely the mechanism behind Simpson's Paradox, formally introduced in Notebook 0, Section 9.2 and revisited as an EDA discipline in Notebook 7, Section 8: a confounding variable distributed unevenly across groups can create or reverse an apparent relationship.

10.2 Checking for Confounding in Practice#

A practical technique: compute the correlation between $X$ and $Y$ within fixed levels or narrow ranges of a suspected confounder $Z$ (directly reusing the multivariate groupby technique from Notebook 7, Section 5.2), and check whether the relationship persists, weakens, or disappears.

Python
CopyAsk AI
df = pd.DataFrame({
    "ice_cream_sales": [50, 80, 120, 150, 90, 60, 140, 100],
    "drownings": [2, 4, 7, 9, 5, 3, 8, 6],
    "temperature": [60, 75, 90, 95, 78, 65, 92, 82]
})

# Overall correlation
print(f"Overall correlation: {df['ice_cream_sales'].corr(df['drownings']):.2f}")

# Partial correlation approach: check correlation after "controlling for"
# temperature using a simple regression-residual method
from scipy import stats

def partial_correlation(x, y, z):
    # Regress x on z, and y on z, then correlate the residuals
    slope_xz, intercept_xz, _, _, _ = stats.linregress(z, x)
    slope_yz, intercept_yz, _, _, _ = stats.linregress(z, y)
    resid_x = x - (slope_xz * z + intercept_xz)
    resid_y = y - (slope_yz * z + intercept_yz)
    return np.corrcoef(resid_x, resid_y)[0, 1]

partial_r = partial_correlation(
    df["ice_cream_sales"].values, df["drownings"].values, df["temperature"].values
)
print(f"Correlation after controlling for temperature: {partial_r:.2f}")

Expected output (illustrative, exact values depend on the simulated data):

›Overall correlation: 0.98
Correlation after controlling for temperature: 0.31

The dramatic drop in correlation after controlling for temperature is precisely the statistical signature of confounding: most (though here, not literally all, given this specific simulated dataset) of the original apparent relationship between ice cream sales and drownings is explained away once the shared cause (temperature) is accounted for.

10.3 Spurious Correlations From Chance Alone#

With a sufficiently large number of variables, some pairs will show a notable correlation purely by random chance, even with no confounding and no genuine relationship of any kind. This is closely related to the multiple-comparisons concern previewed in Notebook 7's Exercise 1 solution and formalized further in Notebook 10; publicly documented examples (such as strong historical correlations between per-capita cheese consumption and the number of people who died by becoming tangled in their bedsheets) exist specifically to illustrate how convincing-looking but entirely coincidental correlations can arise when enough variable pairs are examined.

10.4 A Practical Checklist Before Claiming a Causal Relationship#

1. Is there a plausible, specific mechanism by which $X$ could directly cause $Y$?
2. Has reverse causation (Section 9.2) been considered and ruled out?
3. Has a search for plausible confounding variables (Section 10.1, 10.2) been conducted, and does the relationship persist after accounting for them?
4. Is the correlation based on a large enough number of comparisons that chance alone (Section 10.3) becomes a serious competing explanation?
5. Is there experimental or quasi-experimental evidence (an A/B test, a natural experiment, or a rigorously designed observational study), rather than a purely observational correlation, supporting a causal claim?

Only when a claim survives all five of these checks does treating an observed correlation as evidence of causation become reasonably well justified, and even then, formal statistical inference (Notebook 10) and, ideally, genuine experimentation remain the gold standard for causal claims.

---

11. Hands-On: A Complete Correlation Analysis#

This example applies the full toolkit from this notebook: computing both correlation types, building a correlation matrix, checking for a confounder, and drawing an appropriately cautious conclusion.

Python
CopyAsk AI
import pandas as pd
import numpy as np

rng = np.random.default_rng(42)
n = 200

# Simulate a scenario: "hours of exercise" appears correlated with "happiness score,"
# but both are actually driven by a confounding variable, "sleep quality"
sleep_quality = rng.normal(7, 1.5, n).clip(3, 10)
exercise_hours = 0.8 * sleep_quality + rng.normal(0, 1, n)
happiness_score = 0.9 * sleep_quality + rng.normal(0, 1, n)

df = pd.DataFrame({
    "sleep_quality": sleep_quality,
    "exercise_hours": exercise_hours,
    "happiness_score": happiness_score
})

# Step 1: full correlation matrix
print("Correlation matrix:")
print(df.corr().round(2))

# Step 2: Pearson vs Spearman check on the key relationship
pearson_r = df["exercise_hours"].corr(df["happiness_score"], method="pearson")
spearman_r = df["exercise_hours"].corr(df["happiness_score"], method="spearman")
print(f"\nExercise vs happiness -- Pearson: {pearson_r:.2f}, Spearman: {spearman_r:.2f}")

# Step 3: check whether the relationship survives controlling for the
# suspected confounder (sleep quality), using the technique from Section 10.2
from scipy import stats

def partial_correlation(x, y, z):
    slope_xz, intercept_xz, *_ = stats.linregress(z, x)
    slope_yz, intercept_yz, *_ = stats.linregress(z, y)
    resid_x = x - (slope_xz * z + intercept_xz)
    resid_y = y - (slope_yz * z + intercept_yz)
    return np.corrcoef(resid_x, resid_y)[0, 1]

partial_r = partial_correlation(
    df["exercise_hours"].values, df["happiness_score"].values, df["sleep_quality"].values
)
print(f"Correlation after controlling for sleep quality: {partial_r:.2f}")

Expected output (approximate, exact values depend on random simulation):

code
Ask AICopy
Correlation matrix:
                 sleep_quality  exercise_hours  happiness_score
sleep_quality             1.00            0.63             0.69
exercise_hours             0.63            1.00             0.51
happiness_score            0.69            0.51             1.00

Exercise vs happiness -- Pearson: 0.51, Spearman: 0.50

Correlation after controlling for sleep quality: 0.08

Interpretation: The raw correlation between exercise hours and happiness (0.51) looks meaningful on its own, and Pearson and Spearman agreeing closely suggests the relationship is reasonably linear with no major outlier distortion (Section 7). However, following the full checklist from Section 10.4, checking for a plausible confounder (sleep quality, which independently drives both variables in this simulated scenario) reveals the correlation nearly vanishes (dropping to 0.08) once sleep quality is controlled for, strongly suggesting the original exercise-happiness relationship was substantially confounded rather than reflecting a direct causal link, exactly the pattern illustrated abstractly in the ice cream and drowning example in Section 9.3 and Section 10.2.

---

12. Case Studies#

12.1 Case Study: A Product Feature Correlation That Was Actually Reverse Causation#

A product team observes that customers who use a particular "advanced settings" feature have substantially higher retention rates, and proposes a rollout encouraging all customers to use the feature, reasoning that the feature causes better retention. Further investigation, applying the reverse-causation check from Section 9.2, suggests the opposite direction is more plausible: customers who are already highly engaged and satisfied (and therefore more likely to stay) are the ones who explore advanced settings in the first place, not the other way around.

Diagnosis: This is a textbook case of the reverse causation explanation from Section 9.2: the observed correlation is real, but the presumed causal direction was backward. Aggressively pushing disengaged customers toward the advanced settings feature, based on the incorrect causal assumption, would likely have no effect on their retention (since the feature was never actually driving the outcome), representing wasted product development effort.

Lesson: Before proposing an intervention based on an observed correlation, explicitly considering the reverse-causation explanation from Section 9.2, and ideally testing the proposed causal direction directly via a controlled rollout (an A/B test, Notebook 10), is essential to avoid building a costly feature investment on an incorrect causal assumption.

12.2 Case Study: A Spurious Cross-Country Correlation#

An analyst examining a large public dataset of national economic and social indicators finds a surprisingly strong correlation (r = 0.75) between a country's per-capita consumption of a particular imported beverage and its average life expectancy, and drafts a report suggesting the beverage may have health benefits worth investigating further.

Diagnosis: Applying the confounding check from Section 10.2, a colleague points out that both variables are strongly associated with overall national wealth (GDP per capita): wealthier countries tend to both import more of this particular beverage (a discretionary, often relatively expensive import) and have better healthcare systems, sanitation, and nutrition, which independently drive higher life expectancy. Recomputing the correlation while controlling for GDP per capita (using the partial correlation technique from Section 10.2) reduces the coefficient to a value close to zero.

Lesson: Cross-country and other large-scale observational datasets are especially prone to confounding by broad, pervasive factors like national wealth, education level, or urbanization, since these tend to correlate with an enormous number of otherwise unrelated variables simultaneously; any interesting correlation discovered in this kind of dataset should be checked against the most obvious major confounders before being reported as a substantive, standalone finding.

12.3 Case Study: A Multicollinearity Problem Caught Before Modeling#

A team preparing to build a regression model (in the Machine Learning course's Notebook 3) to predict employee performance runs a correlation matrix (Section 8) on their candidate predictor variables as a standard EDA step and discovers that "years of experience" and "employee age" are correlated at r = 0.91.

Diagnosis: This is precisely the multicollinearity detection use case described in Section 8.3: two candidate predictors are so highly correlated with each other that including both in a linear regression model risks the coefficient instability problem described in the Machine Learning course's Notebook 3, Section 5 and Section 12.1's case study.

Lesson: Catching this during EDA, before any model is built, allowed the team to make a deliberate, informed decision (choosing to keep only one of the two highly correlated variables, or combining them, or applying regularization as covered in that course's Notebook 3, Section 7) rather than discovering unstable, hard-to-interpret coefficients only after a model was already built and needing further diagnosis.

---

13. Common Misconceptions and Pitfalls#

---

14. Glossary#

---

15. Exercises#

Exercise 1 (Conceptual)#

Explain why covariance alone cannot tell you whether a relationship between two variables is "strong," while the Pearson correlation coefficient can.

<details> <summary>Solution</summary>

Covariance's numeric magnitude, per Section 2.3, is directly affected by the units and scale of both variables involved; multiplying one variable by 1,000 (for example, converting from thousands of dollars to dollars) multiplies the covariance by 1,000 as well, even though the underlying real-world relationship has not changed in any meaningful way. This makes a raw covariance value uninterpretable in isolation, since there is no fixed scale against which to judge whether a given covariance value represents a strong or weak relationship. The Pearson correlation coefficient (Section 3.1) divides covariance by the product of both variables' standard deviations, which exactly cancels out this scale-dependence, producing a value always between -1 and 1 that can be interpreted consistently and compared meaningfully across entirely different variable pairs, regardless of their original units.

</details>

---

Exercise 2 (Conceptual)#

A researcher finds a strong positive correlation between the number of firefighters dispatched to a fire and the amount of damage the fire causes, and concludes that firefighters cause fire damage. Identify which of the three classical alternative explanations from Section 9.2 most plausibly applies here, and explain why.

<details> <summary>Solution</summary>

This is a classic illustration of confounding (Section 9.2, explanation 2, and formalized in Section 10.1): fire severity is the confounding variable that independently drives both the number of firefighters dispatched (larger, more severe fires require and receive a larger response) and the amount of damage caused (larger, more severe fires cause more damage, regardless of the response). The correlation between firefighter count and damage is real, but attributing a direct causal role to the firefighters themselves, rather than to the shared underlying cause (fire severity), would be a serious causal reasoning error, precisely analogous to the ice cream and drowning example in Section 9.3, with fire severity playing the role that temperature played in that example.

</details>

---

Exercise 3 (Coding)#

Given two arrays representing years of education and annual income for 50 individuals, where one individual has an anomalously recorded income of $50,000,000 due to a probable data entry error (missing a decimal point), compute both Pearson and Spearman correlation with income, and explain which value is likely more representative of the true underlying relationship for this dataset.

<details> <summary>Solution</summary>

Python
CopyAsk AI
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)
education_years = rng.integers(10, 20, 50)
income = 20000 + education_years * 3000 + rng.normal(0, 5000, 50)
income[10] = 50_000_000  # simulated data entry error

pearson_r = pd.Series(education_years).corr(pd.Series(income), method="pearson")
spearman_r = pd.Series(education_years).corr(pd.Series(income), method="spearman")

print(f"Pearson: {pearson_r:.3f}")
print(f"Spearman: {spearman_r:.3f}")

Given the presence of a single extreme, almost certainly erroneous value (Notebook 4, Section 4.4's "investigate first" principle would flag this immediately in a real analysis), the Spearman correlation is likely far more representative of the true underlying relationship between education and income here, since it is robust to the exact magnitude of the outlier (Section 6.2) and depends only on that individual's income still being ranked as the highest, which it would be with or without the decimal-point error. The Pearson correlation, by contrast, will likely be dramatically distorted toward near-zero or an unstable value by this single extreme point, exactly as demonstrated numerically in Section 5.2. The ideal fix, of course, is to correct or exclude the erroneous data point directly (Notebook 4) rather than relying solely on Spearman to work around it, but Spearman provides a useful, quick diagnostic in the meantime.

</details>

---

Exercise 4 (Applied Thinking)#

A retail analyst finds that stores with more staff scheduled tend to have lower per-employee sales productivity, and a stakeholder proposes cutting staffing levels to boost productivity. Using the checklist in Section 10.4, describe the investigation you would conduct before endorsing this recommendation.

<details> <summary>Solution</summary>

Following the checklist in Section 10.4: first, consider reverse causation — it is plausible that busier stores (with genuinely higher customer traffic and sales volume) are deliberately staffed with more employees because they are busy, meaning high staffing is a response to high demand, not a cause of lower productivity; if so, the causal arrow may run in the opposite direction from what the stakeholder assumes. Second, search for a plausible confounding variable, such as store size or store format (a large-format store naturally has both more staff and different per-employee productivity dynamics than a small-format store, independent of any staffing decision), and check whether the negative relationship persists after controlling for store size, using the partial correlation technique from Section 10.2. Third, check group sizes and the number of stores/comparisons involved to rule out coincidence given a potentially small number of stores. Only if the relationship survives these checks, and ideally following a genuine controlled staffing experiment across a comparable subset of stores (an A/B-test-style approach, Notebook 10) rather than an observational correlation alone, would cutting staffing levels be a well-justified recommendation; acting on the raw correlation alone risks the same reverse-causation and confounding traps illustrated in the case studies in Section 12.1 and 12.2.

</details>

---

16. Quiz#

Q1. Why is the Pearson correlation coefficient generally more useful than raw covariance for comparing relationship strength across different variable pairs? a) Correlation is always a larger number than covariance b) Correlation is normalized by the standard deviations of both variables, removing dependence on their units and scale c) Covariance cannot be computed for numeric data d) There is no meaningful difference between them

Q2. A Pearson correlation of 0 between two variables means: a) The two variables have absolutely no relationship of any kind b) There is no evidence of a linear relationship, though a nonlinear relationship may still exist c) The two variables are identical d) The correlation calculation failed

Q3. What advantage does Spearman correlation have over Pearson correlation? a) It always produces a higher coefficient value b) It is more robust to outliers and captures monotonic, not just linear, relationships c) It cannot be used with continuous data d) It is mathematically identical to Pearson correlation

Q4. In the classic ice cream sales and drowning incidents example, what best explains the observed correlation? a) Ice cream sales directly cause drowning b) Drowning incidents cause people to buy more ice cream c) A confounding variable (hot weather) independently drives both variables d) The correlation is purely coincidental with no explanation

Q5. True or False: If a correlation between two variables persists even after statistically controlling for a suspected confounding variable, this alone proves a direct causal relationship exists.

<details> <summary>Answers</summary>

1. b) Correlation is normalized by the standard deviations of both variables, removing dependence on their units and scale
2. b) There is no evidence of a linear relationship, though a nonlinear relationship may still exist
3. b) It is more robust to outliers and captures monotonic, not just linear, relationships
4. c) A confounding variable (hot weather) independently drives both variables
5. False — surviving a check for one specific suspected confounder rules out that particular explanation, per Section 10.4, but does not by itself rule out other confounders, reverse causation, or the need for genuine experimental evidence to establish causation

</details>

---

17. Summary#

• Covariance measures the direction and joint variability of two variables but is not interpretable in isolation because its magnitude depends on the variables' units and scale.
• The Pearson correlation coefficient normalizes covariance into a scale-independent measure of linear association, ranging from -1 to 1, with $r^2$ interpretable as the proportion of variance linearly explained.
• Pearson correlation can be misleadingly low for strong nonlinear relationships and is highly sensitive to outliers, directly extending the lessons of Anscombe's Quartet from Notebook 8.
• Spearman rank correlation, computed on ranks rather than raw values, is robust to outliers and captures any monotonic relationship, making it a valuable complement to Pearson, especially when the two diverge substantially.
• Correlation matrices summarize pairwise relationships across many variables at once and are the primary EDA tool for detecting multicollinearity before modeling begins.
• Correlation does not imply causation; an observed correlation may instead reflect reverse causation, a confounding variable, or pure coincidence, and a systematic checklist should be applied before treating any correlation as evidence of a causal relationship.
• Checking for confounding using partial correlation or subgroup analysis (directly connecting to the Simpson's Paradox discipline from Notebook 7) is a practical, applicable technique for interrogating a suspicious or consequential correlation before acting on it.

---

18. Further Reading#

• David Spiegelhalter, The Art of Statistics: Learning from Data, particularly its treatment of correlation and causation
• Judea Pearl and Dana Mackenzie, The Book of Why: The New Science of Cause and Effect — a rigorous, accessible treatment of causal reasoning beyond correlation
• Tyler Vigen, Spurious Correlations (spurious correlations.com) — a widely cited collection of amusing, purely coincidental correlations
• SciPy official documentation on statistical functions, including pearsonr and spearmanr: https://docs.scipy.org/doc/scipy/reference/stats.html

---

Next Up#

Notebook 10: Statistical Inference and Hypothesis Testing — p-values, confidence intervals, t-tests, chi-square tests, and A/B testing basics, providing the formal, rigorous framework needed to move from an EDA-discovered pattern (Notebook 7) or a notable correlation (this notebook) to a genuinely tested, defensible conclusion.

---

Course: Data Analytics | Notebook 9 of 13