Notebook 11: Working with Large Datasets and Optimization

Module: Data Analytics Level: Advanced Prerequisites: Notebook 10 (Statistical Inference and Hypothesis Testing) Estimated Time: 3–4 hours

---

Table of Contents#

1. When "Large" Becomes a Real Problem
2. Measuring Memory Usage
3. Dtype Downcasting
4. Categorical Encoding for Memory Savings
5. Reading Data Efficiently From the Start
6. Chunking: Processing Data in Pieces
7. Efficient File Formats
8. Introduction to Out-of-Core and Parallel Tools
9. Profiling Before Optimizing
10. A Decision Framework for Scaling Up
11. Hands-On: Shrinking a Dataset by 80 Percent
12. Case Studies
13. Common Misconceptions and Pitfalls
14. Glossary
15. Exercises
16. Quiz
17. Summary
18. Further Reading

---

Learning Objectives#

By the end of this notebook, you will be able to:

• Diagnose when a dataset is genuinely too large for comfortable in-memory analysis
• Measure a DataFrame's actual memory usage, column by column
• Apply dtype downcasting and categorical encoding to reduce memory usage substantially with no loss of information
• Load large files efficiently using column selection, dtype specification, and chunked reading
• Choose an efficient file format for storage and repeated access
• Explain, at an introductory level, what out-of-core and parallel tools like Dask offer beyond Pandas
• Profile code before optimizing it, avoiding wasted effort on the wrong bottleneck
• Apply a systematic decision framework for scaling an analysis as data size grows

---

1. When "Large" Becomes a Real Problem#

1.1 What "Large" Actually Means in Practice#

"Large" is relative to available memory (RAM), not an absolute row count. A 2-million-row dataset with 5 numeric columns might load instantly on a typical machine, while a 200,000-row dataset with 200 columns of high-cardinality text might already strain memory. The practical definition used throughout this notebook: a dataset is "large" for your current environment when it either fails to load at all, loads but leaves too little memory for the rest of your analysis to run comfortably, or takes long enough to load and process that it meaningfully slows down iterative exploratory work.

1.2 Why This Notebook Comes Last#

Every technique in Notebooks 1 through 10, vectorization (Notebook 1, 3), correct dtypes (Notebook 4), efficient reading (Notebook 2), and appropriate aggregation (Notebook 3) already constitutes the majority of what makes analytics code fast and memory-efficient. This notebook is specifically about the additional techniques needed once those foundational practices alone are not enough, either because the dataset does not fit in memory at all, or because processing time has become a genuine bottleneck to iterative work.

---

2. Measuring Memory Usage#

2.1 Why Default Memory Reporting Can Mislead#

Python
CopyAsk AI
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "id": range(1_000_000),
    "category": np.random.choice(["A", "B", "C", "D"], 1_000_000),
    "value": np.random.rand(1_000_000)
})

print(df.info())  # default: memory usage may show "+" and understate true size
print(df.info(memory_usage="deep"))  # accurate for object/string columns

By default, .info() can significantly understate memory usage for columns with object dtype (Notebook 1, Section 2.2; Notebook 2, Section 3.2), since it may report only the memory used by the pointers to the underlying Python string objects, not the strings themselves. Passing memory_usage="deep" forces an accurate accounting that actually inspects the underlying objects, which is essential before making any real optimization decision.

2.2 Per-Column Memory Breakdown#

Python
CopyAsk AI
memory_by_column = df.memory_usage(deep=True) / 1024**2  # convert bytes to MB
print(memory_by_column.round(2))
print(f"\nTotal: {memory_by_column.sum():.2f} MB")

Expected output (approximate):

code
Ask AICopy
Index         0.00
id            7.63
category     58.00
value         7.63
dtype: float64

Total: 73.26 MB

This immediately reveals the single most valuable piece of information for optimization: which columns dominate memory usage. In this example, the category column, despite holding only 4 distinct values, consumes roughly 8 times more memory than either numeric column, exactly the kind of finding that Section 4's categorical encoding directly addresses.

---

3. Dtype Downcasting#

3.1 The Principle#

Pandas defaults to relatively generous numeric dtypes (int64, float64) unless told otherwise, even when a column's actual value range would fit comfortably in a much smaller dtype. Downcasting switches a column to the smallest dtype that can safely represent its actual data, directly extending the dtype correctness principle from Notebook 4, Section 5.1 specifically toward memory efficiency.

3.2 Integer Downcasting#

Python
CopyAsk AI
df["id_small"] = pd.to_numeric(df["id"], downcast="integer")
print(df["id"].dtype, df["id_small"].dtype)
print(f"Original: {df['id'].memory_usage(deep=True) / 1024**2:.2f} MB")
print(f"Downcast: {df['id_small'].memory_usage(deep=True) / 1024**2:.2f} MB")

Expected output:

code
Ask AICopy
int64 int32
Original: 7.63 MB
Downcast: 3.81 MB

pd.to_numeric(..., downcast="integer") automatically selects the smallest safe integer dtype for the actual data present, here reducing memory by half simply because the values happen to fit within int32's range without any loss of information.

3.3 Float Downcasting#

Python
CopyAsk AI
df["value_small"] = pd.to_numeric(df["value"], downcast="float")
print(df["value"].dtype, df["value_small"].dtype)

Expected output:

float64 float32

float32 uses half the memory of float64 but with reduced precision (roughly 7 significant decimal digits versus 15-16); this tradeoff is acceptable for most exploratory analytics work but should be considered carefully for calculations where precision compounds across many operations (such as certain financial calculations), where the reduced precision could accumulate into a meaningfully incorrect result.

3.4 A General Rule#

Always downcast based on the actual observed range of data, never assume a smaller dtype is safe without checking; pd.to_numeric(..., downcast=...) handles this safety check automatically, refusing to downcast to a dtype that cannot represent the actual values present, but a manual .astype("int8") call would not perform this check and could silently overflow and corrupt data if the actual values exceed that dtype's range.

---

4. Categorical Encoding for Memory Savings#

4.1 Revisiting Notebook 4's Categorical Dtype#

Notebook 4, Section 5.3 introduced the category dtype for memory efficiency; this section quantifies exactly how large that savings can be and clarifies when it does and does not apply.

Python
CopyAsk AI
category_memory_before = df["category"].memory_usage(deep=True) / 1024**2
df["category_encoded"] = df["category"].astype("category")
category_memory_after = df["category_encoded"].memory_usage(deep=True) / 1024**2

print(f"Before (object dtype): {category_memory_before:.2f} MB")
print(f"After (category dtype): {category_memory_after:.2f} MB")
print(f"Reduction: {(1 - category_memory_after/category_memory_before) * 100:.1f}%")

Expected output (approximate):

code
Ask AICopy
Before (object dtype): 58.00 MB
After (category dtype): 0.95 MB
Reduction: 98.4%

4.2 Why the Savings Are So Dramatic for Low-Cardinality Columns#

The category dtype stores each unique value once in a small lookup table, then represents every row as a compact integer code referencing that table, rather than repeating the full string value a million times. The memory savings scale directly with redundancy: a column with very low cardinality (few unique values, Notebook 7, Section 3.3) relative to its row count sees enormous savings, since the same handful of strings are being repeated an enormous number of times.

4.3 When Categorical Encoding Does Not Help#

For a column with very high cardinality relative to its row count (such as a nearly-unique identifier or free-text field), converting to category provides little or no benefit, and can even slightly increase memory usage, since the lookup table itself becomes almost as large as the original column with little redundancy to exploit. Always check cardinality (Notebook 7, Section 3.3) before applying categorical encoding as an optimization.

Python
CopyAsk AI
print(f"Category column cardinality: {df['category'].nunique()} out of {len(df)} rows")

---

5. Reading Data Efficiently From the Start#

5.1 Revisiting and Extending Notebook 2's Efficient Loading#

Notebook 2, Section 8.2 and its case study in Section 11.3 already introduced usecols and dtype at read time. This section combines them explicitly with the downcasting and categorical techniques from Sections 3 and 4, applied at the moment of loading rather than as a separate post-load step:

Python
CopyAsk AI
dtype_spec = {
    "customer_id": "int32",
    "region": "category",
    "amount": "float32"
}

df = pd.read_csv(
    "large_transactions.csv",
    usecols=["customer_id", "region", "amount", "transaction_date"],
    dtype=dtype_spec,
    parse_dates=["transaction_date"]
)

Specifying dtypes directly at load time avoids ever allocating memory for the default, larger dtypes in the first place, which is more memory-efficient than loading with default dtypes and downcasting afterward, since the latter briefly requires memory for both the original and the downcast version simultaneously.

5.2 Loading a Sample First#

Before committing to load an entire very large file, loading a small sample (using nrows) to inspect structure, dtypes, and approximate memory usage per row is a low-cost, high-value first step:

Python
CopyAsk AI
sample = pd.read_csv("large_transactions.csv", nrows=10_000)
print(sample.info(memory_usage="deep"))

estimated_total_memory_mb = (
    sample.memory_usage(deep=True).sum() / 1024**2 / len(sample) * total_row_count
)

This lets you estimate whether the full file will fit comfortably in available memory before attempting to load it, avoiding a potentially long wait followed by an out-of-memory failure.

---

6. Chunking: Processing Data in Pieces#

6.1 The Core Idea#

Chunking processes a large file in smaller pieces sequentially, never holding the entire dataset in memory at once, which is essential when a file is simply too large to fit in available RAM regardless of dtype optimization.

Python
CopyAsk AI
import pandas as pd

chunk_size = 100_000
total_revenue = 0
region_totals = pd.Series(dtype="float64")

for chunk in pd.read_csv("large_transactions.csv", chunksize=chunk_size):
    total_revenue += chunk["amount"].sum()
    region_totals = region_totals.add(
        chunk.groupby("region")["amount"].sum(), fill_value=0
    )

print(f"Total revenue: {total_revenue:,.2f}")
print(region_totals.sort_values(ascending=False))

Each chunk is read, aggregated, and then discarded from memory before the next chunk is loaded, so only one chunk's worth of data (plus the small, running aggregate results) needs to fit in memory at any given moment, regardless of how large the full file is.

6.2 When Chunking Works Well, and When It Does Not#

Chunking works cleanly for operations that can be computed incrementally: sums, counts, and, with the .add(fill_value=0) pattern shown in Section 6.1, groupby aggregations, since a running total can be correctly updated chunk by chunk. Chunking is considerably more awkward, though not always impossible, for operations that inherently require seeing the entire dataset at once, such as computing an exact median (Notebook 6, Section 2.2) or sorting the full dataset, since these cannot simply be combined from independently computed per-chunk results.

6.3 Filtering With Chunking#

Python
CopyAsk AI
matching_rows = []
for chunk in pd.read_csv("large_transactions.csv", chunksize=100_000):
    filtered = chunk[chunk["amount"] > 10000]
    matching_rows.append(filtered)

high_value_transactions = pd.concat(matching_rows, ignore_index=True)

This directly reuses the pd.concat technique from Notebook 3, Section 7.1, but applied specifically to accumulate filtered results across chunks; this pattern is safe as long as the filtered result across all chunks is itself small enough to fit comfortably in memory, even though the original full file was not.

---

7. Efficient File Formats#

7.1 The Limitations of CSV#

CSV, while universally readable and human-inspectable, is a plain-text format with no embedded dtype information (forcing Pandas to either guess dtypes or require them to be specified explicitly every time, per Section 5.1) and no compression, making it both slower to parse and larger on disk than binary alternatives.

7.2 Parquet#

Parquet is a columnar binary file format that stores dtype information directly in the file, supports efficient compression, and, critically, allows reading only specific columns without parsing the entire file, directly extending the usecols benefit from Section 5.1 down to the file format level itself:

Python
CopyAsk AI
df.to_parquet("transactions.parquet", index=False)

# Reading back preserves dtypes exactly, with no need to re-specify them
df_loaded = pd.read_parquet("transactions.parquet")
print(df_loaded.dtypes)

# Reading only specific columns is fast, since Parquet's columnar
# layout allows skipping the data for unneeded columns entirely
df_partial = pd.read_parquet("transactions.parquet", columns=["region", "amount"])

7.3 Comparing Formats#

7.4 When to Convert to a Binary Format#

If a dataset is read repeatedly (across many analysis sessions, or by multiple team members), converting it once to Parquet and reading from that Parquet file thereafter typically pays for itself quickly: faster load times, smaller disk footprint, and no need to re-specify dtypes on every read, directly eliminating an entire category of the dtype-related bugs discussed in Notebook 4.

---

8. Introduction to Out-of-Core and Parallel Tools#

8.1 When Pandas Alone Is Not Enough#

Even with every optimization in this notebook applied, a dataset can still be too large to fit in memory at all, or an analysis can still be too slow on a single CPU core. Out-of-core tools process data that does not fit in memory by operating on it incrementally from disk (extending the manual chunking pattern from Section 6 automatically and more efficiently), and parallel tools split computation across multiple CPU cores or machines.

8.2 Dask: A Pandas-Like Interface for Larger-Than-Memory Data#

Dask provides a DataFrame interface deliberately designed to closely mirror Pandas's own API, while internally operating on data in partitions (conceptually similar to the chunks from Section 6, but managed automatically) and executing operations lazily, building a computation plan that is only actually executed when explicitly requested.

Python
CopyAsk AI
import dask.dataframe as dd

# Reads the file(s) without loading everything into memory immediately
ddf = dd.read_csv("large_transactions_*.csv")

# Operations look nearly identical to Pandas, but build a lazy computation
# graph rather than executing immediately
result = ddf.groupby("region")["amount"].sum()

# .compute() triggers actual execution, processing the data in partitions
final_result = result.compute()
print(final_result)

8.3 Why Lazy Evaluation Matters#

Because Dask builds a computation graph before executing anything, it can optimize the overall plan (for example, recognizing that only certain columns are ultimately needed and skipping the rest, similar in spirit to Parquet's columnar reading in Section 7.2) and can execute independent parts of the computation across multiple CPU cores simultaneously, rather than processing everything strictly sequentially the way the manual chunking loop in Section 6.1 does.

8.4 When to Reach for Dask (or Similar Tools)#

This course introduces Dask only at this conceptual level; a full treatment of distributed computing frameworks is a substantial topic in its own right, generally addressed in a dedicated big-data or data-engineering course rather than a foundational analytics course.

---

9. Profiling Before Optimizing#

9.1 Why Guessing at Bottlenecks Wastes Effort#

Directly extending the benchmarking discipline from Notebook 3, Section 9, it is tempting to optimize whichever piece of code looks slow, but intuition about performance bottlenecks is frequently wrong; a small, overlooked line can dominate total runtime while an obviously complex-looking calculation may already be fast. Profiling measures where time is actually being spent, rather than relying on assumption.

9.2 Simple Timing With time.perf_counter#

Directly reusing the technique from Notebook 1, Section 10.1 and Notebook 3, Section 2.2, applied specifically to isolate one stage of a larger pipeline:

Python
CopyAsk AI
import time

start = time.perf_counter()
df = pd.read_csv("large_file.csv")
print(f"Load time: {time.perf_counter() - start:.2f}s")

start = time.perf_counter()
result = df.groupby("region")["amount"].sum()
print(f"Groupby time: {time.perf_counter() - start:.2f}s")

9.3 Line-by-Line Profiling With %prun (Jupyter)#

For a more detailed breakdown across an entire function or pipeline, Jupyter's built-in %prun magic command provides a function-call-level breakdown of where time is spent:

Python
CopyAsk AI
# %prun result = some_analysis_function(df)

This produces a ranked list of every function call made during execution, sorted by total time spent, immediately revealing whether the actual bottleneck is, for example, a particular string operation, a repeated filtering step, or the initial file load itself, rather than requiring a guess.

9.4 The General Optimization Workflow#

1. Profile first (Section 9.2 or 9.3) to identify the actual bottleneck, not the assumed one.
2. Apply the most targeted fix for that specific bottleneck (dtype optimization for memory issues, vectorization for slow row-wise operations per Notebook 3, Section 2, chunking or Dask for data that does not fit in memory).
3. Re-profile to confirm the fix actually helped and to check whether a new bottleneck has now become the dominant cost.
4. Stop once performance is acceptable for the task at hand; further optimization has diminishing returns and its own cost in code complexity and maintenance burden.

---

10. A Decision Framework for Scaling Up#

---

11. Hands-On: Shrinking a Dataset by 80 Percent#

This example applies every memory-reduction technique from this notebook to a simulated dataset, measuring the cumulative effect at each stage.

Python
CopyAsk AI
import pandas as pd
import numpy as np

rng = np.random.default_rng(42)
n = 2_000_000

df = pd.DataFrame({
    "transaction_id": range(n),
    "customer_id": rng.integers(1, 50_000, n),
    "region": rng.choice(["North", "South", "East", "West"], n),
    "payment_method": rng.choice(["Credit", "Debit", "PayPal", "Cash"], n),
    "amount": rng.uniform(5, 500, n)
})

original_memory = df.memory_usage(deep=True).sum() / 1024**2
print(f"Original memory usage: {original_memory:.1f} MB")

# Step 1: downcast integers (Section 3.2)
df["transaction_id"] = pd.to_numeric(df["transaction_id"], downcast="integer")
df["customer_id"] = pd.to_numeric(df["customer_id"], downcast="integer")

# Step 2: downcast floats (Section 3.3)
df["amount"] = pd.to_numeric(df["amount"], downcast="float")

# Step 3: convert low-cardinality text columns to category (Section 4)
df["region"] = df["region"].astype("category")
df["payment_method"] = df["payment_method"].astype("category")

optimized_memory = df.memory_usage(deep=True).sum() / 1024**2
print(f"Optimized memory usage: {optimized_memory:.1f} MB")
print(f"Reduction: {(1 - optimized_memory/original_memory) * 100:.1f}%")

print("\nFinal dtypes:")
print(df.dtypes)

Expected output (approximate):

code
Ask AICopy
Original memory usage: 209.6 MB
Optimized memory usage: 42.4 MB
Reduction: 79.8%

Interpretation: Without changing a single value in the dataset, purely by choosing dtypes that actually match the data's real range and cardinality, memory usage was reduced by roughly 80 percent, directly extending the correctness-focused dtype work from Notebook 4 into a deliberate, quantified performance optimization; this is frequently enough of a reduction to turn a dataset that previously would not comfortably fit in memory into one that loads and processes without any need for chunking or out-of-core tools at all.

---

12. Case Studies#

12.1 Case Study: A Failed Load Resolved Entirely by dtype Specification#

An analyst attempts to load a 15 GB CSV file of web server logs on a machine with 16 GB of RAM, and the process fails with a memory error partway through loading. Rather than immediately reaching for a more powerful machine or a distributed computing tool, the analyst first checks which columns are actually needed (a handful of the file's 60 total columns) and specifies dtypes explicitly for those columns, following the pattern in Section 5.1.

Diagnosis: The default pd.read_csv() call was loading all 60 columns using generous default dtypes (int64, float64, generic object for text), when the actual analysis needed only 6 columns, several of which were low-cardinality categorical fields that could be stored far more compactly, following the same pattern demonstrated numerically in Section 11.

Lesson: Combining usecols and explicit dtype/category specification at load time reduced the effective memory footprint enough that the file loaded comfortably in available RAM, without needing to reach for chunking (Section 6), a more powerful machine, or Dask (Section 8) at all; per Section 10's decision framework, this kind of load-time optimization should generally be attempted before escalating to more complex tools.

12.2 Case Study: A Slow Nightly Pipeline Traced to an Unexpected Bottleneck#

A team assumes their nightly data pipeline is slow because of a complex statistical calculation performed near the end of the process, and spends a week attempting to optimize that calculation with limited improvement in total runtime. Applying the profiling discipline from Section 9.3, the team discovers that over 70 percent of total runtime was actually being spent re-reading the same large CSV file three separate times at different stages of the pipeline, an inefficiency invisible without profiling and unrelated to the calculation they had assumed was the bottleneck.

Diagnosis: This is a direct illustration of Section 9.1's core warning: intuition about where time is being spent is frequently wrong, and the team's week of effort optimizing the statistical calculation, while not wasted in an absolute sense, addressed a cost that was a small fraction of the true total runtime.

Lesson: Loading the file once, converting it to Parquet (Section 7.2) for the remainder of the pipeline's stages to read from repeatedly, resolved the actual bottleneck in under an hour of work, dramatically outperforming a week spent optimizing the wrong part of the pipeline; profiling first (Section 9.4's workflow) would have identified this immediately rather than after a week of misdirected effort.

12.3 Case Study: Chunking Enabling an Analysis on a Dataset Ten Times Larger Than Available RAM#

An analytics team needs to compute total revenue and average transaction size by region from a 100 GB transaction log, on a machine with roughly 10 GB of available RAM, well beyond what dtype optimization (Section 3, 4) alone could resolve.

Diagnosis: Even aggressive dtype optimization could not bring a genuinely 100 GB dataset down to a size that fits in 10 GB of RAM; this is a case where the underlying data volume itself, not merely inefficient representation, exceeds available memory, exactly the scenario chunking (Section 6) and out-of-core tools (Section 8) are designed for.

Lesson: Because the required aggregations (sums, counts, and group-level averages, which can be computed as a running sum divided by a running count) are all incrementally computable, per Section 6.2, the team was able to use pd.read_csv(..., chunksize=...) to process the file in manageable pieces, maintaining only small running aggregate totals in memory at any point, successfully completing the analysis without needing to provision a larger machine or adopt a more complex distributed tool like Dask, which would likely have been the next reasonable step (per Section 10's decision framework) had the required operations been less straightforwardly incremental.

---

13. Common Misconceptions and Pitfalls#

---

14. Glossary#

---

15. Exercises#

Exercise 1 (Conceptual)#

Explain why calling .astype("int8") directly on a column, without first checking its value range, is riskier than using pd.to_numeric(column, downcast="integer").

<details> <summary>Solution</summary>

.astype("int8") applied directly performs no safety check against the column's actual data; if any value in the column exceeds int8's representable range (-128 to 127), the conversion will silently overflow, wrapping around to an incorrect value rather than raising an error, corrupting the data without any warning. pd.to_numeric(column, downcast="integer"), per Section 3.4, automatically inspects the actual range of values present and selects the smallest safe dtype that can represent all of them without overflow, making it the considerably safer choice for downcasting in practice.

</details>

---

Exercise 2 (Conceptual)#

A colleague converts a column containing unique order confirmation numbers (nearly one distinct value per row) to category dtype, expecting significant memory savings, but observes almost no improvement. Explain why, using Section 4.3.

<details> <summary>Solution</summary>

The category dtype achieves memory savings by storing each unique value once in a lookup table and representing rows as compact integer codes referencing that table (Section 4.2); this savings scales directly with redundancy, since it exploits the fact that the same values repeat many times across rows. A column of nearly unique order confirmation numbers has almost no redundancy to exploit, since nearly every row's value is different from every other row's value, meaning the lookup table itself ends up almost as large as the original column, and the additional overhead of the integer code mapping can even make the categorical version slightly larger than the original, exactly the scenario described in Section 4.3.

</details>

---

Exercise 3 (Coding)#

Given a large CSV file with a region column and an amount column, write chunked processing code that computes the average amount per region across the entire file without loading it all into memory at once, being careful to compute a true average (not simply an average of per-chunk averages, which would be incorrect if chunks have different sizes).

<details> <summary>Solution</summary>

Python
CopyAsk AI
import pandas as pd

running_sum = pd.Series(dtype="float64")
running_count = pd.Series(dtype="int64")

for chunk in pd.read_csv("large_transactions.csv", chunksize=200_000):
    chunk_sum = chunk.groupby("region")["amount"].sum()
    chunk_count = chunk.groupby("region")["amount"].count()

    running_sum = running_sum.add(chunk_sum, fill_value=0)
    running_count = running_count.add(chunk_count, fill_value=0)

true_average_by_region = running_sum / running_count
print(true_average_by_region)

This correctly avoids the trap of averaging per-chunk averages (which would implicitly and incorrectly weight every chunk equally regardless of how many rows of each region happened to land in it); instead, it maintains a running sum and a running count per region separately across all chunks (directly extending the incremental-computation principle from Section 6.2) and divides only once, at the very end, after all chunks have been processed, producing the mathematically correct overall average per region.

</details>

---

Exercise 4 (Applied Thinking)#

Your team's daily reporting script reads the same 8 GB CSV file from scratch every morning, and this load alone takes 12 minutes before any analysis even begins. Using Sections 7 and 10, describe a concrete change you would recommend, and explain why it addresses the actual bottleneck rather than a secondary one.

<details> <summary>Solution</summary>

Since the file is read repeatedly, unchanged, across multiple runs (a "same large file loaded repeatedly across sessions" scenario, directly matching a row in Section 10's decision framework table), converting the source data to a columnar binary format like Parquet (Section 7.2) once, and having the daily script read from that Parquet file instead of the raw CSV, would likely eliminate the majority of the 12-minute load time: Parquet avoids the CSV parsing overhead entirely, preserves dtypes so no time is spent inferring or re-specifying them on every run, and supports efficient compression, all of which directly target the actual bottleneck (repeated, wasteful CSV parsing) rather than, for example, attempting to optimize a downstream calculation that was never the true source of the delay, echoing the misdirected-effort lesson from the case study in Section 12.2.

</details>

---

16. Quiz#

Q1. Why can the default .info() output understate a DataFrame's true memory usage? a) It always reports memory in the wrong units b) It may not accurately account for the memory used by object-dtype columns unless memory_usage='deep' is specified c) It only works for numeric columns d) It reports memory usage from a previous session

Q2. For which type of column does converting to category dtype provide the largest memory savings? a) A column with very high cardinality (nearly unique values per row) b) A column with very low cardinality relative to the number of rows c) A column of floating-point numbers d) Categorical dtype provides identical savings regardless of cardinality

Q3. Which of the following operations is well-suited to being computed incrementally via chunking? a) Computing an exact median of the full dataset b) Sorting the entire dataset c) Computing a running sum and count per group, to later derive an average d) None of the above can be chunked

Q4. What is the primary advantage of a columnar file format like Parquet over CSV? a) It is human-readable in a text editor b) It preserves dtypes, supports compression, and allows efficient reading of only specific columns c) It cannot be read by Pandas d) It requires no disk space at all

Q5. True or False: Optimizing the part of a pipeline that appears slowest, based on intuition alone, is generally as effective as profiling the code first.

<details> <summary>Answers</summary>

1. b) It may not accurately account for the memory used by object-dtype columns unless memory_usage='deep' is specified
2. b) A column with very low cardinality relative to the number of rows
3. c) Computing a running sum and count per group, to later derive an average
4. b) It preserves dtypes, supports compression, and allows efficient reading of only specific columns
5. False — as shown in the case study in Section 12.2, intuition about bottlenecks is frequently wrong, and profiling first avoids wasted optimization effort

</details>

---

17. Summary#

• "Large" is relative to available memory and the specific dtypes involved, not an absolute row count, and accurate memory measurement (memory_usage(deep=True)) is the essential first diagnostic step.
• Dtype downcasting and categorical encoding can reduce memory usage dramatically, often by 70 to 90 percent, with zero loss of information, and should generally be applied before reaching for more complex tools.
• Specifying dtypes, needed columns, and categorical encoding at load time is more efficient than optimizing after a full, default load.
• Chunking processes data too large for memory in sequential pieces, working cleanly for incrementally computable operations like sums and grouped aggregations, but less naturally for operations requiring the full dataset at once.
• Columnar binary file formats like Parquet outperform CSV for repeated access, preserving dtypes and enabling efficient partial reads.
• Dask and similar out-of-core, parallel tools extend Pandas-like operations to datasets and computations that exceed a single machine's memory or a single core's processing capacity, and are best reached for once dtype optimization and chunking are no longer sufficient.
• Profiling code before optimizing it is essential, since intuition about performance bottlenecks is frequently wrong, and misdirected optimization effort can waste substantial time addressing a cost that was never the true bottleneck.

---

18. Further Reading#

• Pandas official documentation on scaling to large datasets: https://pandas.pydata.org/docs/user_guide/scale.html
• Pandas official documentation on enhancing performance: https://pandas.pydata.org/docs/user_guide/enhancingperf.html
• Dask official documentation: https://docs.dask.org/
• Apache Parquet official documentation: https://parquet.apache.org/docs/

---

Next Up#

Notebook 12: End-to-End EDA Project Workflow — a full pipeline from loading through cleaning, transforming, exploring, visualizing, and summarizing findings, integrating every technique from Notebooks 1 through 11 into a single, complete, realistic analytics project.

---

Course: Data Analytics | Notebook 11 of 13