Pandas: DataFrames & Advanced Wrangling

1. Overview#

Pandas is the primary library for data manipulation and analysis in Python. It provides fast, flexible, and expressive data structures (Series and DataFrame) designed to make working with structured (tabular, multidimensional, potentially heterogeneous) and time-series data both easy and intuitive.

Always verify data types after loading. Converting raw strings to category or integer types (e.g. int32) can reduce memory consumption by up to 80% on large datasets.

---

2. Core Data Structures#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
import pandas as pd
import numpy as np

# Creating a DataFrame from scratch
df = pd.DataFrame({
 'experiment_id': [f"EXP_{i:03d}" for i in range(1, 6)],
 'learning_rate': [0.001, 0.01, 0.0005, 0.05, 0.001],
 'optimizer': ['AdamW', 'SGD', 'AdamW', 'RMSprop', 'AdamW'],
 'val_loss': [0.245, 0.512, 0.211, 0.689, 0.198],
 'epoch_time_sec': [45.2, 38.1, 46.0, 39.5, 44.8]
})
print(df.head())

---

3. High-Performance Wrangling Patterns#

3.1 Method Chaining (Clean Pipelines)#

Avoid mutating intermediate variables by chaining operations using .assign(), .query(), and .pipe():

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
cleaned_summary = (
 df
 .query("val_loss < 0.5")
 .assign(
 efficiency_score=lambda x: 1 / (x['val_loss'] * x['epoch_time_sec']),
 is_converged=lambda x: x['val_loss'] <= 0.25
 )
 .groupby('optimizer')
 .agg(
 avg_val_loss=('val_loss', 'mean'),
 best_efficiency=('efficiency_score', 'max'),
 total_runs=('experiment_id', 'count')
 )
 .reset_index()
 .sort_values(by='avg_val_loss', ascending=True)
)

print(cleaned_summary)

3.2 Vectorized Window Functions & Transformations#

🐍 PythonInteractive WebAssembly
EditCopyAsk AIRun Code
# Rolling averages over time windows
ts_df = pd.DataFrame({'daily_requests': np.random.poisson(lam=1000, size=30)})
ts_df['7d_rolling_avg'] = ts_df['daily_requests'].rolling(window=7, min_periods=1).mean()

---

4. Best Practices Checklist#

• Use pd.eval() or .query() for large DataFrames (>1M rows) to benefit from numexpr engine speedups.
• Never iterate over rows with for row in df.iterrows(): — use vectorized operations or .apply() with caution.
• Use df.info(memory_usage='deep') to monitor exact memory footprint.