File I/O

Overview#

Almost every real program needs to read from or write to files — logs, configuration, data exports, user uploads. Python's built-in open() function, combined with the with statement for safe cleanup, is the foundation for all file handling. This notebook covers opening files in different modes, reading and writing text and binary data, working with paths using pathlib, handling common file formats like CSV and JSON, and best practices for encoding and error handling.

1. Opening Files with open()#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
file = open("data.txt", "r")   # opens for reading (text mode)
contents = file.read()
file.close()                    # must remember to close manually

Manually closing files is error-prone — if an exception occurs between open() and close(), the file is never closed. Always use the with statement instead, which guarantees the file closes automatically.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("data.txt", "r") as file:
    contents = file.read()
# file is closed here automatically, even if an exception occurred above

File modes#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# Overwrites the entire file if it exists
with open("output.txt", "w") as f:
    f.write("This replaces everything in the file\n")

# Adds to the end without erasing existing content
with open("output.txt", "a") as f:
    f.write("This line is appended\n")

2. Reading Files#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# Read the entire file as one string
with open("data.txt", "r") as f:
    contents = f.read()

# Read one line at a time
with open("data.txt", "r") as f:
    first_line = f.readline()
    second_line = f.readline()

# Read all lines into a list
with open("data.txt", "r") as f:
    lines = f.readlines()   # ['line1\n', 'line2\n', ...]

# Iterate line by line — memory-efficient for large files
with open("data.txt", "r") as f:
    for line in f:
        print(line.strip())   # strip() removes the trailing newline

Iterating a file object directly (the last example) is the preferred approach for large files — it reads one line at a time instead of loading the entire file into memory at once, similar to how a generator works.

Reading a fixed number of characters#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("data.txt", "r") as f:
    chunk = f.read(100)   # read up to 100 characters

3. Writing Files#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("output.txt", "w") as f:
    f.write("First line\n")
    f.write("Second line\n")

# Write multiple lines at once from a list
lines = ["First line\n", "Second line\n", "Third line\n"]
with open("output.txt", "w") as f:
    f.writelines(lines)

Note that write() and writelines() do not add newlines automatically — you must include \n yourself where needed.

4. Text Encoding#

Text files are stored as bytes on disk, and must be decoded into strings using a specific character encoding. Always specify the encoding explicitly rather than relying on the platform's default, since default encodings can differ across operating systems and cause subtle bugs.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("data.txt", "r", encoding="utf-8") as f:
    contents = f.read()

with open("output.txt", "w", encoding="utf-8") as f:
    f.write("Café, naïve, 日本語")   # non-ASCII text needs correct encoding

utf-8 is the safe, standard default for nearly all modern text files.

5. Binary Files#

Use binary mode ("rb", "wb") for non-text files — images, PDFs, executables — where data should not be decoded as text at all.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("image.png", "rb") as f:
    data = f.read()          # returns bytes, not str

with open("copy.png", "wb") as f:
    f.write(data)

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# Copying a file in chunks (memory-efficient for large files)
with open("source.bin", "rb") as src, open("dest.bin", "wb") as dst:
    while chunk := src.read(4096):
        dst.write(chunk)

6. Working with Paths: pathlib#

The pathlib module offers an object-oriented, cross-platform way to work with file paths, and is preferred over manually building path strings with os.path.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
from pathlib import Path

p = Path("data") / "reports" / "2024.txt"   # cross-platform path joining
print(p)                     # data/reports/2024.txt (or data\reports\2024.txt on Windows)

print(p.name)                 # 2024.txt
print(p.stem)                 # 2024
print(p.suffix)               # .txt
print(p.parent)               # data/reports
print(p.exists())             # True/False
print(p.is_file())            # True/False
print(p.is_dir())             # True/False

# Reading and writing without needing "with open(...)" for simple cases
text = p.read_text(encoding="utf-8")
p.write_text("New content", encoding="utf-8")

# Listing files in a directory
for file in Path("reports").glob("*.txt"):
    print(file)

# Recursively find all .py files
for file in Path(".").rglob("*.py"):
    print(file)

Creating directories#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
Path("logs/2024/january").mkdir(parents=True, exist_ok=True)
# parents=True creates intermediate directories as needed
# exist_ok=True prevents an error if the directory already exists

7. Handling File Errors#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
try:
    with open("missing.txt", "r") as f:
        contents = f.read()
except FileNotFoundError:
    print("The file does not exist")
except PermissionError:
    print("You don't have permission to read this file")

Checking existence before opening (usually unnecessary)#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
from pathlib import Path

path = Path("data.txt")
if path.exists():
    contents = path.read_text()
else:
    print("File not found")

In most cases, it's more robust to simply try opening the file and catch FileNotFoundError, rather than checking existence first — the file could be deleted between the check and the actual open() call (a "race condition"). This principle is often summarized as "easier to ask forgiveness than permission" (EAFP), a common Python idiom.

8. Working with CSV Files#

The csv module handles the quoting and delimiter rules of CSV files correctly, which is safer than manually splitting on commas.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import csv

# Reading rows as lists
with open("people.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.reader(f)
    header = next(reader)
    for row in reader:
        print(row)   # ['Alice', '30', 'Engineer']

# Reading rows as dictionaries (uses the header row automatically)
with open("people.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["name"], row["age"])

# Writing CSV files
with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.writer(f)
    writer.writerow(["name", "age", "role"])
    writer.writerow(["Alice", 30, "Engineer"])

# Writing from dictionaries
with open("output.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["name", "age"])
    writer.writeheader()
    writer.writerow({"name": "Bob", "age": 25})

newline="" is recommended when reading/writing CSV files to prevent the csv module from misinterpreting line endings on some platforms.

9. Working with JSON Files#

The json module converts between Python objects and JSON text.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import json

data = {"name": "Alice", "age": 30, "skills": ["Python", "SQL"]}

# Writing JSON to a file
with open("data.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

# Reading JSON from a file
with open("data.json", "r", encoding="utf-8") as f:
    loaded = json.load(f)
print(loaded["name"])   # Alice

# Converting to/from JSON strings directly (no file involved)
json_string = json.dumps(data, indent=2)
parsed = json.loads(json_string)

10. File Position and Seeking#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
with open("data.txt", "r") as f:
    print(f.tell())        # current position in the file, 0 at start
    f.read(5)                # read 5 characters
    print(f.tell())         # now at position 5
    f.seek(0)                # jump back to the beginning
    print(f.read())          # reads the whole file again from the start

Notes#

• Always use with open(...) as f: instead of manual open()/close() pairs — it guarantees the file is closed even if an error occurs.
• "w" mode erases the existing file content immediately upon opening, even before any write() call — be careful not to overwrite data you meant to keep.
• Always specify encoding="utf-8" explicitly for text files to avoid platform-dependent default encoding issues.
• Use binary mode ("rb"/"wb") for anything that isn't plain text — attempting to open an image or PDF in text mode will corrupt the data or raise decoding errors.
• Prefer pathlib.Path over string concatenation or os.path for new code — it's more readable and handles cross-platform path separators automatically.
• The csv and json modules handle format-specific edge cases (quoting, escaping, nested structures) that manual string parsing gets wrong.

Best Practices Checklist#

• Always use the with statement when working with files
• Specify encoding="utf-8" explicitly for text file operations
• Use binary mode for non-text files (images, PDFs, executables)
• Prefer pathlib.Path over manual string-based path building
• Catch FileNotFoundError/PermissionError rather than checking existence beforehand when possible (EAFP over LBYL)
• Use csv/json modules instead of manual string parsing for those formats
• Pass newline="" when opening files for csv reading/writing
• Read large files line-by-line or in chunks instead of loading the entire file into memory with read()
• Use mkdir(parents=True, exist_ok=True) when creating directories that might already exist or need intermediate folders