Regular Expressions and Text Processing

Overview#

Regular expressions (regex) are a compact language for matching patterns in text — validating formats, extracting data, searching, and replacing. Python's built-in re module implements the full regex engine. This notebook covers regex syntax fundamentals, the core re functions, groups and capturing, common patterns, compiled patterns for performance, and general text-processing techniques that go beyond what plain string methods can handle.

1. When to Use Regex (and When Not To)#

Regex is the right tool when you need to match patterns with variability — an email format, a phone number, any line starting with a date. For simple, fixed substring checks, plain string methods are clearer and faster.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
text = "Order #12345 shipped on 2024-03-15"

# Simple substring check — no regex needed
if "shipped" in text:
    print("Order has shipped")

# Pattern with variability — regex is the right tool
import re
match = re.search(r"\d{4}-\d{2}-\d{2}", text)
if match:
    print(f"Date found: {match.group()}")   # Date found: 2024-03-15

2. Regex Syntax Fundamentals#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

print(re.search(r"\d+", "abc123def").group())       # 123
print(re.search(r"[A-Z]\w+", "hello World").group())  # World
print(re.search(r"cat|dog", "I have a dog").group())    # dog

Raw strings#

Regex patterns should almost always use Python raw strings (r"...") to avoid conflicts between regex escape sequences (\d, \s) and Python's own string escape sequences (\n, \t).

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# Without raw string: \d is not a recognized Python escape, but works by
# accident in older versions; explicit raw strings avoid ambiguity entirely
pattern = r"\d{3}-\d{4}"   # correct, unambiguous

3. Core re Module Functions#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

text = "The rain in Spain falls mainly on the plain"

# re.search: find the FIRST match anywhere in the string
match = re.search(r"\bain\b", text)
print(match)   # None — \b is a word boundary, "ain" alone doesn't appear as a whole word

match = re.search(r"ain", text)
print(match.group())    # 'ain' (from "rain")
print(match.start())     # 4 — index where the match begins
print(match.end())        # 7 — index where the match ends

# re.match: only checks from the START of the string
print(re.match(r"The", text))     # matches, since text starts with "The"
print(re.match(r"rain", text))     # None — "rain" isn't at the start

# re.fullmatch: the ENTIRE string must match the pattern
print(re.fullmatch(r"\d+", "12345"))   # matches
print(re.fullmatch(r"\d+", "123a5"))    # None

# re.findall: return ALL non-overlapping matches as a list
words_with_ain = re.findall(r"\w*ain\w*", text)
print(words_with_ain)   # ['rain', 'Spain', 'mainly', 'plain']

# re.finditer: like findall, but returns Match objects (with position info)
for m in re.finditer(r"\w*ain\w*", text):
    print(m.group(), m.start())

# re.sub: replace matches with a string
print(re.sub(r"ain", "XXX", text))
# "The rXXX in SpXXX falls mXXXly on the plXXX"

# re.split: split a string using a regex pattern as the delimiter
print(re.split(r"\s+", "one   two    three"))   # ['one', 'two', 'three']

4. Groups and Capturing#

Parentheses () create a capturing group, letting you extract specific parts of a match rather than the whole thing.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

text = "Contact: alice@example.com or bob@work.org"

pattern = r"(\w+)@(\w+\.\w+)"
for match in re.finditer(pattern, text):
    print(f"Full match: {match.group()}")     # alice@example.com
    print(f"Username:   {match.group(1)}")      # alice
    print(f"Domain:      {match.group(2)}")      # example.com

Named groups#

Named groups make extracted data self-documenting instead of relying on positional indices.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
pattern = r"(?P<username>\w+)@(?P<domain>\w+\.\w+)"
match = re.search(pattern, "alice@example.com")

print(match.group("username"))   # alice
print(match.group("domain"))      # example.com
print(match.groupdict())           # {'username': 'alice', 'domain': 'example.com'}

Non-capturing groups#

Use (?:...) to group without capturing, useful when you need grouping for alternation or quantifiers but don't need to extract that part.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
pattern = r"(?:https?://)?(\w+\.\w+)"   # scheme is optional, not captured
match = re.search(pattern, "https://example.com")
print(match.group(1))   # example.com

5. Common Real-World Patterns#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

# Email validation (simplified — real-world validation is more nuanced)
email_pattern = r"^[\w.+-]+@[\w-]+\.[a-zA-Z]{2,}$"
print(bool(re.match(email_pattern, "user@example.com")))   # True

# Phone number (US-style)
phone_pattern = r"^\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}$"
print(bool(re.match(phone_pattern, "(555) 123-4567")))    # True

# URL extraction
url_pattern = r"https?://[^\s]+"
text = "Visit https://example.com or http://test.org for details"
print(re.findall(url_pattern, text))
# ['https://example.com', 'http://test.org']

# Extracting numbers, including decimals
numbers_pattern = r"-?\d+\.?\d*"
print(re.findall(numbers_pattern, "Price: $19.99, discount: -5, quantity: 3"))
# ['19.99', '5', '3']   (note: minus sign handling depends on exact context)

# Removing extra whitespace
messy = "This   has\n\nextra    whitespace"
cleaned = re.sub(r"\s+", " ", messy).strip()
print(cleaned)   # "This has extra whitespace"

6. Greedy vs Lazy Quantifiers#

By default, quantifiers (*, +, {n,m}) are greedy — they match as much text as possible. Adding ? after a quantifier makes it lazy, matching as little as possible.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

html = "<b>bold</b> and <i>italic</i>"

# Greedy: matches from the FIRST < to the LAST >
greedy = re.search(r"<.+>", html)
print(greedy.group())   # <b>bold</b> and <i>italic</i>

# Lazy: matches as little as possible, stopping at the first >
lazy = re.search(r"<.+?>", html)
print(lazy.group())   # <b>

# Finding all tags requires the lazy version
tags = re.findall(r"<.+?>", html)
print(tags)   # ['<b>', '</b>', '<i>', '</i>']

This greedy-vs-lazy distinction is one of the most common sources of regex bugs — always consider whether a pattern should stop at the first possible match or consume as much as it can.

7. Flags#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

text = "Hello WORLD"

# re.IGNORECASE (or re.I): case-insensitive matching
print(re.findall(r"hello", text, re.IGNORECASE))   # ['Hello']

# re.MULTILINE (or re.M): ^ and $ match the start/end of EACH line, not just the whole string
multiline_text = "line one\nline two\nline three"
print(re.findall(r"^line", multiline_text, re.MULTILINE))   # ['line', 'line', 'line']

# re.DOTALL (or re.S): . also matches newline characters
text_with_newline = "start\nend"
print(re.search(r"start.end", text_with_newline))              # None (. doesn't match \n by default)
print(re.search(r"start.end", text_with_newline, re.DOTALL))     # matches

Multiple flags can be combined with |: re.IGNORECASE | re.MULTILINE.

8. Compiling Patterns for Reuse#

When the same pattern is used repeatedly (e.g. inside a loop), compiling it once with re.compile() avoids re-parsing the pattern on every call, and makes the code slightly cleaner to read.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import re

email_pattern = re.compile(r"^[\w.+-]+@[\w-]+\.[a-zA-Z]{2,}$")

emails = ["alice@example.com", "not-an-email", "bob@work.org"]
valid_emails = [e for e in emails if email_pattern.match(e)]
print(valid_emails)   # ['alice@example.com', 'bob@work.org']

9. Beyond Regex: General Text Processing#

Not every text-processing task needs regex — Python's string methods and the textwrap/unicodedata modules often handle common cases more simply.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
text = "  The Quick Brown Fox  "

# Basic cleanup without regex
cleaned = text.strip().lower()
print(cleaned)   # "the quick brown fox"

# Splitting on multiple possible delimiters (regex needed here)
import re
parts = re.split(r"[,;|]", "apple,banana;cherry|date")
print(parts)   # ['apple', 'banana', 'cherry', 'date']

# Wrapping long text to a fixed width
import textwrap
long_text = "This is a very long sentence that needs to be wrapped nicely."
print(textwrap.fill(long_text, width=20))
# This is a very long
# sentence that needs
# to be wrapped
# nicely.

# Counting word frequency (combining split + Counter, no regex needed for simple cases)
from collections import Counter
words = "the quick brown fox jumps over the lazy dog the fox runs".split()
print(Counter(words).most_common(2))   # [('the', 3), ('fox', 2)]

Notes#

• Always use raw strings (r"...") for regex patterns to avoid escape sequence conflicts with Python's own string syntax.
• re.match only anchors at the start of the string; re.search looks anywhere; re.fullmatch requires the entire string to match — mixing these up is a common source of bugs.
• Capturing groups add slight overhead; use non-capturing groups (?:...) when you only need grouping for structure, not extraction.
• Greedy quantifiers can accidentally match far more text than intended, especially with . and * combined — test patterns against edge cases, not just the happy path.
• For genuinely complex or security-sensitive parsing (like full email or URL validation per spec), a dedicated parsing library is often more correct and maintainable than a hand-written regex.
• Poorly constructed regex patterns can suffer from catastrophic backtracking on certain inputs, causing severe slowdowns — keep patterns as specific and simple as the task allows.

Best Practices Checklist#

• Always use raw strings (r"...") for regex patterns
• Choose match/search/fullmatch deliberately based on where the pattern should apply
• Use named groups ((?P<name>...)) for patterns with multiple captured parts, to keep extraction code readable
• Use non-capturing groups (?:...) when grouping is only structural
• Be explicit about greedy vs lazy quantifiers when a pattern could match more than intended
• Compile patterns with re.compile() when reused across many calls or inside a loop
• Prefer plain string methods (split, strip, replace) over regex for simple, fixed-pattern tasks
• Test regex patterns against edge cases and malformed input, not just the expected format
• Watch for catastrophic backtracking in complex nested quantifiers