Metaprogramming, Metaclasses, and AST

Overview#

Metaprogramming is writing code that inspects, generates, or modifies code — including classes and functions themselves — at runtime. Python supports this deeply: classes are ordinary objects, created by other objects called metaclasses; functions can be inspected and modified dynamically; and Python can even parse and manipulate its own source code via the Abstract Syntax Tree (AST). This is advanced material — most day-to-day Python code doesn't need it — but understanding it deepens your grasp of how frameworks like Django's ORM, dataclasses, and abc work internally. This notebook covers introspection, dynamic attribute access, descriptors, metaclasses, class decorators as a lighter alternative, and a practical introduction to the ast module.

1. Everything Is an Object: The Foundation#

In Python, classes themselves are objects — instances of type. This is the single most important fact to internalize before metaclasses make sense.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class Dog:
    pass

d = Dog()

print(type(d))       # <class '__main__.Dog'>   — d is an instance of Dog
print(type(Dog))       # <class 'type'>            — Dog is an instance of type!
print(type(type))       # <class 'type'>            — type is its own metaclass

type is the default metaclass for every class in Python. Just as Dog() creates an instance of Dog, type(...) (called with three arguments) can create a class itself.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# Normal class definition
class Dog:
    def bark(self):
        return "Woof!"

# Equivalent, built dynamically with type()
def bark(self):
    return "Woof!"

Dog = type("Dog", (), {"bark": bark})   # name, bases, namespace dict

d = Dog()
print(d.bark())   # Woof!

2. Introspection: Examining Objects at Runtime#

Python provides rich built-in tools for inspecting objects, functions, and classes without needing to know their structure in advance.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class Employee:
    def __init__(self, name, salary):
        self.name = name
        self.salary = salary

    def give_raise(self, amount):
        self.salary += amount


e = Employee("Alice", 50000)

print(dir(e))                    # lists all attributes and methods
print(vars(e))                    # {'name': 'Alice', 'salary': 50000} — instance __dict__
print(hasattr(e, "salary"))        # True
print(getattr(e, "salary"))         # 50000
print(getattr(e, "bonus", 0))        # 0 — default if the attribute doesn't exist
setattr(e, "bonus", 5000)             # dynamically add a new attribute
print(e.bonus)                          # 5000
delattr(e, "bonus")                      # dynamically remove it

import inspect
print(inspect.signature(e.give_raise))   # (amount)
print(inspect.getsource(Employee.give_raise))   # prints the method's source code

Checking callable behavior dynamically#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def process(obj, method_name, *args):
    method = getattr(obj, method_name, None)
    if callable(method):
        return method(*args)
    raise AttributeError(f"{method_name} is not a callable method")

process(e, "give_raise", 1000)
print(e.salary)   # 51000

3. Dynamic Attribute Access with __getattr__ and __setattr__#

Classes can intercept attribute access entirely, enabling proxy objects, lazy-loading, and validation logic that applies to any attribute name.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class LazyLoader:
    def __init__(self, source):
        self._source = source
        self._cache = {}

    def __getattr__(self, name):
        # only called when normal attribute lookup FAILS
        if name not in self._cache:
            print(f"Loading {name} from {self._source}...")
            self._cache[name] = f"data for {name}"
        return self._cache[name]


loader = LazyLoader("database")
print(loader.users)   # Loading users from database... / "data for users"
print(loader.users)   # (no "Loading" message — cached this time)

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class ValidatedModel:
    def __setattr__(self, name, value):
        if name == "age" and value < 0:
            raise ValueError("age cannot be negative")
        super().__setattr__(name, value)   # actually store the attribute


m = ValidatedModel()
m.age = 30       # OK
# m.age = -5       # raises ValueError

__getattr__ is only called when normal lookup fails (i.e. the attribute doesn't already exist); __getattribute__ is called for every attribute access and is far more dangerous to override — an infinite recursion is easy to introduce accidentally.

4. Descriptors#

A descriptor is an object that customizes attribute access via __get__, __set__, and/or __delete__. @property (seen in the OOP notebook) is actually implemented using descriptors under the hood — understanding descriptors explains why property works the way it does.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class PositiveNumber:
    def __init__(self, name):
        self.name = name

    def __get__(self, instance, owner):
        return instance.__dict__.get(self.name)

    def __set__(self, instance, value):
        if value < 0:
            raise ValueError(f"{self.name} must be positive")
        instance.__dict__[self.name] = value


class Product:
    price = PositiveNumber("price")     # a descriptor, reusable across classes
    quantity = PositiveNumber("quantity")

    def __init__(self, price, quantity):
        self.price = price
        self.quantity = quantity


p = Product(10, 5)
print(p.price)        # 10
# p.price = -5          # raises ValueError — enforced automatically

Unlike @property, which is defined once per attribute inside a class, a descriptor object like PositiveNumber can be reused across multiple attributes and multiple classes, since its logic is generic and parameterized by self.name.

5. Metaclasses#

A metaclass controls how classes themselves are created — it's "a class whose instances are classes." You customize class creation by subclassing type and overriding __new__ or __init__.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class UpperAttrMetaclass(type):
    def __new__(mcs, name, bases, namespace):
        # Transform all non-dunder attribute names to uppercase
        uppercase_namespace = {
            (key.upper() if not key.startswith("__") else key): value
            for key, value in namespace.items()
        }
        return super().__new__(mcs, name, bases, uppercase_namespace)


class Config(metaclass=UpperAttrMetaclass):
    timeout = 30
    retries = 3

print(Config.TIMEOUT)   # 30 — 'timeout' was transformed to 'TIMEOUT' at class creation
# print(Config.timeout)  # AttributeError — the original name no longer exists

A practical metaclass: enforcing a required method#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class RequireGreetMeta(type):
    def __new__(mcs, name, bases, namespace):
        cls = super().__new__(mcs, name, bases, namespace)
        if bases and "greet" not in namespace:   # skip check on the base class itself
            raise TypeError(f"{name} must implement a 'greet' method")
        return cls


class Greeter(metaclass=RequireGreetMeta):
    pass   # base class — no check applied here


class Robot(Greeter):
    def greet(self):
        return "Beep boop, hello"


# class BrokenRobot(Greeter):       # raises TypeError immediately at class DEFINITION time
#     pass

This check happens when the class itself is defined, not when an instance is created — catching a missing method far earlier than abc.abstractmethod would (which only errors at instantiation time).

When (not) to use metaclasses#

Metaclasses are genuinely rare in application code. Tim Peters, a prominent Python contributor, is often quoted as saying: if you're wondering whether you need metaclasses, you don't. Most use cases that seem to call for a metaclass — validation, registration, automatic method generation — can be achieved more simply and clearly with class decorators or __init_subclass__.

6. __init_subclass__: A Simpler Alternative to Metaclasses#

Since Python 3.6, __init_subclass__ lets a base class customize how its subclasses are created, without needing a full metaclass.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
class Plugin:
    registry = []

    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        Plugin.registry.append(cls)   # auto-register every subclass


class CSVPlugin(Plugin):
    pass

class JSONPlugin(Plugin):
    pass

print(Plugin.registry)   # [<class 'CSVPlugin'>, <class 'JSONPlugin'>]

This achieves a common metaclass use case (auto-registration) with dramatically less complexity, and should be preferred whenever the goal is customizing subclass creation specifically.

7. Class Decorators as a Lighter Alternative#

For simpler class-modification needs, a class decorator (covered in the Decorators notebook) is often clearer than either a metaclass or __init_subclass__.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
def add_repr(cls):
    def __repr__(self):
        attrs = ", ".join(f"{k}={v!r}" for k, v in vars(self).items())
        return f"{cls.__name__}({attrs})"
    cls.__repr__ = __repr__
    return cls


@add_repr
class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

print(Point(1, 2))   # Point(x=1, y=2)

8. The ast Module: Working with Python's Syntax Tree#

Every piece of Python source code is parsed into an Abstract Syntax Tree before execution. The ast module exposes this tree, enabling code analysis, linters, code transformers, and safer alternatives to eval().

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import ast

source = "x = 1 + 2 * 3"
tree = ast.parse(source)

print(ast.dump(tree, indent=2))

text
Ask AICopy
Module(
  body=[
    Assign(
      targets=[Name(id='x', ctx=Store())],
      value=BinOp(
        left=Constant(value=1),
        op=Add(),
        right=BinOp(
          left=Constant(value=2),
          op=Mult(),
          right=Constant(value=3))))],
  type_ignores=[])

Each node in the tree represents a syntactic construct — assignments, binary operations, function calls, and so on — mirroring the grammar of Python itself.

Walking the tree to analyze code#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import ast

code = """
def greet(name):
    print("Hello", name)

def add(a, b):
    return a + b
"""

tree = ast.parse(code)

for node in ast.walk(tree):
    if isinstance(node, ast.FunctionDef):
        print(f"Found function: {node.name}, args: {[a.arg for a in node.args.args]}")

# Found function: greet, args: ['name']
# Found function: add, args: ['a', 'b']

Safer literal evaluation with ast.literal_eval#

eval() executes arbitrary code and is dangerous on untrusted input. ast.literal_eval only parses safe Python literals (numbers, strings, lists, dicts, tuples, booleans, None) — it cannot execute function calls or arbitrary expressions, making it a much safer choice when parsing structured data from an untrusted or semi-trusted source.

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import ast

# Dangerous — eval() executes ANY code, including malicious code
# result = eval("__import__('os').system('rm -rf /')")

# Safe — only literal data structures are allowed
data = ast.literal_eval("{'name': 'Alice', 'age': 30, 'active': True}")
print(data)   # {'name': 'Alice', 'age': 30, 'active': True}

# ast.literal_eval("print('hacked')")   # raises ValueError — not a literal

A simple use case: finding all imports in a file#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
import ast

code = """
import os
import sys
from collections import Counter
from typing import List, Dict
"""

tree = ast.parse(code)
imports = []

for node in ast.walk(tree):
    if isinstance(node, ast.Import):
        imports.extend(alias.name for alias in node.names)
    elif isinstance(node, ast.ImportFrom):
        imports.extend(f"{node.module}.{alias.name}" for alias in node.names)

print(imports)
# ['os', 'sys', 'collections.Counter', 'typing.List', 'typing.Dict']

This is essentially how tools like linters, import sorters, and static analyzers inspect code without ever running it.

9. exec() and compile(): Executing Code Dynamically#

Python 3.12 (WASM)
EditCopyAsk AIRun Code
# compile() turns source code into a code object; exec() runs it
code_str = "result = 2 + 2"
compiled = compile(code_str, "<string>", "exec")

namespace = {}
exec(compiled, namespace)
print(namespace["result"])   # 4

Like eval(), exec() should never be used on untrusted input — it executes arbitrary code with the same privileges as the running program. Legitimate use cases are rare (plugin systems, code generation tools, certain testing frameworks) and should always sandbox or validate input carefully.

Notes#

• Metaclasses, descriptors, and AST manipulation are powerful but niche — most Python code, even fairly advanced code, never needs them directly. Frameworks and libraries use these tools so application developers don't have to.
• type(obj) returns an object's class; type(name, bases, namespace) (three arguments) dynamically creates a new class — this dual behavior is exactly why type is the default metaclass.
• __init_subclass__ should be preferred over a custom metaclass whenever the goal is simply reacting to or validating subclass creation.
• ast.literal_eval is the safe alternative to eval() for parsing trusted-format but untrusted-source data (e.g. a config value written as a Python literal).
• Never use eval() or exec() on input that could come from an untrusted source — this is a common and serious security vulnerability.

Best Practices Checklist#

• Reach for introspection (getattr, hasattr, inspect) before writing more complex dynamic-attribute logic
• Prefer @property over a custom descriptor unless the same logic needs to be reused across multiple attributes or classes
• Prefer __init_subclass__ or a class decorator over a custom metaclass whenever possible
• Only reach for a metaclass when subclass creation itself needs deep customization that __init_subclass__ can't express
• Use ast.literal_eval instead of eval() when parsing literal data from a file or string
• Never call eval() or exec() on input that could originate from an untrusted source
• Document any metaprogramming code thoroughly — it's inherently harder for other developers (and future you) to follow
• Consider whether a simpler tool (decorator, higher-order function) solves the problem before reaching for metaclasses or AST manipulation