0001 0101 pydoit Build for me.micahrl.com

A pydoit translation of the project’s vars.mk Makefile.

Quick Start

# Install pydoit
pip install doit

# Run from the repo root:
doit -f content/blog/comparing-build-systems/doit/dodo.py build_prod_obverse

# Or copy/symlink to repo root and run without -f:
ln -s content/blog/comparing-build-systems/doit/dodo.py dodo.py
doit build_prod_obverse

Available Tasks

Run doit -f dodo.py list to see all tasks.

How This Compares to Make

What Works Well

1. No parse-time overhead — by default

The Makefile’s core problem: $(shell ...) runs on every make invocation, even make help. pydoit avoids this entirely because dodo.py is a Python module — functions are defined but not called until doit decides a task needs to run.

# Python: _hugo_sources() is a function; it runs only when a task executes
def _hugo_sources():
    return glob.glob("archetypes/**/*", recursive=True) + ...

# Make: $(shell find ...) runs at PARSE TIME on every invocation
HUGOSOURCES = $(shell find archetypes config content ... -type f)

This is the vars.mk pattern’s whole motivation, already solved by the language.

2. calc_dep cleanly replaces -include + $(HUGOSOURCES)

The vars.mk Makefile uses a two-phase trick to get an incremental source list:

1. build/hugo-sources.vars.mk is a target that writes HUGOSOURCES := ...
2. -include build/hugo-sources.vars.mk causes make to re-read the Makefile
3. Hugo targets then depend on $(HUGOSOURCES)

pydoit has calc_dep for exactly this. A task’s action can return a dict of additional dependencies, and any task that declares calc_dep: ["that_task"] gets those deps merged in — no re-parsing needed:

# This task's action returns dynamic file_dep
def task_calc_hugosources():
    def calc():
        return {"file_dep": _hugo_sources()}  # returns a dict!
    return {
        "actions": [calc],
        "file_dep": HUGO_SOURCE_DIRS,  # recompute when dirs change
    }

# Hugo build tasks use calc_dep to get the dynamic source list
def _hugobuild(env):
    return {
        "calc_dep": ["calc_hugosources"],  # merges file_dep from above
        "file_dep": ["build/git-info.json"],
        "targets": [f"public/{env}/.sentinel"],
        ...
    }

Make’s -include dance is essentially reimplementing what calc_dep provides natively.

3. Python functions replace define/endef macros

Make’s macro syntax is awkward. Python closures are natural and handle argument passing cleanly:

# pydoit: regular Python function, env captured by closure
def _hugobuild(env):
    def build(targets):
        ...hugo with env...
    return {"actions": [build], "targets": [f"public/{env}/.sentinel"]}

def task_build_prod_obverse():
    return _hugobuild("prod-obverse")

# Make: define/endef macro, called with $(call ...)
define hugobuild
    @echo "Building for $(1)"
    hugo --destination public/$(1) --environment $(1)
endef

public/prod-obverse/.sentinel: ...
    $(call hugobuild,prod-obverse)

4. JSON replaces .vars.mk fragments

CFN outputs become a JSON file instead of a Makefile fragment. Reading them in Python is straightforward; no $$(cat ...) workarounds needed:

# pydoit: read JSON, pass values directly
base = _read_json("build/cfn-test-obverse-base.json")
subprocess.run([..., f"ContentBucketName={base['ContentBucketName']}", ...])

# Make: read from vars.mk via -include, then use make variable
# (or the original pattern: $$(cat public/cfn/ContentBucketName))
--parameter-overrides ContentBucketName=$(TEST_OBVERSE_ContentBucketName)

5. Generator functions for paired tasks

task_cfn_base and task_cfn_distrib are generator functions that yield named subtasks. This cleanly handles the two nearly-identical stack pairs without copy-paste:

def task_cfn_base():
    yield _cfn_base_task("test-obverse", "test-obverse-base", ...)
    yield _cfn_base_task("test-reverse", "test-reverse-base", ...)

These become cfn_base:test-obverse and cfn_base:test-reverse — the colon grouping is automatic.

6. Explicit task ordering, no left-to-right dependency

Make’s deploy-test-obverse relied on left-to-right ordering of its prerequisites for CFN base → Hugo upload → CFN distribution. In pydoit, task_dep runs tasks in dependency order, and the ordering is made explicit through the dep chain: cfn_distrib:test-obverse has file_dep on the base JSON, so it won’t run before the base task.

What Doesn’t Work Well / Trade-offs

1. No pattern rules (same gap as Task)

Make’s %.exifstripped: | % handles per-image tracking elegantly. pydoit has no pattern rule equivalent — you loop in Python. This loses per-file incremental tracking, processing all images whenever any source directory changes.

2. Task names use underscores, not dashes

Python function names can’t contain hyphens. All task names become underscored (build_prod_obverse, not build-prod-obverse). pydoit does allow dash names via the basename attribute, but it adds noise.

3. .doit.db state file

pydoit stores its dependency graph in a .doit.db SQLite file at the working directory. This is invisible in normal use but needs to be excluded from version control and cleaned with doit forget or by deleting the file. Make uses only filesystem mtimes — no extra state file.

4. calc_dep ordering subtlety

calc_dep tasks run before the dependent task’s up-to-date check. If calc_hugosources itself hasn’t run yet (clean build), pydoit runs it first, then re-evaluates the hugobuild task with the merged deps. This is correct but less obvious than Make’s explicit dependency chain.

5. Must be installed

Like Task, pydoit requires installation (pip install doit). Make is pre-installed on virtually every Unix system.

6. No built-in parallel execution

pydoit runs tasks sequentially by default. Parallel execution requires doit -n 4 (number of workers). Task’s deps: are parallel by default; Make requires -j. Neither default is obviously correct for deploy workflows where ordering matters.

Targets Not Translated

• dev: Interactive hugo server; not interesting for build system comparison.
• macos-dev-*: macOS-specific; not relevant.
• list-pagerefs, list-shortcodes: One-liner rg commands.

Structural Differences Summary

Verdict

pydoit is a natural fit for the vars.mk pattern — it solves the same problem (deferred computation of expensive values) without the two-phase -include dance. calc_dep is a first-class version of what vars.mk implements with Makefile hacks.

The key wins:

1. No parse-time overhead by default — the whole motivation for vars.mk is solved by the language
2. calc_dep replaces -include + $(HUGOSOURCES) cleanly and explicitly
3. Python helper functions are more readable than define/endef macros
4. JSON for stage outputs is cleaner than .vars.mk fragments

The main losses compared to the vars.mk Makefile:

1. No pattern rules (same problem as all other alternatives)
2. .doit.db state file needs management
3. Task names use underscores instead of dashes
4. Must be installed

Compared to the other build systems in this series: pydoit is the closest conceptual match to the vars.mk pattern, because both are fundamentally about staged computation with cached intermediate results. Task is more approachable day-to-day. Shake has the most expressive dependency model. pydoit’s advantage is that Python is already ubiquitous in the toolchain.