Local parallel Enron scans

The pipewhy.execution API prepares immutable shard plans, runs bounded local worker processes, preserves accepted results across restart, and publishes through the same local snapshot store used by ordinary scans. Install the cpu,enron extras and prepare the pinned cache described in the Enron guide before using offline mode. Save this example as a Python script; spawned workers require an importable main module and its execution guard.

from pathlib import Path

from pipewhy.adapters.enron import EnronConfiguration
from pipewhy.execution import execute_run, prepare_enron_execution, publish_execution
from pipewhy.snapshots import LocalSnapshotStore


def main():
    cache = Path(".artifacts/enron/source")
    prepared = prepare_enron_execution(
        EnronConfiguration(
            cache,
            "cfc06c758093d90993abce1a43668fb7357258a6",
            offline=True,
        ),
        ".artifacts/execution/plans",
        backend="cpu",
        workers=3,
    )
    outcome = execute_run(prepared.directory, cache, ".artifacts/execution/runs")
    if not outcome.report.complete:
        raise RuntimeError(f"Incomplete run: {outcome.directory}")
    snapshot = publish_execution(
        outcome.directory,
        prepared.directory,
        LocalSnapshotStore(".artifacts/enron/snapshots"),
    )
    print(snapshot.metadata.snapshot_id)


if __name__ == "__main__":
    main()

resume_execution(run_directory, prepared.directory, cache) resumes an existing run with its original plan and finite attempt budget. Accepted partitions stay counted once. Publication requires every planned partition, including empty ones; repeating publication verifies the existing snapshot and preserves its contents. Parallel capture retains dataset fingerprints and source metadata. Saved snapshots can be queried/exported with the core installation. The command-line scan continues to use the ordinary scan path; parallel execution uses the Python API.