Local execution planning#

pipewhy.execution.prepare_enron_execution(configuration, root, ...) verifies pinned shard schemas and row counts, then atomically persists an immutable plan and its source manifest. This prepares work; it does not dispatch workers or publish a completed observation. The existing CLI scan remains sequential.

Plans bind source/version/scope, C3 context, exact adjacent ranges and execution policy. Their IDs exclude cache locations, and conflicting or corrupt persisted documents fail explicitly. load_plan(directory) and load_source(directory) reopen verified planning metadata without a Parquet dependency.

Planning supports explicit reference or cpu worker profiles. Defaults record 4 workers, 2 attempts, a 600,000 ms operational timeout and 128 Arrow rows per batch. Limits are 61 workers, 10 attempts, a 24-hour timeout and 65,536 Arrow rows; plans/source documents are bounded to 4 MiB. These are implementation policies, not performance guarantees.

execute_partition(directory, cache, run_id, index, attempt=1) executes one prepared Enron shard without network or journal/snapshot writes. It returns a versioned WorkerResult; call result.validate(plan, run_id) before consuming its state. Results bind run/plan/partition, revision, source/context identity and backend, and verify exact C3 count/global range. Failed results carry safe reason codes and progress with no comparable state. The shard API explicitly represents partition scope; it never claims a complete dataset on its own. The same C2 converter serves full and shard reads.

RunJournal.create(root, plan) creates a new journal with its own run ID. journal.accept(result) admits a complete matching result once; an identical retransmission returns False, and conflicting deliveries fail. Failed envelopes have a separate immutable history. Journal payloads have checksums, and reopening validates identities before reconstructing state.

journal.reconcile() returns explicit complete/failed/missing partition IDs and accepted row progress. Its state remains None until every planned partition, including zero-row shards, has a valid complete receipt. Merging follows manifest order. A source failure or missing partition cannot become a complete fingerprint.

Coordinated dispatch, durable attempt budgets, retries/resume, OS ownership and replay-safe C5 publication are implemented. Use execute_run, resume_execution and publish_execution as described in Parallel scans. The CLI scan stays sequential. See blueprint/EXECUTION_CONTRACT.txt for qualification evidence.