Local execution#

Versioned local execution plans with source-bound partition identities.

class pipewhy.execution.PreparedExecution(source: pipewhy.adapters.enron.cache.PinnedSource, plan: pipewhy.execution.plan.ExecutionPlan, directory: pathlib.Path)#
Parameters:

  • source (PinnedSource)

  • plan (ExecutionPlan)

  • directory (Path)

pipewhy.execution.prepare_enron_execution(configuration, root, *, scope=0, profile=Profile.STRICT, mode=Mode.MULTISET, backend='reference', workers=4, max_attempts=2, timeout_ms=600000, batch_size=128)#

Verify shard/schema/count metadata and persist the complete inventory.

This prepares a plan, not a completed execution or a C5 observation. Cache preparation is explicit here; persisted source identities contain no host path.

Parameters:

  • configuration (EnronConfiguration)

  • root (str | Path)

  • scope (int)

  • profile (Profile | str)

  • mode (Mode | str)

  • backend (str)

  • workers (int)

  • max_attempts (int)

  • timeout_ms (int)

  • batch_size (int)

Return type:

PreparedExecution

class pipewhy.execution.ExecutionOutcome(directory: pathlib.Path, report: pipewhy.execution.journal.Reconciliation)#
Parameters:

  • directory (Path)

  • report (Reconciliation)

pipewhy.execution.execute_run(plan_directory, cache, root, *, worker=<function execute_partition>)#

Create an explicit new run; this API never replaces a prior run’s history.

Parameters:

  • plan_directory (str | Path)

  • cache (str | Path)

  • root (str | Path)

  • worker (PartitionWorker)

Return type:

ExecutionOutcome

pipewhy.execution.resume_execution(run_directory, plan_directory, cache, *, worker=<function execute_partition>)#

Resume one run under its OS lock; accepted results and retry counts survive.

Parameters:

  • run_directory (str | Path)

  • plan_directory (str | Path)

  • cache (str | Path)

  • worker (PartitionWorker)

Return type:

ExecutionOutcome

pipewhy.execution.publish_execution(run_directory, plan_directory, store)#

Replay a durable publication intent without replacing an existing snapshot.

All partitions, including empty shards, must have accepted current-attempt results. The run ID is the snapshot ID, making publication restart-idempotent. Source values are not retained and source files are not reread at publication.

Parameters:
Return type:

Snapshot