Enron source preparation

The public corbt/enron-emails dataset is the first real-source connector. Its preparation API downloads and verifies a complete, explicitly pinned revision. It also reuses a verified local cache without network access:

from pathlib import Path
from pipewhy.adapters.enron import EnronConfiguration, prepare_source

source = prepare_source(EnronConfiguration(
    cache=Path("enron-cache"),
    revision="cfc06c758093d90993abce1a43668fb7357258a6",
    offline=False,
))

This revision contains 517,401 records in three Parquet files (493,482,449 bytes). Preparation validates the pinned metadata, sizes and SHA256 digests before publishing a complete cache manifest. Set offline=True to require cached history; missing or corrupt files fail explicitly. Defaults allow 512 MiB of compressed files per requested revision and 4 MiB per metadata response. Cache removal is explicit; preparation never evicts other revisions. Configuration construction does not connect or read ambient authentication tokens. No email text is logged.

After poetry sync --extras enron, stream the verified source into bounded C2 record buffers:

from pipewhy.adapters.enron import iter_record_batches

record_count = sum(
    len(memoryview(batch.offsets)) for batch in iter_record_batches(source)
)

The reader validates all nine Arrow fields, preserves nulls and recipient ordering, and converts UTC microseconds exactly. Defaults bound Arrow batches to 128 rows and emitted C2 pools to 1,048,576 uint64 words with at most 65,536 records. A single over-limit record fails explicitly. A full iteration checks shard contents and file identity before/after reading and requires the declared record count. Stopping iteration early only reads a prefix; it does not establish complete coverage. EnronReadError.records_seen retains progress on a failed read without exposing email values. The optional reader is PyArrow 26.0.0; importing the base package does not import or require it.

EnronAdapter(source) exposes offline capabilities, bounded artifact discovery and pinned reads. discover() returns repository artifact 0 and train artifact 1; discover(1) selects the dataset. lineage() returns no edges with UNKNOWN coverage and UPSTREAM_LINEAGE_NOT_PROVIDED. Shard paths and cache locations do not determine artifact IDs. These methods operate on the prepared source and never make network requests.

To scan the prepared cache from this checkout:

poetry sync --extras cpu --extras enron
poetry run pipewhy scan enron --config examples/enron.toml --source-version cfc06c758093d90993abce1a43668fb7357258a6 --backend cpu --format json

The example requires cached files (offline=true); set offline=false to permit explicit preparation during the scan. Cache and snapshot paths are relative to the configuration file. The source accepts optional positive source_limit and metadata_limit byte limits. reference, cpu, cuda and auto use the existing backend rules. Dataset retention is the default; --retain value explicitly saves normalized email records. Missing optional dependencies or preparation failures exit 3 without a successful capture. A mid-read failure saves an unavailable observation (exit 4) with its reason/progress, discarding partial fingerprints and retained values. Processing failures close active source iterators.

Snapshots include pinned shard identities and unknown-lineage evidence. Reopen them with --store .artifacts/enron/snapshots; saved queries need neither PyArrow, NumPy nor access to the cache. An unchanged complete dataset self-diff exits 0; lineage exits 4 because upstream pipeline lineage is unknown. Source mail dates are never observation/operation clocks. See blueprint/ENRON_CONTRACT.txt and blueprint/ENRON_ACCEPTANCE.txt for exact mapping, qualification and limitations.

Incident exports preserve their lineage context in coordination.lineage_context (cli-export-context-v2), including unknown coverage and its reasons. JSON, CSV companions and text retain these qualifiers; incomplete lineage makes incident export exit 4 even for an empty incident table. DOT retains the complete lineage page as base64 UTF8 JSON in pipewhy_metadata_base64.