# Enron source preparation

The public `corbt/enron-emails` dataset is the first real-source connector.
Its preparation API downloads and verifies a complete, explicitly
pinned revision. It also reuses a verified local cache without network access:

```python
from pathlib import Path
from pipewhy.adapters.enron import EnronConfiguration, prepare_source

source = prepare_source(EnronConfiguration(
    cache=Path("enron-cache"),
    revision="cfc06c758093d90993abce1a43668fb7357258a6",
    offline=False,
))
```

This revision contains 517,401 records in three Parquet files (493,482,449 bytes).
Preparation validates the pinned metadata, sizes and SHA256 digests before
publishing a complete cache manifest. Set `offline=True` to require cached history;
missing or corrupt files fail explicitly. Defaults allow 512 MiB of compressed
files per requested revision and 4 MiB per metadata response. Cache removal is
explicit; preparation never evicts other revisions. Configuration construction
does not connect or read ambient authentication tokens. No email text is logged.

After `poetry sync --extras enron`, stream the verified source into bounded C2
record buffers:

```python
from pipewhy.adapters.enron import iter_record_batches

record_count = sum(
    len(memoryview(batch.offsets)) for batch in iter_record_batches(source)
)
```

The reader validates all nine Arrow fields, preserves nulls and recipient ordering,
and converts UTC microseconds exactly. Defaults bound Arrow batches to 128 rows and
emitted C2 pools to 1,048,576 uint64 words with at most 65,536 records. A single
over-limit record fails explicitly. A full iteration checks shard contents and
file identity before/after reading and requires the declared record count. Stopping
iteration early only reads a prefix; it does not establish complete coverage.
`EnronReadError.records_seen` retains progress on a failed read without exposing
email values. The optional reader is PyArrow 26.0.0; importing the base package
does not import or require it.

`EnronAdapter(source)` exposes offline capabilities, bounded artifact discovery
and pinned reads. `discover()` returns repository artifact 0 and train artifact 1;
`discover(1)` selects the dataset. `lineage()` returns no edges with `UNKNOWN`
coverage and `UPSTREAM_LINEAGE_NOT_PROVIDED`. Shard paths and cache locations do
not determine artifact IDs. These methods operate on the prepared source and
never make network requests.

To scan the prepared cache from this checkout:

```powershell
poetry sync --extras cpu --extras enron
poetry run pipewhy scan enron --config examples/enron.toml --source-version cfc06c758093d90993abce1a43668fb7357258a6 --backend cpu --format json
```

The example requires cached files (`offline=true`); set `offline=false` to permit
explicit preparation during the scan. Cache and snapshot paths are relative to
the configuration file. The source accepts optional positive `source_limit` and
`metadata_limit` byte limits. `reference`, `cpu`, `cuda` and `auto` use the existing
backend rules. Dataset retention is the default; `--retain value` explicitly saves
normalized email records. Missing optional dependencies or preparation failures
exit 3 without a successful capture. A mid-read failure saves an unavailable
observation (exit 4) with its reason/progress, discarding partial fingerprints and
retained values. Processing failures close active source iterators.

Snapshots include pinned shard identities and unknown-lineage evidence. Reopen
them with `--store .artifacts/enron/snapshots`; saved queries need neither PyArrow,
NumPy nor access to the cache. An unchanged complete dataset self-diff exits 0;
lineage exits 4 because upstream pipeline lineage is unknown. Source mail dates
are never observation/operation clocks. See [blueprint/ENRON_CONTRACT.txt](../development/architecture.rst) and
[blueprint/ENRON_ACCEPTANCE.txt](../development/architecture.rst) for exact mapping, qualification and limitations.

Incident exports preserve their lineage context in `coordination.lineage_context`
(`cli-export-context-v2`), including unknown coverage and its reasons. JSON, CSV
companions and text retain these qualifiers; incomplete lineage makes incident
export exit 4 even for an empty incident table. DOT retains the complete lineage
page as base64 UTF8 JSON in `pipewhy_metadata_base64`.