Source adapters#

Pinned public Enron dataset preparation; network access is always explicit.

class pipewhy.adapters.enron.EnronConfiguration(cache, revision, offline=False, source_limit=536870912, metadata_limit=4194304)#

Construction validates configuration without making network requests.

Parameters:

  • cache (Path)

  • revision (str)

  • offline (bool)

  • source_limit (int)

  • metadata_limit (int)

class pipewhy.adapters.enron.PinnedSource(directory: pathlib.Path, manifest: pipewhy.adapters.enron.cache.SourceManifest)#
Parameters:
class pipewhy.adapters.enron.SourceFile(path: str, size: int, sha256: str)#
Parameters:

  • path (str)

  • size (int)

  • sha256 (str)

class pipewhy.adapters.enron.SourceManifest(revision: str, expected_records: int, files: tuple[pipewhy.adapters.enron.cache.SourceFile, ...])#
Parameters:

  • revision (str)

  • expected_records (int)

  • files (tuple[SourceFile, ...])

pipewhy.adapters.enron.prepare_source(configuration)#

Verify the whole requested revision; offline mode never opens a URL.

No file or complete manifest is published before its integrity is checked. Corrupt existing entries fail explicitly rather than being silently repaired.

Parameters:

configuration (EnronConfiguration)

Return type:

PinnedSource

exception pipewhy.adapters.enron.EnronReadError(reason, records_seen)#
Parameters:

  • reason (str)

  • records_seen (int)

pipewhy.adapters.enron.iter_record_batches(source, *, profile=Profile.STRICT, batch_size=128, record_limit=65536, word_limit=1048576)#

A complete iteration checks every shard and the full expected row count.

Early consumer termination is a prefix read, never proof of dataset coverage. Both emitted C2 pools and Arrow row batches have explicit independent limits.

Parameters:

  • source (PinnedSource)

  • profile (Profile | str)

  • batch_size (int)

  • record_limit (int)

  • word_limit (int)

Return type:

Generator[RecordBatch]

class pipewhy.adapters.enron.EnronAdapter(source)#

Operate on an explicitly prepared revision; methods never open network URLs.

Parameters:

source (PinnedSource)

pipewhy.adapters.enron.iter_shard_batches(source, index, expected_records, *, profile=Profile.STRICT, batch_size=128, record_limit=65536, word_limit=1048576)#

Read one explicitly planned partition, never a complete dataset observation.

Parameters:

  • source (PinnedSource)

  • index (int)

  • expected_records (int)

  • profile (Profile | str)

  • batch_size (int)

  • record_limit (int)

  • word_limit (int)

Return type:

Generator[RecordBatch]

Deterministic, captured local JSON fixture source.

class pipewhy.adapters.local.LocalFixtureAdapter(directory)#

Capture validated C4 fixture inputs once; all results own their output buffers.

Parameters:

directory (str | Path)