ORC sources

ORC capture is available in the development source. The published 0.1.0.dev0 package predates this functionality. Install the optional orc extra when using a build that includes this reader. It uses PyArrow; the core package does not need that dependency.

Configure one local ORC file, an explicit schema and resource budgets:

schema_version = 1

[store]
path = "snapshots"

[sources.orders]
kind = "orc"
path = "orders.orc"
source_id = "orders"
schema = { format = "pipewhy.orc-schema", version = 1, fields = [{ id = 0, name = "amount", type = "int64", nullable = false }] }
limits = { source_bytes = 100000, metadata_bytes = 90000, footer_decoded_bytes = 90000, compression_chunk_bytes = 262144, stripe_records = 100, stripe_count = 10, field_bytes = 1000, field_count = 10, postscript_bytes = 255, batch_records = 2, batch_words = 1000, max_records = 100 }
worker = { memory_bytes = 1073741824, timeout_ms = 15000, frame_bytes = 4000, output_bytes = 10000 }

Supply the lowercase SHA-256 of the exact file bytes as --source-version to pipewhy scan. A completed snapshot can be reopened, compared and exported without the original file. Failed reads, source changes, timeouts and corrupt worker results are reported as unavailable observations without retained partial records. Counts describe rows observed after verification; upstream lineage is unknown. The source is one local file, not a table, metastore or remote object.

The initial subset accepts ORC version 0.12, uncompressed or ZLIB-compressed streams, and explicitly selected flat signed 64-bit integers and UTF-8 strings. Every selected field has a stable ID, name, type and nullable policy. Missing columns, nested types anywhere in the schema, unsupported selected types, encryption and unsupported metadata are rejected. The reader checks a declared non-null policy against actual values; ORC does not necessarily preserve the writer’s original required-field declaration. Projection follows file stripe and row order while retaining the configured field identities. Schema evolution is not inferred. Strict normalization is the default; use --profile logical-v1 to opt into logical equivalence.

The limits bound the source, encoded metadata and postscript, decompressed file footer, compression chunk size, stripe and record counts, field bytes and result batches. PyArrow materializes an entire projected stripe before PipeWhy can check its row count, so stripe_records and batch size are not pre-allocation memory bounds. A separate supervised worker enforces a deadline and memory budget during native decoding. On Windows this budget limits job committed memory; on Linux x86_64 it limits process virtual address space. These are different measures, not portable resident-memory limits. Unsupported worker platforms fail closed.

The lower-level read_orc_batches API accepts either a local path or a trusted verified source opener. An opener must enforce the byte limit and digest and verify the source on exit. This interface does not implement an S3-compatible connector; remote storage is separate work.