ORC sources
===========

ORC capture is available in the development source. The published
``0.1.0.dev0`` package predates this functionality. Install the optional
``orc`` extra when using a build that includes this reader. It uses PyArrow;
the core package does not need that dependency.

Configure one local ORC file, an explicit schema and resource budgets:

.. code-block:: toml

   schema_version = 1

   [store]
   path = "snapshots"

   [sources.orders]
   kind = "orc"
   path = "orders.orc"
   source_id = "orders"
   schema = { format = "pipewhy.orc-schema", version = 1, fields = [{ id = 0, name = "amount", type = "int64", nullable = false }] }
   limits = { source_bytes = 100000, metadata_bytes = 90000, footer_decoded_bytes = 90000, compression_chunk_bytes = 262144, stripe_records = 100, stripe_count = 10, field_bytes = 1000, field_count = 10, postscript_bytes = 255, batch_records = 2, batch_words = 1000, max_records = 100 }
   worker = { memory_bytes = 1073741824, timeout_ms = 15000, frame_bytes = 4000, output_bytes = 10000 }

Supply the lowercase SHA-256 of the exact file bytes as ``--source-version`` to
``pipewhy scan``. A completed snapshot can be reopened, compared and exported
without the original file. Failed reads, source changes, timeouts and corrupt
worker results are reported as unavailable observations without retained partial
records. Counts describe rows observed after verification; upstream lineage is
unknown. The source is one local file, not a table, metastore or remote object.

The initial subset accepts ORC version 0.12, uncompressed or ZLIB-compressed
streams, and explicitly selected flat signed 64-bit integers and UTF-8 strings.
Every selected field has a stable ID, name, type and nullable policy. Missing
columns, nested types anywhere in the schema, unsupported selected types,
encryption and unsupported metadata are rejected. The reader checks a declared
non-null policy against actual values; ORC does not necessarily preserve the
writer's original required-field declaration. Projection follows file stripe and
row order while retaining the configured field identities. Schema evolution is
not inferred. Strict normalization is the default; use ``--profile logical-v1``
to opt into logical equivalence.

The limits bound the source, encoded metadata and postscript, decompressed file
footer, compression chunk size, stripe and record counts, field bytes and result
batches. PyArrow materializes an entire projected stripe before PipeWhy can check
its row count, so ``stripe_records`` and batch size are not pre-allocation memory
bounds. A separate supervised worker enforces a deadline and memory budget during
native decoding. On Windows this budget limits job committed memory; on Linux
x86_64 it limits process virtual address space. These are different measures, not
portable resident-memory limits. Unsupported worker platforms fail closed.

The lower-level ``read_orc_batches`` API accepts either a local path or a trusted
verified source opener. An opener must enforce the byte limit and digest and
verify the source on exit. This interface does not implement an S3-compatible
connector; remote storage is separate work.
