Parquet files
=============

The development source can read one verified local Parquet file through the
Python API. The published ``0.1.0.dev0`` package predates this functionality.
From a source checkout, install the optional capability with
``python -m pip install '.[parquet]'``. The core installation does not require
PyArrow.

The first reader supports explicitly selected flat signed 64-bit integer and
UTF-8 string columns. Declare each field's stable ID, name, type and nullability;
missing selected fields, nested columns, duplicate names, unsupported selected
types and incompatible schemas fail explicitly. Extra flat columns need not be
selected. Strict representation is the default; pass ``Profile.LOGICAL`` to opt
in to logical normalization.

.. code-block:: python

   from hashlib import file_digest
   from pathlib import Path

   from pipewhy.formats import (
       ParquetField, ParquetLimits, ParquetSchema, ParquetWorkerLimits,
       read_parquet_batches,
   )

   path = Path("orders.parquet")
   with path.open("rb") as source:
       version = file_digest(source, "sha256").hexdigest()

   schema = ParquetSchema((
       ParquetField(1, "order_id", "int64"),
       ParquetField(2, "label", "text", nullable=True),
   ))
   limits = ParquetLimits(
       source_bytes=10_000_000, footer_bytes=1_000_000,
       thrift_string_bytes=100_000, thrift_container_items=100_000,
       field_bytes=100_000, field_count=100, row_groups=100,
       batch_records=1_000, batch_words=100_000, max_records=100_000,
   )
   worker = ParquetWorkerLimits(
       memory_bytes=1_073_741_824, timeout_ms=30_000,
       frame_bytes=2_000_000, output_bytes=50_000_000,
   )
   batches = read_parquet_batches(
       path, sha256_hex=version, schema=schema, limits=limits,
       worker_limits=worker,
   )

All limits are explicit. The values above are examples rather than portable
defaults: native libraries and datasets need different budgets. The reader
stages the input and bounded output in temporary local files, verifies the
source before and after processing, and returns batches only after successful
worker completion and output validation. Checksums on Parquet pages are verified
when present; the source SHA-256 is checked independently.

The current worker supports Windows and Linux x86-64. Windows must permit job
object assignment. Linux requires access to ``/proc``, process restrictions and
the ability to establish the configured address-space limit. Unsupported or
unenforceable environments fail rather than disabling those limits. Resource
limits are not a security sandbox against a malicious local user.

Configured capture
------------------

Use a source configuration to capture a local Parquet file into a snapshot.
The schema and both sets of limits are required:

.. code-block:: toml

   schema_version = 1

   [store]
   path = "snapshots"

   [sources.orders]
   kind = "parquet"
   path = "orders.parquet"
   source_id = "orders"
   [sources.orders.schema]
   format = "pipewhy.parquet-schema"
   version = 1
   fields = [
     { id = 1, name = "order_id", type = "int64", nullable = false },
     { id = 2, name = "label", type = "text", nullable = true },
   ]

   [sources.orders.limits]
   source_bytes = 10000000
   footer_bytes = 1000000
   thrift_string_bytes = 100000
   thrift_container_items = 100000
   field_bytes = 100000
   field_count = 100
   row_groups = 100
   batch_records = 1000
   batch_words = 100000
   max_records = 100000

   [sources.orders.worker]
   memory_bytes = 1073741824
   timeout_ms = 30000
   frame_bytes = 2000000
   output_bytes = 50000000

Supply the lowercase SHA-256 of the file as ``--source-version``:

.. code-block:: console

   pipewhy scan orders --config pipewhy.toml --source-version FILE_SHA256 --retain value

After a successful capture, snapshots can be reopened and compared without the
source file. Failed reads produce an unavailable observation and do not publish
partial records. Source lineage remains unknown unless separately supported by
evidence. Errors contain stable categories rather than source values.

Table-format semantics, remote object storage and additional Parquet types are
separate capabilities.