Parquet files

The development source can read one verified local Parquet file through the Python API. The published 0.1.0.dev0 package predates this functionality. From a source checkout, install the optional capability with python -m pip install '.[parquet]'. The core installation does not require PyArrow.

The first reader supports explicitly selected flat signed 64-bit integer and UTF-8 string columns. Declare each field’s stable ID, name, type and nullability; missing selected fields, nested columns, duplicate names, unsupported selected types and incompatible schemas fail explicitly. Extra flat columns need not be selected. Strict representation is the default; pass Profile.LOGICAL to opt in to logical normalization.

from hashlib import file_digest
from pathlib import Path

from pipewhy.formats import (
    ParquetField, ParquetLimits, ParquetSchema, ParquetWorkerLimits,
    read_parquet_batches,
)

path = Path("orders.parquet")
with path.open("rb") as source:
    version = file_digest(source, "sha256").hexdigest()

schema = ParquetSchema((
    ParquetField(1, "order_id", "int64"),
    ParquetField(2, "label", "text", nullable=True),
))
limits = ParquetLimits(
    source_bytes=10_000_000, footer_bytes=1_000_000,
    thrift_string_bytes=100_000, thrift_container_items=100_000,
    field_bytes=100_000, field_count=100, row_groups=100,
    batch_records=1_000, batch_words=100_000, max_records=100_000,
)
worker = ParquetWorkerLimits(
    memory_bytes=1_073_741_824, timeout_ms=30_000,
    frame_bytes=2_000_000, output_bytes=50_000_000,
)
batches = read_parquet_batches(
    path, sha256_hex=version, schema=schema, limits=limits,
    worker_limits=worker,
)

All limits are explicit. The values above are examples rather than portable defaults: native libraries and datasets need different budgets. The reader stages the input and bounded output in temporary local files, verifies the source before and after processing, and returns batches only after successful worker completion and output validation. Checksums on Parquet pages are verified when present; the source SHA-256 is checked independently.

The current worker supports Windows and Linux x86-64. Windows must permit job object assignment. Linux requires access to /proc, process restrictions and the ability to establish the configured address-space limit. Unsupported or unenforceable environments fail rather than disabling those limits. Resource limits are not a security sandbox against a malicious local user.

Configured capture

Use a source configuration to capture a local Parquet file into a snapshot. The schema and both sets of limits are required:

schema_version = 1

[store]
path = "snapshots"

[sources.orders]
kind = "parquet"
path = "orders.parquet"
source_id = "orders"
[sources.orders.schema]
format = "pipewhy.parquet-schema"
version = 1
fields = [
  { id = 1, name = "order_id", type = "int64", nullable = false },
  { id = 2, name = "label", type = "text", nullable = true },
]

[sources.orders.limits]
source_bytes = 10000000
footer_bytes = 1000000
thrift_string_bytes = 100000
thrift_container_items = 100000
field_bytes = 100000
field_count = 100
row_groups = 100
batch_records = 1000
batch_words = 100000
max_records = 100000

[sources.orders.worker]
memory_bytes = 1073741824
timeout_ms = 30000
frame_bytes = 2000000
output_bytes = 50000000

Supply the lowercase SHA-256 of the file as --source-version:

pipewhy scan orders --config pipewhy.toml --source-version FILE_SHA256 --retain value

After a successful capture, snapshots can be reopened and compared without the source file. Failed reads produce an unavailable observation and do not publish partial records. Source lineage remains unknown unless separately supported by evidence. Errors contain stable categories rather than source values.

Table-format semantics, remote object storage and additional Parquet types are separate capabilities.