Avro object containers

The development source can read one verified Avro object container file through the Python API. The published 0.1.0.dev0 package predates this capability. Install the optional decoder from a source checkout with python -m pip install '.[avro]'; the core installation does not require it.

The initial subset supports flat writer records containing signed Avro long and UTF-8 string fields, optionally in a union with null. Declare the exact writer schema, selected field IDs and nullability. Nested values, logical types, aliases, defaults, schema resolution and other primitive types are not supported in this subset. An incompatible writer schema or unsupported codec fails explicitly. Strict representation is the default; Profile.LOGICAL is an opt-in normalization profile.

import json
from hashlib import file_digest
from pathlib import Path

from pipewhy.formats import (
    AvroField, AvroLimits, AvroSchema, AvroWorkerLimits,
    read_avro_batches,
)

path = Path("orders.avro")
with path.open("rb") as source:
    version = file_digest(source, "sha256").hexdigest()

writer_schema = {
    "type": "record", "name": "Orders",
    "fields": [
        {"name": "order_id", "type": "long"},
        {"name": "label", "type": ["null", "string"]},
    ],
}
schema = AvroSchema(
    json.dumps(writer_schema),
    (AvroField(1, "order_id", "long"),
     AvroField(2, "label", "string", nullable=True)),
    ("null", "deflate"),
)
limits = AvroLimits(
    source_bytes=10_000_000, metadata_bytes=100_000,
    metadata_entries=100, schema_bytes=50_000, schema_depth=10,
    field_count=100, compressed_block_bytes=1_000_000,
    decompressed_block_bytes=5_000_000, block_records=100_000,
    max_records=1_000_000, field_bytes=100_000,
    batch_records=1_000, batch_words=100_000,
)
worker = AvroWorkerLimits(
    memory_bytes=1_073_741_824, timeout_ms=30_000,
    frame_bytes=2_000_000, output_bytes=50_000_000,
)
batches = read_avro_batches(
    path, sha256_hex=version, schema=schema, limits=limits,
    worker_limits=worker,
)

All limits are explicit; these values are examples, not portable defaults. The reader stages the source locally, validates container metadata, block lengths, counts and sync markers, and verifies the full SHA-256. Results remain provisional until the bounded worker and source checks complete. The worker is supported on Windows and Linux x86-64 when the required platform resource controls are available. These controls are not a security sandbox.

Strict mode rejects bytes following a raw deflate stream. Apache’s Python Avro 1.12.2 writer can produce a three-byte Adler-32 prefix there. To read that specific form, set allow_apache_legacy_deflate=True on AvroSchema. The reader verifies the prefix and binds this option into schema identity; it does not accept arbitrary trailing bytes. A checksum prefix is not authentication.

A trusted verified source opener may provide a binary stream instead of a local path. This interface does not implement remote storage. Snapshot capture currently uses local paths.

Configured capture

A local Avro source can be captured into a snapshot using an explicit schema, codec policy, limits and worker configuration:

schema_version = 1
[store]
path = "snapshots"
[sources.orders]
kind = "avro"
path = "orders.avro"
source_id = "orders"
schema = { format = "pipewhy.avro-schema", version = 1, writer_schema = { type = "record", name = "Order", fields = [{ name = "id", type = "long" }, { name = "label", type = ["null", "string"] }] }, fields = [{ id = 1, name = "id", type = "long", nullable = false }, { id = 2, name = "label", type = "string", nullable = true }], codecs = ["null", "deflate"], allow_apache_legacy_deflate = false }
limits = { source_bytes = 100000, metadata_bytes = 10000, metadata_entries = 10, schema_bytes = 10000, schema_depth = 10, field_count = 10, compressed_block_bytes = 10000, decompressed_block_bytes = 10000, block_records = 100, max_records = 100, field_bytes = 1000, batch_records = 10, batch_words = 1000 }
worker = { memory_bytes = 1073741824, timeout_ms = 15000, frame_bytes = 4000, output_bytes = 10000 }

Supply the file’s lowercase SHA-256 as --source-version:

pipewhy scan orders --config pipewhy.toml --source-version FILE_SHA256 --retain value

Completed snapshots can be reopened and compared without the source file. Incomplete reads record an unavailable observation and do not publish partial records. Upstream lineage remains unknown unless separately supported by evidence.