Avro object containers
======================

The development source can read one verified Avro object container file through
the Python API. The published ``0.1.0.dev0`` package predates this capability.
Install the optional decoder from a source checkout with
``python -m pip install '.[avro]'``; the core installation does not require it.

The initial subset supports flat writer records containing signed Avro ``long``
and UTF-8 ``string`` fields, optionally in a union with ``null``. Declare the
exact writer schema, selected field IDs and nullability. Nested values, logical
types, aliases, defaults, schema resolution and other primitive types are not
supported in this subset. An incompatible writer schema or unsupported codec
fails explicitly. Strict representation is the default; ``Profile.LOGICAL`` is
an opt-in normalization profile.

.. code-block:: python

   import json
   from hashlib import file_digest
   from pathlib import Path

   from pipewhy.formats import (
       AvroField, AvroLimits, AvroSchema, AvroWorkerLimits,
       read_avro_batches,
   )

   path = Path("orders.avro")
   with path.open("rb") as source:
       version = file_digest(source, "sha256").hexdigest()

   writer_schema = {
       "type": "record", "name": "Orders",
       "fields": [
           {"name": "order_id", "type": "long"},
           {"name": "label", "type": ["null", "string"]},
       ],
   }
   schema = AvroSchema(
       json.dumps(writer_schema),
       (AvroField(1, "order_id", "long"),
        AvroField(2, "label", "string", nullable=True)),
       ("null", "deflate"),
   )
   limits = AvroLimits(
       source_bytes=10_000_000, metadata_bytes=100_000,
       metadata_entries=100, schema_bytes=50_000, schema_depth=10,
       field_count=100, compressed_block_bytes=1_000_000,
       decompressed_block_bytes=5_000_000, block_records=100_000,
       max_records=1_000_000, field_bytes=100_000,
       batch_records=1_000, batch_words=100_000,
   )
   worker = AvroWorkerLimits(
       memory_bytes=1_073_741_824, timeout_ms=30_000,
       frame_bytes=2_000_000, output_bytes=50_000_000,
   )
   batches = read_avro_batches(
       path, sha256_hex=version, schema=schema, limits=limits,
       worker_limits=worker,
   )

All limits are explicit; these values are examples, not portable defaults. The
reader stages the source locally, validates container metadata, block lengths,
counts and sync markers, and verifies the full SHA-256. Results remain
provisional until the bounded worker and source checks complete. The worker is
supported on Windows and Linux x86-64 when the required platform resource
controls are available. These controls are not a security sandbox.

Strict mode rejects bytes following a raw deflate stream. Apache's Python Avro
1.12.2 writer can produce a three-byte Adler-32 prefix there. To read that
specific form, set ``allow_apache_legacy_deflate=True`` on ``AvroSchema``. The
reader verifies the prefix and binds this option into schema identity; it does
not accept arbitrary trailing bytes. A checksum prefix is not authentication.

A trusted verified source opener may provide a binary stream instead of a local
path. This interface does not implement remote storage. Snapshot capture
currently uses local paths.

Configured capture
------------------

A local Avro source can be captured into a snapshot using an explicit schema,
codec policy, limits and worker configuration:

.. code-block:: toml

   schema_version = 1
   [store]
   path = "snapshots"
   [sources.orders]
   kind = "avro"
   path = "orders.avro"
   source_id = "orders"
   schema = { format = "pipewhy.avro-schema", version = 1, writer_schema = { type = "record", name = "Order", fields = [{ name = "id", type = "long" }, { name = "label", type = ["null", "string"] }] }, fields = [{ id = 1, name = "id", type = "long", nullable = false }, { id = 2, name = "label", type = "string", nullable = true }], codecs = ["null", "deflate"], allow_apache_legacy_deflate = false }
   limits = { source_bytes = 100000, metadata_bytes = 10000, metadata_entries = 10, schema_bytes = 10000, schema_depth = 10, field_count = 10, compressed_block_bytes = 10000, decompressed_block_bytes = 10000, block_records = 100, max_records = 100, field_bytes = 1000, batch_records = 10, batch_words = 1000 }
   worker = { memory_bytes = 1073741824, timeout_ms = 15000, frame_bytes = 4000, output_bytes = 10000 }

Supply the file's lowercase SHA-256 as ``--source-version``:

.. code-block:: console

   pipewhy scan orders --config pipewhy.toml --source-version FILE_SHA256 --retain value

Completed snapshots can be reopened and compared without the source file.
Incomplete reads record an unavailable observation and do not publish partial
records. Upstream lineage remains unknown unless separately supported by evidence.