TSV sources
===========

TSV capture is available in the development source. The published
``0.1.0.dev0`` package predates this functionality. Configure a local source
with an explicit schema and resource limits::

   schema_version = 1

   [store]
   path = "snapshots"

   [sources.orders]
   kind = "tsv"
   path = "orders.tsv"
   source_id = "orders"
   schema = { format = "pipewhy.tsv-schema", version = 1, encoding = "utf-8", fields = [{ id = 0, name = "amount", type = "int64", nullable = false, null_markers = [] }] }
   limits = { source_bytes = 4096, record_bytes = 1024, field_bytes = 256, field_count = 16, batch_records = 2, batch_words = 1024, max_records = 100 }

The first record is a header whose unique names and order must exactly match the
schema. Values are separated by tabs. Double-quoted fields can contain tabs and
line breaks; use doubled quotes for an embedded quote. UTF-8 is required. To
accept a UTF-8 BOM, set ``encoding`` to ``utf-8-sig``. Null markers are explicit:
an empty field is an empty text value unless it is declared as a null marker.
Missing or extra fields, malformed quotes and invalid scalar values fail the read.
The supported scalar types and conversion rules match :doc:`csv`.

Supply the lowercase SHA-256 of the exact file bytes as ``--source-version`` to
``pipewhy scan``. Source bytes and identity are verified around the complete read;
a changed or invalid source produces an unavailable observation without retaining
partial records. Completed snapshots can be reopened, compared and exported
without the file. The observed record count is not an independent upstream
count, and upstream lineage remains unknown. Local files are supported; remote
object storage is separate work.

The resource limits cover source, record and field bytes, field count, output
batch records and words, and total records. The existing interpreter field limit
also applies. Strict normalization is the default; ``--profile logical-v1``
opts into logical normalization. The lower-level ``iter_tsv_batches`` API uses
``TsvSchema`` and ``CsvLimits`` and returns provisional batches until full
iteration and source verification complete. Its optional custom source opener
is trusted to verify the input and does not add a remote storage connector.