TSV sources

TSV capture is available in the development source. The published 0.1.0.dev0 package predates this functionality. Configure a local source with an explicit schema and resource limits:

schema_version = 1

[store]
path = "snapshots"

[sources.orders]
kind = "tsv"
path = "orders.tsv"
source_id = "orders"
schema = { format = "pipewhy.tsv-schema", version = 1, encoding = "utf-8", fields = [{ id = 0, name = "amount", type = "int64", nullable = false, null_markers = [] }] }
limits = { source_bytes = 4096, record_bytes = 1024, field_bytes = 256, field_count = 16, batch_records = 2, batch_words = 1024, max_records = 100 }

The first record is a header whose unique names and order must exactly match the schema. Values are separated by tabs. Double-quoted fields can contain tabs and line breaks; use doubled quotes for an embedded quote. UTF-8 is required. To accept a UTF-8 BOM, set encoding to utf-8-sig. Null markers are explicit: an empty field is an empty text value unless it is declared as a null marker. Missing or extra fields, malformed quotes and invalid scalar values fail the read. The supported scalar types and conversion rules match CSV sources.

Supply the lowercase SHA-256 of the exact file bytes as --source-version to pipewhy scan. Source bytes and identity are verified around the complete read; a changed or invalid source produces an unavailable observation without retaining partial records. Completed snapshots can be reopened, compared and exported without the file. The observed record count is not an independent upstream count, and upstream lineage remains unknown. Local files are supported; remote object storage is separate work.

The resource limits cover source, record and field bytes, field count, output batch records and words, and total records. The existing interpreter field limit also applies. Strict normalization is the default; --profile logical-v1 opts into logical normalization. The lower-level iter_tsv_batches API uses TsvSchema and CsvLimits and returns provisional batches until full iteration and source verification complete. Its optional custom source opener is trusted to verify the input and does not add a remote storage connector.