PipeWhy Snapshot Contract C5 version 1.0 October 30, 2026 PW-07.01 Manifest fields The first local format captures one explicitly versioned source scope per snapshot. It does not claim a globally atomic enterprise scan. Snapshot metadata records: snapshot_id, optional parent_id, source_namespace, source_version, source_consistency, scope, started_at_ns, finished_at_ns, resolution, coverage and fingerprint context. Snapshot/parent IDs are 32 lowercase hex UUID-style identifiers, never paths. Start/end are exact nonnegative uint64 epoch nanoseconds with finish >= start. Namespace/version/consistency/scope are nonempty UTF-8 strings. Resolution is dataset, record or value and declares the capture's retained detail, not a promise that finer historical data can be recovered later. Coverage includes status COMPLETE/PARTIAL/UNAVAILABLE, expected_records (uint64 or unknown), returned_records (uint64) and explicit failure-reason strings. COMPLETE requires a known matching expected count and no failures. PARTIAL requires failures and cannot return more than a known expected count. UNAVAILABLE requires failures and returned=0. Unknown and zero remain distinct. Producers retain authoritative scan/read evidence; the storage layer checks metadata consistency, not source truth. Fingerprint metadata records algorithm name/full ID, context ID, full normalization profile ID, source schema ID and mode. Resolve the exact current C3 context and check its identity; do not retain only a local integer handle or transient pointer. Parent links require a previously complete snapshot and cannot self-reference. Different historical contexts remain explicit, not silently converted. Manifest shape is {format:"pipewhy.snapshot", version:1, array_schema_version:"1.0", snapshot:{...}, arrays:{...}}. The declared array schema version identifies C1. All field sets are exact. Persist uint64 counts/times/lengths as decimal strings for precision across JSON implementations; the small format version is an integer. No bool/float coercion. Canonical manifest JSON is UTF-8 with sorted keys, compact separators, ensure_ascii=False and allow_nan=False. Reject duplicate object keys, unknown fields, malformed identifiers and unsupported versions before consumption. PW-07.02 Local immutable layout LocalSnapshotStore(root) owns root/snapshots and root/staging on the same local filesystem. Final snapshots live only at snapshots//; unique attempts stage in staging//. Names are not user-supplied relative paths. Each array key matches [a-z][a-z0-9_]{0,63}; file name is array-.bin, avoiding case/path aliasing and Windows reserved base names. Payload metadata records exact file, dtype, count, byte_length, endianness=little and SHA-256. Supported dtypes are signed/unsigned integers of 8/16/32/64 bits. Buffer borrowing follows C1: 1D native contiguous integer CPU input, stable throughout capture, with an owned byte copy. Payloads have no padding/header: count * dtype width bytes. Readers check this identity against both metadata and file size before making read-only owned views. Initial implementation is memory-resident; it does not claim arbitrary-size streaming or mmap behavior. C1 model validators remain the consumer's semantic gate for table relationships, offsets and bitmaps after storage's byte validation. PW-07.03 Stored-byte checksums Use full SHA-256 of every payload's exact stored little-endian bytes. Manifest entries cover dtype/count/endian/file/checksum. complete.sha256 contains SHA-256 of exact manifest.json bytes as 64 lowercase ASCII hex characters plus newline. Validate marker, manifest and every required array before returning a snapshot. Checksum failure is an explicit storage-integrity error, never a successful empty array or unknown source result. Unknown/missing/extra payloads are rejected. These checks address accidental corruption; they do not authenticate an attacker who can rewrite payload, manifest and completion checksum together. PW-07.04 Publication state machine 1. Validate metadata, names, arrays and parent; reject an existing final ID. 2. Create a unique staging attempt with intent.json naming its snapshot ID. 3. Write/flush/fsync payload files and canonical manifest.json into that attempt. 4. Validate every staged payload and metadata, then write/flush/fsync complete.sha256. 5. Validate the complete staged package. Rename the entire nonempty directory to snapshots/ without overwriting an existing snapshot. Only final-directory publication makes a snapshot discoverable as complete. No final directory is populated incrementally and no reader follows staging paths. Pre-publication failures propagate and preserve attempts for inspection; no silent retry into a successful result or automatic deletion is performed. A failure after the directory rename may leave a valid published snapshot even though publish() raises (for example, a destination directory-fsync failure). Inspect the exact ID before retrying; never overwrite it to disguise uncertain commit/durability status. Concurrent writers of one ID can publish at most one nonempty final directory; losing attempts remain inspectable. Staging phases are STAGING (intent but not validated marker), VALIDATED (complete package still in staging) or INVALID/UNAVAILABLE with a concrete diagnostic. An intent-only/partially written attempt never becomes a complete snapshot. Inspection is advisory under concurrent publication and does not mutate or recover data. Retention/deletion and stale-attempt cleanup require explicit policy/action later. Use local filesystem atomic directory rename semantics. Cross-filesystem rename errors remain failures. Fsync files before publication; fsync directories on POSIX. Windows has no equivalent stdlib directory-fsync guarantee, so the first Windows qualification covers process interruption/visibility, not power-loss durability. Remote filesystems, machine-power-failure recovery and adversarial concurrent mutation are outside the initial qualification; no stronger guarantee is inferred. PW-07.05 Compatibility and errors Accept C5 version 1, C1 array schema 1.0 and current C3 algorithm declarations only. C5 stores named arrays without inferring semantic roles from names; C1 table consumers run their validators and C3 state consumers call FingerprintState.from_words before using reopened buffers. Reject unsupported format/dtype/endian/algorithm, mismatched full context identity, truncated/missing files, wrong byte lengths and malformed coverage. Validate full manifests before constructing consumers' array views. Published buffers are read-only owned bytes; mutating original inputs cannot alter the stored or reopened snapshot. File/permission/JSON errors retain context and fail visibly. Incomplete inspection may report specific expected errors with diagnostics; it must never hide them. An unavailable historical resolution remains unavailable rather than re-reading today's source as an alleged reconstruction. Persistence supplies exact source and capture limits to comparison/history consumers without making a retention promise. Primary implementation references: https://docs.python.org/3.14/library/os.html#os.rename https://docs.python.org/3.14/library/os.html#os.fsync https://docs.python.org/3.14/library/json.html PW-07.18 Storage lifecycle and retention hooks Public API: from pipewhy.snapshots import LocalSnapshotStore, SnapshotMetadata, StorageIntegrityError. publish(metadata, arrays) captures one immutable source scope; open(snapshot_id) returns metadata and read-only owned arrays; list_snapshots() validates and lists published IDs. Corrupt published packages make listing fail visibly rather than disappearing as if the source artifact had been removed. Missing IDs raise FileNotFoundError, with no latest-version or live-source fallback. The initial retention policy is retain-all. No age limit, storage budget, automatic pruning, deletion, staged-attempt promotion or recovery rewrite is implemented. An external policy can inspect exact metadata via list/open and staged attempts via inspect_incomplete(); those are actual read-only hooks, not cleanup placeholders. Future cleanup requires an explicit policy/action and must verify resolved paths remain within the selected repository. Preserve required parents, evidence and source-detail retention before removing data. Child snapshots expose parent IDs; opening their own retained arrays does not recursively require parent availability. inspect_incomplete() is non-destructive and returns path, snapshot ID when known, phase and diagnostic. VALIDATED means the staged bytes pass checks at inspection time, not that the attempt was published, externally approved or durably synced. An interrupted write can leave STAGING/INVALID/UNAVAILABLE evidence. A failed complete-file fsync may leave a byte-valid VALIDATED attempt; preserve the original error instead of presenting that as a successful write. Precommit retries create a new attempt and retain the old one. After an uncertain postcommit error, inspect the exact final ID before deciding whether any new action is needed; never overwrite. Resolution declares the detail actually retained by the producer. A dataset-only fingerprint cannot reconstruct old records; record/value capture requires those payloads or access to the exact historical source. Physical storage validates C5 metadata/bytes; source truth and C1/C3 semantic validation remain explicit producer and consumer responsibilities. Retain full schema/profile/algorithm identities. Qualification: Windows units exercise all integer boundary/empty encodings, strict metadata and coverage, immutable reopening, corruption/inventory rejection, each write/sync/finalize failure, concurrent writers and visibility before commit. Separate integration checks round-trip the shared C1 five-table fixture and kill real writer processes immediately before/after directory publication. Linux qualification also passed these checks, POSIX directory-sync execution and installed wheel/sdist checks. Unit coverage is measured separately; integration checks do not inflate that score. Power-loss, remote filesystems, native CPU/CUDA and enterprise-scale retention remain separate qualifications, not inferred from these results.