PipeWhy Snapshot Contract C5 version 1.0
October 30, 2026

PW-07.01 Manifest fields
The first local format captures one explicitly versioned source scope per snapshot.
It does not claim a globally atomic enterprise scan. Snapshot metadata records:
snapshot_id, optional parent_id, source_namespace, source_version, source_consistency,
scope, started_at_ns, finished_at_ns, resolution, coverage and fingerprint context.
Snapshot/parent IDs are 32 lowercase hex UUID-style identifiers, never paths.
Start/end are exact nonnegative uint64 epoch nanoseconds with finish >= start.
Namespace/version/consistency/scope are nonempty UTF-8 strings. Resolution is
dataset, record or value and declares the capture's retained detail, not a promise
that finer historical data can be recovered later.

Coverage includes status COMPLETE/PARTIAL/UNAVAILABLE, expected_records (uint64 or
unknown), returned_records (uint64) and explicit failure-reason strings. COMPLETE
requires a known matching expected count and no failures. PARTIAL requires failures
and cannot return more than a known expected count. UNAVAILABLE requires failures
and returned=0. Unknown and zero remain distinct. Producers retain authoritative
scan/read evidence; the storage layer checks metadata consistency, not source truth.

Fingerprint metadata records algorithm name/full ID, context ID, full normalization
profile ID, source schema ID and mode. Resolve the exact current C3 context and
check its identity; do not retain only a local integer handle or transient pointer.
Parent links require a previously complete snapshot and cannot self-reference.
Different historical contexts remain explicit, not silently converted.

Manifest shape is {format:"pipewhy.snapshot", version:1, array_schema_version:"1.0",
snapshot:{...}, arrays:{...}}. The declared array schema version identifies C1.
All field sets are exact. Persist uint64 counts/times/lengths as decimal strings
for precision across JSON implementations; the small format version is an integer.
No bool/float coercion. Canonical manifest JSON is UTF-8 with sorted keys, compact
separators, ensure_ascii=False and allow_nan=False. Reject duplicate object keys,
unknown fields, malformed identifiers and unsupported versions before consumption.

PW-07.02 Local immutable layout
LocalSnapshotStore(root) owns root/snapshots and root/staging on the same local
filesystem. Final snapshots live only at snapshots/<snapshot_id>/; unique attempts
stage in staging/<random_attempt_id>/. Names are not user-supplied relative paths.
Each array key matches [a-z][a-z0-9_]{0,63}; file name is array-<key>.bin, avoiding
case/path aliasing and Windows reserved base names. Payload metadata records exact
file, dtype, count, byte_length, endianness=little and SHA-256. Supported dtypes are
signed/unsigned integers of 8/16/32/64 bits. Buffer borrowing follows C1: 1D native
contiguous integer CPU input, stable throughout capture, with an owned byte copy.

Payloads have no padding/header: count * dtype width bytes. Readers check this
identity against both metadata and file size before making read-only owned views.
Initial implementation is memory-resident; it does not claim arbitrary-size
streaming or mmap behavior. C1 model validators remain the consumer's semantic
gate for table relationships, offsets and bitmaps after storage's byte validation.

PW-07.03 Stored-byte checksums
Use full SHA-256 of every payload's exact stored little-endian bytes. Manifest
entries cover dtype/count/endian/file/checksum. complete.sha256 contains SHA-256
of exact manifest.json bytes as 64 lowercase ASCII hex characters plus newline.
Validate marker, manifest and every required array before returning a snapshot.
Checksum failure is an explicit storage-integrity error, never a successful empty
array or unknown source result. Unknown/missing/extra payloads are rejected.
These checks address accidental corruption; they do not authenticate an attacker
who can rewrite payload, manifest and completion checksum together.

PW-07.04 Publication state machine
1. Validate metadata, names, arrays and parent; reject an existing final ID.
2. Create a unique staging attempt with intent.json naming its snapshot ID.
3. Write/flush/fsync payload files and canonical manifest.json into that attempt.
4. Validate every staged payload and metadata, then write/flush/fsync complete.sha256.
5. Validate the complete staged package. Rename the entire nonempty directory to
   snapshots/<snapshot_id> without overwriting an existing snapshot.
Only final-directory publication makes a snapshot discoverable as complete.
No final directory is populated incrementally and no reader follows staging paths.
Pre-publication failures propagate and preserve attempts for inspection; no silent
retry into a successful result or automatic deletion is performed. A failure after
the directory rename may leave a valid published snapshot even though publish()
raises (for example, a destination directory-fsync failure). Inspect the exact ID
before retrying; never overwrite it to disguise uncertain commit/durability status.
Concurrent writers of one ID
can publish at most one nonempty final directory; losing attempts remain inspectable.

Staging phases are STAGING (intent but not validated marker), VALIDATED (complete
package still in staging) or INVALID/UNAVAILABLE with a concrete diagnostic. An
intent-only/partially written attempt never becomes a complete snapshot. Inspection
is advisory under concurrent publication and does not mutate or recover data.
Retention/deletion and stale-attempt cleanup require explicit policy/action later.

Use local filesystem atomic directory rename semantics. Cross-filesystem rename
errors remain failures. Fsync files before publication; fsync directories on POSIX.
Windows has no equivalent stdlib directory-fsync guarantee, so the first Windows
qualification covers process interruption/visibility, not power-loss durability.
Remote filesystems, machine-power-failure recovery and adversarial concurrent
mutation are outside the initial qualification; no stronger guarantee is inferred.

PW-07.05 Compatibility and errors
Accept C5 version 1, C1 array schema 1.0 and current C3 algorithm declarations only.
C5 stores named arrays without inferring semantic roles from names; C1 table
consumers run their validators and C3 state consumers call FingerprintState.from_words
before using reopened buffers. Reject unsupported
format/dtype/endian/algorithm, mismatched full context identity, truncated/missing
files, wrong byte lengths and malformed coverage. Validate full manifests before
constructing consumers' array views. Published buffers are read-only owned bytes;
mutating original inputs cannot alter the stored or reopened snapshot.
File/permission/JSON errors retain context and fail visibly. Incomplete inspection
may report specific expected errors with diagnostics; it must never hide them.
An unavailable historical resolution remains unavailable rather than re-reading
today's source as an alleged reconstruction. Persistence supplies exact source and
capture limits to comparison/history consumers without making a retention promise.

Primary implementation references:
https://docs.python.org/3.14/library/os.html#os.rename
https://docs.python.org/3.14/library/os.html#os.fsync
https://docs.python.org/3.14/library/json.html

PW-07.18 Storage lifecycle and retention hooks
Public API: from pipewhy.snapshots import LocalSnapshotStore, SnapshotMetadata,
StorageIntegrityError. publish(metadata, arrays) captures one immutable source scope;
open(snapshot_id) returns metadata and read-only owned arrays; list_snapshots()
validates and lists published IDs. Corrupt published packages make listing fail
visibly rather than disappearing as if the source artifact had been removed.
Missing IDs raise FileNotFoundError, with no latest-version or live-source fallback.

The initial retention policy is retain-all. No age limit, storage budget, automatic
pruning, deletion, staged-attempt promotion or recovery rewrite is implemented.
An external policy can inspect exact metadata via list/open and staged attempts via
inspect_incomplete(); those are actual read-only hooks, not cleanup placeholders.
Future cleanup requires an explicit policy/action and must verify resolved paths
remain within the selected repository. Preserve required parents, evidence and
source-detail retention before removing data. Child snapshots expose parent IDs;
opening their own retained arrays does not recursively require parent availability.

inspect_incomplete() is non-destructive and returns path, snapshot ID when known,
phase and diagnostic. VALIDATED means the staged bytes pass checks at inspection
time, not that the attempt was published, externally approved or durably synced.
An interrupted write can leave STAGING/INVALID/UNAVAILABLE evidence. A failed
complete-file fsync may leave a byte-valid VALIDATED attempt; preserve the original
error instead of presenting that as a successful write. Precommit retries create a
new attempt and retain the old one. After an uncertain postcommit error, inspect the
exact final ID before deciding whether any new action is needed; never overwrite.

Resolution declares the detail actually retained by the producer. A dataset-only
fingerprint cannot reconstruct old records; record/value capture requires those
payloads or access to the exact historical source. Physical storage validates C5
metadata/bytes; source truth and C1/C3 semantic validation remain explicit producer
and consumer responsibilities. Retain full schema/profile/algorithm identities.

Qualification: Windows units exercise all integer boundary/empty encodings, strict
metadata and coverage, immutable reopening, corruption/inventory rejection, each
write/sync/finalize failure, concurrent writers and visibility before commit. Separate
integration checks round-trip the shared C1 five-table fixture and kill real writer
processes immediately before/after directory publication. Linux qualification also
passed these checks, POSIX directory-sync execution and installed wheel/sdist checks.
Unit coverage is measured separately; integration checks do not inflate that score.
Power-loss, remote filesystems, native CPU/CUDA and enterprise-scale retention remain
separate qualifications, not inferred from these results.
