NumPy arrays

Install pipewhy[numpy] for local NumPy array support. The initial adapter accepts one- or two-dimensional signed and unsigned 64-bit integer, 64-bit floating-point and boolean arrays, including noncontiguous views. Schema field IDs, dimension, dtype and byte order are explicit. Boolean arrays use the not-applicable byte-order policy and must use canonical 0 or 1 bytes. Masked, structured, object and other dtypes are rejected by this basic array source. NumPy dtype metadata is also rejected until its semantics are declared. Strict normalization preserves NaN payloads and signed zero; logical normalization treats equivalent NaNs and zeros alike.

from pathlib import Path
import numpy as np
from pipewhy.adapters.numpy import NumpySource
from pipewhy.cli.capture import scan_source
from pipewhy.formats import NumpyLimits, NumpySchema, numpy_source_version
from pipewhy.snapshots import LocalSnapshotStore

values = np.array([[1, 2], [3, 4]], dtype="<i8")
schema = NumpySchema((0, 1), "little", 2)
limits = NumpyLimits(1000, 10, 1000000, 100, 10000)
source = NumpySource("measurements", values, schema, limits)
version = numpy_source_version(values, schema=schema, limits=limits)
snapshot = scan_source(source, LocalSnapshotStore(Path("snapshots")), version)

Source versions include the schema, shape and ordered values. Capture stages records and checks the array again before publishing. Callers must prevent concurrent mutation: these checks detect accidental changes but are not an atomic transaction. The staged-byte limit bounds encoded records and offsets, not memory occupied by the original array. Strict normalization is the default; logical normalization is opt-in. Snapshots reopen without the source array.

Structured arrays

One-dimensional structured arrays can use flat, tightly packed fields with signed and unsigned 64-bit integers, 64-bit floats and booleans. Each field declares a stable ID, name, dtype and byte order. Field titles, padding, aligned layouts, metadata, nested fields, object fields and masks are rejected.

from pipewhy.adapters.numpy import NumpyStructuredSource
from pipewhy.formats import NumpyStructuredField, NumpyStructuredSchema, numpy_structured_source_version

values = np.array([(1, 2.5), (2, 3.0)], dtype=[("id", "<i8"), ("score", ">f8")])
schema = NumpyStructuredSchema((NumpyStructuredField(0, "id", "int64", "little"), NumpyStructuredField(1, "score", "float64", "big")))
source = NumpyStructuredSource("measurements", values, schema, limits)
version = numpy_structured_source_version(values, schema=schema, limits=limits)
snapshot = scan_source(source, LocalSnapshotStore(Path("snapshots")), version)

Field names and layouts are part of the schema identity. The same bounded staging, mutation detection and offline snapshot behavior apply.