pandas DataFrames

Install the optional dependencies with pip install pipewhy[pandas]. The adapter accepts a local, in-memory pandas DataFrame and an explicit schema. It supports signed int64 and nullable Int64, unsigned uint64 and nullable UInt64, float64 and nullable Float64, bool and nullable boolean, Python-backed StringDtype, and UTC-aware datetime64[us, UTC] columns declared as datetime-us-utc. IANA zoneinfo timestamps at microsecond precision use datetime-us-zoneinfo with an explicit timezone_key; daylight saving folds are preserved. Other timezone objects and timestamp precision combinations are rejected until qualified. Native NumPy-backed dtypes cannot declare nullable fields; extension arrays use pandas.NA as the null marker. A floating-point NaN is a value when its extension mask does not mark it missing. NumPy dtype metadata is rejected until its semantics are declared. Boolean storage must use canonical 0 or 1 bytes; the bounded staging budget includes its conversion buffer. Unsupported types, duplicate or mismatched columns, and nulls in required fields are rejected. The DataFrame index is excluded.

from pathlib import Path
import pandas as pd
from pipewhy.adapters.pandas import PandasSource
from pipewhy.cli.capture import scan_source
from pipewhy.formats import PandasField, PandasLimits, PandasSchema, pandas_source_version
from pipewhy.snapshots import LocalSnapshotStore

frame = pd.DataFrame({
    "id": pd.array([1, 2], dtype="Int64"),
    "name": pd.array(["Ada", pd.NA], dtype=pd.StringDtype(storage="python", na_value=pd.NA)),
})
schema = PandasSchema((PandasField(0, "id", "Int64"), PandasField(1, "name", "string-python", True)))
limits = PandasLimits(1000, 10, 1024, 1000000, 100, 10000)
source = PandasSource("people", frame, schema, limits)
version = pandas_source_version(frame, schema=schema, limits=limits)
snapshot = scan_source(source, LocalSnapshotStore(Path("snapshots")), version)

The content version includes the schema and ordered records. Capture stages the records and checks the source again before publishing, detecting accidental changes between observations. It does not provide an atomic transaction against concurrent mutation; callers should keep the DataFrame unchanged during capture. The staged-byte limit bounds encoded records and their offsets, not the memory already owned by the DataFrame. Strict normalization is the default; select profile="logical-v1" explicitly for logical comparisons. Snapshots can be reopened without the original DataFrame. This adapter does not access remote storage or distributed DataFrames.

Series

A named pandas.Series uses a distinct schema and source so its identity does not collide with a one-column DataFrame. The field name must match the Series name; its index is excluded under the same explicit policy. For example:

from pipewhy.adapters.pandas import PandasSeriesSource
from pipewhy.formats import PandasSeriesSchema, pandas_series_source_version

amounts = pd.Series(pd.array([1, pd.NA], dtype="Int64"), name="amount")
series_schema = PandasSeriesSchema(PandasField(0, "amount", "Int64", True))
series_source = PandasSeriesSource("amounts", amounts, series_schema, limits)
series_version = pandas_series_source_version(amounts, schema=series_schema, limits=limits)
series_snapshot = scan_source(series_source, LocalSnapshotStore(Path("snapshots")), series_version)

The same dtype restrictions, resource budgets, mutation caveat and offline snapshot behavior apply.

String categoricals

A category-string field requires an explicit tuple of unique categories and an explicit ordered boolean. Categories must use pandas’ Python-backed StringDtype with pandas.NA as its missing marker. The complete category list and ordering are part of the schema identity, including unused categories. A categorical missing code is recorded as null when the field permits nulls. Other category value types and category storage are rejected. Category labels count towards the configured field and staging limits.

categories = ("red", "blue")
category_index = pd.Index(pd.array(categories, dtype=pd.StringDtype(storage="python", na_value=pd.NA)))
dtype = pd.CategoricalDtype(categories=category_index, ordered=True)
colors = pd.DataFrame({"color": pd.Categorical(["blue", None, "red"], dtype=dtype)})
schema = PandasSchema((PandasField(0, "color", "category-string", True, categories, True),))