pandas DataFrames
=================

Install the optional dependencies with ``pip install pipewhy[pandas]``. The
adapter accepts a local, in-memory pandas DataFrame and an explicit schema.
It supports signed ``int64`` and nullable ``Int64``, unsigned ``uint64`` and
nullable ``UInt64``, ``float64`` and nullable ``Float64``, ``bool`` and nullable
``boolean``, Python-backed ``StringDtype``, and UTC-aware ``datetime64[us, UTC]``
columns declared as ``datetime-us-utc``. IANA zoneinfo timestamps at microsecond
precision use ``datetime-us-zoneinfo`` with an explicit ``timezone_key``; daylight
saving folds are preserved. Other timezone objects and timestamp precision
combinations are rejected until qualified. Native NumPy-backed
dtypes cannot declare nullable fields; extension arrays use ``pandas.NA`` as
the null marker. A floating-point NaN is a value when its extension mask does
not mark it missing. NumPy dtype metadata is rejected until its semantics are
declared. Boolean storage must use canonical 0 or 1 bytes; the
bounded staging budget includes its conversion buffer. Unsupported types,
duplicate or mismatched columns, and
nulls in required fields are rejected. The DataFrame index is excluded.

.. code-block:: python

   from pathlib import Path
   import pandas as pd
   from pipewhy.adapters.pandas import PandasSource
   from pipewhy.cli.capture import scan_source
   from pipewhy.formats import PandasField, PandasLimits, PandasSchema, pandas_source_version
   from pipewhy.snapshots import LocalSnapshotStore

   frame = pd.DataFrame({
       "id": pd.array([1, 2], dtype="Int64"),
       "name": pd.array(["Ada", pd.NA], dtype=pd.StringDtype(storage="python", na_value=pd.NA)),
   })
   schema = PandasSchema((PandasField(0, "id", "Int64"), PandasField(1, "name", "string-python", True)))
   limits = PandasLimits(1000, 10, 1024, 1000000, 100, 10000)
   source = PandasSource("people", frame, schema, limits)
   version = pandas_source_version(frame, schema=schema, limits=limits)
   snapshot = scan_source(source, LocalSnapshotStore(Path("snapshots")), version)

The content version includes the schema and ordered records. Capture stages the
records and checks the source again before publishing, detecting accidental
changes between observations. It does not provide an atomic transaction against
concurrent mutation; callers should keep the DataFrame unchanged during capture.
The staged-byte limit bounds encoded records and their offsets, not the memory
already owned by the DataFrame. Strict normalization is the default; select
``profile="logical-v1"`` explicitly for logical comparisons. Snapshots can be
reopened without the original DataFrame. This adapter does not access remote
storage or distributed DataFrames.

Series
------

A named ``pandas.Series`` uses a distinct schema and source so its identity does
not collide with a one-column DataFrame. The field name must match the Series
name; its index is excluded under the same explicit policy. For example::

   from pipewhy.adapters.pandas import PandasSeriesSource
   from pipewhy.formats import PandasSeriesSchema, pandas_series_source_version

   amounts = pd.Series(pd.array([1, pd.NA], dtype="Int64"), name="amount")
   series_schema = PandasSeriesSchema(PandasField(0, "amount", "Int64", True))
   series_source = PandasSeriesSource("amounts", amounts, series_schema, limits)
   series_version = pandas_series_source_version(amounts, schema=series_schema, limits=limits)
   series_snapshot = scan_source(series_source, LocalSnapshotStore(Path("snapshots")), series_version)

The same dtype restrictions, resource budgets, mutation caveat and offline
snapshot behavior apply.

String categoricals
-------------------

A ``category-string`` field requires an explicit tuple of unique categories and
an explicit ``ordered`` boolean. Categories must use pandas' Python-backed
``StringDtype`` with ``pandas.NA`` as its missing marker. The complete category
list and ordering are part of the schema identity, including unused categories.
A categorical missing code is recorded as null when the field permits nulls.
Other category value types and category storage are rejected. Category labels
count towards the configured field and staging limits.

.. code-block:: python

   categories = ("red", "blue")
   category_index = pd.Index(pd.array(categories, dtype=pd.StringDtype(storage="python", na_value=pd.NA)))
   dtype = pd.CategoricalDtype(categories=category_index, ordered=True)
   colors = pd.DataFrame({"color": pd.Categorical(["blue", None, "red"], dtype=dtype)})
   schema = PandasSchema((PandasField(0, "color", "category-string", True, categories, True),))