mostlyright.core.validator

DataFrame validator with source-identity enforcement.

The Validator is the source-identity invariant gate: every canonical-schema DataFrame flowing through mostlyright must carry df.attrs["source"], and that source must match the schema’s registered canonical source unless the caller explicitly opts out via source_drift_reason.

This is the invariant that prevents the silent train/infer mismatch research() Mode 2 is designed against. When a parquet cache file is loaded the next session and the saved source differs from what the schema expects, the user learns immediately (loud SourceMismatchError) instead of finding out their model trained on AWC data and inferred on IEM.

Validation is entirely pandas-native. Per-column dtype validation is dispatched on ColumnSpec.dtype, and enum-value membership is checked with pandas.Series.isin against ColumnSpec.enum_values (fast, no external schema-validation library or serialization roundtrip).

Functions

register_schema(schema_cls)	Register a Schema subclass by its schema_id.
validate_dataframe(df, schema_id, *[, …])	Validate a DataFrame against the named canonical schema.

mostlyright.core.validator.validate_dataframe(df, schema_id, , source_drift_reason=None)

Validate a DataFrame against the named canonical schema.

Accepts either a raw pd.DataFrame (must carry df.attrs["source"] / df.attrs["retrieved_at"]) or a ProvenancedFrame wrapper (provenance travels on the wrapper). When passed a wrapper, the validator unwraps to pandas via ProvenancedFrame.frame_as_pandas() and re-stamps the provenance onto df.attrs so the four checks below run byte-identically for both shapes.

The Validator runs four checks, in order:

1. Source-identity invariant. Reads df.attrs["source"] and compares against schema_cls._registered_source. If they differ and source_drift_reason is None, raises SourceMismatchError. If source_drift_reason is supplied, the drift is permitted and the audit log records the reason.
2. Required-column presence. Non-nullable columns must be present in df.columns.
3. Per-column dtype. Dispatched on ColumnSpec.dtype.
4. Enum value membership. dtype="enum" columns must have values in ColumnSpec.enum_values. Sample of violating values capped at 10.

• Parameters:
  • df (DataFrame | ProvenancedFrame) – The DataFrame to validate. Must carry df.attrs["source"], OR a ProvenancedFrame wrapper whose source / retrieved_at populate the equivalent attrs on unwrap.
  • schema_id (str) – Canonical schema ID (e.g. "schema.observation.v1").
  • source_drift_reason (str | None) – Reason string. If supplied, source mismatch is allowed; audit log records the reason.
• Return type: SchemaRegistration
• Returns: A SchemaRegistration recording the validation. Carries the full audit log (registered event always; source_drift_allowed event when source_drift_reason is supplied).
• Raises:
  • SchemaValidationError – column / dtype / enum / null-sentinel violations.
  • SourceMismatchError – source identity violated without opt-out.

Examples

The source-identity invariant fires when df.attrs["source"] differs from the schema’s registered canonical source (here schema.observation.v1 is registered to iem.archive):

>>> import pandas as pd
>>> from mostlyright.core import SourceMismatchError, validate_dataframe
>>> df = pd.DataFrame({"date": pd.to_datetime(["2025-01-06"])})
>>> df.attrs["source"] = "awc.live"
>>> try:
...     validate_dataframe(df, "schema.observation.v1")
... except SourceMismatchError as err:
...     print(err.data_source, "!=", err.schema_source)
awc.live != iem.archive

A full passing example requires the canonical column set; see packages/core/tests/core/test_validator.py for end-to-end fixtures.