Concepts

Sources & provenance

Every row states which source produced it. When observation feeds overlap, a deterministic merge in your process picks one survivor. Public-data reads are local-first; the few hosted transports are explicit through delivery=“hosted” and never hidden behind source=.

Raw-as-reported

The observation merge keys reports by (station, event_time_utc, observation_type) and keeps a deterministic source-priority survivor. The SDK does not silently average conflicting providers or relabel one provider’s row as another’s.

This is fetch-time behavior, not a warehouse guarantee: the data lives upstream and providers can restate history. The local cache and fetch-provenance ledger make the exact data used by a run inspectable.

The dedup tiebreak

Multiple sources report the same observation. When fetched feeds overlap on (station, event_time_utc, observation_type), a client-side dedup keeps one row:

tiebreak  source  origin                      why it wins
────────  ──────  ──────────────────────────  ─────────────────────────
   3      awc     aviationweather.gov         The sensor's live feed
   2      iem     Iowa Environmental Mesonet  Near-real-time mirror
   1      ghcnh   NOAA hourly archive         Quality-flagged archive

Rawest source survives; first-reported survives within a source; ties break deterministically. This runs in your process at read time. It is a dedup rule for overlapping fetches, not a hosted “best source” record.

Fused vs pinned: the `source=` axis

The fused default is source-blind: you get the tiebreak survivor per observation. Pinning source="iem" isolates one authority end to end.

source= always means WHO produced the row, never transport or freshness (that is delivery=, a separate axis). Pinning changes row composition (full single-source coverage rather than the post-merge subset), so migrating a trained feature between fused and pinned is a retrain event.

The guard is loud: a model trained on pinned data and served fused (or vice versa) raises SourceMismatchError instead of silently drifting. Historical/live source consistency is the whole point of the axis.

Provenance you can inspect

Every observation row carries its source tag (bare lowercase: awc, iem, ghcnh).
mr.provenance(frame) reports row-level provenance for any composed frame.
The cache keeps a per-source fetch journal on disk (~/.mostlyright/cache/provenance/*.jsonl by default), so “where did this number come from” survives the session that fetched it.

Forecast and covariate sources

Forecast feeds (IEM MOS, gridded NWP) and covariates (satellite, CWOP) use separate source tags. Every row names its source, and satellite or CWOP data cannot produce a label column. Forecast timing varies by source: some runs carry exact issuance, some use a derived bound, and some cannot support a strict as-of claim.

Dataset API index

Dataset family	Python reference	TypeScript reference
Weather observations and climate	`mostlyright.weather`	`@mostlyrightmd/weather`
Forecasts and NWP	`weather.forecasts` and `weather.nwp`	`@mostlyrightmd/weather/forecasts`
Satellite	`weather.satellite()`	`satelliteHosted()`
Citizen weather stations	`weather.cwop`	Python only
Prediction markets	`mostlyright.markets`	`@mostlyrightmd/markets`
Economic indicators	`mostlyright.economy`	`@mostlyrightmd/economy`
Earnings transcripts	`finance.transcripts`	`@mostlyrightmd/finance`