# Data Provenance

Consolidated, human-readable record of every data source feeding the
`hydroclim_risk` pipeline (`R = 100 x P x S x E x V`, see
[methodology.md](methodology.md)). Every acquired GeoTIFF also carries its own
`source`/`license`/`citation` tags (readable via `rasterio.open(path).tags()`)
-- this document exists to give the full list in one place, since no single
GeoTIFF's tags show what was considered and excluded, or how sub-indicators
combine. Machine-readable versions of the same facts live in
`config/exposure_data.yaml` and `config/vulnerability_indicators.yaml`; this
file is generated by hand from those and should be kept in sync when they
change.

Last updated: 2026-07-25 (V_wet terrain/soil/river-proximity additions).

## Hazard / probability inputs

Not acquired by this repo -- produced by a separate, mature sibling project
(`D:\extremes-climate-indices`) and consumed as trusted upstream input via
`hydroclim_risk/ingestion/`. See `references/project-context.md` and the
`project-sibling-pipeline` memory for the full trust/verification history.

| Source | Content | Coverage | Notes |
|---|---|---|---|
| `corrected_1993_2025.nc` | Bias-corrected ensemble precipitation (`pr`, mm/day), dims (lat, lon, time, realization) | 1993-2025, May 2-Oct 31 each year | Ragged ensemble: only the first 25 of up to 51 `realization` slots are valid for the full period -- always slice `[0:25]` |
| `corrected_2026.nc` | Same variable/schema, forecast initialization | 2026, May 2-Oct 31 | Clean 25/25 realizations |
| `et_chirps_pr_r25_1993_2025.nc` | CHIRPS observational precipitation (`precip`, mm/day), dims (time, lat, lon) | 1993-01-01 to 2025-12-31, daily | No ensemble dimension (observational) |
| `outputs/geotiff/*.tif` (sibling repo) | Derived indicators: rainfall total, SPI, CDD, CWD, Rx1day, Rx5day (raw + percentile + climatology forms) | 5 periods (Jun/Jul/Aug/Sep/JJAS) x 2026 | `*_percentile_climatology_mean.tif` and `spi_climatology_mean.tif` were mislabeled (byte-identical to the `_mean.tif` raw mm product) -- quarantined in `outputs/geotiff/_deprecated_mislabeled/` and hard-blocked in `hydroclim_risk.ingestion.geotiff`'s `BLOCKED_PRODUCTS`. Separately, `spi_mean`/`spi_median` have isolated cells beyond +/-4 (max observed \|4.75\|) on disk -- capped to +/-3.09 at consumption time (`apply_spi_cap=True`, the default, in `ingestion.geotiff.load_indicator` and `ingestion.members.load_member_indicator`), not by editing the sibling repo's files, so every real risk map already uses capped SPI even though the raw on-disk files remain uncapped |
| `outputs/netcdf/*.nc` (sibling repo) | Per-member (realization-preserving) exports of the same 6 indicators | 6 indicators x 5 periods = 30 files | Needed because GeoTIFF exports collapse the ensemble; read via `hydroclim_risk.ingestion.members` |
| `boundaries/eth_shapefile/eth_admin{0-3}.shp` | Ethiopia administrative boundaries, admin levels 0-3 | National | Bundled project asset, not a download; EPSG:4326, zero invalid geometries |

## Exposure datasets

All free, no-registration, no-API-key, resampled onto the 0.25 deg analysis
grid. Acquired via `src/hydroclim_risk/acquisition/`.

| Dataset | Source | License | Acquisition module | Output file(s) |
|---|---|---|---|---|
| Population | WorldPop | CC-BY 4.0 | `population.py` | `ethiopia_population.tif` |
| Cropland (total/irrigated/rainfed) | ESA WorldCover 10m 2021 v200 | CC-BY 4.0 | `cropland.py` | `ethiopia_cropland.tif` (+ derived irrigated/rainfed split in `exposure/`) |
| Livestock (cattle/sheep/goats) | FAO Gridded Livestock of the World v4 (2020) | CC-BY 4.0 | `livestock.py` | `ethiopia_livestock_{cattle,sheep,goats}.tif` |
| Built-up surface | JRC GHSL GHS-BUILT-S R2023A | CC-BY 4.0 | `ghs_built.py` | `ethiopia_ghs_built.tif` |
| Health facilities | Global Healthsites Mapping Project (via HDX) | ODbL | `healthsites.py` | `ethiopia_healthsites.tif` |
| Roads, buildings | OpenStreetMap (via Geofabrik) | ODbL | `infrastructure.py` | `ethiopia_roads.tif`, `ethiopia_buildings.tif` |

## Vulnerability datasets

| Dataset | Role | Source | License | Acquisition module | Output file |
|---|---|---|---|---|---|
| Relative Wealth Index | drought + wet sensitivity | Meta Data for Good | CC-BY 4.0 | `poverty.py` | `ethiopia_poverty_rwi.tif` |
| Aridity Index | drought sensitivity | CGIAR-CSI Global Aridity v3.1 | CC-BY 4.0 | `aridity.py` | `ethiopia_aridity.tif` |
| Irrigation (% area) | drought adaptive capacity | FAO/Bonn Global Map of Irrigation Areas v5 | CC-BY 4.0 | `irrigation.py` | `ethiopia_irrigation_gmia.tif` |
| Electrification no-access | drought + wet adaptive capacity | IIASA/Falchetta et al. GDESSA | CC-BY 4.0 | `gdessa.py` | `ethiopia_gdessa_no_access.tif` (rate derived in `vulnerability/indicators.py`) |
| Health facility count | wet adaptive capacity | Global Healthsites Mapping Project | ODbL | `healthsites.py` | `ethiopia_healthsites.tif` (shared with exposure) |
| **Elevation** | **wet sensitivity** | **NOAA ETOPO 2022 (30 arcsec global relief model)** | **Public domain (not for navigation)** | **`terrain.py`** | **`ethiopia_elevation.tif`** |
| **Slope** | **wet sensitivity** | **NOAA ETOPO 2022, derived (native-resolution gradient, then mean-aggregated)** | **Public domain** | **`terrain.py`** | **`ethiopia_slope.tif`** |
| **Topsoil clay content** | **wet sensitivity** | **ISRIC SoilGrids 2.0 (0-5cm mean)** | **CC-BY 4.0** | **`soil_drainage.py`** | **`ethiopia_soil_clay.tif`** |
| **River/water-body distance** | **wet sensitivity** | **OpenStreetMap waterways + water polygons (via Geofabrik)** | **ODbL** | **`river_proximity.py`** | **`ethiopia_river_distance.tif`** |

Bold rows added 2026-07-25 to fix V_wet's sensitivity term, which previously
combined poverty (weight 1.0) with no wetness-specific signal. All five wet-
sensitivity indicators (poverty + the four above) are now weighted equally at
0.20 each -- see `config/vulnerability_indicators.yaml` for the reasoning.

### Why these three sources, specifically

- **ETOPO 2022** was chosen over Copernicus DEM GLO-30 (also verified
  accessible, AWS S3, no auth) because it ships as a single global
  30-arcsecond GeoTIFF with HTTP Range-request support, avoiding a ~180-tile
  download for Ethiopia's domain (Copernicus DEM tiles are 1x1 degree). At
  this project's 0.25 deg analysis resolution, ETOPO's ~900m native
  resolution is more than sufficient.
- **SoilGrids via ISRIC's WebDAV static file server**
  (`files.isric.org/soilgrids/latest/data/`) was used instead of the
  `rest.soilgrids.org` REST API, which is documented as paused/rate-limited
  (5 calls/min) and unsuitable for a bulk grid.
- **River proximity** required no new download: Geofabrik's Ethiopia OSM
  extract, already cached for `infrastructure.py`'s roads/buildings layers,
  also bundles `gis_osm_waterways_free_1.shp` and `gis_osm_water_a_free_1.shp`.

### No full Topographic Wetness Index (TWI)

A true TWI requires flow-accumulation (upslope contributing area), which
needs a library like `richdem`, `pysheds`, or `whitebox` -- none confirmed
available in this environment. Elevation + slope is used instead, a standard,
documented simplification when flow-accumulation tooling isn't available.
This is an explicit approximation, not a full TWI.

## Deliberately excluded sources

| Source | Reason excluded |
|---|---|
| NASA SEDAC Global Gridded Relative Deprivation Index | Requires a NASA Earthdata Login, unlike every other source used |
| Ookla Open Data (connectivity) | License is CC BY-NC-SA 4.0 (non-commercial) -- inconsistent with every other source's commercial-use-permitted license |
| VNP46A4 (VIIRS nighttime lights) | Considered for exposure/wealth proxying; found to add no spatial discrimination beyond existing layers at this project's resolution |
| ND-GAIN Country Index | Country-level only (single value per nation) -- no spatial discrimination within Ethiopia |
| GDL / DHS / World Bank / OCHA HDX admin-level datasets | Sub-national administrative-unit data, not gridded; would require a different aggregation approach than every other layer here. Deliberately out of scope, not integrated. |
| OSM railways | Considered as an infrastructure exposure layer; not extracted (out of scope for this round) |

## Known caveats affecting interpretation

- **Domain rectangle extends beyond Ethiopia's border.** The 0.25 deg
  analysis grid (33-48E, 3-15N) is a bounding rectangle, not Ethiopia's
  actual shape -- it includes slivers of neighboring countries and, at its
  northeastern corner, a piece of the Red Sea. Global raster sources
  (aridity, elevation, soil clay) have real values there (e.g. elevation
  includes Red Sea bathymetry down to several thousand meters below sea
  level). Every indicator is normalized (robust 5th/95th-percentile) over
  the *whole* domain grid, not just Ethiopia's territory, consistent with
  how aridity has always been handled in this pipeline -- final risk output
  is still restricted to Ethiopia's actual footprint downstream (~51%
  domain coverage, driven mainly by population/exposure data being
  Ethiopia-specific).
- **River-proximity edge effect.** Geofabrik's OSM extract is clipped to
  Ethiopia's territory, so a river just across the border may be missed for
  a handful of border cells, slightly overestimating their distance to
  water.