Data Provenance

Data Provenance#

Consolidated, human-readable record of every data source feeding the hydroclim_risk pipeline (R = 100 x P x S x E x V, see methodology.md). Every acquired GeoTIFF also carries its own source/license/citation tags (readable via rasterio.open(path).tags()) – this document exists to give the full list in one place, since no single GeoTIFF’s tags show what was considered and excluded, or how sub-indicators combine. Machine-readable versions of the same facts live in config/exposure_data.yaml and config/vulnerability_indicators.yaml; this file is generated by hand from those and should be kept in sync when they change.

Last updated: 2026-07-25 (V_wet terrain/soil/river-proximity additions).

Hazard / probability inputs#

Not acquired by this repo – produced by a separate, mature sibling project (D:\extremes-climate-indices) and consumed as trusted upstream input via hydroclim_risk/ingestion/. See references/project-context.md and the project-sibling-pipeline memory for the full trust/verification history.

Source	Content	Coverage	Notes
`corrected_1993_2025.nc`	Bias-corrected ensemble precipitation (`pr`, mm/day), dims (lat, lon, time, realization)	1993-2025, May 2-Oct 31 each year	Ragged ensemble: only the first 25 of up to 51 `realization` slots are valid for the full period – always slice `[0:25]`
`corrected_2026.nc`	Same variable/schema, forecast initialization	2026, May 2-Oct 31	Clean 25/25 realizations
`et_chirps_pr_r25_1993_2025.nc`	CHIRPS observational precipitation (`precip`, mm/day), dims (time, lat, lon)	1993-01-01 to 2025-12-31, daily	No ensemble dimension (observational)
`outputs/geotiff/*.tif` (sibling repo)	Derived indicators: rainfall total, SPI, CDD, CWD, Rx1day, Rx5day (raw + percentile + climatology forms)	5 periods (Jun/Jul/Aug/Sep/JJAS) x 2026	`*_percentile_climatology_mean.tif` and `spi_climatology_mean.tif` were mislabeled (byte-identical to the `_mean.tif` raw mm product) – quarantined in `outputs/geotiff/_deprecated_mislabeled/` and hard-blocked in `hydroclim_risk.ingestion.geotiff`’s `BLOCKED_PRODUCTS`. Separately, `spi_mean`/`spi_median` have isolated cells beyond +/-4 (max observed \|4.75\|) on disk – capped to +/-3.09 at consumption time (`apply_spi_cap=True`, the default, in `ingestion.geotiff.load_indicator` and `ingestion.members.load_member_indicator`), not by editing the sibling repo’s files, so every real risk map already uses capped SPI even though the raw on-disk files remain uncapped
`outputs/netcdf/*.nc` (sibling repo)	Per-member (realization-preserving) exports of the same 6 indicators	6 indicators x 5 periods = 30 files	Needed because GeoTIFF exports collapse the ensemble; read via `hydroclim_risk.ingestion.members`
`boundaries/eth_shapefile/eth_admin{0-3}.shp`	Ethiopia administrative boundaries, admin levels 0-3	National	Bundled project asset, not a download; EPSG:4326, zero invalid geometries

Exposure datasets#

All free, no-registration, no-API-key, resampled onto the 0.25 deg analysis grid. Acquired via src/hydroclim_risk/acquisition/.

Dataset	Source	License	Acquisition module	Output file(s)
Population	WorldPop	CC-BY 4.0	`population.py`	`ethiopia_population.tif`
Cropland (total/irrigated/rainfed)	ESA WorldCover 10m 2021 v200	CC-BY 4.0	`cropland.py`	`ethiopia_cropland.tif` (+ derived irrigated/rainfed split in `exposure/`)
Livestock (cattle/sheep/goats)	FAO Gridded Livestock of the World v4 (2020)	CC-BY 4.0	`livestock.py`	`ethiopia_livestock_{cattle,sheep,goats}.tif`
Built-up surface	JRC GHSL GHS-BUILT-S R2023A	CC-BY 4.0	`ghs_built.py`	`ethiopia_ghs_built.tif`
Health facilities	Global Healthsites Mapping Project (via HDX)	ODbL	`healthsites.py`	`ethiopia_healthsites.tif`
Roads, buildings	OpenStreetMap (via Geofabrik)	ODbL	`infrastructure.py`	`ethiopia_roads.tif`, `ethiopia_buildings.tif`

Vulnerability datasets#

Dataset	Role	Source	License	Acquisition module	Output file
Relative Wealth Index	drought + wet sensitivity	Meta Data for Good	CC-BY 4.0	`poverty.py`	`ethiopia_poverty_rwi.tif`
Aridity Index	drought sensitivity	CGIAR-CSI Global Aridity v3.1	CC-BY 4.0	`aridity.py`	`ethiopia_aridity.tif`
Irrigation (% area)	drought adaptive capacity	FAO/Bonn Global Map of Irrigation Areas v5	CC-BY 4.0	`irrigation.py`	`ethiopia_irrigation_gmia.tif`
Electrification no-access	drought + wet adaptive capacity	IIASA/Falchetta et al. GDESSA	CC-BY 4.0	`gdessa.py`	`ethiopia_gdessa_no_access.tif` (rate derived in `vulnerability/indicators.py`)
Health facility count	wet adaptive capacity	Global Healthsites Mapping Project	ODbL	`healthsites.py`	`ethiopia_healthsites.tif` (shared with exposure)
Elevation	wet sensitivity	NOAA ETOPO 2022 (30 arcsec global relief model)	Public domain (not for navigation)	`terrain.py`	`ethiopia_elevation.tif`
Slope	wet sensitivity	NOAA ETOPO 2022, derived (native-resolution gradient, then mean-aggregated)	Public domain	`terrain.py`	`ethiopia_slope.tif`
Topsoil clay content	wet sensitivity	ISRIC SoilGrids 2.0 (0-5cm mean)	CC-BY 4.0	`soil_drainage.py`	`ethiopia_soil_clay.tif`
River/water-body distance	wet sensitivity	OpenStreetMap waterways + water polygons (via Geofabrik)	ODbL	`river_proximity.py`	`ethiopia_river_distance.tif`

Bold rows added 2026-07-25 to fix V_wet’s sensitivity term, which previously combined poverty (weight 1.0) with no wetness-specific signal. All five wet- sensitivity indicators (poverty + the four above) are now weighted equally at 0.20 each – see config/vulnerability_indicators.yaml for the reasoning.

Why these three sources, specifically#

ETOPO 2022 was chosen over Copernicus DEM GLO-30 (also verified accessible, AWS S3, no auth) because it ships as a single global 30-arcsecond GeoTIFF with HTTP Range-request support, avoiding a ~180-tile download for Ethiopia’s domain (Copernicus DEM tiles are 1x1 degree). At this project’s 0.25 deg analysis resolution, ETOPO’s ~900m native resolution is more than sufficient.
SoilGrids via ISRIC’s WebDAV static file server (files.isric.org/soilgrids/latest/data/) was used instead of the rest.soilgrids.org REST API, which is documented as paused/rate-limited (5 calls/min) and unsuitable for a bulk grid.
River proximity required no new download: Geofabrik’s Ethiopia OSM extract, already cached for infrastructure.py’s roads/buildings layers, also bundles gis_osm_waterways_free_1.shp and gis_osm_water_a_free_1.shp.

No full Topographic Wetness Index (TWI)#

A true TWI requires flow-accumulation (upslope contributing area), which needs a library like richdem, pysheds, or whitebox – none confirmed available in this environment. Elevation + slope is used instead, a standard, documented simplification when flow-accumulation tooling isn’t available. This is an explicit approximation, not a full TWI.

Deliberately excluded sources#

Source	Reason excluded
NASA SEDAC Global Gridded Relative Deprivation Index	Requires a NASA Earthdata Login, unlike every other source used
Ookla Open Data (connectivity)	License is CC BY-NC-SA 4.0 (non-commercial) – inconsistent with every other source’s commercial-use-permitted license
VNP46A4 (VIIRS nighttime lights)	Considered for exposure/wealth proxying; found to add no spatial discrimination beyond existing layers at this project’s resolution
ND-GAIN Country Index	Country-level only (single value per nation) – no spatial discrimination within Ethiopia
GDL / DHS / World Bank / OCHA HDX admin-level datasets	Sub-national administrative-unit data, not gridded; would require a different aggregation approach than every other layer here. Deliberately out of scope, not integrated.
OSM railways	Considered as an infrastructure exposure layer; not extracted (out of scope for this round)

Known caveats affecting interpretation#

Domain rectangle extends beyond Ethiopia’s border. The 0.25 deg analysis grid (33-48E, 3-15N) is a bounding rectangle, not Ethiopia’s actual shape – it includes slivers of neighboring countries and, at its northeastern corner, a piece of the Red Sea. Global raster sources (aridity, elevation, soil clay) have real values there (e.g. elevation includes Red Sea bathymetry down to several thousand meters below sea level). Every indicator is normalized (robust 5th/95th-percentile) over the whole domain grid, not just Ethiopia’s territory, consistent with how aridity has always been handled in this pipeline – final risk output is still restricted to Ethiopia’s actual footprint downstream (~51% domain coverage, driven mainly by population/exposure data being Ethiopia-specific).
River-proximity edge effect. Geofabrik’s OSM extract is clipped to Ethiopia’s territory, so a river just across the border may be missed for a handful of border cells, slightly overestimating their distance to water.