Interpretation · September initialization · ONDJ and its months

Interpretation — national aggregate (All Ethiopia)

All tested corrections improve national ONDJ rainfall-amount CRPSS over raw ECMWF in 1993–2016 (raw -0.180; corrected +0.053 to +0.093). Performance varies for monthly tercile probabilities: 3 method–month combinations score below the raw model on RPSS (Detrended Quantile Mapping in Dec, Scaled Distribution Mapping in Dec and CDF-transform in Dec).
Accounting for the whole search. With Holm adjustment across all 60 method–target comparisons, the amount (CRPS) screening passes are Square-root-space Mean-Variance Scaling (Jan) and Gamma Quantile Mapping (Jan), and the tercile (RPS) passes are Square-root-space Mean-Variance Scaling (Nov, Jan), Power Transformation (Jan) and Gamma Quantile Mapping (Jan). Within-method passes that do not survive this check: Empirical Quantile Mapping Jan, Gamma Quantile Mapping ONDJ, Square-root-space Mean-Variance Scaling Dec, Square-root-space Mean-Variance Scaling ONDJ and Variance Scaling (unbounded) Jan.
Square-root-space Mean-Variance Scaling is at least as good as Mean-Variance Scaling in the national aggregate for every tested target and period, on amounts and unblended terciles (ONDJ CRPSS +0.080 vs +0.070 in 1993–2016, +0.189 vs +0.178 in 2017–2024). Its ONDJ amount screening pass does not survive the family-wide correction. Members clipped to 0 mm in the months: 4.5%–12.2% with Mean-Variance Scaling, 1.0%–2.9% with this method. This need not hold in each region; see the tables for the ONDJ R3 (Deyr) domain.
The 0.5–2 limits of Mean-Variance Scaling matter most where the raw model is far too narrow (Jan). With the same 1 mm fallback in both, removing the limits is the only difference: there the upper limit is active over 28% of the area and Mean-Variance Scaling stays too narrow (spread–error 0.83), while the unbounded form reaches 1.00.
Empirical Quantile Mapping reproduces the CHIRPS distribution and, with the dry policy, its dry share (totals ≤ 1 mm: Dec 19.0% vs CHIRPS 18.9%; Jan 16.7% vs CHIRPS 16.6%); it passes the screening for Jan, but its ONDJ amount skill is lower than Mean-Variance Scaling (+0.053 vs +0.070). A possible explanation is sampling noise in an empirical quantile function built from 23 CHIRPS values per fold; the results do not isolate it.
Scaled Distribution Mapping follows the published algorithm (Switanek et al. 2017, steps 1–7; checked numerically against an independent implementation of the steps). Where the model is drier than CHIRPS it cannot add rain: in Jan 46% of model members exceed 1 mm against 83% of CHIRPS years, and SDM, by design, keeps the model’s number of rain members, so its corrected dry share is 55% (CHIRPS 17%).
Delta Change has a high ONDJ CRPSS in 1993–2016 (+0.093) but the gain does not pass the screening (Holm p = 0.38), it is lower in 2017–2024 (+0.114 vs +0.178), its ensemble is too wide (spread–error 1.16) and its tercile skill is low. It is the CHIRPS climatology rescaled by the forecast’s mean: historical scenarios, not corrected members.
The methods that use the forecast’s own distribution (Detrended Quantile Mapping, Quantile Delta Mapping, Scaled Distribution Mapping, CDF-transform) score lower on ONDJ terciles (-0.005 to +0.011 vs +0.035) and none passes the season screening. A possible explanation is that the distribution of a single forecast’s 25–51 members is a noisy estimate of the “future” distribution these methods were designed for; the results do not isolate the cause. Where model quantiles are dry, QDM falls back to the CHIRPS quantile (members affected: Dec 32% and Jan 54%), so its quantile changes are only partly preserved.
Linear Scaling and Local Intensity Scaling correct the mean only (LOCI nearly so for totals that are always wet), so the ensemble stays too narrow (spread–error 0.78 and 0.77 for ONDJ).
Dec remains hard: every method has negative amount skill over 2017–2024.
No tested method demonstrates a statistically significant improvement over Mean-Variance Scaling for the published tercile probabilities under this validation protocol, although several have slightly higher point estimates. Choosing the method each year from training-only scores, the probability procedure (SEL-P) scores ONDJ blended RPSS +0.040 against +0.039, confirmed for none. Amounts: the amount procedure (SEL-A) scores CRPSS +0.093 against +0.070, confirmed for Jan; fixed methods confirmed against the family of all declared methods and targets: Square-root-space Mean-Variance Scaling (Jan) and Gamma Quantile Mapping (Jan).
The climatology blend narrows the differences. Across the methods, ONDJ RPSS (1993–2016) spans 0.025 after the blend against 0.044 for the unblended screening counts. A better amount correction is trusted more by the blend: in Jan the fitted weight on climatology is 0.60 with Square-root-space Mean-Variance Scaling against 0.91 with Mean-Variance Scaling.
Choosing by training scores can backfire. For Oct, SEL-A chose Delta Change from the 1993–2016 scores; over 2017–2024 it has a worse mean CRPS (Δ +0.937 mm, 95% interval -0.229 to +2.405 mm; better than Mean-Variance Scaling in 4 of 8 years, with the largest losses in 2019 and 2023); for ONDJ, SEL-A chose Delta Change from the 1993–2016 scores; over 2017–2024 it has a worse mean CRPS (Δ +3.217 mm, 95% interval -1.217 to +9.206 mm; better than Mean-Variance Scaling in 5 of 8 years, with the largest losses in 2019 and 2023); for Oct, SEL-P chose Gamma Quantile Mapping from the 1993–2016 scores; over 2017–2024 it has a worse mean RPS (Δ +0.003, 95% interval -0.001 to +0.006; better than Mean-Variance Scaling in 4 of 8 years, with the largest losses in 2024 and 2019); for Dec, SEL-P chose Square-root-space Mean-Variance Scaling from the 1993–2016 scores; over 2017–2024 it has a worse mean RPS (Δ +0.001, 95% interval -0.012 to +0.012; better than Mean-Variance Scaling in 2 of 8 years, with the largest losses in 2023 and 2021). These choices fail the no-harm rule; where the interval includes 0 they do not establish a significant deterioration. A method that ranks first on the training years is not, on that alone, a replacement.
Where the confirmed amount gains come from (climate groups from each fold’s training years; contributions add up to the national difference from Mean-Variance Scaling): Square-root-space Mean-Variance Scaling in Jan (Δ CRPS -0.479 mm): all ≤ 1 mm -0.003 mm (1%; 9% of the area), mixed -0.018 mm (4%; 11% of the area), always wet -0.458 mm (96%; 80% of the area); Gamma Quantile Mapping in Jan (Δ CRPS -0.365 mm): all ≤ 1 mm +0.010 mm (-3%; 9% of the area), mixed +0.003 mm (-1%; 11% of the area), always wet -0.378 mm (104%; 80% of the area).
No cell of All Ethiopia has exactly 0 mm in every training year, for any target, at 0.25° with CHIRPS. The all-zero guard (a separate MVS variant, outside the declared family) therefore never acts and equals Mean-Variance Scaling: the non-zero totals that Mean-Variance Scaling gives where every training year is 0 mm (its spread ratio cannot fall below 0.5) do not arise in these data. The guard remains a documented option; a finite history of zero rainfall does not make rain impossible.

Caveats

Screening skill is from 1993–2016 leave-one-year-out (clean) and 2017–2024 (exploratory: these years were seen while the operational method was chosen).
Screening tercile scores use smoothed member counts without the climatology blend; the confirmation scores the blended product.
Two multiplicity families are shown: within one method across its targets (✓) and across all methods and targets (✓✓). The confirmation’s selection-aware procedures include the method choice in the evaluation itself.
Scores are area-weighted over many correlated cells but tested year by year, so each year counts once.
The ordinary CRPS scores the delivered ensemble; the fair CRPS is a diagnostic under a sampling interpretation that DC’s rescaled years and methods tied to the forecast mean or ranks only approximate.
Results are national aggregates unless a table says otherwise; a method can rank differently in a region (see the regional tables).