Method selection
For each season, target and product the platform selects the approach the evidence supports, from every benchmark together: the 13 rainfall-amount corrections (bias correction), the probability-stage methods and the rainfall-distribution models (probability calibration). It uses the historical period (cross-validated 1993–2016, with the 2017 onward test years as a no-harm check) and the independent validation seasons scored since. It is updated automatically each time a new season is scored.
A selection is advice for a decision. The published forecasts use the published approach (Mean-Variance Scaling and the climatology blend) until a change is decided. 8 target–product combinations currently have a selected change; none is recommended yet.
How the selection works
- Candidate: a method is a candidate only if its benchmark confirms it better than the published approach — better over the training years with Benjamini–Hochberg-adjusted p < 0.05 (false discovery rate) across every method and target of its family, and not worse over the test years. The adjusted p-values are recomputed from the p-values each benchmark stored, within the same families; until 17 October 2026 the rule used Holm, which is stricter (decision of the project lead after the sensitivity analyses on the Research overview).
- Choice: a recommended candidate first, then the largest improvement over the training years. One change at a time: an amount correction and a probability or distribution method were each tested against the published approach, not together.
- Validation: every candidate is scored on the seasons after the benchmark (every bias correction through the validation runs; every distribution-model candidate through its own shadow forecast). It becomes recommended after 3 seasons that are not worse, or not supported if worse.
Published approach kept: no method is confirmed better. Candidate: confirmed historically, validation seasons being collected. Recommended: confirmed and not worse over the validation seasons. Not supported by validation: confirmed historically but worse over the validation seasons.
Every method against the published approach
Across all seasons and targets (five per season: the months and the season), All Ethiopia. Confirmed better: the selection rule's test (Benjamini–Hochberg p < 0.05 within the benchmark's family, not worse over the test years). Better 1993–2016 and better 2017+: targets where it is better on average, whatever the significance. Validation: targets of the independent seasons scored after the benchmark (2026 so far). Ranked by confirmed targets, then by the share better over 1993–2016.
Tercile probabilities (RPS against MVS + climatology blend)
| Method | Category | Confirmed better | Better 1993–2016 | Better 2017+ | Better in validation |
|---|---|---|---|---|---|
| Square-root-space Mean-Variance Scaling | Bias correction | 1: FMAM (Feb) Feb | 14 of 20 | 16 of 20 | 11 of 15 |
| Variance Scaling (unbounded) | Bias correction | 0 | 16 of 20 | 12 of 20 | 11 of 15 |
| Gamma Quantile Mapping | Bias correction | 0 | 14 of 20 | 16 of 20 | 10 of 15 |
| Local Intensity Scaling | Bias correction | 0 | 14 of 20 | 6 of 20 | 2 of 15 |
| Linear Scaling | Bias correction | 0 | 13 of 20 | 5 of 20 | 3 of 15 |
| Power Transformation | Bias correction | 0 | 9 of 20 | 17 of 20 | 10 of 15 |
| Temperature scaling | Probability calibration | 0 | 9 of 20 | 8 of 20 | — |
| Platt (sigmoid) calibration, one-vs-rest + normalization | Probability calibration | 0 | 9 of 20 | 6 of 20 | — |
| Beta calibration, one-vs-rest + normalization | Probability calibration | 0 | 8 of 20 | 9 of 20 | — |
| Regularized Dirichlet calibration | Probability calibration | 0 | 7 of 20 | 7 of 20 | — |
| Isotonic regression, one-vs-rest + normalization | Probability calibration | 0 | 5 of 20 | 8 of 20 | — |
| Bayesian model averaging for precipitation | Rainfall-distribution model | 0 | 5 of 20 | 3 of 20 | — |
| Quantile Delta Mapping | Bias correction | 0 | 4 of 20 | 7 of 20 | 5 of 15 |
| Regularized multinomial logistic regression | Probability calibration | 0 | 4 of 20 | 6 of 20 | — |
| Delta Change | Bias correction | 0 | 4 of 20 | 5 of 20 | 7 of 15 |
| CDF-transform | Bias correction | 0 | 4 of 20 | 5 of 20 | 5 of 15 |
| Ordinal (cumulative-logit) logistic regression | Probability calibration | 0 | 4 of 20 | 5 of 20 | — |
| Principal component regression (PCR) | Statistical downscaling | 0 | 3 of 20 | 8 of 20 | 11 of 15 |
| Detrended Quantile Mapping | Bias correction | 0 | 3 of 20 | 7 of 20 | 6 of 15 |
| EMOS, censored shifted gamma | Rainfall-distribution model | 0 | 3 of 20 | 6 of 20 | 9 of 15 |
| Empirical Quantile Mapping | Bias correction | 0 | 3 of 20 | 5 of 20 | 7 of 15 |
| Regularized quantile regression | Rainfall-distribution model | 0 | 3 of 20 | 5 of 20 | 3 of 5 |
| Leung–Dirichlet calibration | Two-stage calibration | 0 | 2 of 15 | 3 of 15 | 1 of 10 |
| Heteroscedastic extended logistic regression | Rainfall-distribution model | 0 | 2 of 20 | 8 of 20 | — |
| Scaled Distribution Mapping | Bias correction | 0 | 2 of 20 | 3 of 20 | 5 of 15 |
| Pseudocount smoothing (no blend) | Probability calibration | 0 | 2 of 20 | 2 of 20 | — |
| Canonical correlation analysis (CCA) | Statistical downscaling | 0 | 1 of 20 | 13 of 20 | 9 of 15 |
| Quantile regression forest | Rainfall-distribution model | 0 | 1 of 20 | 2 of 20 | — |
| Bayesian joint probability | Rainfall-distribution model | 0 | 0 of 20 | 1 of 20 | — |
Rainfall amounts (CRPS against the MVS-corrected ensemble)
| Method | Category | Confirmed better | Better 1993–2016 | Better 2017+ | Better in validation |
|---|---|---|---|---|---|
| Square-root-space Mean-Variance Scaling | Bias correction | 5: JJAS (May) Jun, JJAS (May) Jul, JJAS (May) Aug, JJAS (May) Sep, ONDJ (Sep) Jan | 20 of 20 | 18 of 20 | 10 of 15 |
| Gamma Quantile Mapping | Bias correction | 4: JJAS (May) Jun, JJAS (May) Aug, JJAS (May) Sep, ONDJ (Sep) Jan | 20 of 20 | 16 of 20 | 9 of 15 |
| Variance Scaling (unbounded) | Bias correction | 4: JJAS (May) Jul, JJAS (May) Aug, JJAS (May) Sep, ONDJ (Sep) Jan | 14 of 20 | 15 of 20 | 6 of 15 |
| Principal component regression (PCR) | Statistical downscaling | 3: JJAS (May) Aug, JJAS (May) Sep, JJAS (May) JJAS | 7 of 20 | 12 of 20 | 6 of 15 |
| EMOS, censored shifted gamma | Rainfall-distribution model | 2: FMAM (Jan) FMAM, JJAS (May) Sep | 15 of 20 | 12 of 20 | 5 of 15 |
| Power Transformation | Bias correction | 2: JJAS (May) Jul, JJAS (May) Aug | 12 of 20 | 16 of 20 | 9 of 15 |
| Regularized quantile regression | Rainfall-distribution model | 1: JJAS (May) Sep | 17 of 20 | 14 of 20 | 0 of 5 |
| Leung–Dirichlet calibration | Two-stage calibration | 1: ONDJ (Sep) Jan | 11 of 15 | 9 of 15 | 5 of 10 |
| Canonical correlation analysis (CCA) | Statistical downscaling | 1: JJAS (May) Sep | 7 of 20 | 12 of 20 | 6 of 15 |
| Bayesian model averaging for precipitation | Rainfall-distribution model | 0 | 13 of 20 | 7 of 20 | — |
| Heteroscedastic extended logistic regression | Rainfall-distribution model | 0 | 9 of 20 | 9 of 20 | — |
| Delta Change | Bias correction | 0 | 9 of 20 | 8 of 20 | 6 of 15 |
| Empirical Quantile Mapping | Bias correction | 0 | 6 of 20 | 7 of 20 | 7 of 15 |
| Quantile regression forest | Rainfall-distribution model | 0 | 5 of 20 | 3 of 20 | — |
| Quantile Delta Mapping | Bias correction | 0 | 4 of 20 | 5 of 20 | 3 of 15 |
| Bayesian joint probability | Rainfall-distribution model | 0 | 3 of 20 | 4 of 20 | — |
| Detrended Quantile Mapping | Bias correction | 0 | 2 of 20 | 4 of 20 | 5 of 15 |
| CDF-transform | Bias correction | 0 | 2 of 20 | 3 of 20 | 4 of 15 |
| Linear Scaling | Bias correction | 0 | 1 of 20 | 2 of 20 | 2 of 15 |
| Local Intensity Scaling | Bias correction | 0 | 0 of 20 | 2 of 20 | 2 of 15 |
| Scaled Distribution Mapping | Bias correction | 0 | 0 of 20 | 0 of 20 | 1 of 15 |
Code: scripts/method_summary.py, refreshed after each newly scored season.
January initialization · FMAM and its months
1 change selected · outlook · bias-correction evidence · probability-calibration evidence
| Target | Tercile probabilities | Rainfall amounts |
|---|---|---|
| Feb | Published approach kept | Published approach kept |
| Mar | Published approach kept | Published approach kept |
| Apr | Published approach kept | Published approach kept |
| May | Published approach kept | Published approach kept |
| FMAM | Published approach kept | EMOS, censored shifted gamma Candidate
changes the distribution model · validation 1 of 3 seasons |
Evidence for the selected changes
- FMAM, rainfall amounts: EMOS, censored shifted gamma (distribution model, phase 2) is confirmed better than Mean-Variance Scaling ensemble (published) for FMAM: CRPS (mm) difference -0.7207 over the training years (Benjamini–Hochberg p 0.0126; 19 of 24 years better), -0.7348 over the test years. Independent validation: 1 of 3 season(s), mean difference -1.762.
February initialization · FMAM and its months
1 change selected · outlook · bias-correction evidence · probability-calibration evidence
| Target | Tercile probabilities | Rainfall amounts |
|---|---|---|
| Feb | Square-root-space Mean-Variance Scaling Candidate
changes the amount correction · validation 1 of 3 seasons | Published approach kept |
| Mar | Published approach kept | Published approach kept |
| Apr | Published approach kept | Published approach kept |
| May | Published approach kept | Published approach kept |
| FMAM | Published approach kept | Published approach kept |
Evidence for the selected changes
- Feb, tercile probabilities: Square-root-space Mean-Variance Scaling (bias correction (amount correction)) is confirmed better than Mean-Variance Scaling + climatology blend (published) for Feb: RPS difference -0.03669 over the training years (Benjamini–Hochberg p 0.0264; 18 of 24 years better), -0.006635 over the test years. Independent validation: 1 of 3 season(s), mean difference -0.006004.
May initialization · JJAS and its months
5 changes selected · outlook · bias-correction evidence · probability-calibration evidence
| Target | Tercile probabilities | Rainfall amounts |
|---|---|---|
| Jun | Published approach kept | Gamma Quantile Mapping Candidate
changes the amount correction · validation 1 of 3 seasons |
| Jul | Published approach kept | Variance Scaling (unbounded) Candidate
changes the amount correction · validation 1 of 3 seasons |
| Aug | Published approach kept | Principal component regression (PCR) Candidate
changes the statistical downscaling · validation 1 of 3 seasons |
| Sep | Published approach kept | Regularized quantile regression Candidate
changes the distribution model · validation 1 of 3 seasons |
| JJAS | Published approach kept | Principal component regression (PCR) Candidate
changes the statistical downscaling · validation 1 of 3 seasons |
Evidence for the selected changes
- Jun, rainfall amounts: Gamma Quantile Mapping (bias correction (amount correction)) is confirmed better than Mean-Variance Scaling ensemble (published) for Jun: CRPS (mm) difference -0.1309 over the training years (Benjamini–Hochberg p 0.0488; 18 of 24 years better), -0.1626 over the test years. Independent validation: 1 of 3 season(s), mean difference +0.3062. Also confirmed (one change at a time): Square-root-space Mean-Variance Scaling (amount correction).
- Jul, rainfall amounts: Variance Scaling (unbounded) (bias correction (amount correction)) is confirmed better than Mean-Variance Scaling ensemble (published) for Jul: CRPS (mm) difference -0.1386 over the training years (Benjamini–Hochberg p 0; 23 of 24 years better), -0.1399 over the test years. Independent validation: 1 of 3 season(s), mean difference -0.4168. Also confirmed (one change at a time): Square-root-space Mean-Variance Scaling (amount correction); Power Transformation (amount correction).
- Aug, rainfall amounts: Principal component regression (PCR) (statistical downscaling (docs/53)) is confirmed better than Mean-Variance Scaling ensemble (published) for Aug: CRPS (mm) difference -1.309 over the training years (Benjamini–Hochberg p 0.0246; 16 of 24 years better), -0.7517 over the test years. Independent validation: 1 of 3 season(s), mean difference +3.27. Also confirmed (one change at a time): Variance Scaling (unbounded) (amount correction); Square-root-space Mean-Variance Scaling (amount correction); Power Transformation (amount correction); Gamma Quantile Mapping (amount correction).
- Sep, rainfall amounts: Regularized quantile regression (distribution model, phase 2) is confirmed better than Mean-Variance Scaling ensemble (published) for Sep: CRPS (mm) difference -1.083 over the training years (Benjamini–Hochberg p 0.0127; 18 of 24 years better), -0.5432 over the test years. Independent validation: 1 of 3 season(s), mean difference +1.588. Also confirmed (one change at a time): Gamma Quantile Mapping (amount correction); Variance Scaling (unbounded) (amount correction); Square-root-space Mean-Variance Scaling (amount correction); EMOS, censored shifted gamma (distribution model); Canonical correlation analysis (CCA) (statistical downscaling); Principal component regression (PCR) (statistical downscaling).
- JJAS, rainfall amounts: Principal component regression (PCR) (statistical downscaling (docs/53)) is confirmed better than Mean-Variance Scaling ensemble (published) for JJAS: CRPS (mm) difference -2.904 over the training years (Benjamini–Hochberg p 0.0443; 17 of 24 years better), -3.639 over the test years. Independent validation: 1 of 3 season(s), mean difference +12.53.
September initialization · ONDJ and its months
1 change selected · outlook · bias-correction evidence · probability-calibration evidence
| Target | Tercile probabilities | Rainfall amounts |
|---|---|---|
| Oct | Published approach kept | Published approach kept |
| Nov | Published approach kept | Published approach kept |
| Dec | Published approach kept | Published approach kept |
| Jan | Published approach kept | Square-root-space Mean-Variance Scaling Candidate
changes the amount correction · validation 0 of 3 seasons |
| ONDJ | Published approach kept | Published approach kept |
Evidence for the selected changes
- Jan, rainfall amounts: Square-root-space Mean-Variance Scaling (bias correction (amount correction)) is confirmed better than Mean-Variance Scaling ensemble (published) for Jan: CRPS (mm) difference -0.4787 over the training years (Benjamini–Hochberg p 0.0106; 18 of 24 years better), -0.6902 over the test years. Independent validation: 0 of 3 seasons scored so far. Also confirmed (one change at a time): Variance Scaling (unbounded) (amount correction); Gamma Quantile Mapping (amount correction); Leung–Dirichlet calibration (Leung–Dirichlet calibration).
Rule: config/method_selection_rule.json (fixed before further validation seasons are scored) · code: scripts/method_selection.py · write-up: docs/50.