Leung–Dirichlet calibration
A two-stage calibration: a modified-Leung member-wise affine bias correction followed by an
identity-anchored Dirichlet recalibration of the tercile probabilities. It comes from process_calib.py by
Á. G. Muñoz and T. Demissie (ILRI), implemented here as separate
methods and tested, as the uploaded code and stage by stage, against the published pipeline with the same data, cross-validation
and tests as the other benchmarks.
A comparison, not a published forecast. The published forecasts keep the published approach (Mean-Variance Scaling, smoothed member counts and the climatology blend). A method confirmed here becomes a candidate in the method selection and needs validation seasons before any change is decided.
What the uploaded code does
- Regridding. SEAS5 seasonal totals put on the CHIRPS grid (nearest neighbour). The project's own 0.25° SEAS5 totals are used here.
- Bias correction: modified-Leung member-wise affine. For every member index m its own climatology over the training years (mean μm, SD σm); the corrected value is μobs + rm(x − μm), with rm = σobs / σeff,m limited to 0.5–2, σeff,m = 0.7 σm + 0.3 σpool, r = 1 where both SDs are below 1 mm, and totals clipped at 0.
- Tercile probabilities. The fraction of corrected members below the lower and above the upper CHIRPS tercile (no smoothing, no climatology blend).
- Probability calibration: Identity-Anchored Dirichlet (IAD). p′ = softmax(A log p + b), fitted by penalised log-likelihood on the hindcast probabilities and observed categories pooled over the cells; the penalties pull A towards the identity (off-diagonal 0.01, diagonal and b 0.001), so with little evidence the probabilities are left as they are.
- SPBI (a pasture index from NDVI and the calibrated rainfall): not part of rainfall calibration, not implemented.
January initialization · FMAM and its months
Methods compared
| Method | What it is | Changes |
|---|---|---|
| Leung–Dirichlet (as uploaded) | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | both stages (the uploaded pipeline) |
| Leung + member counts | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | both stages, without the Dirichlet |
| MVS + Dirichlet | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | probability stage |
| Leung + published blend | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | amount correction |
| Leung affine (amounts) | Modified-Leung affine ensemble (amounts) | amount correction |
References: Published probabilities: MVS -> smoothed counts -> climatology blend; MVS-corrected ensemble (published amount correction). Every method is fitted inside the folds: outer leave-one-year-out over 1993–2016, and fitted on all of 1993–2016 for the test years 2017–2025. The Dirichlet is fitted, as in the uploaded code, on the training years' probabilities from the affine correction fitted on those years. Confirmed better: better over the training years with Holm-adjusted p < 0.05 across all methods and targets of the comparison, and not worse over the test years. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026).
Confirmed better than the published approach: none.
Tercile probabilities (RPS)
| Target | Method | RPSS 1993–2016 (LOYO) | RPSS 2017–2025 | ΔRPS vs reference (training) | Holm p | Years better | ΔRPS (test years) | Decision |
|---|---|---|---|---|---|---|---|---|
| Feb | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.033 | +0.020 | — | — | — | — | reference |
| Feb | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.079 | -0.017 | -0.0211 | 0.207 | 17 of 24 | +0.0161 | not confirmed; worse over the test years |
| Feb | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | +0.027 | -0.068 | +0.0025 | 1.000 | 12 of 24 | +0.0378 | not confirmed; worse over the test years |
| Feb | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.076 | +0.033 | -0.0195 | 0.164 | 15 of 24 | -0.0055 | not confirmed |
| Feb | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.050 | +0.017 | -0.0081 | 0.466 | 14 of 24 | +0.0012 | not confirmed; worse over the test years |
| Mar | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.002 | +0.008 | — | — | — | — | reference |
| Mar | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | -0.003 | -0.002 | +0.0023 | 1.000 | 10 of 24 | +0.0042 | not confirmed; worse over the test years |
| Mar | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.080 | -0.045 | +0.0372 | 1.000 | 9 of 24 | +0.0237 | not confirmed; worse over the test years |
| Mar | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | -0.005 | +0.001 | +0.0031 | 1.000 | 9 of 24 | +0.0032 | not confirmed; worse over the test years |
| Mar | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.002 | +0.010 | -0.0001 | 1.000 | 12 of 24 | -0.0012 | not confirmed |
| Apr | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.001 | +0.002 | — | — | — | — | reference |
| Apr | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | -0.002 | +0.003 | +0.0014 | 1.000 | 9 of 24 | -0.0006 | not confirmed |
| Apr | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.027 | -0.033 | +0.0129 | 1.000 | 12 of 24 | +0.0158 | not confirmed; worse over the test years |
| Apr | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | -0.004 | +0.005 | +0.0025 | 1.000 | 11 of 24 | -0.0012 | not confirmed |
| Apr | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.001 | +0.001 | +0.0000 | 1.000 | 12 of 24 | +0.0007 | not confirmed; worse over the test years |
| May | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.028 | +0.004 | — | — | — | — | reference |
| May | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.021 | -0.007 | +0.0033 | 1.000 | 8 of 24 | +0.0047 | not confirmed; worse over the test years |
| May | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | +0.026 | -0.026 | +0.0008 | 1.000 | 13 of 24 | +0.0129 | not confirmed; worse over the test years |
| May | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.022 | -0.007 | +0.0028 | 1.000 | 8 of 24 | +0.0046 | not confirmed; worse over the test years |
| May | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.028 | +0.002 | +0.0002 | 1.000 | 13 of 24 | +0.0009 | not confirmed; worse over the test years |
| FMAM | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.032 | +0.017 | — | — | — | — | reference |
| FMAM | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.019 | +0.018 | +0.0061 | 1.000 | 8 of 24 | -0.0007 | not confirmed |
| FMAM | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | +0.026 | +0.014 | +0.0027 | 1.000 | 8 of 24 | +0.0016 | not confirmed; worse over the test years |
| FMAM | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.021 | +0.016 | +0.0050 | 1.000 | 10 of 24 | +0.0003 | not confirmed; worse over the test years |
| FMAM | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.028 | +0.021 | +0.0020 | 1.000 | 8 of 24 | -0.0018 | not confirmed |
Rainfall amounts (CRPS)
| Target | Method | CRPSS 1993–2016 (LOYO) | CRPSS 2017–2025 | ΔCRPS vs reference (training) | Holm p | Years better | ΔCRPS (test years) | Decision |
|---|---|---|---|---|---|---|---|---|
| Feb | MVS-corrected ensemble (published amount correction) | +0.074 | +0.002 | — | — | — | — | reference |
| Feb | Modified-Leung affine ensemble (amounts) | +0.091 | +0.010 | -0.1373 | 0.080 | 17 of 24 | -0.0748 | not confirmed |
| Mar | MVS-corrected ensemble (published amount correction) | -0.044 | +0.005 | — | — | — | — | reference |
| Mar | Modified-Leung affine ensemble (amounts) | -0.040 | +0.015 | -0.0682 | 0.884 | 13 of 24 | -0.2397 | not confirmed |
| Apr | MVS-corrected ensemble (published amount correction) | -0.016 | -0.006 | — | — | — | — | reference |
| Apr | Modified-Leung affine ensemble (amounts) | -0.016 | -0.004 | -0.0262 | 0.972 | 14 of 24 | -0.0385 | not confirmed |
| May | MVS-corrected ensemble (published amount correction) | +0.007 | -0.007 | — | — | — | — | reference |
| May | Modified-Leung affine ensemble (amounts) | +0.007 | -0.009 | -0.0042 | 0.972 | 13 of 24 | +0.0484 | not confirmed; worse over the test years |
| FMAM | MVS-corrected ensemble (published amount correction) | +0.025 | +0.016 | — | — | — | — | reference |
| FMAM | Modified-Leung affine ensemble (amounts) | +0.022 | +0.021 | +0.1403 | 0.972 | 9 of 24 | -0.2868 | not confirmed |
Negative Δ favours the method. Skill scores against the observed climatology over All Ethiopia.
Independent season 2026
The forecast year scored against CHIRPS once observed: every pipeline fitted on the reference years, as the published forecast, on the same cells. Skill against the observed climatology; the best probability pipeline of each target in bold. One season decides nothing on its own; it is one of the validation seasons the method selection collects.
| Target | Published (MVS + blend) | Leung–Dirichlet (as uploaded) | Leung + member counts | MVS + Dirichlet | Leung + published blend | CRPSS MVS ensemble | CRPSS Leung affine |
|---|---|---|---|---|---|---|---|
| Feb | -0.037 | -0.070 | -0.260 | -0.071 | -0.053 | -0.209 | -0.200 |
| Mar | +0.049 | +0.011 | +0.238 | +0.017 | +0.059 | +0.101 | +0.103 |
| Apr | +0.010 | -0.006 | -0.002 | -0.005 | +0.010 | +0.014 | +0.004 |
| May | +0.028 | +0.016 | +0.039 | +0.014 | +0.031 | +0.032 | +0.028 |
| FMAM | -0.026 | -0.023 | -0.066 | -0.022 | -0.032 | -0.066 | -0.073 |
Reliability of the tercile probabilities
How often the below- and above-normal categories occurred when each pipeline gave a probability in a bin of 10%. Points on the diagonal are reliable; a line flatter than the diagonal is overconfident, steeper is underconfident; marker size shows how many forecasts fall in the bin (sharpness). Hindcasts over All Ethiopia.
Test years 2017–2025
Fitted Dirichlet (on all training years)
| Target | Pipeline | A (rows: below / near / above) | b | Converged |
|---|---|---|---|---|
| Feb | Leung–Dirichlet (as uploaded) | 0.06 0.05 -0.10 / 0.05 0.04 0.12 / 0.04 0.05 0.82 | -0.38 -0.15 0.53 | yes |
| Feb | MVS + Dirichlet | 0.06 0.05 -0.03 / 0.06 0.05 0.07 / 0.06 0.07 0.75 | -0.39 -0.24 0.64 | yes |
| Mar | Leung–Dirichlet (as uploaded) | 0.06 0.05 0.05 / 0.04 0.05 0.05 / 0.05 0.04 0.04 | 0.04 -0.02 -0.02 | yes |
| Mar | MVS + Dirichlet | 0.05 0.04 0.06 / 0.04 0.05 0.04 / 0.05 0.05 0.09 | -0.01 -0.03 0.03 | yes |
| Apr | Leung–Dirichlet (as uploaded) | 0.05 0.11 0.10 / 0.05 -0.01 -0.02 / 0.04 -0.00 0.37 | 0.04 -0.25 0.21 | yes |
| Apr | MVS + Dirichlet | 0.05 0.09 0.07 / 0.05 0.03 -0.00 / 0.04 -0.00 0.34 | -0.01 -0.18 0.18 | yes |
| May | Leung–Dirichlet (as uploaded) | 0.08 -0.01 -0.07 / 0.05 0.06 0.08 / 0.04 0.11 0.82 | -0.45 -0.23 0.69 | yes |
| May | MVS + Dirichlet | 0.05 0.03 -0.05 / 0.05 0.09 0.06 / 0.04 0.07 0.84 | -0.44 -0.23 0.67 | yes |
| FMAM | Leung–Dirichlet (as uploaded) | 0.10 0.01 -0.02 / 0.08 -0.01 0.04 / 0.01 0.09 0.78 | -0.31 -0.27 0.58 | yes |
| FMAM | MVS + Dirichlet | 0.08 0.04 -0.02 / 0.07 0.02 0.05 / 0.02 0.06 0.70 | -0.27 -0.22 0.49 | yes |
A equal to the identity and b equal to 0 leave the probabilities unchanged. Share of 2017+ forecast members using the pooled climatology (members beyond the 25 of the hindcast): 51%.
Forecast-year maps, FMAM 2026 (not published)
Every pipeline fitted on the cycle's reference years and applied to this year's forecast, drawn on the 0.25° grid cells as computed (no interpolation or smoothing), first on the season's rainfall domain (the seasonal mask), then on All Ethiopia. For each target: the rainfall tercile outlook (favoured tercile where it reaches 40%, white where none does) of the published forecast and of each pipeline, then the below-, near- and above-normal probabilities of all of them. The published stage recomputed here matches the published file. Dark grey cells have no probability (too dry for terciles, mostly January in the published forecast); those cells are excluded from the scores of every pipeline.
FMAM rainfall contribution domain
Feb 2026
Mar 2026
Apr 2026
May 2026
FMAM 2026
All Ethiopia
Feb 2026
Mar 2026
Apr 2026
May 2026
FMAM 2026
FMAM main-season rainfall domain (Belg / Gu / Ganna, >=40% of annual)
Feb 2026
Mar 2026
Apr 2026
May 2026
FMAM 2026
Downloads: benchmark.json ld_2026_Apr.nc ld_2026_Feb.nc ld_2026_FMAM.nc ld_2026_Mar.nc ld_2026_May.nc
May initialization · JJAS and its months
Methods compared
| Method | What it is | Changes |
|---|---|---|
| Leung–Dirichlet (as uploaded) | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | both stages (the uploaded pipeline) |
| Leung + member counts | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | both stages, without the Dirichlet |
| MVS + Dirichlet | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | probability stage |
| Leung + published blend | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | amount correction |
| Leung affine (amounts) | Modified-Leung affine ensemble (amounts) | amount correction |
References: Published probabilities: MVS -> smoothed counts -> climatology blend; MVS-corrected ensemble (published amount correction). Every method is fitted inside the folds: outer leave-one-year-out over 1993–2016, and fitted on all of 1993–2016 for the test years 2017–2025. The Dirichlet is fitted, as in the uploaded code, on the training years' probabilities from the affine correction fitted on those years. Confirmed better: better over the training years with Holm-adjusted p < 0.05 across all methods and targets of the comparison, and not worse over the test years. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026).
Confirmed better than the published approach: none.
Tercile probabilities (RPS)
| Target | Method | RPSS 1993–2016 (LOYO) | RPSS 2017–2025 | ΔRPS vs reference (training) | Holm p | Years better | ΔRPS (test years) | Decision |
|---|---|---|---|---|---|---|---|---|
| Jun | Published probabilities: MVS -> smoothed counts -> climatology blend | -0.000 | +0.010 | — | — | — | — | reference |
| Jun | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | -0.009 | +0.000 | +0.0040 | 1.000 | 10 of 24 | +0.0046 | not confirmed; worse over the test years |
| Jun | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.121 | -0.056 | +0.0548 | 1.000 | 6 of 24 | +0.0310 | not confirmed; worse over the test years |
| Jun | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | -0.014 | +0.001 | +0.0062 | 1.000 | 9 of 24 | +0.0043 | not confirmed; worse over the test years |
| Jun | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | -0.001 | +0.011 | +0.0004 | 1.000 | 11 of 24 | -0.0005 | not confirmed |
| Jul | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.015 | +0.042 | — | — | — | — | reference |
| Jul | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | -0.010 | +0.004 | +0.0114 | 1.000 | 6 of 24 | +0.0177 | not confirmed; worse over the test years |
| Jul | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.033 | +0.019 | +0.0216 | 1.000 | 11 of 24 | +0.0107 | not confirmed; worse over the test years |
| Jul | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | -0.017 | +0.003 | +0.0141 | 1.000 | 5 of 24 | +0.0183 | not confirmed; worse over the test years |
| Jul | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.015 | +0.043 | -0.0003 | 1.000 | 15 of 24 | -0.0004 | not confirmed |
| Aug | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.012 | +0.041 | — | — | — | — | reference |
| Aug | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | -0.001 | +0.012 | +0.0060 | 1.000 | 11 of 24 | +0.0135 | not confirmed; worse over the test years |
| Aug | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.039 | +0.065 | +0.0232 | 1.000 | 8 of 24 | -0.0107 | not confirmed |
| Aug | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | -0.003 | +0.008 | +0.0068 | 1.000 | 10 of 24 | +0.0151 | not confirmed; worse over the test years |
| Aug | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.011 | +0.042 | +0.0003 | 1.000 | 11 of 24 | -0.0001 | not confirmed |
| Sep | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.003 | +0.008 | — | — | — | — | reference |
| Sep | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.002 | +0.004 | +0.0003 | 1.000 | 10 of 24 | +0.0021 | not confirmed; worse over the test years |
| Sep | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.068 | -0.042 | +0.0321 | 1.000 | 5 of 24 | +0.0233 | not confirmed; worse over the test years |
| Sep | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.004 | +0.004 | -0.0005 | 1.000 | 10 of 24 | +0.0019 | not confirmed; worse over the test years |
| Sep | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.003 | +0.009 | -0.0000 | 1.000 | 12 of 24 | -0.0004 | not confirmed |
| JJAS | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.027 | +0.039 | — | — | — | — | reference |
| JJAS | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.003 | +0.013 | +0.0106 | 1.000 | 7 of 24 | +0.0121 | not confirmed; worse over the test years |
| JJAS | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.024 | -0.002 | +0.0231 | 1.000 | 7 of 24 | +0.0191 | not confirmed; worse over the test years |
| JJAS | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.007 | +0.012 | +0.0092 | 1.000 | 7 of 24 | +0.0125 | not confirmed; worse over the test years |
| JJAS | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.027 | +0.040 | -0.0000 | 1.000 | 14 of 24 | -0.0001 | not confirmed |
Rainfall amounts (CRPS)
| Target | Method | CRPSS 1993–2016 (LOYO) | CRPSS 2017–2025 | ΔCRPS vs reference (training) | Holm p | Years better | ΔCRPS (test years) | Decision |
|---|---|---|---|---|---|---|---|---|
| Jun | MVS-corrected ensemble (published amount correction) | -0.096 | -0.030 | — | — | — | — | reference |
| Jun | Modified-Leung affine ensemble (amounts) | -0.094 | -0.030 | -0.0211 | 1.000 | 13 of 24 | -0.0021 | not confirmed |
| Jul | MVS-corrected ensemble (published amount correction) | -0.005 | +0.002 | — | — | — | — | reference |
| Jul | Modified-Leung affine ensemble (amounts) | -0.003 | -0.001 | -0.0357 | 1.000 | 13 of 24 | +0.0631 | not confirmed; worse over the test years |
| Aug | MVS-corrected ensemble (published amount correction) | -0.027 | +0.055 | — | — | — | — | reference |
| Aug | Modified-Leung affine ensemble (amounts) | -0.028 | +0.057 | +0.0283 | 1.000 | 7 of 24 | -0.0410 | not confirmed |
| Sep | MVS-corrected ensemble (published amount correction) | -0.060 | -0.034 | — | — | — | — | reference |
| Sep | Modified-Leung affine ensemble (amounts) | -0.060 | -0.033 | -0.0016 | 1.000 | 11 of 24 | -0.0098 | not confirmed |
| JJAS | MVS-corrected ensemble (published amount correction) | -0.028 | -0.018 | — | — | — | — | reference |
| JJAS | Modified-Leung affine ensemble (amounts) | -0.028 | -0.019 | +0.0193 | 1.000 | 10 of 24 | +0.0636 | not confirmed; worse over the test years |
Negative Δ favours the method. Skill scores against the observed climatology over All Ethiopia.
Independent season 2026
The forecast year scored against CHIRPS once observed: every pipeline fitted on the reference years, as the published forecast, on the same cells. Skill against the observed climatology; the best probability pipeline of each target in bold. One season decides nothing on its own; it is one of the validation seasons the method selection collects.
| Target | Published (MVS + blend) | Leung–Dirichlet (as uploaded) | Leung + member counts | MVS + Dirichlet | Leung + published blend | CRPSS MVS ensemble | CRPSS Leung affine |
|---|---|---|---|---|---|---|---|
| Jun | +0.125 | +0.015 | +0.433 | +0.024 | +0.130 | +0.404 | +0.405 |
| Jul | +0.198 | +0.060 | +0.269 | +0.063 | +0.200 | +0.227 | +0.226 |
| Aug | +0.277 | +0.077 | +0.439 | +0.056 | +0.278 | +0.430 | +0.432 |
| Sep | +0.107 | +0.036 | +0.318 | +0.037 | +0.111 | +0.182 | +0.180 |
| JJAS | +0.352 | +0.210 | +0.610 | +0.220 | +0.357 | +0.532 | +0.534 |
Reliability of the tercile probabilities
How often the below- and above-normal categories occurred when each pipeline gave a probability in a bin of 10%. Points on the diagonal are reliable; a line flatter than the diagonal is overconfident, steeper is underconfident; marker size shows how many forecasts fall in the bin (sharpness). Hindcasts over All Ethiopia.
Test years 2017–2025
Fitted Dirichlet (on all training years)
| Target | Pipeline | A (rows: below / near / above) | b | Converged |
|---|---|---|---|---|
| Jun | Leung–Dirichlet (as uploaded) | 0.06 0.05 0.04 / 0.05 0.04 0.05 / 0.05 0.05 0.04 | 0.03 0.00 -0.03 | yes |
| Jun | MVS + Dirichlet | 0.06 0.05 0.04 / 0.05 0.04 0.06 / 0.05 0.05 0.04 | 0.02 -0.01 -0.01 | yes |
| Jul | Leung–Dirichlet (as uploaded) | 0.07 0.05 0.04 / 0.05 0.05 0.06 / 0.04 0.05 0.07 | -0.01 -0.00 0.01 | yes |
| Jul | MVS + Dirichlet | 0.05 0.04 0.04 / 0.05 0.04 0.06 / 0.04 0.05 0.06 | -0.02 0.01 0.01 | yes |
| Aug | Leung–Dirichlet (as uploaded) | 0.15 0.05 0.03 / 0.05 0.04 0.06 / 0.05 0.05 0.07 | 0.06 -0.04 -0.02 | yes |
| Aug | MVS + Dirichlet | 0.09 0.05 0.03 / 0.04 0.04 0.06 / 0.05 0.05 0.07 | 0.00 -0.01 0.01 | yes |
| Sep | Leung–Dirichlet (as uploaded) | 0.09 0.04 0.02 / 0.05 0.04 0.06 / 0.03 0.04 0.09 | -0.01 -0.00 0.02 | yes |
| Sep | MVS + Dirichlet | 0.07 0.05 0.02 / 0.05 0.05 0.07 / 0.04 0.05 0.10 | -0.04 0.02 0.02 | yes |
| JJAS | Leung–Dirichlet (as uploaded) | 0.07 0.04 0.02 / 0.06 0.05 0.07 / 0.04 0.05 0.09 | -0.05 0.03 0.03 | yes |
| JJAS | MVS + Dirichlet | 0.08 0.05 0.02 / 0.05 0.05 0.07 / 0.04 0.05 0.09 | -0.04 0.03 0.01 | yes |
A equal to the identity and b equal to 0 leave the probabilities unchanged. Share of 2017+ forecast members using the pooled climatology (members beyond the 25 of the hindcast): 51%.
Forecast-year maps, JJAS 2026 (not published)
Every pipeline fitted on the cycle's reference years and applied to this year's forecast, drawn on the 0.25° grid cells as computed (no interpolation or smoothing), first on the season's rainfall domain (the seasonal mask), then on All Ethiopia. For each target: the rainfall tercile outlook (favoured tercile where it reaches 40%, white where none does) of the published forecast and of each pipeline, then the below-, near- and above-normal probabilities of all of them. The published stage recomputed here matches the published file. Dark grey cells have no probability (too dry for terciles, mostly January in the published forecast); those cells are excluded from the scores of every pipeline.
JJAS R1+R2 rainfall domain
Jun 2026
Jul 2026
Aug 2026
Sep 2026
JJAS 2026
All Ethiopia
Jun 2026
Jul 2026
Aug 2026
Sep 2026
JJAS 2026
Downloads: benchmark.json ld_2026_Aug.nc ld_2026_JJAS.nc ld_2026_Jul.nc ld_2026_Jun.nc ld_2026_Sep.nc
September initialization · ONDJ and its months
Methods compared
| Method | What it is | Changes |
|---|---|---|
| Leung–Dirichlet (as uploaded) | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | both stages (the uploaded pipeline) |
| Leung + member counts | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | both stages, without the Dirichlet |
| MVS + Dirichlet | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | probability stage |
| Leung + published blend | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | amount correction |
| Leung affine (amounts) | Modified-Leung affine ensemble (amounts) | amount correction |
References: Published probabilities: MVS -> smoothed counts -> climatology blend; MVS-corrected ensemble (published amount correction). Every method is fitted inside the folds: outer leave-one-year-out over 1993–2016, and fitted on all of 1993–2016 for the test years 2017–2024. The Dirichlet is fitted, as in the uploaded code, on the training years' probabilities from the affine correction fitted on those years. Confirmed better: better over the training years with Holm-adjusted p < 0.05 across all methods and targets of the comparison, and not worse over the test years. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026).
Confirmed better than the published approach: Leung affine (amounts) for Jan (CRPS).
Tercile probabilities (RPS)
| Target | Method | RPSS 1993–2016 (LOYO) | RPSS 2017–2024 | ΔRPS vs reference (training) | Holm p | Years better | ΔRPS (test years) | Decision |
|---|---|---|---|---|---|---|---|---|
| Oct | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.013 | +0.027 | — | — | — | — | reference |
| Oct | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.004 | +0.001 | +0.0040 | 1.000 | 8 of 24 | +0.0113 | not confirmed; worse over the test years |
| Oct | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.021 | +0.004 | +0.0153 | 1.000 | 7 of 24 | +0.0099 | not confirmed; worse over the test years |
| Oct | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.003 | +0.006 | +0.0045 | 1.000 | 8 of 24 | +0.0089 | not confirmed; worse over the test years |
| Oct | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.009 | +0.024 | +0.0017 | 1.000 | 8 of 24 | +0.0015 | not confirmed; worse over the test years |
| Nov | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.040 | +0.053 | — | — | — | — | reference |
| Nov | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.035 | +0.014 | +0.0021 | 1.000 | 12 of 24 | +0.0172 | not confirmed; worse over the test years |
| Nov | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | +0.046 | +0.041 | -0.0025 | 1.000 | 10 of 24 | +0.0056 | not confirmed; worse over the test years |
| Nov | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.035 | +0.015 | +0.0022 | 1.000 | 11 of 24 | +0.0170 | not confirmed; worse over the test years |
| Nov | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.041 | +0.053 | -0.0003 | 1.000 | 12 of 24 | +0.0002 | not confirmed; worse over the test years |
| Dec | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.003 | +0.016 | — | — | — | — | reference |
| Dec | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.008 | +0.008 | -0.0024 | 1.000 | 16 of 24 | +0.0037 | not confirmed; worse over the test years |
| Dec | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.064 | -0.040 | +0.0306 | 1.000 | 5 of 24 | +0.0256 | not confirmed; worse over the test years |
| Dec | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.008 | +0.001 | -0.0024 | 1.000 | 16 of 24 | +0.0069 | not confirmed; worse over the test years |
| Dec | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.002 | +0.017 | +0.0004 | 1.000 | 11 of 24 | -0.0007 | not confirmed |
| Jan | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.001 | -0.004 | — | — | — | — | reference |
| Jan | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | -0.004 | +0.021 | +0.0024 | 1.000 | 10 of 24 | -0.0114 | not confirmed |
| Jan | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | -0.132 | -0.213 | +0.0602 | 1.000 | 6 of 24 | +0.0946 | not confirmed; worse over the test years |
| Jan | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.000 | +0.006 | +0.0005 | 1.000 | 9 of 24 | -0.0049 | not confirmed |
| Jan | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.002 | -0.005 | -0.0004 | 1.000 | 12 of 24 | +0.0001 | not confirmed; worse over the test years |
| ONDJ | Published probabilities: MVS -> smoothed counts -> climatology blend | +0.039 | +0.075 | — | — | — | — | reference |
| ONDJ | Leung–Dirichlet calibration as uploaded: modified-Leung affine → member counts → identity-anchored Dirichlet | +0.026 | +0.041 | +0.0062 | 1.000 | 11 of 24 | +0.0155 | not confirmed; worse over the test years |
| ONDJ | Modified-Leung affine → member counts (the uploaded script with --no_dirichlet) | +0.025 | +0.058 | +0.0067 | 1.000 | 8 of 24 | +0.0079 | not confirmed; worse over the test years |
| ONDJ | Published MVS → member counts → identity-anchored Dirichlet (the Dirichlet alone) | +0.031 | +0.049 | +0.0040 | 1.000 | 10 of 24 | +0.0118 | not confirmed; worse over the test years |
| ONDJ | Modified-Leung affine → the published smoothed counts and climatology blend (the Leung correction alone) | +0.034 | +0.071 | +0.0024 | 1.000 | 11 of 24 | +0.0019 | not confirmed; worse over the test years |
Rainfall amounts (CRPS)
| Target | Method | CRPSS 1993–2016 (LOYO) | CRPSS 2017–2024 | ΔCRPS vs reference (training) | Holm p | Years better | ΔCRPS (test years) | Decision |
|---|---|---|---|---|---|---|---|---|
| Oct | MVS-corrected ensemble (published amount correction) | -0.002 | +0.059 | — | — | — | — | reference |
| Oct | Modified-Leung affine ensemble (amounts) | -0.006 | +0.055 | +0.1154 | 1.000 | 7 of 24 | +0.1353 | not confirmed; worse over the test years |
| Nov | MVS-corrected ensemble (published amount correction) | +0.084 | +0.158 | — | — | — | — | reference |
| Nov | Modified-Leung affine ensemble (amounts) | +0.088 | +0.162 | -0.0801 | 1.000 | 9 of 24 | -0.0957 | not confirmed |
| Dec | MVS-corrected ensemble (published amount correction) | +0.029 | -0.033 | — | — | — | — | reference |
| Dec | Modified-Leung affine ensemble (amounts) | +0.032 | -0.040 | -0.0203 | 1.000 | 9 of 24 | +0.0402 | not confirmed; worse over the test years |
| Jan | MVS-corrected ensemble (published amount correction) | -0.040 | -0.066 | — | — | — | — | reference |
| Jan | Modified-Leung affine ensemble (amounts) | -0.022 | -0.050 | -0.1476 | 0.000 | 21 of 24 | -0.0939 | ✓ confirmed better |
| ONDJ | MVS-corrected ensemble (published amount correction) | +0.070 | +0.178 | — | — | — | — | reference |
| ONDJ | Modified-Leung affine ensemble (amounts) | +0.070 | +0.175 | -0.0100 | 1.000 | 5 of 24 | +0.1754 | not confirmed; worse over the test years |
Negative Δ favours the method. Skill scores against the observed climatology over All Ethiopia.
Reliability of the tercile probabilities
How often the below- and above-normal categories occurred when each pipeline gave a probability in a bin of 10%. Points on the diagonal are reliable; a line flatter than the diagonal is overconfident, steeper is underconfident; marker size shows how many forecasts fall in the bin (sharpness). Hindcasts over All Ethiopia.
Test years 2017–2024
Fitted Dirichlet (on all training years)
| Target | Pipeline | A (rows: below / near / above) | b | Converged |
|---|---|---|---|---|
| Oct | Leung–Dirichlet (as uploaded) | 0.19 0.05 0.03 / 0.05 0.12 0.06 / 0.03 0.03 0.14 | 0.06 0.00 -0.06 | yes |
| Oct | MVS + Dirichlet | 0.18 0.06 0.02 / 0.05 0.14 0.05 / 0.03 0.03 0.22 | 0.01 -0.01 0.00 | yes |
| Nov | Leung–Dirichlet (as uploaded) | 0.09 0.04 -0.09 / 0.05 0.05 0.06 / 0.04 0.06 1.18 | -0.47 -0.33 0.80 | yes |
| Nov | MVS + Dirichlet | 0.06 0.04 -0.11 / 0.05 0.04 0.08 / 0.04 0.05 1.19 | -0.51 -0.31 0.82 | yes |
| Dec | Leung–Dirichlet (as uploaded) | 0.09 0.07 0.07 / 0.07 0.07 -0.03 / 0.07 0.07 0.65 | -0.14 -0.26 0.40 | yes |
| Dec | MVS + Dirichlet | 0.08 0.08 0.03 / 0.08 0.08 -0.00 / 0.08 0.08 0.57 | -0.18 -0.21 0.39 | yes |
| Jan | Leung–Dirichlet (as uploaded) | 0.06 0.05 0.03 / 0.05 0.05 0.07 / 0.05 0.05 0.16 | -0.03 -0.01 0.05 | yes |
| Jan | MVS + Dirichlet | 0.05 0.05 0.04 / 0.05 0.04 0.05 / 0.05 0.05 0.12 | -0.04 -0.00 0.04 | yes |
| ONDJ | Leung–Dirichlet (as uploaded) | 0.30 0.08 0.02 / 0.09 0.03 0.07 / -0.02 0.02 0.35 | 0.14 -0.12 -0.02 | yes |
| ONDJ | MVS + Dirichlet | 0.26 0.07 -0.02 / 0.10 0.08 0.06 / 0.01 0.04 0.51 | -0.03 -0.11 0.15 | yes |
A equal to the identity and b equal to 0 leave the probabilities unchanged. Share of 2017+ forecast members using the pooled climatology (members beyond the 25 of the hindcast): 51%.
Forecast-year maps, ONDJ 2026/27 (not published)
Every pipeline fitted on the cycle's reference years and applied to this year's forecast, drawn on the 0.25° grid cells as computed (no interpolation or smoothing), first on the season's rainfall domain (the seasonal mask), then on All Ethiopia. For each target: the rainfall tercile outlook (favoured tercile where it reaches 40%, white where none does) of the published forecast and of each pipeline, then the below-, near- and above-normal probabilities of all of them. The published stage recomputed here matches the published file. Dark grey cells have no probability (too dry for terciles, mostly January in the published forecast); those cells are excluded from the scores of every pipeline.
ONDJ R3 (Deyr) rainfall domain
Oct 2026
Nov 2026
Dec 2026
Jan 2027
ONDJ 2026/27
All Ethiopia
Oct 2026
Nov 2026
Dec 2026
Jan 2027
ONDJ 2026/27
Downloads: benchmark.json ld_2026_Dec.nc ld_2026_Jan.nc ld_2026_Nov.nc ld_2026_Oct.nc ld_2026_ONDJ.nc
Code: scripts/leung_dirichlet.py,
scripts/leung_dirichlet_benchmark.py, leung_affine in
scripts/bias_methods.py · write-up: docs/51.