Phase 1 · February initialization · FMAM and its months
Confirmed better than the incumbent climatology blend for the published tercile probabilities (RPS): none.
✓ = better than the incumbent with Holm-adjusted p < 0.05 across every method and target (SEL: across the targets) and not worse
over 2017–2025. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026). Amount correction fixed: MVS. National aggregate (All Ethiopia). Run at commit 0849f44.
Tercile skill
Tercile RPSS, outer leave-one-year-out 1993–2016 (bold: incumbent)
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.126 +0.011 +0.009 +0.008 +0.021 Pseudocount smoothing (no blend) +0.054 -0.058 -0.001 -0.002 +0.005 Temperature scaling +0.104 +0.005 +0.012 +0.010 +0.023 Regularized Dirichlet calibration +0.164 +0.013 +0.007 +0.003 +0.015 Regularized multinomial logistic regression +0.178 -0.005 -0.017 -0.010 +0.012 Ordinal (cumulative-logit) logistic regression +0.178 -0.006 -0.016 -0.008 +0.013 SEL (method chosen each year by inner cross-validation) +0.178 +0.004 +0.005 -0.018 +0.020
Tercile RPSS, fit 1993–2016, scored 2017–2025 (exploratory)
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.056 +0.029 +0.012 +0.026 +0.059 Pseudocount smoothing (no blend) -0.071 +0.005 +0.007 +0.023 +0.096 Temperature scaling +0.068 +0.019 +0.008 +0.023 +0.051 Regularized Dirichlet calibration +0.049 +0.046 +0.012 +0.021 +0.050 Regularized multinomial logistic regression +0.081 +0.037 +0.001 +0.022 +0.046 Ordinal (cumulative-logit) logistic regression +0.076 +0.036 +0.004 +0.022 +0.044 SEL (method chosen each year by inner cross-validation) +0.076 +0.029 +0.008 +0.026 +0.051
Log loss and Brier skill
Log-loss skill against climatology, 1993–2016
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.071 +0.006 +0.010 +0.006 +0.015 Pseudocount smoothing (no blend) -0.163 -0.187 -0.011 -0.034 -0.010 Temperature scaling +0.061 -0.002 +0.010 +0.006 +0.016 Regularized Dirichlet calibration +0.109 +0.009 +0.010 +0.004 +0.010 Regularized multinomial logistic regression +0.102 -0.002 -0.010 -0.008 +0.006 Ordinal (cumulative-logit) logistic regression +0.105 -0.004 -0.010 -0.006 +0.008 SEL (method chosen each year by inner cross-validation) +0.104 +0.003 +0.007 -0.017 +0.015
Brier skill, below normal, 1993–2016
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.065 +0.016 +0.001 +0.004 +0.017 Pseudocount smoothing (no blend) -0.065 -0.070 -0.021 -0.014 -0.002 Temperature scaling +0.056 +0.001 +0.006 +0.006 +0.020 Regularized Dirichlet calibration +0.137 +0.024 -0.004 -0.002 +0.016 Regularized multinomial logistic regression +0.139 +0.009 -0.028 -0.015 +0.013 Ordinal (cumulative-logit) logistic regression +0.138 +0.007 -0.021 -0.011 +0.014 SEL (method chosen each year by inner cross-validation) +0.139 +0.009 +0.001 -0.025 +0.017
Brier skill, above normal, 1993–2016
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.188 +0.006 +0.018 +0.012 +0.025 Pseudocount smoothing (no blend) +0.174 -0.046 +0.019 +0.009 +0.012 Temperature scaling +0.153 +0.009 +0.018 +0.013 +0.025 Regularized Dirichlet calibration +0.191 +0.002 +0.018 +0.008 +0.013 Regularized multinomial logistic regression +0.217 -0.018 -0.006 -0.004 +0.010 Ordinal (cumulative-logit) logistic regression +0.219 -0.019 -0.010 -0.006 +0.013 SEL (method chosen each year by inner cross-validation) +0.217 +0.000 +0.010 -0.012 +0.022
Tests
Each method against the incumbent (negative Δ favours the method)
Method Target Δ RPS 1993–2016 Years better Holm p Δ RPS 2017–2025 Years better Confirmed Pseudocount smoothing (no blend) Feb +0.0327 6/24 1.0000 +0.0547 2/9 no Pseudocount smoothing (no blend) Mar +0.0313 11/24 1.0000 +0.0109 4/9 no Pseudocount smoothing (no blend) Apr +0.0047 12/24 1.0000 +0.0023 4/9 no Pseudocount smoothing (no blend) May +0.0044 14/24 1.0000 +0.0014 5/9 no Pseudocount smoothing (no blend) FMAM +0.0072 11/24 1.0000 -0.0170 6/9 no Temperature scaling Feb +0.0101 9/24 1.0000 -0.0050 4/9 no Temperature scaling Mar +0.0026 11/24 1.0000 +0.0046 3/9 no Temperature scaling Apr -0.0011 11/24 1.0000 +0.0017 5/9 no Temperature scaling May -0.0008 8/24 1.0000 +0.0015 4/9 no Temperature scaling FMAM -0.0007 11/24 1.0000 +0.0035 1/9 no Regularized Dirichlet calibration Feb -0.0171 15/24 0.2659 +0.0033 5/9 no Regularized Dirichlet calibration Mar -0.0009 13/24 1.0000 -0.0075 7/9 no Regularized Dirichlet calibration Apr +0.0012 13/24 1.0000 +0.0001 4/9 no Regularized Dirichlet calibration May +0.0020 11/24 1.0000 +0.0021 4/9 no Regularized Dirichlet calibration FMAM +0.0028 9/24 1.0000 +0.0040 1/9 no Regularized multinomial logistic regression Feb -0.0234 15/24 0.3064 -0.0107 4/9 no Regularized multinomial logistic regression Mar +0.0071 9/24 1.0000 -0.0034 5/9 no Regularized multinomial logistic regression Apr +0.0118 8/24 1.0000 +0.0051 3/9 no Regularized multinomial logistic regression May +0.0078 5/24 1.0000 +0.0019 3/9 no Regularized multinomial logistic regression FMAM +0.0042 8/24 1.0000 +0.0059 2/9 no Ordinal (cumulative-logit) logistic regression Feb -0.0236 15/24 0.3064 -0.0087 6/9 no Ordinal (cumulative-logit) logistic regression Mar +0.0078 9/24 1.0000 -0.0029 5/9 no Ordinal (cumulative-logit) logistic regression Apr +0.0113 6/24 1.0000 +0.0037 3/9 no Ordinal (cumulative-logit) logistic regression May +0.0073 6/24 1.0000 +0.0017 3/9 no Ordinal (cumulative-logit) logistic regression FMAM +0.0036 8/24 1.0000 +0.0067 2/9 no SEL Feb -0.0233 15/24 0.0662 (targets) -0.0087 6/9 no SEL Mar +0.0030 9/24 1.0000 (targets) +0.0000 0/9 no SEL Apr +0.0019 10/24 1.0000 (targets) +0.0017 5/9 no SEL May +0.0117 3/24 1.0000 (targets) +0.0000 0/9 no SEL FMAM +0.0007 9/24 1.0000 (targets) +0.0035 1/9 no
Fitted parameters and the selection
Fitted parameters (fit on 1993–2016) and the selection-aware choices
Target Blend weight λ Temperature T Ordinal cutpoints Ordinal slopes (mean, log spread) SEL choices (1993–2016 folds) Fallbacks Feb 0.48 3.48 -1.10, +0.41 +0.91, +0.35 Ordinal (cumulative-logit) logistic regression 23, Regularized multinomial logistic regression 1 0 Mar 0.72 4.00 -0.59, +0.70 +0.91, -0.13 Temperature scaling 11, Climatology blend (incumbent) 9, Regularized Dirichlet calibration 4 0 Apr 0.60 4.00 -0.64, +0.64 +0.67, +0.01 Temperature scaling 23, Climatology blend (incumbent) 1 0 May 0.59 4.00 -0.62, +0.66 +0.87, -0.21 Temperature scaling 19, Climatology blend (incumbent) 4, Pseudocount smoothing (no blend) 1 0 FMAM 0.52 2.59 -0.59, +0.70 +0.79, -0.42 Temperature scaling 22, Climatology blend (incumbent) 2 0
Figures
RPSS and log-loss skill by method and target.
Reliability of each method for the season (below and above normal).
