Phase 1 · January initialization · FMAM and its months
Confirmed better than the incumbent climatology blend for the published tercile probabilities (RPS): none.
✓ = better than the incumbent with Holm-adjusted p < 0.05 across every method and target (SEL: across the targets) and not worse
over 2017–2025. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026). Amount correction fixed: MVS. National aggregate (All Ethiopia). Run at commit 0849f44.
Tercile skill
Tercile RPSS, outer leave-one-year-out 1993–2016 (bold: incumbent)
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.027 +0.002 +0.001 +0.028 +0.032 Pseudocount smoothing (no blend) -0.093 -0.111 -0.027 +0.026 +0.034 Temperature scaling +0.009 -0.012 +0.003 +0.022 +0.032 Regularized Dirichlet calibration +0.093 -0.002 +0.001 +0.029 +0.030 Regularized multinomial logistic regression +0.076 -0.017 -0.009 +0.014 +0.022 Ordinal (cumulative-logit) logistic regression +0.078 -0.016 -0.009 +0.015 +0.022 SEL (method chosen each year by inner cross-validation) +0.081 +0.002 +0.003 +0.019 +0.029
Tercile RPSS, fit 1993–2016, scored 2017–2025 (exploratory)
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.022 +0.008 +0.002 +0.004 +0.017 Pseudocount smoothing (no blend) -0.116 -0.067 -0.029 -0.025 +0.009 Temperature scaling +0.027 -0.002 +0.001 +0.008 +0.017 Regularized Dirichlet calibration -0.002 +0.013 +0.007 -0.009 +0.020 Regularized multinomial logistic regression +0.012 -0.002 -0.001 +0.000 +0.009 Ordinal (cumulative-logit) logistic regression +0.015 -0.002 -0.000 +0.001 +0.006 SEL (method chosen each year by inner cross-validation) -0.002 +0.008 +0.001 +0.004 +0.009
Log loss and Brier skill
Log-loss skill against climatology, 1993–2016
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.014 +0.003 +0.004 +0.023 +0.019 Pseudocount smoothing (no blend) -0.337 -0.239 -0.037 +0.004 +0.012 Temperature scaling -0.005 -0.013 +0.004 +0.017 +0.020 Regularized Dirichlet calibration +0.059 -0.000 +0.004 +0.024 +0.020 Regularized multinomial logistic regression +0.042 -0.012 -0.004 +0.008 +0.013 Ordinal (cumulative-logit) logistic regression +0.045 -0.010 -0.006 +0.009 +0.013 SEL (method chosen each year by inner cross-validation) +0.049 +0.003 +0.004 +0.011 +0.010
Brier skill, below normal, 1993–2016
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.009 +0.003 -0.005 +0.027 +0.022 Pseudocount smoothing (no blend) -0.184 -0.132 -0.051 +0.021 +0.020 Temperature scaling -0.020 -0.020 -0.003 +0.020 +0.022 Regularized Dirichlet calibration +0.086 -0.002 -0.006 +0.025 +0.020 Regularized multinomial logistic regression +0.078 -0.018 -0.011 +0.013 +0.013 Ordinal (cumulative-logit) logistic regression +0.081 -0.016 -0.012 +0.015 +0.014 SEL (method chosen each year by inner cross-validation) +0.081 +0.003 -0.003 +0.016 +0.017
Brier skill, above normal, 1993–2016
Method Feb Mar Apr May FMAM Climatology blend (incumbent) +0.045 +0.001 +0.007 +0.029 +0.043 Pseudocount smoothing (no blend) -0.002 -0.091 -0.003 +0.031 +0.047 Temperature scaling +0.038 -0.005 +0.009 +0.024 +0.041 Regularized Dirichlet calibration +0.099 -0.002 +0.008 +0.032 +0.041 Regularized multinomial logistic regression +0.074 -0.016 -0.007 +0.016 +0.031 Ordinal (cumulative-logit) logistic regression +0.074 -0.016 -0.007 +0.016 +0.030 SEL (method chosen each year by inner cross-validation) +0.080 +0.001 +0.009 +0.022 +0.042
Tests
Each method against the incumbent (negative Δ favours the method)
Method Target Δ RPS 1993–2016 Years better Holm p Δ RPS 2017–2025 Years better Confirmed Pseudocount smoothing (no blend) Feb +0.0544 6/24 1.0000 +0.0591 1/9 no Pseudocount smoothing (no blend) Mar +0.0513 8/24 1.0000 +0.0338 5/9 no Pseudocount smoothing (no blend) Apr +0.0127 10/24 1.0000 +0.0139 4/9 no Pseudocount smoothing (no blend) May +0.0009 13/24 1.0000 +0.0125 5/9 no Pseudocount smoothing (no blend) FMAM -0.0005 9/24 1.0000 +0.0037 3/9 no Temperature scaling Feb +0.0081 10/24 1.0000 -0.0023 5/9 no Temperature scaling Mar +0.0065 12/24 1.0000 +0.0043 4/9 no Temperature scaling Apr -0.0008 13/24 1.0000 +0.0003 6/9 no Temperature scaling May +0.0027 7/24 1.0000 -0.0017 5/9 no Temperature scaling FMAM +0.0004 13/24 1.0000 -0.0000 5/9 no Regularized Dirichlet calibration Feb -0.0298 17/24 0.2216 +0.0102 6/9 no Regularized Dirichlet calibration Mar +0.0017 9/24 1.0000 -0.0023 6/9 no Regularized Dirichlet calibration Apr +0.0003 8/24 1.0000 -0.0024 5/9 no Regularized Dirichlet calibration May -0.0003 13/24 1.0000 +0.0053 5/9 no Regularized Dirichlet calibration FMAM +0.0009 9/24 1.0000 -0.0014 5/9 no Regularized multinomial logistic regression Feb -0.0224 17/24 1.0000 +0.0042 3/9 no Regularized multinomial logistic regression Mar +0.0087 8/24 1.0000 +0.0045 3/9 no Regularized multinomial logistic regression Apr +0.0046 4/24 1.0000 +0.0014 5/9 no Regularized multinomial logistic regression May +0.0062 7/24 1.0000 +0.0016 5/9 no Regularized multinomial logistic regression FMAM +0.0046 10/24 1.0000 +0.0036 3/9 no Ordinal (cumulative-logit) logistic regression Feb -0.0230 17/24 1.0000 +0.0031 3/9 no Ordinal (cumulative-logit) logistic regression Mar +0.0082 8/24 1.0000 +0.0045 3/9 no Ordinal (cumulative-logit) logistic regression Apr +0.0049 5/24 1.0000 +0.0010 5/9 no Ordinal (cumulative-logit) logistic regression May +0.0058 6/24 1.0000 +0.0014 5/9 no Ordinal (cumulative-logit) logistic regression FMAM +0.0048 8/24 1.0000 +0.0048 3/9 no SEL Feb -0.0243 17/24 0.2568 (targets) +0.0102 6/9 no SEL Mar +0.0000 0/24 1.0000 (targets) +0.0000 0/9 no SEL Apr -0.0008 13/24 1.0000 (targets) +0.0003 6/9 no SEL May +0.0039 3/24 1.0000 (targets) +0.0000 0/9 no SEL FMAM +0.0013 8/24 1.0000 (targets) +0.0037 3/9 no
Fitted parameters and the selection
Fitted parameters (fit on 1993–2016) and the selection-aware choices
Target Blend weight λ Temperature T Ordinal cutpoints Ordinal slopes (mean, log spread) SEL choices (1993–2016 folds) Fallbacks Feb 0.73 4.00 -0.67, +0.69 +1.48, +0.00 Regularized Dirichlet calibration 23, Ordinal (cumulative-logit) logistic regression 1 0 Mar 0.87 4.00 -0.58, +0.69 +0.28, -0.19 Climatology blend (incumbent) 24 0 Apr 0.73 4.00 -0.61, +0.65 +0.42, -0.14 Temperature scaling 24 0 May 0.49 3.40 -0.67, +0.62 +0.89, +0.31 Climatology blend (incumbent) 15, Pseudocount smoothing (no blend) 9 0 FMAM 0.30 1.76 -0.63, +0.67 +1.20, -0.29 Pseudocount smoothing (no blend) 23, Temperature scaling 1 0
Figures
RPSS and log-loss skill by method and target.
Reliability of each method for the season (below and above normal).
