Phase 1 · September initialization · ONDJ and its months
Confirmed better than the incumbent climatology blend for the published tercile probabilities (RPS): none.
✓ = better than the incumbent with Holm-adjusted p < 0.05 across every method and target (SEL: across the targets) and not worse
over 2017–2024. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026). Amount correction fixed: MVS. National aggregate (All Ethiopia). Run at commit 0849f44.
Tercile skill
Tercile RPSS, outer leave-one-year-out 1993–2016 (bold: incumbent)
Method Oct Nov Jan Dec ONDJ Climatology blend (incumbent) +0.013 +0.040 +0.001 +0.003 +0.039 Pseudocount smoothing (no blend) -0.009 +0.044 -0.213 -0.068 +0.035 Temperature scaling +0.016 +0.027 -0.031 +0.004 +0.043 Regularized Dirichlet calibration +0.010 +0.050 +0.007 +0.010 +0.037 Regularized multinomial logistic regression -0.003 +0.062 -0.016 +0.008 +0.034 Ordinal (cumulative-logit) logistic regression -0.003 +0.063 -0.014 +0.007 +0.037 SEL (method chosen each year by inner cross-validation) +0.016 +0.051 +0.001 +0.002 +0.043
Tercile RPSS, fit 1993–2016, scored 2017–2024 (exploratory)
Method Oct Nov Jan Dec ONDJ Climatology blend (incumbent) +0.027 +0.053 -0.005 +0.015 +0.075 Pseudocount smoothing (no blend) +0.012 +0.041 -0.300 -0.049 +0.065 Temperature scaling +0.027 +0.042 -0.098 +0.007 +0.078 Regularized Dirichlet calibration +0.012 +0.015 +0.057 +0.010 +0.069 Regularized multinomial logistic regression -0.005 +0.060 +0.031 +0.027 +0.064 Ordinal (cumulative-logit) logistic regression -0.004 +0.048 +0.032 +0.023 +0.063 SEL (method chosen each year by inner cross-validation) +0.027 +0.060 -0.005 +0.015 +0.078
Log loss and Brier skill
Log-loss skill against climatology, 1993–2016
Method Oct Nov Jan Dec ONDJ Climatology blend (incumbent) +0.015 +0.021 +0.001 -0.001 +0.026 Pseudocount smoothing (no blend) -0.008 -0.010 -0.529 -0.197 +0.018 Temperature scaling +0.015 +0.017 -0.039 -0.003 +0.029 Regularized Dirichlet calibration +0.013 +0.034 +0.004 +0.008 +0.025 Regularized multinomial logistic regression -0.002 +0.040 -0.011 +0.007 +0.020 Ordinal (cumulative-logit) logistic regression -0.002 +0.040 -0.008 +0.003 +0.024 SEL (method chosen each year by inner cross-validation) +0.015 +0.030 +0.001 -0.005 +0.029
Brier skill, below normal, 1993–2016
Method Oct Nov Jan Dec ONDJ Climatology blend (incumbent) +0.013 +0.003 -0.002 -0.013 +0.028 Pseudocount smoothing (no blend) -0.011 -0.013 -0.331 -0.138 +0.022 Temperature scaling +0.015 +0.004 -0.069 -0.014 +0.031 Regularized Dirichlet calibration +0.013 +0.024 +0.002 -0.003 +0.026 Regularized multinomial logistic regression -0.004 +0.033 -0.011 -0.008 +0.022 Ordinal (cumulative-logit) logistic regression -0.003 +0.034 -0.012 -0.010 +0.024 SEL (method chosen each year by inner cross-validation) +0.015 +0.020 -0.002 -0.010 +0.031
Brier skill, above normal, 1993–2016
Method Oct Nov Jan Dec ONDJ Climatology blend (incumbent) +0.013 +0.078 +0.003 +0.019 +0.051 Pseudocount smoothing (no blend) -0.006 +0.101 -0.095 +0.003 +0.049 Temperature scaling +0.017 +0.050 +0.007 +0.023 +0.054 Regularized Dirichlet calibration +0.007 +0.076 +0.011 +0.024 +0.048 Regularized multinomial logistic regression -0.003 +0.092 -0.020 +0.023 +0.045 Ordinal (cumulative-logit) logistic regression -0.002 +0.091 -0.016 +0.023 +0.050 SEL (method chosen each year by inner cross-validation) +0.017 +0.082 +0.003 +0.014 +0.054
Tests
Each method against the incumbent (negative Δ favours the method)
Method Target Δ RPS 1993–2016 Years better Holm p Δ RPS 2017–2024 Years better Confirmed Pseudocount smoothing (no blend) Oct +0.0097 12/24 1.0000 +0.0066 5/8 no Pseudocount smoothing (no blend) Nov -0.0017 11/24 1.0000 +0.0057 2/8 no Pseudocount smoothing (no blend) Jan +0.0968 6/24 1.0000 +0.1327 1/8 no Pseudocount smoothing (no blend) Dec +0.0319 5/24 1.0000 +0.0296 2/8 no Pseudocount smoothing (no blend) ONDJ +0.0018 10/24 1.0000 +0.0045 3/8 no Temperature scaling Oct -0.0013 11/24 1.0000 -0.0000 2/8 no Temperature scaling Nov +0.0059 12/24 1.0000 +0.0049 3/8 no Temperature scaling Jan +0.0145 11/24 1.0000 +0.0419 4/8 no Temperature scaling Dec -0.0007 15/24 1.0000 +0.0038 3/8 no Temperature scaling ONDJ -0.0014 14/24 1.0000 -0.0016 5/8 no Regularized Dirichlet calibration Oct +0.0013 11/24 1.0000 +0.0063 2/8 no Regularized Dirichlet calibration Nov -0.0044 14/24 1.0000 +0.0171 2/8 no Regularized Dirichlet calibration Jan -0.0027 13/24 1.0000 -0.0275 5/8 no Regularized Dirichlet calibration Dec -0.0034 15/24 1.0000 +0.0027 3/8 no Regularized Dirichlet calibration ONDJ +0.0011 14/24 1.0000 +0.0025 3/8 no Regularized multinomial logistic regression Oct +0.0072 7/24 1.0000 +0.0139 0/8 no Regularized multinomial logistic regression Nov -0.0101 16/24 1.0000 -0.0031 4/8 no Regularized multinomial logistic regression Jan +0.0075 6/24 1.0000 -0.0158 4/8 no Regularized multinomial logistic regression Dec -0.0023 14/24 1.0000 -0.0055 6/8 no Regularized multinomial logistic regression ONDJ +0.0026 11/24 1.0000 +0.0051 3/8 no Ordinal (cumulative-logit) logistic regression Oct +0.0069 7/24 1.0000 +0.0132 0/8 no Ordinal (cumulative-logit) logistic regression Nov -0.0102 16/24 1.0000 +0.0024 4/8 no Ordinal (cumulative-logit) logistic regression Jan +0.0067 7/24 1.0000 -0.0163 4/8 no Ordinal (cumulative-logit) logistic regression Dec -0.0017 16/24 1.0000 -0.0036 4/8 no Ordinal (cumulative-logit) logistic regression ONDJ +0.0011 11/24 1.0000 +0.0052 4/8 no SEL Oct -0.0013 11/24 1.0000 (targets) -0.0000 2/8 no SEL Nov -0.0049 14/24 1.0000 (targets) -0.0031 4/8 no SEL Jan +0.0000 0/24 1.0000 (targets) +0.0000 0/8 no SEL Dec +0.0003 9/24 1.0000 (targets) +0.0000 0/8 no SEL ONDJ -0.0014 14/24 1.0000 (targets) -0.0016 5/8 no
Fitted parameters and the selection
Fitted parameters (fit on 1993–2016) and the selection-aware choices
Target Blend weight λ Temperature T Ordinal cutpoints Ordinal slopes (mean, log spread) SEL choices (1993–2016 folds) Fallbacks Oct 0.45 2.32 -0.60, +0.70 +0.99, -0.18 Temperature scaling 24 0 Nov 0.31 2.73 -0.63, +0.74 +1.93, -0.18 Regularized multinomial logistic regression 12, Ordinal (cumulative-logit) logistic regression 9, Pseudocount smoothing (no blend) 2, Climatology blend (incumbent) 1 0 Jan 0.90 4.00 -0.61, +0.67 +0.56, -0.04 Climatology blend (incumbent) 24 0 Dec 0.70 4.00 -0.63, +0.69 +0.97, +0.03 Temperature scaling 12, Climatology blend (incumbent) 11, Regularized multinomial logistic regression 1 0 ONDJ 0.21 1.52 -0.68, +0.69 +1.39, -0.04 Temperature scaling 24 0
Figures
RPSS and log-loss skill by method and target.
Reliability of each method for the season (below and above normal).
