Phase 1 · May initialization · JJAS and its months
Confirmed better than the incumbent climatology blend for the published tercile probabilities (RPS): none.
✓ = better than the incumbent with Holm-adjusted p < 0.05 across every method and target (SEL: across the targets) and not worse
over 2017–2025. This is the benchmark’s own decision, as run (Holm); the method selection applies Benjamini–Hochberg to the same p-values (since 17 October 2026). Amount correction fixed: MVS. National aggregate (All Ethiopia). Run at commit 0849f44.
Tercile skill
Tercile RPSS, outer leave-one-year-out 1993–2016 (bold: incumbent)
Method Jul Jun Aug Sep JJAS Climatology blend (incumbent) +0.014 -0.000 +0.012 +0.003 +0.027 Pseudocount smoothing (no blend) -0.029 -0.113 -0.031 -0.064 -0.017 Temperature scaling +0.014 -0.001 +0.012 +0.006 +0.027 Regularized Dirichlet calibration +0.008 -0.007 +0.009 +0.001 +0.024 Regularized multinomial logistic regression +0.004 -0.008 +0.008 -0.003 +0.021 Ordinal (cumulative-logit) logistic regression +0.004 -0.008 +0.007 -0.003 +0.020 SEL (method chosen each year by inner cross-validation) +0.009 -0.001 +0.011 +0.006 +0.025
Tercile RPSS, fit 1993–2016, scored 2017–2025 (exploratory)
Method Jul Jun Aug Sep JJAS Climatology blend (incumbent) +0.042 +0.010 +0.041 +0.008 +0.039 Pseudocount smoothing (no blend) +0.022 -0.056 +0.066 -0.041 +0.003 Temperature scaling +0.034 +0.009 +0.042 +0.009 +0.037 Regularized Dirichlet calibration +0.042 +0.012 +0.040 +0.010 +0.035 Regularized multinomial logistic regression +0.040 +0.016 +0.037 +0.005 +0.028 Ordinal (cumulative-logit) logistic regression +0.040 +0.016 +0.038 +0.005 +0.029 SEL (method chosen each year by inner cross-validation) +0.042 +0.009 +0.042 +0.009 +0.039
Log loss and Brier skill
Log-loss skill against climatology, 1993–2016
Method Jul Jun Aug Sep JJAS Climatology blend (incumbent) +0.011 +0.001 +0.009 +0.004 +0.019 Pseudocount smoothing (no blend) -0.053 -0.171 -0.058 -0.074 -0.041 Temperature scaling +0.010 -0.004 +0.008 +0.005 +0.018 Regularized Dirichlet calibration +0.005 -0.004 +0.007 +0.002 +0.015 Regularized multinomial logistic regression +0.002 -0.006 +0.005 -0.002 +0.012 Ordinal (cumulative-logit) logistic regression +0.002 -0.005 +0.004 -0.002 +0.012 SEL (method chosen each year by inner cross-validation) +0.007 -0.004 +0.007 +0.005 +0.018
Brier skill, below normal, 1993–2016
Method Jul Jun Aug Sep JJAS Climatology blend (incumbent) +0.015 +0.001 +0.016 +0.005 +0.034 Pseudocount smoothing (no blend) -0.035 -0.111 -0.027 -0.066 -0.003 Temperature scaling +0.015 -0.001 +0.016 +0.007 +0.032 Regularized Dirichlet calibration +0.007 -0.003 +0.013 +0.002 +0.031 Regularized multinomial logistic regression +0.001 -0.004 +0.013 -0.003 +0.027 Ordinal (cumulative-logit) logistic regression +0.002 -0.005 +0.011 -0.003 +0.026 SEL (method chosen each year by inner cross-validation) +0.011 -0.001 +0.015 +0.007 +0.033
Brier skill, above normal, 1993–2016
Method Jul Jun Aug Sep JJAS Climatology blend (incumbent) +0.014 -0.002 +0.008 +0.001 +0.019 Pseudocount smoothing (no blend) -0.023 -0.114 -0.035 -0.062 -0.030 Temperature scaling +0.014 -0.001 +0.009 +0.005 +0.021 Regularized Dirichlet calibration +0.010 -0.011 +0.005 +0.000 +0.017 Regularized multinomial logistic regression +0.006 -0.012 +0.003 -0.003 +0.014 Ordinal (cumulative-logit) logistic regression +0.005 -0.011 +0.003 -0.003 +0.014 SEL (method chosen each year by inner cross-validation) +0.008 -0.001 +0.007 +0.005 +0.018
Tests
Each method against the incumbent (negative Δ favours the method)
Method Target Δ RPS 1993–2016 Years better Holm p Δ RPS 2017–2025 Years better Confirmed Pseudocount smoothing (no blend) Jul +0.0197 10/24 1.0000 +0.0093 5/9 no Pseudocount smoothing (no blend) Jun +0.0508 5/24 1.0000 +0.0310 4/9 no Pseudocount smoothing (no blend) Aug +0.0194 10/24 1.0000 -0.0115 5/9 no Pseudocount smoothing (no blend) Sep +0.0303 5/24 1.0000 +0.0229 3/9 no Pseudocount smoothing (no blend) JJAS +0.0197 7/24 1.0000 +0.0168 5/9 no Temperature scaling Jul +0.0001 9/24 1.0000 +0.0038 4/9 no Temperature scaling Jun +0.0002 9/24 1.0000 +0.0004 6/9 no Temperature scaling Aug -0.0002 12/24 1.0000 -0.0004 5/9 no Temperature scaling Sep -0.0013 14/24 1.0000 -0.0001 6/9 no Temperature scaling JJAS +0.0001 12/24 1.0000 +0.0013 4/9 no Regularized Dirichlet calibration Jul +0.0027 11/24 1.0000 +0.0000 4/9 no Regularized Dirichlet calibration Jun +0.0029 10/24 1.0000 -0.0006 4/9 no Regularized Dirichlet calibration Aug +0.0012 9/24 1.0000 +0.0006 4/9 no Regularized Dirichlet calibration Sep +0.0009 11/24 1.0000 -0.0007 4/9 no Regularized Dirichlet calibration JJAS +0.0014 10/24 1.0000 +0.0019 3/9 no Regularized multinomial logistic regression Jul +0.0049 6/24 1.0000 +0.0009 4/9 no Regularized multinomial logistic regression Jun +0.0036 9/24 1.0000 -0.0026 7/9 no Regularized multinomial logistic regression Aug +0.0018 8/24 1.0000 +0.0018 4/9 no Regularized multinomial logistic regression Sep +0.0028 11/24 1.0000 +0.0013 4/9 no Regularized multinomial logistic regression JJAS +0.0028 9/24 1.0000 +0.0053 2/9 no Ordinal (cumulative-logit) logistic regression Jul +0.0049 6/24 1.0000 +0.0009 4/9 no Ordinal (cumulative-logit) logistic regression Jun +0.0035 8/24 1.0000 -0.0027 7/9 no Ordinal (cumulative-logit) logistic regression Aug +0.0021 8/24 1.0000 +0.0013 2/9 no Ordinal (cumulative-logit) logistic regression Sep +0.0028 12/24 1.0000 +0.0017 4/9 no Ordinal (cumulative-logit) logistic regression JJAS +0.0030 9/24 1.0000 +0.0049 2/9 no SEL Jul +0.0023 4/24 1.0000 (targets) +0.0000 0/9 no SEL Jun +0.0002 9/24 1.0000 (targets) +0.0004 6/9 no SEL Aug +0.0005 10/24 1.0000 (targets) -0.0004 5/9 no SEL Sep -0.0013 14/24 0.2391 (targets) -0.0001 6/9 no SEL JJAS +0.0006 10/24 1.0000 (targets) +0.0000 0/9 no
Fitted parameters and the selection
Fitted parameters (fit on 1993–2016) and the selection-aware choices
Target Blend weight λ Temperature T Ordinal cutpoints Ordinal slopes (mean, log spread) SEL choices (1993–2016 folds) Fallbacks Jul 0.61 3.85 -0.63, +0.67 +0.46, -0.13 Temperature scaling 15, Climatology blend (incumbent) 9 0 Jun 0.82 4.00 -0.63, +0.63 +0.33, +0.04 Temperature scaling 24 0 Aug 0.73 4.00 -0.63, +0.67 +0.40, -0.14 Temperature scaling 22, Climatology blend (incumbent) 2 0 Sep 0.77 4.00 -0.64, +0.64 +0.19, +0.07 Temperature scaling 24 0 JJAS 0.61 3.36 -0.64, +0.65 +0.47, +0.05 Temperature scaling 22, Climatology blend (incumbent) 2 0
Figures
RPSS and log-loss skill by method and target.
Reliability of each method for the season (below and above normal).
