Historical skill · FMAM 2026
How forecasts made with the same method scored in past years against climatology (the observed frequency of each tercile in the training years, about one-third each). The main number is the cross-validated 1993–2016 score: each year is forecast with parameters fitted without that year. Later years are exploratory: they were seen while the method was chosen.
Takeaway — FMAM 2026
Historical skill is uncertain: cross-validated 1993–2016 RPSS is +0.036, and its interval (−0.003 to +0.071) includes no improvement over climatology. The exploratory 2017–2025 result (+0.014) is encouraging but those years were inspected during method selection. The 2026 verification (−0.054) is one season and does not override the historical uncertainty.
Stronger historical evidence (interval above zero): May +0.035.
| Target | Cross-validated 1993–2016 (main) | In words | Exploratory 2017–2025 |
|---|---|---|---|
| FMAM | +0.036 (−0.003 to +0.071) | Small estimated improvement; skill uncertain | +0.014 (−0.048 to +0.080) |
| Feb | +0.026 (−0.006 to +0.059) | Small estimated improvement; skill uncertain | +0.025 (−0.003 to +0.057) |
| Mar | +0.001 (−0.012 to +0.012) | Small estimated improvement; skill uncertain | +0.011 (−0.004 to +0.024) |
| Apr | −0.002 (−0.020 to +0.013) | No demonstrated improvement over climatology | +0.001 (−0.027 to +0.028) |
| May | +0.035 (+0.009 to +0.060) | Modest improvement over climatology (interval above zero) | −0.006 (−0.089 to +0.051) |
RPSS +0.05 means a 5% lower ranked probability score than climatology. An interval that includes zero means the gain is not established.
Technical detail: years better, p-values and blend weights
| Target | Years better (cross-validated) | p (one target / Holm) | Years better (exploratory) | Blend weight λ |
|---|---|---|---|---|
| FMAM | 18 of 24 | 0.043 / 0.171 | 3 of 9 | 0.34 |
| Feb | 16 of 24 | 0.067 / 0.202 | 7 of 9 | 0.76 |
| Mar | 12 of 24 | 0.451 / 0.901 | 5 of 9 | 0.86 |
| Apr | 12 of 24 | 0.564 / 0.901 | 5 of 9 | 0.76 |
| May | 19 of 24 | 0.009 / 0.044 | 6 of 9 | 0.59 |
Each interval and one-target p-value describes that target alone. Because 5 targets are examined, the Holm-adjusted p is the stricter test: May remains significant after adjustment. λ is the weight given to climatology in the blend (higher = closer to climatology).
Historical verification explorer
The evidence behind the scores above, for FMAM 2026 in this area. Choose the evaluation period; all diagnostics use the blended probabilities against climatology.
Performance by year
Probability skill (RPSS) of each year; better than climatology in 18 of 24 years. Weakest years: 2006 (−0.308), 2003 (−0.070), 2001 (−0.063).
Reliability
Points on the dashed diagonal are reliable: events forecast with probability p happen p of the time. Points flatter than the diagonal mean over-confident probabilities. Open points have fewer than 134 cell-years (0.5% of the sample, at least 50) and are not joined. Hover a point for its count.
Reliability table (forecast bins, observed frequency, counts)
| Category | Bin | Mean forecast | Observed frequency | Cell-years |
|---|---|---|---|---|
| below normal | 0%–10% | 9% | 24% | 21 |
| below normal | 10%–20% | 18% | 21% | 1,478 |
| below normal | 20%–30% | 26% | 26% | 7,352 |
| below normal | 30%–40% | 35% | 37% | 12,048 |
| below normal | 40%–50% | 44% | 44% | 5,017 |
| below normal | 50%–60% | 54% | 43% | 842 |
| below normal | 60%–70% | 63% | 39% | 120 |
| below normal | 70%–80% | 71% | 100% | 2 |
| near normal | 0%–10% | 9% | 21% | 43 |
| near normal | 10%–20% | 16% | 23% | 1,896 |
| near normal | 20%–30% | 25% | 27% | 9,970 |
| near normal | 30%–40% | 35% | 33% | 10,263 |
| near normal | 40%–50% | 44% | 38% | 3,999 |
| near normal | 50%–60% | 53% | 36% | 659 |
| near normal | 60%–70% | 63% | 38% | 50 |
| above normal | 10%–20% | 17% | 26% | 1,328 |
| above normal | 20%–30% | 26% | 22% | 7,252 |
| above normal | 30%–40% | 34% | 34% | 11,130 |
| above normal | 40%–50% | 44% | 49% | 5,586 |
| above normal | 50%–60% | 54% | 57% | 1,380 |
| above normal | 60%–70% | 64% | 46% | 190 |
| above normal | 70%–80% | 72% | 50% | 14 |
How strong are the signals?
Share of cell-years by the leading tercile probability.
| Category | Forecast | Climatology | Brier skill |
|---|---|---|---|
| below normal | 0.2206 | 0.2266 | +0.027 |
| near normal | 0.2128 | 0.2128 | 0.000 |
| above normal | 0.2161 | 0.2262 | +0.045 |
Rainfall amounts
| Rainfall amount skill (CRPSS) | +0.016 |
| CRPS, corrected / climatology | 57.7 / 58.6 mm |
| Average rainfall error, corrected (raw model) | −27.1 mm (−113.4 mm) |
| RMSE, corrected / climatology | 104.5 / 106.1 mm |
Performance by year
Probability skill (RPSS) of each year; better than climatology in 3 of 9 years. Weakest years: 2022 (−0.157), 2025 (−0.027), 2020 (−0.011).
Reliability
Points on the dashed diagonal are reliable: events forecast with probability p happen p of the time. Points flatter than the diagonal mean over-confident probabilities. Open points have fewer than 50 cell-years (0.5% of the sample, at least 50) and are not joined. Hover a point for its count.
Reliability table (forecast bins, observed frequency, counts)
| Category | Bin | Mean forecast | Observed frequency | Cell-years |
|---|---|---|---|---|
| below normal | 10%–20% | 18% | 20% | 233 |
| below normal | 20%–30% | 26% | 26% | 3,717 |
| below normal | 30%–40% | 34% | 25% | 4,106 |
| below normal | 40%–50% | 43% | 24% | 1,804 |
| below normal | 50%–60% | 53% | 32% | 215 |
| below normal | 60%–70% | 61% | 40% | 5 |
| near normal | 10%–20% | 18% | 15% | 389 |
| near normal | 20%–30% | 26% | 24% | 3,380 |
| near normal | 30%–40% | 34% | 29% | 4,637 |
| near normal | 40%–50% | 43% | 34% | 1,555 |
| near normal | 50%–60% | 53% | 38% | 110 |
| near normal | 60%–70% | 62% | 0% | 9 |
| above normal | 10%–20% | 19% | 30% | 272 |
| above normal | 20%–30% | 26% | 38% | 2,493 |
| above normal | 30%–40% | 35% | 46% | 4,684 |
| above normal | 40%–50% | 43% | 61% | 2,523 |
| above normal | 50%–60% | 52% | 76% | 108 |
How strong are the signals?
Share of cell-years by the leading tercile probability.
| Category | Forecast | Climatology | Brier skill |
|---|---|---|---|
| below normal | 0.1997 | 0.1947 | −0.026 |
| near normal | 0.2005 | 0.2024 | +0.009 |
| above normal | 0.2580 | 0.2696 | +0.043 |
Rainfall amounts
| Rainfall amount skill (CRPSS) | +0.016 |
| CRPS, corrected / climatology | 57.7 / 58.6 mm |
| Average rainfall error, corrected (raw model) | −27.1 mm (−113.4 mm) |
| RMSE, corrected / climatology | 104.5 / 106.1 mm |
Historical performance by rainfall region
Probability skill by rainfall region (full regional matrix)
| Region | FMAM | Feb | Mar | Apr | May |
|---|---|---|---|---|---|
| All Ethiopia | +0.032 (−0.003 to +0.067) 18/24 yrs | +0.027 (−0.002 to +0.055) 16/24 yrs | +0.002 (−0.009 to +0.012) 13/24 yrs | +0.001 (−0.014 to +0.014) 15/24 yrs | +0.028 (+0.005 to +0.051) 17/24 yrs |
| R0 arid / marginal | +0.051 (−0.012 to +0.113) 14/24 yrs | +0.026 (−0.023 to +0.076) 14/24 yrs | +0.002 (−0.024 to +0.025) 15/24 yrs | −0.019 (−0.050 to +0.007) 11/24 yrs | +0.033 (0.000 to +0.066) 15/24 yrs |
| R1 western unimodal | −0.003 (−0.053 to +0.043) 13/24 yrs | +0.040 (+0.016 to +0.062) 18/24 yrs | +0.003 (−0.008 to +0.013) 14/24 yrs | +0.013 (−0.004 to +0.030) 15/24 yrs | −0.014 (−0.041 to +0.014) 10/24 yrs |
| R2 Belg–Kiremt | +0.061 (+0.011 to +0.113) 17/24 yrs | +0.039 (+0.010 to +0.066) 18/24 yrs | +0.008 (−0.001 to +0.017) 16/24 yrs | −0.008 (−0.036 to +0.014) 14/24 yrs | +0.045 (+0.014 to +0.075) 19/24 yrs |
| R3 Gu–Deyr | +0.036 (−0.012 to +0.078) 18/24 yrs | +0.008 (−0.032 to +0.048) 13/24 yrs | −0.003 (−0.022 to +0.015) 10/24 yrs | +0.002 (−0.018 to +0.020) 14/24 yrs | +0.047 (+0.004 to +0.088) 19/24 yrs |
| FMAM rainfall contribution domain | +0.036 (−0.003 to +0.071) 18/24 yrs | +0.026 (−0.006 to +0.059) 16/24 yrs | +0.001 (−0.012 to +0.012) 12/24 yrs | −0.002 (−0.020 to +0.013) 12/24 yrs | +0.035 (+0.009 to +0.060) 19/24 yrs |
| FMAM main-season rainfall domain (Belg / Gu / Ganna, >=40% of annual) | +0.041 (−0.008 to +0.083) 17/24 yrs | +0.006 (−0.034 to +0.047) 13/24 yrs | −0.002 (−0.021 to +0.014) 10/24 yrs | +0.001 (−0.021 to +0.020) 13/24 yrs | +0.050 (+0.007 to +0.092) 20/24 yrs |
Positive values mean lower RPS than climatology. Intervals resample whole years; regions are the fixed 1993–2025 rainfall regimes, and the last rows are this cycle's rainfall domains. Regional results are noisier than national ones.