Historical skill · FMAM 2026
How forecasts made with the same method scored in past years against climatology (the observed frequency of each tercile in the training years, about one-third each). The main number is the cross-validated 1993–2016 score: each year is forecast with parameters fitted without that year. Later years are exploratory: they were seen while the method was chosen.
Takeaway — FMAM 2026
Historical skill is uncertain: cross-validated 1993–2016 RPSS is +0.017, and its interval (−0.038 to +0.067) includes no improvement over climatology. The exploratory 2017–2025 result (+0.061) is encouraging but those years were inspected during method selection. The 2026 verification (+0.006) is one season and does not override the historical uncertainty.
Stronger historical evidence (interval above zero): Feb +0.137.
| Target | Cross-validated 1993–2016 (main) | In words | Exploratory 2017–2025 |
|---|---|---|---|
| FMAM | +0.017 (−0.038 to +0.067) | Small estimated improvement; skill uncertain | +0.061 (+0.029 to +0.095) |
| Feb | +0.137 (+0.050 to +0.234) | Moderate improvement over climatology (interval above zero) | +0.049 (−0.017 to +0.128) |
| Mar | +0.011 (−0.020 to +0.042) | Small estimated improvement; skill uncertain | +0.041 (+0.007 to +0.080) |
| Apr | +0.011 (−0.032 to +0.048) | Small estimated improvement; skill uncertain | +0.015 (−0.009 to +0.042) |
| May | +0.007 (−0.051 to +0.050) | Small estimated improvement; skill uncertain | +0.017 (−0.033 to +0.057) |
RPSS +0.05 means a 5% lower ranked probability score than climatology. An interval that includes zero means the gain is not established.
Technical detail: years better, p-values and blend weights
| Target | Years better (cross-validated) | p (one target / Holm) | Years better (exploratory) | Blend weight λ |
|---|---|---|---|---|
| FMAM | 15 of 24 | 0.276 / 1.000 | 8 of 9 | 0.40 |
| Feb | 16 of 24 | 0.002 / 0.012 | 5 of 9 | 0.50 |
| Mar | 15 of 24 | 0.251 / 1.000 | 7 of 9 | 0.73 |
| Apr | 16 of 24 | 0.305 / 1.000 | 5 of 9 | 0.58 |
| May | 17 of 24 | 0.434 / 1.000 | 8 of 9 | 0.55 |
Each interval and one-target p-value describes that target alone. Because 5 targets are examined, the Holm-adjusted p is the stricter test: Feb remains significant after adjustment. λ is the weight given to climatology in the blend (higher = closer to climatology).
Historical verification explorer
The evidence behind the scores above, for FMAM 2026 in this area. Choose the evaluation period; all diagnostics use the blended probabilities against climatology.
Performance by year
Probability skill (RPSS) of each year; better than climatology in 15 of 24 years. Weakest years: 1996 (−0.289), 2006 (−0.221), 2011 (−0.173).
Reliability
Points on the dashed diagonal are reliable: events forecast with probability p happen p of the time. Points flatter than the diagonal mean over-confident probabilities. Open points have fewer than 134 cell-years (0.5% of the sample, at least 50) and are not joined. Hover a point for its count.
Reliability table (forecast bins, observed frequency, counts)
| Category | Bin | Mean forecast | Observed frequency | Cell-years |
|---|---|---|---|---|
| below normal | 10%–20% | 18% | 27% | 613 |
| below normal | 20%–30% | 26% | 26% | 8,774 |
| below normal | 30%–40% | 35% | 38% | 11,145 |
| below normal | 40%–50% | 44% | 44% | 5,434 |
| below normal | 50%–60% | 53% | 33% | 906 |
| below normal | 60%–70% | 61% | 37% | 8 |
| near normal | 10%–20% | 18% | 21% | 784 |
| near normal | 20%–30% | 26% | 28% | 11,765 |
| near normal | 30%–40% | 34% | 33% | 12,019 |
| near normal | 40%–50% | 43% | 39% | 2,235 |
| near normal | 50%–60% | 52% | 51% | 74 |
| near normal | 60%–70% | 64% | 0% | 3 |
| above normal | 10%–20% | 19% | 53% | 432 |
| above normal | 20%–30% | 26% | 29% | 8,049 |
| above normal | 30%–40% | 35% | 29% | 11,932 |
| above normal | 40%–50% | 44% | 49% | 5,164 |
| above normal | 50%–60% | 54% | 50% | 933 |
| above normal | 60%–70% | 62% | 53% | 370 |
How strong are the signals?
Share of cell-years by the leading tercile probability.
| Category | Forecast | Climatology | Brier skill |
|---|---|---|---|
| below normal | 0.2231 | 0.2266 | +0.016 |
| near normal | 0.2116 | 0.2128 | +0.006 |
| above normal | 0.2222 | 0.2262 | +0.018 |
Rainfall amounts
| Rainfall amount skill (CRPSS) | +0.071 |
| CRPS, corrected / climatology | 54.5 / 58.6 mm |
| Average rainfall error, corrected (raw model) | −28.0 mm (−102.8 mm) |
| RMSE, corrected / climatology | 99.9 / 106.1 mm |
Performance by year
Probability skill (RPSS) of each year; better than climatology in 8 of 9 years. Weakest years: 2021 (−0.005), 2018 (+0.008), 2025 (+0.009).
Reliability
Points on the dashed diagonal are reliable: events forecast with probability p happen p of the time. Points flatter than the diagonal mean over-confident probabilities. Open points have fewer than 50 cell-years (0.5% of the sample, at least 50) and are not joined. Hover a point for its count.
Reliability table (forecast bins, observed frequency, counts)
| Category | Bin | Mean forecast | Observed frequency | Cell-years |
|---|---|---|---|---|
| below normal | 10%–20% | 19% | 6% | 16 |
| below normal | 20%–30% | 27% | 6% | 2,764 |
| below normal | 30%–40% | 35% | 29% | 6,028 |
| below normal | 40%–50% | 43% | 46% | 1,268 |
| below normal | 50%–60% | 52% | 100% | 4 |
| near normal | 10%–20% | 19% | 23% | 17 |
| near normal | 20%–30% | 27% | 24% | 2,949 |
| near normal | 30%–40% | 34% | 28% | 6,248 |
| near normal | 40%–50% | 43% | 37% | 857 |
| near normal | 50%–60% | 51% | 0% | 9 |
| above normal | 10%–20% | 20% | 0% | 7 |
| above normal | 20%–30% | 27% | 29% | 2,888 |
| above normal | 30%–40% | 34% | 50% | 6,092 |
| above normal | 40%–50% | 43% | 82% | 1,070 |
| above normal | 50%–60% | 52% | 100% | 23 |
How strong are the signals?
Share of cell-years by the leading tercile probability.
| Category | Forecast | Climatology | Brier skill |
|---|---|---|---|
| below normal | 0.1823 | 0.1947 | +0.064 |
| near normal | 0.2006 | 0.2024 | +0.009 |
| above normal | 0.2537 | 0.2696 | +0.059 |
Rainfall amounts
| Rainfall amount skill (CRPSS) | +0.071 |
| CRPS, corrected / climatology | 54.5 / 58.6 mm |
| Average rainfall error, corrected (raw model) | −28.0 mm (−102.8 mm) |
| RMSE, corrected / climatology | 99.9 / 106.1 mm |
Historical performance by rainfall region
Probability skill by rainfall region (full regional matrix)
| Region | FMAM | Feb | Mar | Apr | May |
|---|---|---|---|---|---|
| All Ethiopia | +0.021 (−0.031 to +0.069) 15/24 yrs | +0.126 (+0.043 to +0.217) 18/24 yrs | +0.011 (−0.017 to +0.038) 15/24 yrs | +0.009 (−0.034 to +0.044) 15/24 yrs | +0.008 (−0.042 to +0.047) 16/24 yrs |
| R0 arid / marginal | +0.023 (−0.072 to +0.103) 15/24 yrs | +0.057 (−0.081 to +0.206) 13/24 yrs | −0.005 (−0.058 to +0.044) 15/24 yrs | −0.028 (−0.102 to +0.038) 13/24 yrs | −0.015 (−0.079 to +0.040) 13/24 yrs |
| R1 western unimodal | +0.019 (−0.035 to +0.067) 18/24 yrs | +0.156 (+0.068 to +0.248) 18/24 yrs | +0.009 (−0.018 to +0.034) 15/24 yrs | +0.016 (−0.024 to +0.047) 17/24 yrs | +0.009 (−0.033 to +0.046) 15/24 yrs |
| R2 Belg–Kiremt | +0.037 (−0.050 to +0.108) 17/24 yrs | +0.166 (+0.076 to +0.264) 20/24 yrs | +0.023 (−0.017 to +0.058) 16/24 yrs | −0.001 (−0.063 to +0.052) 14/24 yrs | +0.009 (−0.051 to +0.059) 16/24 yrs |
| R3 Gu–Deyr | +0.012 (−0.042 to +0.064) 15/24 yrs | +0.082 (−0.006 to +0.180) 14/24 yrs | +0.006 (−0.030 to +0.041) 14/24 yrs | +0.017 (−0.034 to +0.064) 16/24 yrs | +0.009 (−0.066 to +0.062) 19/24 yrs |
| FMAM rainfall contribution domain | +0.017 (−0.038 to +0.067) 15/24 yrs | +0.137 (+0.050 to +0.234) 16/24 yrs | +0.011 (−0.020 to +0.042) 15/24 yrs | +0.011 (−0.032 to +0.048) 16/24 yrs | +0.007 (−0.051 to +0.050) 17/24 yrs |
| FMAM main-season rainfall domain (Belg / Gu / Ganna, >=40% of annual) | +0.013 (−0.040 to +0.066) 16/24 yrs | +0.080 (−0.005 to +0.176) 14/24 yrs | +0.007 (−0.029 to +0.042) 15/24 yrs | +0.013 (−0.041 to +0.060) 17/24 yrs | +0.009 (−0.066 to +0.062) 18/24 yrs |
Positive values mean lower RPS than climatology. Intervals resample whole years; regions are the fixed 1993–2025 rainfall regimes, and the last rows are this cycle's rainfall domains. Regional results are noisier than national ones.