Probability and rainfall-distribution calibration

Can a different probability method improve the tercile probabilities users receive? The published pipeline is MVS-corrected members → smoothed member counts → climatology blend (the incumbent). Phase 1 keeps the amount correction fixed (Pathway A) and replaces only the probability stage; Phase 2 adds one-vs-rest binary calibrators and the first full rainfall-distribution models (Pathway B); Phase 3 adds Bayesian joint probability, BMA for precipitation and quantile regression forests. Every parameter is fitted only on training years within each left-out year. The published forecasts are unchanged.

Cycles

Plan

Implementation plan (the order is a plan, not a ranking of performance)
Phase	Identifier	Method	Family	Output	Status
1	`pseudocount`	Pseudocount smoothing	baseline / stabilization	tercile probabilities	implemented, results on each cycle page
1	`climatology_blend`	Climatology blending (incumbent)	baseline / shrinkage	tercile probabilities	implemented, results on each cycle page
1	`temperature`	Temperature scaling	tercile calibration	tercile probabilities	implemented, results on each cycle page
1	`dirichlet`	Regularized Dirichlet calibration	tercile calibration	tercile probabilities	implemented, results on each cycle page
1	`multinomial_logistic`	Regularized multinomial logistic regression	tercile prediction	tercile probabilities	implemented, results on each cycle page
1	`ordinal_logistic`	Ordinal (cumulative-logit) logistic regression	tercile prediction	tercile probabilities	implemented, results on each cycle page
2	`platt_ovr`	Platt (sigmoid) calibration, one-vs-rest + normalization	binary calibration	tercile probabilities	implemented, results on each cycle page
2	`beta_ovr`	Beta calibration, one-vs-rest + normalization	binary calibration	tercile probabilities	implemented, results on each cycle page
2	`isotonic_ovr`	Isotonic regression, one-vs-rest + normalization	binary calibration	tercile probabilities	implemented, results on each cycle page
2	`emos_csg`	EMOS, censored shifted gamma	rainfall distribution	CDF, quantiles, terciles	implemented, results on each cycle page
2	`extended_logistic`	Extended logistic regression	rainfall distribution	CDF, terciles	implemented, results on each cycle page
2	`quantile_regression`	Regularized quantile regression	rainfall quantiles	quantiles (CDF by interpolation), terciles	implemented, results on each cycle page
3	`bjp`	Bayesian joint probability	rainfall distribution	CDF, quantiles, terciles	implemented, results on each cycle page
3	`bma_precipitation`	Bayesian model averaging (precipitation)	rainfall distribution	CDF, quantiles, terciles	implemented, results on each cycle page
3	`quantile_regression_forest`	Quantile regression forest	rainfall quantiles	conditional distribution, terciles	implemented, results on each cycle page

Definitions kept distinct

Pseudocount smoothing p_k = (n_k + α)/(M + 3α) stabilizes count probabilities; the climatology blend p = (1 − λ)·p_smooth + λ·p_clim learns how much forecast information to keep. Both are kept as references.
Dirichlet calibration (a regularized linear map of the log probabilities, then softmax) is not pseudocount smoothing.
Dirichlet uses only the log probabilities; multinomial logistic uses only the ensemble predictors, so they are different models. Ordinal logistic uses the same predictors with ordered cutpoints (below → near → above).
All methods are pooled over the grid cells of a target with equal weight per year; a method that cannot be fitted falls back to the incumbent and the fallback is counted.

Full write-up: docs/48. Code: scripts/prob_calibration.py, scripts/prob_benchmark.py, scripts/dist_calibration.py, scripts/dist_benchmark.py, scripts/dist_shadow.py.

References

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning, PMLR 70, 1321–1330. proceedings.mlr.press/v70/guo17a.html
Kull, M., Perello-Nieto, M., Kängsepp, M., Silva Filho, T., Song, H., & Flach, P. (2019). Beyond temperature scaling: obtaining well-calibrated multiclass probabilities with Dirichlet calibration. Advances in Neural Information Processing Systems 32 (NeurIPS 2019). arxiv.org/abs/1910.12656
McCullagh, P. (1980). Regression models for ordinal data. Journal of the Royal Statistical Society, Series B, 42, 109–127. doi.org/10.1111/j.2517-6161.1980.tb01109.x
Rajagopalan, B., Lall, U., & Zebiak, S. E. (2002). Categorical climate forecasts through regularization and optimal combination of multiple GCM ensembles. Monthly Weather Review, 130, 1792–1811. doi.org/10.1175/1520-0493(2002)130<1792:CCFTRA>2.0.CO;2
Wilks, D. S. (2019). Statistical Methods in the Atmospheric Sciences (4th ed.). Elsevier.
Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers (pp. 61–74). MIT Press.
Kull, M., Silva Filho, T., & Flach, P. (2017). Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. Proceedings of AISTATS 2017, PMLR 54. proceedings.mlr.press/v54/kull17a.html
Zadrozny, B., & Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the 8th ACM SIGKDD, 694–699. doi.org/10.1145/775047.775151
Scheuerer, M., & Hamill, T. M. (2015). Statistical postprocessing of ensemble precipitation forecasts by fitting censored, shifted gamma distributions. Monthly Weather Review, 143, 4578–4596. doi.org/10.1175/MWR-D-15-0061.1
Wilks, D. S. (2009). Extending logistic regression to provide full-probability-distribution MOS forecasts. Meteorological Applications, 16, 361–368. doi.org/10.1002/met.134
Messner, J. W., Mayr, G. J., Zeileis, A., & Wilks, D. S. (2014). Heteroscedastic extended logistic regression for postprocessing of ensemble guidance. Monthly Weather Review, 142, 448–456. doi.org/10.1175/MWR-D-13-00271.1
Koenker, R., & Bassett, G. (1978). Regression quantiles. Econometrica, 46, 33–50. doi.org/10.2307/1913643
Wang, Q. J., Robertson, D. E., & Chiew, F. H. S. (2009). A Bayesian joint probability modeling approach for seasonal forecasting of streamflows at multiple sites. Water Resources Research, 45, W05407. doi.org/10.1029/2008WR007355
Wang, Q. J., & Robertson, D. E. (2011). Multisite probabilistic forecasting of seasonal flows for streams with zero value occurrences. Water Resources Research, 47, W02546. doi.org/10.1029/2010WR009333
Wang, Q. J., Shrestha, D. L., Robertson, D. E., & Pokhrel, P. (2012). A log-sinh transformation for data normalization and variance stabilization. Water Resources Research, 48, W05514. doi.org/10.1029/2011WR010973
Sloughter, J. M., Raftery, A. E., Gneiting, T., & Fraley, C. (2007). Probabilistic quantitative precipitation forecasting using Bayesian model averaging. Monthly Weather Review, 135, 3209–3220. doi.org/10.1175/MWR3441.1
Fraley, C., Raftery, A. E., & Gneiting, T. (2010). Calibrating multimodel forecast ensembles with exchangeable and missing members using Bayesian model averaging. Monthly Weather Review, 138, 190–202. doi.org/10.1175/2009MWR3046.1
Meinshausen, N. (2006). Quantile regression forests. Journal of Machine Learning Research, 7, 983–999. jmlr.org/papers/v7/meinshausen06a.html