September initialization · ONDJ and its months
Research benchmark. The published forecasts still use Mean-Variance Scaling (operational). This benchmark compares 12 other rainfall-amount bias-correction methods with it, member by member, for every grid cell and target: a screening of the amount correction alone, then a confirmation through the full probability pipeline that also accounts for how a method is chosen.
What the evidence supports
- Published tercile probabilities: no tested method demonstrates a statistically significant improvement over Mean-Variance Scaling under this validation protocol (several have slightly higher point estimates); the selection-aware procedure is confirmed for none.
- Rainfall amounts: confirmed against Mean-Variance Scaling (family-wide, full pipeline): Square-root-space Mean-Variance Scaling for Jan; Gamma Quantile Mapping for Jan. The selection-aware procedure is confirmed for Jan. These comparisons establish improvement against the reference, not that one candidate is better than another.
- The operational method stays Mean-Variance Scaling until a change is decided by initialization, target and product, and recorded in the docs, on this evidence; confirmed improvements limited to some months may justify a per-target choice.
- If amounts and probabilities come to use different correction paths, the product metadata must record both methods so that every output can be traced.
Suggested method
The platform suggests a bias-correction method for every target and product from the historical evidence (the confirmation against CHIRPS, counting every method tried) and the independent validation seasons scored since (the shadow and validation runs), by a rule fixed before those seasons are scored. A suggestion is advice for a decision: the published forecast always uses Mean-Variance Scaling until a change is decided. 1 of 10 target–product combinations have a candidate.
The rule
- Candidate: A method is a candidate only if the confirmation (CHIRPS, outer leave-one-year-out over the training years, full pipeline) finds it better than the operational method with Benjamini-Hochberg-adjusted p < 0.05 (false discovery rate) across every declared method and target, and not worse over the test years. Variants outside the declared family are not candidates.
- Choice: Among candidates, the method the selection-aware procedure chooses from all training years (training-only information); if that method is not a candidate, the candidate with the best training-period score. Other candidates are listed as alternatives: a confirmation establishes improvement against the operational method, not a ranking among candidates.
- Validation: Independent validation seasons are forecast years scored after the benchmark (bias_validation.py). A candidate is 'recommended' once at least minimum_seasons have been scored for it and the operational method, and its mean score over them is not worse; with fewer seasons it stays a 'candidate'; with enough seasons and a worse mean it is 'not supported by validation'. Minimum: 3 seasons.
| Target | Product | Suggested method | Status | Evidence |
|---|---|---|---|---|
| Oct | Published tercile probabilities | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Oct on the historical evidence. |
| Oct | Rainfall amounts | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Oct on the historical evidence. |
| Nov | Published tercile probabilities | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Nov on the historical evidence. |
| Nov | Rainfall amounts | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Nov on the historical evidence. |
| Dec | Published tercile probabilities | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Dec on the historical evidence. |
| Dec | Rainfall amounts | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Dec on the historical evidence. |
| Jan | Published tercile probabilities | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for Jan on the historical evidence. |
| Jan | Rainfall amounts | Square-root-space Mean-Variance Scaling | Candidate — validation 0 of 3 seasons | Square-root-space Mean-Variance Scaling is confirmed better than Mean-Variance Scaling for Jan (CRPS (mm) difference -0.4787 over the training years, Benjamini–Hochberg p 0.0106; -0.6902 over the test years). Independent validation: 0 of 3 season(s). Also confirmed: Variance Scaling (unbounded), Gamma Quantile Mapping. |
| ONDJ | Published tercile probabilities | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for ONDJ on the historical evidence. |
| ONDJ | Rainfall amounts | Mean-Variance Scaling | Keep the operational method | No method is confirmed better than Mean-Variance Scaling for ONDJ on the historical evidence. |
Explore the evidence
- Interpretation
What the results mean, statement by statement (each checked against the numbers), and the caveats. - Data and methods
The data, the 13 methods and the reference, and the evaluation designs. - Screening
The amount correction alone: skill by method and target, every region, statistics, diagnostics. - Confirmation
Every method through the full probability pipeline, and the selection-aware procedures. - Ratio limits
The operational MVS limits (0.5–2) against wider limits and none. - Figures
Summary figures, per-cell maps, rank histograms and the forecast members. - Downloads
Result files, provenance, code and references.