When a rainfall model reports 99.7% accuracy, the right reaction isn’t celebration — it’s suspicion.
While building Stormwatch AI, a set of models forecasting cyclones, heatwaves, and rainfall across 15 Indian cities, one of the rainfall models kept scoring implausibly high. Real-world rainfall prediction from weather station data does not look like that.
Finding the leak
The leakage traced back to features that encoded information only available after the event being predicted — a classic case of the training pipeline peeking into the future. Three models across the pipeline shared the same feature-engineering step, so the bug wasn’t isolated to one notebook; it was systemic.
The fix
- Rebuilt the feature pipeline so every feature was computed strictly from data available at prediction time.
- Re-ran the walk-forward validation split to confirm no temporal overlap between train and test windows.
- Re-trained all three models and re-measured.
The rainfall model landed at 88.3% accuracy — lower, but real. That number now ships behind a 5-endpoint FastAPI service with an MLflow registry, and KS-test drift monitoring keeps watch for the next time a feature quietly starts leaking the future.
Stack: Python, XGBoost, FastAPI, MLflow, PySpark, GitHub Actions