Demand Forecasting at Scale
SKU-level demand forecasting across 10K+ products and thousands of stores — handling promotional uplift and seasonality to cut waste 22% and drive ~£2M annual savings.
What the platform does
The platform forecasts demand at SKU × store granularity across multi-horizon windows, feeding automated reordering. It fuses point-of-sale and stock data with external signals — weather, promotions, seasonality — through Spark feature pipelines and an ensemble forecast, orchestrated by Airflow.
Why coarse forecasting failed
Over-ordering perishables drives waste and margin loss; under-ordering drives stockouts. Coarse models couldn't attribute demand shifts to local drivers, so buyers over-corrected. The requirement was granular, explainable, signal-aware forecasting at scale.
System architecture
Spark pipelines process 50M+ daily transaction records into features; an ensemble of Prophet (seasonality/trend) and XGBoost (signal interactions) is stacked by a meta-learner; Airflow DAGs replace fragile manual workflows with monitored automation.
Forecasting methodology
Prophet captures trend and seasonality per SKU; XGBoost captures interactions between promotions, weather, and price. A stacking meta-learner combines them, and forecasts are backtested on held-out horizons before driving reorders.
def forecast(sku: str, horizon: int) -> Forecast:
base = prophet_models[sku].predict(horizon) # trend + seasonality
signal = xgb.predict(feature_frame(sku, horizon)) # promo × weather × price
# meta-learner blends base + signal, weighted by recent backtest error
return meta_learner.combine(base, signal)Architecture decision records
Ensemble over a single model
Decision: combine Prophet and XGBoost via stacking rather than one model.
Consequence: Prophet handles seasonality it's strong at; XGBoost handles promotional interactions; the blend beats either alone on backtest.
Airflow-orchestrated, monitored pipelines
Decision: replace manual workflows with Airflow DAGs.
Consequence: retraining and validation are scheduled, monitored, and recoverable — production MLOps rather than fragile scripts.