Complexity-Aware Geographic Fairness in Crop Yield Prediction Using Diversity-Weighted Learning

Abstract

Machine-learning models for crop yield prediction are usually judged by overall accuracy alone, which hides large differences in prediction quality between regions. This study shows that a standard Random Forest with an aggregate R² of 0.935 can be misleading: state-level MAE ranges from 0.168 t/ha to 2.505 t/ha, a 14.9-fold gap that the overall score conceals. Two independent structural factors underlie this imbalance: the number of records per state (data-volume imbalance) and the agricultural complexity of each state, both of which can be observed in any census or statistical record. To address both at once, a Complexity-Aware Geographic Fairness (CAGF) framework is presented that combines a Crop Diversity Index based on Shannon entropy with inverse-frequency sample re-weighting. Four architectures (Random Forest, XGBoost, SVR and MLP) are evaluated under three training regimes (unweighted baseline, inverse-frequency weighting and CAGF) using expanding-window temporal cross-validation, and TreeSHAP attribution shows that CAGF directs model attention toward crop-specific regional patterns. For XGBoost, CAGF gives the best overall accuracy among the three regimes (R² = 0.918; MAE = 1.299 t/ha) together with a lower geographic Gini of prediction error than the unweighted baseline (GGPE = 0.289 versus 0.293), so fairness improves without any loss of accuracy.