The World Food Programme estimates (pdf) that approximately 3.5 million Malawians are chronically food insecure. That number rises, and hunger becomes more acute, in the lean season between harvests. Seasonal hunger is further aggravated by limited storage options and extreme weather events, such as the recent weak rains that rendered 25% of the population acutely food insecure. The impacts of crop losses go beyond rural subsistence producers; they also affect urban consumers through the resulting food price shocks.
In a recent article, we asked whether data related to crop production or markets could allow for greater precision in identifying which communities would face food insufficiency, which we define as a substantial proportion of households reporting “running out” of food in each month. Our data come from four waves of the Malawi Integrated Household Survey (IHS). Our goals were to understand whether using publicly available data derived from satellites and geo-coded market food prices with machine learning models could produce more accurate predictions than simpler approaches like classical regression or non-modeled human predictions based on past occurrences. To test the models, we first fit them using information from one or more survey rounds, then tested whether they could classify each community in the next survey round as food sufficient or food insufficient using updated predictor variables.
Our key takeaways were:
- Producing accurate forecasts can require multiple years of data and, even in their simplest form, machine learning models still require some expertise
- Prices, which reflect multiple factors on the supply and demand side, work as well as weather data
- When food insufficiency recurs with some spatial and temporal predictability, machine learning models may not add substantial improvements to overall accuracy, but the distribution of communities predicted as food sufficient or food insufficient varies substantially by model, even at similar accuracy rates.
More details, including considerations for constructing indicators, specifying models and determining relevant predictors, are available in our paper.
1. Evaluating the Comparative Accuracy of Machine Learning Models
Machine learning models can discern subtle patterns in large datasets, but the size requirement can restrict where the models can be effectively used. Using datasets that are too small can lead to low generalizability – fits are highly accurate on the sample used to develop the models, but extrapolations on new data may be weaker.
To evaluate whether machine-learning was performing well given the available public data, we trained ML models on one to three rounds of the survey, testing on the second through fourth survey rounds, and compared them to classical regression and a non-modeled measure of checking the food security situation of the nearest neighboring community in the previous year. In addition to accuracy, we compared rates of false positive predictions (classifying food-sufficient communities as food-insufficient) and false negative predictions (classifying food-insufficient communities as food-sufficient) using two additional metrics: recall and precision. Recall represents the ratio of true positive predictions out of all observations of food insufficiency, and precision represents the ratio of true positive predictions to false positive predictions. Low recall scores indicate a bias toward producing false negative predictions, and low precision scores indicate a bias toward false positive predictions. We found that machine learning models tended to outperform on recall but under-perform on precision compared to the classical approaches and that model accuracy did not approach the simple non-modeled approach until three waves of survey data were used for training, a finding that indicates that although food insufficiency has high recurrence rates, the underlying reasons in a given year may vary.
2. Comparing Prices to Weather as Leading Indicators
Although the demand for staple crops is relatively stable, prices are affected by international trade, policy, weather, input prices, and other factors that affect supply. Hence, price movements offer a convenient summary of current and forecast changes in the availability of food relative to demand.
Using a primary measure of food affordability (observed maize market price over the prior year), combined with indicators of direction (maize price inflation and overall CPI inflation), we found that the predictions had similar or better accuracy compared to models that used variables that might influence agricultural productivity over the previous growing season, such as precipitation and temperature.
In our datasets, food insufficiency was highest in December, January, and February, typically dropping in March and April. In Wave 4 (2020-2021), March was an unusually severe month due to shocks to production in the previous growing season, and price-based models were slightly better at reacting to that difference than weather-based models, which may have been more reliant on seasonal variations. The price-based models were also more sensitive to the transition back to widespread food insufficiency at the end of the dry season.
One disadvantage of using machine learning compared to classical regression is that the former lack easily interpretable coefficients to assess the relative contributions of each variable. The Shapley Additive Values Framework was developed to help demystify modeling results and shows the effect each variable has on the final predicted value for each data point. Applying the Shapley framework to our fitted models suggested that prices in the past month and inflation in the previous year both had substantial, if mixed influences on predicted values. The most influential variable is the maize price in the previous month, but there is a mixed impact: low maize prices could be either a strong or a weak signal of future food insufficiency depending on the month in which the observation took place, while high maize prices tended to have a generally positive influence on predictions of insufficiency on positive predictions.
3. Recurrence of Food Insufficiency
Despite the weather and price volatility during the observation period, temporal stability, captured in a dummy variable representing the quarter when the observation occurred, was a highly influential factor in the model predictions. A large portion of the observed variation comes from seasonal swings in food availability. Much of the remainder is associated with the drought in southern Malawi, although spikes in flood-associated shocks to production also occur in the third survey round (Figure 2). These features of the agricultural sector in Malawi create a predictable spatio-temporal path for food insufficiency, making non-modeling prediction approaches based on historical occurrences of food insufficiency effective, although differences exist in which communities are flagged as food insufficient depending on the modeling approach being used.
Conclusion
While model accuracy may be higher in the absence of shocks like flooding and cyclones, historical observations of food insufficiency suggest that insufficiency in one year can arise from the consequences of a poor harvest in the previous year. Therefore, where recurring spatial patterns exist, decision makers could already have the information needed to act. Instead, modeling may offer benefits like increasing the utility of ongoing data collection for extrapolating to the rest of the country or providing visualizations of spatial patterns. The decreasing cost and increasing resolution of spatial data products could allow for detailed analyses to help inform policy-making, but only with frequent collection of training data. In resource-constrained environments, timely interventions based on informed priors may be preferable to gathering more information.
The Data and Policy open-access manuscript is available here.
Blog written by C. Leigh Anderson, Didier Alia, and Andrew Tomes.