No Time to Waste: A New Model for Agricultural Forecasting

EXECUTIVE SUMMARY

In a world of rising trade tensions and climate volatility, global agriculture is reliant on a forecasting model that is dangerously out of date. While other sectors profit from data delivered by the nanosecond, the agricultural commodities sector still depends on data delivered monthly.

This data – from the USDA World Agricultural Supply and Demand Estimates (WASDE) – is comprehensive and robust, yet the method by which estimates are being generated is inadvertently jeopardizing the stability of key global crops such as corn and soy.

The markets urgently need parallel forecasting models delivering consistent and comprehensive forecasts in a transparent way in real time.

In this paper, we assess the current status of agricultural commodities forecasting, and reveal how new models could:

enable traders to predict price shifts more accurately

improve the relationships between the stages of the food supply chain

reduce the risk of domino effects in agricultural markets causing food shortages and crises.

INTRODUCTION

Agricultural production, and the price of key agricultural commodities such as corn, soybeans, and wheat , has always been volatile. As the world’s population surges towards 7.5 billion, that volatility is felt more acutely than ever.

The number of mouths to feed continues to rise, yet the amount of arable land on which to grow the food to feed them stays the same. In years when there are bumper crops, supply continues to meet demand, but crop yields are becoming more unpredictable as a result of climate change and over-farming. Crop failures, or even relatively small reductions in projected yields in key crops such as corn, can destabilize world food markets, knocking supply and demand out of sync and triggering food shortages.

Effective forecasting is more crucial than ever in the fight to ensure global food security. Yet the benchmark for the production and price of agricultural commodities has remained largely unchanged for nearly 40 years.

The US Department of Agriculture’s (USDA) World Agricultural Supply and Demand Estimates (WASDE), published around the 10th of every month, provides the most accurate and comprehensive forecasts of global supply and demand for key crops and livestock.

The WASDE reports, especially in key global crops such as corn and soybeans, are the data points that traders, producers and major buyers use to determine their actions in the market. Policy-makers working to ensure food security within the world’s most vulnerable countries also rely on WASDE data.

The WASDE reports are closely guarded until publication, which means that decision-makers in global food markets must wait a whole month before receiving the next forecast. As a result, the WASDE reports can cause serious volatility in agricultural supply chains.

A new forecasting model delivering data in real time would have a significant positive impact globally. It would:

enable traders to predict price shifts more accurately

improve the relationships between the stages of the food supply chain

reduce the risk of the domino effect in agricultural markets causing food shortages and crises.

WHAT IS WASDE?

The US government has produced analysis on global agricultural supply and demand since 1893, with reports appearing in various forms and with varying regularity through to the 1960s, when they became regular ‘circulars’. In 1973 the first Agricultural Supply and Demand Estimates were published, analyzing solely the US market. The first World Agricultural Supply and Demand Estimates appeared in October 1980.

Today WASDE has changed little from its original 1980 form; it supplies monthly forecasts for the supply and demand of agricultural commodities in the US and globally. The exact formula and inputs used by the USDA to produce WASDE are a closely guarded secret. Farm production data gets assembled at the local level and aggregated through a complex series of analysts and agencies. The Office of the USDA Chief Economist performs the final analysis of the data in an overnight lockup (no one is allowed to enter, leave, or use electronic devices) before it is collated, formatted, and released.  

THE UNINTENDED CONSEQUENCES OF WASDE

The WASDE remains a hugely important and helpful tool for the global agricultural commodities sector. Over the last century it has enabled better financing of the farming industry, powered large-scale commercialization, and helped to avert food crises.

But problems with the analogue WASDE model are increasingly evident. WASDE forecasts have remained remarkably accurate over the past 40 years, but even marginal errors in forecasts, especially in key crops, can trigger huge volatility.

And even when accuracy is not an issue, the monthly schedule of the reports can create serious problems. A WASDE report could predict a failure in US corn yields perfectly, but the opportunity to mitigate the fallout of such a failure could be lost in the weeks before publication.

The USDA’s WASDE reports have also given rise to an industry of analysts and consultants who have created a fragmented and privatized ecosystem of forecasting models and tools. These aim to give companies and investors an edge in hedging and profiting against the monthly WASDE numbers. Private analysis tends to smooth price paths by allowing some traders to anticipate changes in USDA opinion. But private forecasters are routinely wrong-footed by USDA, and brutal volatility ensues as the market reassesses its collective price projections in the minutes after the report.

THE USDA’s WASDE URGENTLY NEEDS REFORM AND INNOVATION

The way to improve forecasting in agriculture is not to change the WASDE model – as we have shown, the WASDE has remained resilient to changes in market fundamentals and continues to deliver the most comprehensive and trusted public forecasting data for agricultural commodities.

But it does have a number of weaknesses, two of which are particularly notable. First, its rigid monthly schedule, which creates information deficits between reports; and second, the monthly estimation process itself, which is based on subjective inputs, meaning the approach taken cannot be reproduced.

Markets would greatly benefit from the establishment of parallel forecasting models that offer dynamic, real-time, and accurate assessments of the future supply and demand of key crops.

Several initiatives have made progress towards meeting this challenge. Yet so far, the market has lacked a systematic, reproducible, quantitative approach to modelling – one that specifies its forecasts in full and has transparent methodology, allowing users to understand and scrutinize the entire modelling process.

THE GRO MODEL

Using US corn and Argentine soybean yields, we have been working on a new set of models. In 2017 we decided to share the structure and output of our cutting-edge yield model, which marries agronomic process expertise with machine learning. We wanted to help demonstrate how this could be done, and how some of the drawbacks of the WASDE approach could be overcome. Our aim was to develop an approach to yield modelling that was relevant to all commodities, not only corn and soybeans, in a way that would add value to a much wider audience.

From May to November 2017, Gro’s website published the weekly forecasts for all to comment on. Subscribers had access to daily updates of yield forecasts at county level and access to all the signals in the model. In the forthcoming US 2018 corn and soybean season, we anticipate supplying them with corn and soybean yields modelled on the same basis.

We initially made a US corn-yield model as ‘proof of concept’ for a more general method that could deliver analysis of a broad array of agricultural commodities globally. Our data product “Gro” encompasses global agricultural information on all crops.

We also created a companion trading system, which gave conclusive evidence of our model’s forecasting prowess. Our trading method follows the most simple rules we could find for a systematic model: generate a “buy” or “sell” signal based on the level of our corn-yield model relative to the trade estimate when it’s available, or to last month’s USDA yield when the trade estimate is not compiled (May, June, and July.) If our model has a lower yield than the trade estimate or old USDA number, we bought corn, because a lower crop yield should lead to higher prices. If our yield estimate was higher than the trade or USDA number, we sold it. All trades were executed in liquid nearby futures at the closing price the day before the WASDE report was released. Trades were reversed and closed out at the daily closing price after the report.

Using our public forecasts in this simple trading strategy shows how effective they are. The strategy made money in 71% of the 2017 trades, ending the forecast season in November up $0.1775/bushel (or +$887.50/contract). In a back test, it profited in 61% of 76 trades from May 2006 to November of 2016 and earned a cumulative $2.79/bushel (or +$13,950/contract). In contrast, the well-known trade estimate numbers, if traded the same way through 2016, lost a cumulative $0.72/bushel (or - $3,600/contract.)

THE IMPLICATIONS OF THE GRO MODEL

The advantage of the Gro model to commodity traders is clear. The ability to gain a more accurate forecast of key crop prices ahead of the WASDE numbers gives traders multiple opportunities to profit. But the Gro model goes beyond this.

It offers the opportunity to democratize agricultural knowledge. Gro pulls trillions of data points onto one platform and curates them to deliver the most comprehensive assessment available of supply and demand in key agricultural commodities. Such analysis is currently only available to a handful of large traders. Gro offers the opportunity to level the playing field.

Benefits to the global agricultural sector  

Better data equals better forecasting, which in turn leads to more reliable production. This is particularly relevant to the global agriculture sector, where local and regional planting decisions are inextricably linked with global crop yields, and where very small incremental improvements in forecasting can have huge knock-on effects.

With more and better data, Argentina, for example, could have managed its ongoing drought-related problems with soybean and corn production more effectively. The US ‘s concentration on corn and soybean production, which accounts for 80% of agricultural output, makes it equally vulnerable to volatility.

An agricultural sector that is so dependent on external and unpredictable factors simply cannot function optimally without harnessing the vast data and predictive analytics now available.

Greater global food security

Finally, a more accessible, dynamic, and timely forecasting model could significantly improve food security. Keeping 7.5 billion people nourished is clearly a significant challenge, but better data on potential shocks or spikes in supply and demand of key commodities such as corn and wheat would allow international development and humanitarian agencies as well as nation states to make earlier and better decisions on the procurement and supply of food. A supply- or demand-side issue could escalate into a crisis before it was even flagged by the next WASDE report. A new dynamic forecasting model could help avert such a crisis entirely, by issuing a warning two weeks earlier.

Children are alarmingly vulnerable to malnutrition. Small periods of food crisis have particularly devastating effects on young children, who can suffer permanent physical and mental stunting after a brief period of fasting. Inadequate nutrition leaves children highly susceptible to infection, which can lead to further negative outcomes. Earlier warning of an impending food crisis would have great value. Speeding intervention by days can boost a society’s prospects for decades.

A new, accessible forecasting model

WASDE remains a critical tool in global agricultural markets, but its totemic position within the market also means that it can create huge volatility. The answer is to introduce a secondary, open, and accessible forecasting model that can offer insurance and insight to all players in the market.

This paper has presented one model that has proven over a short timeframe to deliver better crop-yield forecasts. This model is constantly evolving, and we expect its accuracy to increase still further as we add, analyze, and interpret an ever-wider range of data sources.

The data revolution could transform the agricultural commodities sector, and the sector is ripe for positive disruption. We have a real opportunity right now to help actors across the global agricultural industry, while also improving food security around the world.