No Time to Waste: A New Model for Agricultural Forecasting

10 May 2018

EXECUTIVE SUMMARY

In a world of rising trade tensions and climate volatility, global agriculture is reliant on a forecasting model that is dangerously out of date. While other sectors profit from data delivered by the nanosecond, the agricultural commodities sector still depends on data delivered monthly.

This data – from the USDA World Agricultural Supply and Demand Estimates (WASDE) – is comprehensive and robust, yet the method by which estimates are being generated is inadvertently jeopardizing the stability of key global crops such as corn and soy.

The markets urgently need parallel forecasting models delivering consistent and comprehensive forecasts in a transparent way in real time.

In this paper, we assess the current status of agricultural commodities forecasting, and reveal how new models could:

  1. enable traders to predict price shifts more accurately

  2. improve the relationships between the stages of the food supply chain

  3. reduce the risk of domino effects in agricultural markets causing food shortages and crises.

 

INTRODUCTION

Agricultural production, and the price of key agricultural commodities such as corn, soybeans, and wheat , has always been volatile. As the world’s population surges towards 7.5 billion, that volatility is felt more acutely than ever.

The number of mouths to feed continues to rise, yet the amount of arable land on which to grow the food to feed them stays the same. In years when there are bumper crops, supply continues to meet demand, but crop yields are becoming more unpredictable as a result of climate change and over-farming. Crop failures, or even relatively small reductions in projected yields in key crops such as corn, can destabilize world food markets, knocking supply and demand out of sync and triggering food shortages.

Effective forecasting is more crucial than ever in the fight to ensure global food security. Yet the benchmark for the production and price of agricultural commodities has remained largely unchanged for nearly 40 years.

The US Department of Agriculture’s (USDA) World Agricultural Supply and Demand Estimates (WASDE), published around the 10th of every month, provides the most accurate and comprehensive forecasts of global supply and demand for key crops and livestock.

Focusing on the fundamentals

Agricultural supply and demand are affected by the weather, atmospheric change, and issues related to the land itself. This makes forecasting in agricultural commodities heavily reliant on ‘fundamental’ analysis rather than the more accessible and consistent ‘technical’ analysis based on price and volume. 

WASDE’s granular fundamental analysis of farming conditions in the US and in other top producer countries from Argentina to Russia gives the market a benchmark from which to measure all other analysis. Historically, the WASDE reports have functioned as the most realistic and reliable forecast of supply and demand available to the market.  

 

The WASDE reports, especially in key global crops such as corn and soybeans, are the data points that traders, producers and major buyers use to determine their actions in the market. Policy-makers working to ensure food security within the world’s most vulnerable countries also rely on WASDE data.

The WASDE reports are closely guarded until publication, which means that decision-makers in global food markets must wait a whole month before receiving the next forecast. As a result, the WASDE reports can cause serious volatility in agricultural supply chains.

A new forecasting model delivering data in real time would have a significant positive impact globally. It would:

  • enable traders to predict price shifts more accurately

  • improve the relationships between the stages of the food supply chain

  • reduce the risk of the domino effect in agricultural markets causing food shortages and crises.

 

WHAT IS WASDE?

The US government has produced analysis on global agricultural supply and demand since 1893, with reports appearing in various forms and with varying regularity through to the 1960s, when they became regular ‘circulars’. In 1973 the first Agricultural Supply and Demand Estimates were published, analyzing solely the US market. The first World Agricultural Supply and Demand Estimates appeared in October 1980.

Today WASDE has changed little from its original 1980 form; it supplies monthly forecasts for the supply and demand of agricultural commodities in the US and globally. The exact formula and inputs used by the USDA to produce WASDE are a closely guarded secret. Farm production data gets assembled at the local level and aggregated through a complex series of analysts and agencies. The Office of the USDA Chief Economist performs the final analysis of the data in an overnight lockup (no one is allowed to enter, leave, or use electronic devices) before it is collated, formatted, and released.  

How is WASDE constructed?

Several agencies within USDA are responsible for preparing crop statistics. The National Agricultural Statistics Service (NASS) forecasts US crop production based on data collected from farm operations and field observations. 

Forecasts for each crop season begin with a winter wheat and rye seedings report in early January followed by a March report that gives a first look at what farmers intend to plant. This is followed in late June by a report of the acreage actually planted. Monthly yield and production forecasts begin in May for winter wheat, in July for spring wheat and other small grains, and in August for other spring-planted crops, concluding with estimates of actual production at the end of the various harvesting seasons. NASS also conducts quarterly surveys of grain and soybeans stored on and off farms. 

The World Agricultural Outlook Board (WAOB) coordinates an interagency process that prepares monthly forecasts of supply and demand for major crops, both for the US and the world, and follows a balance-sheet approach to account for supplies and utilization. 

The major components of the supply and demand balance sheet are beginning stocks, production, domestic use, trade, and end-of-season carryout stocks. Whereas forecasts of US crop production and estimates of US stocks on hand are independently prepared by NASS, US and foreign supply and demand forecasts are developed jointly by several USDA agencies. 

Understanding USDA Crop Forecasts, US Department of Agriculture, 1999



THE UNINTENDED CONSEQUENCES OF WASDE

The WASDE remains a hugely important and helpful tool for the global agricultural commodities sector. Over the last century it has enabled better financing of the farming industry, powered large-scale commercialization, and helped to avert food crises.

But problems with the analogue WASDE model are increasingly evident. WASDE forecasts have remained remarkably accurate over the past 40 years, but even marginal errors in forecasts, especially in key crops, can trigger huge volatility.

And even when accuracy is not an issue, the monthly schedule of the reports can create serious problems. A WASDE report could predict a failure in US corn yields perfectly, but the opportunity to mitigate the fallout of such a failure could be lost in the weeks before publication.

The USDA’s WASDE reports have also given rise to an industry of analysts and consultants who have created a fragmented and privatized ecosystem of forecasting models and tools. These aim to give companies and investors an edge in hedging and profiting against the monthly WASDE numbers. Private analysis tends to smooth price paths by allowing some traders to anticipate changes in USDA opinion. But private forecasters are routinely wrong-footed by USDA, and brutal volatility ensues as the market reassesses its collective price projections in the minutes after the report.

The impact of WASDE beyond the trading floor

The impact of the WASDE reports goes well beyond the US borders. WASDE data is not only used by traders and investors; public and private organizations in some of the world’s most vulnerable countries rely on WASDE numbers to plan their procurement of key crops such as wheat and corn. Even a small variation or delay in yield forecasts can have a significant impact on food security.  

The Middle East and North Africa are the world’s largest net importers of grains. They depend on exports from around the world, including North America, Europe and Central Asia. ****The civil unrest that erupted in Tunisia in late 2010 and Egypt in early 2011 coincided with a rise in global food prices, with the World Bank deeming food prices at the time to be at a “dangerous level”. 

Political instability spread across the region, with protests in Egypt and Tunisia focusing on the price of food staples such as bread. 

According to the World Bank, most Arab countries import 50% of what they consume. Between 2008 and 2010 local food prices in Egypt rose by over 20%, which increased pressure on a government already under strain, fuelling public anger and protest and contributing to the uprising in 2010. The steep rise in the food price index created widespread food insecurity for middle-class urban populations of poorer countries, which in turn resulted in widespread political instability. 

The WASDE process failed to warn of this possible development. Had the supply-and-demand risk been identified earlier, in maize specifically, global donor bodies and recipient states could have managed the situation better, and averted the price spikes that led to public unrest. 

Food crises in South Sudan, Northern Nigeria and in Yemen have sparked similar, more localized scenarios.



THE USDA’s WASDE URGENTLY NEEDS REFORM AND INNOVATION

The way to improve forecasting in agriculture is not to change the WASDE model – as we have shown, the WASDE has remained resilient to changes in market fundamentals and continues to deliver the most comprehensive and trusted public forecasting data for agricultural commodities.

But it does have a number of weaknesses, two of which are particularly notable. First, its rigid monthly schedule, which creates information deficits between reports; and second, the monthly estimation process itself, which is based on subjective inputs, meaning the approach taken cannot be reproduced.

Markets would greatly benefit from the establishment of parallel forecasting models that offer dynamic, real-time, and accurate assessments of the future supply and demand of key crops.

Several initiatives have made progress towards meeting this challenge. Yet so far, the market has lacked a systematic, reproducible, quantitative approach to modelling – one that specifies its forecasts in full and has transparent methodology, allowing users to understand and scrutinize the entire modelling process.

 

THE GRO MODEL

Using US corn and Argentine soybean yields, we have been working on a new set of models. In 2017 we decided to share the structure and output of our cutting-edge yield model, which marries agronomic process expertise with machine learning. We wanted to help demonstrate how this could be done, and how some of the drawbacks of the WASDE approach could be overcome. Our aim was to develop an approach to yield modelling that was relevant to all commodities, not only corn and soybeans, in a way that would add value to a much wider audience.

From May to November 2017, Gro’s website published the weekly forecasts for all to comment on. Subscribers had access to daily updates of yield forecasts at county level and access to all the signals in the model. In the forthcoming US 2018 corn and soybean season, we anticipate supplying them with corn and soybean yields modelled on the same basis.

We initially made a US corn-yield model as ‘proof of concept’ for a more general method that could deliver analysis of a broad array of agricultural commodities globally. Our data product “Gro” encompasses global agricultural information on all crops.

We also created a companion trading system, which gave conclusive evidence of our model’s forecasting prowess. Our trading method follows the most simple rules we could find for a systematic model: generate a “buy” or “sell” signal based on the level of our corn-yield model relative to the trade estimate when it’s available, or to last month’s USDA yield when the trade estimate is not compiled (May, June, and July.) If our model has a lower yield than the trade estimate or old USDA number, we bought corn, because a lower crop yield should lead to higher prices. If our yield estimate was higher than the trade or USDA number, we sold it. All trades were executed in liquid nearby futures at the closing price the day before the WASDE report was released. Trades were reversed and closed out at the daily closing price after the report.

Using our public forecasts in this simple trading strategy shows how effective they are. The strategy made money in 71% of the 2017 trades, ending the forecast season in November up $0.1775/bushel (or +$887.50/contract). In a back test, it profited in 61% of 76 trades from May 2006 to November of 2016 and earned a cumulative $2.79/bushel (or +$13,950/contract). In contrast, the well-known trade estimate numbers, if traded the same way through 2016, lost a cumulative $0.72/bushel (or - $3,600/contract.)

 

THE IMPLICATIONS OF THE GRO MODEL

The advantage of the Gro model to commodity traders is clear. The ability to gain a more accurate forecast of key crop prices ahead of the WASDE numbers gives traders multiple opportunities to profit. But the Gro model goes beyond this.

It offers the opportunity to democratize agricultural knowledge. Gro pulls trillions of data points onto one platform and curates them to deliver the most comprehensive assessment available of supply and demand in key agricultural commodities. Such analysis is currently only available to a handful of large traders. Gro offers the opportunity to level the playing field.

 

Benefits to the global agricultural sector  

Better data equals better forecasting, which in turn leads to more reliable production. This is particularly relevant to the global agriculture sector, where local and regional planting decisions are inextricably linked with global crop yields, and where very small incremental improvements in forecasting can have huge knock-on effects.

With more and better data, Argentina, for example, could have managed its ongoing drought-related problems with soybean and corn production more effectively. The US ‘s concentration on corn and soybean production, which accounts for 80% of agricultural output, makes it equally vulnerable to volatility.

An agricultural sector that is so dependent on external and unpredictable factors simply cannot function optimally without harnessing the vast data and predictive analytics now available.

 

Greater global food security

Finally, a more accessible, dynamic, and timely forecasting model could significantly improve food security. Keeping 7.5 billion people nourished is clearly a significant challenge, but better data on potential shocks or spikes in supply and demand of key commodities such as corn and wheat would allow international development and humanitarian agencies as well as nation states to make earlier and better decisions on the procurement and supply of food. A supply- or demand-side issue could escalate into a crisis before it was even flagged by the next WASDE report. A new dynamic forecasting model could help avert such a crisis entirely, by issuing a warning two weeks earlier.

Children are alarmingly vulnerable to malnutrition. Small periods of food crisis have particularly devastating effects on young children, who can suffer permanent physical and mental stunting after a brief period of fasting. Inadequate nutrition leaves children highly susceptible to infection, which can lead to further negative outcomes. Earlier warning of an impending food crisis would have great value. Speeding intervention by days can boost a society’s prospects for decades.

 

A new, accessible forecasting model

WASDE remains a critical tool in global agricultural markets, but its totemic position within the market also means that it can create huge volatility. The answer is to introduce a secondary, open, and accessible forecasting model that can offer insurance and insight to all players in the market.

This paper has presented one model that has proven over a short timeframe to deliver better crop-yield forecasts. This model is constantly evolving, and we expect its accuracy to increase still further as we add, analyze, and interpret an ever-wider range of data sources.

The data revolution could transform the agricultural commodities sector, and the sector is ripe for positive disruption. We have a real opportunity right now to help actors across the global agricultural industry, while also improving food security around the world.

Contact sales