In a world of rising trade tensions and climate volatility, global agriculture is reliant on a forecasting model that is dangerously out of date. While other sectors profit from data delivered by the nanosecond, the agricultural commodities sector still depends on data delivered monthly.
This data – from the USDA World Agricultural Supply and Demand Estimates (WASDE) – is comprehensive and robust, yet the method by which estimates are being generated is inadvertently jeopardizing the stability of key global crops such as corn and soy.
The markets urgently need parallel forecasting models delivering consistent and comprehensive forecasts in a transparent way in real time.
In this paper, we assess the current status of agricultural commodities forecasting, and reveal how new models could:
enable traders to predict price shifts more accurately
improve the relationships between the stages of the food supply chain
reduce the risk of domino effects in agricultural markets causing food shortages and crises.
Agricultural production, and the price of key agricultural commodities such as corn, soybeans, and wheat , has always been volatile. As the world’s population surges towards 7.5 billion, that volatility is felt more acutely than ever.
The number of mouths to feed continues to rise, yet the amount of arable land on which to grow the food to feed them stays the same. In years when there are bumper crops, supply continues to meet demand, but crop yields are becoming more unpredictable as a result of climate change and over-farming. Crop failures, or even relatively small reductions in projected yields in key crops such as corn, can destabilize world food markets, knocking supply and demand out of sync and triggering food shortages.
Effective forecasting is more crucial than ever in the fight to ensure global food security. Yet the benchmark for the production and price of agricultural commodities has remained largely unchanged for nearly 40 years.
The US Department of Agriculture’s (USDA) World Agricultural Supply and Demand Estimates (WASDE), published around the 10th of every month, provides the most accurate and comprehensive forecasts of global supply and demand for key crops and livestock.
Focusing on the fundamentals
Agricultural supply and demand are affected by the weather, atmospheric change, and issues related to the land itself. This makes forecasting in agricultural commodities heavily reliant on ‘fundamental’ analysis rather than the more accessible and consistent ‘technical’ analysis based on price and volume.
WASDE’s granular fundamental analysis of farming conditions in the US and in other top producer countries from Argentina to Russia gives the market a benchmark from which to measure all other analysis. Historically, the WASDE reports have functioned as the most realistic and reliable forecast of supply and demand available to the market.
The WASDE reports, especially in key global crops such as corn and soybeans, are the data points that traders, producers and major buyers use to determine their actions in the market. Policy-makers working to ensure food security within the world’s most vulnerable countries also rely on WASDE data.
The WASDE reports are closely guarded until publication, which means that decision-makers in global food markets must wait a whole month before receiving the next forecast. As a result, the WASDE reports can cause serious volatility in agricultural supply chains.
A new forecasting model delivering data in real time would have a significant positive impact globally. It would:
enable traders to predict price shifts more accurately
improve the relationships between the stages of the food supply chain
reduce the risk of the domino effect in agricultural markets causing food shortages and crises.
WHAT IS WASDE?
The US government has produced analysis on global agricultural supply and demand since 1893, with reports appearing in various forms and with varying regularity through to the 1960s, when they became regular ‘circulars’. In 1973 the first Agricultural Supply and Demand Estimates were published, analyzing solely the US market. The first World Agricultural Supply and Demand Estimates appeared in October 1980.
Today WASDE has changed little from its original 1980 form; it supplies monthly forecasts for the supply and demand of agricultural commodities in the US and globally. The exact formula and inputs used by the USDA to produce WASDE are a closely guarded secret. Farm production data gets assembled at the local level and aggregated through a complex series of analysts and agencies. The Office of the USDA Chief Economist performs the final analysis of the data in an overnight lockup (no one is allowed to enter, leave, or use electronic devices) before it is collated, formatted, and released.
How is WASDE constructed?
Several agencies within USDA are responsible for preparing crop statistics. The National Agricultural Statistics Service (NASS) forecasts US crop production based on data collected from farm operations and field observations.
Forecasts for each crop season begin with a winter wheat and rye seedings report in early January followed by a March report that gives a first look at what farmers intend to plant. This is followed in late June by a report of the acreage actually planted. Monthly yield and production forecasts begin in May for winter wheat, in July for spring wheat and other small grains, and in August for other spring-planted crops, concluding with estimates of actual production at the end of the various harvesting seasons. NASS also conducts quarterly surveys of grain and soybeans stored on and off farms.
The World Agricultural Outlook Board (WAOB) coordinates an interagency process that prepares monthly forecasts of supply and demand for major crops, both for the US and the world, and follows a balance-sheet approach to account for supplies and utilization.
The major components of the supply and demand balance sheet are beginning stocks, production, domestic use, trade, and end-of-season carryout stocks. Whereas forecasts of US crop production and estimates of US stocks on hand are independently prepared by NASS, US and foreign supply and demand forecasts are developed jointly by several USDA agencies.
Understanding USDA Crop Forecasts, US Department of Agriculture, 1999
THE UNINTENDED CONSEQUENCES OF WASDE
The WASDE remains a hugely important and helpful tool for the global agricultural commodities sector. Over the last century it has enabled better financing of the farming industry, powered large-scale commercialization, and helped to avert food crises.
But problems with the analogue WASDE model are increasingly evident. WASDE forecasts have remained remarkably accurate over the past 40 years, but even marginal errors in forecasts, especially in key crops, can trigger huge volatility.
And even when accuracy is not an issue, the monthly schedule of the reports can create serious problems. A WASDE report could predict a failure in US corn yields perfectly, but the opportunity to mitigate the fallout of such a failure could be lost in the weeks before publication.
The USDA’s WASDE reports have also given rise to an industry of analysts and consultants who have created a fragmented and privatized ecosystem of forecasting models and tools. These aim to give companies and investors an edge in hedging and profiting against the monthly WASDE numbers. Private analysis tends to smooth price paths by allowing some traders to anticipate changes in USDA opinion. But private forecasters are routinely wrong-footed by USDA, and brutal volatility ensues as the market reassesses its collective price projections in the minutes after the report.
The impact of WASDE beyond the trading floor
The impact of the WASDE reports goes well beyond the US borders. WASDE data is not only used by traders and investors; public and private organizations in some of the world’s most vulnerable countries rely on WASDE numbers to plan their procurement of key crops such as wheat and corn. Even a small variation or delay in yield forecasts can have a significant impact on food security.
The Middle East and North Africa are the world’s largest net importers of grains. They depend on exports from around the world, including North America, Europe and Central Asia. ****The civil unrest that erupted in Tunisia in late 2010 and Egypt in early 2011 coincided with a rise in global food prices, with the World Bank deeming food prices at the time to be at a “dangerous level”.
Political instability spread across the region, with protests in Egypt and Tunisia focusing on the price of food staples such as bread.
According to the World Bank, most Arab countries import 50% of what they consume. Between 2008 and 2010 local food prices in Egypt rose by over 20%, which increased pressure on a government already under strain, fuelling public anger and protest and contributing to the uprising in 2010. The steep rise in the food price index created widespread food insecurity for middle-class urban populations of poorer countries, which in turn resulted in widespread political instability.
The WASDE process failed to warn of this possible development. Had the supply-and-demand risk been identified earlier, in maize specifically, global donor bodies and recipient states could have managed the situation better, and averted the price spikes that led to public unrest.
Food crises in South Sudan, Northern Nigeria and in Yemen have sparked similar, more localized scenarios.
THE USDA’s WASDE URGENTLY NEEDS REFORM AND INNOVATION
The way to improve forecasting in agriculture is not to change the WASDE model – as we have shown, the WASDE has remained resilient to changes in market fundamentals and continues to deliver the most comprehensive and trusted public forecasting data for agricultural commodities.
But it does have a number of weaknesses, two of which are particularly notable. First, its rigid monthly schedule, which creates information deficits between reports; and second, the monthly estimation process itself, which is based on subjective inputs, meaning the approach taken cannot be reproduced.
Markets would greatly benefit from the establishment of parallel forecasting models that offer dynamic, real-time, and accurate assessments of the future supply and demand of key crops.
Several initiatives have made progress towards meeting this challenge. Yet so far, the market has lacked a systematic, reproducible, quantitative approach to modelling – one that specifies its forecasts in full and has transparent methodology, allowing users to understand and scrutinize the entire modelling process.
THE GRO MODEL
Using US corn and Argentine soybean yields, we have been working on a new set of models. In 2017 we decided to share the structure and output of our cutting-edge yield model, which marries agronomic process expertise with machine learning. We wanted to help demonstrate how this could be done, and how some of the drawbacks of the WASDE approach could be overcome. Our aim was to develop an approach to yield modelling that was relevant to all commodities, not only corn and soybeans, in a way that would add value to a much wider audience.
From May to November 2017, Gro’s website published the weekly forecasts for all to comment on. Subscribers had access to daily updates of yield forecasts at county level and access to all the signals in the model. In the forthcoming US 2018 corn and soybean season, we anticipate supplying them with corn and soybean yields modelled on the same basis.
We initially made a US corn-yield model as ‘proof of concept’ for a more general method that could deliver analysis of a broad array of agricultural commodities globally. Our data product “Gro” encompasses global agricultural information on all crops.
We also created a companion trading system, which gave conclusive evidence of our model’s forecasting prowess. Our trading method follows the most simple rules we could find for a systematic model: generate a “buy” or “sell” signal based on the level of our corn-yield model relative to the trade estimate when it’s available, or to last month’s USDA yield when the trade estimate is not compiled (May, June, and July.) If our model has a lower yield than the trade estimate or old USDA number, we bought corn, because a lower crop yield should lead to higher prices. If our yield estimate was higher than the trade or USDA number, we sold it. All trades were executed in liquid nearby futures at the closing price the day before the WASDE report was released. Trades were reversed and closed out at the daily closing price after the report.
Using our public forecasts in this simple trading strategy shows how effective they are. The strategy made money in 71% of the 2017 trades, ending the forecast season in November up $0.1775/bushel (or +$887.50/contract). In a back test, it profited in 61% of 76 trades from May 2006 to November of 2016 and earned a cumulative $2.79/bushel (or +$13,950/contract). In contrast, the well-known trade estimate numbers, if traded the same way through 2016, lost a cumulative $0.72/bushel (or - $3,600/contract.)
THE IMPLICATIONS OF THE GRO MODEL
The advantage of the Gro model to commodity traders is clear. The ability to gain a more accurate forecast of key crop prices ahead of the WASDE numbers gives traders multiple opportunities to profit. But the Gro model goes beyond this.
It offers the opportunity to democratize agricultural knowledge. Gro pulls trillions of data points onto one platform and curates them to deliver the most comprehensive assessment available of supply and demand in key agricultural commodities. Such analysis is currently only available to a handful of large traders. Gro offers the opportunity to level the playing field.
Benefits to the global agricultural sector
Better data equals better forecasting, which in turn leads to more reliable production. This is particularly relevant to the global agriculture sector, where local and regional planting decisions are inextricably linked with global crop yields, and where very small incremental improvements in forecasting can have huge knock-on effects.
With more and better data, Argentina, for example, could have managed its ongoing drought-related problems with soybean and corn production more effectively. The US ‘s concentration on corn and soybean production, which accounts for 80% of agricultural output, makes it equally vulnerable to volatility.
An agricultural sector that is so dependent on external and unpredictable factors simply cannot function optimally without harnessing the vast data and predictive analytics now available.
Greater global food security
Finally, a more accessible, dynamic, and timely forecasting model could significantly improve food security. Keeping 7.5 billion people nourished is clearly a significant challenge, but better data on potential shocks or spikes in supply and demand of key commodities such as corn and wheat would allow international development and humanitarian agencies as well as nation states to make earlier and better decisions on the procurement and supply of food. A supply- or demand-side issue could escalate into a crisis before it was even flagged by the next WASDE report. A new dynamic forecasting model could help avert such a crisis entirely, by issuing a warning two weeks earlier.
Children are alarmingly vulnerable to malnutrition. Small periods of food crisis have particularly devastating effects on young children, who can suffer permanent physical and mental stunting after a brief period of fasting. Inadequate nutrition leaves children highly susceptible to infection, which can lead to further negative outcomes. Earlier warning of an impending food crisis would have great value. Speeding intervention by days can boost a society’s prospects for decades.
A new, accessible forecasting model
WASDE remains a critical tool in global agricultural markets, but its totemic position within the market also means that it can create huge volatility. The answer is to introduce a secondary, open, and accessible forecasting model that can offer insurance and insight to all players in the market.
This paper has presented one model that has proven over a short timeframe to deliver better crop-yield forecasts. This model is constantly evolving, and we expect its accuracy to increase still further as we add, analyze, and interpret an ever-wider range of data sources.
The data revolution could transform the agricultural commodities sector, and the sector is ripe for positive disruption. We have a real opportunity right now to help actors across the global agricultural industry, while also improving food security around the world.
What Information Do We Collect?
The information we gather enables us to personalize, improve and continue to operate the Services. We collect the following types of information from our users.
IP Address Information and Other Information Collected Automatically:
· We automatically receive and record information from your web browser when you interact with the Services, including your IP address and cookie information. This information is used for fighting spam/malware and also to facilitate collection of data concerning your interaction with the Services (e.g., what links you have clicked on).
· Generally, the Services automatically collect usage information, such as the number and frequency of visitors to the Site. We may use this data in aggregate form, that is, as a statistical measure, but not in a manner that would identify you personally. This type of aggregate data enables us and third parties authorized by us to figure out how often individuals use parts of the Services so that we can analyze and improve them.
Information Collected Using Cookies:
· Most browsers have an option for turning off the cookie feature, which will prevent your browser from accepting new cookies, as well as (depending on the sophistication of your browser software) allowing you to decide on acceptance of each new cookie in a variety of ways.
We collect statistical information about how users collectively use the Services (“Aggregate Information”). Some of this information may be derived from Personal Information. This statistical information is not Personal Information and cannot be tied back to you or your web browser.
How, and With Whom, Is My Information Shared?
IP Address Information:
Information You Elect to Share:
We share Aggregate Information with our partners, service providers and other persons with whom we conduct business. We share this type of statistical data so that our partners can understand how and how often people use our Services and their services or websites, which facilitates improving both their services and how our Services interface with them. In addition, these third parties may share with us non-private, aggregated or otherwise non Personal Information about you that they have independently developed or acquired.
Information Shared with Our Agents:
We employ and contract with people and other entities that perform certain tasks on our behalf and who are under our control (our “Agents”). We may need to share Personal Information with our Agents in order to provide products or services to you. Unless we tell you differently, our Agents do not have any right to use Personal Information or other information we share with them beyond what is necessary to assist us. You hereby consent to our sharing of Personal Information with our Agents.
Information Disclosed Pursuant to Business Transfers:
In some cases, we may choose to buy or sell assets. In these types of transactions, user information is typically one of the transferred business assets. Moreover, if we, or substantially all of our assets, were acquired, or if we go out of business or enter bankruptcy, user information would be one of the assets that is transferred or acquired by a third party. You acknowledge that such transfers may occur, and that any acquirer of us or our assets may continue to use your Personal Information as set forth in this policy.
Information Disclosed for Our Protection and the Protection of Others:
Information We Share With Your Consent:
Except as set forth above, you will be notified when your Personal Information may be shared with third parties, and will be able to prevent the sharing of this information.
Is Information About Me Secure?
We store all of our information, including your IP address information, using industry-standard techniques. We do not guarantee or warrant that such techniques will prevent unauthorized access to information about you that we store, Personal Information or otherwise.
What Information of Mine Can I Access?
You can access and delete cookies through your web browser settings.
California Privacy Rights: Under California Civil Code sections 1798.83-1798.84, California residents are entitled to ask us for a notice identifying the categories of personal customer information which we share with our affiliates and/or third parties for marketing purposes, and providing contact information for such affiliates and/or third parties. If you are a California resident and would like a copy of this notice, please submit a written request to the following address: 1156 6th Ave, 7th Floor, New York, NY 10036
What If I Have Questions or Concerns?
If you have any questions or concerns regarding privacy using the Services, please send us a detailed message to email@example.com. We will make every effort to resolve your concerns.
Effective Date: March 11, 2014
b. You shall not (directly or indirectly):i. take any action that imposes or may impose (as determined by us in our sole discretion) an unreasonable or disproportionately large load on our (or our third party providers’) infrastructure;ii. interfere or attempt to interfere with the proper working of the Services or any activities conducted on the Services;iii. bypass, circumvent or attempt to bypass or circumvent any measures we may use to prevent or restrict access to the Services (or other accounts, computer systems or networks connected to the Services);iv. use manual or automated software, devices, or other processes to “crawl” or “spider” any page of the Site;
v. harvest or scrape any Content from the Services;
vi. otherwise take any action in violation of our guidelines and policies;vii. decipher, decompile, disassemble, reverse engineer or otherwise attempt to derive any source code or underlying ideas or algorithms of any part of the Services (including without limitation any application), except to the limited extent applicable laws specifically prohibit such restriction;viii. modify, translate, or otherwise create derivative works of any part of the Services; orix. copy, rent, lease, distribute, or otherwise transfer any of the rights that you receive hereunder.c. We also reserve the right to access, read, preserve, and disclose any information as we reasonably believe is necessary to:i. satisfy any applicable law, regulation, legal process or governmental request;ii. enforce these Terms of Service, including investigation of potential violations hereof;
iii. detect, prevent, or otherwise address fraud, security or technical issues;
iv. respond to user support requests; or
v. protect the rights, property or safety of us, our users and the public.4. Third Party Services. The Services may permit you to link to other websites, services or resources on the Internet, and other websites, services or resources may contain links to the Services. When you access third party resources on the Internet, you do so at your own risk. These other resources are not under our control, and you acknowledge that we are not responsible or liable for the content, functions, accuracy, legality, appropriateness or any other aspect of such websites or resources. The inclusion of any such link does not imply our endorsement or any association between us and their operators. You further acknowledge and agree that we shall not be responsible or liable, directly or indirectly, for any damage or loss caused or alleged to be caused by or in connection with the use of or reliance on any such content, goods or services available on or through any such website or resource.5. Termination. We may terminate your access to all or any part of the Services at any time, with or without cause, with or without notice, effective immediately. All provisions of these Terms of Service which by their nature should survive termination shall survive termination, including, without limitation, ownership provisions, warranty disclaimers, indemnity and limitations of liability.6. Warranty Disclaimer.a. You release us from all liability for you having acquired or not acquired Content through the Services. We make no representations concerning any Content contained in or accessed through the Services, and we will not be responsible or liable for the accuracy, copyright compliance, or legality of material or Content contained in or accessed through the Services.b. THE SERVICES AND CONTENT ARE PROVIDED “AS IS”, “AS AVAILABLE” AND WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, AND ANY WARRANTIES IMPLIED BY ANY COURSE OF PERFORMANCE OR USAGE OF TRADE, ALL OF WHICH ARE EXPRESSLY DISCLAIMED. WE, AND OUR DIRECTORS, EMPLOYEES, AGENTS, SUPPLIERS, PARTNERS AND CONTENT PROVIDERS DO NOT WARRANT THAT: (I) THE SERVICES WILL BE SECURE OR AVAILABLE AT ANY PARTICULAR TIME OR LOCATION; (II) ANY DEFECTS OR ERRORS WILL BE CORRECTED; (III) ANY CONTENT AVAILABLE AT OR THROUGH THE SERVICES IS FREE OF VIRUSES OR OTHER HARMFUL COMPONENTS; OR (IV) THE RESULTS OF USING THE SERVICES WILL MEET YOUR REQUIREMENTS.7. Limitation of Liability. IN NO EVENT SHALL WE, NOR OUR DIRECTORS, EMPLOYEES, AGENTS, PARTNERS, SUPPLIERS OR CONTENT PROVIDERS, BE LIABLE UNDER CONTRACT, TORT, STRICT LIABILITY, NEGLIGENCE OR ANY OTHER LEGAL OR EQUITABLE THEORY WITH RESPECT TO THE SERVICES FOR ANY (I) LOST PROFITS, DATA LOSS, COST OF PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES, OR SPECIAL, INDIRECT, INCIDENTAL, PUNITIVE, COMPENSATORY OR CONSEQUENTIAL DAMAGES OF ANY KIND WHATSOEVER, SUBSTITUTE GOODS OR SERVICES (HOWEVER ARISING), (II) BUGS, VIRUSES, TROJAN HORSES, OR THE LIKE (REGARDLESS OF THE SOURCE OF ORIGINATION), OR (III) DIRECT DAMAGES IN EXCESS OF $50.00.8. Governing Law and Jurisdiction. These Terms of Service shall be governed by and construed in accordance with the laws of the State of New York, including its conflicts of law rules, and the United States of America. You agree that any dispute arising from or relating to the subject matter of these Terms of Service shall be governed by the exclusive jurisdiction and venue of the state and Federal courts of New York County, New York.9. Miscellaneous.a. Modification. We reserve the right, in our sole discretion, to modify or replace any of these Terms of Service, or change, suspend, or discontinue the Services at any time. Your continued use of the Services following notification of any changes to these Terms of Service constitutes acceptance of those changes.b. Entire Agreement and Severability. These Terms of Service are the entire agreement between you and us with respect to the Services, including use of the Site, and supersede all prior or contemporaneous communications and proposals (whether oral, written or electronic) between you and us with respect to the Services. If any provision of these Terms of Service is found to be unenforceable or invalid, that provision will be limited or eliminated to the minimum extent necessary so that these Terms of Service will otherwise remain in full force and effect and enforceable. The failure of either party to exercise in any respect any right provided for herein shall not be deemed a waiver of any further rights hereunderc. Force Majeure. We shall not be liable for any failure to perform our obligations hereunder where such failure results from any cause beyond our reasonable control, including, without limitation, mechanical, electronic or communications failure or degradation.d. Assignment. These Terms of Service are personal to you, and are not assignable, transferable or sublicensable by you except with our prior written consent. We may assign, transfer or delegate any of our rights and obligations hereunder without consent.e. Agency. No agency, partnership, joint venture, or employment relationship is created as a result of these Terms of Service and neither party has any authority of any kind to bind the other in any respect.f. Notices. Unless otherwise specified in these Term of Service, all notices under these Terms of Service will be in writing and will be deemed to have been duly given when received, if personally delivered or sent by certified or registered mail, return receipt requested; when receipt is electronically confirmed, if transmitted by facsimile or e-mail; or the day after it is sent, if sent for next day delivery by recognized overnight delivery service. Electronic notices should be sent to firstname.lastname@example.org. No Waiver. Our failure to enforce any part of these Terms of Service shall not constitute a waiver of our right to later enforce that or any other part of these Terms of Service. Waiver of compliance in any particular instance does not mean that we will waive compliance in the future. In order for any waiver of compliance with these Terms of Service to be binding, we must provide you with written notice of such waiver through one of our authorized representatives.h. Headings. The section and paragraph headings in these Terms of Service are for convenience only and shall not affect their interpretation.Contact. You may contact us at the following address: 1156 6th Ave, 7th Floor, New York, NY 10036.