How is Gro tackling things?
We've been at this since 2014, focusing intently on climate and food security problems and how to articulate and present the solution. In January of 2021 we raised our Series B and are now building to solve the problem at scale. Scale in terms of users; scale in terms of volume of data; scale in terms of enabling a platform to be concurrently expanded by a rapidly-growing organization.
With an ever-growing platform, we need to be cognizant of the true costs of the various options available to us (e.g. buy vs build, even in a strictly public=cloud context)
Rapid expansion of data and data sources which affects the data pipeline, indexing and serving
Increase in the number of users and queries against our platform
We’ve more than doubled the number of employees in 2021, which means more concurrent changes to the platform
We need to be creative about organizing a wide variety of data and data formats that don’t obviously fit together in a seamless way
Static vs dynamic (e.g. computed on the fly) data
Tabular, textual (e.g. prose), and raster (image) data
Historical vs future (modeled) data
Single origin vs ensemble data
Almost all of our code was originally written in Python. Python is a great language in terms of its ease-of-use and expressiveness but it’s not well-suited for high-performance computing. We won’t be moving away from Python at the edges (e.g. for some of our data scientists and rapid prototypers) but we need to shift to something more performant for the inner nodes and foundational layers.
Given the nature of the technical challenges at Gro (i.e. Big Data) and that we had the luxury of choosing a language after the initial end-to-end platform was built out, the choice for us was clear.
Plays nicely with Python both as the caller and the callee
Designed from the onset to be low-overhead and a viable competitor of C/C++ with modern features
Explicit data ownership concepts and lack of garbage collection yield remarkable performance
Compile-time problems are always preferable to run-time problems