A major open source photo repository developed at North Carolina State University could be a significant step forward in unlocking the potential of artificial intelligence to solve agriculture's stubborn challenges.
The Ag Image Repository (AgIR), led by the U.S. Department of Agriculture's Agricultural Research Service and NCSU, continues to grow its collection of 1.5 million high-quality photographs and related data of plants collected at various stages of growth. It was launched nationally this fall on the computing cluster SCINet.
The team of researchers who developed the photo repository uses images to create “cutouts” (plants removed from the background), which are key to AI development. These include 16 cover crop species, 38 weed species, and a growing number of cash crops such as corn, soybeans, and cotton.
Alexander Allen, head of AgIR's systems software development team, said the repository is designed to support researchers who want to create AI-based solutions for agricultural stakeholders, such as farmers and plant breeders.
It grew out of research by Precision Sustainable Agriculture (PSA), a national network of scientists, engineers, farmers and others with a shared interest in helping farmers make data-driven business decisions in real time.
“Access to this data will be a game-changer for plant intelligence technologies around the world and will lead to an increase in precision agriculture design techniques, allowing farmers to get the most out of their fields while protecting them and nature from ecological damage,” Allen said.
AgIR could be particularly useful for those looking to develop agricultural tools and technologies that employ computer vision, a type of AI that allows machines to “see,” understand, and respond to the world around them.
Computer vision is widely used today for everything from driverless taxis stopping pedestrians to surgical robots assisting with appropriate cuts. Farmers are using computer vision, especially machines mounted on tractors to identify weeds and spray herbicides.
AgIR project leaders said achieving effective computer vision solutions in agriculture can be more difficult than in some other sectors. Fields are complex, varied environments that can be difficult to navigate. Subtle differences in a plant's appearance can have a significant impact on how you care for it.
“Even within a single plant species, there are genetic varieties with different visible traits and very different responses to environmental factors,” he added. “Not only that, but plants can look different if they're grown in a drier, hotter climate than if they're grown in a cooler climate, or if they're subjected to moisture stress, etc.”
Agriculture does not have the big, well-labeled image that other sectors take for granted. AgIR bridges that gap, allowing you to train models that can be maintained across farms, seasons, and applications.
To train AI models that are robust enough to account for variations in the field and the plants growing there, agricultural researchers need large numbers of carefully curated and annotated images depicting plants in different growing conditions and at different stages of growth, Allen explains.
Computational agronomist Matthew Kutugata believes AgIR is an important step forward in meeting that need.
“Agriculture lacks the large labeled image sets that other fields take for granted. AgIR fills that gap, allowing us to train models that can be maintained across farms, seasons, and applications,” said Kutsugata, who leads PSA's data engineering and computer vision efforts. “With open data and baselines, researchers, their students, small labs, and even growers have a clear starting point to build, test, and improve their tools without starting from scratch.”
Kutugata, Allen, and North Carolina agronomist Chris Leberg Horton see AgIR as a step toward realizing the full benefits of precision agriculture.
In precision agriculture, growers provide plants with exactly what they need, when and where, and in the amount they need. For example, instead of spraying the entire field against insects, farmers can spray only problem areas. This protects crops, limits chemical use, and protects the environment from overspraying.
Although smart devices are now available that apply most inputs variably, we have been particular about creating enough intelligence to tell the device what to do. Computer vision is the technology that makes this possible.
Precision agriculture has been talked about for more than 40 years, but “it's mostly been aspirational,” said Leberg Horton, professor in the Department of Crop and Soil Sciences, director of the NC Plant Science Initiative's Resilient Agricultural Systems Platform, and co-director of the PSA.
“Smart devices are now available that apply most inputs variably, but we are stuck creating enough intelligence to tell that device what to do,” he says. “Computer vision is the technology that makes that possible.”
To begin building his desired tool, Reberg Horton needed a stack of solid images of every plant he might encounter on his farm, including cash crops, cover crops, and weeds.
He and his team quickly realized that collecting images of individual plants in the field was tedious, time-consuming, and labor-intensive, and that labeling the images with the data needed to build artificial intelligence would be a challenge. They've built tools to streamline the process. This means robotic hardware that collects plant images that meet rigorous standards, and software that makes it easy to label images with the data needed to train reliable AI models.
To automate the photography, the team developed three robots that operate in “semi-field” environments at a USDA research facility in Beltsville, Maryland, Texas A&M University, and outside the North Carolina State Plant Science Building in Raleigh.
Each of these wheel-mounted benchbots is equipped with a camera that can take photos with the level of detail necessary to make them usable for scientific research.
“We're using a camera that can take incredibly detailed images of plants, something no one has ever attempted before,” Allen said.
Each Benchbot is programmed to move through a “field” of hundreds of potted plants arranged in rows, allowing a camera moving along an overhead trajectory to take a photo of each pot.
“For example, if you're imaging peas, you bring in a bunch of peas of the same type, plant them, and do at least three passes each week. This ultimately gives you a time series of the plants as they grow and develop throughout their lifespan,” Allen says.
While building the hardware, the team developed software that automates the process of cropping plants from images, performing color correction, and attaching detailed descriptions about each image.
While developing the hardware and software solution, PSA team members realized that the database they built could also be useful to those looking to develop AI-driven tools and technology for farmers. Plant breeders can use computer vision to perform high-throughput phenotyping, the automated, rapid and accurate measurement of organismal traits at scale.
“We're excited to see this being used in applications we never imagined, by teams we've never heard of, to impact problems for farmers.”The goal is to make machine learning algorithms less tedious in the plant breeding process by tackling tasks like counting fruit, scoring disease resistance, and estimating yield without harvesting.
By adding the image repository to USDA SCINet, the team hopes to have an even bigger impact, allowing those who want to develop AI-powered tools for agriculture to have what Kutugata calls “a proven baseline rather than starting from scratch.”
