Artificial intelligence and deep machine learning will enable UAS to scale and enhance its impact across many sectors.
In the world, large-scale explosions of innovation can occur due to random alignments of favorable conditions in technology, business, and society. The explosive growth of innovative applications for unmanned aerial vehicles (UAVs) over the past decade is rivaled only by the World Wide Web and the iPhone.
Remotely controlled flight has existed since 1935, when the Royal Air Force launched the Queen Bee aircraft. Drones then evolved to a more advanced level, but were still limited to military use. Perhaps his two turning points in turning drones into true consumer products were in 2010, when Parrot used his iPhone app to fly his AR drone at CES, and in 2013. It happened when DJI put a camera on its Phantom drone.
The ease of navigation, image capture capabilities, and miniaturization of ARM processors have provided fertile ground for AI applications in all fields. This powerful combination has allowed drones to penetrate various fields, including his ISR (intelligence, surveillance, and reconnaissance) and targeting, cinema, agriculture, logistics, engineering, and disaster response. iPhones, GoPro cameras, GPS, and AI were the flashpoints of technology. The number of problems awaiting solutions for UAVs is only limited by the imagination.
Today's drones come pre-packaged with GPS sensing and navigation capabilities, video capture, command and control applications, as well as several interfaces that allow for the implementation of AI and special purpose programs. This combination makes the UAS a “system.”
The benefits of AI
Computers have enabled massive automation and expansion of information processing. On the other hand, AI has moved computation to a much higher level, allowing for large-scale automation of inference, or more precisely solution discovery and parameter optimization. These two operations allowed the machine to learn.
For example, an AI thermostat can “learn” optimal temperature settings that minimize the number of times a person enters and exits a home or office to heat or cool it. To do this, machine learning algorithms collect many data points, such as the room temperature and the actions a person takes each time they set the thermostat. This provides valuable feedback to the algorithm regarding appropriate setpoints considering ambient temperature, time of day, etc. All machine learning algorithms follow the same principles, but use hundreds or thousands of “thermostats”, each learning an obscure parameter within the algorithm. The cumulative learning from these units will help you recognize cats in images, parallel park trailer trucks, translate text from Chinese to Arabic without prior knowledge of vocabulary or grammar, and help you with social networks. Machine intelligence will be created, such as identifying terrorist cells.
Currently, the most successful practical applications of drones leverage AI for image recognition and image stitching. Although this is far from the full promise of AI, it is helping automate and scale a huge number of applications at minimal cost and is having a significant impact. Drones fly inside European cathedrals and architectural heritage sites to build comprehensive 3D models. Drones are counting sheep in Israel, but surprisingly, it's not happening. Civil engineers use drones to continuously scan large structures such as bridges, dams, and oil rigs to detect structural defects before they become serious problems. And, of course, the military has been using drones for decades to gather field intelligence. Drone photography is becoming so popular that it's virtually impossible to find an area that hasn't been penetrated.
Conditions are important
For example, a drone scanning the interior of a cathedral enjoys many luxuries rarely available in other fields: it can easily return to its charging station whenever it needs power; it can use broadband to upload high-resolution video to the cloud for processing on giant server farms; it is protected from the elements, theft, and attack, and there is little cybersecurity risk. Without these luxuries, the AI could not do its job.
The challenge of operating applications in hostile environments such as war zones or forest fires requires significant amounts of support infrastructure and larger, more powerful drones. This challenge presents a great opportunity for AI to provide “operational support” capabilities. While less sexy than object identification, navigation, and context determination, these support capabilities optimize and pre-process data to mitigate the lack of readily available powerful computing and high-speed data transfer.
For example, rather than uploading high-resolution images to a server or processing them on the drone, an AI algorithm is used to sample the image and capture only relevant features, such as edges, the position of each eye relative to the nose, etc. These features can then be uploaded to the server with much lower bandwidth and much less power. Another powerful approach, often used in movies, is to shoot several high-resolution still images and a lower-resolution video, and an AI algorithm uses the two to build a high-resolution video. Once the high-resolution video is uploaded to the server, advanced AI can use the virtually infinite computing power. For example, a drone with FHD video captures an image containing 1920×1080 pixels (about 2 million). The image produces 240×135 (or 32,400 pixels), one-eighth the FHD resolution, but still provides a recognizable image, and reduces the power and required bandwidth required to capture, store, and transmit the image by a factor of 64. Many AI applications only use 64×32 images, so the size is reduced by a factor of 1024.
Advances in computing and battery technology, as well as increased availability of bandwidth, will make these support functions less relevant for many applications. However, the drive toward miniaturization and the continued expansion of drones into new application areas in harsh environments will always require additional support capabilities.
From individual drones to drone systems
The model of using a small number of drones to capture images and other data and uploading them to a server to run advanced AI tends to quickly become obsolete. The problem with this model is that it is not operational. It is neither robust nor scalable. If a drone malfunctions, gets lost, or comes under physical or cyber attack, the entire operation can fail. The cost of manually replacing a drone on the battlefield or on an offshore oil rig is operationally prohibitive. The future model is one that uses hundreds or thousands of drones as a system rather than individual drones.
Transforming “individuals” into systems is the basis of many disciplines, including physics, biology, social science, and technology. The key difference is that a system is greater than the sum of its parts; it is a universal property of sustainable systems. The science of managing large numbers of individuals as systems has been applied with great success to marketing, finance, web search, language translation, environmental remediation, and AI itself. Transforming an army of drones into a system is a natural application of this science.
To give a public example, in 2012, Ars Electronica Futurelab created the first drone light show, programming 49 drones to adopt swarming behavior inspired by birds and bees, resulting in artistic light patterns in the sky. Since then, hundreds of light shows have been held using thousands of drones. Most of these shows use highly coordinated drones, such as the one that showed a man walking in space during the 2020 New Year celebrations in Shanghai.
As impressive as it is, the light show is simply a rendering of a 3D image, with each drone occupying a pixel location. Technology-related challenges should not be underestimated, but they remain more in the realm of personal coordination than system management.
AI exemplifies the power of harnessing many individuals as a system. In 1956, Oliver Selfridge and Marvin Minsky founded what we know today as AI with the concept of daemons, or background computer programs, that act as systems to solve complex problems. Since then, there has been a proliferation of distributed technology systems based on the principle of agents (daemons) working together to address complex problems at scale. Today, the “Hadoop technology”, widely used to mine vast amounts of text data such as web logs, is based on agents, where each agent takes a subset of the data and divides the task into thousands of subtasks that run in parallel. Agent-based modeling is widely used in biology, for example, to learn about ecosystem sustainability. In these models, thousands of agents are programmed with specific behaviors, such as deer that eat vegetation at a certain rate, trees that grow at their own rate, predators that eat deer, etc. These agents then interact randomly, producing many plausible outcomes.
deep learning
As a result, deep learning, which is at the heart of many of the most influential applications of AI in drones, involves hundreds or thousands of very basic mathematical functions working together systematically to learn from experience. It gets its power from a neural network that adapts parameters while These neural networks are constructed in layers, with each layer feeding a condensed version of the input to the next layer, allowing the network to dive deeper and deeper into the features that really matter. This is why it was named deep learning.
Drones are no different than nodes in a neural network, but much more sophisticated. Today's technology allows small, simple drones to be manufactured at very low cost. Instead of a few powerful drones with a single point of failure, thousands of simple drones can operate as a system. Drone systems are much more effective than individuals. A small number of drones may be shot down on the battlefield. Drone dust, on the other hand, is much more difficult to prevent.
Operating drones as a multi-drone system is key to enabling the next leap. Fortunately, there is strong precedent that suggests success is likely. In 2006, Amazon launched Amazon Web Services (AWS), giving birth to cloud computing, forever transforming IT operations and making data centers obsolete. At the core of AWS are the many supporting processes that bring thousands of computers together in the cloud to work as one. Before AWS, most AI applications were limited to a few powerful computers. Now, AI with virtually unlimited power is readily available at an astonishingly low cost.
Many companies, including Amazon, are currently working hard to make drones a reality. There is an urgent need to use drones on a large scale for package delivery logistics. They address many technical and regulatory challenges, such as air traffic control. They have the know-how, tools and, crucially, the imagination to build the support infrastructure for drones to work as a system.
Let there be light
Drone Light Show is a pioneer in today's drone systemization. These are creating market conditions that encourage miniaturization, cost reduction, interoperability and, importantly, the development of supporting processes. In fact, companies like Intel, EHang, and HighGreat have developed a vast amount of technology and capabilities that can quickly scale up the production of drone light shows. Latvian software engineering company SPH Engineering provides open source and premium software that allows anyone to create their own drone light show.
It is only a matter of time before these light show functions are repurposed for other activities. The availability of open source software and standard drone interfaces provides very favorable conditions for implementing AI applications. While light shows involve mapping drones to pixels, these more generalized applications focus on mapping drones to sub-functions. For example, some drones track movements on the battlefield while others count soldiers. While many of these applications address the task at hand, such as mapping a war zone, much of the AI will focus on dividing up and distributing drones to perform tasks among thousands of drones and stitching together the parts to make a whole.
