At the time of writing, Ghana is the 27th most polluted country in the world and faces significant challenges due to air pollution. Recognizing the important role of air quality monitoring, many African countries, including Ghana, are deploying low-cost air quality sensors.
The West African Sensor Evaluation and Training Centre (Afri-SET) aims to address these challenges through technology. Afri-SET partners with air quality sensor manufacturers to provide critical evaluations tailored to the African context. Through sensor evaluation and informed decision support, Afri-SET is empowering governments and private society for effective air quality management.
December 6Number-8Number In 2023, nonprofit Tech to the Rescue, in collaboration with AWS, hosted the world's largest air quality hackathon, aimed at tackling one of the world's most pressing health and environmental challenges: air pollution. More than 170 technology teams built 33 solutions using the latest cloud, machine learning, and artificial intelligence technologies. The solution featured in this blog solved the Afri-SET challenge and was ranked among the top three winning solutions.
In this post, we present a solution that uses generative artificial intelligence (AI) to standardize air quality data from low-cost sensors in Africa. Specifically, we address the problem of low-cost sensor air quality data consolidation. The solution leverages the capabilities of generative AI, specifically large-scale language models (LLMs), to address the challenges posed by diverse sensor data and automatically generate Python functions based on different data formats. The fundamental objective is to build a manufacturer-agnostic database, leveraging the capabilities of generative AI to standardize sensor outputs, synchronize data, and facilitate precise corrections.
Current Issues
Afri-SET currently integrates data from numerous sources, with a customized approach for each sensor manufacturer. This manual synchronization process is resource-intensive due to different data formats, limiting the potential for extensive data orchestration. Although functional, the platform requires significant effort for data ingest, dealing with CSV and JSON files containing hundreds of thousands of rows from various manufacturers.
The goal is to automate data integration from various sensor manufacturers in Accra, Ghana, with scalability across West Africa. Despite the challenges, Afri-SET, with limited resources, aims to provide accurate data from low-cost sensors and envisions a comprehensive data management solution for stakeholders seeking sensor hosting on their platform. This endeavor is handicapped as the current focus is on cleaning the data, which takes away valuable skills from building ML models for sensor calibration. Additionally, they aim to report calibrated data from low-cost sensors, but to do so they need more than just specific pollutants.
The solution had the following requirements:
- Cloud Hosting – The solution must reside in the cloud to ensure scalability and accessibility.
- Automated Data Ingestion – Automated systems are essential to recognize and synchronize new (and invisible) diverse data formats with minimal human intervention.
- Format flexibility – The solution should accommodate both CSV and JSON input and be flexible with regards to formatting (proper column names, units of measure, nested structures, or malformed CSVs such as missing or extra columns)
- Golden Copy Preserved – It is essential to keep an intact copy of the data for reference and verification purposes.
- Cost-effective – For maximum cost efficiency, solutions should call LLM where necessary to generate reusable code rather than directly manipulating data.
The goal was to build a one-click solution that would take different data structures and formats (CSV and JSON) and automatically convert and consolidate them into a database with uniform headers, as shown in the following diagram, allowing data to be aggregated for further manufacturer-agnostic analysis.
Figure 1: Converting data from different data formats into a destination data format with a unified header
Solution overview
The proposed solution uses Anthropic's Claude 2.1 foundational model via Amazon Bedrock to generate Python code. This code converts input data into a unified data format. LLM excels at writing code and inferencing on text, but tends to perform poorly when working directly with time series data. The solution leverages LLM's inference and coding capabilities to create reusable extract, transform, and load (ETL) files that transform sensor data files that do not adhere to universal standards and store them together for downstream calibration and analysis. Additionally, it leverages LLM's inference capabilities to understand the meaning of labels in the context of air quality sensors, such as particulate matter (PM), relative humidity, and temperature.
The following diagram illustrates the conceptual architecture:
Figure 2: AWS reference architecture and workflow for data transformation using Amazon Bedrock
Solution Walkthrough
The solution reads raw data files (CSV and JSON files) from Amazon Simple Storage Service (Amazon S3) (step 1) and checks if the device type (or data format) has been seen before. If so, the solution retrieves and runs the previously generated Python code (step 2) and stores the converted data in S3 (step 10). The solution only calls LLM for new device data file types (for which the code has not yet been generated). This is done to optimize performance and minimize the cost of calling LLM. If no Python code is available for a particular device data, the solution notifies the operator to check for a new data format (steps 3 and 4). At this point, the operator checks the new data format and validates whether the new data format is for a new manufacturer (step 5). In addition, the solution checks whether the file is CSV or JSON. For CSV files, the data can be directly converted to a Pandas data frame by a Python function without calling LLM. For JSON files, the LLM is invoked to generate a Python function that creates a Pandas DataFrame from the JSON payload, taking into account the schema and nesting method (step 6).
It calls LLM to generate a Python function that manipulates the data using three different prompts (input strings).
- The first call (step 6) generates a Python function that converts the JSON file into a Pandas data frame. The manufacturer JSON files have different schemas. Some input data uses value type and value pairs for measurements. The latter format generates a data frame with one column of value type and one column of values. Such columns need to be pivoted.
- The second call (step 7) determines if the data needs to be pivoted and generates a Python function for pivoting if necessary. Another issue with the input data is that the same air quality measurement may have different names depending on the manufacturer. For example, “P1” and “PM1” For the same type of measurement.
- The third call (step 8) focuses on cleaning the data: it generates a Python function that converts the data frame into a common data format. The Python function might include steps to standardize column names for measurements of the same type, or remove columns.
All Python code generated by the LLM is stored in a repository (step 9) and can be used to process and convert daily raw device data files into a common format.
The data is then stored in Amazon S3 (step 10) and published to OpenAQ, making the calibrated air quality data available to other organizations.
The following screenshots show the proposed front-end for illustrative purposes only, as the solution is designed to integrate with Afri-SET’s existing back-end systems.
result
The proposed method minimizes the invocation of LLM, optimizing costs and resources. The solution invokes LLM only when a new data format is detected. The generated code is stored so that input data of the same format (seen previously) can reuse the code for data processing.
Data ingestion is protected by human involvement mechanisms, which are only performed when new data formats are discovered to avoid overtaxing scarce Afri-SET resources. Human involvement to validate each data transformation step is optional.
Automatic code generation reduces data engineering efforts from months to days. With this solution, Afri-SET can automatically generate Python code based on the format of the input data. The output data is converted into a standardized format and stored in Parquet format (a columnar, efficient storage format) in a single location on Amazon S3. If it helps, it can be further extended into a data lake platform that uses AWS Glue (a serverless data integration service for data preparation) and Amazon Athena (a serverless, interactive analytics service) to analyze and visualize the data. AWS Glue custom connectors make it easy to transfer data between Amazon S3 and other applications. Moreover, this is a no-code experience for Afri-SET's software engineers, making it easy to build data pipelines.
Conclusion
The solution facilitates data integration and helps scale air quality monitoring cost-effectively, providing data-driven and informed law enforcement, empowering communities, and encouraging innovation.
This effort to collect accurate data is an important step toward a cleaner, healthier environment. We believe AWS technology can help solve poor air quality through technical solutions similar to those described here. If you would like to prototype a similar solution, please apply to the AWS Health Equity initiative.
As always, AWS welcomes your feedback, so please leave your thoughts and questions in the comments section.
About the Author
Sandra Topic Sandra is the Environmental Equity Leader at AWS. In this role, she leverages her engineering background to find new ways to use technology to solve the world's “to-do list” and make a positive impact on society. Sandra's background includes social entrepreneurship and leading sustainability and AI efforts at technology companies.
Jang KyungShe is a PhD and a Senior Partner Solutions Architect at AWS, specializing in AI/ML. Her current interests are federated learning, distributed training, and generative AI. She holds 30+ patents, has co-authored 100+ journal/conference papers, and has won best paper awards at IEEE NetSoft 2016, IEEE ICC 2011, ONDM 2010, and IEEE GLOBECOM 2005.
Gabriel Verot Gabriel is a Senior Partner Solutions Architect for Industrial Manufacturing at AWS. He works with AWS partners to define, build, and evolve solutions around smart manufacturing, sustainability, and AI/ML. He also has expertise in industrial data platforms, predictive maintenance, and combining AI/ML with industrial workloads.
Venkatavaradhan (Venkath) Viswanathan is a Global Partner Solutions Architect at Amazon Web Services. Venkat is a technical strategy leader for Data, AI, ML, Generative AI and Advanced Analytics. Venkat is a Global SME at Databricks, helping AWS customers architect, build, secure and optimize their Databricks workloads on AWS.
