VANCOUVER — What did Apache Airflow, an open source project for building machine learning pipelines, reveal? A discussion with three Amazon Web Services technologists on “The New Stack Makers.” As you can see, the experience just got easier.
Apache Airflow is a Python-based platform for programmatically creating, scheduling, and monitoring workflows. Suitable for machine learning to build pipelines, manage data, train models, and deploy.
Airflow is common enough for the entire machine learning pipeline. Airflow takes data and performs Extract, Transform, and Load (ETL). Tag your data, run training, deploy and test your model, and send it to production.
On the On the Road episode of Makers, recorded at The Linux Foundation’s Open Source Summit North America, guests all working with the AWS Managed Service for Airflow team reflected on Apache Airflow’s efforts to improve their overall experience. rice field.
Dennis Ferruzzi, Software Developer at AWS, is an Airflow contributor working on project API-49, which updates Airflow’s logging and metrics backend to the OpenTelemetry standard. The API allows for more granular metrics, giving you better visibility into your Airflow environment.
Niko Oliveira, Senior Software Development Engineer at AWS, is a committer/maintainer of Apache Airflow. He spends a lot of time reviewing, approving and merging pull his requests. Recent projects included writing and implementing his AIP-51 (Airflow Improvement Proposal) which changed and updated his Executor interface in Airflow. This makes Airflow a more pluggable architecture and makes it easier for users to build and create their own Airflow Executor.
Raphaël Vandon, Senior Software Engineer at AWS, is a contributor to Apache Airflow, working to improve Airflow’s performance and leveraging the asynchronous capabilities of the AWS Operators, part of Airflow that enable seamless interaction with AWS. I’m here.
“The great thing about Airflow is that it’s very popular. It’s very easy,” says Oliveira. “First, it’s Python. Python is easy to learn and master. And second, we have this ecosystem of operators. Contributing to these operators that wrap to
“That blueprint exists for everyone.”
Operators are like general building blocks. According to Ferruzzi, each operator performs her one specific task.
“You just connect them in different ways,” he said. “So, for example, we have an operator that writes data. [Amazon Simple Storage Service]. And then there’s an operator that sends the data to a SQL server, etc. And basically, the community develops and contributes to these operators, so eventually users basically say the task I want to do is pull data from here. So I want to use that operator to send the data to another location.
“So we’re going to look at the Google Cloud operators, for example, and find one that fits what you want to do there. It’s cross-cloud. You can interact with so many different services and cloud providers, and it’s just growing. I think we’re at 2,500 contributors now, and it’s just like people find a need and deliver it, and now that block, that blueprint, is for everyone. To do.”
Bandon said Airflow 2.6 has an alpha version of the sensor. A sensor is an operator that waits for something to happen. There is also a notifier that is placed at the end of the workflow. They act on the success (or failure) of the workflow.
“It just makes things easier for users,” Bandon said.
