Modernize your Apache Spark pipeline with Snowpark Connect
Upgrading your data platform shouldn’t mean rebuilding everything from scratch. Snowpark Connect provides a practical on-ramp for teams by bringing their existing Spark-based pipelines to Snowflake’s modern managed infrastructure without a complete rewrite. Engineers can move away from older and expensive Spark clusters to a platform built for today’s data scale with native governance, flexible compute, and seamless access to Snowflake’s complete ecosystem. It’s a modernization that addresses teams where they are and eliminates the operational overhead of the past.
Since the release of Snowpark Connect last year, Snowflake has been hard at work on a number of updates, including:
- Spark Scala and Java clients For Scala 2.12/2.13 and Java 11/17
snowpark-submitCLI for production deployment with zero code changes - bronze layer file treatment Uses permissive mode, complex data types, schema evolution, and parallel reading of large compressed files
- Unified observability Helps discover, diagnose, and alert users to Spark jobs with complete details (status, duration, resources, queries, logs) from Jupyter, Airflow, or external sources.
For the past decade, the definition of business existed outside the pipeline. Metrics were defined in BI tools, features were defined in ML stores, and every team had their own version of the truth. In the semantic view, that is changing. Data engineers can now add meaning directly to pipelines. Incorporate this into your dbt workflow using the Snowflake Semantic View dbt Package. The team defines the semantic layer directly in the dbt model file using standard DDL syntax. CoCo helps you create that definition. Running dbt build materializes or updates Snowflake’s semantic view to keep it in step with the rest of the pipeline. Horizon Context takes that even further, making these definitions available to all AI agents, BI tools, and applications that automatically access your data.
We’ve known for years that you can’t just hire your way out of systemic problems. After all, it’s the same with the use of AI. When data engineers use AI to deliver solutions to weak legacy platforms, technical debt is accelerated rather than eliminated. As a result, pipelines break, infrastructure becomes difficult to maintain, and data products cannot keep up with the business. In this new AI era, there is a danger that the speed of creation will outpace the quality of the underlying foundation.
Snowflake offers both an agent coding experience purpose-built for data engineering, aligned with the managed platform that AI workloads demand. Whether you’re adopting an open lakehouse architecture, migrating Spark workloads, building a large-scale ML inference pipeline, or launching an entirely new data platform, Snowflake gives all data engineers the tools to move faster, ship with confidence, and spend less time grappling with infrastructure. The agent age of data engineering has arrived.
Get started by downloading our free ebook, Build Pipelines for AI: An Essential Guide to Smarter Data Engineering, and read more about the exciting releases and announcements from Snowflake Summit 2026.
- Based on ADE bench results compared to Claude Code. ↩
- Note: Efficiency scores are based on internal testing using ADE-bench, a framework created by dbt to evaluate AI-on-agent agents in real-world analytics and data engineering tasks. ↩
- Based on customer production use cases and proof-of-concept exercises comparing Snowpark speed and cost from November 2022 to May 2026. Actual speed and cost improvements will vary depending on specific customer environment and workload patterns. ↩
