Hina Gandhi, software engineering technical lead at Cisco, offered tips and techniques for paving the way to autonomous and efficient data pipelines that continuously adapt to changing workload and infrastructure dynamics in her Data Summit 2026 session, “Tune Spark for Itself: Reinforcement Learning for Smarter Optimization.”
The Annual Data Summit Conference was held in Boston on May 6-7, 2026, with a pre-conference workshop held on May 5th.
Existing optimization techniques leave gaps in pre-execution intelligence, she said.
Manual tuning requires deep domain expertise, is very time consuming, and is immediately interrupted when the underlying data patterns change.
Adaptive query execution begins after initial overhead and improper task scheduling have already occurred. The missing link, she explained, was pre-execution intelligence.
Gandhi introduced an approach that allows Spark to learn how to optimize itself, using reinforcement learning (RL), specifically Q-learning, to dynamically choose the most efficient partitioning strategy at runtime.
She considered how RL agents can observe key performance signals such as shuffle size, task duration, data skew, and executor utilization and iteratively adjust splitting decisions to minimize latency and resource costs.
The agent instantly recognizes the state and chooses actions based on its memory. The agent applies configuration and learns strictly from execution time.
The agent completes a continuous learning cycle by executing the job, measuring accurate performance, and updating its internal memory with mathematical reward signals.
“Infrastructure must learn from experience rather than relying on static rules,” Gandhi said.
Many of the Data Summit 2026 presentations can be reviewed here: https://www.dbta.com/datasummit/2026/presentations.aspx.
