Data Practitioners in the Age of AI – MIT Technology Review

The rise of generative AI and the rapid adoption and democratization of AI across industries over the last decade has highlighted the importance of data. Effective management of data has become crucial for businesses in this era, making data practitioners such as data engineers, analytics engineers, and ML engineers key players in the data and AI revolution.

Organizations that don't leverage their data will fall behind competitors that do and miss opportunities to unlock new value for themselves and their customers. As data volume and complexity grows, so do the challenges, forcing organizations to adopt new data tools and infrastructure, resulting in changes to the roles and duties of technology staff.

Data practitioners are among the people whose roles are changing the most as organizations expand their responsibilities. Rather than working in siloed data teams, data engineers are now developing platforms and tools designed to increase data visibility and transparency for employees across the organization, including analytics engineers, data scientists, data analysts, machine learning engineers, and business stakeholders.

Through a series of interviews with data professionals, this report explores key shifts in data engineering, the evolving skill sets required of data professionals, data infrastructure and tooling options to support AI, and the data challenges and opportunities emerging alongside generative AI. Key findings from the report include:

• The fundamental importance of data is creating new demands on data practitioners. As the rise of AI makes the business importance of data clearer than ever, data practitioners are faced with new data challenges, increasing data complexity, evolving team structures, emerging tools and technologies, and establishing new organizational imperatives.

• Data practitioners get closer to the business and the business gets closer to the data. Pressure to create value from data is driving executives to invest heavily in data-related capabilities. Data professionals are being asked to broaden their business knowledge, engage more deeply with business units, and support the use of data within their organizations, while functional teams are realizing that they need their own in-house data expertise to leverage data.

• Data and AI strategies have become a vital part of business strategies. As data becomes increasingly the key differentiator for every business, business leaders must invest in their data and AI strategies, including making key decisions around data team organizational structure, data platform and architecture, and data governance.

• Data practitioners will decide how generative AI will be deployed within their enterprises. Key considerations in deploying generative AI – producing high-quality results, avoiding bias and illusions, establishing governance, designing data workflows, and ensuring regulatory compliance – are the province of data practitioners, who have a major influence on how this powerful technology is leveraged.

Download the full report.

This content was produced by Insights, MIT Technology Review's custom content division; it was not written by MIT Technology Review's editorial staff.