Cloud giants cash in on DSML platforms while GenAI adds fuel to the fire: Gartner

(Valery Brodzinski/Shutterstock)

Gartner's latest Magic Quadrant report found that the emergence of generative AI is driving enterprises to adopt unified data science and machine learning (DSML) platforms that can handle traditional ML as well as new GenAI use cases, with cloud giants rapidly gaining market share.

Gartner's definition of a DSML platform is an integrated set of libraries designed to help data scientists complete all aspects of the data science lifecycle through a low-code or code-based approach. Running on the web or installed on a PC, the platform not only helps clean and prepare data, but also enables data scientists to analyze and understand it, build ML and AI models, and deploy them in production.

While traditional ML focuses on structured data, such as tables of numbers in a database, newer AI approaches such as GenAI are based on unstructured data, such as text and images, and today's DSML platforms can accommodate both types of data.

“Supported machine learning techniques range from classical regression and decision trees to more complex deep learning, reinforcement learning and GenAI,” wrote Gartner analysts Afraz Jafri, Ola Popa, Peter Krensky, Jim Hare, Raghubender Bhati, Mariam Hassanloo and Tong Zhang. “Models built using these techniques can be used for tasks within business processes such as credit scoring, churn prediction, predictive maintenance, recommendations and image classification.”

GenAI has recently driven a significant increase in DSML platform adoption: Gartner noted that 53% of respondents in a recent survey said demand for GenAI will “drive significant growth in DSML platform spending beyond 2024.” However, building GenAI products is very difficult, and the number of GenAI projects far outstrips actual GenAI deployments.

“The surge in demand for AI solutions, including GenAI, is reaching a peak,” Gartner analysts wrote, “but assembling the raw materials of data, models, code and infrastructure into reliable, scalable products has never been more complex.”

(greenbutterfly/Shutterstock)

The good news for GenAI enthusiasts (i.e. all of us) is that the DSML Platform is gearing up to incorporate GenAI into the AI ​​and ML space. The DSML Platform has developed an established process for building all kinds of ML and AI products, and new GenAI workloads can benefit from that advancement.

But there's a bit of a gap between what organizations want from GenAI and how DSML tools can deliver that in terms of the personas who use these tools. Gartner says that's because GenAI is bringing more talent into data science from the business side of the company, who typically don't have the advanced skills of a full-fledged data scientist.

The rapid pace of AI development is changing the role of humans orchestrating AI: Gartner predicts that 50% of data analysts will be retrained as data scientists by 2027. Meanwhile, today's data scientists will become tomorrow's AI engineers.

But according to Gartner, there's good news here: capabilities like AutoML (where the software makes decisions about the features, weights and ML models to use) are becoming commonplace in DSML platforms.

Additionally, these AutoML capabilities are complemented by GenAI-based features such as coding assistants and natural language querying, further lowering the barrier to entry and helping democratize data science.

As GenAI drives demand for AI and draws more people into the AI ​​business, Gartner said the DSML platform will play a key role.

“The challenge for data science and AI leaders is how to manage and govern the activities of distributed DSML teams and maximize efficiency through collaboration with centralized resources,” Gartner analysts said.

Cloud giants

(Source: Gartner)

Gartner, in its latest Magic Quadrant report, said that cloud giants are making major gains in the data science and machine learning platforms market, but smaller software companies are expected to continue to innovate and thrive, driven by the momentum of GenAI and the need for collaboration across teams.

Amazon Web Services, Google Cloud and Microsoft Azure all ranked in the Leaders quadrant of Gartner's latest Magic Quadrant for Data Science and Machine Learning Platforms, joining Databricks, Dataiku, DataRobot, SAS and Altair.

The report authors note that hyperscaler offerings are gaining increasing traction in the DSML platform market “due to the availability of compute, data, and infrastructure required for DSML development.”

“But there is still room for other areas to develop, especially when it comes to enabling collaboration across teams, a key pillar of DSML and GenAI development,” the authors continue. “Adopting DSML technology to more companies, and across all areas of the enterprise, is an opportunity that vendors and end-users alike can seize. The fundamental use case of data science for insight-driven decision-making should not be lost in the noise of GenAI. The DSML platform provides a perfect home for converging advanced analytics and AI development.”

It's worth noting how quickly the DSML leaderboard has changed in just a few years: Of the eight vendors currently ranked in Gartner's Leaders quadrant, only SAS and RapidMiner (now owned by Altair) were ranked in 2019. Even in 2021, none of the cloud giants are ranked in the Leaders quadrant, although Databricks, Dataiku, and SAS are ranked.

Vendors that were considered leaders in the 2021 DSML Magic Quadrant have fallen back in terms of ability to execute and completeness of vision, including KNIME, TIBCO, Mathworks, and IBM. Alteryx, which was in the Challenger quadrant, is now in the Niche Player quadrant along with MathWorks and KNIME.

Meanwhile, Cloudera has moved from the Niche Player quadrant in 2021 to the Visionary quadrant in 2024, alongside H20.ai and Domino Data Lab. Meanwhile, Alibaba Cloud has been promoted from the Niche Player category to the Challenger quadrant, a quadrant that IBM is now also in. Anaconda has been in the Niche Player quadrant since 2019.

Related Products:

How broad is your database data ecosystem? Gartner Survey

A “plethora” of innovation discovered in data science and ML platforms

The “big bang” of data science and ML tools