The 2022 State of Workflow Orchestration report shows 43% of data scientists, engineers and developers use an orchestrator to handle over half of their recurring tasks

Prefect Technologies, Inc., the company that produces the open-source data workflow automation platform Prefect, announced the release of a new report by leading data and AI consulting firm Gradient Flow titled the “2022 State of Workflow Orchestration.” Sponsored by Prefect, the survey report reveals a growing demand for workflow orchestration solutions, which is being driven by ongoing company initiatives to modernize their data infrastructures and enhance datasets for machine learning (ML) and AI models. Data professionals at these organizations are looking to orchestrators to help write, schedule, monitor, and manage their data and ML pipelines.

Marketing Technology News: Mediaocean and Boostr Partner to Accelerate Deal Workflow

Usage of orchestration

For the report, Gradient Flow surveyed 581 respondents from a variety of industries, with roles including data scientists and analysts, data and ML engineers, software engineers and developers, and DevOps engineers. The responses indicated strong adoption of workflow orchestration and dataflow automation tools, with 43% of respondents stating that they currently use an orchestrator to handle over half of their recurring tasks. Notably, among the data scientists and analysts surveyed, 52% reported using an orchestrator for up to a quarter of recurring tasks. Moreover, 91% of the data and ML engineers said they use orchestrators for more than a quarter of their tasks.

When asked about their primary use cases for workflow orchestration, 29% of respondents all cited data science, making it the most popular application of workflow orchestration. The use cases shifted when surveying data and ML engineers, given their focus on moving and transforming data. For these respondents, data movement (21%), MLOps (16%), and data transformation (14%) were their top use cases for orchestration.

Ben Lorica, Founder and Principal of Gradient Flow, said, “More organizations now want to enable analytics and AI to benefit their businesses, creating demand for not only data talent but also foundational data technologies. This includes everything from data integration and DataOps to data governance and platforms—as well as orchestration. Organizations need orchestrators to make data available for downstream applications including data science and AI systems. The growing demand for workflow orchestration has led to the emergence of a host of open source and SaaS solutions in an area ripe for innovation.”

Marketing Technology News: MarTech Interview with Michael Kraut, VP of OEM for Automotive at Experian

Orchestration solutions

The two most popular orchestration solutions among the respondents are open-source workflow management platform Apache Airflow—used by 36% of the respondents—and the Prefect dataflow automation platform—used by 14% of all respondents.

While 46% of the surveyed data scientists and analysts reported they use Airflow, a notable 82% of respondents in these roles indicated that they plan to use a new orchestration tool in the next year. The data and ML engineers in the survey also expressed strong interest in trying Prefect and Dagster (both 18%) over Airflow (15%), while Prefect (at 14%) was the top solution that software engineers, developers and those in DevOps wanted to try for their next project.

“This report shows that the market for orchestration is growing significantly and starting to evolve beyond legacy products,” added Jeremiah Lowin, Founder and CEO of Prefect. “Data scientists, engineers, and developers all ultimately need dataflow automation that will ensure their code runs and their data arrives as expected, while most importantly making it easy to identify when a failure occurs and how to fix it. The result is less time spent writing defensive code to protect against failure, and more time on productive objectives.”

Marketing Technology News: In a Privacy-First World, A Data Management Strategy is Essential