As enterprises fiercely compete for data engineers, a new global poll out today by Wakefield Research and Fivetran, the leading provider of automated data integration, shows that, on average, 44 percent of their time is wasted building and rebuilding data pipelines, which connect data lakes and warehouses with databases and applications. This significant outlay of human capital isn’t adding up: 71 percent of respondents say end users are making business decisions with old or error-prone data — with 66 percent saying their C-suite doesn’t know this is even happening. As a result, 85 percent of enterprises have made bad decisions that have cost them money.

Marketing Technology News:  MarketSquare.io Announces Blockchain-based NFT Metaverse Game and Loyalty Program

Companies are paying large sums to reach these bad outcomes. Data and analytics leaders report a median of 12 data engineers earning an average of $98,400 a year each. Spending 44 percent of their time away from working on advanced models and analytics adds up to $520,000 a year.

“What we’re seeing from this study is that data and analytics leaders are really struggling to keep up,” said George Fraser, CEO of Fivetran. “It would be one thing if the processes companies used for manually building and managing pipelines were optimized, but 80 percent of those surveyed admit they have to rebuild data pipelines after deployment — from changing APIs, for example. For 39 percent, they say this happens often or all of the time.”

In addition to being error-prone, the process for deriving perceived value from the data also moves at a snail’s pace. Only 13 percent report being able to derive value from newly-collected data in minutes or hours. Instead for 76 percent of companies, it takes days or up to a week to prepare the data for revenue-impacting decisions, including 74 percent of companies with $500 million-plus in revenue.

In other findings:

• 69 percent of data and analytics leaders say that business outcomes at their company would be somewhat or significantly improved if their data team were able to contribute more to business decisions rather than manual pipeline management.
• 90 percent of those polled in the U.S. say hiring staff would be a major component if they could effectively scale up data management capacity, compared with 74 percent outside the U.S.
• Nearly all (97 percent) say business outcomes would be improved if their data team could spend more time devoted to the analytics behind data-driven business decisions.

Marketing Technology News:  What Can Brands Do to Protect Themselves From Social Media Outages?