Immuta State of Data Engineering Survey Reveals Emerging Challenges with Data Security and Privacy Amid Shift to the Cloud

Immuta, the leader in universal cloud data access control, today announced the findings from its 2022 State of Data Engineering Survey, which examined the changing landscape of data engineering and emerging challenges as companies shift data management workloads to the cloud.

Marketing Technology News: MarTech Interview with Johann Wrede, CMO at Xactly

Organizations are increasingly adopting multiple cloud technologies to keep up with the scale, speed, and use cases required by modern data teams. Nearly two-thirds (65%) of respondents characterized their company as either 100% cloud-based or primarily cloud-based, indicating a large market need for automated cloud data access control. Additionally, 81% of organizations expect to be fully or primarily cloud-based within the next 24 months, an increase from 71% in last year’s survey and a sign that cloud adoption is accelerating.

To gather these findings, Immuta and Gradient Flow surveyed nearly 400 data leaders, architects, engineers, and consumers across industries, organization sizes, and geographies. The survey revealed that as data becomes more complex, and data rules and regulations expand and evolve, new challenges such as data privacy, security, and quality have emerged that threaten to hinder data initiatives.

Further, the survey revealed that sensitive data use is on the rise, and the tasks associated with managing sensitive data – data cataloging, data discovery, and access control – are the most challenging. Increasing rules and regulations further complicate data management as new data platforms and BI/analytics tools are adopted.

Marketing Technology News: Five Companies Offering Conversational AI Solutions Named IDC Innovators

Key findings include:

• Shift to Cloud Computing: 65% of respondents characterized their company as already either 100% cloud-based or primarily cloud-based. Looking ahead 12-24 months, 81% of respondents projected they will be 100% cloud-based or primarily cloud-based.
• Need for Data Privacy and Security: 64% of respondents came from companies that already collect and store sensitive data, while 88% indicated their organizations are subject to one or more data use rules or regulations, with GDPR, HIPAA, CCPA, and SOC 2 cited as the most common.
• Challenge of Data Quality: Respondents cited data quality and validation as the most challenging area they face today, with 27% indicating they were unsure what (if any) data quality solution their organization is using.
• Rise of Data Catalogs and Data Discovery: 60% of organizations are now using data catalog and data discovery tools.
• Accelerated Adoption of Cloud Data Platforms: 62% of respondents plan to adopt at least one of the following five cloud databases and platforms in the next 12-24 months (in order of priority): Amazon Athena, Google BigQuery, Databricks, Snowflake, and Amazon Redshift. The biggest mover compared to the 2021 Impact Report was Google BigQuery, which moved up 5 spots this year.
• Top BI and Analytics Tools: The most popular BI and Analytics solutions cited by survey respondents were (in order of priority): Jupyter Notebooks, Tableau, Microsoft Power BI, Looker, and Google Colab.

“The move to the cloud is accelerating and organizations are making huge bets on the platforms and tools they need to manage their data,” said Matthew Carroll, CEO, Immuta. “As this survey shows, the data engineering landscape is getting more complex and data security and privacy are top concerns. We’ve seen huge demand from the public and private sector and are working with organizations large and small across industries who are focused on addressing this head-on.”

The report includes year-over-year comparisons to findings from Immuta’s 2020 Data Engineering Survey and 2021 Impact Report, which took place during September and October 2020.

Marketing Technology News: CLIPr Hires Durk Stelter as Chief Revenue Officer