The leading data observability platform’s new functionalities stop broken data pipelines before bad data impacts the business

Monte Carlo, the data reliability company, announced the release of a new suite of data observability capabilities to help data teams automatically stop broken data pipelines before they impact the business.

Data engineers spend upwards of 30 percent of their time tackling data downtime, meaning periods of time when data is missing, erroneous, or otherwise inaccurate. These issues cost companies millions of dollars per year, eroding trust in the data that informs decision-making and powers digital services.

Marketing Technology News: Deborah Besemer to Retire as Chairman of Brightcove’s Board of Directors

“Red Ventures is committed to delivering reliable and accurate data products for teams across our portfolio of media companies. With Monte Carlo’s data observability, our data engineers and analysts can automatically and collaboratively detect, alert on, and resolve data downtime before it becomes a problem for the business”

Monte Carlo’s Circuit Breakers automate testing and manage all quality checks across the entire data pipeline within a single interface. Circuit Breakers limit the downstream impact of bad data by stopping data quality issues closer to the source. With this release, Monte Carlo becomes the first Data Observability provider to deliver both automatic and custom orchestration-based tests to monitor for data quality issues.

A better way forward for managing data reliability at the orchestration level

Traditionally, data teams leverage manual tests to ensure that poor quality data doesn’t get passed downstream to stakeholders before going to production systems. To troubleshoot, teams must toggle between dozens of tools and platforms, slowing down the root cause analysis (RCA) process and making it difficult to develop a central source of truth about their data.

By stopping data processing jobs when data quality rules fail, Monte Carlo’s Circuit Breakers reduce time to detection and resolution of data issues, ensuring that teams avoid backfilling costs associated with cascading data failures. Simultaneously, Apache Airflow logs, dbt models, and other metadata related to an incident are made available in Monte Carlo, alongside additional root cause analysis capabilities.

Marketing Technology News: MarTech Interview with Werner Kunz-Cho, CEO at Fareportal

In addition to helping teams prevent backfilling costs and improve engineering efficiency, Circuit Breakers allow data teams to:

• Deploy data reliability checks directly into your data pipeline: For the first time, data teams can integrate Monte Carlo’s data quality tests directly into the orchestration layer, helping identify and stop bad data at its source.
• Manage rules and tests within a single platform: Data engineering teams can create and manage unscheduled data quality checks with Monte Carlo. Updates to the rules are automatically reflected in Airflow DAGs and other data processing jobs.
• More easily triage data quality issues: Once a job is stopped, Monte Carlo alerts data engineering teams via Slack, Microsoft Teams, email, webhooks, and other communication channels so they can prioritize associated issues and reduce time-to-resolution.
• Prevent broken dashboards and reports downstream: Circuit Breakers that fail rules automatically “circuit break” or stop the data processing job, preventing tables, downstream dashboards and reports from being populated with bad data. As a result, negative business impact from wrong decisions made with inaccurate, stale, or duplicate data are eliminated.
• Set both automatic and custom rules for Circuit Breakers: Monte Carlo automatically generates Circuit Breakers based on common data quality checks, including null values, freshness, and distribution, with the option to set custom rules and alerts based on the needs of your business.

With Circuit Breakers, data teams can more easily achieve data quality KPIs and SLAs, reducing time-to-detection (TTD) and time-to-resolution (TTR) for data issues from hours or days to minutes. By automating Circuit Breakers, data engineering teams can better scale data reliability across their entire data stack, without needing to update code each time a change is required.

Circuit Breakers reduce data downtime by 88 percent

Circuit Breakers take the promise of reliable data a step further by making it easier for Monte Carlo customers like Optoro and Red Ventures to identify, solve, prevent, and communicate data quality problems before they affect the business, reducing data downtime by 88 percent.