Data Quality: The Journey That Never Ends

Introduction

This blog won’t dive too deeply into the importance of high-quality data. It’s well understood that reliable systems—whether recommendation engines, curated playlists, or growth forecasts—are built on a foundation of solid data. Instead, we’ll share our enterprise data warehouse journey, including how we monitor data health, empower business users to define quality rules, and build trust through proactive communication with a diverse community of data consumers.

Where we started

In the early days, Calendly’s data warehouse maturity lagged behind the company’s growth. But as customers arrived in droves and brand awareness soared, those forces converged to fuel the product-led growth (PLG) machine we know today.

As we entered the next phase of growth, we needed to unlock the power of our rich datasets to fuel product development, sharpen our understanding of customers, and elevate the overall user experience. But our immature data infrastructure simply wasn’t up to the challenge.

At the time, we were on Redshift (judgments of this warehouse service withheld), and data was ingested directly into the warehouse via a third-party SaaS tool. That ingestion system controlled everything: it dictated table schemas, handled backfills, and served as the only pipeline feeding data into Redshift. The catch was that our only safety net was ‘scream tests’—problems only surfaced when something broke loudly in production. For a team determined to make data quality and observability guiding principles, it was a frustrating way to operate and a clear signal that change was overdue.

Journey to a Data Platform

What we had built on Redshift was a data warehouse, but it was neither consistent nor trustworthy. That realization became the turning point where we committed to building a reliable data platform and invested in the engineering infrastructure and talent required to make it a reality. 

Calendly invested in Google Cloud Platform (GCP) and naturally decided to work with BigQuery as well. Our new stack also included Google Cloud Storage (GCS), Dataflow, Pub/Sub, and Cloud Composer (Airflow), each playing a critical role in the data ecosystem. Below is a high-level view of our architecture, contrasting the old with the new:

Quality of Data == Quality of Life

As we scaled up our data platform, good data quality became table stakes. Stronger governance and observability meant fewer surprises for both pipelines and stakeholders. So, we introduced strict schema definitions, routine data quality checks, and service-level agreements for each data source. While this was all hand-written and fairly lightweight, it established key patterns, to orchestrate our jobs, dataclasses for our sources to house our `expectation_spec`, and guardrails to ensure confidence in our system.

This was great for a while. Ultimately, we had many more users onboarded to our system and they asked very legitimate questions for data consumers. They wanted to know about data lineage and how often data is refreshed, they wanted to define their own monitors and to have a better understanding generally when something was ‘up’ with our data.

We knew we were outgrowing our own custom solution (with the very original name of ‘dq’). Since we were already invested in Google Cloud, we turned to Dataplex—Google's all-in-one metadata management system.  On paper, it had all the pieces we wanted: lineage tracking, data quality checks, profiling, producer visibility, and more. But, as the old adage goes, if something is too good to be true, it probably is. Dataplex had the right framework, but was still immature. Key automation features were missing and we did not want to be in a position where we replied to click-ops to add lineage for every table. We quickly realized it wouldn’t meet our scale needs nor automatically track new tables as they came online.

Around the same time, we started really honing in on our platform’s product market fit. Customer feedback, via quarterly surveys, made it clear that they wanted more transparency into data quality. Our Stakeholders knew we had a pulse on our data quality via proactive communications, but that was all they knew.What they really wanted was the ability to check the health of the data themselves to  have confidence in the insights they were sharing with others.

In what ways can we improve across Data Platform, Product Analytics, and AI?

• Establish robust data quality frameworks
  Regularly monitor and maintain those for accuracy and reliability.

• Ensure the system can handle real-time data ingestion and processing, and equally ensure the platform can handle increasing data volume as the appetite for more data grows.

• Finish the goal of access to all meaningful data in data warehouse. Currently, still missing part of the customer lifecycle that's not fully represented in data warehouse.

• Data quality, or at least some sort of understanding on what data quality is. (Percentage of clean data, alerts when data quality might be poor for specific fields/tables)

• Data Timeliness - We'd like to eventually get to a place where we have transactional data. Less of an issue than quality, but still worth optimizing.

• Some times I run into stale data in BigQuery.

Ultimately, the decision was made to look to the market for a new solution. Folks on the team had experience with Monte Carlo and a few other vendors. We did our due diligence and landed on Monte Carlo as our new vendor.

Monte Carlo Meets Calendly

We started off with a proof-of-concept of Monte Carlo at Calendly and realized quickly that the tool was going to do everything we needed it to. Out of the box, it already covered our homegrown solution’s capabilities and the product had additional capabilities  that were essential to take our Data Quality journey to the next level.

To help evaluate and compare, we broke down our needs as must have, should have and could have.  The below table shows the detail of what we needed, and wanted, for our data quality tools

Must Have

• Freshness - Ensuring tables are being updated on a routine basis

• Completeness - Ensuring the updates of a table contains the expected number of records

• Threshold - As new data enters our ecosystem, is the data what we expect? E.g. manually entered data doesn’t have an extra zero

• Calendly Data Stack Integrations - BigQuery, Airflow, dbt, Hex, GCS

Should Have

• Alert Routing - Who should receive and when

• RBAC - Different access levels for different users in our organization

• Lineage - Both on the table-to-table lineage as well as through Hex (our BI tool)

• Bundling of Tables - Looking at a set of tables that make up a critical pipeline

Could Have

• Warehouse Cost - Monitors for our warehouse spend, bytes scanned, and job times

When we started to really unpack the features they offered that we didn’t have— things like stakeholder enablement and the extra credit features—we knew we picked the right vendor for our needs.

During the implementation process, we had to decide when to use  dbt (Data Build Tool) tests versus Monte Carlo tests. Applying the KISS (Keep It Simple Stupid) principle led to a straightforward decision rule: if a test failure requires stopping the DAG (Directed Acyclic Graph) pipeline, implement it as a dbt test; otherwise, use Monte Carlo. This logic ensured that critical data quality issues that could corrupt downstream processes halt execution immediately, while softer anomalies (like new product names, values slightly outside expected ranges, or day-over-day ARR (Annual Recurring Revenue) variations) instead trigger monitoring alerts without disrupting pipeline flow. The approach effectively separates pipeline-blocking failures from monitoring scenarios thereby maintaining both data integrity and operational continuity.

We currently have over 350 tests running in Monte Carlo that were migrated from our dbt test suite.

​​A critical component of our testing framework is a custom SQL test that validates referential integrity for one of our most fundamental business relationships: ensuring every user is properly associated with an organization. This test serves as a crucial safeguard because any breakdown in this core relationship—whether due to replication issues, data drift, or upstream system changes—cascades throughout our entire data ecosystem corrupting downstream models and analytics. This single test has consistently caught referential integrity failures early in the pipeline, preventing them from snowballing into widespread data quality issues that could compromise business reporting, user analytics, and operational dashboards across the organization.

Many of these tests are subjective and will change over time with our business - but this is the guidance data platform team has provided to the team and our stakeholders. 

As the team scaled up Monte Carlo implementation building data products, migrating over 200 tests from dbt, integrating with our existing stack, and onboarding users—the value exceeded initial expectations. Most notably, we achieved a 40%+ reduction in data-related bug tickets, primarily due to proactive communication that demonstrated our team's awareness,and monitoring of, potential issues before they escalated to user-reported problems. This shift from reactive firefighting to proactive issue management has been transformative for our team’s credibility and operational efficiency. We actively track our incident acknowledgement rates as a key performance indicator, maintaining a 60-70% alert acknowledgement rate that reflects both our engagement with the monitoring system and the relevance of the alerts being generated.

Here is how we have broken down our Data Products:

Another important part of monitoring data quality is how it impacts all parts of our data stack. This is where integration plays a key role. Through Monte Carlo’s recently released Hex integration, we are now able to view complete lineage coverage across our entire data ecosystem. Our stakeholders can immediately identify when their specific dashboards or notebooks are affected by upstream data issues—with the added capability of self-service discovery through intelligent alert routing within our data products. This end-to-end visibility eliminates guesswork around impact assessment and empowers business users to proactively understand how data quality incidents affect their specific analytical work, creating a more transparent and efficient incident response process.

Conclusion

A data platform only delivers value when it’s backed by robust monitoring and observability. Without oversight, even the most advanced architecture can quickly turn from asset to liability. The very foundation of reliability is comprehensive visibility into system health, data quality, and operational performance—allowing stakeholders to trust the insights driving their decisions.

At Calendly, building that trust meant extending transparency beyond engineering. Clear communication channels and self-service monitoring gave business stakeholders the ability to see behind the curtain and even configure their own quality checks. This shared accountability reinforced a simple principle: just as code is only as good as its tests, data pipelines are only as reliable as the checks that safeguard them.

And, the benefits extend beyond trust. Monte Carlo’s Performance Dashboard gave us visibility into high-cost jobs, helping us prioritize technical debt remediation and target optimizations with the greatest impact. That cost transparency now informs our resource allocation and platform strategy—delivering both operational efficiency and financial gains.

Appreciations

First and foremost, we want to thank our colleagues at Calendly who contribute to data quality every day. Their commitment to maintaining business-critical tests, communicating data impact to stakeholders, and amplifying the value of Monte Carlo across the organization has been essential to our success. Today, users across multiple departments are embracing the data platform, extracting meaningful insights, and building confidence not only in the data they rely on, but also in the systems our Data Platform team delivers.

We also want to recognize our partnership with Monte Carlo. While their tooling provides the foundation, it’s the ongoing support, optimization guidance, and intelligent monitoring recommendations that have elevated our entire approach. Their collaboration has reduced configuration overhead, sharpened our focus on the most impactful metrics and tests, and ultimately strengthened the reliability of our data platform.

Next Steps

Looking ahead, we plan to expand data quality investments beyond the data warehouse. Several use cases currently managed with custom testing are strong candidates for Monte Carlo deployment. These include PostgreSQL datamarts and document stores like Elasticsearch. Extending automated monitoring into these systems will bring the same level of visibility and trust to more of our critical data sources.

We are also collaborating with Monte Carlo on deeper cost optimization features for BigQuery. Also, enhanced performance monitoring within the GCP ecosystem presents a clear opportunity to uncover additional savings and efficiency gains. Together, these efforts mark the next phase of our journey toward a more reliable, transparent, and cost-effective data platform.