Your data drifts silently — and no one sees it
One Monday morning, the executive dashboard shows revenue down 30%. Panic. After 2 days of investigation, the team discovers that an API connector changed its date format on Friday evening — and that all weekend orders were ignored. Data observability would have detected the volume anomaly in less than 10 minutes and sent an alert before the offices opened. It continuously monitors 5 pillars: freshness, volume, distribution, schema and lineage.
The problem
Data doesn't break like a server — it drifts silently. A field that goes from "98% filled" to "85% filled" over three weeks triggers no alarm. A file that usually arrives at 6 a.m. and starts arriving at 11 a.m. raises no ticket. Yet these drifts corrupt your analyses and your AI models.
The most frequent symptoms in SMBs and mid-market companies:
- The incident discovered by the business — The sales director notices that the month's sales look abnormally low in the dashboard. They alert the data team, which discovers an ingestion problem 5 days old. The missing data has to be reloaded manually. Cost: 3 days of work + loss of confidence.
- The AI model that drifts without warning — Your customer scoring model worked well at deployment. Three months later, its predictions are increasingly wrong. The cause: the distribution of the input data changed gradually (data drift) without anyone monitoring it. The model classifies data it never saw during training.
- The schema that changes without warning — A developer adds a column in the CRM or renames a field in the ERP. The extraction pipeline doesn't handle the change and stops silently. The data stops updating, but the dashboard keeps showing the old figures — with no error indication.
The common thread: the data team is always the last to know. Data observability reverses this dynamic by detecting problems before they reach users. To go further, discover our observability and traceability offer.
The AI solution
Data observability rests on the continuous monitoring of 5 pillars. AI plays a key role in detecting anomalies without predefined rules, by learning the normal behavior of your data and alerting on deviations.
Monitoring of the 5 pillars
Freshness (does the data arrive on time?), volume (is the number of rows consistent?), distribution (do the values follow the usual patterns?), schema (has the structure changed?) and lineage (are the dependencies intact?). Each pillar is monitored around the clock with adaptive thresholds.
ML-based anomaly detection
A machine learning model learns the normal behavior of each table and column — daily patterns, seasonality, trends. It automatically detects significant deviations without the need to define thresholds manually. The result: fewer false positives and detection of "unknown" anomalies.
Contextualized and prioritized alerts
Each alert includes the context: which table, which pillar, since when, what downstream impact (via lineage). Alerts are prioritized by business criticality: an anomaly on the orders table is more urgent than an anomaly on the logs table. Notification via Slack, email or PagerDuty.
Implementation
Data observability rollout happens in three phases over 4 to 8 weeks. The approach is progressive: start with the critical tables and expand coverage.
Identification of critical tables (weeks 1-2)
List the 10 to 20 most important tables in your data warehouse: those that feed executive dashboards, production AI models and regulatory reports. For each, document the expected update frequency, the average volume and the critical columns. This is your starting scope.
Deployment of monitors (weeks 3-5)
Install an observability tool (Elementary for dbt, Soda, or Monte Carlo). Configure monitors on the 5 pillars for your critical tables. Run a 2-week learning period to calibrate thresholds and reduce false positives. Connect the alerts to your dedicated Slack or Teams channel.
Response process and extension (weeks 6-8)
Define an alert response process: who is notified, what is the resolution SLA, how to document the incident. Train the Data Stewards to interpret the alerts. Gradually extend coverage to secondary tables. Review the thresholds each month to adjust sensitivity.
Results
Here are the results observed at our clients after 3 months of operational data observability.
Frequently asked questions
What is the difference between data quality and data observability?
Data quality checks that data complies with predefined rules (format, completeness, uniqueness). Data observability goes further: it continuously monitors the behavior of data — volumes, distributions, freshness, schema — and detects anomalies even without an explicit rule. It's the difference between a quality check at the end of the chain and real-time monitoring of the entire chain.
When should you invest in data observability?
As soon as you have more than 5 data pipelines in production or you're feeding an AI model with data that changes regularly. If your team has already spent more than 3 days diagnosing a data problem in the last 6 months, the return on investment will be immediate.
Can data observability replace data testing?
No, the two are complementary. Tests (Great Expectations, dbt tests) check known rules: "this column must not be null", "this amount must be positive". Observability detects unknown anomalies: "the data volume dropped by 40% compared to the average", "the age distribution changed abruptly". Keep both.
What is the cost of a data observability solution?
Open source solutions (Elementary, re_data, Soda open source) are free apart from hosting costs. SaaS solutions (Monte Carlo, Bigeye, Soda Cloud) cost between 1,000 and 5,000 euros per month for an SMB. The cost is easily justified against the price of an undetected data incident — typically 10,000 to 100,000 euros per major incident.
For tech profiles
Data observability architecture
Complete SaaS platform
ML anomaly detection on the 5 pillars, automatic lineage, native integration with Snowflake, BigQuery, Redshift, dbt and Airflow. Graphical interface for Data Stewards. Automated root cause analysis. Market reference, but high cost.
Open source and hybrid solutions
Elementary integrates natively into dbt and generates monitoring reports in your warehouse. Soda offers a declarative language (SodaCL) to define checks in YAML. Both offer a cloud version for alerts and dashboards. Ideal for teams of 2 to 5 data engineers.
The 5 technical pillars
Comparison of solutions
| Criterion | Monte Carlo | Elementary (OSS) | Soda Cloud |
|---|---|---|---|
| Monthly cost | 3,000-10,000 € | Free (self-hosted) | 500-2,000 € |
| ML detection | Advanced | Basic | Intermediate |
| dbt integration | Native | Native (dbt package) | Native |
| Initial setup | 1 day (SaaS) | 2-3 days | 1 day (SaaS) |