Risk & Compliance
Data Observability & Quality Management
Data observability explained: five monitoring dimensions, three pipeline quality gates, a worked data contract in YAML, and a four-phase rollout plan.
Data observability is the continuous monitoring of data health across five dimensions — freshness, volume, schema, distribution, and lineage — so problems are caught before they reach dashboards, models, and regulatory reports. Paired with data contracts and automated gates, it is what makes it defensible to automate decisions on top of your data.
#When Data Quality Becomes a Board-Level Metric
What this means for you: the exposure is no longer a wrong dashboard. It is a wrong number in a regulatory filing, a model trained on corrupted history, and no way to say when either started.
Data now powers revenue-generating systems, AI models, and decisions taken at board level. Most organisations still discover quality problems after the business has already acted on them.
The pattern is consistent: an incorrect forecast presented to the board, a model trained on a corrupted period, a customer-facing dashboard showing the wrong figure, a regulatory report filed on bad data.
The real question is whether the first person to notice a broken pipeline is an engineer with a runbook or an executive with a wrong number.
Traditional testing catches syntax errors and schema violations. It misses the subtle drift that causes most of the damage: distribution shifts that degrade model accuracy, late-arriving data that breaks time-series analysis, silent schema evolution, and completeness decay that skews everything downstream.
#What Data Observability Actually Measures
What this means for you: five signals, and the final column is the one to agree with your on-call team before any of them are switched on.
| Dimension | What it answers | The failure it catches that tests miss | What to do when it fires |
|---|---|---|---|
| Freshness | Is data arriving on time? | Upstream jobs that silently stop, and late batches that quietly shift a reporting window | Hold downstream publication, notify the producing team, and check whether any consumer already read the stale partition |
| Volume | Are row counts within expected ranges? | Partial loads, duplicate ingestion, and partitions that arrive empty without erroring | Quarantine the batch rather than propagating it, and compare against the same weekday in prior weeks before escalating |
| Schema | Is structure evolving without breaking consumers? | Additive changes that are safe and type changes that are not, plus fields whose meaning shifts while the type holds | Check the contract's compatibility mode, and route breaking changes to the producer as a diff rather than an incident |
| Distribution | Are value patterns statistically normal? | Null-rate spikes, category drift, and gradual shifts that degrade model accuracy long before anything fails | Alert the model owner as well as the pipeline owner, because the consequence lands in inference before it lands in a dashboard |
| Lineage | Where did this come from, and what depends on it? | Blast radius that nobody can enumerate during an incident | Use it to scope the incident: which reports, models, and downstream tables consumed the affected data, and who needs telling |
The first four detect that something is wrong. Lineage tells you what is affected and where to look for the cause, which is why it is the dimension most worth investing in before an incident rather than during one.
Coverage is a decision about blast radius, not a badge to collect.
#Three Quality Gates Between Ingestion and Consumption
What this means for you: deciding where a check runs is deciding who gets interrupted and how far bad data travels before someone notices.
The ingest gate
Contract validation at the boundary: schema compatibility, required fields, freshness, and volume bounds. Failures quarantine the batch rather than propagating it. This is the cheapest place to stop a problem and the only place where the producer is unambiguously accountable.
The transform gate
Business-rule and referential checks after transformation, where domain meaning exists. Failures block promotion from validated to business-ready. This is where most genuine quality logic lives, and where tests and observability overlap most usefully.
The publish gate
Consumer-facing service level checks before a dataset is exposed to reports, models, or downstream teams. Failures hold publication and notify named consumers. This is the gate a regulator or auditor will ask about.
Placing every check at the publish gate is the common shortcut, and it fails predictably: the problem is detected accurately and far too late to be cheap to fix.
#Formalising Expectations Between Producers and Consumers
What this means for you: a data contract converts arguments about who broke what into a version-controlled diff with a named owner.
Data contracts prevent downstream failures by codifying expectations, validating data at ingestion, and creating explicit accountability between teams. A contract is most useful when it is machine-readable, stored in version control next to the pipeline code, and enforced by the same checks in every environment.
#Example Data Contract: Customer Events Stream
# data-contract.yaml — customer_events, v1.2
dataset: analytics.customer_events
owner: product-analytics
schema:
format: avro
compatibility: BACKWARD # additive changes only
fields:
- name: user_id
type: string
nullable: false
- name: event_timestamp
type: timestamp
nullable: false
- name: email
type: string
checks: [valid_email_format]
- name: geo_lat
type: double
checks: [{ accepted_range: { min: -90, max: 90 } }]
freshness:
max_lag_minutes: 5 # event time -> availability
volume:
expected_rows_per_hour: { min: 10000, max: 50000 }
alert_on_deviation_pct: 20
sla:
max_error_rate_pct: 1
max_end_to_end_latency_minutes: 10
on_violation:
notify: [producer_team, consumer_teams]
action: quarantine_batch # hold, don't propagate
When a contract breaks, automated alerting notifies both producer and consumer teams. Impact analysis shows which downstream pipelines, dashboards, and models are affected.
Version control tracks contract evolution, so a breaking change is reviewed before it ships rather than diagnosed after it lands. Expressing these expectations as executable, reviewable artefacts is the same discipline described in governance as code, and the freshness and volume clauses are where data quality service level objectives become concrete.
#What Modern Observability Platforms Deliver
What this means for you: these are the capabilities we implement through our DQ Sentinel service, whether delivered on a commercial platform or assembled from open-source components. The first three detect; the second three explain and prioritise.
Detection capabilities:
Anomaly detection from learned baselines
Models learn normal behaviour for each dataset and flag deviations, which reduces manual threshold tuning rather than removing it. The value is catching seasonality shifts and correlation breaks nobody wrote a test for.
In-flight quality validation
Validate data as it moves through pipelines and quarantine batches that fail their contract, so faulty records are held at the boundary instead of propagating into downstream tables and reports.
Automated lineage tracking
Table-level lineage across the estate with column-level on regulated and AI-feeding paths, traced from source systems through transformations to consumption, so blast radius is known before an incident rather than reconstructed during one.
Explanation and prioritisation capabilities:
Alerting with context
Alerts that arrive with the affected datasets, the likely upstream cause, and the consumers at risk, routed to the team that can act. Context is what separates a useful page from noise.
Assisted root cause analysis
Correlation of incidents across systems, with likely causes proposed from historical resolutions. It shortens diagnosis; it does not remove the need for someone who understands the domain.
Business impact quantification
Quality incidents tied to the reports, models, and decisions they touched, so remediation is prioritised by consequence rather than by whichever alert fired most recently.
#Building Observability Into Your Data Platform
What this means for you: the collection layer is the part you cannot skip, because everything above it inherits its coverage and its gaps.
┌─────────────────────────────────────────────────────────────────────────┐
│ DATA OBSERVABILITY PLATFORM │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ COLLECTION & INSTRUMENTATION │ │
│ ├──────────────────────────────────────────────────────────┤ │
│ │ │ │
│ │ ┌────────────┐ ┌────────────┐ ┌───────────────┐ │ │
│ │ │ OpenTelemetry│ │ Agents │ │ SQL Queries │ │ │
│ │ │ (Metrics, │ │ (Profilers)│ │ (Metadata) │ │ │
│ │ │ Traces) │ │ │ │ │ │ │
│ │ └──────┬───────┘ └─────┬──────┘ └───────┬───────┘ │ │
│ │ │ │ │ │ │
│ │ └───────────────┼─────────────────┘ │ │
│ │ │ │ │
│ └─────────────────────────┼───────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────▼───────────────────────────────┐ │
│ │ CENTRALISED TELEMETRY STORE │ │
│ │ • Time-series DB (Prometheus/InfluxDB/TimescaleDB) │ │
│ │ • Distributed Tracing (Jaeger/Tempo) │ │
│ │ • Log Aggregation (Loki/Elasticsearch) │ │
│ └─────────────────────────┬───────────────────────────────┘ │
│ │ │
├────────────────────────────┼───────────────────────────────────────────┤
│ ANALYSIS & DETECTION LAYER │
├────────────────────────────┼───────────────────────────────────────────┤
│ │ │
│ ┌─────────────────────────▼───────────────────────────────┐ │
│ │ ML ANOMALY DETECTION ENGINE │ │
│ │ • Statistical Models (Z-score, IQR, MAD) │ │
│ │ • Time-Series Forecasting (Prophet, ARIMA) │ │
│ │ • Autoencoders for Multivariate Anomalies │ │
│ │ • Isolation Forests, DBSCAN for Outliers │ │
│ └─────────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────▼───────────────────────────────┐ │
│ │ DATA QUALITY VALIDATION │ │
│ │ • Great Expectations / Soda / Deequ │ │
│ │ • Contract Enforcement (Schema, SLA) │ │
│ │ • Custom Business Rules Engine │ │
│ └─────────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────▼───────────────────────────────┐ │
│ │ LINEAGE & IMPACT ANALYSIS │ │
│ │ • Column-Level Lineage (OpenLineage) │ │
│ │ • Dependency Graph (Neo4j/Graph DB) │ │
│ │ • Blast Radius Calculator │ │
│ └─────────────────────────┬───────────────────────────────┘ │
│ │ │
├────────────────────────────┼───────────────────────────────────────────┤
│ ALERTING & ACTION LAYER │
├────────────────────────────┼───────────────────────────────────────────┤
│ │ │
│ ┌─────────────────────────▼───────────────────────────────┐ │
│ │ INTELLIGENT ALERTING ENGINE │ │
│ │ • Context-Aware Routing │ │
│ │ • Alert Fatigue Reduction (ML-based prioritisation) │ │
│ │ • Auto-Remediation Triggers │ │
│ └─────────────────────────┬───────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────▼───────────────────────────────┐ │
│ │ INCIDENT MANAGEMENT INTEGRATION │ │
│ │ • PagerDuty, Slack, Email, Webhooks │ │
│ │ • Runbook Automation │ │
│ │ • Post-Mortem Generation │ │
│ └───────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ OBSERVABILITY DASHBOARD │ │
│ │ • Real-time Data Health Score │ │
│ │ • Anomaly Timeline & Correlation │ │
│ │ • SLA Compliance Tracking │ │
│ │ • Cost Attribution & Waste Analysis │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
The cost attribution panel at the bottom is not decoration. The same telemetry that tells you a pipeline is unhealthy tells you which pipelines are expensive, which is why observability and data platform FinOps are usually built by the same team.
#Four Phases to Comprehensive Observability
What this means for you: the durations below are indicative for a first wave across business-critical pipelines, not a commitment across the whole estate.
#Phase 1: Instrumentation and Baseline
Deploy collectors, instrument business-critical pipelines, and establish baseline metrics for freshness and volume. Implement alerting on service level breaches. Indicative duration for a first wave: 6–8 weeks.
#Phase 2: Automated Quality Checks
Implement a validation framework, define contracts for critical datasets, and enable quality gates at ingestion. Indicative duration: 8–10 weeks.
#Phase 3: Detection and Lineage
Deploy anomaly detection for distribution drift, build automated lineage, and stand up assisted root cause analysis. Indicative duration: 10–14 weeks.
#Phase 4: Prevention and Policy
Enable early warning where patterns are stable enough to support it, automate remediation for well-understood failures, and move policy enforcement into code. Indicative duration: 12–16 weeks.
#What This Buys the Person Accountable for the Numbers
What this means for you: the business case is avoided incidents and reclaimed engineering capacity, and both are measurable in your own environment rather than borrowed from a benchmark.
The value shows up in three places. Unused datasets are identified and retired instead of retained by default. Pipeline failures are caught at ingestion instead of after downstream consumption. Engineering time moves from firefighting towards building.
Each of those is measurable where you work: incidents detected before consumer impact, mean time to detection, and storage retired. That is a firmer basis for a business case than any industry statistic.
A business case built on your own baseline survives a board challenge. One built on an industry benchmark does not.
The reason this matters beyond reporting hygiene is what sits above it. Governed, monitored data is the first of the four layers that autonomous execution depends on, as set out in the autonomous enterprise architecture — and the pipeline discipline underneath it is where our data engineering work starts.
#Frequently Asked Questions
#How is data observability different from data quality testing?
Testing validates conditions you already know to check — schema shape, null constraints, referential integrity — at defined points in a pipeline. Observability continuously monitors data behaviour in production across freshness, volume, schema, distribution, and lineage, catching failures nobody wrote a test for. Mature platforms need both: tests are the contract, observability tells you when reality has left it behind.
#What are the five dimensions of data observability?
Freshness, whether data arrives on time. Volume, whether row counts sit within expected ranges. Schema, whether structure evolves without breaking consumers. Distribution, whether value patterns remain statistically normal. And lineage, where data comes from and what depends on it. The first four detect that something is wrong; lineage tells you what is affected and where to look.
#What is a data contract and who owns it?
A data contract is a machine-readable agreement between a data producer and its consumers covering schema, freshness, volume, quality rules, and what happens on violation. The producing team owns it, because they control the changes that break it. Consumers negotiate its terms and are notified automatically when it is violated. Stored in version control, it turns pipeline arguments into a diff review.
#How long does it take to implement data observability?
Instrumenting business-critical pipelines and baselining freshness and volume typically takes weeks rather than months and delivers alerting value immediately. The full progression through validation frameworks, contracts, anomaly detection, and policy enforcement is a phased build measured in quarters. Sequence by consequence: instrument what feeds regulated reporting first, then let detected incidents drive the order of everything after.
#How do you stop data quality alerts becoming noise?
Tier your datasets before you tier your thresholds. Business-critical pipelines page an on-call rota; everything else routes to a queue reviewed on a cadence. Attach context to every alert — affected consumers, likely upstream cause, and the owning team — and retire any rule that has fired repeatedly without producing an action. Alert volume is a metric worth tracking in its own right.
Unolabs is a Data and AI first engineering consultancy, headquartered in the United Kingdom with engineering operations in Pune and active engagements across the UK, Australia, and Hong Kong. We help enterprises build the architectural foundation for autonomous AI execution — governed data platforms, semantic intelligence, and agentic systems that enterprises can stand behind.
If you are deciding which pipelines to instrument first and what a quality gate should be allowed to block, book a discovery call and we will map the critical paths with you.
Continue reading
- Risk & ComplianceSecuring Enterprise RAG: PII Masking, Token-Level Access Control, and Boundary-Aware RetrievalHow to secure enterprise RAG: PII masking, chunk-level access control, boundary-aware retrieval, and audit trails regulators can follow.14 min read
- Agentic ArchitecturesDesigning Production Agentic AI Systems: Architecture Patterns, Guardrails, and EvaluationHow production agentic AI is built: the agent loop, typed tool contracts, guardrail config, evaluation harnesses, and the gates that grant autonomy safely.15 min read
- Data Engineering Trends 2026AI-Powered Autonomous Data Operations: What to Automate, and What to Keep Under ReviewAutonomous data operations explained: six AI DataOps capabilities, five levels of autonomy, and the guardrails that decide what may run without a human.13 min read
Find out what your pipelines are not telling you
We will review your current monitoring against the five dimensions and three gates above, and show you which failures would still reach a consumer undetected.
Book an Observability Review