Cloud Operations

Data Platform FinOps: Controlling Databricks, Snowflake, and AI Compute Costs

A data platform FinOps playbook: cost attribution, right-sized compute, storage tiering, query tuning, and GPU controls for Databricks and Snowflake.

By Akshay RajJune 9, 202615 min read

Categories where platform spend leaks

Optimisation tactics with durable payback

GPU cost controls for AI workloads

Phases in a FinOps rollout

Data platform FinOps is the practice of making cloud spend attributable, then engineering it down. Attribution comes first: until every warehouse, cluster, and query maps to an owner, optimisation is guesswork. The order that works is measure, attribute, optimise, then commit — and commitments last.

#When the Data Platform Bill Reaches the Board

What this means for you: the question you will be asked is not "why is it high" but "who decided", and most data platforms cannot answer that today.

After a decade of cloud-first enthusiasm in which data teams were largely insulated from financial consequence, cost has returned as a first-class engineering concern. Data workloads are among the most expensive lines in a modern estate: multi-petabyte warehouses, always-on streaming, GPU-accelerated training, cross-region transfer, and duplicate storage held for compliance.

The underlying problem is rarely extravagance. It is that nobody can say where the money went.

A cloud bill is not a finance artefact. It is a running record of architectural decisions that nobody reviewed.

Engineers provision without seeing a price. Storage accumulates because no retention policy was ever agreed. Queries run inefficiently because computational waste is not measured. Architecture optimises for scalability because that is what the design review asked about.

In our engagements, the first month of a FinOps programme is almost entirely archaeology, and the finding that changes the conversation is usually not a single expensive workload but the share of spend that nobody claims.

#Where Data Platform Money Leaks

What this means for you: six categories account for most recoverable spend. The last column is the control to put in place first, before any tuning work begins.

Waste category	What it looks like in your estate	Why it persists	First control to put in place
Idle compute	Clusters running around the clock for workloads that execute for a couple of hours; non-production environments sized like production	Nobody is billed for the hours nobody used	Auto-suspend defaults on every warehouse and cluster, with an explicit exception list that needs an owner's name
Inefficient queries	Full table scans, missing partition pruning, unoptimised joins, repeated identical aggregations	Query cost is invisible at the point of authorship	Query-level cost attribution surfaced back to the author, with the top spenders reviewed weekly
Storage sprawl	No retention policy, duplicate datasets per environment, hot tiers holding cold data, abandoned experiments	Deleting data feels riskier than keeping it	Lifecycle rules and a default retention class assigned at dataset creation rather than negotiated later
Data movement	Cross-region transfer, repeated egress, full copies where incremental would serve	Egress is priced separately from the pipeline that causes it	Egress attributed to the pipeline that triggers it, and a review gate on any new cross-region dependency
Over-provisioning	Resources sized for peak and running far below it, with headroom added defensively	Under-sizing causes incidents; over-sizing causes nothing visible	Published utilisation reporting per workload, with right-sizing proposals raised as reviewable changes
Tool duplication	Overlapping SaaS platforms across teams, unused licences, expired proofs of concept still billing	Renewals are administrative rather than architectural	An owned inventory of platform tooling with renewal dates and a named approver per line

The pattern across all six is the same. The cost is created by one team and paid by another, and the feedback loop between the two is missing.

#Making Spend Visible and Attributable

What this means for you: this is the instrumentation you need before a single optimisation is attempted, and it is the part most programmes under-resource.

Attribution is not one dashboard. It is a set of joins between cloud billing data, platform metadata, and organisational ownership — and each of those three sources fails in a different way.

Cost line	What actually drives it	Where the number lives	What to instrument first
Warehouse and cluster compute	Concurrency, warehouse size, and how long resources stay warm	Platform query history and credit consumption tables	Cost per query joined to user, warehouse, and originating pipeline
Object and table storage	Retention, replication, and format choices; uncompacted small files	Cloud storage inventory plus table metadata	Storage by dataset and age band, with an owner tag on every dataset
Streaming infrastructure	Partition count, replication factor, and always-on broker capacity	Cluster metrics and managed-service billing	Throughput per topic against provisioned capacity
Orchestration	Scheduler footprint and the worker capacity idle jobs hold open	Orchestrator run history	Cost per DAG or job, attributed to the consuming team
Transfer and egress	Cross-region reads, external table access, and repeated extracts	Cloud billing detail, usually the least granular source	Egress mapped to the pipeline or consumer that triggers it
Tooling and licences	Seat counts, tiers, and renewals nobody reviewed	Procurement records rather than the platform	A single inventory with owner and renewal date per line

Every untagged resource is a cost you will be arguing about six months from now.

Once those joins exist, optimisation becomes a ranked list rather than a debate. Finance gets transparency, engineers see the consequence of their design choices, and both parties argue from the same number.

#Six Optimisation Tactics With Durable Payback

What this means for you: these are ordered by how quickly they pay back against how much engineering effort they take. The first group needs configuration; the second needs design work.

Configuration-led tactics, achievable without redesigning pipelines:

Storage tiering and lifecycle rules

Move cold data to cheaper tiers automatically, compress with columnar formats, and expire data on a published retention policy. Lifecycle rules beat scheduled cleanup because they do not depend on anyone remembering.

Right-sized, auto-suspending compute

Warehouses and clusters that scale down and suspend when idle, separate pools for different service levels, and batch work scheduled outside peak windows. The saving is usually larger than any query rewrite.

Reserved and committed capacity

Cloud providers publish meaningful discounts for one- and three-year commitments against on-demand pricing. Commit only to the baseline you can evidence from at least two quarters of steady usage.

Design-led tactics, which change how pipelines work:

Query optimisation

Partition pruning on time-series data, materialised views for repeated aggregations, result caching, and clustering keys applied where the access pattern justifies them. Target the queries that appear most often, not the ones that look worst.

Reduced data movement

Federated queries and in-place transformation where a copy adds no value, and change data capture instead of full replication. The cheapest transfer is the one that does not happen.

Incremental processing

Process changed data rather than full refreshes, using watermarks and checkpoints. Daily full recomputation of a large table is often the single largest avoidable line in a warehouse bill.

Query-level work has more depth than a single section allows, and the engine-specific tuning is covered separately in lakehouse performance tuning. Where the underlying question is which platform to standardise on in the first place, the Microsoft Fabric versus Databricks decision framework is the more useful starting point.

#Containing the GPU Bill Before It Lands

What this means for you: accelerated compute grows faster than any other line and is the easiest to waste, because an idle GPU looks exactly like a busy one on a dashboard nobody reads.

AI scaling adds a dimension to platform economics that the usual controls handle badly. Training and inference capacity is expensive per hour, provisioned in advance, and frequently left running between experiments.

The same attribution-first discipline applies, extended with four controls built for accelerated compute.

Spot and preemptible orchestration

Route non-critical training and batch inference onto interruptible capacity with checkpointed restarts. The published discounts for interruptible instances are substantial, and checkpointing is the engineering work that makes them usable.

Dynamic team quotas

Allocate GPU and warehouse capacity per team against project priority, reviewed on a fixed cadence, so experimental workloads cannot quietly consume production budget.

Tagging enforced at provisioning

Resource metadata required at creation, with untagged provisioning rejected by the pipeline rather than flagged afterwards. This is what makes attribution across clouds hold up over time.

Orphan reclamation

Continuous scanning for accelerator instances and clusters with no recent activity, with automatic suspension after a published grace period. Notification alone does not work; the reclaim has to be the default.

#Reference Architecture: Cost Management Platform

What this means for you: the layer worth building first is telemetry and attribution. Everything above it is only as trustworthy as the tags underneath.

┌──────────────────────────────────────────────────────────────────────────┐
│                    COST VISIBILITY & CONTROL LAYER                        │
├──────────────────────────────────────────────────────────────────────────┤
│                                                                           │
│  ┌───────────────────────────────────────────────────────────────┐      │
│  │              Real-Time Cost Dashboard                          │      │
│  │  • Spend by Team/Product/Pipeline                              │      │
│  │  • Budget Utilisation & Forecasting                            │      │
│  │  • Cost Anomaly Detection (ML-based)                           │      │
│  │  • Savings Recommendations & Opportunities                     │      │
│  └───────────────────────────────────────────────────────────────┘      │
│                                                                           │
│  ┌───────────────┐  ┌────────────────┐  ┌──────────────────┐           │
│  │  Budget       │  │  Chargeback    │  │  Cost Alerts     │           │
│  │  Allocation   │  │  Reporting     │  │  (Slack/Email)   │           │
│  └───────────────┘  └────────────────┘  └──────────────────┘           │
│                                                                           │
├───────────────────────────────────────────────────────────────────────────┤
│                       TELEMETRY & ATTRIBUTION                             │
├───────────────────────────────────────────────────────────────────────────┤
│                                                                           │
│  ┌───────────────────────────────────────────────────────────────┐      │
│  │           Cloud Cost APIs & Tag Propagation                    │      │
│  │  • AWS Cost Explorer / Azure Cost Management / GCP Billing     │      │
│  │  • Resource Tagging (Team, Product, Environment)               │      │
│  │  • Unified Cost Data Lake (15-min granularity)                │      │
│  └──────────────────────────┬────────────────────────────────────┘      │
│                             │                                            │
│  ┌──────────────────────────▼────────────────────────────────────┐      │
│  │        Query-Level Cost Attribution (Snowflake/BigQuery)       │      │
│  │  • QUERY_HISTORY with credit consumption                       │      │
│  │  • User/Warehouse/Pipeline attribution                         │      │
│  │  • Cost per query (compute + storage + egress)                │      │
│  └──────────────────────────┬────────────────────────────────────┘      │
│                             │                                            │
├─────────────────────────────┼──────────────────────────────────────────┤
│                  OPTIMISATION ENGINES                                    │
├─────────────────────────────┼──────────────────────────────────────────┤
│                             │                                            │
│  ┌──────────────────────────▼────────────────────────────────────┐      │
│  │          Intelligent Query Optimiser                           │      │
│  │  • Pre-execution cost estimation                               │      │
│  │  • Rewrite suggestions (partition pruning, indexes)            │      │
│  │  • Materialised view recommendations                           │      │
│  │  • Query result caching with TTL                               │      │
│  └──────────────────────────┬────────────────────────────────────┘      │
│                             │                                            │
│  ┌──────────────────────────▼────────────────────────────────────┐      │
│  │           Auto-Scaling & Scheduling Engine                     │      │
│  │  • Predictive scaling based on historical patterns             │      │
│  │  • Auto-pause/resume for idle resources                        │      │
│  │  • Workload scheduling for off-peak execution                  │      │
│  │  • Spot instance orchestration                                 │      │
│  └──────────────────────────┬────────────────────────────────────┘      │
│                             │                                            │
│  ┌──────────────────────────▼────────────────────────────────────┐      │
│  │          Storage Lifecycle Management                          │      │
│  │  • Automated tiering (Hot → Warm → Cold → Archive)            │      │
│  │  • Retention policy enforcement                                │      │
│  │  • Compression optimisation (Parquet/ORC)                      │      │
│  │  • Duplicate detection & deduplication                         │      │
│  └──────────────────────────┬────────────────────────────────────┘      │
│                             │                                            │
│  ┌──────────────────────────▼────────────────────────────────────┐      │
│  │              FinOps Policy Engine                              │      │
│  │  • Budget threshold alerts & auto-throttling                   │      │
│  │  • Approval workflows for expensive operations                 │      │
│  │  • Resource tagging enforcement                                │      │
│  │  • Reserved capacity recommendations                           │      │
│  └────────────────────────────────────────────────────────────────┘      │
│                                                                           │
├───────────────────────────────────────────────────────────────────────────┤
│                    WASTE DETECTION & CLEANUP                              │
├───────────────────────────────────────────────────────────────────────────┤
│                                                                           │
│  ┌───────────────────────────────────────────────────────────────┐      │
│  │            Automated Waste Scanner                             │      │
│  │  • Idle compute detection (>7 days no activity)                │      │
│  │  • Orphaned storage (datasets unused >90 days)                 │      │
│  │  • Over-provisioned resources (avg utilisation <20%)           │      │
│  │  • Duplicate datasets across environments                      │      │
│  │  • Expensive queries running repeatedly (optimise candidates)  │      │
│  └───────────────────────────────────────────────────────────────┘      │
│                                                                           │
└───────────────────────────────────────────────────────────────────────────┘

The waste scanner at the bottom is the component teams most often build first and benefit from least. Without the attribution layer above it, a scanner produces a list of orphans with no owner to send it to.

#Building Cost Awareness Into Your Platform

What this means for you: four phases, each with an outcome you can defend in a business case. The timings are indicative for a first workstream, not a commitment across an estate.

Visibility foundation (roughly 4–8 weeks). Connect cloud cost APIs, establish resource tagging, and stand up a dashboard showing spend by team and service. Enable query-level cost tracking in the warehouse. Outcome: the first honest picture of where money goes.
Attribution and budgets (roughly 8–12 weeks). Tag resources with team, product, and environment. Build chargeback reporting finance will accept. Introduce team-level budgets and alerts. Outcome: spend has owners.
Automated optimisation (roughly 12–16 weeks). Auto-suspend and auto-scale compute, apply storage lifecycle policies, and surface query optimisation recommendations to authors. Outcome: reduction from configuration alone, before any redesign.
Continuous FinOps (roughly 16–24 weeks). Automated waste detection and reclaim, reserved capacity review against evidenced baselines, and cost tracked against business outcomes. Outcome: savings that compound rather than a one-off cut.

#Who Actually Owns the Number

What this means for you: cost reduction is an engineering deliverable with a finance sponsor, and programmes that invert that relationship stall within two quarters.

Finance can report spend. Only engineering can reduce it.

That division of labour has a consequence for how the programme is staffed and incentivised. Engineers need visibility at the point of authorship, decision rights over architectural trade-offs, and an explicit mandate that says a slower-but-cheaper design is an acceptable answer.

Without that mandate, cost review becomes a monthly meeting where a number is reported and nothing changes. With it, the platform starts making cost-aware choices by default — which is the same cultural shift described in platform engineering and DataOps culture, applied to spend.

The measurement discipline is also shared. The instrumentation that tells you a pipeline is expensive is the same instrumentation that tells you it is broken, which is why continuous data observability and FinOps tend to be built by the same team. If you are standing up either capability, our cloud platform engineering and data platform build work covers the underlying instrumentation.

#Frequently Asked Questions

#What is data platform FinOps?

Data platform FinOps applies financial accountability to data and AI infrastructure: attributing cloud spend to teams, products, and pipelines, then engineering it down through configuration and design changes. It differs from general cloud FinOps in emphasis, because warehouse credits, query behaviour, storage retention, and accelerator utilisation drive most of the bill rather than virtual machine fleets.

#How do you attribute Snowflake or Databricks cost to a team?

Join platform query history and credit consumption to an ownership map, using warehouse, user, role, and pipeline identifiers as the keys. The gap is usually ownership rather than telemetry: the platform knows which role ran the query, but not which team owns that role. Fixing the mapping and enforcing tags at provisioning closes most of it.

#Should we buy reserved capacity for our data platform?

Commit only to the baseline you can evidence from at least two quarters of steady usage, and keep volatile workloads on-demand. Reservations convert variable overspend into fixed overspend when the workload later changes shape. Review commitments at each renewal against actual utilisation rather than rolling them forward, and treat a reservation as an architectural decision.

#How do we control GPU costs for AI workloads?

Four controls carry most of the value: route interruptible work to spot capacity with checkpointed restarts, allocate quotas per team against project priority, reject untagged provisioning in the pipeline rather than flagging it later, and reclaim accelerator instances automatically after a published idle period. Instrument utilisation per accelerator before the first large training run.

#Who should own cost reduction, finance or engineering?

Finance owns reporting, forecasting, and the budget envelope. Engineering owns the changes that move the number, because queries, storage layout, and compute sizing are engineering decisions. The workable model gives engineers cost visibility at the point of authorship and explicit permission to trade performance for cost, with finance sponsoring the programme rather than running it.

Unolabs is a Data and AI first engineering consultancy, headquartered in the United Kingdom with engineering operations in Pune and active engagements across the UK, Australia, and Hong Kong. We help enterprises build the architectural foundation for autonomous AI execution — governed data platforms, semantic intelligence, and agentic systems that enterprises can stand behind.

If you are weighing where your data platform spend is actually going before committing to reserved capacity, book a discovery call and we will work through the attribution with you.

Continue reading

See where your platform spend is actually going

We will map your Databricks or Snowflake spend to the four waste categories above and show you which controls pay back first.

Book a Platform FinOps Review