Designing an OCI Data Platform That Does Not Leak Money

Nobody designs a wasteful data platform. Waste is what happens between designs: the workload grows, the team provisions for the new peak, the peak passes, and nothing scales back down. The pipeline gets built for a dashboard, the dashboard owner changes roles, and the pipeline keeps running because turning things off has an owner and leaving them on does not. Two years later the platform bill has doubled and no single line item looks wrong.

OCI gives the data platform unusually good raw material for cost discipline, per second billing, genuine auto scaling on Autonomous, serverless engines that scale to zero, cheap object storage with lifecycle tiers, and egress pricing that undercuts the other hyperscalers. But the raw material is not the result. Every one of those features has a default that leaks, and the platform that does not leak is the one where somebody made seven specific design decisions deliberately. Here they are, with the leak each one closes.

The seven leaks and their design answers

1. Compute that idles between jobs

The classic: a Spark or Hadoop cluster sized for the nightly batch, billing 24 hours to work four. The design answer is to match the engine's billing model to the workload's duty cycle. Scheduled batch belongs on serverless engines like Data Flow, where idle time simply is not billed. Clusters that must persist, Big Data Service among them, get lifecycle automation: scale profiles for the batch window, stop schedules for development environments, and a standing rule that any cluster idle beyond a threshold gets flagged. Development and test environments are the deepest version of this leak; they are needed forty hours a week and billed for 168.

2. Warehouses sized for a peak that passed

Autonomous Database auto scales, but auto scaling works from a base OCPU count somebody chose, and bases ratchet upward. The quarter close needed more, the base went up, the close passed, the base stayed. The design answer is calendar scaling plus quarterly review: the base reflects the normal week, scheduled scale ups cover the known peaks, and auto scaling absorbs the surprises. The utilisation data to make this honest is already in the console; what platforms lack is the standing appointment to read it.

3. Hot storage holding cold data

Object storage is cheap, which is exactly why it fills with data nobody reads. The design answer is lifecycle policy at bucket creation, not as a cleanup project: raw zones tier to Infrequent Access and then Archive on a schedule, curated zones keep only current versions plus a defined history, and versioned buckets get version expiry. The same discipline applies inside the warehouse, where partitioned history tables hold years that no query touches; partition level archiving to object storage keeps the warehouse paying only for the data it serves. A lakehouse pattern makes archived data queryable without rehydrating it, which removes the usual objection to tiering.

4. Pipelines without consumers

Every pipeline was built for someone. Platforms that never trace lineage from pipeline to consumer cannot answer the only question that matters at review time: who reads this output? The design answer is a consumer register, pipeline, output, owning consumer, review date, kept in Data Catalog alongside the lineage that makes it verifiable. Our field experience is consistent: in a mature estate, between one in ten and one in five pipelines feed nothing anyone still uses.

5. Duplicate data answering the same question

The same customer table copied into five marts because copying was easier than sharing. Each copy carries storage, pipeline compute, and reconciliation risk. The design answer is sharing over copying wherever the consumer can query in place, Cloud Links inside the Oracle estate, Delta Sharing beyond it, and a default that new consumers get a view or a share before they get a pipeline.

6. Egress nobody designed

OCI egress is cheap relative to the competition, but cross region replication, external consumers pulling full snapshots, and chatty multicloud integrations still add a line that grows quietly. The design answer is locality review at design time: consumers near the data, snapshot pulls replaced by incremental shares, and replication scoped to what disaster recovery actually requires.

7. Nobody owns the number

The platform bill is everyone's context and no one's responsibility. The design answer is structural: compartments and tagging that map spend to workloads and owners, budgets with alerts per compartment, and a monthly review where each owner sees their number and the trend. Cost governance is not a tool, it is a meeting with data, and a cost governance setup that makes the data unavoidable.

The leak table

A seven step cost design framework

What this is worth

Across the estates we have run this discipline on, the pattern repeats: a platform that has grown organically for two years or more carries 25% to 45% removable spend, and our engagements average a 40% reduction in OCI spend without retiring a single capability the business uses. The first pass, duty cycles, warehouse bases, and storage tiers, usually lands within a quarter. The structural items, deduplication and pipeline retirement, take longer and pay longer. We run this as a defined optimization engagement priced on a percentage of verified savings, no savings, no fee, which keeps the incentive exactly where it belongs, or as part of a managed monthly retainer where the review cadence is built in.

Bringing it together

A data platform that does not leak is not a cheaper platform, it is a designed one. Every leak above is the default behaviour of a reasonable team under delivery pressure, and every answer is a decision that costs little at design time and compounds for years. If the OCI data platform bill has grown faster than the workload, the gap is almost certainly in these seven places, and an assessment will locate it precisely: duty cycles measured, bases compared to utilisation, storage aged, pipelines traced. The findings fund the fix.