Walk through most data estates and you find a Spark cluster that is busy four hours a day. It was sized for the nightly batch, it runs the nightly batch well, and for the other twenty hours it burns compute waiting. The team knows, but the cluster is load bearing and nobody wants to touch the scheduling, so the waste gets renewed annually like an unread subscription.
OCI Data Flow is the platform's answer: Apache Spark as a fully managed, serverless service. There is no cluster to provision, patch, or babysit. You define an application, the code, the Spark version, the driver and executor shapes, and each run spins up exactly the requested capacity, executes, writes logs and results, and releases everything. The economics invert: instead of paying for a cluster's calendar, you pay for the job's runtime. This article covers how it works, the real cost behaviour, where it beats Big Data Service and self managed Spark on OKE, and the migration framework we use, all in the context of the wider OCI data platform.
How Data Flow works
The unit of definition is the application: a Spark artefact in Java, Scala, PySpark, or Spark SQL, stored in Object Storage, plus default sizing, driver shape, executor shape, executor count. The unit of execution is the run: an invocation of the application, with parameters and sizing overridable per run. Runs are submitted from the console, the CLI, the API, or orchestrators, and each run gets an isolated, ephemeral Spark environment inside Oracle managed infrastructure, with output and Spark UI history preserved after the run ends.
Data access is where the integration shows. Runs read and write Object Storage natively, which makes Data Flow the natural transformation engine for a lakehouse architecture: raw zone in, curated Parquet out, with the heavy shuffles happening on capacity that exists only while the job does. Private endpoints let runs reach databases and services inside your VCN, so a job can land its result straight into Autonomous or read from a source system without public exposure. Resource Principal authentication removes credential files from the picture.
Pools, streaming, and the rough edges
Cold starts are the serverless tax: a run pays a provisioning delay before the first task executes. For a nightly batch nobody notices. For a chain of short jobs or an SLA bound pipeline, Data Flow pools answer it, pre created capacity that runs start into immediately, billed while the pool is up, schedulable to match the batch window. Pools are also the budget control: a pool caps the aggregate capacity concurrent runs can consume, which turns a runaway scheduling mistake from a billing event into a queue.
Structured Streaming is supported, and a streaming run is effectively a long lived job, which means always on capacity and none of the scale to zero economics. Consuming from OCI Streaming's Kafka compatible endpoints works well, but estates with very heavy continuous streaming should price the always on pattern honestly against dedicated infrastructure before assuming serverless wins.
Data Flow against the alternatives
| Dimension | Data Flow | Big Data Service | Spark on OKE |
|---|---|---|---|
| Operating model | Serverless, per run | Managed Hadoop and Spark cluster | Self managed on Kubernetes |
| Idle cost | None between runs | Cluster runs unless stopped | Node pool runs unless scaled down |
| Ecosystem breadth | Spark only | Full Hadoop stack, Hive, HBase, more | Anything you containerise |
| Operational burden | Minimal | Moderate, cluster lifecycle | High, you own the stack |
| Latency profile | Cold start per run, pools mitigate | Always warm | Always warm if provisioned |
| Customisation depth | Configuration and dependencies | Cluster level control | Total control |
| Best fit | Batch transforms, scheduled heavy lifts | Hadoop estates, mixed stack needs | Engineering teams standardised on Kubernetes |
The decision pattern that holds up: if the workload is Spark jobs with a schedule, Data Flow wins on economics and operations. If the estate genuinely uses the broader Hadoop stack, Hive metastore centric tooling, HBase, long lived services, Big Data Service is the honest home. If a platform engineering team already runs everything on OKE and wants Spark as one more workload under their GitOps regime, Spark on Kubernetes is coherent, provided someone prices the engineering hours as part of the TCO. What rarely survives scrutiny is a permanent cluster whose only job is nightly batch, which is exactly the workload serverless was built to absorb.
What it costs, and how it leaks
Data Flow bills the OCPUs and memory of driver and executors for the duration of each run, plus the Object Storage underneath. The headline win is structural, idle time simply is not billed, but three leak patterns deserve attention. Oversized defaults: an application defined with twenty executors because the first developer guessed, then never revisited, pays the guess on every run; the Spark history UI shows real utilisation, and right sizing runs is the cheapest optimization on the platform. Unbounded concurrency: schedulers that fan out dozens of simultaneous runs multiply capacity silently, which is what pool caps are for. And forgotten streaming runs: a long lived run someone started as a test and left is a small cluster on permanent rent. All three are visibility problems before they are engineering problems, the theme we expand in data platform cost design.
A six step migration framework
- Inventory the Spark estate by runtime, not by cluster. List every job, its schedule, duration, and the capacity it actually uses at peak. The gap between cluster size and job need is the prize.
- Classify the workloads. Scheduled batch goes to Data Flow first. Hadoop stack dependencies stay on Big Data Service. Continuous streaming gets priced both ways before moving.
- Port one heavy nightly job end to end. Dependencies packaged, Object Storage paths wired, private endpoint tested, logs landing where operations expects them. The first job carries all the learning.
- Right size from the history server. Run the ported job, read the actual executor utilisation, and cut the allocation to fit. Repeat after data volumes shift.
- Wire orchestration and pools. Schedule through your orchestrator, group SLA bound jobs onto a pool sized for the batch window, and cap concurrency at the pool.
- Decommission the idle cluster. The business case completes when the old cluster stops billing, not when the new jobs run. Track the before and after spend and bank the difference.
Bringing it together
Data Flow is one of the clearest wins on the OCI data platform for a common, expensive pattern: heavy Spark batch on infrastructure that idles between runs. The serverless model fits the workload's actual shape, the lakehouse integration is native, and the operational surface is a fraction of cluster ownership. The judgment calls, what moves first, what stays on Big Data Service, how pools map to batch windows, and what the right sized runs actually need, are exactly what we settle in a platform engagement, on a fixed project fee or as part of an optimization engagement paid only on verified savings. An assessment of the current Spark spend against job runtime is usually the fastest way to see the size of the prize, and our OCI optimization practice runs that analysis as standard.
Free white paper
Go deeper on this topic with The Exadata Cloud Decision Guide, Database Service vs Cloud@Customer vs Autonomous, and how to choose. An independent analyst style report with comparison tables and recommendations, free with a work email. Prefer a monthly summary instead? The OCI Brief delivers one practical OCI briefing a month.
Part of a series
This guide is part of Data & AI on OCI — our complete pillar guide on the topic.
Moving Oracle workloads to OCI, or already running on OCI and not sure the architecture or the spend is right? Most teams bring in a specialist before they commit to a region, a shape, or a Universal Credits number. OCISpecialists.com plans the landing zone, runs the migration, and manages the estate after go live, on a fixed project fee, a managed monthly retainer, or a cost optimization fee paid only on verified savings.