OCI Data Integration: ETL Without Servers

Somewhere in almost every estate there is a VM whose entire job is running ETL scripts at 2am, and whose entire documentation is the memory of whoever wrote them. It gets patched nervously, sized for its busiest hour, and billed for all of them. OCI Data Integration exists to retire that VM. It is a fully managed, serverless ETL service: you design extract, transform, and load logic in a visual canvas, schedule it, and the service provisions whatever compute the run needs, executes, and disappears. No cluster, no agent, no operating system, and no bill between runs. This guide explains how the service is put together, what it costs, where it fits among OCI's other movement tools, and how to adopt it without recreating the script swamp in a prettier interface.

Data Integration is the batch movement layer of the OCI data platform, sitting alongside GoldenGate for continuous change and Data Flow for code heavy transformation.

How the service is put together

The working unit is a workspace, which holds connections to sources and targets, the data flows you design, and the tasks that execute them. A data flow is the visual pipeline: sources in, a chain of operators for joining, filtering, shaping, and cleansing, targets out. Tasks wrap data flows, or simpler operations like straight data loads and SQL executions, with parameters and runtime configuration, and pipelines chain tasks into orchestrated sequences with branching and error handling. Schedules trigger the lot. Connectivity covers the obvious estate: Autonomous Database, Object Storage, MySQL and HeatWave, other relational engines, and generic JDBC reach for the rest.

Two properties matter more than the feature list. First, design is declarative: the canvas describes what should happen, and the service compiles it into distributed execution, with schema drift handled through rules rather than hand edits. Second, execution is genuinely serverless: capacity appears for the run and vanishes after it, so an estate of nightly jobs consumes nothing all day.

Where it sits among the movement tools

The boundaries are practical. If the business needs data fresh in seconds, no batch schedule will save you, and the answer is change data capture through GoldenGate. If the transformation is hundreds of lines of logic, machine learning preparation, or anything a visual canvas would strangle, write it as Spark and run it on Data Flow. The large middle, scheduled extracts, conformance, warehouse and lakehouse loading, belongs to Data Integration, and the fourth row of the table belongs to history.

What it costs, and how the bill behaves

Pricing follows execution rather than existence. Charges accrue from workspace usage and the compute consumed by runs, so cost tracks how much processing you do, not how big a box you reserved. The behaviour to design for is concentration: jobs that read efficiently, push work down to capable sources where the optimizer allows, and avoid reprocessing unchanged data through incremental extracts keep run consumption proportional to new data rather than total data. The classic anti pattern is the full reload that grows linearly with history, costing a little more every night forever, the kind of slow leak we catalogue in data platform cost design.

Designing pipelines that age well

The service removes servers, not discipline. Pipelines age well when extracts are incremental by design, transformations are built from parameterised, reusable flows rather than copies, and every task writes to a defined zone in the lakehouse structure we set out in building a lakehouse on OCI. They age badly when the visual canvas becomes a museum of one off jobs, each cloned from the last and edited in place, which is the script swamp reborn with better graphics. Treat data flows as code: named conventions, versioned exports, a development workspace separate from production, and reviews for changes that touch shared logic.

Limits and honest caveats

Independence means listing the edges. Data Integration is a batch tool, and no amount of aggressive scheduling turns it into a streaming platform; if the requirement is seconds, that is GoldenGate's job, and pretending otherwise produces an expensive metronome. Complex procedural logic fights the visual paradigm, and pipelines that bury hundreds of business rules inside expression operators become as opaque as the scripts they replaced; that work belongs in Spark on Data Flow or in SQL inside the database, orchestrated rather than reimplemented. Connector coverage is solid for the OCI native estate and standard relational sources, but exotic SaaS sources may need a landing step through object storage first. And as with every visual tool, the governance burden shifts from reading code to reviewing canvases, which is only an improvement if someone actually reviews them. None of these limits argues against the service; they argue for placing it correctly, which is most of what good platform design is.

A six step adoption framework

Bringing it together

OCI Data Integration takes the least glamorous layer of the data platform, scheduled batch movement, and makes it declarative, serverless, and billed only when working. The wins are operational as much as financial: no ETL servers to patch, no capacity to guess, and pipelines that live in a governed service instead of one engineer's home directory. The discipline still has to be brought: incremental extracts, reusable flows, and lakehouse zone targets are design choices the service enables but does not enforce. Our implementation practice builds these estates on a fixed project fee, migration of the existing job swamp included, and an assessment is the quickest way to size that effort honestly.