OCI Streaming: Kafka Compatible Event Pipelines

Event streaming has a strange cost profile in most organisations. The value is real: order events, telemetry, audit trails, and integration messages all want a durable, ordered, replayable log between producers and consumers. But the infrastructure that classically delivers it, a self managed Apache Kafka cluster, demands a level of operational care that small and medium platform teams struggle to fund. Brokers need patching, ZooKeeper or KRaft needs babysitting, partitions need rebalancing, disks need watching, and version upgrades need rehearsing. Many teams end up paying a full engineer's attention for a messaging layer that moves a few hundred gigabytes a day.

OCI Streaming is Oracle's answer to that mismatch. It is a fully managed event streaming service with a Kafka compatible API, billed by partition and by data moved, with no brokers, no cluster, and no version upgrades visible to you. This article explains how the service is structured, how the Kafka compatibility actually behaves, what the bill looks like, where Streaming fits among the other movement services in the OCI data platform, and when you should still run Kafka yourself.

The shape of the service

The core object is a stream, which is the equivalent of a Kafka topic: an append only, durable log divided into partitions. Producers write messages to a stream, the service assigns them to partitions by key, and consumers read from offsets at their own pace. Messages persist for a retention window you choose, from 24 hours up to 7 days, and within that window any consumer can replay from any offset. Ordering is guaranteed within a partition, exactly as Kafka consumers expect.

Streams live inside stream pools, which group related streams and carry shared settings: the encryption key, whether endpoints are public or private, and the Kafka compatibility configuration. A private endpoint places the streaming API inside your VCN, which is the right default for anything carrying business data. Each partition supports a defined throughput, 1 MB per second of writes and 2 MB per second of reads, so capacity planning is a multiplication exercise: estimate peak write volume, divide by per partition throughput, add headroom, and that is your partition count.

Consumer groups work the way Kafka consumer groups work. A group of consumer instances divides the partitions of a stream among themselves, the service tracks committed offsets per group, and when an instance disappears its partitions are reassigned to the survivors. This is the behaviour that makes streaming pipelines resilient, and it is the part that is genuinely tedious to operate yourself.

What Kafka compatibility really means

OCI Streaming exposes a Kafka compatible endpoint, which means standard Kafka client libraries, producers, and consumers can connect to it by changing configuration rather than code: bootstrap servers point at the OCI endpoint, authentication uses SASL with an OCI user and auth token, and the application is otherwise unchanged. For the common case, producing and consuming messages with standard clients, the compatibility is real and the migration is a configuration exercise.

The honest caveats sit at the edges. Streaming implements the core produce and consume protocol, not the entire Kafka ecosystem surface. Applications that depend on Kafka Streams state stores, exactly once transactional semantics, log compaction, or very long retention will find gaps: retention tops out at 7 days, compacted topics are not part of the model, and transactions are not supported. A Kafka Connect harness exists for running standard connectors against Streaming, which covers most integration patterns, but a platform built deeply on Kafka Streams topology or KSQL belongs on a real Kafka cluster. Knowing which side of that line your workload sits on is most of the decision.

Managed Streaming vs running Kafka yourself

The cost comparison usually surprises teams in Streaming's favour. A self managed cluster needs at least three brokers on compute shapes with fast block storage, doubled for any serious availability story, plus the quiet salary cost of whoever keeps it healthy. Streaming charges partition hours and data transfer, and a pipeline moving even a terabyte a month costs a small fraction of the infrastructure it replaces. The exception is very high, sustained throughput with long retention, where dedicated infrastructure can win back the gap, but most estates are nowhere near that line.

Where Streaming sits among OCI's movement services

OCI has several services that move data, and placing each correctly avoids both gaps and overlaps. Streaming is the buffer and transport layer for events: it decouples producers from consumers and holds the log. GoldenGate is change data capture from databases, and it can publish captured changes into a stream, which is a powerful combination: database changes become events that any consumer can subscribe to. Data Integration is scheduled batch ETL and does not compete with Streaming at all; it complements it on the slow path. Data Flow runs Spark, including Spark structured streaming jobs that consume from Streaming for continuous transformation. And Service Connector Hub is the glue service that moves data from Streaming into Object Storage, Logging, Monitoring, or Functions without custom consumers, which is the cheapest way to land an event archive in the lakehouse.

For small, point to point messaging needs, a work queue between two services with no replay requirement, OCI Queue is simpler and cheaper than a stream. The distinction is replay: if more than one consumer needs the data, or anyone needs history, it is a stream. If exactly one consumer needs each message exactly once and history is irrelevant, it is a queue.

How the bill behaves

Streaming pricing has two meters: partition hours and data transferred. Partitions are provisioned, so they bill whether or not data flows, which makes partition count the number to govern. The common waste pattern is partition sprawl: a developer sizes a stream for an imagined peak, the peak never comes, and twenty idle partitions bill around the clock. Because per partition throughput is fixed and known, right sizing is straightforward arithmetic from observed peak rates, and resizing is an operational decision rather than a redesign. Data transfer charges track actual volume and are rarely the dominant line.

The discipline that keeps streaming estates cheap is the same one that keeps the rest of the platform cheap: observe actual throughput, size to measured peaks with sensible headroom, and review quarterly. We cover the platform wide version of this in designing an OCI data platform that does not leak money.

A six step adoption framework

Limits to respect

An independent view requires the list of edges. Retention is capped at 7 days, so Streaming is transport, not storage; anything needing history must land in Object Storage or a database. Kafka Streams applications, transactions, and compacted topics do not port. Per partition throughput is fixed, so a single very hot key can saturate one partition while others idle, which is a data design problem the service will not solve for you. And while the Kafka compatibility covers standard clients well, every migration should include a validation phase that exercises failover and rebalancing behaviour under load, because compatible is not identical.

None of this argues against the service. It argues for placing it as what it is: a managed, durable event transport with Kafka semantics, priced for pipelines rather than platforms.

Bringing it together

OCI Streaming removes the least defensible piece of operational burden in a modern data estate, the hand run Kafka cluster, and replaces it with partitioned capacity you buy by the hour. Combined with GoldenGate for database change capture, Service Connector Hub for archiving, and Data Flow for stream processing, it completes the real time path of the OCI data platform. The design work that remains is the work the service cannot do: choosing what becomes an event, keying data so partitions load evenly, and sizing to measurement rather than ambition. Our implementation practice builds these pipelines as part of platform engagements on a fixed project fee, and an assessment will tell you quickly whether your messaging estate belongs on Streaming, Queue, or a cluster you actually need.