Centralized Logging Architecture on OCI

Logging starts simply. An application writes lines to a file on the machine it runs on, and when something goes wrong you log into that machine and read the file. This works until it does not, and it stops working at exactly the moment you most need it. The machine you want to investigate may have been replaced, the container may have been destroyed, the incident may span several machines whose logs you would have to gather and align by hand, and in any case logging into production systems to read files does not scale past a handful of servers. Centralised logging solves all of this by gathering log data from everywhere into a single store, where it persists independently of the things that produced it and where it can be searched and correlated as one body of evidence.

Why local logs fail you

The case for centralisation is really a list of the ways local logs let you down. They disappear when the machine does, which on a platform of ephemeral instances and containers is constantly. They are scattered, so investigating anything that touches more than one machine means collecting files from each and lining them up by hand. They are hard to search, because grepping through files on many machines is slow and clumsy. And they require access to production systems to read, which is both a security concern and an operational drag. Each of these is an inconvenience on a small system and a serious impediment on a large one. Centralisation removes all of them at once by separating where logs are produced from where they are kept.

The shape of a logging pipeline

A centralised logging architecture is a pipeline with a few distinct stages, and understanding them helps you reason about where things can go wrong. At the source, applications and infrastructure emit log data. Collection gathers that data from the sources, ideally as it is produced, so that nothing is lost when a source disappears. Transport carries the collected data to the central store, and along the way it may be parsed and enriched, turning raw lines into structured records that are easier to query. Storage holds the data, balancing how much is kept against the cost of keeping it. And finally search and analysis is where humans actually use the logs, querying across the whole estate to investigate problems. A weakness at any stage undermines the whole, so each deserves attention.

Structure beats raw text

A line of free form text in a log file carries information that a human can read but a machine struggles with. The same event recorded as a structured record, with named fields for the things that matter such as the time, the severity, the service, and the relevant identifiers, becomes something you can query precisely and aggregate meaningfully. Centralised logging is far more powerful when the data flowing into it is structured, because then you can ask questions like show me all errors from this service for this customer in this window and get a precise answer rather than a pile of text to read. Investing in structured logging at the source, or parsing logs into structure during transport, is what turns a log store from an archive into an investigative tool. The deeper analytical capabilities this unlocks are explored in OCI logging analytics.

A framework for designing centralised logging

A logging architecture holds up when it is designed around a few deliberate decisions rather than assembled by accident.

These decisions, taken up front, are what separate a logging pipeline that is useful and affordable from one that is either too sparse to help or too expensive to justify.

Retention and the cost question

Logs accumulate relentlessly, and storage is usually the largest ongoing cost of a logging platform, so retention is a decision that deserves real thought rather than a default. The value of a log falls off with age. Recent logs are investigated constantly, logs from last month occasionally, logs from last year almost never except for compliance reasons. A sensible architecture reflects this by keeping recent data readily searchable and moving or expiring older data according to how likely it is to be needed and what rules require it to be kept. Treating all logs the same, holding everything in fast searchable storage forever, is the most common way a logging bill grows out of control. The discipline here connects to the wider practice of monitoring and observability and to keeping the cost of operations in proportion to its value.

One place where evidence survives

The reason centralised logging matters is that it makes the evidence available when you need it, which is the moment something has gone wrong, often on a machine that no longer exists. By separating where logs are produced from where they are kept, gathering them as they are written, structuring them so they can be queried, and retaining them sensibly, you turn a scattered and perishable resource into a single durable body of evidence that spans the whole estate. This is foundational to operating anything on OCI seriously, and it underpins everything from incident investigation to tracing requests across services. It is a central part of the practice described in the complete monitoring and observability guide. When your logs are scattered and vanish when you need them, our OCI monitoring and observability practice designs the central pipeline that keeps them.