Setting Up OCI Alarms and Alerts

There is a particular kind of dread that comes with a phone buzzing at three in the morning. If the team trusts its alerts, that buzz means something real is wrong and needs attention now, and they get up and deal with it. If the team does not trust its alerts, because nine times out of ten the buzz is noise, then the one time it matters gets ignored along with the rest. Everything good about alerting and everything bad about it flows from this question of trust, and trust is built or destroyed in how the alarms are set up. This guide is about setting up alarms that earn trust, the kind that fire when they should, stay quiet when they should, and carry what a responder needs.

Alert on symptoms, not causes

The most important principle in alerting is to alert on what users feel, not on every internal condition that might contribute to it. Users do not experience CPU utilisation or memory pressure directly. They experience slow responses, errors, and unavailability. An alarm on high CPU might fire when nothing is actually wrong, because the system is busy but coping fine, or it might stay silent while users suffer for a reason CPU never showed. An alarm on the symptom, such as elevated error rate or response time, fires when and only when users are actually affected, regardless of which underlying cause produced it. Symptom based alerts are both more reliable and less noisy, because they track the thing you actually care about rather than a proxy for it.

Thresholds and dwell time

Two settings decide whether an alarm is steady or jumpy. The threshold is the value that counts as a problem, and the dwell time is how long the condition must hold before the alarm fires. Both need thought. A threshold set too close to normal operating values will trip on routine variation, while one set too far away will let real problems run before anyone is told. The dwell time guards against transient spikes. A metric that briefly crosses a line and immediately falls back is usually noise, and firing on it teaches people to ignore the alarm. Requiring the condition to persist for a sensible period before firing filters out these blips while still catching sustained problems quickly. Tuning threshold and dwell time together is most of the work of making an alarm trustworthy.

Notice that severity belongs in the same conversation. If every alarm is marked critical, the marking carries no information and responders cannot triage. Reserve the highest severity for the alerts that truly demand someone wake up, and let the rest sit at a level that says look at this during the day.

Give the alert context to act on

An alert that says something is wrong without saying what or where forces the responder to start every investigation from scratch, which wastes the most expensive minutes of an incident. A good alert carries context. It names the affected resource, states the condition that fired, indicates the severity, and ideally points toward the next step, such as a relevant dashboard or runbook. The goal is that a responder reading the alert on their phone already knows roughly what they are dealing with and where to look, rather than having to log in and hunt for the basics. This context is set up when the alarm is created, in the message it sends, and the small effort of writing a clear alarm message pays back many times over during every incident it fires for.

Route alerts to the right place

An alert is only useful if it reaches someone who can act on it, which makes routing as important as the alarm itself. OCI alarms send to notification topics, and those topics fan out to destinations such as email, messaging, or an on call paging system. The setup should ensure that the right people get the right alerts at the right urgency. A critical production alert should page whoever is on call now, through a channel that will wake them, while a low priority informational alert should land somewhere it can be reviewed without interrupting anyone. Mixing these up, by paging people for trivia or by sending genuine emergencies to an unread mailbox, undoes all the care that went into the alarm. Routing is covered in more depth alongside notifications and events.

A framework for setting up an alarm

Each new alarm benefits from a consistent thought process. The steps below keep the result trustworthy.

Running every alarm through this sequence produces a set of alerts that responders trust, which is the whole point. An alarm built carelessly is a future false alarm waiting to erode that trust.

Alarms are never finished

The final thing to understand is that alarm setup is not a one time task. Systems change, thresholds that were right become wrong, alarms that mattered become obsolete, and new failure modes appear that need new alerts. An alarm set that is created once and never revisited slowly drifts into noise as the world moves on around it. The healthy practice is to review alarms regularly, retiring the ones that no longer earn their keep and tuning the ones that fire too often or too late. This ongoing tuning is the subject of the deeper discussion of alert fatigue and tuning, and it connects to defining SLOs and SLIs, which give alarms a principled target to aim at. Alarms sit on top of the Monitoring service and form part of the wider practice in the complete monitoring and observability guide. When you want an alerting setup people actually trust, our OCI monitoring and observability practice builds it to the discipline described here.