Count completed orders

The dashboard is green, every process is running and checkout returns HTTP 200. Customers still cannot finish orders because the payment result never reaches the order record. Monitoring must observe completed customer work, not only healthy components. Start with that outcome, then collect the signals that explain why it fails.

Choose an SLI for quality and an SLO target. An example service might require 99.9% correct order confirmations over 30 days. That is the example's promise, not a standard for every service. Define which attempts enter the denominator and which outcomes count as bad.

Alongside the user-facing measure, we track request rate, errors, latency and resource saturation. They show what is happening to the workload and where headroom is running out. High CPU is useful context, but paging depends on product impact and how quickly the situation can worsen.

Google SRE’s four golden signals are traffic, latency, errors and saturation. Use them to explain checkout health, while a separate completion measure checks whether customers actually receive correct orders within the promised time.

Latency, percentiles and queue age

Separate successful operations from failures. Fast rejections can improve average latency during an outage. Percentiles describe a distribution, but p99 on a small sample is unstable. Interpret latency alongside request volume and important routes.

Averaging the p99 values from several instances does not give us the service p99. We need to combine observations appropriately, for example through compatible histograms. Bucket boundaries should match the objective: coarse buckets around a one-second limit make that limit difficult to assess.

Queues need arrival rate, completion rate and oldest-job age. A hundred jobs can mean seconds or hours. Scheduled work also needs the last successful completion time; no new error does not prove a job ran.

Keep diagnostics affordable

Add pool waits, locks, dependency duration, memory and disk use. These help trace slow checkout to a particular part of the path. Each metric needs an owner, unit and understood observation point.

Label cardinality counts combinations. Twenty routes, five statuses and ten instances allow 1000 combinations for one metric. A user identifier makes growth depend on audience size; histogram buckets multiply the resulting series further.

For route metrics, we keep a template such as /orders/:id. Individual IDs and request context belong in logs or traces when needed. We collect context that helps investigation, rather than copying bodies and secrets because they are easy to capture. Monitoring has a storage and processing budget too.

An error budget for checkout

An SLO of 99.9% allows 0.1% bad operations over its window. In an example with a million requests, that means 1000 bad ones. An observed 1% failure ratio consumes the allowance ten times faster than its permitted pace: burn rate 10.

With uniform traffic over 30 days, that pace would spend a full budget in roughly three days. Actual traffic varies, some budget may already be spent and a short window can be noisy. Use the calculation with observations and service conditions.

Combining a longer window with a shorter one helps: the first shows meaningful budget consumption, and the second checks whether the problem is still happening. Low traffic needs different conditions. One failure out of ten requests is already 10%, so sample size, operation importance and additional checks matter.

The SRE Workbook combines longer and shorter windows to detect significant budget consumption that is still happening. It also calls out low-traffic services: a ratio from a handful of requests needs different handling from a busy checkout.

When to wake someone up

Page when delay materially increases harm and a person can intervene. Gradual disk growth may initially need a ticket; a few hours until exhaustion can require immediate action. Severity depends on remaining time and available mitigation.

An alert needs the affected workflow, scale, start time, owner and a runbook. The runbook explains how to check the cause, limit damage and confirm recovery. "Restart the service" is risky without context: a restart may interrupt work, clear local state or increase load on dependencies.

Group symptoms of one failure and define repeat and escalation rules. A separate page from every instance about the same dependency impedes response. Verify suppression relationships so that grouping does not hide an independent problem.

Try the alert before relying on it

Create a known failure in a safe environment. Confirm the measurement changes, the rule triggers, the on-call receives it, instructions help and recovery is detected. Test missing data and monitoring failure separately.

After an incident, we review missed signals, noise and time to useful action. A noisy alert might have a bad threshold, a bad measurement or a condition that no longer matters. Raising the threshold without finding the cause risks hiding the next failure. Monitoring earns its place when it helps us understand and recover the product.

  • Define a correct checkout outcome and its time limit. State which attempts count, where they are measured and which failures the observation point cannot detect.
  • Separate successful requests, service failures and invalid input. Keep payment visible even when fast catalogue reads dominate the service-wide average.
  • Use bounded route labels and buckets near the latency target. Put individual request histories in controlled logs or traces rather than unbounded metric labels.
  • Choose paging conditions that identify ongoing harm and a useful human action. Give gradual capacity issues a ticket when immediate intervention is unnecessary.
  • Introduce a known safe fault and record measurement, rule firing, delivery and recovery times. Test absent data, grouping and escalation as well as the threshold.
  • Review the first real incident for missed signals and noise. Improve the measurement or response instructions before simply moving a threshold to silence it.
Back