Designing observability for a small, high-trust firm

An alert makes a demand on someone’s attention. In a small team, that demand should come with enough context to act: the affected service, the consequence for the firm, and the next useful step. I would start an observability program with the work people need to complete, then choose the signals that help us notice a problem and understand it.

Source

Monitor the work people are trying to do

Start with a short list of observable outcomes. Can an authorized person sign in? Can they open a current document? Can a scheduled transfer complete with the expected records? A platform can report healthy infrastructure while one of these workflows is broken.

A useful check follows enough of the path to reveal a meaningful failure. For a recurring export, that might mean confirming that a new file exists, its timestamp is current, and its contents meet a few agreed checks. A successful scheduler event covers only the attempt.

Google’s monitoring guidance distinguishes externally visible behavior from internal signals and explains why noisy alerts become expensive. I would apply that principle by giving every alert a clear relationship to the work at risk.

Give different signals different jobs

Availability checks tell us whether a service appears usable. Diagnostic logs help explain how an operation behaved. Security detections identify activity that deserves investigation under a different set of assumptions. These signals can inform one another, but they need distinct owners, retention decisions, and response paths.

Figure 1. Give each kind of signal its own job: an availability check leads to the next useful check, a diagnostic log is collected so it can be searched during an incident, and a security detection goes to an investigation path separate from availability checks. The open rings are the decisions the piece leaves to you, such as how long the evidence is useful.

Describe this figure

Three kinds of signal side by side, each with its owner, its evidence and retention, and its response path. Availability checks tell us whether a service appears usable. Their owner is the workflow owner, reached through the agreed channel; their evidence and retention are left for the reader to decide; their response path is the next useful check, with an urgency of page now or next business day. Diagnostic logs help explain how an operation behaved. Their owner is left for the reader to decide; their evidence is a record identifier, outcome, timestamp, and error category; their response path is to collect the causes so they can be searched during an incident. Security detections identify activity that deserves investigation. Their owner is the person responsible for investigation; their evidence is preserved for investigation and collected regardless of alerting; their response path, highlighted, is an investigation path separate from availability checks.

For example, repeated failed sign-ins might indicate a forgotten password, a configuration change, or hostile activity. A service-health dashboard cannot settle that question. The detection needs enough context for the person responsible for investigation, and that context must be handled with appropriate access controls.

Avoid collecting sensitive document contents simply because they are available to log. Prefer a record identifier, outcome, timestamp, and relevant error category where those are sufficient. Decide how long the evidence is useful and who can see it.

Write the response before enabling the alert

An actionable alert should identify the affected workflow, the observed symptom, the time, the person or service expected to respond, and the next useful check. It should also distinguish urgency. A missed overnight synchronization and a firm-wide sign-in failure may deserve very different handling.

Imagine a daily relationship-data export that is needed before the next working day. The response could be: confirm the source is available, check the latest run, inspect whether the destination already received the data, then retry only if doing so will not create duplicates. If the deadline is at risk, notify the workflow owner through the agreed channel.

That small amount of preparation changes an alert from a notification into a usable starting point.

Exercise the whole path

Create a controlled failure and follow it through. Did the check detect it? Did the intended person receive the alert? Could they understand the consequence and find the instructions? Was recovery confirmed in the service itself?

Review alerts that repeatedly require no action. Adjust thresholds, improve context, or move the signal into a periodic review. Preserve signals needed for investigation even when they do not warrant an immediate interruption.

The result should be a monitoring matrix that a second person can understand. Each row connects a workflow to a symptom, a signal, a response, and an owner. That is a more useful foundation than a large dashboard whose significance depends on one person being available.

Proposal

What I would test next

I would measure how quickly a second operator can recognize and resolve one controlled failure before and after adding the proposed signals. If the extra data increases investigation time, the design needs better context or fewer distractions.