Use one row per signal. Identify whether it supports availability, security investigation, or diagnosis. Record times with a timezone.
Review owner: Date:
Services in scope and their tier:
What "healthy" means for each, stated as a user-visible outcome (for example, a document opens within a stated time, a sign-in completes, a scheduled job finishes before the business needs its output):
Signal matrix
| Workflow | Symptom / consequence | Signal and threshold | Type (availability, security, diagnostic) | Urgency (page now, next business day, periodic review) | Response owner / escalation | First useful action | Evidence and retention |
|---|---|---|---|---|---|---|---|
For each Tier 1 service, check that the matrix covers the four basic questions: is it slow, is it busy, is it failing, is it running out of something. Alert on what the user experiences; collect the causes for diagnosis.
Sources that must be collected for security investigation regardless of alerting (sign-in, administrative change, sharing, mail-flow rule, device enrollment):
Clock source, log retention period, and how logs are protected from alteration:
Exercise record
Controlled failure: Stop condition:
| Milestone | Expected | Observed / time | Exception and owner |
|---|---|---|---|
| Problem detected | |||
| Intended responder notified | |||
| Instructions located and understood | |||
| Service recovered and checked | |||
| Alert cleared and record closed |
Review decisions
- [ ] Each immediate alert has an actionable response and a named responder outside business hours, or is downgraded.
- [ ] Noisy signals have been tuned, retired, or moved to periodic review; the count of alerts per week is recorded.
- [ ] Silence is tested: a stopped collector or expired credential produces its own alert.
- [ ] Logs avoid secrets and unnecessary sensitive content.
- [ ] Access to monitoring data and retention are documented.
- [ ] Security detections have an investigation path separate from ordinary availability checks.
- [ ] Vendor-hosted services have a status source and a way to confirm the vendor's claim independently.
- [ ] Every incident in the period has been checked against this matrix: was it detected by a signal here, by a user, or not at all?
Quarterly summary
| Measure | This period | Previous period | Note |
|---|---|---|---|
| Alerts raised / acted on | |||
| Incidents detected by monitoring vs reported by people | |||
| Median time from symptom to responder acknowledgement | |||
| Signals added / retired |
Common failure modes
- Alerts go to a shared mailbox nobody watches after hours.
- Thresholds are copied from a vendor default and never compared to actual usage.
- Diagnostic detail is collected but nobody can search it during an incident.
- A collector fails silently and the dashboard stays green.
Next exercise / trigger:
This matrix covers detection and first response. It does not cover the incident decisions themselves; use the incident tabletop and decision log for that.