When to run it
A production alert - latency, error rate, saturation, a failing health check - during hours when the person on call is the least rested engineer on the team.
The page fires.Ana is already investigating.
From a 3 a.m. page to root cause and fix
The playbook in one paragraph
An alert pages the on-call engineer. By the time they open the laptop, Ana has correlated metrics, logs, traces and recent deploys, named the likely cause, and drafted the remediation for approval.
Works with
- Observability: Datadog, Prometheus/Grafana, New Relic, CloudWatch, Splunk
- Incident tools: PagerDuty, Opsgenie
- Kubernetes and cloud APIs (read-only)
- Git, CI/CD and IaC
What Ana needs from you
- Which alerts Ana investigates automatically
- Runbooks, if you have them
- Which remediation actions are allowed and who approves them
How Ana runs it
Step by step.
5 steps from trigger to done - each one visible to your team as it happens.
- 01
Correlate
Ana pulls the alert context and correlates metrics, logs and traces across the affected services with recent deploys, config changes and infrastructure events.
- 02
Form and test hypotheses
Ana lists candidate causes and checks each against the evidence - ruling them in or out instead of guessing - and posts its reasoning in the incident channel.
- 03
Propose remediation
Ana drafts the fix: a rollback, a config change, a scaling adjustment, or a code or IaC change, with the expected effect and how to undo it.
- 04
Act with approval
The on-call engineer approves in Slack or the incident tool; Ana executes through your pipelines and watches the signals recover.
- 05
Write it up
Ana drafts the incident timeline and post-incident review and updates the runbook so the next occurrence is faster.
Who approves what
Approval gates
- Any production change
- On-call engineer approval
- Rollbacks outside business hours
- As your policy defines
- Runbook updates
- Service owner review
What you get
Deliverables
- 01Root-cause analysis with evidence links
- 02Executed, verified remediation
- 03Incident timeline and post-incident review draft
- 04Updated runbook
What usually goes wrong
And how Ana handles it.
Risk
Correlation mistaken for cause
Ana tests hypotheses against evidence and says how confident it is, including what it could not verify.
Risk
Automated actions making things worse
Every action is gated, reversible, and verified against the original signal.
FAQ
Common questions.
Only actions your policy explicitly allows. By default, production changes need an engineer’s approval.
Keep going
Related playbooks.
SRE / DevOps
Reconciling infrastructure drift
Hot-fixes in the console and manual changes pile up until Terraform plans become scary. Ana finds every drifted resource, decides with you whether code or cloud is right, and brings them back in line.
Read the playbook →SRE / DevOps
Rightsizing cloud spend without breaking SLOs
The cloud bill keeps climbing and nobody is sure which resources are safe to shrink. Ana finds idle and oversized resources, checks them against real utilization and SLOs, and ships the changes as reviewed IaC.
Read the playbook →Software
Upgrading a framework across dozens of services
A major framework or runtime version is going end-of-life and it touches every service. Ana plans the migration, upgrades each service in its own branch, fixes what breaks, and hands your team small, tested pull requests instead of one terrifying diff.
Read the playbook →See it in your stack
Run this playbook on your systems.
Thirty minutes, your repositories and cloud, one real instance of this job.