How to use this tool

Failure & Fallback Playbook Generator — Guide

A walkthrough of the Failure Playbook Generator: what each question controls in the runbook, what the five failure scenarios cover, and how to use the exported Markdown in your on-call and incident response process.

What is this tool for?

Payment agents fail in predictable ways. An API times out. A circuit breaker opens. A duplicate payment slips through an idempotency gap. DynamoDB gets throttled under load. An agent hits its spend cap. Without a documented runbook, every incident is an improvised scramble — and that scramble is how small failures become major outages or compliance events.

The Failure Playbook Generator covers Agent Safety Layer 7 — the incident response layer. It takes 8 questions about your agent's configuration and generates a complete Markdown runbook covering all 5 standard payment agent failure scenarios, with escalation matrices and CloudWatch diagnostics queries included.

The output is a production-ready runbook Markdown file — drop it into your wiki, your on-call tool, or your incident management system.

Step-by-step walkthrough

Step 1 — Select agent type
  1. Click the chip that best describes the agent you are generating the runbook for. The 6 options are: Payment Router, Fraud Detector, Settlement Agent, Refund Agent, Compliance Checker, and Webhook Processor.
  2. The agent type changes the transaction types shown in Step 2, the default downstream services in Step 2, and the specificity of the generated runbook language.
Step 2 — Answer the 8 configuration questions
  1. Transaction types (multi-select): what types of payment operations this agent executes. Affects which failure scenarios are most relevant and the runbook's priority order.
  2. Downstream services (multi-select): the AWS services and external APIs your agent depends on. Each selection adds a dependency-specific section to the runbook. Select all services the agent genuinely calls — even indirectly.
  3. Timeout behaviour: what the agent does when a downstream call exceeds its timeout. Options are Retry Immediately, Retry with Backoff, Circuit Break, or Dead Letter Queue. This appears in the runbook as the primary timeout response procedure.
  4. Retry window: the total time window over which retries are attempted before giving up. Used in the runbook to define when escalation is triggered.
  5. Customer impact: the visible effect on customers when this agent fails. Used to set the severity level and communication steps in the runbook.
  6. On-call team: the team or rotation handle for first-line escalation. Appears in the escalation matrix. Replace the default placeholder before publishing.
  7. Monitoring tool: where your team monitors alerts. Options include CloudWatch, Datadog, PagerDuty, Grafana, and New Relic. Changes the alert detection steps in the runbook.
  8. RTO: your Recovery Time Objective — the maximum acceptable time to restore service. Sets the urgency level and time-boxed steps in the runbook procedures.
Step 3 — Generate and export the runbook
  1. Click Generate Runbook. The output panel populates with a complete Markdown document.
  2. Review the Escalation Matrix section — ensure the team handles and severity levels match your actual on-call rotation.
  3. Copy the three CloudWatch Logs Insights queries and test them against your agent's log group in the AWS Console before an incident happens.
  4. Click Export Runbook to download {agentType}-failure-runbook.md. Store it in your team wiki, Confluence, or PagerDuty runbook library.
  5. Update the [payments-oncall — update before publishing] placeholder with your actual on-call handle before sharing with your team.

The five failure scenarios covered

These scenarios are derived from the most common payment agent failure modes observed in production AWS environments. Every generated runbook covers all five — the content adapts based on your configuration answers.
API TimeoutPayment processor or downstream API does not respond within the configured timeout. Runbook covers: detection via CloudWatch, retry logic, circuit breaker trigger, customer communication.
Circuit Breaker OpenFailure threshold exceeded — the agent has stopped sending requests to a degraded service. Runbook covers: detecting circuit state, fallback execution path, recovery probing, re-enabling traffic.
Duplicate / Idempotency BreachThe same payment was triggered twice and the idempotency check failed to prevent duplication. Runbook covers: detecting the duplicate, reversing the second charge, root cause analysis.
DynamoDB ThrottleDynamoDB provisioned capacity was exhausted — spend counters, idempotency records, or config reads are failing. Runbook covers: on-demand capacity switch, backoff config review, SLA impact assessment.
Spend BreachAgent hit a spend limit (per-transaction, daily, or hard cap). Runbook covers: immediate freeze verification, manual review of recent transactions, re-enabling procedure.

Key terms

RTORecovery Time Objective — the maximum acceptable duration of a service disruption. Defined by business requirements and SLA commitments. Sets the urgency level for runbook procedures.
Escalation MatrixA table in the runbook specifying who to contact at each severity level and via which channel (phone, Slack, PagerDuty). Ensures the right person is involved without improvisation during an incident.
Dead Letter Queue (DLQ)An SQS queue where messages are sent after failing all retry attempts. For payment agents, a DLQ is essential — it means failed payment intents are not lost and can be replayed or investigated after the incident.
Post-incident ReviewA structured template (also called a blameless postmortem) completed after an incident to document root cause, timeline, impact, and preventive actions. Included in the generated runbook. Required by PCI DSS Req 12.10.6.
CloudWatch Logs InsightsAWS service for querying structured log data. The generated queries can be run directly in the Logs Insights console to diagnose failures using your agent's actual log data.
Agent Safety Layer 7The incident response layer in an agent defence-in-depth architecture. Assumes all other layers (identity, network, idempotency, secrets, observability, spend controls) may occasionally fail and provides the human response procedures for when they do.

Frequently asked questions

Ready to generate your failure runbook?
Answer 8 questions about your agent and export a complete on-call runbook in under 5 minutes.