How to use this tool

Agent Observability Pack Generator — Guide

A walkthrough of the Observability Pack Generator: what each setting controls, what the generated alarms monitor, how the three output formats differ, and how to deploy your observability stack to AWS.

What is this tool for?

You cannot operate a payment agent in production without knowing when it is failing, slowing down, or approaching its limits. CloudWatch alarms, X-Ray tracing, and DLQ monitoring are the minimum observability requirements for any payment agent — but setting them up from scratch is time-consuming and error-prone.

The Agent Observability Pack Generator produces a complete observability configuration for your payment agent stack in seconds. Configure your stack name, environment, and feature toggles — the tool generates CloudWatch alarms, a dashboard, X-Ray tracing config, and DLQ monitoring for all 6 standard payment agent types.

The output satisfies PCI DSS Requirements 10.2, 10.3, and 10.5 — audit event logging, log protection, and log retention. File headers in the generated output reference these requirements explicitly so your QSA can map each resource to the control it satisfies.


Step-by-step walkthrough

Step 1 — Configure stack settings
  1. Stack name: the identifier used as a prefix for all generated AWS resource names. Use lowercase and hyphens — e.g. payment-agents-prod. The tool auto-sanitises the name (converts uppercase to lowercase, removes invalid characters) to ensure it is a valid AWS resource name.
  2. Environment: Production, Staging, or Dev. This controls log retention (365 / 90 / 14 days respectively), whether SNS alerts are generated, and the alarm evaluation periods. Production settings satisfy PCI DSS Req 10.5 log retention requirements.
  3. X-Ray tracing: toggle on to include X-Ray tracing configuration in the output. Adds an X-Ray group resource and enables active tracing on Lambda functions. Recommended for Production — enables distributed trace analysis per payment execution.
  4. DLQ monitoring: toggle on to add CloudWatch Alarm and dashboard widget for Dead Letter Queue message count. If the DLQ receives any messages, the alarm fires immediately — a non-zero DLQ count means at least one payment intent has failed all retries.
Step 2 — Select agent cards to include
  1. The 6 agent cards (Payment Router, Fraud Detector, Settlement Agent, Compliance Checker, Notification Agent, Webhook Processor) can each be toggled on or off.
  2. Each selected agent adds its own set of alarms and a dashboard row to the output. Only select agents you are actually deploying — unused alarm resources add noise and cost.
  3. The alarm preview panel (right side of each agent card) shows the colour-coded severity level (red = critical, amber = warning, purple = info) for each alarm before you generate.
Step 3 — Export and deploy
  1. Switch between Terraform HCL, CloudWatch Dashboard JSON, and CloudFormation YAML tabs. All output is real-time — no generate button needed.
  2. Click Export to download the file as {stackName}-observability.{tf|json|yaml}.
  3. For Terraform: place the .tf file in your infrastructure repo and run terraform apply. For Production, the SNS topic ARN will be output — subscribe your on-call tool (PagerDuty, Slack, email) to it.
  4. For CloudWatch Dashboard JSON: go to CloudWatch → Dashboards → Create Dashboard → Source, paste the JSON, and save. The dashboard will populate with data once your agents are running.
  5. For CloudFormation: run aws cloudformation deploy --template-file observability.yaml --stack-name {stackName}.

Standard alarms included per agent

Each agent type generates this standard set of alarms. Alarm thresholds are tuned for production payment workloads — review and adjust them to match your actual traffic volume before treating them as definitive.
PaymentFailureRatecriticalFires when payment execution failures exceed threshold. First alarm any on-call engineer should check.
AgentInvocationErrorscriticalLambda or ECS task errors for the agent. Indicates the agent runtime itself is failing, not just a downstream service.
AgentThrottleswarningLambda concurrency throttles. Agent invocations are being queued or dropped because concurrency limit is hit.
DLQMessageCountwarningMessages accumulating in the Dead Letter Queue — failed payment intents that need investigation and potential replay.
AgentDurationwarningAgent execution time approaching timeout limit. Indicates slow downstream calls or logic that needs optimisation.
SpendLimitApproachinfoAgent spend is approaching the configured daily alert threshold. Early warning before the limit is hit.

Key terms

CloudWatch AlarmAn AWS monitoring resource that watches a single metric and transitions between OK, ALARM, and INSUFFICIENT_DATA states based on a threshold. Can trigger SNS notifications, Lambda functions, or EC2 actions.
X-Ray TracingAWS distributed tracing service. Instruments Lambda, API Gateway, and DynamoDB to produce per-request traces showing exactly where time is spent across services.
DLQ (Dead Letter Queue)An SQS queue that receives messages after all retry attempts have been exhausted. A non-empty DLQ means payment intents have failed and need manual review or replay.
Log RetentionHow long CloudWatch Logs stores log events. PCI DSS Req 10.5 requires 12 months minimum for in-scope systems, with 3 months immediately accessible. Production environment sets 365 days.
PCI Req 10.2Requires audit logs to record all access to cardholder data, all actions taken by any user with root or admin privileges, and all failures of access control mechanisms.
PCI Req 10.3Requires audit logs to be protected against modification and destruction, using mechanisms such as append-only storage, file integrity monitoring, or centralised log management.
SNS TopicAWS Simple Notification Service topic — a pub/sub endpoint that fan-outs messages to subscribers (email, Lambda, SQS, PagerDuty). The generated Terraform creates an SNS topic for Production alarm notifications.

Frequently asked questions

Ready to generate your observability stack?
Configure your stack, select your agents, and export production-ready CloudWatch infrastructure in under 2 minutes.