How to Set Up Budgets and Anomaly Detection Before the Bill Surprises You

Finding out about a spend problem from the invoice means finding out weeks late. The thing that caused it, a forgotten test cluster, a log group with no retention, a NAT gateway suddenly carrying replication traffic, has been running the whole time. The point of alerting on spend is to shorten that gap from weeks to about a day, which is usually the difference between a conversation and an incident.

AWS gives you two mechanisms and they are not interchangeable. A budget is a threshold you set yourself, and it fires when the number crosses the line you drew. Cost Anomaly Detection has no line. It builds a model of what your spend normally looks like and tells you when the shape changes, which catches the thing you never thought to draw a line around.

What a budget actually watches

AWS Budgets tracks cost, usage, Reserved Instance utilization and coverage, and Savings Plans utilization and coverage. For each one you can be alerted on actual spend after it has accrued, or on forecasted spend before it accrues. The forecasted alert is the one worth having, because an actual alert at your limit tells you the money is already gone.

The timing is easy to get wrong. AWS Budgets information is updated up to three times a day, and updates typically occur 8 to 12 hours after the previous update. AWS also states plainly that there can be a delay between when you incur a charge and when you receive a notification, because there is a delay between when a resource is used and when that usage is billed. A budget is not a circuit breaker. Treat it as a signal that arrives within a day, not within a minute.

Create one on the account before you build anything clever. The budgets endpoint lives in us-east-1 regardless of where your workloads run.

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
read -r -p "Monthly cost limit to track against: " LIMIT
read -r -p "Alert email: " ALERT_EMAIL

cat > /tmp/budget.json <<JSON
{
  "BudgetName": "monthly-account-cost",
  "BudgetLimit": { "Amount": "$LIMIT", "Unit": "USD" },
  "TimeUnit": "MONTHLY",
  "BudgetType": "COST"
}
JSON

cat > /tmp/notifications.json <<JSON
[
  {
    "Notification": {
      "NotificationType": "FORECASTED",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 100,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "$ALERT_EMAIL" }]
  },
  {
    "Notification": {
      "NotificationType": "ACTUAL",
      "ComparisonOperator": "GREATER_THAN",
      "Threshold": 80,
      "ThresholdType": "PERCENTAGE"
    },
    "Subscribers": [{ "SubscriptionType": "EMAIL", "Address": "$ALERT_EMAIL" }]
  }
]
JSON

aws budgets create-budget \
  --region us-east-1 \
  --account-id "$ACCOUNT_ID" \
  --budget file:///tmp/budget.json \
  --notifications-with-subscribers file:///tmp/notifications.json

A single notification takes one SNS topic and up to ten email addresses, and a budget takes up to five notifications. That is enough to send the forecast warning to the team that can act on it and the hard threshold to whoever owns the number.

The daily budget is the underrated one. A monthly budget on a growing account spends most of the month comfortably under its limit, so a runaway that starts on the fourth has three weeks to run before anything fires. Change TimeUnit to DAILY in the same JSON and you get a far shorter fuse for the same effort.

What anomaly detection catches that a budget cannot

Cost Anomaly Detection runs machine learning models over your net unblended cost. After billing data is processed it runs approximately three times a day, and because it uses data from Cost Explorer, which has a delay of up to 24 hours, it can take up to 24 hours to detect an anomaly after the usage occurs. A new monitor takes 24 hours before it begins detecting, and a newly used service needs 10 days of historical usage data before anomalies can be detected for it.

You choose what it watches. A monitor is either AWS managed, which tracks every value in a dimension automatically and picks up new values as they appear, or customer managed, which tracks up to ten values you select. The dimensions are AWS services, linked accounts, cost allocation tags and cost categories. Start with the services monitor, since it needs no decisions from you.

aws ce create-anomaly-monitor \
  --region us-east-1 \
  --anomaly-monitor '{
    "MonitorName": "all-services",
    "MonitorType": "DIMENSIONAL",
    "MonitorDimension": "SERVICE"
  }'

If you tag by team or by environment, a managed tag monitor is what turns an alert into an owner. Give it the tag key and it evaluates every value of that key independently.

aws ce create-anomaly-monitor \
  --region us-east-1 \
  --anomaly-monitor '{
    "MonitorName": "per-team",
    "MonitorType": "DIMENSIONAL",
    "MonitorDimension": "TAG",
    "MonitorSpecification": { "Tags": { "Key": "team" } }
  }'

Who should receive which alert

The subscription decides the audience, and the frequency you pick decides the channel. IMMEDIATE notifies as soon as an anomaly is detected and requires an SNS topic. DAILY and WEEKLY are email summaries, the daily one being the top ten alerts from the previous day sorted by cost impact, generated at 00:00 UTC.

Route the immediate ones through SNS into the channel where the engineers who own that workload already are. Send the weekly summary to whoever owns the number. Keep a monthly budget threshold out of the on-call pager, because nobody can usefully act on it at three in the morning and it teaches people to ignore the pager.

The threshold on a subscription is either the anomaly's total cost impact or its total impact percentage, written as a threshold expression. The percentage form is the one that survives growth, since a fixed figure that was meaningful last year quietly becomes noise as the account gets bigger.

aws ce create-anomaly-subscription \
  --region us-east-1 \
  --anomaly-subscription '{
    "SubscriptionName": "engineering-immediate",
    "Frequency": "IMMEDIATE",
    "MonitorArnList": ["MONITOR_ARN"],
    "Subscribers": [{ "Type": "SNS", "Address": "SNS_TOPIC_ARN" }],
    "ThresholdExpression": {
      "Dimensions": {
        "Key": "ANOMALY_TOTAL_IMPACT_PERCENTAGE",
        "MatchOptions": ["GREATER_THAN_OR_EQUAL"],
        "Values": ["50"]
      }
    }
  }'

Anomalies below your threshold are still detected and still recorded. They simply do not notify anyone, and you can review them on the Detected anomalies tab when you have time.

The gaps to know about

Cost Anomaly Detection does not monitor third-party products and services bought through AWS Marketplace, with the exception of third-party foundation models on Amazon Bedrock. If Marketplace is a real line on your bill, a cost budget with the billing entity filter is what covers it.

A budget can also act rather than only alert. Budget actions can apply an IAM policy or a service control policy, or target specific EC2 or RDS instances, running either automatically or after your manual approval. Manual approval first is the sane default. An automatic deny policy that fires on a forecast can stop a deploy at the worst possible moment.

Neither mechanism replaces knowing what your resources are. They tell you something changed. Working out what changed still comes from consistent tagging and from Cost Explorer, which is where a cloud cost optimization engagement usually starts, and the alerting itself is a small part of ongoing AWS cloud management.