Service quotas, machine images and other gaps in the recovery region

Why can a correct template still fail?

Infrastructure as code describes what should exist. It does not guarantee that the recovery region will accept it. Three things commonly stop a rebuild that looks correct on paper.

1. Service quotas. Cloud accounts carry per-region limits on instances, IP addresses, load balancers and many other resources. A recovery region that has never run production load often has lower limits than the primary.
2. Machine images. An image built and stored in the primary region is not automatically available in the recovery region. If the template refers to it, the rebuild stops.
3. Drift. Settings changed by hand in production after the template was applied will not be in the rebuilt environment, and nobody may notice until the application misbehaves.

What does AWS recommend?

The AWS whitepaper on testing disaster recovery puts the principle plainly: "the only error recovery that works is the path you test frequently". It tells teams to manage configuration drift at the DR region and to check that machine images and service quotas there match what the workload needs.

What does CISA add for ransomware cases?

The CISA #StopRansomware Guide recommends golden images of critical systems and using infrastructure as code to deploy and update cloud resources, with backups of the template files kept offline. A template that an attacker can delete or alter is not a recovery asset.

How do the tools here help?

None of the tools scored here publishes a quota check on the pages we reviewed, so ask about it directly. Drift is where they differ most: Firefly documents drift detection and change tracking, ControlMonkey documents drift detection and remediation, and Gambit Security detects infrastructure drift; Arpio does not publish drift detection, and Commvault Cloud Rewind's page mentions drift analysis only in a third-party quote. Arpio documents failover testing that does not disrupt production, which is the most direct way to find these gaps before an incident.

How often should you check?

Quotas and images change whenever production changes, so a check done once at setup goes stale. Tie the checks to two events: every scheduled recovery test, and every change that adds a new resource type or a new image to a critical application. Keep the result with the test record, so the next test starts from a known list rather than from memory.

A short list of what to compare is enough: instance and IP limits for the resource types the application uses, the image identifiers its templates reference, and the settings that differ between the live environment and its template.

What should you take from this lesson?

Add three lines to every recovery test: compare service quotas between regions, confirm every image the templates reference exists in the recovery region, and compare the live environment with its template before you start.