Roll Back a Failed Deploy

When a resource fails partway through cdkd deploy, cdkd undoes what that deploy had already done to the stack and exits with a non-zero code. The stack returns to how it was before the deploy. This is called a rollback, and it happens without you asking for it.

For example, suppose MyStack is already deployed and you add a queue and a Lambda function to it. On the next deploy the queue is created and the function fails. cdkd deletes the new queue, leaves the rest of the stack as it was, and reports the function's error. You fix the cause and deploy again.

The rest of this page is for when you want something else:

cdkd deploy MyStack                 # a failure rolls the stack back
cdkd deploy MyStack --no-rollback   # a failure leaves the partial stack
cdkd rollback MyStack               # roll back a partial stack later

What the automatic rollback undoes

The rollback handles each resource according to what the failed deploy had done to it. It works through them in the reverse of the order the deploy used.

cdkd also updates the stack's state, its record of what is deployed, so that state matches AWS after the rollback.

What a rollback cannot bring back

A rollback restores the stack's resources and their configuration. Three things are outside its reach:

• A deleted resource. Once the deploy has deleted a resource, the rollback cannot restore it. It prints a warning and continues.
• Data in a replaced resource. The rollback re-creates the old resource, but a re-created bucket, table or other data-holding resource starts empty. cdkd warns when this happens.
• A change made outside cdkd. An updated resource is set back to the properties cdkd recorded at the last deploy. cdkd does not read the live resource first. If someone edited the resource in the console before the failed deploy, that edit is not restored.

Resources with a DeletionPolicy

A DeletionPolicy on a resource the failed deploy created changes how the rollback removes it:

A resource with no DeletionPolicy is deleted. The exception is an RDS cluster or a standalone RDS instance, which gets a final snapshot first, as it does in CloudFormation.

cdkd records a retained resource in state. What the next deploy does with it depends on the stack:

On a legacy stack, the next deploy takes the resource back only when all of these hold:

• the template still declares the resource, with the same type,
• the template leaves the physical name for cdkd to generate,
• no other cdkd stack has the same resource in its state.

For a resource type that cdkd cannot take back this way, cdkd reports the resource and leaves it for you to delete.

A stack that another state prefix also records

The rollback keeps a resource the failed deploy created, and does not delete it, in one more case: the same stack is also recorded under another state prefix, or cdkd cannot check whether it is. For a nested stack, the check is on its Parent~Child record.

The reason is that a create can take over a resource that already existed under the same name (a declared one, or a generated one in a stack first deployed before generated names carried a suffix). That resource may belong to the other deployment. cdkd warns, names the other prefix when it found one, and leaves the resource in the rollback journal. One stack name per account and region explains the rule and the fix.

A replacement under UpdateReplacePolicy: Retain

When the resource declares UpdateReplacePolicy: Retain, the rollback deletes nothing. The new copy is left running and removed from state. If the deploy had kept the old resource under that policy, cdkd takes the old resource back under management, with its data.

A replacement whose old resource was not deleted

If the deploy could not delete the old resource, the rollback first checks whether the old resource is still there. If it is gone, cdkd re-creates it. If it is still there and AWS had refused the delete (access denied, for example) before cdkd changed anything, or the delete was never sent (a failed UpdateReplacePolicy: Snapshot snapshot), cdkd takes it back under management instead of creating a second one, and warns you to check it with cdkd drift. Otherwise — the delete may still be running, it had already removed part of the resource (an IAM role's policies, for example) before AWS refused it, or cdkd cannot tell — it changes nothing for that resource, keeps the rollback journal, and prints what to do next.

Nested stacks

A nested stack that the deploy updated is rolled back from cdkd's record of what the deploy did to the child stack. When that record is missing, cdkd reports the nested stack as a failed rollback operation.

Keep the partial stack: --no-rollback

With --no-rollback, a failed deploy stops where it failed and keeps everything it created:

cdkd deploy MyStack --no-rollback

Use it when you want to inspect or repair the stack by hand. State describes the partial stack, so you can see which resources were created:

cdkd state show MyStack

In CI, the default rollback is the safer choice.

cdkd saves state after each resource completes. So even if cdkd itself is killed between the failure and the rollback, state still matches what is in AWS, and a later cdkd destroy deletes all of it.

After a deploy that was not rolled back

A stack is left partial in three cases: a failure under --no-rollback, a deploy you interrupted with Ctrlc, and an automatic rollback that itself failed partway. From there you have three ways forward:

cdkd state destroy MyStack also deletes the stack, and it does not need the CDK app.

Roll back later with cdkd rollback

cdkd rollback does for a partial stack what the automatic rollback would have done. It prints what it is about to undo, asks you to confirm, and then undoes it:

# print the plan, confirm, roll back
cdkd rollback MyStack

# the only stack that has a journal
cdkd rollback

# skip the confirmation prompt
cdkd rollback MyStack --force

# roll back everything except MyBucket
cdkd rollback MyStack --orphan MyBucket

The command works from the rollback journal. This is a file in the state bucket, next to the stack's state, that lists every operation the failed deploy completed. cdkd writes it whenever a deploy ends without a completed rollback. cdkd rollback goes through the journal in reverse and applies the rules described above.

Because the journal has everything the command needs, cdkd rollback does not synthesize your app. It works even when the app no longer compiles, which is a common reason to roll back.

The next successful deploy deletes the journal, and so does cdkd destroy. A nested stack's journal is kept until the top-level stack deploys successfully.

Flags you are likely to need

By default the rollback skips the resource whose operation failed, because that resource is in an unknown condition in AWS. --revert-failed includes it. The flag also works after a clean automatic rollback, because cdkd keeps the failed resource's journal entry for this purpose.

Exit codes

After an exit 2 caused by a failed operation, cdkd keeps the journal. Run cdkd rollback again, and it skips what is already rolled back.

Edge cases

A resource you imported after the failed deploy

If you run cdkd import on a resource after the failed deploy, the rollback leaves that resource alone. cdkd import marks the resource in the journal, and the rollback skips the journal's operations for it. The plan lists each skipped operation as adopted by cdkd import after this deploy.

The rollback also leaves an operation alone, with a warning, when the journal recorded a different resource under the same logical ID. Operations of a deploy that ran after the import are rolled back as usual.

A failed create that still made its resource

A create call can fail after AWS has already made the resource. Such a resource is not in the stack's state, and only the journal lists it.

The automatic rollback deletes it, following its DeletionPolicy. The one exception is a stack that another state prefix also records.

When the automatic rollback did not run, a later cdkd rollback, cdkd destroy or successful deploy deletes it instead, but only when cdkd can tell that the resource in AWS is the one the failed create made. That requires two things:

• no other stack's state holds the resource,
• for a resource identified by a name you can choose, AWS still reports the identity that the failed create recorded.

The second check depends on the resource type:

When cdkd keeps the resource, cdkd rollback and cdkd destroy print a warning with its physical ID and exit 2. Delete the resource by hand if nothing uses it.

cdkd rollback covers the remaining cases, and --drop-failed for a journal entry cdkd cannot act on.

Related

• cdkd rollback: every flag, the journal, the DeletionPolicy table and the known limitations
• cdkd destroy: tearing a stack down instead of rolling it back
• Review Past Deploys: reading back what a failed run did