Skip to content

aws: tag EC2 instance with detach timestamp on DetachInstance#140

Open
priyeshgpatel wants to merge 1 commit into
atlassian-labs:masterfrom
priyeshgpatel:feat/tag-detached-instances
Open

aws: tag EC2 instance with detach timestamp on DetachInstance#140
priyeshgpatel wants to merge 1 commit into
atlassian-labs:masterfrom
priyeshgpatel:feat/tag-detached-instances

Conversation

@priyeshgpatel

@priyeshgpatel priyeshgpatel commented May 7, 2026

Copy link
Copy Markdown

When a CycleNodeRequest fails mid-cycle and the healing path is unable to re-attach the detached instance (e.g. due to missing IAM permissions or a transient error), the instance continues running outside the ASG — invisible to the autoscaler, not terminated, and accumulating cost.

Tag the instance with cyclops:detached-at=<RFC3339 timestamp> immediately after a successful DetachInstances call. This makes orphaned instances trivially discoverable:

aws ec2 describe-instances
--filters "Name=tag-key,Values=cyclops:detached-at"

Tag failure is non-fatal — logged but does not affect the detach result. Also add a no-op CreateTags implementation to the fake EC2 client so tests continue to pass.


IAM permissions required

To use this feature, add ec2:CreateTags to the cyclops-operator IAM role policy, scoped to cluster-owned instances and restricted to the cyclops:detached-at tag key only:

{
"Sid": "EC2TagPermission",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": "*",
"Condition": {
"StringEquals": {
"ec2:ResourceTag/kubernetes.io/cluster/": "owned"
},
"ForAllValues:StringEquals": {
"aws:TagKeys": ["cyclops:detached-at"]
}
}
}

Tag failure is non-fatal — if the permission is missing, cyclops logs the error and continues normally.

When a CycleNodeRequest fails mid-cycle and the healing path is unable
to re-attach the detached instance (e.g. due to missing IAM permissions
or a transient error), the instance continues running outside the ASG —
invisible to the autoscaler, not terminated, and accumulating cost.

Tag the instance with `cyclops:detached-at=<RFC3339 timestamp>`
immediately after a successful DetachInstances call. This makes orphaned
instances trivially discoverable:

  aws ec2 describe-instances \
    --filters "Name=tag-key,Values=cyclops:detached-at"

Tag failure is non-fatal — logged but does not affect the detach result.
Also add a no-op CreateTags implementation to the fake EC2 client so
tests continue to pass.
@mwhittington21 mwhittington21 force-pushed the feat/tag-detached-instances branch from 5df2a81 to 464409c Compare May 18, 2026 23:51
@mwhittington21

Copy link
Copy Markdown
Collaborator

@priyeshgpatel hi, thanks for the contribution! The change looks simple, useful, and well contained. It should have an associated test for the behaviour so it is ensured to remain working. If you add tests it should be good to go.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants