aws: tag EC2 instance with detach timestamp on DetachInstance#140
Open
priyeshgpatel wants to merge 1 commit into
Open
aws: tag EC2 instance with detach timestamp on DetachInstance#140priyeshgpatel wants to merge 1 commit into
priyeshgpatel wants to merge 1 commit into
Conversation
When a CycleNodeRequest fails mid-cycle and the healing path is unable
to re-attach the detached instance (e.g. due to missing IAM permissions
or a transient error), the instance continues running outside the ASG —
invisible to the autoscaler, not terminated, and accumulating cost.
Tag the instance with `cyclops:detached-at=<RFC3339 timestamp>`
immediately after a successful DetachInstances call. This makes orphaned
instances trivially discoverable:
aws ec2 describe-instances \
--filters "Name=tag-key,Values=cyclops:detached-at"
Tag failure is non-fatal — logged but does not affect the detach result.
Also add a no-op CreateTags implementation to the fake EC2 client so
tests continue to pass.
5df2a81 to
464409c
Compare
Collaborator
|
@priyeshgpatel hi, thanks for the contribution! The change looks simple, useful, and well contained. It should have an associated test for the behaviour so it is ensured to remain working. If you add tests it should be good to go. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a CycleNodeRequest fails mid-cycle and the healing path is unable to re-attach the detached instance (e.g. due to missing IAM permissions or a transient error), the instance continues running outside the ASG — invisible to the autoscaler, not terminated, and accumulating cost.
Tag the instance with
cyclops:detached-at=<RFC3339 timestamp>immediately after a successful DetachInstances call. This makes orphaned instances trivially discoverable:aws ec2 describe-instances
--filters "Name=tag-key,Values=cyclops:detached-at"
Tag failure is non-fatal — logged but does not affect the detach result. Also add a no-op CreateTags implementation to the fake EC2 client so tests continue to pass.
IAM permissions required
To use this feature, add ec2:CreateTags to the cyclops-operator IAM role policy, scoped to cluster-owned instances and restricted to the cyclops:detached-at tag key only:
{
"Sid": "EC2TagPermission",
"Effect": "Allow",
"Action": "ec2:CreateTags",
"Resource": "*",
"Condition": {
"StringEquals": {
"ec2:ResourceTag/kubernetes.io/cluster/": "owned"
},
"ForAllValues:StringEquals": {
"aws:TagKeys": ["cyclops:detached-at"]
}
}
}
Tag failure is non-fatal — if the permission is missing, cyclops logs the error and continues normally.