[Bug]: Namespace resources block namespace deletion

[sjiekak]

What happened?

When a namespace containing Crossplane managed resources (e.g. roles.iam.aws.m.upbound.io, lbs.elbv2.aws.m.upbound.io, securitygroups.ec2.aws.m.upbound.io, vpcendpointservices.ec2.aws.m.upbound.io) is deleted, the namespace gets stuck in Terminating indefinitely.

I initially taught it is an issue with upbound aws and filed crossplane-contrib/provider-upjet-aws#2091 but I see many projects are using Track, so it might be related to the runtime behavior

kubectl describe on the namespace shows that the managed resources still carry finalizer.managedresource.crossplane.io, and content cannot be removed:

Status:       Terminating
Conditions:
  NamespaceDeletionDiscoveryFailure            False   ResourcesDiscovered   All resources successfully discovered
  NamespaceDeletionGroupVersionParsingFailure  False   ParsedGroupVersions   All legacy kube types successfully parsed                                                                                                                                                                                                                                                            
  NamespaceDeletionContentFailure              False   ContentDeleted        All content successfully deleted, may be waiting on finalization                                                                                                                                                                                                                                     
  NamespaceContentRemaining                    True    SomeResourcesRemain   Some resources are remaining: securitygroups.ec2.aws.m.upbound.io has 1 resource instances, vpcendpointservices.ec2.aws.m.upbound.io has 1 resource instances                                                                                                                                        
  NamespaceFinalizersRemaining                 True    SomeFinalizersRemain  Some content in the namespace has finalizers remaining: finalizer.managedresource.crossplane.io in 2 resource instances                                                                                                                                                                              

Looking at the resource conditions, the controller is trying to create a providerconfigusages.aws.m.upbound.io in the namespace that is being terminated, which the API server (correctly) rejects:

cannot initialize the Terraform plugin SDK async external client:
    cannot get terraform setup: cannot track ProviderConfig usage:
        cannot apply ProviderConfigUsage:                                                                                                                                                                                                                                                                                                                                         
            cannot create object: providerconfigusages.aws.m.upbound.io "2bb7202e-df40-4352-9702-9a3a27f56a91" is forbidden:
                   unable to create new content in namespace sample-bug-ns because it is being terminated                                                                                                                                                                                                                                                                         

Because this happens on every reconcile of the MR — including the deletion reconcile — the provider never reaches the actual delete call, never removes the finalizer, and the namespace cannot finalize.

How can we reproduce it?

It reproduces with any namespaced (*.m.upbound.io) managed resource, and is easier to observe when more than one MR lives in the namespace.

1. Create a namespace.
2. Create one (or more) Crossplane managed resource(s) in it
3. Delete the namespace.

Expected: the namespace finalizes once the MRs are gone.

Actual: the namespace stays in Terminating forever; MRs remain with finalizer.managedresource.crossplane.io; provider logs show repeated ProviderConfigUsage create failures against the terminating namespace.

What environment did it happen in?

• Crossplane version: 2.2.1
• provider-upjet-aws version: 2.3.0
• Kubernetes version: v1.35.3
• Kubernetes distribution: EKS

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

roles.iam.aws.m.upbound.io
lbs.elbv2.aws.m.upbound.io                                                                                                                                                                                                                                                                                                                                                        
securitygroups.ec2.aws.m.upbound.io
vpcendpointservices.ec2.aws.m.upbound.io                                                                                                                                                                                                                                                                                                                                          

(any namespaced *.m.upbound.io MR is affected)