Skip to content

Expiring assumed role token after 6 hours when using FargateCluster with max duration set to 12 hours #422

@jimhonetech

Description

@jimhonetech

Describe the issue:
I was advised on SO to post here, not sure its a bug, I may just not know something...

Im running dask locally and spinning up fargate cluster in AWS.
Its assuming a role correctly and starting up the cluster.
After six hours its permissions fail, I have set the maximum session duration on the role to 12 hours but it seems to be getting a defailt 6 hour token anyway and not asking for a 12 hour token.

Is there a way to instruct it to get a 12 hour token or do I need to refresh the session token somehow?

Minimal Complete Verifiable Example:
Cant post a full Minimal Complete Verifiable Example, would need something that goes over 6 hrs.
But our cluster is setup like:

cluster = FargateCluster(
        cloudwatch_logs_group=f"{project}_log_group",
        cluster_name_template=f"{project}-{uuid}",
        environment={"DEFAULT_GSF_VERSION": gsf_version},
        image=f"{aws_account_id}.dkr.ecr.eu-west-2.amazonaws.com/{project}:latest",
        n_workers=9,
        scheduler_cpu=4096,
        scheduler_mem=30720,
        scheduler_timeout="240 minutes",
        security_groups=[aws_security_group_id],
        task_role_arn=f"arn:aws:iam::{aws_account_id}:role/{project}ECSTaskRole",
        worker_cpu=4096,
        worker_mem=30720,
        worker_extra_args=algorithm_extra_cluster_args,
        fargate_use_private_ip=use_private_ip,
        vpc=vpc,
        subnets=subnets,
        skip_cleanup=True,
        platform_version="LATEST",
        shutdown_on_close=True,
        mount_points=mount_points,
        volumes=volumes,
    )
    cluster.adapt(minimum=9, maximum=80)
    logger.info(f"dashboard_link is: {cluster.dashboard_link}")

Anything else we need to know?:

I have set the maximum session duration on my role to 43200(12 hours), seems it still only gets a 6 hour token.
I tried adding duration_seconds=43200 to AWS config files hoping to override the default 6 hour token, still lost perms after 6 hours

Environment:

  • Dask version:022.12.0
  • Python version:3.8.3
  • Operating System:linux
  • Install method (conda, pip, source):poetry

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions