[BUG] `@aws-mdaa/dataops-job` never applies `AmazonDataZoneProject` tag

[marcinc]

Summary

Glue jobs created by @aws-mdaa/dataops-job are not visible in the SageMaker Unified Studio (SMUS) project UI even when projectName is configured. SMUS uses the AmazonDataZoneProject: <datazone-project-id> resource tag to associate Glue jobs with projects.
The @aws-mdaa/dataops-job module never emits this tag, so every job it creates is invisible in the SMUS Data analytics → Data processing jobs section.

Affected version

@aws-mdaa/dataops-job v1.5.0
@aws-mdaa/dataops-project v1.5.0

Component

packages/constructs/L3/dataops/dataops-job-l3-construct/lib/dataops-job-l3-construct.ts
packages/constructs/L3/dataops/dataops-project-l3-construct/lib/dataops-project-l3-construct.ts

Symptom

Glue jobs deployed via the dataops-job MDAA module are visible in the AWS Glue console and in CloudTrail, but do not appear under the SMUS project UI (Data analytics → Data processing jobs). This is true even when projectName is set in the job configuration, which correctly wires up S3, KMS, and the Glue security configuration, but has no effect on SMUS visibility.

Jobs created directly through the SMUS Visual ETL interface appear correctly, so the issue is specific to jobs provisioned by MDAA.

Root cause

SMUS associates a Glue job with a project via the AmazonDataZoneProject: <datazone-project-id> resource tag. The DataZone project ID changes every time the domain is rebuilt -- it is not the same as the project name.

dataops-job has a projectName config field that resolves project-level SSM parameters (S3 bucket, KMS key, security configuration, SNS topic), but it never resolves or applies the DataZone project ID. The AmazonDataZoneProject tag is structurally absent from every CreateJob call issued by MDAA.

Proposed Fix

1. dataops-project-l3-construct.ts — publish DataZone project ID to SSM

After createDataZoneSageMakerResources creates the DataZone/SageMaker project, store the project ID in SSM under a cross-module-readable path so other modules can resolve it at synthesis time:

if (resources?.datazoneProject) {
  this.createProjectSSMParam(
    'ssm-sagemaker-project-id',
    'sagemaker/project/id/default',
    resources.datazoneProject.project.attrId,
  );
}

This uses the existing createProjectSSMParam helper with the cross-module (non-project-internal) path convention, making the parameter available at:

/<org>/<domain>/<projectName>/sagemaker/project/id/default

2. dataops-job-l3-construct.ts — read SSM and tag each Glue job

Inside createJob, after the job is created and within the existing if (job.name && projectName) block, resolve the DataZone project ID from SSM and apply the tag:

const projectIdSsmPath = this.props.naming.ssmPath(
  `${this.props.projectName}/sagemaker/project/id/default`,
  false,
  false,
);
const datazoneProjectId = MdaaStringParameter.valueForStringParameter(this.scope, projectIdSsmPath);
Tags.of(job).add('AmazonDataZoneProject', datazoneProjectId);

The SSM path is resolved as a CloudFormation parameter at deploy time, following the same pattern used by the existing project SSM lookups (kmsArn, projectBucket, etc.). The tag value in the CloudFormation template is a { "Ref": "SsmParameter..." } reference that CloudFormation resolves against the live SSM value during stack deployment.

Workaround

Apply the tag manually after each deploy:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION=eu-west-2
DOMAIN_ID=$(aws datazone list-domains --query 'items[?contains(name,`your-name-here`)].id' --output text --region eu-west-2)
PROJECT_ID=$(aws datazone list-projects --domain-identifier $DOMAIN_ID --query 'items[?contains(name,`project1`)].id' --output text --region $REGION)

aws glue tag-resource \
  --resource-arn "arn:aws:glue:${REGION}:${ACCOUNT_ID}:job/<job-name>" \
  --tags-to-add "{\"AmazonDataZoneProject\": \"${PROJECT_ID}\"}" \
  --region $REGION

The tag must be reapplied whenever the job is recreated, as CreateJob is called fresh and the tag is not in the CDK template without this fix.

Impact

• Severity: High -- every Glue job deployed via dataops-job with a projectName is invisible in the SMUS project UI, breaking the ETL workflow discovery experience.
• Scope: Any deployment using dataops-job with projectName configured and SMUS / DataZone enabled.

[suddash24]

Hi @marcinc,

We confirm the gap. The createJob path in dataops-job-l3-construct.ts never applies DataZone resource tags, and the upstream dataops-project construct does not publish the DataZone project ID to SSM, so downstream modules have no way to resolve it today.

We do want to recalibrate severity here. The Glue jobs execute correctly, produce expected output, and remain fully accessible through the Glue console and API. What breaks is visibility within the SageMaker Unified Studio project view. This is a real gap in the integrated experience, but it does not block data processing or affect IAM permissions. We would classify it as Medium.

On PR #53: the approach is sound. Publishing the project ID via createProjectSSMParam and resolving it with MdaaStringParameter.valueForStringParameter follows the same cross-module pattern the codebase uses for bucket names, KMS ARNs, and security configurations.

One open question before we merge: the Athena workgroup in dataops-project applies all three DataZone tags (AmazonDataZoneProject, AmazonDataZoneEnvironment, AmazonDataZoneDomain). This fix applies only AmazonDataZoneProject. For Glue jobs provisioned outside a DataZone environment, the project tag alone may be sufficient for SMUS association, but we want to validate that before proceeding.

Detailed review to follow on the PR.