Skip to content

Remove unnecessary VPC endpoints created with SQL schemas #3409

@iliapolo

Description

@iliapolo

Environment information

System:
  OS: macOS 26.2
  CPU: (12) arm64 Apple M3 Pro
  Memory: 2.37 GB / 36.00 GB
  Shell: /bin/zsh
Binaries:
  Node: 22.16.0 - /Users/epolon/.nvm/versions/node/v22.16.0/bin/node
  Yarn: 1.22.22 - /Users/epolon/.nvm/versions/node/v22.16.0/bin/yarn
  npm: 10.9.2 - /Users/epolon/.nvm/versions/node/v22.16.0/bin/npm
  pnpm: 10.29.2 - /Users/epolon/.nvm/versions/node/v22.16.0/bin/pnpm
NPM Packages:
  @aws-amplify/auth-construct: 1.11.0
  @aws-amplify/backend: 1.20.0
  @aws-amplify/backend-ai: Not Found
  @aws-amplify/backend-auth: 1.9.1
  @aws-amplify/backend-cli: 1.8.2
  @aws-amplify/backend-data: 1.6.3
  @aws-amplify/backend-deployer: 2.1.5
  @aws-amplify/backend-function: 1.16.0
  @aws-amplify/backend-output-schemas: 1.8.0
  @aws-amplify/backend-output-storage: 1.3.3
  @aws-amplify/backend-secret: 1.4.2
  @aws-amplify/backend-storage: 1.4.3
  @aws-amplify/cli-core: 2.2.3
  @aws-amplify/client-config: 1.10.0
  @aws-amplify/data-construct: 1.17.0
  @aws-amplify/data-schema: 1.23.0
  @aws-amplify/deployed-backend-client: 1.8.1
  @aws-amplify/form-generator: 1.2.6
  @aws-amplify/model-generator: 1.2.2
  @aws-amplify/platform-core: 1.10.4
  @aws-amplify/plugin-types: 1.11.2
  @aws-amplify/sandbox: 2.1.4
  @aws-amplify/schema-generator: 1.4.1
  @aws-cdk/toolkit-lib: 1.6.1
  aws-amplify: 6.16.2
  aws-cdk-lib: 2.234.1
  typescript: 5.9.3
AWS environment variables:
No CDK environment variables

Data packages

amplify-rds-gen2@1.0.0 /Users/epolon/dev/src/local/amplify-rds-gen2
├─┬ @aws-amplify/backend-cli@1.8.2
│ └─┬ @aws-amplify/schema-generator@1.4.1
│   └── @aws-amplify/graphql-schema-generator@0.11.13
└─┬ @aws-amplify/backend@1.20.0
  └─┬ @aws-amplify/backend-data@1.6.3
    └── @aws-amplify/data-construct@1.17.0

Description

SQL as part of the data schema causes the creation of the following VPC endpoints:

Image

Out of these, only ssm is actually needed since the lambda reads the connection string from SSM. Lets remove the other endpoints.

Context

The function that creates the endpoints mentions that:

// Although the Lambda function will only invoke SSM directly, internally the SDK makes calls to other services as well
const services = ['ssm', 'ssmmessages', 'ec2', 'ec2messages', 'kms'];

This isn't true, removing all endpoints except ssm and invoking the lambda function with this payload:

{
  "operation": "RAW_SQL",
  "statement": "show databases;"
}

Still results in a success:

[
  {
    "Database": "amplify"
  },
  {
    "Database": "information_schema"
  },
  {
    "Database": "mysql"
  },
  {
    "Database": "performance_schema"
  },
  {
    "Database": "sys"
  }
]

In fact, the function that creates these endpoint only returns the SSM DNS entry back to the caller:

// Replace the default SSM endpoint with the VPC endpoint
const ssmEndpoint = Fn.select(0, endpointEntries);
return ssmEndpoint;

Which means that the lambda function itself has no knowledge about the other endpoints, and therefore cannot make use of them.

The SSM endpoint DNS however is indeed used in the handler code:

const endpoint = process.env.SSM_ENDPOINT?.split(PORT_SEPERATOR).pop();
ssmClient = new SSMClient({
endpoint: `https://${endpoint}`,
});

I believe the initial implementation mistakenly created endpoints for all service mentioned Creating VPC endpoints for Systems Manager. However, ssmmessages, ec2messages and ec2 are only required when using an SSM agent with EC2 instances, not for the Parameter Store retrieval use case.

The kms endpoint description is kind of vague though:

(Optional) com.amazonaws.region.kms – Create this endpoint if you want to use AWS Key Management Service (AWS KMS) encryption for Session Manager or Parameter Store parameters

Im not sure what this means exactly, but my test successfully retrieved the secret MySQL connection string from SSM even without the kms endpoint. This is because the decryption happens by SSM on the server side, not on the client.

Workaround

You can remove the unnecessary endpoints using CDK escape hatches:

const sqlLambdaNestedStack = backend.stack.node.findAll().filter(c => NestedStack.isNestedStack(c) && c.node.path.includes('SQLApi'))[0];
const vpcEndpoints = sqlLambdaNestedStack.node.findAll().filter(c => CfnResource.isCfnResource(c) && c.cfnResourceType === 'AWS::EC2::VPCEndpoint');
const endpointsToRemove = ['ssmmessages', 'ec2messages', 'ec2', 'kms'];

for (const vpce of vpcEndpoints) {

  if (endpointsToRemove.find(e => vpce.node.id.endsWith(e))) {
    const removed = sqlLambdaNestedStack.node.tryRemoveChild(vpce.node.id);
    if (removed) {
      console.log(`Removed ${vpce.node.path}`);
    }
  }

}

Reproduce

  1. Deploy an RDS MySQL engine with the following CDK code:
const vpc = new ec2.Vpc(this, 'Vpc', { maxAzs: 2 });

const securityGroup = new ec2.SecurityGroup(this, 'SecurityGroup', {
  vpc,
});

securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(3306));
securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.HTTPS);
securityGroup.addEgressRule(ec2.Peer.securityGroupId(securityGroup.securityGroupId), ec2.Port.tcp(3306));
securityGroup.addEgressRule(ec2.Peer.securityGroupId(securityGroup.securityGroupId), ec2.Port.HTTPS);

new rds.DatabaseInstance(this, 'MySqlDatabase', {
  engine: rds.DatabaseInstanceEngine.mysql({
    version: rds.MysqlEngineVersion.VER_8_0,
  }),
  vpc,
  securityGroups: [securityGroup],
  vpcSubnets: vpc.selectSubnets({ subnetType: ec2.SubnetType.PUBLIC }),
  publiclyAccessible: true,
  instanceType: ec2.InstanceType.of(
    ec2.InstanceClass.T3,
    ec2.InstanceSize.MICRO,
  ),
  allocatedStorage: 20,
  removalPolicy: cdk.RemovalPolicy.DESTROY,
  credentials: rds.Credentials.fromGeneratedSecret('admin'),
});
  1. Install the mysql CLI
brew install mysql@8.0
  1. Create a simple table with a primary key.
mysql -h <instance-domain>.us-east-1.rds.amazonaws.com -P 3306 -u admin 

Grab the password from the secret the CDK created.

CREATE TABLE users (
  id INT PRIMARY KEY AUTO_INCREMENT,
  name VARCHAR(255)
);

CREATE

  1. Follow https://docs.amplify.aws/react/build-a-backend/data/connect-to-existing-data-sources/connect-postgres-mysql-database/ to connect the databases into the data construct.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions