AWS Account Setup Guide

1. Creating an AWS Account

To start working with Terraform and AWS resources, you’ll need to set up an AWS account and configure administrative permissions. Follow the steps below:

Step 1: Register for an AWS Account

1. Go to the AWS Sign-Up Page and create a new AWS account.
2. Complete the registration form with your details and payment information. Once registered, you can log in to the AWS Console.

Step 2: Set Up IAM (Identity and Access Management)

1. Log into AWS Console
   Once logged in, navigate to the AWS Management Console.

2. Access IAM
   In the console, search for "IAM" in the search bar at the top and select IAM under Services.

3. Create an Admin User

   • In the left sidebar under Access Management, click Users.
   • Click Add User and enter a username (e.g., Admin).
   • For AWS Management Console access, select Provide user access to the AWS Management Console.
   • Set a password preference (either autogenerated or custom).

4. Assign Administrator Permissions

   • In the Permissions step, select Attach policies directly.
   • Find and check the box for the AdministratorAccess policy. This grants full access to AWS services, which is required for setting up resources via Terraform.

5. Review and Create User

   • Click Next through the remaining steps to Review and create the user.
   • Download the credentials file containing the Access Key ID and Secret Access Key. Store this file securely as it will be required to sign in and authenticate for Terraform operations.

Step 3: Configure Billing Access (Optional)

To monitor costs and usage within your account:

1. In the AWS Console, go to Billing and Cost Management from the Account menu (top-right corner).
2. Enable billing and cost management access if prompted, which allows you to view detailed usage statistics and avoid unexpected expenses.

Note: Use the Admin IAM user you created for logging in and managing resources rather than the root account for better security and access control.

Setting Up Terraform

1. Preparing Your Development Environment

MacOS Users:

• Open Terminal on your Mac.

Windows Users:

• Open Command Prompt or PowerShell on your Windows computer.

---

2. Setting Up Your Project Directory

1. Navigate to the Desired Directory:
   Use the cd command to navigate to the directory where you want to set up the project.

   • Example command:

     cd Documents

2. Create a New Directory:
   Create a folder to store your Terraform code by using mkdir.

   • Example command:

     mkdir rev-parallel-cluster

3. Move into the New Directory:
   Change into the newly created project folder.

   • Example command:

     cd rev-parallel-cluster

4. Clone the Project Repository:
   Use git to clone the project repository and get the necessary code files.

   • Example command:

     git clone https://github.com/switchbox-data/rev-parallel-cluster.git

1. Install AWS CLI

macOS:

1. Install AWS CLI using Homebrew:

   brew install awscli

2. Configure AWS CLI with your credentials:

   aws configure

   Follow the prompts to enter your Access Key ID, Secret Access Key, Region, and Output Format.

Windows:

1. Download the AWS CLI installer from the official site and run it.
2. After installation, open a terminal (PowerShell or Command Prompt) and run:

   aws configure

   Follow the prompts to set up your Access Key ID, Secret Access Key, Region, and Output Format.

---

2. Install Terraform

macOS:

1. Install Terraform using Homebrew:

   brew install terraform

2. Verify the installation:

   terraform -version

Windows:

1. Download Terraform from the official site.
2. Extract the Terraform executable and add it to your PATH:
   • Open System Properties > Environment Variables > Path and add the folder containing terraform.exe.
3. Verify the installation:

   terraform -version

---

3. Navigate to the Project Directory

Once both tools are installed, navigate to your project directory:

cd rev-parallel-cluster/

Running Terraform Commands

1. Setting Up terraform.tfvars

1. Open your project in an Editor/IDE:
   Open the rev-parallel-cluster directory in your preferred text editor or IDE.

2. Create a terraform.tfvars file:
   In the rev-parallel-cluster directory, create a file named terraform.tfvars to define the necessary Terraform variables.

---

2. Create a Key Pair for the Parallel Cluster

Generate a new SSH key pair:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

---

3. Define Variables in terraform.tfvars

Add the following variable definitions in your terraform.tfvars file:

terraform_state_s3_bucket_name = "<insert-bucket-name-here>"
subnet_id = "<insert-public-subnet-id-here>"
parallel_cluster_api_stack_name = "ParallelCluster"
parallel_cluster_api_stack_version = "3.10.0"
region = "us-west-2"
sns_alert_emails = ["<insert-emails-here>"]
spending_alert_threshold = <insert-spending-alert-threshold> # spending limit for alerts
monthly_spend_limit = <insert-monthly-spend-limit>
post_install_scripts_bucket_name = "<insert-bucket-name-post-install>"
output_files_bucket_name = "<insert-output-files-bucket-name>"

Key Variables and Their Uses

1. terraform_state_s3_bucket_name:

   • Purpose: Stores the Terraform state file, which keeps track of the current state of your infrastructure.
   • Explanation: This bucket name must be unique because S3 bucket names are globally unique across AWS. Storing the Terraform state in S3 allows multiple team members or systems to access and maintain the infrastructure’s state consistently.
   • Creation Command:

     terraform-state-storage-$(uuidgen | tr '[:upper:]' '[:lower:]')

2. subnet_id:

   • Purpose: Specifies the public subnet where the ParallelCluster and its resources, like EC2 instances, will be deployed.
   • Explanation: This should be a subnet ID in the us-west-2a availability zone (or the relevant zone you’re using). The subnet must have internet access for resources that need to communicate externally. In the AWS Console, go to VPC > Subnets to find an existing subnet or create a new one if necessary.

3. parallel_cluster_api_stack_name:

   • Purpose: Defines the name for the AWS CloudFormation stack that deploys the ParallelCluster API.
   • Explanation: This stack handles the API layer for managing the ParallelCluster. Naming it makes it easy to locate in AWS CloudFormation and ensures it won’t conflict with other stacks. You can leave this as "ParallelCluster" or change it if needed.

4. parallel_cluster_api_stack_version:

   • Purpose: Specifies the version of ParallelCluster you want to use.
   • Explanation: Make sure the version you specify is compatible with any other configurations or dependencies you have. Using the latest stable version (like 3.10.0 here) is recommended unless you need a specific version for compatibility.

5. region:

   • Purpose: Sets the AWS region where all resources, including the ParallelCluster, will be deployed.
   • Explanation: Ensure that this matches the region where your resources (like S3 buckets, subnets, etc.) reside. Here, us-west-2 is used, but this can be any region depending on where your data and operations are centered.

6. sns_alert_emails:

   • Purpose: A list of email addresses that will receive alerts if the cluster usage exceeds the defined spending limits.
   • Explanation: This should include any email addresses of stakeholders who need to monitor budget usage. AWS Simple Notification Service (SNS) will use this to send alert emails. Add valid email addresses in a list format, for example:

     sns_alert_emails = ["admin@example.com", "finance@example.com"]

7. spending_alert_threshold:

   • Purpose: The spending limit that triggers alerts, measured in USD.
   • Explanation: Once the threshold is met, an alert will be sent to the emails in sns_alert_emails. Set this to a realistic amount based on expected usage patterns, for example:

     spending_alert_threshold = 500

8. monthly_spend_limit:

   • Purpose: Sets the monthly spending limit for the entire ParallelCluster setup.
   • Explanation: If the total usage cost exceeds this amount, it may trigger additional actions or notifications. Set this to align with your monthly budget cap, such as:

     monthly_spend_limit = 1000

9. post_install_scripts_bucket_name:

   • Purpose: The unique bucket used to store post-install scripts that run on EC2 instances during cluster setup.
   • Explanation: These scripts, which could include commands for configuring services like HSDS on EC2 instances, need to be accessible by the cluster upon initialization. Since this bucket stores scripts, it also needs a unique name.
   • Creation Command: bash post-install-bucket-$(uuidgen | tr '[:upper:]' '[:lower:]') Here’s a similar setup for output_files_bucket_name:

10. output_files_bucket_name

• Purpose: A unique bucket used by the head node to upload files to S3 manually after completing a run. This allows easy access and management of output files generated during the run.
• Explanation: The bucket is specifically used for storing files from completed runs, making it easier to retrieve and organize results.
• Creation Command:

  output-files-bucket-$(uuidgen | tr '[:upper:]' '[:lower:]')

---

Example Configuration

Once you have your unique bucket names, email addresses, and budget limits defined, your configuration might look like this:

terraform_state_s3_bucket_name = "terraform-state-storage-123abc456def"
subnet_id = "subnet-0abcd1234efgh5678"
parallel_cluster_api_stack_name = "ParallelCluster"
parallel_cluster_api_stack_version = "3.10.0"
region = "us-west-2"
sns_alert_emails = ["admin@example.com", "finance@example.com"]
spending_alert_threshold = 500   # spending limit for alerts in USD
monthly_spend_limit = 1000       # monthly spending cap in USD
post_install_scripts_bucket_name = "post-install-bucket-789ghi012jkl"
output_files_bucket_name = "output-files-12345"

---

Running Terraform Commands

1. Initialize Terraform

terraform init

• Purpose: Initializes the Terraform working directory by downloading necessary provider plugins and setting up the backend configuration.
• Explanation: This command is essential before running any other Terraform command, as it ensures that Terraform has access to the correct providers (e.g., AWS) and prepares the environment to handle your infrastructure setup.

2. Preview Changes with terraform plan

terraform plan

• Purpose: Generates an execution plan that shows you exactly what Terraform will do when you apply changes.
• Explanation: This command is useful for reviewing changes before they’re applied, so you can verify that the planned operations align with your expectations. It provides a preview of resources that will be added, modified, or destroyed without making any changes to actual infrastructure.

3. Apply Changes to Infrastructure

terraform apply

• Purpose: Deploys the infrastructure as specified in the configuration files.
• Explanation: This command applies all defined resources in your Terraform configuration files to the specified environment. After running terraform apply, Terraform will prompt for confirmation (unless you pass the -auto-approve flag), allowing you to verify the changes one final time. After confirmation, Terraform creates or modifies resources to match the configuration.

---

Removing Placeholder Comments

For the initial run, some code lines need to be commented out to allow Terraform to set up the foundational resources without trying to build dependencies that aren't yet ready. After the initial run is complete, follow these steps to finalize your configuration:

1. Remove the Placeholder Comments: Uncomment the lines labeled # RemoveComment in your configuration files to enable the dependent resources to be created.

2. Reinitialize and Apply the Configuration:

   • Re-run terraform init (if necessary) to refresh the environment and ensure Terraform acknowledges the uncommented lines.
   • Apply the configuration again to build the dependent resources:

     terraform apply

This staged approach ensures that Terraform can reference the newly created resources and then proceed to build out the dependent infrastructure.

Example: Replace the S3 Bucket Name

• Replace the placeholder with terraform_state_s3_bucket_name in the specified line (e.g., line 24) in your Terraform configuration file to point to the secure S3 bucket specified in terraform.tfvars. This will ensure Terraform’s state is stored securely in your S3 bucket.

---

Handling Errors like Provider produced inconsistent final plan

If you encounter an error message such as Provider produced inconsistent final plan, it may be due to dependencies between resources that need to be refreshed or re-evaluated. Simply re-running the command can often resolve this issue, as Terraform will try to re-plan and verify resources before applying changes.

Using the ParallelCluster

1. SSH into the Head Node

To access the ParallelCluster’s head node, use the SSH command below with your private key and the head node’s public IP:

ssh -i ~/.ssh/id_rsa ec2-user@<head-node-ip-address>

• Finding the Head Node IP: In the AWS Console, navigate to EC2, locate your head node, and copy its public IP address.
• Once connected, continue with the following commands to set up the environment.

2. Install Miniconda on the Head Node

We’ll install Miniconda to manage Python dependencies:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
sh Miniconda3-latest-Linux-x86_64.sh

• Note: During installation, press Enter to proceed, then type yes when prompted.
• After installation, activate the Miniconda environment:

source ~/.bashrc

3. Set Up the reV Environment

Now we’ll create a Python environment specifically for reV and install the necessary packages:

conda create --name rev python=3.9
conda activate rev
pip install NREL-reV
git clone https://github.com/switchbox-data/rev-parallel-cluster-example-files.git

4. Run a Simple SLURM Command to Verify Setup

To verify that SLURM and the environment are set up correctly, we’ll run a simple SLURM job:

echo -e '#!/bin/bash\nsleep 30\necho "Hello World from $(hostname)"' > my_script.sh
chmod +x my_script.sh
sbatch my_script.sh

After submitting the job, check the SLURM controller status:

sudo systemctl status slurmctld

• Monitor the Job: Use squeue to check job status. If auto-provisioning is enabled, the job will remain in the CF (configuring) state until nodes are provisioned. Once running, the status changes to R.

5. Running the reV Jobs

For detailed instructions on running reV jobs, please refer to the example repo that has information regarding how to use reV with the parallel cluster

https://github.com/switchbox-data/rev-parallel-cluster-example-files

How to Make Changes to the Cluster

To customize your ParallelCluster configuration, there are a few key parameters and settings you can adjust, depending on your requirements. Below are the main configuration options you can modify.

Cluster Configuration Parameters

Region: ${region}
Image:
  Os: alinux2
HeadNode:
  InstanceType: t2.large
  Networking:
    SubnetId: ${subnet_id}
  Ssh:
    KeyName: ${ssh_key}
  Iam:
    AdditionalIamPolicies:
      - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
      - Policy: ${pass_and_attach_role_policy}

Scheduling:
  Scheduler: slurm
  SlurmQueues:
    - Name: compute
      CapacityType: ONDEMAND
      Networking:
        SubnetIds:
          - ${subnet_id}
        AssignPublicIp: true
      Iam:
        AdditionalIamPolicies:
          - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
          - Policy: arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
          - Policy: ${s3_readonly_post_install_scripts_policy}
          - Policy: ${pass_and_attach_role_policy}
      ComputeResources:
        - Name: queue1
          InstanceType: c5.2xlarge
          MinCount: 1
          MaxCount: 100
      CustomActions:
        OnNodeConfigured:
          Script: s3://${post_install_bucket}/hsds_post_install.sh
  SlurmSettings:
    QueueUpdateStrategy: TERMINATE
SharedStorage:
  - Name: ebs1
    MountDir: /shared
    StorageType: Ebs
    EbsSettings:
      VolumeType: gp2
      Size: 20
      DeletionPolicy: Delete

Explanation of Parameters

• Region: Defines the AWS region for the cluster deployment.
• InstanceType: Determines the instance type for the head and compute nodes. For higher compute needs, consider increasing this to a larger instance type.
• SubnetId: Specifies the subnet ID for network configuration. Adjust this based on network requirements and security settings.
• Iam Policies: Sets additional IAM policies. Policies like AmazonSSMManagedInstanceCore and AmazonEC2ContainerRegistryReadOnly provide permissions for management and container registry access. Modify these policies if there are any specific security or access requirements.
• ComputeResources: Controls compute node resources. The InstanceType, MinCount, and MaxCount control the node type and scaling limits. Increase the InstanceType or MaxCount to handle more intensive workloads.
• SharedStorage: Defines storage settings, such as EBS volumes. Increasing the EBS Size can help if the workload requires more storage.
• CapacityType: Specifies the type of compute nodes to be used in your parallel cluster. Setting this to SPOT enables the use of spot instances, which utilize unused compute capacity at a significantly reduced cost—up to 70% less than on-demand instances. Spot instances are quick to provision and are ideal for cost-sensitive workloads. However, they can be reclaimed by AWS when needed for other services, which may interrupt your jobs. In such cases, jobs will either restart or resume from the last saved checkpoint, depending on your configuration.

Updating and Redeploying the Cluster

To apply configuration changes, you’ll need to redeploy the cluster:

1. Comment Out the ParallelCluster Module
   In main.tf, temporarily comment out the module "parallel_cluster" { ... } block.

2. Reinitialize and Apply Changes
   Run the following commands:

   terraform init
   terraform apply

   If you encounter an error related to resource conflicts or other issues, follow the next steps.

3. Check CloudFormation Stack Status

   • Go to the CloudFormation section in the AWS Console.
   • Look for the rev-cluster stack to monitor its status and identify any errors.

4. If Necessary, Manually Clean Up Resources

   • If you encounter persistent errors, consider checking for any dangling resources that may need manual intervention.

5. Uncomment the ParallelCluster Module

   • Uncomment the module "parallel_cluster" block in main.tf.
   • Make any desired changes to rev_cluster_config.yaml.tpl.

6. Deploy the Updated Cluster

   • Run the following commands to redeploy:

     terraform init
     terraform apply

Your cluster will redeploy with the updated configuration. For any issues, check the AWS Console logs for more detailed troubleshooting.

Important Notes

Stopping the Cluster

To avoid incurring additional or unnecessary costs while not using the cluster, make sure to turn it off. Follow these steps:

1. Run the Command to Stop the Compute Fleet: Execute the following command in your terminal:

   pcluster update-compute-fleet --cluster-name revcluster --status STOP_REQUESTED

2. Manually Stop the Head Node:

   • Log in to the AWS Management Console.
   • Navigate to the EC2 Dashboard.
   • In the left sidebar, click on Instances.
   • Find the instance corresponding to your head node (you can identify it by the instance name, which typically follows the format revcluster-head-*).
   • Select the instance and click on the Instance state dropdown button.
   • Choose Stop instance from the options.
   • Confirm the action when prompted.

Restarting the Cluster

When you are ready to use the cluster again, follow these steps to restart it:

1. Run the Command to Start the Compute Fleet: Execute the following command in your terminal:

   pcluster update-compute-fleet --cluster-name revcluster --status START_REQUESTED

2. Start the Head Node:

   • Log in to the AWS Management Console.
   • Navigate to the EC2 Dashboard.
   • In the left sidebar, click on Instances.
   • Find the instance corresponding to your head node.
   • Select the instance and click on the Instance state dropdown button.
   • Choose Start instance from the options.
   • Confirm the action when prompted.

Adding Authorized SSH Keys via AWS Console

1. Log in to the AWS Management Console: Navigate to the AWS Management Console.

2. Access the EC2 Dashboard: In the top search bar, type "EC2" and select EC2 from the drop-down menu.

3. Locate Your Instance:

   • In the left-hand sidebar, click on Instances.
   • Find your head node instance in the list.

4. Modify Instance Settings:

   • Select the instance, and in the Instance details pane, look for the Actions dropdown menu at the top right.
   • Click on Instance Settings > Edit User Data.

5. Add User Data Script: You can add a script that will run when the instance starts. This script can be used to add the SSH key. Here’s an example script you can use:

   #!/bin/bash
   echo "<paste-your-public-key-here>" >> /home/ec2-user/.ssh/authorized_keys

   Replace <paste-your-public-key-here> with the actual public key of the user you want to grant access to.

6. Save Changes:

   • Click on Update User Data or Save to apply the changes.

7. Reboot the Instance: You may need to reboot the instance for the user data script to execute. You can do this from the Instance state dropdown by selecting Reboot.

Important Notes

• User Data Execution: The user data script runs only during the initial launch of the instance or upon reboot if modified. Make sure to add the correct public key.
• Security: Ensure that you only add trusted public keys and regularly manage your authorized_keys file.

Debugging Tips

1. Job Failures

If one of your jobs fails, navigate to the logs/stdout directory and locate the corresponding .e file for the failed run.

• Common Error: If you see an error related to 503, it indicates an issue with the HSDS service setup. This can often be caused by the max_task_count being too low.
  • Solution: You can either increase the max_task_count in the post-install script or utilize more nodes to reduce the number of concurrent requests.

2. Incorrect HSDS Path

• Common Error: If you encounter an error stating Unexpected Path, it likely means that your ~/.hscfg file is incorrectly formatted or located.
  • Solution: Ensure that the ~/.hscfg file is correctly formatted and copied to the right location, with all passwords and settings accurately defined.

3. Head Node Wait Condition Failure

If you receive an error related to the HeadNodeWait condition when deploying your cluster, this suggests a problem running the post-install script.

• Solution: SSH into your head node and attempt to run the post-install script directly from there to identify any issues. Ensure that the head node has the necessary S3 permissions in its role. You can retrieve the full post-install script from the S3 bucket.

4. Verifying HSDS Docker Container

To verify that your Docker container is functioning correctly, execute the following commands:

• Run:

  curl http://localhost:5101/about

  This command checks if the HSDS service is responsive.

• Next, to run a simple test script, ensure you have the h5pyd package installed in your Conda environment:

  pip install h5pyd

• Then run the following test script to access a known file:

  import h5pyd
  
  try:
      f = h5pyd.File("/nrel/wtk/conus/wtk_conus_2008.h5", "r")
      print("File accessed successfully")
  except Exception as e:
      print(f"Error accessing file: {e}")

Common Error Messages

• 403 Forbidden: Indicates an authorization issue, potentially due to incorrect permissions or the bucket_name not being set correctly in the configuration.
• Unexpected Path: This error means the ~/.hscfg file is incorrectly formatted or located, leading to issues in accessing the required paths.
• HeadNodeWait Condition Failure: Suggests problems running the post-install script, indicating the need for additional permissions or configurations.