wvchallenges · ju-ponectus · Feb 13, 2017 · Feb 13, 2017 · Feb 13, 2017 · Feb 13, 2017
diff --git a/README.md b/README.md
@@ -1,58 +1,35 @@
 # Wave Operations Engineering Development Challenge
 
-Applicants for the [Operations Engineering team](https://wave.bamboohr.co.uk/jobs/) at Wave must complete the following challenge, and submit a solution prior to the interviewing process. 
+This is my (Julien) submission for Wave ops challenge.
 
-The purpose of this exercise is to create something that we can discuss during the on-site interview, and that's representative of the kind of things we do here on a daily basis.
+## What I have done
 
-There isn't a hard deadline for this exercise; take as long as you need to complete it. However, in terms of total time spent actively working on the challenge, we ask that you not spend more than a few hours, as we value your time and expect to leave things open to discussion in the on-site interview.
+This project only consists in some shell scripts, I just wanted to keep it simple and be as efficient as possible.
+I'm somewhat proud of acheiving this challenge because I had never used AWS until now. And I have also learned a lot of thing about Python web apps since it was also the first time I have to deploy this kind of things.
 
-Send your completed submission to your contact at Wave. Feel free to email [ops.careers@waveapps.com](ops.careers@waveapps.com) if you have any questions.
+## How it works
 
-## Submission Instructions
+Simply run aws-app.sh to deploy the app. There are also some parameters :
 
-1. Fork this project on GitHub -  you'll need to create an account if you don't already have one
-1. Complete the project as described below within your fork
-1. Push all of your changes to your fork on GitHub and submit a pull request
-1. Email your contact at Wave to let them know you have submitted a solution, and make sure to include your GitHub username in your email (so we can match applicants with pull requests)
+```bash
+usage: ./aws-app.sh [-b|--branch-or-tag <branch-or-tag>][-h|--help]
 
-## Alternate Submission Instructions (if you don't want to publicize completing the challenge)
-
-1. Clone the repository
-1. Complete your project as described below within your local repository
-1. Email a patch file to your contact at Wave
-
-## Project Description
-
-There's a basic Python app available [here](https://github.com/wvchallenges/opseng-challenge-app). Your task is to host this app on AWS, using the current `HEAD` of the `master` branch as of when we test your submission.
-
-The OS used for hosting, and the tools & techniques used to accomplish this are up to you. Once you're done, please submit a paragraph or two in your `README` about what you're particularly proud of in your implementation, and why. Be deliberate in your choices and design, as we'll use them as a starting point for our discussions.   
-
-### Deliverables
-
-You should provide at least an executable bash script called `aws-app.sh`. You're welcome to include other files and install/use other tools in your repo as needed, but `aws-app.sh` is what we'll run to test your submission (see the evaluation section).
-
-#### Notes
-
-* **Do not check AWS keys or any other secret credentials into git**
-* Prefix all of your AWS resources (when possible) with your first name (example: joanne.domain.com)
+-b|--branch-or-tag <branch-or-tag>  Sets the branch or tag name of the GitHub repository to deploy to AWS
+-h|--help                           Prints this help message
+```
 
-## Evaluation
+## Some more info
 
-We'll do the following, using on a stock OSX machine with Python 2.7.10 or higher (but <3.0), the `awscli` Python package installed, and appropriate AWS environment variables set:
-```
-$ git clone <your username>/<repo name>  # Or we'll apply your patch file to a checked-out branch
-$ cd <repo name>
-$ ./aws-app.sh
-```
-We expect that this will output a URL, and we'll then visit that URL to confirm it has the output generated by the current `HEAD` of the `master` branch of the repo linked to above. 
+I chose to use an Ubuntu EC2 instance because I use Debian-based distros every day, so I really feel comfortable using then.
+It is the same for the Nginx server I used, I know it is a quite solid and easy to install solution when you need a reverse proxy to publish a web app.
 
-When we're evaluating your submission, some of the questions we'll be asking are:
+## Improvements
 
-* If we follow the steps above, do we end up with a working app at the URL specified?
-* Does the working app reflect what's at the `HEAD` of the `master` branch right now, or at a point in the past? 
-* If we wanted to push out an updated version of the app's code, how much work would that be? 
-* Which application(s) and OS were chosen to host the app, and why?
-* Which hosting strategy was selected, and did you have a good reason to pick that one?
-* Are the decisions and strengths/weaknesses of this strategy discussed?
-* How much of the hosting infrastructure is created when calling `aws-app.sh`, and how much does the script assume already exists or is created by hand in the console?
+Some things could be improved in what I have done:
 
+- Use something like [RunIt](https://wiki.debian.org/runit) to manage Gunicorn deamon and to be sure it runs after the EC2 instance is restarted for instance
+- Extract parameters from main shell script in order to store them in a Yaml file for instance, which is more user-friendly
+- Store the opseng-challenge-app files in a S3 bucket and mount this storage from EC2 instance in order to be able to quickly scale up the system by adding some other EC2 instances sharing the same bucket.
+- Use Route53 to define an easy to remember DNS name for the app
+- Use virtualenv to run Python web app in a box and to be able to publish other Python apps using different versions of the same libraries
+- Use Docker to containerize the app and to be able to deploy it everywhere (different OSes, different cloud providers...)
diff --git a/aws-app.sh b/aws-app.sh
@@ -0,0 +1,89 @@
+#!/bin/bash
+
+# Basic Python app git parameters
+HELLO_APP_GIT_URL=https://github.com/wvchallenges/opseng-challenge-app
+HELLO_APP_DEFAULT_BRANCH_OR_TAG=master
+
+# SSH Key Pair name
+KEY_PAIR_NAME=julien.HelloAppKey
+
+# Security group
+SECGRP_NAME=julien.HelloAppSecGrp
+SECGRP_DESC="Security group for Hello App access"
+SECGRP_RULES_PORT=( 22 80 )
+
+# Variables related to EC2 Instance installation and configuration
+REQUIRED_PKGS="git-core nginx python python-pip"
+HELLOAPP_INSTALLDIR=/var/www/helloapp
+GUNICORN_PORT=8000
+# Base Nginx configuration to use for app publication
+NGINX_TEMPLATE=resources/nginx/helloapp
+NGINX_CONFDIR=/etc/nginx
+
+# AWS EC2 Instance parameters
+AWS_EC2_INSTANCE_TYPE=t2.micro
+AWS_EC2_INSTANCE_TAG_NAME=Name
+AWS_EC2_INSTANCE_TAG_VALUE=julien.HelloAppInstance
+
+# Ubuntu Xenial Amazon image
+AWS_EC2_UBUNTU_AMI_ID=ami-7c803d1c
+AWS_EC2_UBUNTU_LOGIN=ubuntu
+
+# SSH related commands for EC2 Instance
+SSH_CMD="ssh -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/${KEY_PAIR_NAME}.pem -l ${AWS_EC2_UBUNTU_LOGIN}"
+SCP_CMD="scp -q -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i ~/.ssh/${KEY_PAIR_NAME}.pem"
+
+# Get this script install dir
+SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
+
+source "${SCRIPT_DIR}/inc/toolbox.sh"
+source "${SCRIPT_DIR}/inc/aws.sh"
+source "${SCRIPT_DIR}/inc/aws-app-steps.sh"
+
+HELLO_APP_BRANCH_OR_TAG=
+while [ $# -ge 1 ]
+do
+  key="$1"
+
+  case $key in
+      -b|--branch-or-tag)
+      HELLO_APP_BRANCH_OR_TAG="$2"
+      shift # past argument
+      ;;
+      -h|--help)
+      printHelp
+      exit 0
+      ;;
+      *)
+              # unknown option
+      ;;
+  esac
+  shift # past argument or value
+done
+
+if [ -z "${HELLO_APP_BRANCH_OR_TAG}" ]; then
+  HELLO_APP_BRANCH_OR_TAG=$HELLO_APP_DEFAULT_BRANCH_OR_TAG
+fi
+
+echoStep "Starting HelloApp deployment"
+
+# 1- Manage SSH key pair
+manageKeypair ${KEY_PAIR_NAME}
+
+# 2- Manage security group
+manageSecGrp ${SECGRP_NAME} "${SECGRP_DESC}"
+
+# 3- Create EC2 instance
+manageInstance ${AWS_EC2_UBUNTU_AMI_ID} ${AWS_EC2_INSTANCE_TYPE} ${KEY_PAIR_NAME} ${SECGRP_NAME} ${AWS_EC2_INSTANCE_TAG_NAME} ${AWS_EC2_INSTANCE_TAG_VALUE} "${SSH_CMD}"
+
+INSTANCE_ID=$(cat ./instance.id)
+INSTANCE_PUBLIC_DNS_NAME=$(getInstancePublicDnsName ${INSTANCE_ID})
+echoInfo "EC2 instance public DNS name: ${INSTANCE_PUBLIC_DNS_NAME}"
+
+# 4- Configure instance and install app
+setupAndDeployApp ${INSTANCE_PUBLIC_DNS_NAME} $HELLO_APP_BRANCH_OR_TAG $HELLO_APP_GIT_URL \
+                  $HELLOAPP_INSTALLDIR $AWS_EC2_UBUNTU_LOGIN "$REQUIRED_PKGS" "${SCRIPT_DIR}/${NGINX_TEMPLATE}" \
+                  $GUNICORN_PORT $NGINX_CONFDIR "$SSH_CMD" "$SCP_CMD"
+
+# 5- Exit
+cleanupAndExit 0
diff --git a/inc/aws-app-steps.sh b/inc/aws-app-steps.sh
@@ -0,0 +1,224 @@
+#!/bin/bash
+
+manageKeypair() {
+  local _KEY_PAIR_NAME=$1
+
+  echoStep "Manage SSH key pair"
+  checkKeypair ${_KEY_PAIR_NAME}
+  if [ $? -ne 0 ]; then
+    echoWarning "${_KEY_PAIR_NAME} key pair does not exist. We are going to create it."
+    createAndInstallKeypair ${_KEY_PAIR_NAME}
+    if [ $? -ne 0 ]; then
+      echoError "Could not create and install new key pair. Exiting."
+      cleanupAndExit 1
+    fi
+  else
+    echoSuccess "${_KEY_PAIR_NAME} key pair already exists."
+    if [ ! -f ~/.ssh/${_KEY_PAIR_NAME}.pem ]; then
+      echoWarning "Private key file does not exist locally. We are going to recreate key pair."
+      deleteKeypair ${_KEY_PAIR_NAME}
+      if [ $? -ne 0 ]; then
+        echoError "Could not delete existing key pair. Exiting."
+        cleanupAndExit 1
+      fi
+      createAndInstallKeypair ${_KEY_PAIR_NAME}
+      if [ $? -ne 0 ]; then
+        echoError "Could not create and install new key pair. Exiting."
+        cleanupAndExit 1
+      fi
+    fi
+  fi
+}
+
+manageSecGrp() {
+  local _SECGRP_NAME=$1
+  local _SECGRP_DESC="$2"
+
+  echoStep "Manage security group"
+  local _CREATE_RULES=0
+  checkSecGrp ${_SECGRP_NAME}
+  if [ $? -ne 0 ]; then
+    echoWarning "${_SECGRP_NAME} security group does not exist. We are going to create it."
+    createSecGrp ${_SECGRP_NAME} "${_SECGRP_DESC}"
+    if [ $? -ne 0 ]; then
+      echoError "Could not create security group. Exiting."
+      cleanupAndExit 1
+    fi
+    _CREATE_RULES=1
+  else
+    echoSuccess "${_SECGRP_NAME} security group already exists. Will check rules to be sure they exist."
+    _CREATE_RULES=1
+  fi
+
+  if [ $_CREATE_RULES -eq 1 ]; then
+    for i in "${SECGRP_RULES_PORT[@]}"
+    do
+    	createSecGrpRule ${_SECGRP_NAME} ${i}
+      if [ $? -ne 0 ]; then
+        echoError "Could not create inbound rule for port ${i}. Exiting."
+        cleanupAndExit 1
+      fi
+    done
+  fi
+}
+
+manageInstance() {
+  local _EC2_AMI_ID=$1
+  local _EC2_INSTANCE_TYPE=$2
+  local _KEY_PAIR_NAME=$3
+  local _SECGRP_NAME=$4
+  local _TAG_NAME=$5
+  local _TAG_VALUE=$6
+  local _SSH_CMD="$7"
+
+  if [ -f ./instance.id ]; then
+    rm -f ./instance.id
+  fi
+
+  echoStep "Create EC2 instance"
+  checkInstance ${_TAG_NAME} ${_TAG_VALUE}
+  if [ $? -eq 0 ]; then
+    echoInfo "EC2 instance already exists. We are going to try to ssh it."
+    local _INSTANCE_ID=$(getInstanceId ${_TAG_NAME} ${_TAG_VALUE})
+    if [ $? -ne 0 ]; then
+      echoError "Could not get EC2 instance ID"
+      cleanupAndExit 1
+    fi
+    local _INSTANCE_PUBLIC_DNS_NAME=$(getInstancePublicDnsName ${_INSTANCE_ID})
+    if [ $? -ne 0 ]; then
+      echoError "Could not get EC2 instance public DNS name"
+      cleanupAndExit 1
+    fi
+    echoInfo "Existing EC2 instance public DNS name: ${_INSTANCE_PUBLIC_DNS_NAME}"
+    ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} exit
+    if [ $? -ne 0 ]; then
+      echoWarning "Could not ssh EC2 instance. We are going to delete and recreate it."
+      deleteInstance ${_INSTANCE_ID}
+      if [ $? -ne 0 ]; then
+        echoError "Could not terminate existing instance. Exiting."
+        cleanupAndExit 1
+      fi
+      echoSuccess "EC2 instance successfully deleted."
+      echoInfo "Create new EC2 instance."
+      createAndTagInstance ${_EC2_AMI_ID} ${_EC2_INSTANCE_TYPE} ${_KEY_PAIR_NAME} ${_SECGRP_NAME} ${_TAG_NAME} ${_TAG_VALUE}
+    else
+      echo $_INSTANCE_ID > ./instance.id
+      echoSuccess "SSH connection to EC2 instance successfully acheived."
+    fi
+  else
+    echoWarning "EC2 instance not found, we are going to create it."
+    createAndTagInstance ${_EC2_AMI_ID} ${_EC2_INSTANCE_TYPE} ${_KEY_PAIR_NAME} ${_SECGRP_NAME} ${_TAG_NAME} ${_TAG_VALUE}
+  fi
+}
+
+createAndTagInstance() {
+  local _EC2_AMI_ID=$1
+  local _EC2_INSTANCE_TYPE=$2
+  local _KEY_PAIR_NAME=$3
+  local _SECGRP_NAME=$4
+  local _TAG_NAME=$5
+  local _TAG_VALUE=$6
+
+  local _INSTANCE_ID=$(createAndRunInstance ${_EC2_AMI_ID} ${_EC2_INSTANCE_TYPE} ${_KEY_PAIR_NAME} ${_SECGRP_NAME})
+  if [ $? -ne 0 ]; then
+    echoError "Could not create and run new EC2 instance. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "EC2 instance successfully created and launched."
+  echo $_INSTANCE_ID > ./instance.id
+  echoStep "Wait some time so as to let EC2 instance start..."
+  sleep 5
+  echoInfo "Create tag for new instance"
+  tagInstance ${_INSTANCE_ID} ${_TAG_NAME} ${_TAG_VALUE}
+  if [ $? -ne 0 ]; then
+    echoError "Could not create tag EC2 instance. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "EC2 instance successfully tagged."
+}
+
+setupAndDeployApp() {
+  local _INSTANCE_PUBLIC_DNS_NAME=$1
+  local _APP_BRANCH_OR_TAG=$2
+  local _APP_URL=$3
+  local _APP_INSTALLDIR=$4
+  local _INSTANCE_LOGIN=$5
+  local _REQUIRED_PKGS="$6"
+  local _NGING_TPL="$7"
+  local _GUNICORN_PORT=$8
+  local _NGINX_CONFDIR=$9
+  local _SSH_CMD="${10}"
+  local _SCP_CMD="${11}"
+
+  echoStep "Configure instance and install app"
+  echoInfo "Update available packages list"
+  ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} "sudo apt-get update -q  > /dev/null 2>&1"
+  if [ $? -ne 0 ]; then
+    echoError "Unable to update packages list. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "Packages list successfully updated."
+  echoInfo "Install required packages on instance (${_REQUIRED_PKGS})"
+  ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} "sudo apt-get install -yq ${_REQUIRED_PKGS} > /dev/null 2>&1"
+  if [ $? -ne 0 ]; then
+    echoError "Something went wrong during packages installation. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "Required packages successfully installed."
+  echoInfo "Get HelloApp code (branch or tag = ${_APP_BRANCH_OR_TAG})"
+  ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} "sudo rm -rf ${_APP_INSTALLDIR} 2> /dev/null; \
+                                          sudo mkdir -p ${_APP_INSTALLDIR} \
+                                          && sudo chown ${_INSTANCE_LOGIN} ${_APP_INSTALLDIR} \
+                                          && cd ${_APP_INSTALLDIR} \
+                                          && git clone --branch ${_APP_BRANCH_OR_TAG} --depth 1 ${_APP_URL} . > /dev/null 2>&1"
+  if [ $? -ne 0 ]; then
+    echoError "Something went wrong during Hello App source code installation. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "Hello App source code successfully downloaded."
+  echoInfo "Install HelloApp Python requirements"
+  ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} "cd ${_APP_INSTALLDIR} \
+                                            && sudo pip install -r requirements.txt > /dev/null 2>&1"
+  if [ $? -ne 0 ]; then
+    echoError "Something went wrong during Hello App Python requirements installation. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "Hello App source Python requirements successfully installed."
+  echoInfo "Launch Gunicorn"
+  ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} "sudo killall -q gunicorn ; \
+                                            cd ${_APP_INSTALLDIR} \
+                                            && sudo gunicorn -b 127.0.0.1:${_GUNICORN_PORT} -D app:app > /dev/null 2>&1"
+  if [ $? -ne 0 ]; then
+    echoError "An error occured while trying to launch Gunicorn. Exiting."
+    cleanupAndExit 1
+  fi
+  echoSuccess "Gunicorn successfully started."
+  echoInfo "Generate Nginx configuration from template"
+  sed "s/##SERVERNAME##/${_INSTANCE_PUBLIC_DNS_NAME}/g;\
+       s@##WEBROOT##@${_APP_INSTALLDIR}@g;\
+       s/##GUNICORN_PORT##/${_GUNICORN_PORT}/g" "${_NGING_TPL}" > /tmp/helloapp.nginx 2> /dev/null
+  if [ $? -ne 0 ]; then
+   echoError "Something went wrong during Nginx configuration generation. Exiting."
+   cleanupAndExit 1
+  fi
+  echoSuccess "NGinx configuration successfully generated."
+  echoInfo "Send Nginx configuration to EC2 instance"
+  ${_SCP_CMD} /tmp/helloapp.nginx ${_INSTANCE_LOGIN}@${_INSTANCE_PUBLIC_DNS_NAME}:/tmp/helloapp
+  if [ $? -ne 0 ]; then
+   echoError "Something went wrong during Nginx configuration SCP transfer. Exiting."
+   cleanupAndExit 1
+  fi
+  rm -f /tmp/helloapp.nginx
+  echoSuccess "NGinx configuration successfully sent to EC2 instance."
+  echoInfo "Configure Nginx on EC2 instance"
+  ${_SSH_CMD} ${_INSTANCE_PUBLIC_DNS_NAME} "sudo cp /tmp/helloapp ${_NGINX_CONFDIR}/sites-available \
+                                          && sudo rm -f ${_NGINX_CONFDIR}/sites-enabled/* \
+                                          && sudo ln -s ${_NGINX_CONFDIR}/sites-available/helloapp ${_NGINX_CONFDIR}/sites-enabled/helloapp \
+                                          && sudo service nginx restart > /dev/null 2>&1"
+  if [ $? -ne 0 ]; then
+   echoError "Something went wrong during Nginx configuration. Exiting."
+   cleanupAndExit 1
+  fi
+  echoSuccess "NGinx successfully configured."
+  echoStep "HelloApp deployment successfully finished, it is available at http://${_INSTANCE_PUBLIC_DNS_NAME}"
+}