From 90e886ba9ea6dd416ea3c34062f55b045049a478 Mon Sep 17 00:00:00 2001 From: Daniel Sont Date: Fri, 1 Apr 2022 13:07:32 +0700 Subject: [PATCH 1/3] monitoring with github actions post --- .../monitoring-with-github-actions/index.md | 101 ++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 content/posts/monitoring-with-github-actions/index.md diff --git a/content/posts/monitoring-with-github-actions/index.md b/content/posts/monitoring-with-github-actions/index.md new file mode 100644 index 00000000..98669852 --- /dev/null +++ b/content/posts/monitoring-with-github-actions/index.md @@ -0,0 +1,101 @@ ++++ +date = "2022-03-30T17:10:49+07:00" +author = "ds0nt" +description = "Using Github Actions to run some of your monitoring tasks on a schedule." +title = "Using Github Actions for Daily Monitoring Tasks" +categories = ["DevSecOps", "Automation", "CI"] +tags = ["DevOps", "Github Actions", "Monitoring", "CRON", "Schedule"] +slug = "github-actions-for-monitoring-tasks" ++++ + +# Using Github Actions for Daily Monitoring Tasks + +Github actions has a CRON schedule trigger we can use run scripts on a custom schedule. It's pretty cool that Github Actions can run jobs on a CRON schedule, and we can use this ability to cover some of our dev-ops needs. + +At Manabie, we used this to check if any of our domain certificates were close to their expiry. Each day, a job is starts at 10:45AM, checks the certificates of each cluster, and posts to our `#monitoring` Slack if any certificate is too close to expiring. + + +### The Workflow + +This is a simple monitoring flow really; Run Check -> Alert if Problem. + +To do this, were going to use three Github actions. + +1. an action to checkout our code +2. an action to execute a script +3. and an action to send slack message based on our condition. + +```yaml +# github/workflows/check_weather.yml` +name: check_weather +on: + schedule: # run at 010:45 UTC daily + - cron: '45 10 * * *' + workflow_dispatch: # runnable manually. + +jobs: + check-weather: + runs-on: ubuntu-latest + steps: + + # checkout code + - uses: actions/checkout@v2 + + # run script + - name: Check Weather + shell: bash + run: | + ./check-weather.sh + + # post results + - if: failure() + name: Slack Notification + uses: rtCamp/action-slack-notify@v2 + env: + SLACK_WEBHOOK: https://hooks.slack.com/services/*****/*****/***** + SLACK_CHANNEL: '#monitoring' + SLACK_USERNAME: "weather-checker" + SLACK_TITLE: Bad Weather Warning +``` + +This job runs a `./check-weather.sh` script every day at 10:45AM UTC and sends a message on slack if we want it to. + +We're using this action [rtCamp/action-slack-notify](https://github.com/rtCamp/action-slack-notify) to send the slack notification. We set most of the variables for the message already, but we still want to set the `SLACK_MESSAGE` variable from our weather-checking script. It's not hard to set env vars from prior steps, check it out [in the docs](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-environment-variable). + + +### Writing the Check Script + +We need to perform some check (like the weather), and exit with status 0 if it's ok, or exit with a non-zero status if theres a problem. + +```bash +#!/bin/bash + +# fetch the temperature +TEMPERATURE=$(curl 'wttr.in/Ho_Chi_Minh?format=3' | cut -d' ' -f4 | grep -oE '[+-][0-9]+') + +# test if the temperature is ok +if [[ $TEMPERATURE -le 20 ]]; then + MESSAGE="Dress warm, it's a chilly $TEMPERATURE outside" +elif [[ $TEMPERATURE -ge 30 ]]; then + MESSAGE="Its going to be hot. The temperature is $TEMPERATURE" +fi + + +# if it was ok, we can exit 0 +[[ -n $MESSAGE ]] || exit 0 + +# otherwise, we set the SLACK_MESSAGE env var for the next step, and exit 1. +echo 'SLACK_MESSAGE<> $GITHUB_ENV +echo -e "$MESSAGE" >> $GITHUB_ENV +echo 'EOF' >> $GITHUB_ENV +exit 1 +``` +Once your ready to run your workflow, make sure it's merged into your main github branch, for us it's `develop`. + +Then it will run on it's CRON schedule, and you can also dispatch it manually from the actions page to run the check immediately. + +### Conclusion + +Github Actions is capable of running jobs on a timer. It's native to your github repository so you don't need to muck about with maintaining a deployment of it in the cloud, or writing infrastructure code. + +Next time you ask yourself how to deploy one of these simple scripted checks, consider github actions. \ No newline at end of file From a63797008cc243444a43ec0d104c898897ca7a61 Mon Sep 17 00:00:00 2001 From: Daniel Sont Date: Fri, 1 Apr 2022 13:12:07 +0700 Subject: [PATCH 2/3] monitoring with github actions post --- content/posts/monitoring-with-github-actions/index.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/content/posts/monitoring-with-github-actions/index.md b/content/posts/monitoring-with-github-actions/index.md index 98669852..7803d8f2 100644 --- a/content/posts/monitoring-with-github-actions/index.md +++ b/content/posts/monitoring-with-github-actions/index.md @@ -27,6 +27,7 @@ To do this, were going to use three Github actions. ```yaml # github/workflows/check_weather.yml` + name: check_weather on: schedule: # run at 010:45 UTC daily @@ -38,14 +39,13 @@ jobs: runs-on: ubuntu-latest steps: - # checkout code + # checkout code - uses: actions/checkout@v2 # run script - name: Check Weather shell: bash - run: | - ./check-weather.sh + run: ./check-weather.sh # post results - if: failure() @@ -68,6 +68,8 @@ We're using this action [rtCamp/action-slack-notify](https://github.com/rtCamp/a We need to perform some check (like the weather), and exit with status 0 if it's ok, or exit with a non-zero status if theres a problem. ```bash +# check-weather.sh + #!/bin/bash # fetch the temperature From 415bdd27582e177d9d1a67d4ccf000d9b7af7906 Mon Sep 17 00:00:00 2001 From: Daniel Sont Date: Mon, 4 Apr 2022 13:49:47 +0700 Subject: [PATCH 3/3] slight edits to monitoring-with-github-actions --- content/posts/monitoring-with-github-actions/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/posts/monitoring-with-github-actions/index.md b/content/posts/monitoring-with-github-actions/index.md index 7803d8f2..cd7ada06 100644 --- a/content/posts/monitoring-with-github-actions/index.md +++ b/content/posts/monitoring-with-github-actions/index.md @@ -12,7 +12,7 @@ slug = "github-actions-for-monitoring-tasks" Github actions has a CRON schedule trigger we can use run scripts on a custom schedule. It's pretty cool that Github Actions can run jobs on a CRON schedule, and we can use this ability to cover some of our dev-ops needs. -At Manabie, we used this to check if any of our domain certificates were close to their expiry. Each day, a job is starts at 10:45AM, checks the certificates of each cluster, and posts to our `#monitoring` Slack if any certificate is too close to expiring. +For example, we used this to check if any of our domain certificates were close to their expiry. Each day, a github action runs at 10:45, checks if our certificates are dangerously close to expiring, and posts to our `#monitoring` Slack if any of them are. ### The Workflow