Course project for Engineering Production Ready ML/AI Systems

This is a full production pipeline + demo for exercise prediction that does the following:

• Download the RecGym dataset: RecGym Dataset Page, RecGym Kaggle Page with working download link
• Feature-engineering with PySpark (add delta columns that measure rate of change), and data-processing (preparing training data in a sliding-window pattern that ingests 4s of 20Hz sensor data and predicts the exercise type)
• Training a simple ConvNet to predict the exercises, log training to Weights & Biases.
• JIT-compile the trained model and expose it through a FastAPI web server
• Provide a phone demo (HTML page) that logs the phone's IMU, sends the data to the prediction endpoint, and continuously outputs the predicted exercise.
• Everything is dockerized and starts with docker-compose up.
• Includes an Airflow DAG container that redoes the feature engineering and model training nightly
• Includes a Prometheus server for crawling and storing log data of the app
• Includes a Grafana server for visualizing the log data and sending alerts.

SETUP

Get a Weights & Biases api key from here: https://wandb.ai/authorize, have it in your clipboard.

[OPTIONAL] If you have a GMail account and want to use Grafana Alerts, generate an app password in Google: https://myaccount.google.com/apppasswords. The result will be a 16 character string that can be used where indicated below.

cd containers

# copy the secrets file
cp TEMPLATE.env .env

# ! now edit the `.env` and paste the wandb api key under `WANDB_API_KEY=...`, save.
# [OPIONAL] if using the GMail alerts, set your email address in `GRAFANA_EMAIL_ACCT=...` 
# ...and add the app password you just generated in `GRAFANA_EMAIL_PASSWD=...`

# change the email recipient in the grafana provisioning to the one from the `.env`
bash change-email-recipient.sh 

# start the contaienrs
docker compose up --build

...and wait until you see the FastAPI server start. This is indicated by a line like

app-1         |     server   Server started at http://0.0.0.0:8000

This will start 4 containers: Prometheus, Grafana, Airflow, and my app. The names of the containers are prometheus, grafana, airflow, app, respectively.

Now, install and authenticate Ngrok. This is important to use the mobile app demo. https://ngrok.com/download

Sign up for a free account if you don't have an ngrok account yet.

To expose the web app and make it reachable from any phone:

ngrok http 8000

This will generate a few outputs, one of them is a line similar to:

...
Forwarding   https://2422996caa9a.ngrok-free.app -> http://localhost:8000

IMPORTANT the 2422996caa9a will be a different string every time you launch ngrok.

Finally, open a browser on your phone (currently only tested with Chrome on Android) and type in the address from above:

https://wwwxxxyyyzzz.ngrok-free.app and in the resulting page, click on the link to the demo.

---

USE

The servers can be reached at the following addresses.

• FastAPI webserver: http://localhost:8000/
• Airflow: http://localhost:8080/
• Prometheus: http://localhost:9090/
• Grafana: http://localhost:3000/

Prometheus is set up to just scrape the data from the web app every 3s and nothing else.

Grafana is provisioned to:

• use the Prometheus data source,
• create a dashboard showing CPU, RAM, requests, & predictions, and make that the home screen
• auto-login
• have a default email contact
• send alerts when the RAM usage exceeds 2GB

Airflow has 3 steps:

• feature-engineering and preprocessing the data in PySpark
• re-training the model
• hot-reloading the model through the FastAPI server

Data

• For training on pocket data only, do and replace appropriate filename in the PathsConfig data_file variable (see below) awk -F, 'NR==1 || $2 == "pocket"' RecGym.csv > RecGym_pocket.csv head -n 100000 RecGym_pocket.csv > RecGym_pocket_small.csv
• For "wrist":
  • awk -F, 'NR==1 || $2 == "wrist"' RecGym.csv > RecGym_wrist.csv
  • head -n 100000 RecGym_wrist.csv > RecGym_wrist_small.csv
• For "leg":
  • awk -F, 'NR==1 || $2 == "leg"' RecGym.csv > RecGym_leg.csv
  • head -n 100000 RecGym_leg.csv > RecGym_leg_small.csv
• Currently the data preprocessing does not do one hot encoding for position so it's useful to filter anyway

App

• On startup, the app runs a download script to download the full dataset RecGym.csv and a smaller version for testing RecGym_small.csv. No need to manually download the dataset.
• The data_file variable on line 19 in src > engproj > configs of the PathsConfig class is set to the testing dataset. You may change the variable to RecGym.csv to process and train on the full file, but it will take longer (RecGym has ~4.7mil rows vs 100k rows in RecGym_small)
• RecGym.csv was created by running head -n 100000 RecGym.csv > RecGym_small.csv (catches only position=wrist data)
• To run the scripts inside this container, connect a shell to it (inside the containers directory): docker compose exec app bash
  • To manually run data preprocessing, make sure you have the RecGym.csv file in your /data directory, then run python3 scripts/02-process-data.py, which will generate /data/processed_data.npz
  • To manually train the model, run python3 scripts/06-train.py
  • To manually reload the model, visit http://localhost:8000/reload on the host machine or inside the app container
  • To see a few sample API requests, run bash scripts/12-api-request-examples.sh. You can see the telemetry metrics change at http://localhost:8000/metrics or on the Grafana dashboard
  • Finally, if the phone live demo is not working, you can run pre-recorded phone data through the prediction endpoint. To do this, run python3 scripts/14-send-mock-data.py
• Used adb for mirroring the phone to laptop for styling & debugging, access through chrome://inspect#devices

---

TESTING

To test:

# start a bash inside the app container
docker compose exec app bash   

cd tests
pytest  

Test coverage report

Name                                         Stmts   Miss  Cover   Missing
--------------------------------------------------------------------------
/app/scripts/04-fastapi-app.py                  50      2    96%   70, 75
/app/src/engproj/__init__.py                     0      0   100%
/app/src/engproj/configs.py                     42      0   100%
/app/src/engproj/data_processing.py            176     18    90%   109-111, 118-120, 448-468, 471-473
/app/src/engproj/fastapi/middleware.py           8      0   100%
/app/src/engproj/fastapi/utils.py               25      3    88%   18-21
/app/src/engproj/net.py                         77     32    58%   62-84, 89-109
conftest.py                                      7      0   100%
integration/test_airflow_integration.py         15      0   100%
integration/test_fastapi_integration.py         51      0   100%
integration/test_prometheus_integration.py       8      0   100%
unit/test_data_processing.py                   181      3    98%   128, 135-136
unit/test_model.py                               8      0   100%
--------------------------------------------------------------------------
TOTAL                                          648     58    91%