Skip to content

Development Guide

Jump to bottom
Felix-Deng edited this page May 2, 2023 · 1 revision

This page hosts resources that may be valuable to developers and maintainers who would like to replicate and/or extend our work for other educational or research purposes.

First time setup

Local Testing and Development

Follow the instructions below for first time local setup for testing and development. If you are not planning to make any modifications to the app and would like to use it as is, you may skip this section and proceed to web deployment directly.

  1. Git clone this repository to your local directory and open the repository in a local IDE, where Python needs to be available and supported.
  2. Make sure you have HomeBrew installed, instructions to install can be found here.
  3. Install the Postgres.app here to enable PostgreSQL in your computer. After installation, open the app and initiate the database. Try running which psql in a terminal to check if installation is successful.
  4. Install Redis here locally to allow background job queue. After installation, try running redis-server to start the server as a check of successful installation.
  5. Create a local virtual environment in your repository by running the command python3 -m venv venv in a terminal window.
  6. Switch into the created virtual environment with command source venv/bin/activate.
  7. Download all required packages in the virtual environment pip install -r requirements.txt.
  8. Migrate all migration files in the repository to set up your local database for the app with command python3 manage.py migrate.
  9. With a new database, you need to create an intial access credential to the admin portal of the app. Run the command python3 manage.py createsuperuser and follow the prompts to complete the process.
  10. Follow instructions for a local launch of the app.
  11. Before adding any questions to the app as detailed in this section, make sure you first assign at least one "main admin" from the reviewers. Details can be found in this section.

Web Development

We currently deploy the CAD Challenges app through Heroku as the deployment platform, and all the code is prepared to fit Heroku's deployment preference. However, you are not limited to using Heroku as the platform, although certain changes to the code structure may be required.

Follow the instructions below to deploy the web application and connect it to Onshape:

  1. Make sure you have git cloned this repository to your own GitHub space.
  2. You may follow instructions from Heroku on deploying a Django app here.
    • All required packages for deployment should've already been prepared in the current repository.
    • We use the Heroku PostgreSQL database
    • We utilize one web dyno (for the Questioner) and one worker dyno (for the Data_Miner). You may not need a worker dyno if you are not planning to collect any design data for research from the users
  3. After deployment on Heroku, you should now own a URL for your app, in the form of https://<app_name>.herokuapp.com/.
  4. Go to the Onshape Developer Portal to create a new OAuth application for this app:
    1. Under "OAuth applications" of the left panel, create a new OAuth application
      • For "Redirect URLs", put https://<app_name>.herokuapp.com/oauthRedirect.
      • For "OAuth URL", put https://<app_name>.herokuapp.com/oauthSignin.
      • For "Permissions", only read and write access is required for this app.
      • All other settings don't really matter to the deployment of the app.
    2. Once you click "Create application", a pop-up window will display the application's client secret. Make sure this secret key is recorded, as it will not show up again after the window is closed.
      • The client ID of the application can be found under the "Keys and secret" tab in the OAuth application.
    3. Click the "Extensions" tab in your new OAuth application to create a tab-based extension.
      • Assign a "Name" for the extension that will show up in the Onshape user interface.
      • For "Location", choose "Element right panel".
      • For "Context", choose "Inside part studio".
      • For "Action URL", enter the following URL:
      https://<app_name>.herokuapp.com/oauthSignin?etype=partstudios&did={$documentId}&wvm={$workspaceOrVersion}&wvmid={$workspaceOrVersionId}&eid={$elementId}
      • You will also need an SVG "Icon" for your app. This can be any icon that may help you identifing your app in the Onshape user interface.
    4. Follow the same procedure to create a second extension for Assemblies. You can re-use all the settings for the extension that you just created, except the following:
      • You may want to use a different "Name" for the second extension, so that you know which one to edit if needed in the future.
      • Choose "Inside assembly" for "Context".
      • For "Action URL", enter the following URL:
      https://<app_name>.herokuapp.com/oauthSignin?etype=assemblies&did={$documentId}&wvm={$workspaceOrVersion}&wvmid={$workspaceOrVersionId}&eid={$elementId}
  5. Go to the "Details" tab of your OAuth application, and click "Create store entry". Such that, you can subscribe to your app in the Onshape app store.
    • Unless you are trying to make this app public, the information required on this page does not really matter, and this app will only be visible and accessible to the team that you specify in "Team Visibility".
    • With your Onshape account signed in, you should now be able to find your app in the Onshape [App Store] under the "Category" you specified. The app should be marked with a "Private" label, and you can now "Subscribe" to your app.
  6. Either use the Heroku CLI in your local IDE or type manually through the Heroku web UI, set the following configuration variables for your app: 
    $ heroku config:set OAUTH_CLIENT_ID=<client-id-from-created-app-in-dev-portal>
$ heroku config:set OAUTH_CLIENT_SECRET=<client-secret-from-created-app-in-dev-portal>
$ heroku config:set OAUTH_URL=https://oauth.onshape.com

Daily maintenance and use

Add new questions to database

See instructions on Wiki page Quesiton Preperation Guide and add questions through the admin portal of the app (base_url/admin/). It also works the same way for localhost database.

If this is your first time accessing the admin portal of the web app, first run heroku run python manage.py createsuperuser in a local IDE with Heroku CLI connected and follow the prompts to create the initial super user.

Add reviewer to the app

See instructions on Wiki page User Management Guide and assign reviewers through the admin portal of the app. It also works the same way for localhost database.

Local launch of the app

  1. Make sure you are in the virtual environment of the repository. I.e., you should see (venv) (base) User@usercomputer at the beginning of your command line. Otherwise, start the environment by running the command source venv/bin/activate.
  2. If you started a new terminal session, add configuration variables to your virtual environment by running the following commands:
    • export OAUTH_URL=https://oauth.onshape.com
    • export OAUTH_CLIENT_ID=xxxxxxxxxxxx
    • export OAUTH_CLIENT_SECRET=xxxxxxxxxxxxxxxxx
  3. Double check if uninstalled packages are used by the app, and install them if any, with the command pip install -r requirements.txt.
  4. Make sure you migrate any new changes to your local database with the command python3 manage.py migrate.
  5. Launch the web with command python3 manage.py runserver.
  6. Open a new terminal and launch your local redis library for background job queue with command redis-server.
  7. Open a third terminal window and complete step 1 and 2 again in the new terminal window. Then, launch the web worker with command python3 manage.py rqworker high default low.
  8. Make sure you have three terminals running steps 5 - 8, respectively. Once finished with local testing, you can quit all of them with ctrl + C.

Local development

If changes are made to the fields of any Django models (typically changes made to the attributes in classes in any one of the models.py files), you need to create new migration files to initiate updates to the database. If unsure, always run step 1 below.

  1. First run python3 manage.py makemigrations to scan if any changes are made.
  2. If changes are detected, new migration files are automatically created. Run python3 manage.py migrate to apply those new changes to your local database.
  3. No changes are required for the cloud database, as automatic actions are already specified in the Procfile.

If new Python packages are installed and used for the app, make sure it is recorded in the requirements.txt, so the cloud version also has access to the library.

  1. First run pip install xxx in the virtual environment, so that the package is installed locally.
  2. Run pip freeze > requirements.txt to update the requirement file.

If unnecessary or unused packages are installed or added to the requirements.txt, you should uninstall the packages to keep the cloud repository as light-weight as possible.

  1. First run pip uninstall xxx in the virtual environment to uninstall the package.
  2. Run pip freeze > requirements.txt to update the requirement file.

When new packages are used by other contributors and added to the requirements.txt, or when packages are upgraded to newer versions, you may want to update/upgrade your local development environment:

  1. Make sure you pull the updated changes to your local development environment.
  2. In the virtual environment, run pip install -r requirements.txt to install all the packages (and versions) specified in the requirements.txt file. Packages that are already installed (and up-to-date) will not be installed with duplicates.
    • For security reasons, you may sometimes need to add --trusted-host pypi.org --trusted-host files.pythonhosted.org to the end of the command above.

Manually modify database

While changes to some fields and entries of the database are locked or forbidden through the user interface of the app, changes can be made manually through the shell. This can be done for both the local database and the database in the cloud. However, this should be done with extra caution!

  1. Launch the shell in the terminal with command
    • python3 manage.py shell for local database
    • heroku run python manage.py shell for Heroku cloud database
  2. Now you should see >>> at the front of the next available command line, and any Python codes you commit will be applied to the database.

Example of how you would update an entry:

  1. Open the database in one window with the data table you would to modify opened.
  2. Import the Django model that the data belongs to: e.g., from questioner.models import Question_SPPS
  3. Query for the entry (row) of the data (more documentation can be found here): e.g., q = Question_SPPS.objects.get(question_id=3)
  4. Make the changes you would like to make to the data: e.g., q.completion_feature_cnt = [10,2,4,10,20,3]
  5. Make sure the changes are saved to the database, otherwise all changes will be discarded. E.g., q.save()
  6. Quit the shell when finished: quit()

Other Maintenance Issues and Solutions

DEBUG mode

In mysite/settings.py, we can specify the DEBUG mode to be either True or False. When an error occurs in the app:

  • If DEBUG = True, the detailed error message will be shown
  • If DEBUG = False, only an error code (e.g., 404, 500, 503) will be shown

When an app is deployed and made public, DEBUG should be set to be False. In other words, you should not ask the users to debug for you. However, if the app functions properly when you launched it locally but returns error when deployed in Heroku, you can set DEBUG = True temporarily to see the error returned to better understand the problem. Always remember to set DEBUG = False when you finish debugging.

Heroku DATABASE_URL maintenance

For essential-tier (mini and basic) Heroku PostgreSQL database, Heroku will automatically initiate a maintenance to the database occasionally without much prior notice. Email notifications will be sent to the maintainers of the Heroku app a few minutes before the maintenance starts and right after the maintenance completes, and the maintenance should normally only take ~10 to 20 minutes. During maintenance, the database will be read-only. However, every log in to our app through Onshape requires some form of modifications (write) to the database, the app is essentially not available for use during maintenance.

After maintenance, a new DATABASE_URL should be generated by Heroku for the PostgreSQL database, and this should be automatically updated to the app's "Config Vars" under the "Settings" tab of the Heroku app's page. However, there had been occurrences when this does not happen.

Every time after an automatic update, as noticed by email notifications from Heroku, there should be a new entry under the "Activity" tab of the Heroku app, saying heroku-postgresql: Update DATABASE by heroku-postgresql. If this action is not automatically shown, the database will remain read-only, and launching the app through Onshape will keep returning either a request timeout error or a 500 server error. When DEBUG in mysite/settings.py is set to be True, you will see the error encountered being a ProgramError, saying that your app does not have access to the database. To fix this, use the command heroku pg:credentials:rotate through your terminal to force re-generate the DATABASE_URL for your app.

If the app runs with a production-tier Heroku PostgreSQL database, you can also schedule maintenance window or initiate the maintenance manually before the upgrade deadline. For more information, see the official documentation here.

Clone this wiki locally