Skip to content

Data Collection Documentation

Jump to bottom
Felix-Deng edited this page Apr 14, 2023 · 9 revisions

Documentation on Data Collection

While the front-end of the app supports CAD modelling challenge questions with automatic geometric evaluation of correctness, the back-end of the app also collects a selection of design data from the users to support research studies.

Data Types

Each of the following sections details one type of design data that the app is currently collecting. However, at different occurrences, different types of data are being collected (see the data collection section for more details).

Microversion descriptions

For every change made in an Onshape document, an immutable microversion is automatically created and logged by Onshape in the document’s history. When the data is recorded in the app, the username of every microversion entry (as obtained from Onshape's API) is automatically removed. This data type is retrieved using the Document/getDocumentHistory API endpoint (ref).

The microversion history is stored as a list of JSON objects in reversed chronological order. Every element of the list stores the description of one microversion. An example of a microversion description is shown below:

{
    "date": "2022-12-15T16:11:34.613+00:00",
    "canBeRestored": true,
    "description": "Part Studio 1 :: Insert feature : Sketch 1",
    "userId": "888888888888888888888888",
    "microversionId": "b54e0c116a5957a5e8f16a8f",
    "nextMicroversionId": "97c088c56cc53473e43d9bfb"
}

Feature list definition

In an Onshape Part Studio, the models in the Part Studio are constructed and defined by a list of parametric features (i.e., the feature list). The definition of all features in the list, including the semantic naming of and the parameters specified to the features, can be retrieved using the PartStudio/getPartStudioFeatures API endpoint (ref).

In an Onshape Assembly, the feature list refers to the list of mate features used to define the relationship between part instances. The feature list definition can be retrieved using the Assembly/getFeatures API endpoint (ref).

All feature list definitions are stored in JSON format, as it is output from the API response.

Shaded views of the workspace

In both Onshape Part Studio and Assembly, a shaded view image of the workspace can be taken using the API with the view angle specified by a view matrix. For every occurrence of shaded view image capture in the app, two isometric views are taken from the model. Every isometric projection is defined by a $\pm 45 \degree$ rotation of a vertical axis, followed by a rotation of $\arcsin{(\frac{1}{\sqrt{3}})} \degree$ about a horizontal axis. By default, all images are in the size of 128 $\times$ 128 pixels, and pixel size is automatically sized to fit the output image dimensions. With two images, all six faces of the view cube are captured:

  • FRT image captures the front, right, and top face of the view cube
  • BLB image Capture the back, left, and bottom face of the view cube

Shaded view images are captured using the PartStudio/getPartStudioShadedViews API endpoint (ref) in Part Studio and the Assembly/getAssemblyShadedViews API endpoint (ref) in Assembly. Data stored in the app is in the following JSON format, where all images are base64 encoded, replacing the xxxxxx:

{
    "FRT": "data:image/png;base64,xxxxxx", 
    "BLB": "data:image/png;base64,xxxxxx"
}

To view images in Python for example, given imgs being the dictionary of shaded view images above:

import base64 
from IPython.display import display, Image 

img = base64.b64decode(imgs['FRT'].split(',')[1])
display(Image(img))

Mesh representation of the model

For Part Studios only, all parts in the model can exported in mesh representation to allow geometric analysis of the product. The mesh data is first retrieved using the PartStudio/exportPartStudioGltf API endpoint (ref). Then, the GLTF file is automatically converted to STL format to be stored. All stored STL files are base64 encoded to be stored as strings.

To open and view a collected STL file in Python for example, given an STL file is stored in the variable stl_data as a string:

import trimesh 
import base64

stl_data = base64.b64decode(stl_data)
mesh = trimesh.load(trimesh.util.wrap_as_stream(stl_data), file_type="stl", force="mesh")
mesh.show()

Assembly definition

For Assemblies only, information of all part instances and mating features in the Assembly are exported for analysis. This is retrieved using the Assembly/getAssemblyDefinition API endpoint (ref).

Data Collection

At different occurrences, as detailed in the following sections, different types of design data (as introduced in the data type section) are being collected.

At first failure attempt

After the first submission attempt, if the submitted model fails to pass the automatic evaluation, the following data are collected:

For "single-part part studio" questions and "multi-part part studio" questions:

For "assembly mating" questions:

No data are collected for "multi-step part studio" questions.

At successful completion

After the successful attempt, which the submitted model passes the automatic evaluation, the following data are collected:

For all question types:

  • The microversion history of the entire document from the microversion logged when the user started the question to the microversion logged when the model submission is evaluated to be correct

For "single-part part studio" questions and "multi-part part studio" questions:

  • The feature list definition at the point of submission
  • Two shaded view images of the Part Studio
  • One mesh representation of the entire Part Studio, stored in the format of [(-1, "xxx")], where xxx is the base64 encoded STL as string
    • Ideally, one mesh can be exported after every non-sketch and non-plane feature in the feature list, where the first tuple item of every list item specifies the rollback bar index. However, due to technical constraints, this feature is currently not implemented.

For "assembly mating" questions:

For "multi-step part studio" questions:

  • The time of completion for every step in a list of string
  • The feature list definition for every completed step in a list of JSON
  • Two shaded view images of the Part Studio for every completed step in a list of dictionary

At give-up confirmation

After the first failed submission attempt, the user is given the option to "give-up" on the question. The "give-up" button is available if and only if at least one failed submission has been made. At the point when the user gives up the question, the same types of data are collected as if the submission attempt was successful, as detailed in the previous section. However, the field for the boolean field is_final_failure will be set to be True.

No data are collected for "multi-step part studio" questions.

Data Inclusion Criteria

Since data storage space is limited and can be expensive in the cloud, there are inclusion criteria set in the code to limit data collection from the data_miner of the app. Generally, we aim to collect a large enough amount of data for every question from a larger variety of users. In other words, we try to avoid collecting a lot of data for a few "old" questions that have been available for a long time, while losing the space available for newer questions. Meanwhile, we also try to avoid collecting too much data from only a few users, especially who keep attempting the same questions without much improvements.

Unless otherwise over-ridden by the maintainers, the following data inclusion criteria are implemented in the app:

  • Every question collects a maximum of 250 entries of successful completion (i.e., data collection for a question stops once question.completion_count >= 250).
  • The first 3 successful completions of a user for every question are collected (i.e., when len(user.completed_history[str(question)]) <= 3).
  • After the first 3 entries for a question, new successful attempts are only recorded if and only if the new attempt spent less than 20% of the time spent for the fastest successful attempt in the user's history. In other words, we only collect data for new attempts when the user gains big improvements on the question.

Access to Database

All collected data are stored in the Heroku PostgreSQL database in data.heroku.com. For basic access, simply log in to your Heroku account, given you have the access permission. "Dataclips" are available on the website to run PostgreSQL queries and access data.

If more advanced data analysis is to be performed, you may need to use (e.g., download, modify, transform) the data. This should be done through an external script with connection established to the cloud database. To access the database, the "URI" of the database can be found in the "Datastore" that is accessible as stated in the previous paragraph. Under the "Settings" tab, you can view "Database Credentials". The "URI" should start with postgres:// and contain compute-1.amazonaws.com:5432. Using the URI, a remote connection to the database can be established. Use the following code in Python to retrieve all collected design data in Part Studios from the users of the app:

import pandas as pd 
from sqlalchemy import create_engine, text 

database_url = "postgresql://xxx.compute-1.amazonaws.com:5432/xxx"
engine = create_engine(database_url)
df = pd.DataFrame(engine.connect().execute(text('''
    SELECT * 
    FROM data_miner_historydata hd
    LEFT JOIN data_miner_historydata_ps ON hd.id = historydata_ptr_id
    ORDER BY hd.final_query_complete_time DESC; 
''')))
print(df.info())

Notes:

  • Replace the database_url with the actual URI of the database. Then, make sure to replace the starting postgres:// with postgresql://. Everything following // should stay the same.
  • You may modify the query inside pd.read_sql() with your own PostgreSQL query to be executed.
  • If everything works as intended, the queried database will be returned and stored in your df variable.
  • The database_url for the app is NOT permanent. Whenever an automatic maintenance is initiated and completed by Heroku, the database_url will change without prior notice.

You can either work within your local runtime of the Python script to conduct any research analysis you need to, or you can also export the collected data to be used in the future without querying the database every time you come back to this work.

However, there are two points should be noted:

  • Data cannot be exported in CSV format. While CSV format separates columns with a separating character (usually , by default), the dataset itself contains many of the separator (even if you try other characters), hence the columns are not well separated when exported. However, exporting the data in Excel format works fine.
  • When exporting the returned pandas Dataframe with the code above to Excel, the pandas package cannot export datetime objects with timezone information, hence we need to first clean the data columns with datetime data to remove the timezone information before exporting.

Following the code block from above, use the following code block to export the returned data to Excel:

datetime_cols = ['start_time', 'time_of_completion', 'final_query_complete_time', 'first_failed_time']
for col in datetime_cols: 
    df[col] = df[col].dt.tz_localize(None)
df.to_excel("data.xlsx")

Clone this wiki locally