From b87c3284a4fdc72126b56909e1e3b495e6782705 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:02:36 +0800 Subject: [PATCH 01/28] SOF-7907: doc on contributing new application --- .../docs/tutorials/bring-new-application.md | 368 ++++++++++++++++++ 1 file changed, 368 insertions(+) create mode 100644 lang/en/docs/tutorials/bring-new-application.md diff --git a/lang/en/docs/tutorials/bring-new-application.md b/lang/en/docs/tutorials/bring-new-application.md new file mode 100644 index 000000000..97ee98669 --- /dev/null +++ b/lang/en/docs/tutorials/bring-new-application.md @@ -0,0 +1,368 @@ +# Bring Your Application to Mat3ra Platform + +This page explains how developers and advanced users can contribute a new +application to the Mat3ra platform so that it becomes a first-class option in +both the web-interface and the [Command Line Interface (CLI)]( +../../cli/overview.md). + +The task involves adding necessary configurations to two repositories via pull +requests. A basic understanding of container technologies (Apptainer, +Singularity, or Docker), a GitHub account, Node.js installed locally, and a +working Apptainer `.def` file are required before proceeding. If you need help +with preparing Apptainer definition, please consult the [Add Software]( +../../cli/actions/add-software.md) page first. + +## 1. Overview + +### 1.1. Understand the two-repository architecture + +Contributing a new application involves two repositories. + +[**application-containers-public**](https://github.com/Exabyte-io/application-containers-public) holds the Apptainer definition files and a +`manifest.yml` that drives a GitHub Actions (GHA) workflow. On merge, GHA +builds each image and pushes it to the GitHub Container Registry (GHCR). + +[**standata**](https://github.com/Exabyte-io/standata) holds the platform +metadata, including application name, version, build flavor, the GHCR image tag +to pull, and the runtime environment variables. The platform reads this +repository to populate the application dropdown in the web-interface. Necessary +modulefiles are also generated based this repository for use from the CLI. + +The two repositories are coupled by `imageName` and `imageTag`: the value +provided to `standata` must exactly match the value registered in +`application-containers-public`. As a result, the container pull request must +be merged and the image published before the `standata` pull request can be +merged. + +``` +Developer machine + │ + ├─ PR: .def + manifest.yml → application-containers-public + │ │ + │ GitHub Actions + │ │ + │ oras push → GHCR images + │ + └─ PR: applications YAML → standata + │ + imageName + imageTag ← GHCR + │ + Mat3ra platform +``` + + +## 2. Container repository (`application-containers-public`) + +### 2.1. Fork and clone the repository + +First, fork `github.com/Exabyte-io/application-containers-public` on GitHub, +then clone the fork locally. The top-level layout is: + +``` +application-containers-public/ +├── base/ +├── espresso/ +├── lammps/ +├── nwchem/ +├── manifest.yml +├── inheritance-tree.png +└── .github/workflows/cicd.yml +``` + +Under `base/` are foundational images: an AlmaLinux base, a GNU toolchain +variant, an Intel OneAPI variant, and an NVIDIA HPC SDK variant. Each +application has its own subdirectory containing one `.def` file per build +flavor. + +### 2.2. Add the `.def` file + +Add the Apptainer definition file under the appropriate application directory, +e.g. `espresso/espresso-7.5-gnu.def`. Two conventions must be followed. + +First, bootstrap from an existing base image using `Bootstrap: oras`: + +```singularity +Bootstrap: oras +From: ghcr.io/exabyte-io/application-containers-public/almalinux-apptainer-gnu:9.7-2 +``` + +This reuses tested toolchains and keeps build times short. + +Second, source the parent environment file at the top of `%post`: + +```singularity +%post + if [ -f /.singularity.d/env/91-environment.sh ]; then + . /.singularity.d/env/91-environment.sh + fi +``` + +This picks up environment variables (e.g. OpenMPI `PATH` and `LD_LIBRARY_PATH`) +set by the parent base image. Application-specific runtime variables belong in +the `%environment` section. + +!!!tip "Multi-stage builds" + In order to keep image size small, use a first stage that compiles against + the large toolchain, then copy only the compiled executables into a + lightweight final stage. The Mat3ra platform bind-mounts the required + runtime libraries from the host compute node at job submission time. + +!!!warning "Do not bake large toolchains into the image" + Libraries such as NVIDIA HPC SDK and Intel OneAPI are pre-installed on the + clusters and available via NFS. Embedding them would produce images tens of + GB in size. Declare them instead as `environmentVariables` in `standata` + (see [Section 3.2](#32-add-the-application-version-block)). + +### 2.3. Register the image in `manifest.yml` + +Open `manifest.yml` at the repository root and add an entry: + +```yaml +- name: espresso + path: espresso/espresso-7.5-gnu.def + tag: 7.5-gnu-1 +``` + +The three fields are: + +- `name` — the image name in the registry. +- `path` — location of the `.def` file relative to the repository root. +- `tag` — follows the convention `--` where `N` is a + build iteration starting from `0`. Bump `N` whenever the recipe changes + without a version change; the CI treats tags as immutable and skips a build + if the tag already exists. + +### 2.4. Open the pull request and verify the build + +Open a pull request against the `main` branch of +`application-containers-public`. The CI workflow iterates over `manifest.yml`, +checks whether each tag already exists in GHCR, and if not, runs +`apptainer build` and `oras push`. On pull requests the push step is skipped +(dry run); the image is published only after the PR is merged. + +After merge, the image is available at: + +```bash +apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso:7.5-gnu-1 +``` + +The exact `ghcr.io` URL, image name, and tag are needed in the next section. + + +## 3. Metadata repository (`standata`) + +### 3.1. Fork and clone the repository + +Fork `github.com/Exabyte-io/standata` and clone locally. The relevant +subtree is: + +``` +assets/applications/ +├── applications/ +│ ├── application_data.yml +│ ├── espresso.yml +│ ├── lammps.yml +│ └── ... +├── executables/ +└── templates/ +``` + +Each application has its own YAML file under `applications/`, and +`application_data.yml` is the index that the build process reads. + +### 3.2. Add the application version block + +Create or extend the YAML file for the application, e.g. +`assets/applications/applications/espresso.yml`. Each version block follows +this structure: + +```yaml +- version: '7.5' + isDefault: true + build: GNU + hasAdvancedComputeOptions: true + buildConfig: + moduleName: '7.5-gnu' + imageName: 'espresso' + imageTag: '7.5-gnu-1' + bio: 'Quantum ESPRESSO 7.5 (GCC 11.5.0, OpenMPI 4.1.1 and OpenBLAS)' + dependencies: + - 'mpi/ompi-4.1.1' + environmentVariables: {} +``` + +The `imageName` and `imageTag` fields must exactly match what was registered in +`manifest.yml` in the container repository. This is the link between the two +repositories. + +For applications that require large host-side toolchains (e.g. NVIDIA HPC SDK +or Intel OneAPI), declare the Apptainer environment forwarding variables under +`environmentVariables`. The prefix `APPTAINERENV_` instructs Apptainer to +inject the variable into the container at runtime: + +```yaml + environmentVariables: + APPTAINERENV_PREPEND_PATH: '${SOFTWARE_LIBRARIES_PATH}/nvidia/hpc-sdk/bin' + APPTAINERENV_LD_LIBRARY_PATH: '${SOFTWARE_LIBRARIES_PATH}/nvidia/hpc-sdk/lib' +``` + +`SOFTWARE_LIBRARIES_PATH` is a platform-native variable that resolves to the +correct host directory for the cluster the job lands on. Consult an existing +CUDA or Intel entry in `espresso.yml` as a template when the exact paths are +unknown. + +The remaining fields are: + +- `isDefault: true` → marks the version selected by default in the UI. +- `hasAdvancedComputeOptions: true` → exposes the advanced compute settings + panel. +- `build` → the flavor label shown in the version submenu (e.g. `GNU`, + `Intel`, `CUDA`). + +### 3.3. Register in `application_data.yml` + +Add a single `!include` statement to +`assets/applications/applications/application_data.yml`: + +```yaml +espresso: !include 'applications/espresso.yml' +``` + +In order to expose executables in the UI or provide starter input templates, +also populate `assets/applications/executables//` and +`assets/applications/templates//` respectively. These are optional for a +minimal first contribution. + +### 3.4. Add an executable + +The executable YAML describes the command that the platform runs, the input +files it expects, and the results and monitors it produces. Create +`assets/applications/executables/myapp/myapp.yml` following the LAMMPS +pattern: + +```yaml +isDefault: true +monitors: + - standard_output +results: [] +flavors: + myrun: + isDefault: true + input: + - name: flavor.in + results: [] + monitors: + - standard_output + applicationName: myapp + executableName: myexec +``` + +Each key in `flavors` corresponds to one flavor visible in the workflow +designer unit editor. The `input` list names the input files the template +system will render. `monitors` controls which output streams the platform +captures in real time (at minimum `standard_output`). + +Then register the executable in `assets/applications/executables/tree.yml`: + +```yaml +myapp: + myapp: !include 'executables/myapp/myapp.yml' +``` + +### 3.5. Add an input file template and flavor + +The template system connects the executable flavor to a rendered input file. +This is what the user sees and edits inside the workflow designer unit editor. + +**Step 1: Write the raw input file.** Create the actual input content under +`assets/applications/input_files_templates/myapp/flavor.in`. This is the +default input script that a user starts from. It may be a static file or +contain [Jinja](https://jinja.palletsprojects.com/) template variables if the +platform should substitute material-specific values at runtime. + +**Step 2: Write the flavor YAML.** Create +`assets/applications/templates/myapp/flavor.yml`: + +```yaml +- content: !readFile 'input_files_templates/myapp/flavor.in' + name: flavor.in + contextProviders: [] + applicationName: myapp + executableName: myexec +``` + +The `!readFile` tag inlines the raw input file at build time. `contextProviders` +is an optional list of platform context plugins that inject material or job +properties into the template at render time; leave it empty for a static +template. + +**Step 3: Register the flavor in `templates.yml`.** Add a line to +`assets/applications/templates/templates.yml`: + +```yaml +# myapp +- !include 'templates/myapp/flavor.yml' +``` + +After these three steps, the `flavor` flavor appears in the unit editor when a +user selects `myapp` as the application and `myexec` as the executable. + +### 3.6. Build and validate locally + +Run the following commands in the `standata` checkout to verify that the YAML +parses correctly and the generated data looks as expected: + +```bash +npm install +npm run build:applications +npm run build # to build all assets +``` + +`build:applications` generates per-application JSON under `data/applications/`. +Inspect the diffs to confirm the version block is present, the `imageTag` +matches the container repository, the executable flavor appears, and the +template content renders correctly. + +### 3.7. Open the pull request + +Open a pull request against `standata` only after the container pull request +has been merged and the image is live in GHCR. Commit the generated files +under `data/` and `dist/` produced by the build step above. + + +## 4. Merge order and checklist + +Merge order is mandatory: the container pull request must be merged first so +that the image tag referenced in `standata` is valid when that PR is reviewed. + +``` +1. application-containers-public PR + ✓ .def file added under the correct application directory + ✓ manifest.yml entry with correct name, path, and tag + ✓ CI passes (dry-run build succeeds) + ✓ Merged first + +2. standata PR + ✓ applications/myapp.yml with matching imageName and imageTag + ✓ !include added to application_data.yml + ✓ executables/myapp/myapp.yml with at least one flavor + ✓ myapp entry added to executables/tree.yml + ✓ input_files_templates/myapp/flavor.in created + ✓ templates/myapp/flavor.yml created + ✓ !include added to templates/templates.yml + ✓ npm run build outputs committed + ✓ Opened after the container PR is merged +``` + +Once both PRs are merged and the next platform release ships, the application +appears in the application dropdown for every user. The container image is +pulled from GHCR on first use, the version block drives the runtime environment, +and the flavor/template pair appears in the workflow designer unit editor. + + +## 5. References + +- Local recipe authoring: [Add Software](add-software.md) +- Container repository: [github.com/Exabyte-io/application-containers-public](https://github.com/Exabyte-io/application-containers-public) +- Metadata repository: [github.com/Exabyte-io/standata](https://github.com/Exabyte-io/standata) +- Published images: [Exabyte-io packages on GHCR](https://github.com/orgs/Exabyte-io/packages?repo_name=application-containers-public) From 3e607d858416c3acf52ebdee64a64e072850d263 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:17:47 +0800 Subject: [PATCH 02/28] add mermaid diagram --- ...ation.md => contribute-new-application.md} | 35 +++++++++++-------- mkdocs.yml | 6 +++- 2 files changed, 26 insertions(+), 15 deletions(-) rename lang/en/docs/tutorials/{bring-new-application.md => contribute-new-application.md} (95%) diff --git a/lang/en/docs/tutorials/bring-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md similarity index 95% rename from lang/en/docs/tutorials/bring-new-application.md rename to lang/en/docs/tutorials/contribute-new-application.md index 97ee98669..266e7f098 100644 --- a/lang/en/docs/tutorials/bring-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -34,20 +34,27 @@ provided to `standata` must exactly match the value registered in be merged and the image published before the `standata` pull request can be merged. -``` -Developer machine - │ - ├─ PR: .def + manifest.yml → application-containers-public - │ │ - │ GitHub Actions - │ │ - │ oras push → GHCR images - │ - └─ PR: applications YAML → standata - │ - imageName + imageTag ← GHCR - │ - Mat3ra platform +```mermaid +flowchart TB + Dev[Developer machine] + + subgraph containers["application-containers-public"] + AppRepo[manifest.yml + .def files] + GHA[GitHub Actions] + GHCR[GHCR images] + AppRepo --> GHA --> GHCR + end + + subgraph standata["standata"] + Meta[applications YAML\n+ executables + templates] + end + + Platform[Mat3ra platform] + + Dev -->|"PR: .def + manifest.yml"| AppRepo + Dev -->|"PR: apps YAML + executables + templates"| Meta + GHCR -->|"imageName + imageTag"| Meta + Meta --> Platform ``` diff --git a/mkdocs.yml b/mkdocs.yml index 49ae33a99..01cdc1f64 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,7 +73,11 @@ markdown_extensions: linenums: true - pymdownx.snippets - pymdownx.striphtml # Strip html comments before processing - - pymdownx.superfences # Allow for code blocks "fencing" + - pymdownx.superfences: # Allow for code blocks "fencing" + custom_fences: + - name: mermaid + class: mermaid + format: !!python/name:pymdownx.superfences.fence_code_format - pymdownx.tabbed: alternate_style: true From 2c35d0af329ed27166b5a6596b6cafc839c33278 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:23:07 +0800 Subject: [PATCH 03/28] chore --- lang/en/docs/tutorials/contribute-new-application.md | 8 ++++---- .../tutorials/ml/run-mlff-python-workflows-mattersim.json | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 266e7f098..22f73e802 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -1,4 +1,4 @@ -# Bring Your Application to Mat3ra Platform +# Contribute New Application to Mat3ra Platform This page explains how developers and advanced users can contribute a new application to the Mat3ra platform so that it becomes a first-class option in @@ -132,9 +132,9 @@ Open `manifest.yml` at the repository root and add an entry: The three fields are: -- `name` — the image name in the registry. -- `path` — location of the `.def` file relative to the repository root. -- `tag` — follows the convention `--` where `N` is a +- `name` → the image name in the registry. +- `path` → location of the `.def` file relative to the repository root. +- `tag` → follows the convention `--` where `N` is a build iteration starting from `0`. Bump `N` whenever the recipe changes without a version change; the CI treats tags as immutable and skips a build if the tag already exists. diff --git a/lang/en/docs/tutorials/ml/run-mlff-python-workflows-mattersim.json b/lang/en/docs/tutorials/ml/run-mlff-python-workflows-mattersim.json index 6492eff26..5fcd97267 100644 --- a/lang/en/docs/tutorials/ml/run-mlff-python-workflows-mattersim.json +++ b/lang/en/docs/tutorials/ml/run-mlff-python-workflows-mattersim.json @@ -1,6 +1,6 @@ { "descriptionLinks": [ - "Running Python-based ML workflows in Mat3ra platform: https://docs.mat3ra.com/tutorials/ml/mattersim/" + "Running Python-based ML workflows in Mat3ra platform: https://docs.mat3ra.com/tutorials/ml/run-mlff-python-workflows-mattersim/" ], "description": "How to run Python-based ML workflows in Mat3ra platform.", "tags": [ From e3410e5560a92821d665a109aeb8eabd20abca14 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:41:00 +0800 Subject: [PATCH 04/28] add application selection screenshot --- images/tutorials/new-application/application-modules-cli.webp | 3 +++ .../new-application/application-selection-web-ui.webp | 3 +++ lang/en/docs/tutorials/contribute-new-application.md | 4 ++++ mkdocs.yml | 2 +- 4 files changed, 11 insertions(+), 1 deletion(-) create mode 100644 images/tutorials/new-application/application-modules-cli.webp create mode 100644 images/tutorials/new-application/application-selection-web-ui.webp diff --git a/images/tutorials/new-application/application-modules-cli.webp b/images/tutorials/new-application/application-modules-cli.webp new file mode 100644 index 000000000..5f05b09df --- /dev/null +++ b/images/tutorials/new-application/application-modules-cli.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c4903a02c3ad4cce9576ee256d0e4cbb723dda76fa8a87e79b39862d1d0ad53e +size 94700 diff --git a/images/tutorials/new-application/application-selection-web-ui.webp b/images/tutorials/new-application/application-selection-web-ui.webp new file mode 100644 index 000000000..1287e11f1 --- /dev/null +++ b/images/tutorials/new-application/application-selection-web-ui.webp @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:022a0f5c756d0adf9271013dde48289e10100a826b7a6b870fb6e3d61e5305d6 +size 62588 diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 22f73e802..18226e81d 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -5,6 +5,8 @@ application to the Mat3ra platform so that it becomes a first-class option in both the web-interface and the [Command Line Interface (CLI)]( ../../cli/overview.md). +![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") + The task involves adding necessary configurations to two repositories via pull requests. A basic understanding of container technologies (Apptainer, Singularity, or Docker), a GitHub account, Node.js installed locally, and a @@ -12,6 +14,8 @@ working Apptainer `.def` file are required before proceeding. If you need help with preparing Apptainer definition, please consult the [Add Software]( ../../cli/actions/add-software.md) page first. +![Load application with modulefile](../../../../images/tutorials/new-application/application-modules-cli.webp "Load application with modulefile") + ## 1. Overview ### 1.1. Understand the two-repository architecture diff --git a/mkdocs.yml b/mkdocs.yml index 01cdc1f64..6afbb6da7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,7 +73,7 @@ markdown_extensions: linenums: true - pymdownx.snippets - pymdownx.striphtml # Strip html comments before processing - - pymdownx.superfences: # Allow for code blocks "fencing" + - pymdownx.superfences: # Allow for code blocks "fencing" custom_fences: - name: mermaid class: mermaid From 2c3ce63ebadfba065c9066948655f45472083d4b Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:51:58 +0800 Subject: [PATCH 05/28] chore: checklist --- .../tutorials/contribute-new-application.md | 34 +++++++++---------- 1 file changed, 16 insertions(+), 18 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 18226e81d..75cd527ac 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -346,24 +346,22 @@ under `data/` and `dist/` produced by the build step above. Merge order is mandatory: the container pull request must be merged first so that the image tag referenced in `standata` is valid when that PR is reviewed. -``` -1. application-containers-public PR - ✓ .def file added under the correct application directory - ✓ manifest.yml entry with correct name, path, and tag - ✓ CI passes (dry-run build succeeds) - ✓ Merged first - -2. standata PR - ✓ applications/myapp.yml with matching imageName and imageTag - ✓ !include added to application_data.yml - ✓ executables/myapp/myapp.yml with at least one flavor - ✓ myapp entry added to executables/tree.yml - ✓ input_files_templates/myapp/flavor.in created - ✓ templates/myapp/flavor.yml created - ✓ !include added to templates/templates.yml - ✓ npm run build outputs committed - ✓ Opened after the container PR is merged -``` +### 4.1. application-containers-public PR Checklist + ✅ .def file added under the correct application directory
+ ✅ manifest.yml entry with correct name, path, and tag
+ ✅ CI passes (dry-run build succeeds)
+ ✅ Merged first + +### 4.2. standata PR Checklist + ✅ applications/myapp.yml with matching imageName and imageTag
+ ✅ !include added to application_data.yml
+ ✅ executables/myapp/myapp.yml with at least one flavor
+ ✅ myapp entry added to executables/tree.yml
+ ✅ input_files_templates/myapp/flavor.in created
+ ✅ templates/myapp/flavor.yml created
+ ✅ !include added to templates/templates.yml
+ ✅ npm run build outputs committed
+ ✅ Opened after the container PR is merged Once both PRs are merged and the next platform release ships, the application appears in the application dropdown for every user. The container image is From c9174d64eaf848e732591b381bbe4da3461ab1a0 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:56:08 +0800 Subject: [PATCH 06/28] chore: codeblock --- .../tutorials/contribute-new-application.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 75cd527ac..ad1c73c6e 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -347,21 +347,21 @@ Merge order is mandatory: the container pull request must be merged first so that the image tag referenced in `standata` is valid when that PR is reviewed. ### 4.1. application-containers-public PR Checklist - ✅ .def file added under the correct application directory
- ✅ manifest.yml entry with correct name, path, and tag
+ ✅ `.def` file added under the correct application directory
+ ✅ `manifest.yml` entry with correct name, path, and tag
✅ CI passes (dry-run build succeeds)
✅ Merged first ### 4.2. standata PR Checklist - ✅ applications/myapp.yml with matching imageName and imageTag
- ✅ !include added to application_data.yml
- ✅ executables/myapp/myapp.yml with at least one flavor
- ✅ myapp entry added to executables/tree.yml
- ✅ input_files_templates/myapp/flavor.in created
- ✅ templates/myapp/flavor.yml created
- ✅ !include added to templates/templates.yml
- ✅ npm run build outputs committed
- ✅ Opened after the container PR is merged + ✅ `applications/myapp.yml` with matching `imageName` and `imageTag`
+ ✅ `!include` added to `application_data.yml`
+ ✅ `executables/myapp/myapp.yml` with at least one flavor
+ ✅ `myapp` entry added to `executables/tree.yml`
+ ✅ `input_files_templates/myapp/flavor.in` created
+ ✅ `templates/myapp/flavor.yml` created
` + ✅ `!include` added to `templates/templates.yml`
+ ✅ `npm run build` outputs committed
+ ✅ Merged after the container PR is merged Once both PRs are merged and the next platform release ships, the application appears in the application dropdown for every user. The container image is From 7187343c3c0498bc9c142e5ca563e89cfcb94e80 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 19 May 2026 18:58:45 +0800 Subject: [PATCH 07/28] chore --- lang/en/docs/tutorials/contribute-new-application.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index ad1c73c6e..8a4bfa708 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -358,7 +358,7 @@ that the image tag referenced in `standata` is valid when that PR is reviewed. ✅ `executables/myapp/myapp.yml` with at least one flavor
✅ `myapp` entry added to `executables/tree.yml`
✅ `input_files_templates/myapp/flavor.in` created
- ✅ `templates/myapp/flavor.yml` created
` + ✅ `templates/myapp/flavor.yml` created
✅ `!include` added to `templates/templates.yml`
✅ `npm run build` outputs committed
✅ Merged after the container PR is merged From 0689a8a292c29c068228db1f1b272ba2b0e1a971 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 20 May 2026 20:40:50 +0800 Subject: [PATCH 08/28] add example pr + other edits --- .../application-selection-web-ui.webp | 4 +- .../tutorials/contribute-new-application.md | 77 +++++++++++-------- 2 files changed, 49 insertions(+), 32 deletions(-) diff --git a/images/tutorials/new-application/application-selection-web-ui.webp b/images/tutorials/new-application/application-selection-web-ui.webp index 1287e11f1..1f63e1f13 100644 --- a/images/tutorials/new-application/application-selection-web-ui.webp +++ b/images/tutorials/new-application/application-selection-web-ui.webp @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:022a0f5c756d0adf9271013dde48289e10100a826b7a6b870fb6e3d61e5305d6 -size 62588 +oid sha256:be20ace90fc1a47772dc1f96301659b11ab5e03a9fdbfbc9e7b968d77d6d8e4f +size 61782 diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 8a4bfa708..86820af37 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -1,36 +1,38 @@ # Contribute New Application to Mat3ra Platform -This page explains how developers and advanced users can contribute a new +This page explains how developers and advanced users can contribute new application to the Mat3ra platform so that it becomes a first-class option in both the web-interface and the [Command Line Interface (CLI)]( ../../cli/overview.md). -![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") - -The task involves adding necessary configurations to two repositories via pull +This task involves adding necessary configurations to two repositories via pull requests. A basic understanding of container technologies (Apptainer, Singularity, or Docker), a GitHub account, Node.js installed locally, and a working Apptainer `.def` file are required before proceeding. If you need help with preparing Apptainer definition, please consult the [Add Software]( ../../cli/actions/add-software.md) page first. -![Load application with modulefile](../../../../images/tutorials/new-application/application-modules-cli.webp "Load application with modulefile") +![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") + ## 1. Overview ### 1.1. Understand the two-repository architecture -Contributing a new application involves two repositories. +Contributing a new application involves creating Pull Requests (PRs) to two +repositories: -[**application-containers-public**](https://github.com/Exabyte-io/application-containers-public) holds the Apptainer definition files and a -`manifest.yml` that drives a GitHub Actions (GHA) workflow. On merge, GHA -builds each image and pushes it to the GitHub Container Registry (GHCR). +[**application-containers-public**]( +https://github.com/Exabyte-io/application-containers-public) holds the Apptainer +definition files and a `manifest.yml` that drives a GitHub Actions (GHA) +workflow. On merge, GHA builds each image and pushes it to the GitHub Container +Registry (GHCR). [**standata**](https://github.com/Exabyte-io/standata) holds the platform metadata, including application name, version, build flavor, the GHCR image tag to pull, and the runtime environment variables. The platform reads this repository to populate the application dropdown in the web-interface. Necessary -modulefiles are also generated based this repository for use from the CLI. +modulefiles are also generated based on these metadata for CLI use. The two repositories are coupled by `imageName` and `imageTag`: the value provided to `standata` must exactly match the value registered in @@ -62,7 +64,7 @@ flowchart TB ``` -## 2. Container repository (`application-containers-public`) +## 2. application-containers-public repository ### 2.1. Fork and clone the repository @@ -80,17 +82,17 @@ application-containers-public/ └── .github/workflows/cicd.yml ``` -Under `base/` are foundational images: an AlmaLinux base, a GNU toolchain -variant, an Intel OneAPI variant, and an NVIDIA HPC SDK variant. Each -application has its own subdirectory containing one `.def` file per build -flavor. +Under `base/` are foundational images: AlmaLinux base, GNU toolchain variant, +Intel OneAPI variant, NVIDIA HPC SDK variant, and so on. Each application has +its own subdirectory containing one `.def` file per build variant. ### 2.2. Add the `.def` file Add the Apptainer definition file under the appropriate application directory, e.g. `espresso/espresso-7.5-gnu.def`. Two conventions must be followed. -First, bootstrap from an existing base image using `Bootstrap: oras`: +First, bootstrap from an existing base image using `Bootstrap: oras` wherever +possible: ```singularity Bootstrap: oras @@ -116,7 +118,7 @@ the `%environment` section. In order to keep image size small, use a first stage that compiles against the large toolchain, then copy only the compiled executables into a lightweight final stage. The Mat3ra platform bind-mounts the required - runtime libraries from the host compute node at job submission time. + runtime libraries from the host compute node at job execution time. !!!warning "Do not bake large toolchains into the image" Libraries such as NVIDIA HPC SDK and Intel OneAPI are pre-installed on the @@ -126,7 +128,8 @@ the `%environment` section. ### 2.3. Register the image in `manifest.yml` -Open `manifest.yml` at the repository root and add an entry: +Open `manifest.yml` at the repository root and add an entry for the new +application: ```yaml - name: espresso @@ -157,15 +160,19 @@ After merge, the image is available at: apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso:7.5-gnu-1 ``` -The exact `ghcr.io` URL, image name, and tag are needed in the next section. +The image name, and tag are needed in the next section. +### 2.5. Example Pull Request +- [GNU build of Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/application-containers-public/pull/7/files) +- [Intel build of LAMMPS](https://github.com/Exabyte-io/application-containers-public/pull/9/changes) -## 3. Metadata repository (`standata`) + +## 3. standata repository ### 3.1. Fork and clone the repository -Fork `github.com/Exabyte-io/standata` and clone locally. The relevant -subtree is: +Fork `github.com/Exabyte-io/standata` and clone locally. The relevant subtree +is: ``` assets/applications/ @@ -181,9 +188,9 @@ assets/applications/ Each application has its own YAML file under `applications/`, and `application_data.yml` is the index that the build process reads. -### 3.2. Add the application version block +### 3.2. Add application version block -Create or extend the YAML file for the application, e.g. +Create or extend the YAML file for the new application, e.g. `assets/applications/applications/espresso.yml`. Each version block follows this structure: @@ -206,7 +213,7 @@ The `imageName` and `imageTag` fields must exactly match what was registered in `manifest.yml` in the container repository. This is the link between the two repositories. -For applications that require large host-side toolchains (e.g. NVIDIA HPC SDK +For applications that require mapping host-side toolchains (e.g. NVIDIA HPC SDK or Intel OneAPI), declare the Apptainer environment forwarding variables under `environmentVariables`. The prefix `APPTAINERENV_` instructs Apptainer to inject the variable into the container at runtime: @@ -241,8 +248,7 @@ espresso: !include 'applications/espresso.yml' In order to expose executables in the UI or provide starter input templates, also populate `assets/applications/executables//` and -`assets/applications/templates//` respectively. These are optional for a -minimal first contribution. +`assets/applications/templates//` respectively. ### 3.4. Add an executable @@ -315,8 +321,8 @@ template. - !include 'templates/myapp/flavor.yml' ``` -After these three steps, the `flavor` flavor appears in the unit editor when a -user selects `myapp` as the application and `myexec` as the executable. +After these three steps, the `flavor` appears in the unit editor when a user +selects `myapp` as the application and `myexec` as the executable. ### 3.6. Build and validate locally @@ -327,6 +333,7 @@ parses correctly and the generated data looks as expected: npm install npm run build:applications npm run build # to build all assets +npm run test # to run the tests ``` `build:applications` generates per-application JSON under `data/applications/`. @@ -340,6 +347,13 @@ Open a pull request against `standata` only after the container pull request has been merged and the image is live in GHCR. Commit the generated files under `data/` and `dist/` produced by the build step above. +### 3.8. Example Pull Request +- [Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/standata/pull/109/changes) +- [LAMMPS](https://github.com/Exabyte-io/standata/pull/91/changes) + +One may ignore the auto-generated files under `data/`, `dist/`, and `src/` +directories while reviewing the PR changes. + ## 4. Merge order and checklist @@ -366,7 +380,10 @@ that the image tag referenced in `standata` is valid when that PR is reviewed. Once both PRs are merged and the next platform release ships, the application appears in the application dropdown for every user. The container image is pulled from GHCR on first use, the version block drives the runtime environment, -and the flavor/template pair appears in the workflow designer unit editor. +and the flavor/template pair appears in the workflow designer unit editor. The +application is also available via modulefile for CLI use. + +![Load application with modulefile](../../../../images/tutorials/new-application/application-modules-cli.webp "Load application with modulefile") ## 5. References From 164603fce63b780a0c9a36fecf04d8ffba45d5b7 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 20 May 2026 20:41:07 +0800 Subject: [PATCH 09/28] chore: bump deps --- requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index dcdbf54d4..f06001e9f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -21,7 +21,7 @@ googleapis-common-protos==1.74.0 grpcio==1.80.0 grpcio-status==1.80.0 httplib2==0.22.0 -idna==3.11 +idna==3.15 importlib_metadata==8.5.0 Jinja2==3.1.6 latexcodec==3.0.1 @@ -47,7 +47,7 @@ pyasn1==0.6.3 pyasn1_modules==0.4.2 pybtex==0.24.0 Pygments==2.20.0 -pymdown-extensions==10.21.2 +pymdown-extensions==10.21.3 pypandoc==1.16.2 pyparsing==3.1.4 python-dateutil==2.9.0.post0 From a03d727237682527d8e265020305af6972b39ca7 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Fri, 22 May 2026 17:58:33 +0800 Subject: [PATCH 10/28] note on yaml --- .../tutorials/contribute-new-application.md | 33 ++++++++++--------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 86820af37..52d3a33ef 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -36,9 +36,8 @@ modulefiles are also generated based on these metadata for CLI use. The two repositories are coupled by `imageName` and `imageTag`: the value provided to `standata` must exactly match the value registered in -`application-containers-public`. As a result, the container pull request must -be merged and the image published before the `standata` pull request can be -merged. +`application-containers-public`. As a result, the container pull request must be +merged and the image published before the `standata` pull request can be merged. ```mermaid flowchart TB @@ -169,10 +168,14 @@ The image name, and tag are needed in the next section. ## 3. standata repository +Although, the `standata` repository contains JavaScript code, the only changes +required are in the YAML files. YAML is a human-readable data serialization +format, containing key-value pairs, lists, and nested structures, very similar +to JSON or dictionaries in Python and other programming languages. + ### 3.1. Fork and clone the repository -Fork `github.com/Exabyte-io/standata` and clone locally. The relevant subtree -is: +Fork `github.com/Exabyte-io/standata` and clone locally. The relevant subtree is: ``` assets/applications/ @@ -191,8 +194,8 @@ Each application has its own YAML file under `applications/`, and ### 3.2. Add application version block Create or extend the YAML file for the new application, e.g. -`assets/applications/applications/espresso.yml`. Each version block follows -this structure: +`assets/applications/applications/espresso.yml`. Each version block follows this +structure: ```yaml - version: '7.5' @@ -215,8 +218,8 @@ repositories. For applications that require mapping host-side toolchains (e.g. NVIDIA HPC SDK or Intel OneAPI), declare the Apptainer environment forwarding variables under -`environmentVariables`. The prefix `APPTAINERENV_` instructs Apptainer to -inject the variable into the container at runtime: +`environmentVariables`. The prefix `APPTAINERENV_` instructs Apptainer to inject +the variable into the container at runtime: ```yaml environmentVariables: @@ -274,10 +277,10 @@ flavors: executableName: myexec ``` -Each key in `flavors` corresponds to one flavor visible in the workflow -designer unit editor. The `input` list names the input files the template -system will render. `monitors` controls which output streams the platform -captures in real time (at minimum `standard_output`). +Each key in `flavors` corresponds to one flavor visible in the workflow designer +unit editor. The `input` list names the input files the template system will +render. `monitors` controls which output streams the platform captures in real +time (at minimum `standard_output`). Then register the executable in `assets/applications/executables/tree.yml`: @@ -344,8 +347,8 @@ template content renders correctly. ### 3.7. Open the pull request Open a pull request against `standata` only after the container pull request -has been merged and the image is live in GHCR. Commit the generated files -under `data/` and `dist/` produced by the build step above. +has been merged and the image is live in GHCR. Commit the generated files under +`data/` and `dist/` produced by the build step above. ### 3.8. Example Pull Request - [Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/standata/pull/109/changes) From f6e8668437052e71c1cdc678a1ece9990d8a0b92 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Fri, 22 May 2026 18:07:45 +0800 Subject: [PATCH 11/28] put yaml note on an info callout --- lang/en/docs/tutorials/contribute-new-application.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 52d3a33ef..728523382 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -168,10 +168,12 @@ The image name, and tag are needed in the next section. ## 3. standata repository -Although, the `standata` repository contains JavaScript code, the only changes -required are in the YAML files. YAML is a human-readable data serialization -format, containing key-value pairs, lists, and nested structures, very similar -to JSON or dictionaries in Python and other programming languages. +!!!info + Although the `standata` repository contains JavaScript code, the only + changes required to add new application are in the YAML files. YAML is a + human-readable data serialization format, containing key-value pairs, lists, + and nested structures, very similar to JSON or dictionaries in Python and + other programming languages. ### 3.1. Fork and clone the repository From cbdb52525e72488565c36064e7832576eac4cbf0 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Fri, 22 May 2026 19:41:14 +0800 Subject: [PATCH 12/28] format --- .../bring-your-application-planning.md | 397 ++++++++++++++++++ .../tutorials/contribute-new-application.md | 6 +- 2 files changed, 400 insertions(+), 3 deletions(-) create mode 100644 lang/en/docs/tutorials/bring-your-application-planning.md diff --git a/lang/en/docs/tutorials/bring-your-application-planning.md b/lang/en/docs/tutorials/bring-your-application-planning.md new file mode 100644 index 000000000..c76463c75 --- /dev/null +++ b/lang/en/docs/tutorials/bring-your-application-planning.md @@ -0,0 +1,397 @@ +# Bring Your Application to Mat3ra + +A video tutorial transcript for advanced users and developers who want to bring +their own application to the Mat3ra platform as a first-class, selectable +option in the UI and via the CLI. + + +## Scene 1 — Intro and goal + +**On screen.** Mat3ra platform UI; "New Workflow" dialog showing the application +dropdown with `Quantum ESPRESSO`, `LAMMPS`, `NWChem`, etc. Cursor opens the +version submenu. Different versions and builds for the same application. +Applications can also be used via the CLI. Same modulefile is used behind the +scenes. Why containerize? All dependencies, libraries, and custom code are +locked in the image, guaranteeing exact reproducibility of scientific research. + +**Narration.** Welcome. In this tutorial we will discuss how developers and +advanced users can add new applications to Mat3ra platform. Such applications +can be used from both the web-interface and command-line interface. We assume +that you have basic understanding of container technologies such as apptainer, +singularity or docker. It is recommended that you have Node.js is installed in +your local machine. A GitHub account is required to create pull requests and +contribute to Mat3ra repositories. You should have a working Apptainer recipe. +If you do not, please first watch our "Build containerized applications with GPU +support" tutorial, as covered in [add-software.md](add-software.md). + +--- + +## Scene 2 — The two-repo architecture + +**On screen.** A simple architecture diagram: + +```mermaid +flowchart LR + Dev[Developer machine] + AppRepo[application-containers-public] + GHA[GitHub Actions] + GHCR[GHCR images] + Standata[standata] + Platform[Mat3ra platform] + + Dev -->|"PR: .def + manifest.yml"| AppRepo + AppRepo --> GHA + GHA -->|"oras push"| GHCR + Dev -->|"PR: applications YAML"| Standata + GHCR -->|"imageName + imageTag"| Standata + Standata --> Platform +``` + +Then briefly show `inheritance-tree.png` from `application-containers-public`. + +**Narration.** Bringing an application to Mat3ra is a two-repository workflow. +The first repository, `application-containers-public`, is where the Apptainer +definition files live and where GitHub Actions builds and publishes container +images to the GitHub Container Registry. The second repository, `standata`, +is where the platform learns *about* your application: its name, version, +build flavor, the image tag to pull, and the environment variables to set at +runtime. You will open one pull request to each repository. The container +must be built first, because `standata` references it by tag. + +--- + +## Scene 3 — Recap: local prototyping + +**On screen.** Terminal session running `apptainer build --sandbox`, then +`apptainer shell --writable --fakeroot`, then +`apptainer build espresso.sif espresso.def`. Briefly highlight a section of +[add-software.md](add-software.md). + +**Narration.** As a quick recap: you can prototype your recipe locally using +Apptainer sandbox mode, iterate on the build steps interactively, and once +the steps are stable, capture them in a `.def` file. The detailed steps are +in our `add-software` documentation page. From here on we will assume you +have a working `.def` file ready to contribute. + +--- + +## Scene 4 — Forking `application-containers-public` + +**On screen.** GitHub page for +`github.com/Exabyte-io/application-containers-public`; click "Fork". Then a +local clone in the terminal. `tree -L 2` showing: + +``` +application-containers-public/ +├── base/ +├── espresso/ +├── lammps/ +├── nwchem/ +├── manifest.yml +├── inheritance-tree.png +└── .github/workflows/cicd.yml +``` + +Open [base/almalinux-gnu.def](../../../../../application-containers-public/base/almalinux-gnu.def) +and [espresso/espresso-7.5-gnu.def](../../../../../application-containers-public/espresso/espresso-7.5-gnu.def) +side by side. + +**Narration.** Fork the `application-containers-public` repository and clone +your fork locally. The layout is straightforward. Under `base/` you will find +foundational images: an AlmaLinux base, a GNU toolchain variant, an Intel +OneAPI variant, and an NVIDIA HPC SDK variant. Each application has its own +folder with one `.def` file per build flavor. A single `manifest.yml` at the +root drives the GitHub Actions workflow that builds and pushes images. + +--- + +## Scene 5 — Adding the `.def` file + +**On screen.** Editor showing a new file `espresso/espresso-7.5-gnu.def` +based on the existing one: + +```singularity +Bootstrap: oras +From: ghcr.io/exabyte-io/application-containers-public/almalinux-apptainer-gnu:9.7-2 + +%labels + Maintainer Mat3ra.com + Version espresso-7.5-gnu-1 + +%environment + export PATH=/opt/qe-7.5/bin:$PATH + +%post + if [ -f /.singularity.d/env/91-environment.sh ]; then + . /.singularity.d/env/91-environment.sh + fi + + # build steps for Quantum ESPRESSO 7.5 + # ... +``` + +**Narration.** Add your `.def` file under the appropriate application folder. +Two important conventions: first, bootstrap from one of our existing base +images using `Bootstrap: oras` and the `ghcr.io` URI. This reuses already +tested toolchains and keeps builds fast. Second, in your `%post` section, +source `/.singularity.d/env/91-environment.sh` if it exists. This uses +the environment variables defined by the parent base image, for example, +the OpenMPI `PATH` and `LD_LIBRARY_PATH` set up by `almalinux-apptainer-gnu`. +Application-specific runtime variables go into the `%environment` section. + +If your application depends on a large library like NVIDIA HPC SDK or Intel +OneAPI, do not bake it into the image. We will bind-mount it from the host +at runtime; we'll come back to this when we configure `standata`. + +Multi-stage builds: In your definition file, configure a first stage that uses +a base container with the heavy toolchains. Then, define a second, lightweight +final stage and copy only the compiled executables over from the first stage. +Because you leave the heavy compilers behind, the resulting image is small and +fast to pull. Then, at runtime, the Mat3ra platform will simply bind-mount the +required runtime libraries from the host compute node to satisfy your +executables, exactly as we configure in the standata repository. + +--- + +## Scene 6 — Registering in `manifest.yml` + +**On screen.** Open [manifest.yml](../../../../../application-containers-public/manifest.yml) +and add an entry: + +```yaml +- name: espresso + path: espresso/espresso-7.5-gnu.def + tag: 7.5-gnu-1 +``` + +Highlight the tag convention text in the +[README](../../../../../application-containers-public/README.md). + +**Narration.** Next, register your image in `manifest.yml`. Each entry has +three fields: `name`, which becomes the image name in the registry; `path`, +the location of the `.def` file; and `tag`, which follows the convention +`---N`, where `N` is the build iteration +starting from zero. Whenever you change the recipe without changing the version, +bump `N`. The workflow treats the tag as immutable: if the tag already exists in +the registry, the build is skipped, so bumping `N` is how you trigger a rebuild. + +--- + +## Scene 7 — CI builds and publishes + +**On screen.** Open [.github/workflows/cicd.yml](../../../../../application-containers-public/.github/workflows/cicd.yml). +Highlight the loop that iterates over `manifest.yml`, the +`oras manifest fetch` skip check, the `apptainer build` step, and the +`oras push` step. Then switch to the GitHub Actions UI showing a successful +run, then to the `Packages` page on GHCR. + +**Narration.** Open a pull request against `application-containers-public`. +The CI workflow reads `manifest.yml` and, for each entry, first checks +whether the image and tag already exist in the registry. If so, it skips. +Otherwise it runs `apptainer build` to produce a `.sif` file and `oras push` +to publish it to GHCR. On pull requests the build runs as a dry run; the +push only happens after the PR is merged to `main`. Once merged, you can +pull your image like this: + +```bash +apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso:7.5-gnu-1 +``` + +That URL which contains registry, image name, and tag, is the details we need to +provide to `standata` in the next step. + +--- + +## Scene 8 — Forking `standata` + +**On screen.** Fork `github.com/Exabyte-io/standata`. Clone locally. `tree -L 2 assets/applications`: + +``` +assets/applications/ +├── applications/ +│ ├── application_data.yml +│ ├── espresso.yml +│ ├── lammps.yml +│ └── ... +├── executables/ +├── templates/ +├── methods/ +├── models/ +└── categories.yml +``` + +**Narration.** Now we move to the second repository, `standata`. This is +where the Mat3ra platform learns the metadata of every application it offers: +its name, versions, build flavors, default settings, and runtime environment. We +will focus on `assets/applications/applications/`, where each application has +its own YAML file, and `application_data.yml` is the index. + +--- + +## Scene 9 — Adding the application config + +**On screen.** Open [assets/applications/applications/espresso.yml](../../../../../standata/assets/applications/applications/espresso.yml). +Walk through a version block: + +```yaml +- version: '7.5' + isDefault: true + build: GNU + hasAdvancedComputeOptions: true + buildConfig: + moduleName: '7.5-gnu' + imageName: 'espresso' + imageTag: '7.5-gnu-1' + bio: 'Quantum ESPRESSO 7.5 (GCC 11.5.0, OpenMPI 4.1.1 and OpenBLAS)' + dependencies: + - 'mpi/ompi-4.1.1' + environmentVariables: {} +``` + +Then scroll down to a CUDA flavor to show `APPTAINERENV_PREPEND_PATH`, +`APPTAINERENV_LD_LIBRARY_PATH`, and the `SOFTWARE_LIBRARIES_PATH` pattern. + +**Narration.** The application file describes one or more `versions`. For each +version you specify a `build` flavor, for example `GNU`, `Intel`, or `CUDA`, and +a `buildConfig` block. Two fields are critical here: `imageName` and `imageTag`. +They must match exactly what you registered in `manifest.yml` in the container +repository. This is connects the two repositories. + +If your application binds large libraries from the host, for example NVIDIA HPC +SDK or Intel OneAPI, declare them under `environmentVariables` using +`APPTAINERENV_PREPEND_PATH` and `APPTAINERENV_LD_LIBRARY_PATH`. The platform +will set these on the host before invoking `apptainer exec`, and Apptainer will +forward them into the container. Use the existing CUDA and Intel entries in +`espresso.yml` as a template. + +`isDefault: true` marks the version that the UI selects when a user first +picks your application. `hasAdvancedComputeOptions: true` exposes the +advanced compute settings panel in the UI. + +Note on mapping host libraries: + +If your application requires large toolchains—like the NVIDIA HPC SDK for CUDA, +or Intel OneAPI—do not bake these libraries into your container image. Doing so +creates large (tens of GB) images that are slow to pull and hard to manage. + +Instead, Mat3ra pre-installs these libraries directly on the clusters and +available to the compute nodes via network file system (NFS). Your job in +standata is to tell the platform how to connect the host libraries to your +container at runtime. + +We do this in the environmentVariables block. By using the APPTAINERENV_ prefix, +we instruct Apptainer to inject these variables directly into the container's +environment. Notice the variable ${SOFTWARE_LIBRARIES_PATH}. This is a +platform-native variable that automatically resolves to the correct host +directory for the cluster your job lands on. + +In this CUDA example, we append the NVIDIA bin directory to the container's +PATH, and the lib directory to the LD_LIBRARY_PATH. Finally, if your application +needs explicit directory mounting, you can pass an APPTAINER_BIND instruction. +If you are unsure of the exact paths to use, copy the environmentVariables block +from an existing application like LAMMPS or Quantum ESPRESSO. + +--- + +## Scene 10 — Registering in `application_data.yml` + +**On screen.** Open [assets/applications/applications/application_data.yml](../../../../../standata/assets/applications/applications/application_data.yml): + +```yaml +deepmd: !include 'applications/deepmd.yml' +espresso: !include 'applications/espresso.yml' +lammps: !include 'applications/lammps.yml' +nwchem: !include 'applications/nwchem.yml' +python: !include 'applications/python.yml' +shell: !include 'applications/shell.yml' +vasp: !include 'applications/vasp.yml' +``` + +Then briefly show the sibling folders `assets/applications/executables/` and +`assets/applications/templates/`, each containing an `espresso/` subfolder. + +**Narration.** Add a one-line `!include` entry for your application in +`application_data.yml`. That hooks your YAML into the build. For a richer +integration, also populate `assets/applications/executables//` +with the executables you want exposed in the UI, and +`assets/applications/templates//` with starter input file +templates. These are optional for a minimal first contribution but make the +experience smoother for end users. + +--- + +## Flavor/Sub-workflow/Workflow + +## Scene 11 — Building and validating locally + +**On screen.** Terminal in the `standata` checkout: + +```bash +npm install +npm run build:applications +npm run build:runtime-data +``` + +Then `git status` showing changes under `data/applications/` and +`dist/js/runtime_data/`. + +**Narration.** Before pushing, build the application data locally. `npm run +build:applications` parses your YAML and generates the per-application JSON +under `data/applications/`. Then `npm run build:runtime-data` regenerates +the minified runtime data that ships to the client. Inspect the diffs to +confirm your version block landed correctly and the `imageTag` matches the +container repository. + +--- + +## Scene 12 — Opening both pull requests + +**On screen.** Side-by-side GitHub PR pages: one against +`application-containers-public`, one against `standata`. A small checklist +overlay: + +``` +1. application-containers-public PR + - .def file added + - manifest.yml entry with correct tag + - CI green + - Merged first + +2. standata PR + - YAML version block with matching imageName + imageTag + - !include added to application_data.yml + - npm run build outputs committed + - Opens after the container PR is merged +``` + +**Narration.** Open two pull requests. Merge order matters: the container +pull request must merge first, because that is what publishes the image to +GHCR. Only then is the `imageTag` in your `standata` change valid. Once the +image is live, the `standata` pull request can be reviewed and merged +safely. + +--- + +## Scene 13 — After merge + +**On screen.** Mat3ra platform UI; user opens the application dropdown and +sees the new application; selects it; the version submenu shows the new +flavor. The user creates and submits a job. + +**Narration.** After both pull requests are merged and the next platform +release ships, your application appears in the dropdown for every user, the +container is pulled from GHCR on first use, and your version block drives +the runtime environment. From the user's perspective it is a first-class +application — selectable in the UI, runnable from the CLI, and reproducible +across the cluster. + +Thanks for watching, and welcome to the Mat3ra ecosystem. + +--- + +## Links + +- Local recipe authoring details: [add-software.md](add-software.md) +- Container repository: [github.com/Exabyte-io/application-containers-public](https://github.com/Exabyte-io/application-containers-public) +- Standata repository: [github.com/Exabyte-io/standata](https://github.com/Exabyte-io/standata) +- Browse existing images on GHCR: [Exabyte-io packages](https://github.com/orgs/Exabyte-io/packages?repo_name=application-containers-public) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 728523382..84cb042c1 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -224,9 +224,9 @@ or Intel OneAPI), declare the Apptainer environment forwarding variables under the variable into the container at runtime: ```yaml - environmentVariables: - APPTAINERENV_PREPEND_PATH: '${SOFTWARE_LIBRARIES_PATH}/nvidia/hpc-sdk/bin' - APPTAINERENV_LD_LIBRARY_PATH: '${SOFTWARE_LIBRARIES_PATH}/nvidia/hpc-sdk/lib' +environmentVariables: + APPTAINERENV_PREPEND_PATH: '${SOFTWARE_LIBRARIES_PATH}/nvidia/hpc-sdk/bin' + APPTAINERENV_LD_LIBRARY_PATH: '${SOFTWARE_LIBRARIES_PATH}/nvidia/hpc-sdk/lib' ``` `SOFTWARE_LIBRARIES_PATH` is a platform-native variable that resolves to the From dccc7f42402e2c4e5849a4ebd9e606f49547aa68 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Mon, 8 Jun 2026 21:01:50 +0800 Subject: [PATCH 13/28] add to index pages --- lang/en/docs/index-developers.md | 4 ++++ lang/en/docs/index-guide.md | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/lang/en/docs/index-developers.md b/lang/en/docs/index-developers.md index cb8b20f11..2ee838977 100644 --- a/lang/en/docs/index-developers.md +++ b/lang/en/docs/index-developers.md @@ -18,3 +18,7 @@ Programmatic access to the Mat3ra platform. - [Endpoints](rest-api/endpoints.md) — available resources - [API client library](rest-api/api-client.md) - [API examples](rest-api/api-examples.md) — common usage patterns + +## Miscellaneous + +- [Contribute new applications](tutorials/contribute-new-applications.md) — How to contribute new applications to the Mat3ra platform diff --git a/lang/en/docs/index-guide.md b/lang/en/docs/index-guide.md index 0c6604f31..1000dfcc2 100644 --- a/lang/en/docs/index-guide.md +++ b/lang/en/docs/index-guide.md @@ -160,6 +160,7 @@ Platform access, notebook environments, and software management. | Restart from previous job | Resume a calculation from the output of a prior job | [Link](tutorials/other/restart-job.md) | | TensorFlow (GPU) | Run TensorFlow workloads on GPU-enabled compute nodes | [Link](tutorials/general-functionality/tensorflow-gpu.md) | | Add new software | Install custom packages via the CLI environment | [Link]({{ cli_url }}/cli/actions/add-software/) | +| Contribute new applications | Contribute new applications to the Mat3ra platform | [Link](tutorials/contribute-new-applications.md) | \ No newline at end of file +--> From 5a5b3a4b637646454aa4108b0c65f3f5862cf704 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Mon, 8 Jun 2026 21:11:17 +0800 Subject: [PATCH 14/28] chore --- lang/en/docs/index-developers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lang/en/docs/index-developers.md b/lang/en/docs/index-developers.md index 2ee838977..0e4d3eae4 100644 --- a/lang/en/docs/index-developers.md +++ b/lang/en/docs/index-developers.md @@ -21,4 +21,4 @@ Programmatic access to the Mat3ra platform. ## Miscellaneous -- [Contribute new applications](tutorials/contribute-new-applications.md) — How to contribute new applications to the Mat3ra platform +- [Contribute new applications](tutorials/contribute-new-applications.md) — How to bring your own applications to the Mat3ra platform From 99d2f6819ddc3154a16bdae2f4ad6fd79a857702 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Mon, 8 Jun 2026 21:16:08 +0800 Subject: [PATCH 15/28] fix link --- lang/en/docs/index-developers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lang/en/docs/index-developers.md b/lang/en/docs/index-developers.md index 0e4d3eae4..e0bec1a68 100644 --- a/lang/en/docs/index-developers.md +++ b/lang/en/docs/index-developers.md @@ -21,4 +21,4 @@ Programmatic access to the Mat3ra platform. ## Miscellaneous -- [Contribute new applications](tutorials/contribute-new-applications.md) — How to bring your own applications to the Mat3ra platform +- [Contribute new applications](/guide/tutorials/contribute-new-applications) — How to bring your own applications to the Mat3ra platform From 2b6a1b04ee1291411aa52e6e5c9a6de0d275b210 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Mon, 8 Jun 2026 21:23:38 +0800 Subject: [PATCH 16/28] fix link x2 --- lang/en/docs/index-developers.md | 2 +- lang/en/docs/index-guide.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lang/en/docs/index-developers.md b/lang/en/docs/index-developers.md index e0bec1a68..257ffce33 100644 --- a/lang/en/docs/index-developers.md +++ b/lang/en/docs/index-developers.md @@ -21,4 +21,4 @@ Programmatic access to the Mat3ra platform. ## Miscellaneous -- [Contribute new applications](/guide/tutorials/contribute-new-applications) — How to bring your own applications to the Mat3ra platform +- [Contribute new applications]({{ guide_url }}/tutorials/contribute-new-application) — How to bring your own applications to the Mat3ra platform diff --git a/lang/en/docs/index-guide.md b/lang/en/docs/index-guide.md index 1000dfcc2..924fc4054 100644 --- a/lang/en/docs/index-guide.md +++ b/lang/en/docs/index-guide.md @@ -160,7 +160,7 @@ Platform access, notebook environments, and software management. | Restart from previous job | Resume a calculation from the output of a prior job | [Link](tutorials/other/restart-job.md) | | TensorFlow (GPU) | Run TensorFlow workloads on GPU-enabled compute nodes | [Link](tutorials/general-functionality/tensorflow-gpu.md) | | Add new software | Install custom packages via the CLI environment | [Link]({{ cli_url }}/cli/actions/add-software/) | -| Contribute new applications | Contribute new applications to the Mat3ra platform | [Link](tutorials/contribute-new-applications.md) | +| Contribute new applications | Contribute new applications to the Mat3ra platform | [Link](tutorials/contribute-new-application.md) | |"PR: .def + manifest.yml"| AppRepo - AppRepo --> GHA - GHA -->|"oras push"| GHCR - Dev -->|"PR: applications YAML"| Standata - GHCR -->|"imageName + imageTag"| Standata - Standata --> Platform -``` - -Then briefly show `inheritance-tree.png` from `application-containers-public`. - -**Narration.** Bringing an application to Mat3ra is a two-repository workflow. -The first repository, `application-containers-public`, is where the Apptainer -definition files live and where GitHub Actions builds and publishes container -images to the GitHub Container Registry. The second repository, `standata`, -is where the platform learns *about* your application: its name, version, -build flavor, the image tag to pull, and the environment variables to set at -runtime. You will open one pull request to each repository. The container -must be built first, because `standata` references it by tag. - ---- - -## Scene 3 — Recap: local prototyping - -**On screen.** Terminal session running `apptainer build --sandbox`, then -`apptainer shell --writable --fakeroot`, then -`apptainer build espresso.sif espresso.def`. Briefly highlight a section of -[add-software.md](add-software.md). - -**Narration.** As a quick recap: you can prototype your recipe locally using -Apptainer sandbox mode, iterate on the build steps interactively, and once -the steps are stable, capture them in a `.def` file. The detailed steps are -in our `add-software` documentation page. From here on we will assume you -have a working `.def` file ready to contribute. - ---- - -## Scene 4 — Forking `application-containers-public` - -**On screen.** GitHub page for -`github.com/Exabyte-io/application-containers-public`; click "Fork". Then a -local clone in the terminal. `tree -L 2` showing: - -``` -application-containers-public/ -├── base/ -├── espresso/ -├── lammps/ -├── nwchem/ -├── manifest.yml -├── inheritance-tree.png -└── .github/workflows/cicd.yml -``` - -Open [base/almalinux-gnu.def](../../../../../application-containers-public/base/almalinux-gnu.def) -and [espresso/espresso-7.5-gnu.def](../../../../../application-containers-public/espresso/espresso-7.5-gnu.def) -side by side. - -**Narration.** Fork the `application-containers-public` repository and clone -your fork locally. The layout is straightforward. Under `base/` you will find -foundational images: an AlmaLinux base, a GNU toolchain variant, an Intel -OneAPI variant, and an NVIDIA HPC SDK variant. Each application has its own -folder with one `.def` file per build flavor. A single `manifest.yml` at the -root drives the GitHub Actions workflow that builds and pushes images. - ---- - -## Scene 5 — Adding the `.def` file - -**On screen.** Editor showing a new file `espresso/espresso-7.5-gnu.def` -based on the existing one: - -```singularity -Bootstrap: oras -From: ghcr.io/exabyte-io/application-containers-public/almalinux-apptainer-gnu:9.7-2 - -%labels - Maintainer Mat3ra.com - Version espresso-7.5-gnu-1 - -%environment - export PATH=/opt/qe-7.5/bin:$PATH - -%post - if [ -f /.singularity.d/env/91-environment.sh ]; then - . /.singularity.d/env/91-environment.sh - fi - - # build steps for Quantum ESPRESSO 7.5 - # ... -``` - -**Narration.** Add your `.def` file under the appropriate application folder. -Two important conventions: first, bootstrap from one of our existing base -images using `Bootstrap: oras` and the `ghcr.io` URI. This reuses already -tested toolchains and keeps builds fast. Second, in your `%post` section, -source `/.singularity.d/env/91-environment.sh` if it exists. This uses -the environment variables defined by the parent base image, for example, -the OpenMPI `PATH` and `LD_LIBRARY_PATH` set up by `almalinux-apptainer-gnu`. -Application-specific runtime variables go into the `%environment` section. - -If your application depends on a large library like NVIDIA HPC SDK or Intel -OneAPI, do not bake it into the image. We will bind-mount it from the host -at runtime; we'll come back to this when we configure `standata`. - -Multi-stage builds: In your definition file, configure a first stage that uses -a base container with the heavy toolchains. Then, define a second, lightweight -final stage and copy only the compiled executables over from the first stage. -Because you leave the heavy compilers behind, the resulting image is small and -fast to pull. Then, at runtime, the Mat3ra platform will simply bind-mount the -required runtime libraries from the host compute node to satisfy your -executables, exactly as we configure in the standata repository. - ---- - -## Scene 6 — Registering in `manifest.yml` - -**On screen.** Open [manifest.yml](../../../../../application-containers-public/manifest.yml) -and add an entry: - -```yaml -- name: espresso - path: espresso/espresso-7.5-gnu.def - tag: 7.5-gnu-1 -``` - -Highlight the tag convention text in the -[README](../../../../../application-containers-public/README.md). - -**Narration.** Next, register your image in `manifest.yml`. Each entry has -three fields: `name`, which becomes the image name in the registry; `path`, -the location of the `.def` file; and `tag`, which follows the convention -`---N`, where `N` is the build iteration -starting from zero. Whenever you change the recipe without changing the version, -bump `N`. The workflow treats the tag as immutable: if the tag already exists in -the registry, the build is skipped, so bumping `N` is how you trigger a rebuild. - ---- - -## Scene 7 — CI builds and publishes - -**On screen.** Open [.github/workflows/cicd.yml](../../../../../application-containers-public/.github/workflows/cicd.yml). -Highlight the loop that iterates over `manifest.yml`, the -`oras manifest fetch` skip check, the `apptainer build` step, and the -`oras push` step. Then switch to the GitHub Actions UI showing a successful -run, then to the `Packages` page on GHCR. - -**Narration.** Open a pull request against `application-containers-public`. -The CI workflow reads `manifest.yml` and, for each entry, first checks -whether the image and tag already exist in the registry. If so, it skips. -Otherwise it runs `apptainer build` to produce a `.sif` file and `oras push` -to publish it to GHCR. On pull requests the build runs as a dry run; the -push only happens after the PR is merged to `main`. Once merged, you can -pull your image like this: - -```bash -apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso:7.5-gnu-1 -``` - -That URL which contains registry, image name, and tag, is the details we need to -provide to `standata` in the next step. - ---- - -## Scene 8 — Forking `standata` - -**On screen.** Fork `github.com/Exabyte-io/standata`. Clone locally. `tree -L 2 assets/applications`: - -``` -assets/applications/ -├── applications/ -│ ├── application_data.yml -│ ├── espresso.yml -│ ├── lammps.yml -│ └── ... -├── executables/ -├── templates/ -├── methods/ -├── models/ -└── categories.yml -``` - -**Narration.** Now we move to the second repository, `standata`. This is -where the Mat3ra platform learns the metadata of every application it offers: -its name, versions, build flavors, default settings, and runtime environment. We -will focus on `assets/applications/applications/`, where each application has -its own YAML file, and `application_data.yml` is the index. - ---- - -## Scene 9 — Adding the application config - -**On screen.** Open [assets/applications/applications/espresso.yml](../../../../../standata/assets/applications/applications/espresso.yml). -Walk through a version block: - -```yaml -- version: '7.5' - isDefault: true - build: GNU - hasAdvancedComputeOptions: true - buildConfig: - moduleName: '7.5-gnu' - imageName: 'espresso' - imageTag: '7.5-gnu-1' - bio: 'Quantum ESPRESSO 7.5 (GCC 11.5.0, OpenMPI 4.1.1 and OpenBLAS)' - dependencies: - - 'mpi/ompi-4.1.1' - environmentVariables: {} -``` - -Then scroll down to a CUDA flavor to show `APPTAINERENV_PREPEND_PATH`, -`APPTAINERENV_LD_LIBRARY_PATH`, and the `SOFTWARE_LIBRARIES_PATH` pattern. - -**Narration.** The application file describes one or more `versions`. For each -version you specify a `build` flavor, for example `GNU`, `Intel`, or `CUDA`, and -a `buildConfig` block. Two fields are critical here: `imageName` and `imageTag`. -They must match exactly what you registered in `manifest.yml` in the container -repository. This is connects the two repositories. - -If your application binds large libraries from the host, for example NVIDIA HPC -SDK or Intel OneAPI, declare them under `environmentVariables` using -`APPTAINERENV_PREPEND_PATH` and `APPTAINERENV_LD_LIBRARY_PATH`. The platform -will set these on the host before invoking `apptainer exec`, and Apptainer will -forward them into the container. Use the existing CUDA and Intel entries in -`espresso.yml` as a template. - -`isDefault: true` marks the version that the UI selects when a user first -picks your application. `hasAdvancedComputeOptions: true` exposes the -advanced compute settings panel in the UI. - -Note on mapping host libraries: - -If your application requires large toolchains—like the NVIDIA HPC SDK for CUDA, -or Intel OneAPI—do not bake these libraries into your container image. Doing so -creates large (tens of GB) images that are slow to pull and hard to manage. - -Instead, Mat3ra pre-installs these libraries directly on the clusters and -available to the compute nodes via network file system (NFS). Your job in -standata is to tell the platform how to connect the host libraries to your -container at runtime. - -We do this in the environmentVariables block. By using the APPTAINERENV_ prefix, -we instruct Apptainer to inject these variables directly into the container's -environment. Notice the variable ${SOFTWARE_LIBRARIES_PATH}. This is a -platform-native variable that automatically resolves to the correct host -directory for the cluster your job lands on. - -In this CUDA example, we append the NVIDIA bin directory to the container's -PATH, and the lib directory to the LD_LIBRARY_PATH. Finally, if your application -needs explicit directory mounting, you can pass an APPTAINER_BIND instruction. -If you are unsure of the exact paths to use, copy the environmentVariables block -from an existing application like LAMMPS or Quantum ESPRESSO. - ---- - -## Scene 10 — Registering in `application_data.yml` - -**On screen.** Open [assets/applications/applications/application_data.yml](../../../../../standata/assets/applications/applications/application_data.yml): - -```yaml -deepmd: !include 'applications/deepmd.yml' -espresso: !include 'applications/espresso.yml' -lammps: !include 'applications/lammps.yml' -nwchem: !include 'applications/nwchem.yml' -python: !include 'applications/python.yml' -shell: !include 'applications/shell.yml' -vasp: !include 'applications/vasp.yml' -``` - -Then briefly show the sibling folders `assets/applications/executables/` and -`assets/applications/templates/`, each containing an `espresso/` subfolder. - -**Narration.** Add a one-line `!include` entry for your application in -`application_data.yml`. That hooks your YAML into the build. For a richer -integration, also populate `assets/applications/executables//` -with the executables you want exposed in the UI, and -`assets/applications/templates//` with starter input file -templates. These are optional for a minimal first contribution but make the -experience smoother for end users. - ---- - -## Flavor/Sub-workflow/Workflow - -## Scene 11 — Building and validating locally - -**On screen.** Terminal in the `standata` checkout: - -```bash -npm install -npm run build:applications -npm run build:runtime-data -``` - -Then `git status` showing changes under `data/applications/` and -`dist/js/runtime_data/`. - -**Narration.** Before pushing, build the application data locally. `npm run -build:applications` parses your YAML and generates the per-application JSON -under `data/applications/`. Then `npm run build:runtime-data` regenerates -the minified runtime data that ships to the client. Inspect the diffs to -confirm your version block landed correctly and the `imageTag` matches the -container repository. - ---- - -## Scene 12 — Opening both pull requests - -**On screen.** Side-by-side GitHub PR pages: one against -`application-containers-public`, one against `standata`. A small checklist -overlay: - -``` -1. application-containers-public PR - - .def file added - - manifest.yml entry with correct tag - - CI green - - Merged first - -2. standata PR - - YAML version block with matching imageName + imageTag - - !include added to application_data.yml - - npm run build outputs committed - - Opens after the container PR is merged -``` - -**Narration.** Open two pull requests. Merge order matters: the container -pull request must merge first, because that is what publishes the image to -GHCR. Only then is the `imageTag` in your `standata` change valid. Once the -image is live, the `standata` pull request can be reviewed and merged -safely. - ---- - -## Scene 13 — After merge - -**On screen.** Mat3ra platform UI; user opens the application dropdown and -sees the new application; selects it; the version submenu shows the new -flavor. The user creates and submits a job. - -**Narration.** After both pull requests are merged and the next platform -release ships, your application appears in the dropdown for every user, the -container is pulled from GHCR on first use, and your version block drives -the runtime environment. From the user's perspective it is a first-class -application — selectable in the UI, runnable from the CLI, and reproducible -across the cluster. - -Thanks for watching, and welcome to the Mat3ra ecosystem. - ---- - -## Links - -- Local recipe authoring details: [add-software.md](add-software.md) -- Container repository: [github.com/Exabyte-io/application-containers-public](https://github.com/Exabyte-io/application-containers-public) -- Standata repository: [github.com/Exabyte-io/standata](https://github.com/Exabyte-io/standata) -- Browse existing images on GHCR: [Exabyte-io packages](https://github.com/orgs/Exabyte-io/packages?repo_name=application-containers-public) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 84cb042c1..ae4ab190a 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -2,15 +2,14 @@ This page explains how developers and advanced users can contribute new application to the Mat3ra platform so that it becomes a first-class option in -both the web-interface and the [Command Line Interface (CLI)]( -../../cli/overview.md). +both the web-interface and the [Command Line Interface (CLI)](/cli/overview). This task involves adding necessary configurations to two repositories via pull requests. A basic understanding of container technologies (Apptainer, Singularity, or Docker), a GitHub account, Node.js installed locally, and a working Apptainer `.def` file are required before proceeding. If you need help with preparing Apptainer definition, please consult the [Add Software]( -../../cli/actions/add-software.md) page first. +/cli/actions/add-software) page first. ![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") @@ -393,7 +392,7 @@ application is also available via modulefile for CLI use. ## 5. References -- Local recipe authoring: [Add Software](add-software.md) +- Local recipe authoring: [Add Software](/cli/actions/add-software) - Container repository: [github.com/Exabyte-io/application-containers-public](https://github.com/Exabyte-io/application-containers-public) - Metadata repository: [github.com/Exabyte-io/standata](https://github.com/Exabyte-io/standata) - Published images: [Exabyte-io packages on GHCR](https://github.com/orgs/Exabyte-io/packages?repo_name=application-containers-public) From 7dc224f81b76b37510e0cae25d9dbf34c35d59b7 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 20:37:46 +0800 Subject: [PATCH 18/28] chore: quotes --- lang/en/docs/tutorials/contribute-new-application.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index ae4ab190a..d38edcb1a 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -62,7 +62,7 @@ flowchart TB ``` -## 2. application-containers-public repository +## 2. `application-containers-public` repository ### 2.1. Fork and clone the repository @@ -160,12 +160,12 @@ apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso: The image name, and tag are needed in the next section. -### 2.5. Example Pull Request +### 2.5. Example Pull Requests - [GNU build of Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/application-containers-public/pull/7/files) - [Intel build of LAMMPS](https://github.com/Exabyte-io/application-containers-public/pull/9/changes) -## 3. standata repository +## 3. `standata` repository !!!info Although the `standata` repository contains JavaScript code, the only @@ -351,7 +351,7 @@ Open a pull request against `standata` only after the container pull request has been merged and the image is live in GHCR. Commit the generated files under `data/` and `dist/` produced by the build step above. -### 3.8. Example Pull Request +### 3.8. Example Pull Requests - [Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/standata/pull/109/changes) - [LAMMPS](https://github.com/Exabyte-io/standata/pull/91/changes) @@ -364,13 +364,13 @@ directories while reviewing the PR changes. Merge order is mandatory: the container pull request must be merged first so that the image tag referenced in `standata` is valid when that PR is reviewed. -### 4.1. application-containers-public PR Checklist +### 4.1. `application-containers-public` PR Checklist ✅ `.def` file added under the correct application directory
✅ `manifest.yml` entry with correct name, path, and tag
✅ CI passes (dry-run build succeeds)
✅ Merged first -### 4.2. standata PR Checklist +### 4.2. `standata` PR Checklist ✅ `applications/myapp.yml` with matching `imageName` and `imageTag`
✅ `!include` added to `application_data.yml`
✅ `executables/myapp/myapp.yml` with at least one flavor
From 77055eb9897c9c04b93a2e28c5ad197ae0d30b7a Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 21:31:44 +0800 Subject: [PATCH 19/28] update application selection screenshot --- .../new-application/application-selection-web-ui.webp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/images/tutorials/new-application/application-selection-web-ui.webp b/images/tutorials/new-application/application-selection-web-ui.webp index 1f63e1f13..94f4e378a 100644 --- a/images/tutorials/new-application/application-selection-web-ui.webp +++ b/images/tutorials/new-application/application-selection-web-ui.webp @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:be20ace90fc1a47772dc1f96301659b11ab5e03a9fdbfbc9e7b968d77d6d8e4f -size 61782 +oid sha256:676ee0ae90f7f5485c54ab3d6a23d3c226677f035a3d489e3547808c76988ab3 +size 93176 From cacd1e293b186ba99f9bbafe3f9b003766dcddd0 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 21:42:40 +0800 Subject: [PATCH 20/28] chore --- .../new-application/application-selection-web-ui.webp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/images/tutorials/new-application/application-selection-web-ui.webp b/images/tutorials/new-application/application-selection-web-ui.webp index 94f4e378a..02f43bac1 100644 --- a/images/tutorials/new-application/application-selection-web-ui.webp +++ b/images/tutorials/new-application/application-selection-web-ui.webp @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:676ee0ae90f7f5485c54ab3d6a23d3c226677f035a3d489e3547808c76988ab3 -size 93176 +oid sha256:3892a08b00350939adb9f232a73459833f092db99352d0c6332d10e8ee73874a +size 93040 From cba71f64f7ba6baccc558b26003800aca3589636 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 22:02:06 +0800 Subject: [PATCH 21/28] chore --- .../new-application/application-selection-web-ui.webp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/images/tutorials/new-application/application-selection-web-ui.webp b/images/tutorials/new-application/application-selection-web-ui.webp index 02f43bac1..2649b2176 100644 --- a/images/tutorials/new-application/application-selection-web-ui.webp +++ b/images/tutorials/new-application/application-selection-web-ui.webp @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:3892a08b00350939adb9f232a73459833f092db99352d0c6332d10e8ee73874a -size 93040 +oid sha256:e26c6ae0e46b16aefd13254e73d0cf6956797e55e51fbe4684ba9d6b45eaa1bb +size 90700 From 9ac16b99da4eaf08682be7b60cd648a7cc5a9ffb Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 22:13:35 +0800 Subject: [PATCH 22/28] minor edits --- lang/en/docs/tutorials/contribute-new-application.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index d38edcb1a..f689adbe2 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -364,13 +364,13 @@ directories while reviewing the PR changes. Merge order is mandatory: the container pull request must be merged first so that the image tag referenced in `standata` is valid when that PR is reviewed. -### 4.1. `application-containers-public` PR Checklist +### 4.1. "`application-containers-public` PR" Checklist ✅ `.def` file added under the correct application directory
✅ `manifest.yml` entry with correct name, path, and tag
✅ CI passes (dry-run build succeeds)
✅ Merged first -### 4.2. `standata` PR Checklist +### 4.2. "`standata` PR" Checklist ✅ `applications/myapp.yml` with matching `imageName` and `imageTag`
✅ `!include` added to `application_data.yml`
✅ `executables/myapp/myapp.yml` with at least one flavor
From 1e4ae860b69ced474aad186adecbdf61a2daf105 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 23:15:12 +0800 Subject: [PATCH 23/28] minor edits x2 --- lang/en/docs/tutorials/contribute-new-application.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index f689adbe2..4c081f5f0 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -160,8 +160,8 @@ apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso: The image name, and tag are needed in the next section. -### 2.5. Example Pull Requests -- [GNU build of Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/application-containers-public/pull/7/files) +### 2.5. Example Pull Request +- [GNU build of Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/application-containers-public/pull/7/changes) - [Intel build of LAMMPS](https://github.com/Exabyte-io/application-containers-public/pull/9/changes) @@ -351,7 +351,7 @@ Open a pull request against `standata` only after the container pull request has been merged and the image is live in GHCR. Commit the generated files under `data/` and `dist/` produced by the build step above. -### 3.8. Example Pull Requests +### 3.8. Example Pull Request - [Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/standata/pull/109/changes) - [LAMMPS](https://github.com/Exabyte-io/standata/pull/91/changes) @@ -364,13 +364,13 @@ directories while reviewing the PR changes. Merge order is mandatory: the container pull request must be merged first so that the image tag referenced in `standata` is valid when that PR is reviewed. -### 4.1. "`application-containers-public` PR" Checklist +### 4.1. `application-containers-public` PR Checklist ✅ `.def` file added under the correct application directory
✅ `manifest.yml` entry with correct name, path, and tag
✅ CI passes (dry-run build succeeds)
✅ Merged first -### 4.2. "`standata` PR" Checklist +### 4.2. `standata` PR Checklist ✅ `applications/myapp.yml` with matching `imageName` and `imageTag`
✅ `!include` added to `application_data.yml`
✅ `executables/myapp/myapp.yml` with at least one flavor
From 3a931d775009c63667e05e11beed3d26f7c02e5d Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Tue, 9 Jun 2026 23:25:04 +0800 Subject: [PATCH 24/28] minor edits x3 --- lang/en/docs/tutorials/contribute-new-application.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 4c081f5f0..d76cf5b5c 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -160,7 +160,7 @@ apptainer pull oras://ghcr.io/exabyte-io/application-containers-public/espresso: The image name, and tag are needed in the next section. -### 2.5. Example Pull Request +### 2.5. Example Pull Requests - [GNU build of Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/application-containers-public/pull/7/changes) - [Intel build of LAMMPS](https://github.com/Exabyte-io/application-containers-public/pull/9/changes) @@ -351,7 +351,7 @@ Open a pull request against `standata` only after the container pull request has been merged and the image is live in GHCR. Commit the generated files under `data/` and `dist/` produced by the build step above. -### 3.8. Example Pull Request +### 3.8. Example Pull Requests - [Quantum ESPRESSO 7.5](https://github.com/Exabyte-io/standata/pull/109/changes) - [LAMMPS](https://github.com/Exabyte-io/standata/pull/91/changes) From 9e21a003066189768e95cd4210b8ee313a483939 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 10 Jun 2026 00:42:03 +0800 Subject: [PATCH 25/28] add tl;dr --- .../docs/tutorials/contribute-new-application.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index d76cf5b5c..3daf5c7cb 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -11,6 +11,20 @@ working Apptainer `.def` file are required before proceeding. If you need help with preparing Apptainer definition, please consult the [Add Software]( /cli/actions/add-software) page first. +!!!note "TL;DR: How to Contribute a New Application" + Bringing your own application to the Mat3ra platform is a two-step process: + + 1. **Build the Image:** Package your application into an Apptainer + container. Create a PR to the `application-containers-public` repository + with the Apptainer `.def` file and register it in `manifest.yml`. Merge this + PR first so the container image is built and published to the registry. + + 2. **Update the Metadata:** Add the application's YAML metadata, templates, + and executables, ensuring your image tag matches Step 1 exactly. Create a PR + to the `standata` repository. Once merged and deployed, the application will + be available on both the web-interface and the CLI. + + ![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") From 4afc6b9cf24f72b0e7736424040c707f384da5a1 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 10 Jun 2026 00:47:03 +0800 Subject: [PATCH 26/28] move tl;dr to the top --- .../tutorials/contribute-new-application.md | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index 3daf5c7cb..a0e2e8840 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -1,17 +1,6 @@ # Contribute New Application to Mat3ra Platform -This page explains how developers and advanced users can contribute new -application to the Mat3ra platform so that it becomes a first-class option in -both the web-interface and the [Command Line Interface (CLI)](/cli/overview). - -This task involves adding necessary configurations to two repositories via pull -requests. A basic understanding of container technologies (Apptainer, -Singularity, or Docker), a GitHub account, Node.js installed locally, and a -working Apptainer `.def` file are required before proceeding. If you need help -with preparing Apptainer definition, please consult the [Add Software]( -/cli/actions/add-software) page first. - -!!!note "TL;DR: How to Contribute a New Application" +!!!abstract "TL;DR: How to Contribute a New Application" Bringing your own application to the Mat3ra platform is a two-step process: 1. **Build the Image:** Package your application into an Apptainer @@ -24,6 +13,17 @@ with preparing Apptainer definition, please consult the [Add Software]( to the `standata` repository. Once merged and deployed, the application will be available on both the web-interface and the CLI. +This page explains how developers and advanced users can contribute new +application to the Mat3ra platform so that it becomes a first-class option in +both the web-interface and the [Command Line Interface (CLI)](/cli/overview). + +This task involves adding necessary configurations to two repositories via pull +requests. A basic understanding of container technologies (Apptainer, +Singularity, or Docker), a GitHub account, Node.js installed locally, and a +working Apptainer `.def` file are required before proceeding. If you need help +with preparing Apptainer definition, please consult the [Add Software]( +/cli/actions/add-software) page first. + ![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") From 48f1f2f1bcea2a44b9beafe1573769ff80da3a48 Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Wed, 10 Jun 2026 00:52:35 +0800 Subject: [PATCH 27/28] use repo url --- .../en/docs/tutorials/contribute-new-application.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index a0e2e8840..d1cd25151 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -1,17 +1,20 @@ # Contribute New Application to Mat3ra Platform -!!!abstract "TL;DR: How to Contribute a New Application" +!!!abstract "TL;DR" Bringing your own application to the Mat3ra platform is a two-step process: 1. **Build the Image:** Package your application into an Apptainer - container. Create a PR to the `application-containers-public` repository + container. Create a PR to the [application-containers-public]( + https://github.com/Exabyte-io/application-containers-public) repository with the Apptainer `.def` file and register it in `manifest.yml`. Merge this - PR first so the container image is built and published to the registry. + PR first so the container image is built and published to the GitHub + Container Registry (GHCR). 2. **Update the Metadata:** Add the application's YAML metadata, templates, and executables, ensuring your image tag matches Step 1 exactly. Create a PR - to the `standata` repository. Once merged and deployed, the application will - be available on both the web-interface and the CLI. + to the [standata](https://github.com/Exabyte-io/standata) repository. Once + merged and deployed, the application will be available on both the + web-interface and the CLI. This page explains how developers and advanced users can contribute new application to the Mat3ra platform so that it becomes a first-class option in From 3ba67c5aaa43158e683c1455c4334f133cbc1f8e Mon Sep 17 00:00:00 2001 From: Pranab Das <31024886+pranabdas@users.noreply.github.com> Date: Thu, 11 Jun 2026 09:35:07 +0800 Subject: [PATCH 28/28] minor edits --- .../tutorials/contribute-new-application.md | 25 +++++++++++-------- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/lang/en/docs/tutorials/contribute-new-application.md b/lang/en/docs/tutorials/contribute-new-application.md index d1cd25151..267a4ef0b 100644 --- a/lang/en/docs/tutorials/contribute-new-application.md +++ b/lang/en/docs/tutorials/contribute-new-application.md @@ -21,11 +21,12 @@ application to the Mat3ra platform so that it becomes a first-class option in both the web-interface and the [Command Line Interface (CLI)](/cli/overview). This task involves adding necessary configurations to two repositories via pull -requests. A basic understanding of container technologies (Apptainer, -Singularity, or Docker), a GitHub account, Node.js installed locally, and a -working Apptainer `.def` file are required before proceeding. If you need help -with preparing Apptainer definition, please consult the [Add Software]( -/cli/actions/add-software) page first. +requests. A basic understanding of container technologies (such as Apptainer, +Singularity, or Docker), a GitHub account, [Node.js](https://nodejs.org) +installed in local development machine, and a working Apptainer `.def` file are +required before proceeding. If you need help with preparing Apptainer +definition, please consult the [Add Software](/cli/actions/add-software) page +first. ![Application Selection in Web-Interface](../../../../images/tutorials/new-application/application-selection-web-ui.webp "Application Selection in Web-Interface") @@ -83,8 +84,9 @@ flowchart TB ### 2.1. Fork and clone the repository -First, fork `github.com/Exabyte-io/application-containers-public` on GitHub, -then clone the fork locally. The top-level layout is: +First, fork [github.com/Exabyte-io/application-containers-public]( +https://github.com/Exabyte-io/application-containers-public) on GitHub, then +clone the fork locally. The top-level layout is: ``` application-containers-public/ @@ -158,8 +160,8 @@ The three fields are: - `path` → location of the `.def` file relative to the repository root. - `tag` → follows the convention `--` where `N` is a build iteration starting from `0`. Bump `N` whenever the recipe changes - without a version change; the CI treats tags as immutable and skips a build - if the tag already exists. + without a version change for the application; the CI treats tags as immutable + and skips a build if the tag already exists. ### 2.4. Open the pull request and verify the build @@ -193,7 +195,8 @@ The image name, and tag are needed in the next section. ### 3.1. Fork and clone the repository -Fork `github.com/Exabyte-io/standata` and clone locally. The relevant subtree is: +Fork [github.com/Exabyte-io/standata](https://github.com/Exabyte-io/standata) +and clone locally. The relevant subtree is: ``` assets/applications/ @@ -409,7 +412,7 @@ application is also available via modulefile for CLI use. ## 5. References -- Local recipe authoring: [Add Software](/cli/actions/add-software) +- Apptainer Definition and container building: [Adding New Software](/cli/actions/add-software) - Container repository: [github.com/Exabyte-io/application-containers-public](https://github.com/Exabyte-io/application-containers-public) - Metadata repository: [github.com/Exabyte-io/standata](https://github.com/Exabyte-io/standata) - Published images: [Exabyte-io packages on GHCR](https://github.com/orgs/Exabyte-io/packages?repo_name=application-containers-public)