What is the Best Process Model Representation? A Comparative Analysis for Process Modeling with Large Language Models

This repository contains the code and data for the paper "What is the Best Process Model Representation ? A Comparative Analysis for Process Modeling with Large Language Models" submitted to AI4BPM'2025.

Authors:

• Alexis Brissard
• Frédéric Cuppens
• Amal Zouaq

Project Organization

The repository is organized as follows:

├── config.ini           <- Configuration file for project paths
├── pyproject.toml       <- Python project configuration
├── README.md            <- Project documentation
├── conversion/          <- PMR conversion outputs organized by conversion type
├── data/                <- Project datasets and cached data
│   ├── pmo_dataset/     <- PMo Dataset
│   ├── pmr/             <- Process Model Representations (PMR) data (examples and schemas)
│   └── similarity_cache/ <- Cached text similarity computations
├── experiments/         <- Experiment predictions and analytics
│   ├── pmg/             <- Process Model Generation experiment (generated models)
├── prompts/             <- Prompts for different tasks and model configurations
│   ├── basic/           <- Basic PMG prompts
│   ├── pmg.md           <- PMG prompt template
│   └── pmg_without_modeling_instructions.md <- PMG prompt without modeling instructions
├── results/             <- Results and analysis outputs
│   ├── intrinsic_comparison/  <- Intrinsic (PMR for PMo) evaluation results
│   └── pmg_comparison/        <- PMG (PMR for PMG) comparison results
└── src/                 <- Source code
    ├── conversion/      <- PMR conversion code
    ├── evaluation/      <- Evaluation metrics and methods
    ├── generation/      <- LLM generation code
    ├── pmo_dataset/     <- PMo Dataset utilities
    ├── promoai/         <- PromoAI code (for POWL code)
    ├── similarity/      <- Similarity computation methods (for text, lists and process model elements)
    ├── utils/           <- General utility functions
    ├── intrinsic_comparison.py <- Intrinsic (PMR for PMo) evaluation script
    ├── pmg_comparison.py       <- PMG (PMR for PMG) comparison script
    └── pmg_experiment.py       <- PMG experiment runner

Setup

Requirements

• Python 3.8 or higher

Installation

1. Clone the repository:

   git clone https://github.com/Lama-West/Process_Model_Representations.git
   cd Process_Model_Representations

2. Install the package in development mode:

   pip install -e .

   This will install all required dependencies including:

   • google_genai - Google Generative AI API
   • pm4py - For process model conversions
   • rich - Rich text and console formatting
   • sentence_transformers - Text embeddings for similarity computations
   • tabulate - Pretty-print tabular data

3. Optional: Install development dependencies:

   pip install -e ".[dev]"

   This includes additional tools for development:

   • ruff - Fast Python linter and formatter

Google Vertex AI API Setup

If you want to run the PMG experiments with Google Vertex AI, you need to set up the Google Cloud SDK and authenticate your account:

1. Install Google Cloud SDK: Follow the instructions at Google Cloud SDK Installation.

2. Authenticate your account:

    gcloud auth login

Run the Experiments

To run the PMG experiments, execute the following command:

python src/pmg_experiment.py

You can modify the main function to run specific LLMs and/or PMRs. Generated process models will be saved in the experiments/pmg/ directory and automatically converted to PME.

Run the Intrinsic (PMR for PMo) Comparison

To run the intrinsic comparison of PMRs for PMo, execute the following command:

python src/intrinsic_comparison.py

This will evaluate the PMRs using the PMo Dataset and save the results in the results/intrinsic_comparison/ directory.

Run the PMG (PMR for PMG) Comparison

To run the comparison of PMRs for PMG, execute the following command:

python src/pmg_comparison.py

This will evaluate the PMRs using the PMo Dataset and save the results in the results/pmg_comparison/ directory.