Conceptual Design of Multidimensional Cubes with LLMs: An Investigation

A research project by the Business Intelligence Group.

Index of this repository.

• Project structure: gives an overview of the repository.
• Research data: shows the input used and output obtained in our research activity.
• Usage: explains how to use the code to run new experiments.

IMPORTANT: to run the project via API, an active account on OpenAI is needed to connect to ChatGPT, while for import, Huggingface is the provider used to import models.

Project structure

datasets/                   -- where case studies are stored
    *-text.yml                -- input of the case study
    *-ground-truth.yml        -- expected result
inputs/                     -- where prompt templates are stored
outputs/                    -- where generated outputs are stored (not committed)
results/                    -- where final results are stored (committed)
llm4dfm/                    -- source code
tests/                      -- test code

The module llm4dfm/pipeline contains Python files implementing a pipeline enabling the systematic interaction with an LLM.

• models.py -- contains model's utils
• pipeline.py -- contains the whole process of importing, batching, querying and storing metrics
• metrics.py -- contains the process of calculating metrics (precision, recall, f1-measure)
• preprocess.py -- contains the preprocessing phase (e.g., to remove spaces and underscores)
• utils.py -- contains general utils
• csv_graph.py -- contains the process to generate graphs
• merge_graph.py -- contains the process to generate graphs for each exercise aggregating all times and f1 accuracy of all models used
• graph_utils.py -- contains utils used to work with graph, such as metrics calculation
• .env -- contains information about program's paths - must be created when cloning the repo, based on .env-example

Configuration files and scripts to automate multiple executions of the pipeline are collected in the llm4dfm/resources module. If not present when cloning the repo, the file must be created based on the respective (filename)-example.yml version.

• credentials.yml -- contains configuration needed to connect to cloud APIs
• preprocess.yml -- contains preprocessing rules to apply (thesaurus)
• pipeline-config.yml -- contains configuration of a single run
• metrics-config.yml -- contains configuration of metrics
• csv-graph-config.yml -- contains configuration of graphs generated from the resulting csv file
• automatic-run.sh -- script to automate the execution of multiple pipeline
• automatic-metrics.sh -- script to automate the recalculation of metrics on obtained outputs
• yml.html -- script to compare ground-truth and output via visualisation
• conf.json -- optional file to provide pipeline configurations in json fashion, a template is provided in conf-example.json

Research data

The data you want to look at is summarized here.

• Prompt templates for the research questions are in the input folder.
• Case studies are in the datasets folder, called "exercises"; for each of them, you can find:
  • The supply-driven requirements with tables described with logical schema: original-text
  • The supply-driven requirements with tables described with SQL DDL: sql-text
  • The demand-driven requirements: demand-text
  • The ground truth for both supply- and demand-driven requirements: ground-truth
• The thesaurus used for equivalence rules in demand-driven design is in the llm4dfm/resources/preprocess.yml file
• The results of all interactions with ChatGPT are in the results folder; for each research question, you can find:
  • The 10 interactions for every case study
  • A CSV file summarizing all obtained outputs and the calculated metrics
  • Charts showing, for each case study:
    • The average F1-measure
    • Average precision and recall on nodes
    • Average precision and recall on edges
    • Boxplot of F1-measures on nodes
    • Boxplot of F1-measures on edges
• An example of a full interaction with ChatGPT (taken from one of the interactions for RQ5 on case study 7) is shown in results/full-chat-example-rq5-exercise7.json

Analyzing results

To analyze the output obtained from ChatGPT, the static web application at llm4dfm/resources/yml.html can be used.

• Open the page on your browser
• Load one of the output .yml files
• Use the available buttons to either draw the ground truth's DFM, draw the LLM's DFM, or draw a comparison of the two DFMs.
  • True positives (on both nodes and edges) are indicated in green
  • False positives (on both nodes and edges) are indicated in red
  • False negatives (on both nodes and edges) are indicated in grey
  • Nodes are indicated in yellow if they are present in the LLM's output with contradicting roles (e.g., both as dimensional attribute and measure), one of them being correct and the other wrong
  • Comparison metrics (precision, recall, and F1-measure) are indicated to the right

Usage

Installation

The project has been developed with Python 3.12.

Venv

It is recommended to manage Python dependencies through virtual environments. See here.

The .venv module provides support for creating lightweight "virtual environments" with their own site directories, optionally isolated from system site directories. Each virtual environment has its own Python binary (which matches the version of the binary that was used to create this environment) and can have its own independent set of installed Python packages in its site directories.

In case .venv folder is not created, venv can be created through

python -m venv .venv

To activate venv in Windows (with bash shell; e.g., git bash)

source .venv/Scripts/activate

To activate venv in Linux

source .venv/bin/activate

Requirements

Poetry is used to manage all dependencies, packages and tasks to provide easier execution.

• Install dependencies with Poetry

curl -sSL https://install.python-poetry.org | python3 -
poetry install

In pyproject.toml file, under the [tool.poe.tasks] section, all the provided tasks can be found.

Configuration

Required configuration parameters

Inside the pipeline module, a .env file must be provided with the following configurations:

• DATASETS -- path to folder containing exercise (case study) texts
• OUTPUTS -- path to folder in which outpust are stored
• AUTO_OUTPUTS -- path to folder in which outpust of automatic runs are stored
• RESULTS -- path to folder in which results are stored
• INPUTS -- path to folder containing exercise prompts
• SAVE_MODELS -- path to folder in which store imported models

If using Azure to interact with model's API, these configurations must be provided too (model_name must be uppercase)

• ENDPOINT_{model-name}

Authentication key

The authentication key to connect to the API endpoint must be stored in llm4dfm/resources/credentials.yml; an example of how the config is structured can be found in llm4dfm/resources/credentials-example.yml.

Pipeline parameters

The following parameters can be configured in llm4dfm/resources/pipeline-config.yml file.

Imported model (to be set if using a locally-imported LLM model)

• name -- model's name (can be a generalization, such as llama-2, the exact name is stored in "models.py" file, if not present you must add it there)
• temperature -- threshold between 0 and 2 that specifies willing to generate more random answers as growing to 1 *if used do_sample must be true
• max_new_tokens -- limit the maximum number of tokens generated in a single call
• do_sample -- boolean, if set specifies to generate more creative output
• top_p -- threshold between 0 and 1 that specifies willing to use a wider set of words as growing to 1 *if used do_sample must be true
• device -- set device to use between cpu and gpu if available, if not cpu is used instead

Api model (to be set if using APIs to connect to a remote endpoint)

• name -- model's name (can be a generalization, such as llama-2, the exact name is stored in "models.py" file, if not present you must add it there)
• label -- model's name used in output file name generated, if not specified it uses name
• deployment -- Deployment name for azure distribution [test-gpt-35, test-gpt-4o]
• api_version -- api model's version
• max_tokens -- it's the maximum length of the generated output
• n_response -- regulates number of responses the model generates
• temperature -- threshold between 0 and 2 that specifies willing to generate more random answers as growing to 2 [0, ..., 1] [Default 0.1]
• stop -- set the stop character(s, if list) that terminate the response when encountered
• top_p -- threshold between 0 and 1 that specifies willing to use a wider set of words as growing to 1 [0, ..., 1]
• top_k -- threshold between 1 and 40 that specifies the number of tokens (with the highest probability) considered for the next generation. Less randomness for lower values [could be only a gemini parameter]

Exercise

• name -- the list of exercises' name (part before version, exercise-N)
• version -- the exercise version (part between exercise-N- and text.yml) [sql, original, demand]
• prompt_version -- the prompt version (part between prompts- and .yml)
• number -- the list of exercises' number [optional, if not given, obtained as last digit of each exercise in name config]

General

• use -- the model to use between import and api
• debug_prints -- enable output prints during execution

Output

• dir_label -- the label used in output directory name

CSV-Graph

The following parameters can be configured in llm4dfm/resources/csv-graph-config.yml file.

• dir -- full directory name, in case it's specified all other parameters will be ignored
• v -- the exercise version (part between exercise-N- and text.yml) [sql, original, demand]
• prompt_v -- the prompt version (part between prompts-v and .yml)
• model_label -- model's label name
• dir_label -- directory in which store file name
• model_loading -- model's loading technique used [import, api]
• device -- device set to be used in case of import, given that in case of gpu not found cpu is set, if no file with gpu tag is found, cpu is checked [cpu, gpu]

Merge-graphs

The script can be configured with these settings:

• root -- directory path where the recursive search for csv files starts, if not given results folder is used
• label -- set the output's directory label that is saved in root path in which store the graphs

Usage example:

python merge_graphs.py --root ../../results/ --label example

As result, a directory ../../results/example is created, containing files exercise-N_f1_vs_time.pdf for each exercise-N found.

It is also possible to state via code include and exclude rules to choose sub-directories to include (only those would be included) or to exclude (only those would be excluded) in recursive search for csv files

def exclude(dir_name):
    return dir_name.endswith('dir-to-exclude')

merged_df = collect_csvs(root_directory, exclude_dir=exclude)

def include(dir_name):
    return dir_name.endswith('dir-to-include')

merged_df = collect_csvs(root_directory, include_dir=include)

Aggregate-times

The script can be configured with these settings:

• root -- directory path where the recursive search for csv files starts, if not given results folder is used

Usage example:

python aggregate_times.py --root ../../results/

As result, a file in root directory {root}/aggregate_times.pdf is created, containing boxplot graph aggregating times on y-axis and model, f1 measure on average for nodes and edges and the device used.

Metrics

The following parameters can be configured in llm4dfm/resources/metrics-config.yml file, under the exercise section.

• dir -- the exercise's directory inside outputs folder
• name -- the exercise name without .yml extension
• verion -- the exercise version [sql, original, demand]
• gt -- the ground truth's exercise
• number -- the exercise number

Preprocessing

In order to apply equality or ignore rules, llm4dfm/resources/preprocess.yml file has been provided. It is split in 2 sections, the first one is the common, that is applied to all exercises, and then rules for each exercise.

Structure of each section is as follows:

common:
  demand:
    equals:
    - Date:
      - day
    ignore:
    - count
    - month
    - year
  supply:
    equals: []
    ignore: []

N_EXERCISE:
  demand:
    equals:
    - item_to_keep_1:
      - elem_1_equal_item_to_keep_1
      - elem_2_equal_item_to_keep_1
    - item_to_keep_2:
      - elem_1_equal_item_to_keep_2
      - elem_2_equal_item_to_keep_2
    ignore:
    - elem_to_ignore_1
    - elem_to_ignore_2
  supply:
    equals:
    - item_to_keep_1:
      - elem_1_equal_item_to_keep_1
      - elem_2_equal_item_to_keep_1
    ignore: []

This state that in exercise N_EXERCISE all elem_1_equal_item_to_keep_1 and elem_2_equal_item_to_keep_1 found in demand exercise will be preprocessed in item_to_keep_1 and so on, and all dependencies that will have elem_to_ignore_1 or elem_to_ignore_2 will be ignored after preprocessing. Thesaurus rules are applied here.

Single run

• Setup authentication
• Configure pipeline, and graph
• Run python pipeline/pipeline.py from llm4dfm directory. If no Exceptions raised, in outputs directory a new directory with a file /{exercise-version}-{exercise-prompt_version}-{model-label}-{device}-{dir_label}/{exercise.name}-{exercise.version}-{exercise.prompt_version}-{model.label}-{new_timestamp}.yml is generated, where device is actually an empty string if model loading is api, "-gpu" if gpu is set as device and available, "-cpu" otherwise. Its structure is as follows:

config:
  name: gpt
  version: 3.5-turbo
  max_tokens: null 
  n_responses: 1
  stop: null
  top_p: 0.9
  top_k: 5

errors:
- dependencies:
    reversed: [number>=0] (reversed edges aren't counted in missing and extra)
    missing: [number>=0]
    extra: [number>=0]
  measures:
    missing: [number>=0]
    extra: [number>=0]
  fact:
    incorrect: [boolean]
    false_fact: [number>=0]
  attributes:
    shared_missing: [number>=0] (bigger than 0 only if extra = 0)
    shared_extra: [number>=0] (bigger than 0 only if missing = 0)
    shared_with_fact_root_missing: [number>=0] (bigger than 0 only if with_fact_root_extra = 0)
    shared_with_fact_root_extra: [number>=0] (bigger than 0 only if with_fact_root_missing = 0)
  miscellaneous:
    extra_disconnected_components: [number>=0] (0 means no extra components)
    extra_tags: [boolean]
- dependencies:
    reversed: [number>=0] (reversed edges aren't counted in missing and extra)
    missing: [number>=0]
    extra: [number>=0]
  measures:
    missing: [number>=0]
    extra: [number>=0]
  fact:
    incorrect: [boolean]
    false_fact: [number>=0]
  attributes:
    shared_missing: [number>=0] (bigger than 0 only if extra = 0)
    shared_extra: [number>=0] (bigger than 0 only if missing = 0)
    shared_with_fact_root_missing: [number>=0] (bigger than 0 only if with_fact_root_extra = 0)
    shared_with_fact_root_extra: [number>=0] (bigger than 0 only if with_fact_root_missing = 0)
  miscellaneous:
    extra_disconnected_components: [number>=0] (0 means no extra components)
    extra_tags: [boolean]

output:
- fact:
    name: FACT_NAME
  measures: 
  - name: MEASURE1_NAME
  - name: MEASURE2_NAME
  dependencies:
  - from: TABLE1.Attr
    to: TABLE2.Attr
  - ...
- fact:
    name: FACT_NAME
  measures:
  - name: MEASURE1_NAME
  - name: MEASURE2_NAME
  dependencies:
  - from: TABLE1.Attr
    to: TABLE2.Attr
  - ...

output_preprocessed:
- fact:
    name: FACT_NAME
  measures:
  - name: MEASURE1_NAME
  - name: MEASURE2_NAME
  dependencies:
  - from: TABLE1.Attr
    label: tp
    to: TABLE2.Attr
  - ...
  ground_truth_labels:
    dependencies:
    - from: TABLE1.Attr
      label: tp
      to: TABLE2.Attr
    - ...
    fact:
      name: FACT_NAME
    measures:
    - name: MEASURE1_NAME
    - name: MEASURE2_NAME
- fact:
    name: FACT_NAME
  measures:
  - name: MEASURE1_NAME
  - name: MEASURE2_NAME
  dependencies:
  - from: TABLE1.Attr
    label: tp
    to: TABLE2.Attr
  - ...
  ground_truth_labels:
    dependencies:
    - from: TABLE1.Attr
      label: tp
      to: TABLE2.Attr
    - ...
    fact:
      name: FACT_NAME
    measures:
    - name: MEASURE1_NAME
    - name: MEASURE2_NAME
gt_preprocessed:
- fact:
    name: FACT_NAME
  measures:
  - name: MEASURE1_NAME
  - name: MEASURE2_NAME
  dependencies:
  - from: TABLE1.Attr
    to: TABLE2.Attr
  - ...
- fact:
    name: FACT_NAME
  measures:
  - name: MEASURE1_NAME
  - name: MEASURE2_NAME
  dependencies:
  - from: TABLE1.Attr
    to: TABLE2.Attr
  - ...
metrics:
- edges:
    precision: [0.0 - 1]
    recall: [0.0 - 1]
    f1: [0.0 - 1]
  nodes:
    precision: [0.0 - 1]
    recall: [0.0 - 1]
    f1: [0.0 - 1]
- edges:
    precision: [0.0 - 1]
    recall: [0.0 - 1]
    f1: [0.0 - 1]
  nodes:
    precision: [0.0 - 1]
    recall: [0.0 - 1]
    f1: [0.0 - 1]

• Configure graph

• Run python pipeline/csv_graph.py from llm4dfm directory. If no Exceptions raised, in outputs/{csv_graph-v}-{csv_graph-prompt_v}-{csv_graph-model_label}-{device}-{csv_graph-dir_label}/ directory, new graph files named graph-boxplot_f1_edges.pdf, graph-boxplot_f1_nodes.pdf, graph-f1_scores_edges_nodes.pdf, graph-precision_recall_edges.pdf, graph-precision_recall_nodes.pdf are generated aggregating precision, recall and f1-measure collected in the csv file inside outputs/{csv_graph-v}-{csv_graph-prompt_v}-{csv_graph-model_label}-{csv_graph-dir_label}/ directory. Example of run: python pipeline/csv_graph.py --exercise_v sql --prompt_version v4 --model_label gpt4o --dir_label test-dir --model_loading import --device cpu

• Configure metrics

• Run python pipeline/metrics.py from llm4dfm directory. If no Exceptions raised, in selected exercise file, metrics section is added/overridden. In case preprocessed ground truth and output are not present, standard ones are used.

Import model

Imported model are set up in models.py from llm4dfm/pipeline directory. Specifically, in method load_model_and_tokenizer there is a match case in which a key name is bound with exact model name used in AutoModelForCausalLM.from_pretrained(model_name). Using Huggingface models, it's required to put in file credentials.yml from llm4dfm/resources model key name and its key used, to make it easier to configure import models from HuggingFace, a single hf clause is provided, and each run with hf in model_name will use these credentials:

my_model:
  key:
    api: my_api_key
    import: my_import_key
hf:
  key:
    api: my_api_key
    import: my_import_key

In case model or tokenizer require additional chat template, it has to be configured in get_chat_template(model_name, tokenizer) method. Moreover, function to batch input has to be provided in load_generate_import_function(name, model, tokenizer, config, debug_print) with the structure function(str) -> str.

Specific prompts has to be stated in inputs folder, a base configuration can be set as follows, while specific ones can be detailed with model's name. If no model name matches, base will be picked.

base:
  - role: system
    content: Prompt content
  - role: user
    content: Prompt content
gpt:
  - role: system
    content: Prompt content
  - role: user
    content: Prompt content
falcon:
  - role: system
    content: Prompt content
  - role: user
    content: Prompt content

It is suggested to check chat constraints of each specific model.

Automatic run

First of all, execution privileges must be granted by means of chmod 700 ./resources/automatic-run.sh.

After activating venv, a task triggered by poetry poe automatic_run run the pipeline with the following configurations:

• number_of_runs -- set number of runs, 1 by default
• ex_version -- set exercise version [sql, original, demand], sql by default
• prompt_version -- set prompt version [rq2, rq3-alg, rq3-dec, rq4, rq5], rq5 by default
• model -- model to use in run
• model_loading -- model loading technique to use in run [api, import]
• model_label -- an optional model label used in yml output generated, empty string by default, if empty model name (configuration's model) is used
• "exercises" -- set exercises to run, if not provided all files matching previous configurations are used [" ... "]
• dir_label -- an optional label used in output directory generated, if not provided a timestamp is generated
• device -- set device to use if model imported, if gpu not available cpu is set as default [cpu, gpu]
• --debug_print -- an optional arg enabling debug prints

This could also be achieved by directly run ./resources/automatic-run.sh from llm4dfm directory, with configurations as stated before.

All configurations specified as argument override the ones provided by configuration file ones. If not specified, optional parameters are read by configuration files instead, all except dir_label, that in place of automatic run is generated if not given.

It is also possible to set configurations through ./resources/conf.json file, following the structure of ./resources/conf-example.json. To pass this, execution has to be done like this poetry poe automatic_run -f ./llm4dfm/resources/conf.json

Example of run: poetry poe automatic_run 1 sql rq3-alg-base gpt api gpt4o "1 2 3 4 5 6 7 8 9" example --debug_print ./resources/automatic-run.sh 1 sql rq3-alg-base gpt api gpt4o "1 2 3 4 5 6 7 8 9" example ./resources/automatic-run.sh 1 sql rq3-alg-base falcon-3-7B-inst-hf import falcon-3 "1 2 3 4 5 6 7 8 9" example gpu ./resources/automatic-run.sh -f my_conf.json

Output: Generate one output file for each run on each file as described before inside outputs/{file_version}-{prompt_version}-{model_label}-{device}-{dir_label}/. Additionally, a csv file output-{file_version}-{prompt_version}-{model_label}-{device}-{dir_label}.csv is generated if not present, else is enriched with run output. Moreover, pipeline/csv_graph.py is run too, generating graphs.

Automatic metrics

First of all, execution privileges must be granted by means of chmod 700 ./resources/automatic_metrics.sh.

It is noticeable to say that exercise number has to be extracted from exercises' names. Depending on exercises name convention in directory where to iterate, it's provided inside script a default function collecting first numeric occurrence, but also a commented one enabling last occurrence in file names

After activating venv from llm4dfm root directory via source .venv/bin/activate, a task triggered by poetry poe automatic_metrics run the program with the following configurations:

• dir -- the directory name inside 'outputs' folder
• version -- the prompt version used [sql, demand], sql by default

This could also be achieved by directly run ./resources/automatic-metrics.sh from llm4dfm directory, with configurations as stated before.

Example of run: poetry poe automatic_metrics demand-rq5-example-gpt4o-demand demand ./resources/automatic-metrics.sh 1 demand-rq5-example-gpt4o-demand demand

Output: File preprocess, metrics calculation and error detection will be executed, results will be overridden in same file.

Aggregate results

In order to aggregate results, llm4dfm/pipeline/[aggregate_times.py, aggregate_times_cpu_gpu.py, aggregate_times_gpts.py], scripts have been provided.

For the llm4dfm/pipeline/aggregate_times.py one, a root parameter should be supplied to state where the recursive search for csv files begins; once found, these are loaded and aggregated for a boxplot for each couple model - device, calculating average F1 for edges and nodes (differing on cpu and gpu execution) and plotting boxplot based on execution times, saved in {root}/aggregate_times.pdf An example of run is poetry run python llm4dfm/pipeline/aggregate_times.py --root results.

For the llm4dfm/pipeline/aggregate_times_cpu_gpu.py one, a root parameter should be supplied to state where the recursive search for csv files begins; once found, these are loaded and aggregated for a boxplot for each couple model - device, calculating average F1 for edges and nodes (considering cpu and gpu execution the same) and plotting boxplot based on execution times, saved in {root}/aggregate_times_cpu_gpu.pdf An example of run is poetry run python llm4dfm/pipeline/aggregate_times_cpu_gpu.py --root results.

For the llm4dfm/pipeline/aggregate_times_gpts.py one, a root parameter should be supplied to state where the recursive search for csv files involving gpt4 and gpt5 begins; once found, these are loaded and aggregated for each couple model - prompt version, calculating average F1 for edges and nodes on y-axis and execution times on x-axis, saved in {root}/aggregate_times_gpts.pdf An example of run is poetry run python llm4dfm/pipeline/aggregate_times_gpts.py --root results.

Deployment on Portainer

To enable deploy for container management platforms such as Portainer, Dockerfile and setup-container.sh have been provided.

The first one is used for image building, provided via docker buildx build --platform linux/amd64,linux/arm64 -t llm4dfm:latest . and once built, it has to be pushed in a container registry. An example is provided via Github registry ghcr, by means of tagging with docker tag llm4dfm:latest ghcr.io/{my-github-username}/llm4dfm:{my-version-tag} and then pushed via docker push ghcr.io/{my-github-username}/llm4dfm:{my-version-tag}. Then, in the platform, the image has to be specified as ghcr.io/{my-github-username}/llm4dfm:{my-version-tag} and once the container is configured and running, credentials in llm4dfm/resources/credentials.yml has to be specified and then setup script setup-container.sh has to be executed. After this setup, the scripts can be executed as stated before.

Tests

To execute the tests' suite, inside llm4dfm root directory, run

poetry poe test

Add new model

In order to add a new model, it's required to set up model loading [and keys if required], prompt and batching functions.

In case a key is required, in llm4dfm/resources/credentials.yml the key must be added in api if model is accessed via api or else in import, as previously reported, in case of a HuggingFace model, ensure hf is in model_name to load credentials from the hf clause

my-new-model-name:
  key:
    api: my-api-key
    import: my-import-key

In place of an API model, in llm4dfm/pipeline/models.py the model loading function must be added

def load_model_api(name, key):
    match name:
        case 'my-new-model-name':
            model = ... # add loading function here
            return model

In place of an import model, in llm4dfm/pipeline/models.py the model and tokenizer loading function must be added

def load_model_and_tokenizer(model_name, key, quantization):
    match model_name:
        case 'my-new-model-name':
            model, tokenizer = ... # add loading function here
    
    return model, tokenizer

Then, model's prompt must be added in inputs/{prompt-version-to-use}.yml. It's worth state that a base prompt is provided in each prompt file, a specific prompt should be added as follows only if the base one doesn't satisfy model's requirement.

my-new-model-name:
  - role: my-first-role
    content: |
      my multi line comment
  - role: my-second-role
    content: |
      my multi line comment
  - ...

Lastly, the generation function must be provided in llm4dfm/pipeline/models.py, in case of an API model

def load_generate_api_function(name, model, config, debug_print) -> Callable[[List[str]], str]:
    def my-new-model-generating-function(chat):
        ...
        output_generated = model.batch(chat) # Substitute specific generating function here
        ...
        return output_generated

    match name:
        case 'my-new-model-name':
            return my-new-model-generating-function

In case of an import model instead, in llm4dfm/pipeline/models.py, if model loading requires further steps (i.e. huggingface models), it's suggested to load it outside the generation function, for example:

def load_generate_import_function(name, model, tokenizer, config, debug_print) -> Callable[[str], str]:
    ...
    def my-new-model-generating-function_from_model(model_to_use):
        
        def my-new-model-generating-function(chat):
            ...
            output_generated = model_to_use.batch(chat) # Substitute specific generating function here
            ...
            return output_generated

        return my-new-model-generating-function

    match name:
        case 'my-new-model-name':
            model_to_use = load(model, tokenizer)
            return my-new-model-generating-function(model_to_use)

Or, if not required to load model:

def load_generate_import_function(name, model, tokenizer, config, debug_print) -> Callable[[str], str]:
    ...

    def my-new-model-generating-function(chat):
        ...
        output_generated = model.batch(tokenizer(chat)) # Substitute specific generating function here
        ...
        return output_generated

    match name:
        case 'my-new-model-name':
            return my-new-model-generating-function