Appeng 4536 multi prompt/model factory

README.md

@@ -883,6 +883,44 @@ In any configuration file, locate the `llms` section to see the current settings
 ```
 Please note that the prompts have been tuned to work best with the Llama 3.1 70B NIM and that when using other LLM models it may be necessary to adjust the prompting.
 
+## Customizing the llm prompt
+### Overview
+All the LLM prompts are defined in a single, dedicated file (`multi_prompt_factory.py`). This decouples the application logic from prompt engineering, simplifying maintenance.
+
+The `get_prompt` function allows the caller to fetch a prompt based on three core attributes:
+* **Prompt ID:** Defined by the `PromptId` type, this identifies the specific prompt to be used (e.g., checklist generation, justification, etc.).
+* **LLM:** Specifies the target language model. This can be a simple model name or include additional attributes such as the version.
+* **Language:** Defined by `LanguageAdjustments`, this ensures the prompt uses one of the supported languages.
+
+> [!NOTE]
+This function also supports receiving an additional dictionary for template variables.
+
+Based on these attributes, the function returns the prompt defined in the internal `_CATALOG` dictionary.
+
+### 1. Review and/or Update an Existing Prompt
+To update an existing prompt for a specific combination of attributes (`id`, `llm`, `language`), look for the dedicated function that adds those prompts to the catalog.
+
+Currently, the catalog contains prompts for Llama, Gemma, and Granite using the following functions respectively:
+* `_llama_catalog()`
+* `_gemma_catalog()`
+* `_granite_catalog()`
+
+Each of these functions returns a dictionary that is appended to the main catalog. Simply modify the returned dictionary of the relevant function based on your needs.
+
+### 2. Adding a Prompt for a New LLM or Existing LLM with Different Attributes (e.g., Version)
+* **For an existing LLM with a new version/attribute:** Find the existing function (e.g., `_granite_catalog()`) and append your new prompt configurations directly to it.
+* **For a completely new LLM:** Create a new function that returns the desired dictionary structure (following the pattern of `_granite_catalog()`). Then, ensure it gets merged into the main catalog by adding the following loop:
+
+```python
+for _pid, _entry in _YOUR_NEW_SUB_CATALOG().items():
+    _CATALOG.setdefault(_pid, {}).update(_entry)
+```
+
+> [!NOTE]
+LLM multi-pronpt factory does not determine the LLM to be used.  
+The location that defines the LLM to be used for a specific request type (e.g., checklist_llm, justify_llm, etc.) is defined in the main configuration under llms.  
+For example, for checklist_llm, the configuration is located at llms.checklist.model_name
+
 ### Customizing the embedding model
 Vector databases are used by the agent to fetch relevant information for impact analysis investigations. The embedding model used to vectorize your documents can significantly affect the agent's performance. The default embedding model used by the workflow is the NIM [nvidia/nv-embedqa-e5-v5](https://build.nvidia.com/nvidia/nv-embedqa-e5-v5) model, but you can experiment with different embedding models of your choice.
 

src/vuln_analysis/configs/config-gemma-test.yml

@@ -0,0 +1,269 @@
+# Test config for Google Gemma 4 31B via inference gateway.
+# Based on config-http-openai.yml with these changes:
+#   - Uses cve_file_output (writes .tmp/output.json) instead of cve_http_output
+#   - intel_plugin_config kept (required by schema) but points to localhost:8080;
+#     SimpleHttpIntelPlugin catches ConnectionError and returns empty intel gracefully
+#   - Removes Code/Docs Semantic Search from agent tools (no embedder available)
+#   - ignore_code_embedding: true (skip VDB build, use lexical + call-chain only)
+#   - cve_check_vuln_deps: skip: true
+#   - cve_generate_cvss: skip: true
+#
+# Required env vars:
+#   GHSA_API_KEY          SERPAPI_API_KEY     NVD_API_KEY
+#   NVIDIA_API_BASE       (all model names use RedHatAI/gemma-4-31B-it-FP8-block)
+#   CHECKLIST_MODEL_NAME  CVE_AGENT_EXECUTOR_MODEL_NAME
+#   CODE_VDB_RETRIEVER_MODEL_NAME  DOC_VDB_RETRIEVER_MODEL_NAME
+#   SUMMARIZE_MODEL_NAME  JUSTIFY_MODEL_NAME
+#
+# Start with:
+#   nat serve --config_file src/vuln_analysis/configs/config-gemma-test.yml --port 26466
+
+general:
+  front_end:
+    _type: fastapi
+    endpoints:
+      - path: /health
+        method: GET
+        description: Perform a health check.
+        function_name: health_check
+  use_uvloop: true
+  telemetry:
+    tracing:
+      phoenix:
+        _type: phoenix
+        endpoint: ${OTEL_TRACES_ENDPOINT:-http://localhost:6006/v1/traces}
+        project: cve_agent_gemma
+
+functions:
+  cve_clone_and_deps:
+    _type: cve_clone_and_deps
+    base_git_dir: .cache/am_cache/git
+    base_pickle_dir: .cache/am_cache/pickle
+    base_rpm_dir: .cache/am_cache/rpms
+
+  cve_segmentation:
+    _type: cve_segmentation
+    agent_name: cve_agent_executor
+    embedder_name: nim_embedder
+    base_git_dir: .cache/am_cache/git
+    base_vdb_dir: .cache/am_cache/vdb
+    base_code_index_dir: .cache/am_cache/code_index
+    base_pickle_dir: .cache/am_cache/pickle
+    ignore_code_embedding: true
+
+  cve_fetch_intel:
+    _type: cve_fetch_intel
+    # intel_plugin_config is required by the schema; pointing to a non-existent endpoint
+    # is safe - SimpleHttpIntelPlugin catches all RequestException and returns empty intel.
+    intel_plugin_config:
+      plugin_name: vuln_analysis.data_models.plugins.intel_plugin.SimpleHttpIntelPlugin
+      plugin_config:
+        source: Product Security research
+        endpoint: http://localhost:8080/api/v1/vulnerabilities/{vuln_id}/comments
+
+  cve_process_sbom:
+    _type: cve_process_sbom
+
+  cve_verify_vuln_package:
+    _type: cve_verify_vuln_package
+    skip: true
+    base_git_dir: .cache/am_cache/git
+    llm_name: checklist_llm
+
+  cve_fetch_patches:
+    _type: cve_fetch_patches
+
+  cve_checklist:
+    _type: cve_checklist
+    llm_name: checklist_llm
+    model_family: gemma
+
+  Call Chain Analyzer:
+    _type: transitive_code_search
+    enable_transitive_search: true
+
+  Function Caller Finder:
+    _type: calling_function_name_extractor
+    enable_functions_usage_search: true
+
+  Function Locator:
+    _type: package_and_function_locator
+
+  Function Library Version Finder:
+    _type: calling_function_library_version_finder
+
+  Code Keyword Search:
+    _type: lexical_code_search
+    top_k: 5
+
+  CVE Web Search:
+    _type: serp_wrapper
+    max_retries: 5
+
+  Container Analysis Data:
+    _type: container_image_analysis_data
+
+  cve_agent_executor:
+    _type: cve_agent_executor
+    llm_name: cve_agent_executor_llm
+    model_family: gemma
+    tool_names:
+      # Semantic search tools removed (no embedder available)
+      - Code Keyword Search
+      - CVE Web Search
+      - Call Chain Analyzer
+      - Function Caller Finder
+      - Function Locator
+      - Function Library Version Finder
+    max_concurrency: null
+    max_iterations: 10
+    prompt_examples: false
+    replace_exceptions: true
+    replace_exceptions_value: "I do not have a definitive answer for this checklist item."
+    return_intermediate_steps: false
+    cve_web_search_enabled: true
+    verbose: false
+
+  cve_generate_cvss:
+    _type: cve_generate_cvss
+    skip: true
+    llm_name: generate_cvss_llm
+    tool_names:
+      - Code Keyword Search
+      - Container Analysis Data
+    max_concurrency: null
+    max_iterations: 10
+    prompt_examples: true
+    replace_exceptions: false
+    replace_exceptions_value: "Failed to generate CVSS for this analysis."
+    return_intermediate_steps: false
+    verbose: false
+
+  cve_summarize:
+    _type: cve_summarize
+    llm_name: summarize_llm
+    model_family: gemma
+
+  cve_justify:
+    _type: cve_justify
+    llm_name: justify_llm
+    model_family: gemma
+
+  cve_generate_vex:
+    _type: cve_generate_vex
+    skip: false
+
+  cve_file_output:
+    _type: cve_file_output
+    file_path: .tmp/output.json
+    markdown_dir: .tmp/vulnerability_markdown_reports
+    overwrite: true
+
+  cve_calculate_intel_score:
+    _type: cve_calculate_intel_score
+    llm_name: intel_source_score_llm
+    generate_intel_score: true
+    intel_low_score: 51
+    insist_analysis: false
+
+  health_check:
+    _type: health_check
+
+llms:
+  checklist_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${CHECKLIST_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 2000
+    top_p: 0.01
+
+  code_vdb_retriever_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${CODE_VDB_RETRIEVER_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 2000
+    top_p: 0.01
+
+  doc_vdb_retriever_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${DOC_VDB_RETRIEVER_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 2000
+    top_p: 0.01
+
+  cve_agent_executor_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${CVE_AGENT_EXECUTOR_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 2000
+    top_p: 0.01
+
+  generate_cvss_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${GENERATE_CVSS_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 1024
+    top_p: 0.01
+
+  summarize_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${SUMMARIZE_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 1024
+    top_p: 0.01
+
+  justify_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${JUSTIFY_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 1024
+    top_p: 0.01
+
+  intel_source_score_llm:
+    _type: openai
+    api_key: "EMPTY"
+    base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}
+    model_name: ${JUSTIFY_MODEL_NAME:-RedHatAI/gemma-4-31B-it-FP8-block}
+    temperature: 0.0
+    max_tokens: 1024
+    top_p: 0.01
+
+# Embedder definition kept but not used (ignore_code_embedding: true means no VDB build;
+# the semantic search tools are removed from the agent tool list).
+embedders:
+  nim_embedder:
+    _type: nim
+    base_url: ${NIM_EMBED_BASE_URL:-https://integrate.api.nvidia.com/v1}
+    model_name: ${EMBEDDER_MODEL_NAME:-nvidia/nv-embedqa-e5-v5}
+    truncate: END
+    max_batch_size: 128
+
+workflow:
+  _type: cve_agent
+  cve_clone_and_deps_name: cve_clone_and_deps
+  cve_segmentation_name: cve_segmentation
+  cve_fetch_intel_name: cve_fetch_intel
+  cve_calculate_intel_score_name: cve_calculate_intel_score
+  cve_process_sbom_name: cve_process_sbom
+  cve_verify_vuln_package_name: cve_verify_vuln_package
+  cve_checklist_name: cve_checklist
+  cve_agent_executor_name: cve_agent_executor
+  cve_generate_cvss_name: cve_generate_cvss
+  cve_generate_vex_name: cve_generate_vex
+  cve_summarize_name: cve_summarize
+  cve_justify_name: cve_justify
+  cve_output_config_name: cve_file_output