diff --git a/README.md b/README.md index 008b9a323..139070c7c 100644 --- a/README.md +++ b/README.md @@ -64,6 +64,21 @@ limitations under the License. - [Adding Test Time Compute (TTC)](#adding-test-time-compute-ttc) - [Customizing the TTC configuration](#customizing-the-ttc-configuration) - [Writing a custom TTC strategy](#writing-a-custom-ttc-strategy) +- [Evaluation](#evaluation) + - [Inputting Evaluation Datasets](#inputting-evaluation-datasets) + - [Running Multiple Evaluation Repetitions](#running-multiple-evaluation-repetitions) + - [Disable Caching](#disable-caching) + - [Option 1: Disable Caching Globally](#option-1-disable-caching-globally) + - [Option 2: Disable Caching on Specific Components](#option-2-disable-caching-on-specific-components) + - [Example Custom Evaluators](#example-custom-evaluators) + - [Simple Accuracy Evaluator](#simple-accuracy-evaluator) + - [Configuration Options](#configuration-options) + - [Output Format](#output-format) + - [average\_score Field](#average_score-field) + - [eval\_output\_items Field](#eval_output_items-field) + - [Consistency Evaluator](#consistency-evaluator) + - [Configuration Options](#configuration-options-1) + - [Output Format](#output-format-1) - [Troubleshooting](#troubleshooting) - [Git LFS issues](#git-lfs-issues) - [Container build issues](#container-build-issues) @@ -570,13 +585,13 @@ The configuration defines how the workflow operates, including functions, LLMs, - `cve_summarize`: Generates concise, human-readable summarization paragraph from agent results. - `llm_name`: Name of LLM (`summarize_llm`) configured in `llms` section. - `cve_justify`: Assigns justification label and reason to each CVE based on summary. - - `llm_name`: Name of LLM (`justify_llm`) configured in `llms` section. + - `llm_name`: Name of LLM (`justify_llm`) configured in `llms` section.concurrency. - Postprocessing/Output functions - `cve_file_output`: Outputs workflow results to a file. - `file_path`: Defines the path to the file where the output will be saved. - `markdown_dir`: Defines the path to the directory where the output will be saved in individual navigable markdown files per CVE-ID. - `overwrite`: Indicates whether the output file should be overwritten when the workflow starts if it already exists. Will throw an error if set to `False` and the file already exists. Note that the overwrite behavior only occurs on workflow initialization. For pipelines started in HTTP mode, each new request will append the existing file until the workflow is restarted. -3. **LLMs** (`llms`): The `llms` section contains the LLMs used by the workflow. Functions can reference LLMs in this section to use. The supported LLM API types in NeMo Agent toolkit are `nim` and `openai` The models in this workflow use `nim`. +1. **LLMs** (`llms`): The `llms` section contains the LLMs used by the workflow. Functions can reference LLMs in this section to use. The supported LLM API types in NeMo Agent toolkit are `nim` and `openai` The models in this workflow use `nim`. - Configured models in this workflow: `checklist_llm`, `code_vdb_retriever_llm`, `doc_vdb_retriever_llm`, `cve_agent_executor_llm`, `summarize_llm`, `justify_llm` - Each `nim` model is configured with the following attributes defined in the NeMo Agent toolkit's [NimModelConfig](https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/llm/nim_llm/index.html#nat.llm.nim_llm.NIMModelConfig). Use [OpenAIModelConfig](https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/llm/openai_llm/index.html#nat.llm.openai_llm.OpenAIModelConfig) for `openai` models. - `base_url`: Optional attribute to override `https://integrate.api.nvidia.com/v1` @@ -589,7 +604,7 @@ The configuration defines how the workflow operates, including functions, LLMs, - `num_retries`: Number of times to retry a method call that fails with a retryable error. - `retry_on_status_codes`: List of HTTP status codes that should trigger a retry. - `retry_on_errors`: List of error messages that should trigger a retry. We recommend including both the error message and error code to accommodate libraries that don’t expose a HTTP status code attribute in raised exceptions. -4. **Embedding models** (`embedders`): The `embedders` section contains the embedding models used by the workflow. Functions can reference embedding models in this section to use. The supported embedding model API types in NeMo Agent toolkit are `nim` and `openai`. +2. **Embedding models** (`embedders`): The `embedders` section contains the embedding models used by the workflow. Functions can reference embedding models in this section to use. The supported embedding model API types in NeMo Agent toolkit are `nim` and `openai`. - The models uses `nim` model, `nvidia/nv-embedqa-e5-v5`. - Each `nim` embedding model is configured with the following attributes defined in the NeMo Agent Toolkit\'s [NimEmbedderModelConfig](https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/embedder/nim_embedder/index.html). Use [OpenAIEmbedderModelConfig](https://docs.nvidia.com/nemo/agent-toolkit/latest/api/nat/embedder/openai_embedder/index.html#nat.embedder.openai_embedder.OpenAIEmbedderModelConfig) for `openai` embedding models. - `base_url`: Optional attribute to override `https://integrate.api.nvidia.com/v1` @@ -601,16 +616,17 @@ The configuration defines how the workflow operates, including functions, LLMs, - `num_retries`: Number of times to retry a method call that fails with a retryable error. - `retry_on_status_codes`: List of HTTP status codes that should trigger a retry. - `retry_on_errors`: List of error messages that should trigger a retry. We recommend including both the error message and error code to accommodate libraries that don’t expose a HTTP status code attribute in raised exceptions. -5. **Workflow** (`workflow`): The `workflow` section ties the previous sections together by defining the tools and LLM models to use in the workflow. +3. **Workflow** (`workflow`): The `workflow` section ties the previous sections together by defining the tools and LLM models to use in the workflow. - `_type`: This is set to `cve_agent` indicating to NeMo Agent toolkit to use the function defined in [register.py](src/vuln_analysis/register.py) for the workflow. - `missing_source_action`: Action when source analysis is unavailable in the agent due to: missing source_info, VDB generation failures, or inaccessible repositories. Options are: - `error`: Fail pipeline with validation error - `skip_agent`: Collect intel and check dependencies, but skip agent - `continue_with_warning`: Run full pipeline with warning log (degraded analysis quality). Default is "continue_with_warning" for backwards compatibility. - The remaining configuration items correspond to attributes in [CVEWorkflowConfig](src/vuln_analysis/register.py) to specify the registered tools to use in the workflow. -6. **Evaluations and Profiling** (`eval`): The `eval` section contains the evaluation settings for the workflow. Refer to [Evaluating NVIDIA Agent Intelligence Toolkit Workflows](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/evaluate.html) for more information about NeMo Agent toolkit built-in evaluators as well as the plugin system to add custom evaluators. The CVE workflow uses the `eval` section to configure a profiler that uses the NeMo Agent toolkit evaluation system to collect usage statistics and store them to the local file system. You can find more information about NeMo Agent toolkit profiling and performance monitoring [here](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/profiler.html). +4. **Evaluations and Profiling** (`eval`): The `eval` section contains the evaluation settings for the workflow. Refer to [Evaluating NVIDIA Agent Intelligence Toolkit Workflows](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/evaluate.html) for more information about NeMo Agent toolkit built-in evaluators as well as the plugin system to add custom evaluators. An example evaluation pipeline has been provided in the [Evaluation](#evaluation) section of this README. Additionally, the CVE workflow uses the `eval` section to configure a profiler that uses the NeMo Agent Toolkit evaluation system to collect usage statistics and store them to the local file system. You can find more information about NeMo Agent toolkit profiling and performance monitoring [here](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/profiler.html). - `general.output_dir`: Defines the path to the directory where profiling results will be saved. - `general.dataset`: Defines file path and format of dataset used to run profiling. + - `evaluators`: Custom evaluators defined in `eval/evaluators`. See [Evaluation](#evaluation) for more details on each custom evaluator and their configurations. - `profiler`: The profiler for this workflow is configured with the following options. - `token_uniqueness_forecast`: Compute inter query token uniqueness - `workflow_runtime_forecast`: Compute expected workflow runtime @@ -779,6 +795,7 @@ The workflow can then be updated to use the new function: Additional output options will be added in the future. + ### Adding Test Time Compute (TTC) **Test time compute (TTC)** is a technique that can improve agentic workflow performance by allocating more compute at inference time. There are a variety of available TTC strategies. This blueprint includes an implementation of the **majority voting** TTC strategy using the NeMo Agent Toolkit's [Test Time Compute](https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/test-time-compute.html) module. Majority voting is a simple strategy that executes the workflow multiple times, then takes a majority vote on the output, using the most frequent justification status as the final output status. This improves both the accuracy and the consistency of the workflow predictions. @@ -837,6 +854,205 @@ workflow: You can also implement custom TTC strategies other than majority voting by following the NeMo Agent Toolkit [Test Time Compute reference](https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/test-time-compute.html). +## Evaluation +Evaluation is an essential part of developing agentic workflows. In this section, we use NeMo Agent Toolkit's Evaluation feature to implement an example evaluation pipeline that demonstrates best practices for performing evaluation on the vulnerability analysis workflow. There are two custom evaluators included in this example, which provide an extensible template for developers to add their own custom evaluators that target their desired metrics. + +To run evaluation, run `nat eval` using the `config-eval.yml` configuration. + +``` +nat eval --config_file=configs/config-eval.yml +``` + +This will run the two custom evaluators, which measure accuracy and consistency. Evaluation results can be viewed in `.tmp/evaluators`. + +Visit the [Example Custom Evaluators](#example-custom-evaluators) section to learn more details about these evaluators. + +To extend this evaluation example, developers should also write their own custom evaluators to calculate desired metrics. Read more at the NeMo Agent Toolkit documentation for [custom evaluators](https://docs.nvidia.com/nemo/agent-toolkit/1.2/reference/evaluate.html#adding-custom-evaluators). Additionally, developers should provide their own evaluation datasets, which contain the ground truth data that the pipeline's output is compared against. See [Inputting Evaluation Datasets](#inputting-evaluation-datasets) for details. + +> [!NOTE] +> The custom evaluators and evaluation dataset format shown below are example implementations and not part of the stable API. Teams should use these as a starting point and adapt them for their specific evaluation needs. These examples may be updated in future releases. + +### Inputting Evaluation Datasets +By default, evaluation datasets containing ground truth labels should be put in the `data/eval_datasets/` directory. This file path is configurable via the `eval.general.dataset.filepath` field. + +Each dataset is a JSON file which represents an evaluation test set. The structure of the dataset is as follows: + +``` +{ + "dataset_id": "...", + "dataset_description": "...", + "containers": { + "containerA": { + "container_image": { + "name": "...", + "tag": "...", + "source_info": [...], + "sbom_info": {...} + }, + "ground_truth": [ + { + "vuln_id": "CVE-XXXX-XXXX", + "status": "NOT AFFECTED", + "label": "..." + } + ] + }, + "containerB": { + "container_image": {...}, + "ground_truth": [...] + } + } +} +``` + +Evaluation datasets can contain ground truth data across multiple containers. Each entry in the `containers` field consists of input info for a container, along with a list of CVEs and their corresponding ground truth labels. An example test set `eval_dataset.json` is provided, which includes two containers (`morpheus:23.11-runtime` and `morpheus:24.03-runtime`) which have a few CVEs each. Developers can input their own test sets by following this file structure. + +A custom parsing script (`eval/parse_eval_input.py`) is used to transform test sets into a format ingestible by the workflow. The path to this script is configured in `eval.general.dataset.function`. Advanced developers can choose to modify the test set input structure by creating their own parsing script, and then updating the configuration field with their own script. See the NAT documentation on [Custom Dataset Format](https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/evaluate.html#custom-dataset-format) to learn more about customizing datasets for your evaluation needs. + +You can additionally configure the output directory of evaluation results via the `eval.general.output_dir` field. + +> [!NOTE] +> For large input test sets, developers may encounter `429: Too Many Requests` errors if using NVIDIA-hosted model endpoints. See [Troubleshooting](#429-errors---too-many-requests) for how to mitigate this. + + +### Running Multiple Evaluation Repetitions + +The NeMo Agent Toolkit provides a `--reps` flag, which can be used to conveniently run an evaluation experiment multiple times. This is useful for capturing metrics such as workflow consistency. Note that **caching must be disabled first** (see [Disable Caching](#disable-caching) below) in order to force evaluation repetitions to re-run the workflow (as opposed to relying on cached results). + +Below is an example of how to run an evaluation experiment with 3 repetitions: + +``` +nat eval --config_file=configs/config-eval.yml --reps=3 +``` + +> [!NOTE] +> Running multiple repetitions may take some extra time. Additionally, developers may encounter `429: Too Many Requests` errors if using hosted model endpoints, due to the concurrency introduced when running multiple repetitions. See [Troubleshooting](#429-errors---too-many-requests) for how to mitigate this. + + +#### Disable Caching +LLM and embedder caching must be disabled for multiple evaluation runs. Two options for disabling caching are provided below. + +##### Option 1: Disable Caching Globally +Globally disable all LLM and embedding caches by setting the `NVIDIA_API_BASE` variable to directly access the hosted model endpoint instead of going through NGINX. + +``` +export NVIDIA_API_BASE=https://integrate.api.nvidia.com/v1 +``` + +Re-enable caching by setting the `NVIDIA_API_BASE` variable back to NGINX. +``` +export NVIDIA_API_BASE=http://nginx-cache/nim_llm/v1 +``` + + +##### Option 2: Disable Caching on Specific Components +This option enables developers to disable caching for specific parts of the workflow. This approach is useful for in-depth evaluation experiments; for example, a developer may want to disable caching for just the `justify_llm` in order to test consistency of the justification part of the workflow without re-running everything before that point. + +In `config-eval.yml`, assign the `base_url` field of the target components to `"https://integrate.api.nvidia.com/v1"`. We recommend creating multiple config files to keep track of different evaluation experiments. To re-enable caching, simply assign the `base_url` back to `${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1}`. + +For quick testing, developers can also change configuration in the CLI using the `--override` flag, which uses dot notation. This change will not persist across future runs. Below is an example that disables caching on the `checklist_llm`. + +``` +nat eval --config_file=configs/config-eval.yml --reps=3 --override llms.checklist_llm.base_url https://integrate.api.nvidia.com/v1 +``` + +### Example Custom Evaluators +In this section, the implementation of the two custom evaluators for accuracy and consistency are discussed in detail. + +#### Simple Accuracy Evaluator +For the vulnerability analysis workflow, the most simple evaluation we can begin with is evaluating the accuracy of the `status` and `label` fields which the workflow outputs per CVE. + +Each CVE has a ground truth `status`, as well as a `label` which corresponds to the status. +Valid statuses include: `AFFECTED` or `NOT AFFECTED`. +Valid labels include: `vulnerable` if the CVE status is `AFFECTED`, or one of 10 VEX statuses if the status is `NOT AFFECTED` (see **Justification Assignment** under the [Key Components section](#key-components)). + +For each CVE, if the workflow output matches the ground truth, a score of 1 is assigned. Otherwise, a score of 0 is assigned. The evaluator then takes the mean of all CVEs' scores to produce an accuracy score. + + +##### Configuration Options +The accuracy evaluator is implemented in `eval/evaluators/accuracy.py` and registered in NeMo Agent Toolkit with the unique identifer (`_type`) of `accuracy`. + +The evaluator takes in a target field (`status` or `label`), which can be configured in `config-eval.yml` under `evaluators.[evaluator name].field`. This design follows a common NeMo Agent Toolkit design pattern to promote code reusability. In `config-eval.yml`, a `status_accuracy_evaluator` and a `label_accuracy_evaluator` are set up as two separate evaluators, but under the hood, they share the same `_type` and therefore the same source code. + +The evaluator also takes in a `duplicates_policy` field. This is to account for instances where, if an input set is an aggregation of ground truths from different sources, the set may contain duplicate CVEs with conflicting answers. For example, a CVE may be listed twice in the dataset as both `AFFECTED` and `NOT AFFECTED`, with different label justifications. We provide three policies for handling duplicates: +- `drop_all`: Any CVE with conflicting ground truth will be omitted and not contribute towards evaluation results. +- `keep_first`: Use the first encountered instance of a CVE as the truth, disregard all other instances. +- `keep_all`: Any answer listed in the ground truth for a CVE is considered valid (eg. if the ground truth has two listings of a CVE, with one being `AFFECTED` and the other being `NOT AFFECTED`, either answer given by the pipeline will be considered correct). + + +##### Output Format +The final scores are written to the files `status_accuracy_output.json` and `label_accuracy_output.json`. The output will include an `average_score` overview field, and a an `eval_output_items` field which gives a more detailed breakdown. + +###### average_score Field +The `average_score` field contains a dict with metrics corresponding to `pandas.Dataframe.describe` output. The goal of this field is to aggregate the accuracy score across all runs, e.g., when using the `--reps` option. If only one run is performed, `count` and `mean` would be the only meaningful metrics; for example, if the run yielded accuracy of 0.8, the `average_score` field would look like this: +```json +"average_score": { + "count": 1, + "mean": 0.8, + "std": null, + "min": 0.8, + "25%": 0.8, + "50%": 0.8, + "75%": 0.8, + "max": 0.8 +} +``` +Meanwhile, if there were multiple runs, the `average_score` field would capture the accuracy distribution across runs. For example, with 5 runs, the output might look like this: +```json +"average_score": { + "count": 5, + "mean": 0.74, + "std": 0.07, + "min": 0.65, + "25%": 0.7, + "50%": 0.75, + "75%": 0.8, + "max": 0.8 +} +``` +A script is provided in `eval/visualizations/box_and_whisker_plot.py` to visualize this distribution as a box and whisker plot. It is especially helpful for comparing the plots of multiple experiments side-by-side, or comparing `status` accuracy against `label` accuracy. It can be run like so: + +```bash +python3 src/vuln_analysis/eval/visualizations/box_and_whisker_plot.py \ + .tmp/evaluators/accuracy-output-1.json \ + .tmp/evaluators/accuracy-output-2.json \ + --title "Accuracy: Experiment 1 vs. Experiment 2" \ + --save src/vuln_analysis/eval/visualizations/my-plot.png \ + --labels my-experiment1 my-experiment2 +``` +Users can list any number of output files, as well as configure chart title, save location, and X-axis labels. + +###### eval_output_items Field +The `eval_output_items` field gives a more detailed output of the experiment. It lists the runs, giving each run a unique `id`, a `score` (the accuracy score for that run), and a `reasoning` field. + +This last field breaks the run down by container, and lists the specific CVEs that were run in each container (`question`), the ground truth answers `[target field]_answer`, and the answers generated by the workflow (`generated_[target field]_answer`). + + + +#### Consistency Evaluator +This evaluator measures the consistency of results across repeated runs. It can be configured to measure consistency of the `status` or `label` fields. + +Consistency is defined as: `∑p²` per CVE, where *p* is the probability of each unique status or label. For example, if the status for a CVE across 3 runs are `(AFFECTED, NOT_AFFECTED, NOT_AFFECTED)`, then the probabilities become `p(AFFECTED) = 1/3`, `p(NOT_AFFECTED) = 2/3`. Therefore, the consistency score is `(1/3)² + (2/3)² = 0.56`. A score of 1.0 means perfect consistency (all runs yielded the exact same values). The evaluator outputs a consistency score for each CVE. + +> [!NOTE] +> This evaluator should only be used if evaluation is being run with multiple repetitions. An error will occur if this criteria is not met. To enable this evaluator, uncomment the last section in `config-eval.yml`. + + +##### Configuration Options +The accuracy evaluator is implemented in `eval/evaluators/consistency.py` and registered in NeMo Agent Toolkit with the unique identifer (`_type`) of `consistency`. + +The evaluator takes in a configurable target field (`status` or `label`), following a similar design pattern as the [Simple Accuracy Evaluator](#configuration-options) + +##### Output Format +The final scores are written to the files `status_consistency_output.json` and `label_consistency_output.json`. + +The output will include an `average_score` field which gives the average consistency score across all containers, and an `eval_output_items` field which lists the input containers. + +For each container in the `eval_output_items` field, there is an unique `id` and a consistency `score`, which is the mean of the consistency scores of each CVE within that container. There is also a `reasoning` field that displays the following details: + - `status_consistencies`: consistency score for a specific CVE + - `total_cves`: the total number of CVEs in the container + - `num_reps`: number of evaluation repetitions that were run + ## Troubleshooting Several common issues can arise when running the workflow. Here are some common issues and their solutions. @@ -936,10 +1152,13 @@ Error requesting [1/10]: (Retry 0.1 sec) https://services.nvd.nist.gov/rest/json ##### 429 Errors - Too Many Requests -429 errors can occur when your requests exceed the rate limit for the model. Try setting the `cve_agent_executor.max_concurrency` in the [config.yml](src/vuln_analysis/configs/config.yml) to a low value such as 5 to reduce the rate of requests. -``` -Exception: [429] Too Many Requests -``` +`Exception: [429] Too Many Requests` can occur when using NVIDIA-hosted model endpoints, if your requests exceed the rate limit. + +These errors most commonly occur during the `cve_agent_executor` part of the workflow, which has high default concurrency. Try setting the `cve_agent_executor.max_concurrency` in the [config.yml](src/vuln_analysis/configs/config.yml) to a low value such as 5 to reduce the rate of requests. + +With a large number of input CVEs, you may also run into `429` errors from the concurrency of other parts of the workflow. In this case, developers should consider switching to [self-hosting](#using-self-hosted-nims) to accommodate larger inputs for production use cases. + +Additionally, when running evaluation, `nat eval` parallelizes evaluation input items and evaluation repetitions. Therefore, evaluation pipelines are especially prone to triggering 429 errors. You can set the `eval.general.max_concurrency` to a low value such as 1 to reduce parallelization and the rate of requests. However, note that this will reduce throughput. If your evaluation test set is large, or you are running many repetitions, we also recommend switching to [self-hosting](#using-self-hosted-nims) for faster evaluation. #### Authentication errors @@ -976,7 +1195,7 @@ We recommend that teams looking to test or optimize their CVE analysis system cu - [x] Upgrade to [NAT v1.2](https://docs.nvidia.com/nemo/agent-toolkit/1.2/index.html) - [x] Configurable NIM error handling - [x] Configurable missing source info and VDB handling -- [ ] [NAT Evaluation](https://docs.nvidia.com/nemo/agent-toolkit/1.2/workflows/evaluate.html) integration +- [x] [NAT Evaluation](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/evaluate.html) integration - [ ] [NAT W&B Weave](https://docs.nvidia.com/nemo/agent-toolkit/1.2/workflows/observe/observe-workflow-with-weave.html) integration - [x] [NAT Test Time Compute](https://docs.nvidia.com/nemo/agent-toolkit/latest/reference/test-time-compute.html) integration - [ ] Upgrade to NAT v1.3 diff --git a/src/vuln_analysis/configs/config-eval.yml b/src/vuln_analysis/configs/config-eval.yml new file mode 100644 index 000000000..f43edae2c --- /dev/null +++ b/src/vuln_analysis/configs/config-eval.yml @@ -0,0 +1,199 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +general: + use_uvloop: true + +functions: + cve_generate_vdbs: + _type: cve_generate_vdbs + agent_name: cve_agent_executor # Used to determine which tools are enabled + embedder_name: nim_embedder + base_git_dir: .cache/am_cache/git + base_vdb_dir: .cache/am_cache/vdb + base_code_index_dir: .cache/am_cache/code_index + cve_fetch_intel: + _type: cve_fetch_intel + cve_process_sbom: + _type: cve_process_sbom + cve_check_vuln_deps : + _type: cve_check_vuln_deps + skip: false + cve_checklist: + _type: cve_checklist + llm_name: checklist_llm + Container Image Code QA System: + _type: local_vdb_retriever + embedder_name: nim_embedder + llm_name: code_vdb_retriever_llm + vdb_type: code + return_source_documents: false + Container Image Developer Guide QA System: + _type: local_vdb_retriever + embedder_name: nim_embedder + llm_name: doc_vdb_retriever_llm + vdb_type: doc + return_source_documents: false + Lexical Search Container Image Code QA System: + _type: lexical_code_search + top_k: 5 + Internet Search: + _type: serp_wrapper + max_retries: 5 + cve_agent_executor: + _type: cve_agent_executor + llm_name: cve_agent_executor_llm + tool_names: + - Container Image Code QA System + - Container Image Developer Guide QA System + # - Lexical Search Container Image Code QA System # Uncomment to enable lexical search + - Internet Search + max_concurrency: null + max_iterations: 10 + prompt_examples: false + replace_exceptions: true + replace_exceptions_value: "I do not have a definitive answer for this checklist item." + return_intermediate_steps: false + verbose: false + cve_summarize: + _type: cve_summarize + llm_name: summarize_llm + cve_justify: + _type: cve_justify + llm_name: justify_llm + cve_file_output: + _type: cve_file_output + file_path: .tmp/output.json + markdown_dir: .tmp/vulnerability_markdown_reports + overwrite: true + +llms: + checklist_llm: + _type: nim + base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1} + model_name: ${CHECKLIST_MODEL_NAME:-meta/llama-3.1-70b-instruct} + temperature: 0.0 + max_tokens: 2000 + top_p: 0.01 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + code_vdb_retriever_llm: + _type: nim + base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1} + model_name: ${CODE_VDB_RETRIEVER_MODEL_NAME:-meta/llama-3.1-70b-instruct} + temperature: 0.0 + max_tokens: 2000 + top_p: 0.01 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + doc_vdb_retriever_llm: + _type: nim + base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1} + model_name: ${DOC_VDB_RETRIEVER_MODEL_NAME:-meta/llama-3.1-70b-instruct} + temperature: 0.0 + max_tokens: 2000 + top_p: 0.01 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + cve_agent_executor_llm: + _type: nim + base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1} + model_name: ${CVE_AGENT_EXECUTOR_MODEL_NAME:-meta/llama-3.1-70b-instruct} + temperature: 0.0 + max_tokens: 2000 + top_p: 0.01 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + summarize_llm: + _type: nim + base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1} + model_name: ${SUMMARIZE_MODEL_NAME:-meta/llama-3.1-70b-instruct} + temperature: 0.0 + max_tokens: 1024 + top_p: 0.01 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + justify_llm: + _type: nim + base_url: ${NVIDIA_API_BASE:-https://integrate.api.nvidia.com/v1} + model_name: ${JUSTIFY_MODEL_NAME:-meta/llama-3.1-70b-instruct} + temperature: 0.0 + max_tokens: 1024 + top_p: 0.01 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + +embedders: + nim_embedder: + _type: nim + base_url: ${NIM_EMBED_BASE_URL:-https://integrate.api.nvidia.com/v1} + model_name: ${EMBEDDER_MODEL_NAME:-nvidia/nv-embedqa-e5-v5} + truncate: END + max_batch_size: 128 + do_auto_retry: True + max_retries: 5 + retry_on_status_codes: [429, 500, 502, 503, 504] + retry_on_errors: ["Too Many Requests", "Internal Server Error", "Bad Gateway", "Service Unavailable", "Gateway Time-out", "Gateway Timeout"] + +workflow: + _type: cve_agent + cve_generate_vdbs_name: cve_generate_vdbs + cve_fetch_intel_name: cve_fetch_intel + cve_process_sbom_name: cve_process_sbom + cve_check_vuln_deps_name: cve_check_vuln_deps + cve_checklist_name: cve_checklist + cve_agent_executor_name: cve_agent_executor + cve_summarize_name: cve_summarize + cve_justify_name: cve_justify + cve_output_config_name: cve_file_output + +eval: + general: + output_dir: ./.tmp/evaluators + dataset: + _type: custom + file_path: data/eval_datasets/eval_dataset.json + function: vuln_analysis.eval.parse_eval_input.parse_input + + evaluators: + status_accuracy: + _type: accuracy + field: status + duplicates_policy: drop_all # Options: drop_all, keep_first, keep_all + label_accuracy: + _type: accuracy + field: label + duplicates_policy: drop_all + + # Uncomment to enable consistency evaluator + # NOTE: Only use this evaluator if evaluation is being run with multiple repetitions + # status_consistency: + # _type: consistency + # field: status + # label_consistency: + # _type: consistency + # field: label diff --git a/src/vuln_analysis/configs/config-tracing.yml b/src/vuln_analysis/configs/config-tracing.yml index 1f9be18f8..ade78f678 100644 --- a/src/vuln_analysis/configs/config-tracing.yml +++ b/src/vuln_analysis/configs/config-tracing.yml @@ -185,10 +185,10 @@ workflow: eval: general: - output_dir: ./.tmp/eval/cve_agent + output_dir: ./.tmp/profiler dataset: _type: json - file_path: data/eval_datasets/eval_dataset.json + file_path: data/profiler_datasets/profiler_dataset.json profiler: token_uniqueness_forecast: true diff --git a/src/vuln_analysis/configs/config.yml b/src/vuln_analysis/configs/config.yml index c9a65665d..692bdbd7e 100644 --- a/src/vuln_analysis/configs/config.yml +++ b/src/vuln_analysis/configs/config.yml @@ -175,10 +175,10 @@ workflow: eval: general: - output_dir: ./.tmp/eval/cve_agent + output_dir: ./.tmp/profiler dataset: _type: json - file_path: data/eval_datasets/eval_dataset.json + file_path: data/profiler_datasets/profiler_dataset.json profiler: token_uniqueness_forecast: true diff --git a/src/vuln_analysis/data/eval_datasets/eval_dataset.json b/src/vuln_analysis/data/eval_datasets/eval_dataset.json index 1f4597f4e..a0107930d 100644 --- a/src/vuln_analysis/data/eval_datasets/eval_dataset.json +++ b/src/vuln_analysis/data/eval_datasets/eval_dataset.json @@ -1,7 +1,103 @@ -[ - { - "id": 1, - "question": "{\"image\":{\"name\":\"nvcr.io\/nvidia\/morpheus\/morpheus\",\"tag\":\"23.11-runtime\",\"source_info\":[{\"type\":\"code\",\"git_repo\":\"https:\/\/github.com\/nv-morpheus\/Morpheus.git\",\"ref\":\"v23.11.01\",\"include\":[\"**\/*.cpp\",\"**\/*.cu\",\"**\/*.cuh\",\"**\/*.h\",\"**\/*.hpp\",\"**\/*.ipynb\",\"**\/*.py\",\"**\/*Dockerfile\"],\"exclude\":[\"tests\/**\/*\"]},{\"type\":\"doc\",\"git_repo\":\"https:\/\/github.com\/nv-morpheus\/Morpheus.git\",\"ref\":\"v23.11.01\",\"include\":[\"**\/*.md\",\"docs\/**\/*.rst\"]}],\"sbom_info\":{\"_type\":\"file\",\"file_path\":\"data\/sboms\/nvcr.io\/nvidia\/morpheus\/morpheus:v23.11.01-runtime.sbom\"}},\"scan\":{\"vulns\":[{\"vuln_id\":\"GHSA-3f63-hfp8-52jq\"},{\"vuln_id\":\"CVE-2023-50782\"}]}}", - "answer": "N/A" +{ + "dataset_id": "example-eval-dataset", + "dataset_description": "Example eval dataset with two containers", + "containers": { + "morpheus:23.11-runtime": { + "container_image": { + "name": "nvcr.io/nvidia/morpheus/morpheus", + "tag": "23.11-runtime", + "source_info": [ + { + "type": "code", + "git_repo": "https://github.com/nv-morpheus/Morpheus.git", + "ref": "branch-23.11", + "include": [ + "**/*.cpp", + "**/*.cu", + "**/*.cuh", + "**/*.h", + "**/*.hpp", + "**/*.ipynb", + "**/*.py", + "**/*Dockerfile" + ], + "exclude": [ + "tests/**/*" + ] + }, + { + "type": "doc", + "git_repo": "https://github.com/nv-morpheus/Morpheus.git", + "ref": "branch-23.11", + "include": [ + "**/*.md", + "docs/**/*.rst" + ] + } + ], + "sbom_info": { + "_type": "file", + "file_path": "data/sboms/nvcr.io/nvidia/morpheus/morpheus:v23.11.01-runtime.sbom" + } + }, + "ground_truth": [ + { + "vuln_id": "GHSA-3f63-hfp8-52jq", + "status": "NOT AFFECTED", + "label": "code_not_reachable" + }, + { + "vuln_id": "CVE-2023-5388", + "status": "NOT AFFECTED", + "label": "false_positive" + } + ] + }, + "morpheus:24.03-runtime": { + "container_image": { + "name": "nvcr.io/nvidia/morpheus/morpheus", + "tag": "v24.03.02-runtime", + "source_info": [ + { + "type": "code", + "git_repo": "https://github.com/nv-morpheus/Morpheus.git", + "ref": "v24.03.02", + "include": [ + "**/*.cpp", + "**/*.cu", + "**/*.cuh", + "**/*.h", + "**/*.hpp", + "**/*.ipynb", + "**/*.py", + "**/*Dockerfile" + ], + "exclude": [ + "tests/**/*" + ] + }, + { + "type": "doc", + "git_repo": "https://github.com/nv-morpheus/Morpheus.git", + "ref": "v24.03.02", + "include": [ + "**/*.md", + "docs/**/*.rst" + ] + } + ], + "sbom_info": { + "_type": "file", + "file_path": "data/sboms/nvcr.io/nvidia/morpheus/morpheus:v24.03.02-runtime.sbom" + } + }, + "ground_truth": [ + { + "vuln_id": "CVE-2023-5388", + "status": "NOT AFFECTED", + "label": "false_positive" + } + ] + } } -] +} diff --git a/src/vuln_analysis/data/profiler_datasets/profiler_dataset.json b/src/vuln_analysis/data/profiler_datasets/profiler_dataset.json new file mode 100644 index 000000000..1f4597f4e --- /dev/null +++ b/src/vuln_analysis/data/profiler_datasets/profiler_dataset.json @@ -0,0 +1,7 @@ +[ + { + "id": 1, + "question": "{\"image\":{\"name\":\"nvcr.io\/nvidia\/morpheus\/morpheus\",\"tag\":\"23.11-runtime\",\"source_info\":[{\"type\":\"code\",\"git_repo\":\"https:\/\/github.com\/nv-morpheus\/Morpheus.git\",\"ref\":\"v23.11.01\",\"include\":[\"**\/*.cpp\",\"**\/*.cu\",\"**\/*.cuh\",\"**\/*.h\",\"**\/*.hpp\",\"**\/*.ipynb\",\"**\/*.py\",\"**\/*Dockerfile\"],\"exclude\":[\"tests\/**\/*\"]},{\"type\":\"doc\",\"git_repo\":\"https:\/\/github.com\/nv-morpheus\/Morpheus.git\",\"ref\":\"v23.11.01\",\"include\":[\"**\/*.md\",\"docs\/**\/*.rst\"]}],\"sbom_info\":{\"_type\":\"file\",\"file_path\":\"data\/sboms\/nvcr.io\/nvidia\/morpheus\/morpheus:v23.11.01-runtime.sbom\"}},\"scan\":{\"vulns\":[{\"vuln_id\":\"GHSA-3f63-hfp8-52jq\"},{\"vuln_id\":\"CVE-2023-50782\"}]}}", + "answer": "N/A" + } +] diff --git a/src/vuln_analysis/data_models/eval_input.py b/src/vuln_analysis/data_models/eval_input.py new file mode 100644 index 000000000..9954a5f70 --- /dev/null +++ b/src/vuln_analysis/data_models/eval_input.py @@ -0,0 +1,89 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Literal + +from pydantic import BaseModel +from pydantic import Field +from pydantic import field_validator +from pydantic import model_validator + +from vuln_analysis.data_models.input import ImageInfoInput +from vuln_analysis.utils.justification_parser import JustificationParser +from vuln_analysis.utils.string_utils import is_valid_cve_id +from vuln_analysis.utils.string_utils import is_valid_ghsa_id + +# Use label/status mappings from JustificationParser +ALLOWED_LABELS: tuple[str, ...] = tuple(JustificationParser.JUSTIFICATION_TO_AFFECTED_STATUS_MAP.keys()) + +# Map justification status (TRUE/FALSE/UNKNOWN) to eval dataset statuses (AFFECTED/NOT AFFECTED/UNKNOWN) +_JUST_STATUS_TO_EVAL_STATUS: dict[str, str] = { + "TRUE": "AFFECTED", + "FALSE": "NOT AFFECTED", + "UNKNOWN": "UNKNOWN", +} + +ALLOWED_STATUSES: tuple[str, ...] = tuple(_JUST_STATUS_TO_EVAL_STATUS.values()) + + +class GroundTruthItem(BaseModel): + vuln_id: str + status: str + label: str + + @field_validator("vuln_id") + @classmethod + def validate_vuln_id(cls, v: str) -> str: + if not (is_valid_cve_id(v) or is_valid_ghsa_id(v)): + raise ValueError(f"{v} is not a valid CVE ID or GHSA ID.") + return v + + @field_validator("label") + @classmethod + def validate_label(cls, v: str) -> str: + if v not in ALLOWED_LABELS: + raise ValueError(f"{v} is not a valid justification label. Allowed: {sorted(ALLOWED_LABELS)}") + return v + + @field_validator("status") + @classmethod + def validate_status(cls, v: str) -> str: + if v not in ALLOWED_STATUSES: + raise ValueError(f"{v} is not a valid status. Allowed: {sorted(ALLOWED_STATUSES)}") + return v + + @model_validator(mode="after") + def validate_label_status_consistency(self): + # Expected status from label via mapping from JustificationParser + expected_status = JustificationParser.JUSTIFICATION_TO_AFFECTED_STATUS_MAP[self.label] + expected_status_transformed = _JUST_STATUS_TO_EVAL_STATUS[expected_status] + + if self.status != expected_status_transformed: + raise ValueError( + f"Label/status mismatch: label '{self.label}' implies '{expected_status_transformed}' but got '{self.status}'." + ) + + return self + + +class AgentMorpheusEvalInputItem(BaseModel): + container_image: ImageInfoInput + ground_truth: list[GroundTruthItem] + + +class AgentMorpheusEvalDataset(BaseModel): + dataset_id: str + dataset_description: str | None = None + containers: dict[str, AgentMorpheusEvalInputItem] diff --git a/src/vuln_analysis/eval/evaluators/accuracy.py b/src/vuln_analysis/eval/evaluators/accuracy.py new file mode 100644 index 000000000..b54d902d1 --- /dev/null +++ b/src/vuln_analysis/eval/evaluators/accuracy.py @@ -0,0 +1,240 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import json +import typing +from typing import Literal + +import pandas as pd +from nat.builder.builder import EvalBuilder +from nat.builder.evaluator import EvaluatorInfo +from nat.cli.register_workflow import register_evaluator +from nat.data_models.evaluator import EvaluatorBaseConfig +from nat.eval.evaluator.evaluator_model import EvalInput +from nat.eval.evaluator.evaluator_model import EvalInputItem +from nat.eval.evaluator.evaluator_model import EvalOutput +from nat.eval.evaluator.evaluator_model import EvalOutputItem +from nat.eval.utils.tqdm_position_registry import TqdmPositionRegistry +from pydantic import Field +from tqdm import tqdm + +_STATUS_MAP = {"TRUE": "AFFECTED", "FALSE": "NOT AFFECTED", "UNKNOWN": "UNKNOWN"} + + +class AccuracyEvaluatorConfig(EvaluatorBaseConfig, name="accuracy"): + """Configuration for unified accuracy evaluator""" + field: typing.Literal["status", "label"] = Field(default="status", description="The field to evaluate accuracy on.") + duplicates_policy: typing.Literal["drop_all", "keep_first", "keep_all"] = Field( + default="keep_first", description="Policy for handling duplicate CVEs in ground truth.") + + +@register_evaluator(config_type=AccuracyEvaluatorConfig) +async def accuracy_evaluator(config: AccuracyEvaluatorConfig, builder: EvalBuilder): + """Register unified accuracy evaluator""" + evaluator = AccuracyEvaluator(builder.get_max_concurrency(), + field=config.field, + duplicates_policy=config.duplicates_policy) + yield EvaluatorInfo(config=config, + evaluate_fn=evaluator.evaluate, + description=f"{config.field.title()} Accuracy Evaluator") + + +class AccuracyEvaluator: + '''Configurable accuracy evaluator for either the status or label field.''' + + def __init__(self, + max_concurrency: int, + field: str = "status", + duplicates_policy: Literal["drop_all", "keep_first", "keep_all"] = "keep_first"): + self.max_concurrency = max_concurrency + self.semaphore = asyncio.Semaphore(self.max_concurrency) + # field must be either "status" or "label" + self.field = field + # duplicates_policy: "drop_all" | "keep_first" | "keep_all" + self.duplicates_policy = duplicates_policy + + async def evaluate(self, eval_input: EvalInput) -> EvalOutput: + + def _preprocess_ground_truth(ground_truth_list, field: str, policy: str): + """Build a mapping of vuln_id -> acceptable answer set based on policy. + + - drop_all: only vuln_ids with a single unique answer are kept + - keep_first: take the first encountered answer + - keep_all: any answer in ground truth is considered acceptable + """ + # Collect ground truth answers per vuln_id + answers_per_id = {} + for gt in ground_truth_list: + vuln_id = gt.get("vuln_id") + if not vuln_id: + continue + ans = gt.get(field) + if ans is None: + continue + answers_per_id.setdefault(vuln_id, []) + answers_per_id[vuln_id].append(ans) + + acceptable_map = {} + + if policy == "drop_all": + for vuln_id, answers in answers_per_id.items(): + distinct = list(dict.fromkeys(answers)) # preserve order, unique + if len(distinct) == 1: + acceptable_map[vuln_id] = {distinct[0]} + return acceptable_map + + if policy == "keep_first": + # Keep first encountered answer per vuln_id + for vuln_id, answers in answers_per_id.items(): + first_answer = answers[0] + acceptable_map[vuln_id] = {first_answer} + return acceptable_map + + # keep_all: any of the distinct answers are acceptable + for vuln_id, answers in answers_per_id.items(): + acceptable_map[vuln_id] = set(answers) + return acceptable_map + + def _extract_rep_number(item_id: str) -> str: + """Extract rep number from item id (e.g., 'container1_rep1' -> 'rep1')""" + if "_rep" in item_id: + return "rep" + item_id.split("_rep")[-1] + return "rep1" # Default to rep1 if no rep suffix + + def _group_by_rep(items: list[EvalInputItem]) -> dict[str, list[EvalInputItem]]: + """Group items by rep number""" + groups = {} + for item in items: + rep_id = _extract_rep_number(item.id) + groups.setdefault(rep_id, []) + groups[rep_id].append(item) + return groups + + async def process_rep_group(items: list[EvalInputItem]): + """Process all containers in a single rep together""" + # Aggregate results across all containers in this rep + all_per_item_accuracies = [] + acceptable_answers_map_combined = {} + test_set_file = "unknown" + container_data = {} + + if self.field == "status": + answer_key = "status_answer" + gen_key = "generated_status_answer" + else: + answer_key = "label_answer" + gen_key = "generated_label_answer" + + for item in items: + # Extract container name from item id (e.g., "container1_rep1" -> "container1") + container_name = item.id.split('_rep')[0] if '_rep' in item.id else item.id + + # Build acceptable answers map from the original ground truth list + original_ground_truth = item.full_dataset_entry.get("ground_truth", []) + acceptable_answers_map = _preprocess_ground_truth(original_ground_truth, self.field, self.duplicates_policy) + acceptable_answers_map_combined.update(acceptable_answers_map) + + pipeline_result = json.loads(item.output_obj)["output"] + test_set_file = item.full_dataset_entry.get("test_set_file", "unknown") + + # Track data per container + container_questions = [] + container_generated_answers = [] + + for result in pipeline_result: + vuln_id = result["vuln_id"] + + if self.field == "status": + justification_status = result.get("justification", {}).get("status", "MISSING") + pred = _STATUS_MAP.get(justification_status, "MISSING") + acceptable_set = acceptable_answers_map.get(vuln_id) + else: # label + justification_label = result.get("justification", {}).get("label", "MISSING") + pred = justification_label + acceptable_set = acceptable_answers_map.get(vuln_id) + + # If this vuln_id was filtered out by the duplicates policy, skip + if acceptable_set is None: + continue + + container_questions.append(vuln_id) + container_generated_answers.append(pred) + all_per_item_accuracies.append(float(pred in acceptable_set)) + + # Output the data per-container + container_data[container_name] = { + "question": container_questions, + answer_key: [sorted(list(acceptable_answers_map_combined.get(q, set()))) for q in container_questions], + gen_key: container_generated_answers, + } + + total_correct = sum(all_per_item_accuracies) + total_items = len(all_per_item_accuracies) + avg = round(total_correct / total_items, 2) if total_items > 0 else 0.0 + reasoning = {**container_data, "test_set_file": test_set_file} + + return avg, reasoning, total_items, total_correct + + async def wrapped_process_rep(rep_id: str, items: list[EvalInputItem]) -> tuple[str, float, dict, int, int]: + async with self.semaphore: + result = await process_rep_group(items) + pbar.update(len(items)) + return (rep_id, *result) + + # Group items by rep number + rep_groups = _group_by_rep(eval_input.eval_input_items) + + try: + tqdm_position = TqdmPositionRegistry.claim() + pbar = tqdm(total=len(eval_input.eval_input_items), + desc=f"Evaluating {self.field.title()} Accuracy", + position=tqdm_position) + results = await asyncio.gather(*[wrapped_process_rep(rep_id, items) for rep_id, items in rep_groups.items()]) + finally: + pbar.close() + TqdmPositionRegistry.release(tqdm_position) + + # Unpack results: rep_id, avg, reasoning, total_items, total_correct + if results: + rep_ids, all_averages, all_reasonings, _, _ = zip(*results) + else: + rep_ids, all_averages, all_reasonings = [], [], [] + + # Output dict format with .describe() summary + scores_sequential = list(all_averages) + if scores_sequential: + series = pd.Series(scores_sequential) + desc = series.describe() # count, mean, std, min, 25%, 50%, 75%, max + # Round values + final_output = {k: (int(v) if k == "count" else round(float(v), 2)) for k, v in desc.items()} + else: + final_output = { + "count": 0, + "mean": 0.0, + "std": 0.0, + "min": 0.0, + "25%": 0.0, + "50%": 0.0, + "75%": 0.0, + "max": 0.0, + } + + eval_output_items = [ + EvalOutputItem(id=rep_id, score=score, reasoning=reasoning) + for rep_id, score, reasoning in zip(rep_ids, all_averages, all_reasonings) + ] + + return EvalOutput(average_score=final_output, eval_output_items=eval_output_items) diff --git a/src/vuln_analysis/eval/evaluators/consistency.py b/src/vuln_analysis/eval/evaluators/consistency.py new file mode 100644 index 000000000..42520c2fe --- /dev/null +++ b/src/vuln_analysis/eval/evaluators/consistency.py @@ -0,0 +1,151 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import json +import typing +from collections import Counter + +from nat.builder.builder import EvalBuilder +from nat.builder.evaluator import EvaluatorInfo +from nat.cli.register_workflow import register_evaluator +from nat.data_models.evaluator import EvaluatorBaseConfig +from nat.eval.evaluator.evaluator_model import EvalInput +from nat.eval.evaluator.evaluator_model import EvalOutput +from nat.eval.evaluator.evaluator_model import EvalOutputItem +from nat.eval.utils.tqdm_position_registry import TqdmPositionRegistry +from pydantic import Field +from tqdm import tqdm + + +class ConsistencyEvaluatorConfig(EvaluatorBaseConfig, name="consistency"): + """Configuration for consistency evaluator""" + field: typing.Literal["label", "status"] = Field(default="label", + description="The field to evaluate consistency on.") + + +@register_evaluator(config_type=ConsistencyEvaluatorConfig) +async def consistency_evaluator(config: ConsistencyEvaluatorConfig, builder: EvalBuilder): + """Register consistency evaluator""" + evaluator = ConsistencyEvaluator(builder.get_max_concurrency(), field=config.field) + yield EvaluatorInfo(config=config, + evaluate_fn=evaluator.evaluate, + description=f"{config.field.title()} Consistency Evaluator") + + +class ConsistencyEvaluator: + '''Evaluator class for measuring consistency of the specified field via the agreement probability metric ∑p², where p is the probability of each unique value''' + + def __init__(self, max_concurrency: int, field: str = "label"): + self.max_concurrency = max_concurrency + self.semaphore = asyncio.Semaphore(self.max_concurrency) + self.field = field # "label" or "status" + + async def evaluate(self, eval_input: EvalInput) -> EvalOutput: + '''Evaluate consistency across multiple runs per CVE, broken down by container''' + + async def extract_run_data(item): + """Extract data from a single evaluation run""" + async with self.semaphore: + evaluation_run = json.loads(item.output_obj)['output'] + pbar.update(1) + return evaluation_run, item + + try: + tqdm_position = TqdmPositionRegistry.claim() + pbar = tqdm(total=len(eval_input.eval_input_items), desc="Evaluating Consistency", position=tqdm_position) + + # Extract data from all evaluation runs in parallel + all_runs_with_items = await asyncio.gather( + *[extract_run_data(item) for item in eval_input.eval_input_items]) + finally: + pbar.close() + TqdmPositionRegistry.release(tqdm_position) + + # Group runs by container (extract container name from item id) + container_runs = {} + for evaluation_run, item in all_runs_with_items: + container_name = item.id.split('_rep')[0] + + if container_name not in container_runs: + container_runs[container_name] = [] + container_runs[container_name].append(evaluation_run) + + # Validate that each container has multiple reps + for container_name, runs in container_runs.items(): + if len(runs) <= 1: + raise ValueError( + f"Consistency evaluator requires multiple runs per container. Container '{container_name}' has only {len(runs)} run(s). Consistency cannot be measured with a single run per container." + ) + + # Calculate consistency for each container + eval_output_items = [] + all_cve_consistencies = [] # Collect all CVE consistency scores across all containers + + for container_name, runs in container_runs.items(): + # Aggregate results across all reps for this container + vuln_responses = {} + for evaluation_run in runs: + for cve_result in evaluation_run: # Group by CVE + vuln_id = cve_result['vuln_id'] + # Measure consistency on the specified field + field_value = cve_result.get('justification', {}).get(self.field, 'MISSING') + + if vuln_id not in vuln_responses: + vuln_responses[vuln_id] = [] + + vuln_responses[vuln_id].append(field_value) + + # Calculate consistency score per CVE for this container + consistencies_dict = {} # Stores consistency scores mapped to CVE + consistencies = [] # Stores all consistency scores in an array + + for vuln_id, values in vuln_responses.items(): + if len(values) > 1: + value_counts = Counter(values) + n = len(values) + probs = [count / n for count in value_counts.values()] + consistency = sum(p * p for p in probs) + else: + consistency = 1.0 # Perfect consistency if there is only one response + + consistencies_dict[vuln_id] = round(consistency, 2) + consistencies.append(round(consistency, 2)) + + # Add all CVE consistency scores from this container to the global collection + all_cve_consistencies.extend(consistencies) + + # Calculate average consistency for this container + container_avg = sum(consistencies) / len(consistencies) if consistencies else 1.0 + container_score = round(container_avg, 2) + + # Create output with per-CVE scores for this container + output = { + f"{self.field}_consistencies": consistencies_dict, + "total_cves": len(vuln_responses), + "num_reps": len(runs), + } + + # Create EvalOutputItem for this container + eval_output_items.append( + EvalOutputItem(id=f"{self.field.title()} Consistency per CVE - {container_name}", + score=container_score, + reasoning=output)) + + # Calculate overall average across all CVEs from all containers + overall_avg = sum(all_cve_consistencies) / len(all_cve_consistencies) if all_cve_consistencies else 1.0 + overall_score = round(overall_avg, 2) + + return EvalOutput(average_score=overall_score, eval_output_items=eval_output_items) diff --git a/src/vuln_analysis/eval/parse_eval_input.py b/src/vuln_analysis/eval/parse_eval_input.py new file mode 100644 index 000000000..baf6478b8 --- /dev/null +++ b/src/vuln_analysis/eval/parse_eval_input.py @@ -0,0 +1,190 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json +from pathlib import Path + +from nat.eval.evaluator.evaluator_model import EvalInput +from nat.eval.evaluator.evaluator_model import EvalInputItem + +from vuln_analysis.data_models.eval_input import AgentMorpheusEvalDataset + + +def parse_input(file_path: Path) -> EvalInput: + """ + Transform a human-readable eval dataset in the eval_datasets/ directory into a series of EvalInputItems + that the NAT evaluation harness accepts. Each EvalInputItem corresponds to a container. + EvalInputItem `id` is the container name, `question` encodes the source code and input CVEs, + and `answer` encodes the CVE ground truth mapping. + + Usage (outside of the full NAT pipeline): + python3 parse_eval_input.py [test_set_file].json --preview -o [your output file].json + Example: + python3 parse_eval_input.py ../data/eval_datasets/eval_dataset.json --preview -o my_eval_string.json + Run this inside the vuln_analysis container, from the eval directory. + + Example output: + [ + { + "id": "morpheus:23.11-runtime", + "question": "{\"image\":{\"name\":\"nvcr.io/nvidia/morpheus/morpheus\",\"tag\":\"23.11-runtime\",\"source_info\":[{\"type\":\"code\",\"git_repo\":\"https://github.com/nv-morpheus/Morpheus.git\",\"ref\":\"branch-23.11\",\"include\":[\"**/*.cpp\",\"**/*.cu\",\"**/*.cuh\",\"**/*.h\",\"**/*.hpp\",\"**/*.ipynb\",\"**/*.py\",\"**/*Dockerfile\"],\"exclude\":[\"tests/**/*\"]},{\"type\":\"doc\",\"git_repo\":\"https://github.com/nv-morpheus/Morpheus.git\",\"ref\":\"branch-23.11\",\"include\":[\"**/*.md\",\"docs/**/*.rst\"]}],\"sbom_info\":{\"_type\":\"file\",\"file_path\":\"data/sboms/nvcr.io/nvidia/morpheus/morpheus:v23.11.01-runtime.sbom\"}},\"scan\":{\"vulns\":[{\"vuln_id\":\"GHSA-3f63-hfp8-52jq\"},{\"vuln_id\":\"CVE-2023-36632\"}]}}", + "answer": "{\"GHSA-3f63-hfp8-52jq\":\"NOT AFFECTED\",\"GHSA-3f63-hfp8-52jq_label\":\"code_not_reachable\",\"CVE-2023-36632\":\"AFFECTED\",\"CVE-2023-36632_label\":\"vulnerable\"}" + }, + { + "id": "morpheus:24.03-runtime", + "question": "{\"image\":{\"name\":\"nvcr.io/nvidia/morpheus/morpheus\",\"tag\":\"v24.03.02-runtime\",\"source_info\":[{\"type\":\"code\",\"git_repo\":\"https://github.com/nv-morpheus/Morpheus.git\",\"ref\":\"v24.03.02\",\"include\":[\"**/*.cpp\",\"**/*.cu\",\"**/*.cuh\",\"**/*.h\",\"**/*.hpp\",\"**/*.ipynb\",\"**/*.py\",\"**/*Dockerfile\"],\"exclude\":[\"tests/**/*\"]},{\"type\":\"doc\",\"git_repo\":\"https://github.com/nv-morpheus/Morpheus.git\",\"ref\":\"v24.03.02\",\"include\":[\"**/*.md\",\"docs/**/*.rst\"]}],\"sbom_info\":{\"_type\":\"file\",\"file_path\":\"data/sboms/nvcr.io/nvidia/morpheus/morpheus:v24.03.02-runtime.sbom\"}},\"scan\":{\"vulns\":[{\"vuln_id\":\"GHSA-3f63-hfp8-52jq\"},{\"vuln_id\":\"CVE-2023-36632\"}]}}", + "answer": "{\"GHSA-3f63-hfp8-52jq\":\"NOT AFFECTED\",\"GHSA-3f63-hfp8-52jq_label\":\"code_not_reachable\",\"CVE-2023-36632\":\"AFFECTED\",\"CVE-2023-36632_label\":\"vulnerable\"}" + } + ] + """ + # Load and validate the test set file using Pydantic models + with open(file_path, 'r', encoding='utf-8') as f: + config_data = json.load(f) + dataset = AgentMorpheusEvalDataset.model_validate(config_data) + + # Extract containers + containers = dataset.containers + + # Get dataset metadata from the file + dataset_id = dataset.dataset_id + dataset_description = dataset.dataset_description + test_set_file = file_path.name # Extract filename from the file path + print(f"Processing dataset: {dataset_id} (file: {test_set_file})") + + # Create evaluation items for each container + eval_items = [] + + for container_id, container_data in containers.items(): + container_image = container_data.container_image.model_dump() + ground_truth = [gt.model_dump() for gt in container_data.ground_truth] + + # Create scan structure from ground truth + scan_vulns = [] + ground_truth_mapping = {} + seen_vuln_ids = set() # Ensures vuln_id uniqueness + + for gt_item in ground_truth: + vuln_id = gt_item.get('vuln_id') + status = gt_item.get('status') + label = gt_item.get('label') + + if not (vuln_id and status): + continue + + if vuln_id in seen_vuln_ids: # skip duplicate vuln_ids + continue + + seen_vuln_ids.add(vuln_id) + + # Add to scan structure + scan_vulns.append({"vuln_id": vuln_id}) + + # Add to ground truth mapping + ground_truth_mapping[vuln_id] = status + + # Add label to ground truth mapping if available + if label: + ground_truth_mapping[f"{vuln_id}_label"] = label + + # Create the question structure (container_image + scan) + question_structure = {"image": container_image, "scan": {"vulns": scan_vulns}} + + # Convert to stringified format compatible with NAT EvalInputItem format + question_string = json.dumps(question_structure, separators=(',', ':')) + answer_string = json.dumps(ground_truth_mapping, separators=(',', ':')) + + # Create evaluation item + eval_item = EvalInputItem(id=container_id, + input_obj=question_string, + expected_output_obj=answer_string, + full_dataset_entry={ + "id": container_id, + "question": question_string, + "answer": answer_string, + "test_set_file": test_set_file, + "dataset_id": dataset_id, + "dataset_description": dataset_description, + "container_id": container_id, + "ground_truth": ground_truth + }) + + eval_items.append(eval_item) + + if not eval_items: + print("No valid evaluation items created") + return EvalInput(eval_input_items=[]) + + print(f"Created {len(eval_items)} evaluation items for dataset '{dataset_id}'") + return EvalInput(eval_input_items=eval_items) + + +# For debugging - saves the EvalInputItem string to a file +def save_as_baseline_string_format(eval_input: EvalInput, output_path: Path) -> None: + baseline_data = [] + + for item in eval_input.eval_input_items: + baseline_item = {"id": item.id, "question": item.input_obj, "answer": item.expected_output_obj} + baseline_data.append(baseline_item) + + with open(output_path, 'w', encoding='utf-8') as f: + json.dump(baseline_data, f, indent=4) + + print(f"Saved baseline string format to: {output_path}") + + +def main(): + import argparse + + parser = argparse.ArgumentParser( + description='Parse consolidated evaluation dataset with multiple containers and test sets', + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + parser.add_argument('input_file', help='Path to eval-dataset.json file') + parser.add_argument('-o', '--output', help='Output path for baseline string format (optional)') + parser.add_argument('--preview', action='store_true', help='Show preview of first converted item') + + args = parser.parse_args() + + try: + input_path = Path(args.input_file) + + # Convert to EvalInput format + eval_input = parse_input(input_path) + + # Save as baseline string format if output is specified + if args.output: + output_path = Path(args.output) + save_as_baseline_string_format(eval_input, output_path) + else: + # Default output path in eval directory + eval_dir = Path(__file__).parent + default_output = eval_dir / "eval_input_string.json" + save_as_baseline_string_format(eval_input, default_output) + + except FileNotFoundError as e: + print(f"Error: Input file not found - {e}") + return 1 + except json.JSONDecodeError as e: + print(f"Error: Invalid JSON in input file - {e}") + return 1 + except Exception as e: + print(f"Unexpected error: {e}") + return 1 + + return 0 + + +if __name__ == "__main__": + exit(main()) diff --git a/src/vuln_analysis/eval/visualizations/box_and_whisker_plot.py b/src/vuln_analysis/eval/visualizations/box_and_whisker_plot.py new file mode 100644 index 000000000..7e1c3befd --- /dev/null +++ b/src/vuln_analysis/eval/visualizations/box_and_whisker_plot.py @@ -0,0 +1,131 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import argparse +import json +import os + +import matplotlib.pyplot as plt +""" +Box-and-whisker plot utility for evaluator outputs. + +Example usage: + + python3 src/vuln_analysis/eval/visualizations/box_and_whisker_plot.py \ + .tmp/evaluators/llama8b-experiment/status_accuracy_output.json \ + .tmp/evaluators/llama70b-experiment/status_accuracy_output.json \ + --title "Status Accuracy: llama8b vs llama70b" \ + --save src/vuln_analysis/eval/visualizations/plot.png \ + --labels llama8b llama70b +""" + + +def load_box_values(path: str) -> tuple[list[float], str]: + """Load average_score as [mean, min, q1, median, q3, max] from accuracy evaluator output JSON. + Returns (values, label) where label is the input file. + """ + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + + avg = data.get("average_score") + if not isinstance(avg, dict): + raise ValueError(f"File '{path}' does not contain pandas describe() format. Expecting dict in 'average_score'.") + + # Extract values from pandas describe() format: [mean, min, q1, median, q3, max] + values = [ + float(avg.get("mean", 0)), + float(avg.get("min", 0)), + float(avg.get("25%", 0)), + float(avg.get("50%", 0)), + float(avg.get("75%", 0)), + float(avg.get("max", 0)) + ] + + label = os.path.splitext(os.path.basename(path))[0] + return values, label + + +def main() -> int: + parser = argparse.ArgumentParser(description=( + "Plot box-and-whisker charts from evaluator output JSON files (average_score = [mean, min, q1, median, q3, max])." + )) + parser.add_argument("files", nargs="+", help="Paths to output files") + parser.add_argument("--title", default="Accuracy Distribution Across Runs", help="Chart title.") + parser.add_argument("--save", + metavar="OUTPUT_PATH", + help="If provided, save the figure to this path instead of displaying it.") + parser.add_argument("--dpi", type=int, default=160, help="Figure DPI when saving (default: 160).") + parser.add_argument( + "--labels", + nargs='*', + help="Optional custom x-axis labels (one per file, in order). If omitted, .json file names are used.") + + args = parser.parse_args() + + # Load all files + stats_values: list[dict] = [] + means_overlay: list[tuple[float, float]] = [] + labels: list[str] = [] + + for idx, path in enumerate(args.files): + values, default_label = load_box_values(path) + mean_value, min_value, q1, median, q3, max_value = values + + # Matplotlib boxplot stats dictionary + stats_values.append({'whislo': min_value, 'q1': q1, 'med': median, 'q3': q3, 'whishi': max_value, 'fliers': []}) + if args.labels and len(args.labels) == len(args.files): + labels.append(args.labels[idx]) + else: + labels.append(default_label) + means_overlay.append((len(labels), mean_value)) + + if not stats_values: + print("No valid files provided.") + return 1 + + # Plot + fig, ax = plt.subplots(figsize=(max(6, 1.8 * len(stats_values)), 5)) + + # Create boxplot from precomputed stats + bp = ax.bxp(stats_values, showfliers=False) + + # Overlay means as black diamonds + for x_pos, mean_value in means_overlay: + ax.plot(x_pos, mean_value, marker='D', color='black', linestyle='None', label='_nolegend_') + + ax.set_title(args.title) + ax.set_ylabel('Accuracy') + ax.set_xticks(range(1, len(labels) + 1)) + ax.set_xticklabels(labels, rotation=20, ha='right') + ax.set_ylim(0.0, 1.0) + ax.grid(axis='y', linestyle='--', alpha=0.3) + + # Legend entry for mean marker + ax.plot([], [], marker='D', color='black', linestyle='None', label='Mean') + ax.legend() + + fig.tight_layout() + + if args.save: + fig.savefig(args.save, dpi=args.dpi, bbox_inches='tight') + print(f"Saved figure to {args.save}") + else: + plt.show() + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/vuln_analysis/register.py b/src/vuln_analysis/register.py index df8a6321c..e123a8d70 100644 --- a/src/vuln_analysis/register.py +++ b/src/vuln_analysis/register.py @@ -31,6 +31,8 @@ from vuln_analysis.data_models.output import AgentMorpheusOutput from vuln_analysis.data_models.state import AgentMorpheusEngineState # pylint: disable=unused-import +from vuln_analysis.eval.evaluators import accuracy +from vuln_analysis.eval.evaluators import consistency from vuln_analysis.functions import cve_agent from vuln_analysis.functions import cve_check_vuln_deps from vuln_analysis.functions import cve_checklist