KnowVal

Paper | Zhongyu Xia, Wenhao Chen, Yongtao Wang, Ming-Hsuan Yang

This is the official implementation of KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System

KnowVal releases the code for a knowledge-aware Value Model for autonomous driving research. The model scores a driving scene against rule embeddings and can be used as a learned value signal in planning- or policy-evaluation pipelines.

Method Overview

KnowVal follows the paper's overall architecture. It first performs retrieval-guided open-world perception to extract instance-, panoramic-, and concept-level information. It then conducts perception-guided retrieval from the knowledge graph to obtain relevant knowledge. Finally, decision-making is achieved through planning, world prediction, and value assessment. Feature propagation across modules keeps perception, retrieval, and planning connected while allowing the core perception and planning components to be adapted from existing driving methods.

Repository Scope

This code release covers two components:

• the scene-feature, per-knowledge-entry subscore Value Model;
• a knowledge retrieval module that maps scene perception into ranked traffic-law knowledge entries for the Value Model.

Environment

If you need to use the retrieval function, we recommend installing LightRAG first. If you want to use the Value Model, you need to install mmdetection3d. We recommend that you install HENet directly, which is a derivative package of mmdetection3d v1.0 and includes mmdet3d.

This release has been tested with:

• Python 3.8
• PyTorch 2.0.1 + CUDA 11.8
• MMCV 1.6.0
• MMDetection 2.28.2
• MMDetection3D 1.0.0rc4

The paper uses Qwen2.5-3B for retrieval and knowledge embedding. In this public repository, paper-aligned retrieval wires Qwen2.5-3B/vLLM into the two-layer keyword extraction stage; the knowledge-embedding checkpoint is not distributed in Git. The public offline_demo mode is a deterministic local test path for interface examples; it does not reproduce the paper's LLM-based retrieval stage.

Quick Start

Set up the repository from source. The default retrieval demo and public release check use only the Python standard library; pytest is needed only for tests.

git clone https://github.com/VDIGPKU/KnowVal.git
cd KnowVal
conda create -n knowval-public python=3.8 pytest
conda activate knowval-public

Run the offline perception-guided knowledge retrieval demo:

python -m knowval_retrieval.cli \
  --knowledge-base knowval_retrieval/examples/traffic_law_sample_kg.json \
  --perception-file knowval_retrieval/examples/perception_case_crosswalk.json \
  --top-k 4

Run the retrieval tests:

python -m pytest -q \
  scripts/test_knowval_retrieval.py

Qwen/vLLM keyword extraction for the paper-aligned retrieval stage is documented in docs/RETRIEVAL.md. Value Model weight placement and checksum verification are documented in docs/VALUE_MODEL.md.

Run the public release check. This default check does not import the OpenMMLab Value Model stack:

python scripts/evaluate_public_release.py

In an environment with the MMDetection/MMCV/MMDetection3D dependencies used by the released implementation, the Value Model test can be added explicitly:

python scripts/evaluate_public_release.py --include-value-model

Component Documentation

• Retrieval.md: perception-guided retrieval from the knowledge graph, the public traffic-rule sample KG, Qwen/vLLM usage, and the rule_embed handoff.
• Value Model.md: OpenMMLab plugin usage, feature schema, weight placement, and test commands.

Citation

If this work is helpful for your research, please consider citing KnowVal.

@article{xia2025knowval,
  title={KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System},
  author={Xia, Zhongyu and Chen, Wenhao and Wang, Yongtao and Yang, Ming-Hsuan},
  journal={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2026}
}

License

This repository is released under the Apache License 2.0. See LICENSE for details. In addition, the project is free only for academic research purposes; it requires authorization for commercial use. For collaboration or commerce permission, please contact wyt@pku.edu.cn.