['25 VLDB] Déjà Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reuse

This repository provides the artifact for the paper "Déjà Vu: Efficient Video-Language Query Engine with Learning-based Inter-Frame Computation Reuse". It walks through dataset preparation, feature extraction, training, evaluation, and plotting for the NextQA experiments.

Table of Contents

1. Prerequisites
2. Downloading videos
3. Transcode videos
4. Extracting input features for training
5. Extracting compressed features
6. Training ReuseViT
7. Extracting ReuseViT and baselines for testing
8. Running end tasks
   • Original
   • ReuseViT
   • Eventful
   • CMC
9. Plotting graph

1. Prerequisites

1-1. Clone and build docker image

git clone https://github.com/anonymous-dejavu/dejavu
cd dejavu
git submodule update --init --recursive

cd docker
./build.sh

1-2. Launch docker container

We require a large amount of storage to keep the extracted features, so we recommend launching the Docker container with a mounted volume.

# Refer to the example script to launch the Docker container.
./launch.sh

1-3. Set data root path

From this point on, unless noted otherwise, we assume the commands are run inside the Docker container.

cd /workspace/configs/paths
vim data.yaml

Set data_dir to your desired path. The storage should be at least 2 TB.

Preparing Datasets

2. Downloading videos

Download the NextQA dataset from the official NextQA repository. Place them under ${data_dir}/datasets/nextqa/videos.

mkdir -p ${data_dir}/datasets/nextqa/videos
cd ${data_dir}/datasets/nextqa/videos
unzip NExTVideo.zip
ls # Should show 0000/ 0001/ 0002/

3. Transcode videos

cd /workspace
python -m src.scripts.transcode dry_run=false
python -m src.scripts.transcode split=val dry_run=false
python -m src.scripts.transcode split=test dry_run=false

4. Extracting input features for training

cd /workspace
python -m src.scripts.extract dry_run=false
python -m src.scripts.extract split=val dry_run=false
python -m src.scripts.extract split=test target_features='i,o' num_gpus=4 num_workers=32 dry_run=false

5. Extracting compressed features

cd /workspace
python -m src.scripts.extract_compressed dry_run=false
python -m src.scripts.extract_compressed split=val dry_run=false
python -m src.scripts.extract_compressed split=test dry_run=false

Check that dataset info is populated correctly.

python -m src.data.components.nextqa +split=train +fps=2 +base_model_name=openai/clip-vit-large-patch14 +return_compressed=true +regenerate_dataset_info=false
python -m src.data.components.nextqa +split=val +fps=2 +base_model_name=openai/clip-vit-large-patch14 +return_compressed=true +regenerate_dataset_info=false
python -m src.data.components.nextqa +split=test +fps=2 +base_model_name=openai/clip-vit-large-patch14 +return_compressed=true +regenerate_dataset_info=false

6. Training ReuseViT

One can train the ReuseViT oneself, or use the checkpoint we provide.

6-1. Training from scratch

items=(0.95 0.9 0.85 0.8)
for item in "${items[@]}"; do
    python src/train.py model.compile=true experiment=nextqa model.loss.target_reuse_rate=${item} model.dry_run=false
done

6-2. Checkpoints

• ReuseViT checkpoints

Place the checkpoints under ${data_dir}/checkpoints/nextqa.

mkdir -p ${data_dir}/checkpoints/nextqa
# Download the checkpoints and place them under ${data_dir}/checkpoints/nextqa
ls ${data_dir}/checkpoints/nextqa # should look like nextqa-68.ckpt nextqa-85.ckpt nextqa-93.ckpt nextqa-95.ckpt

7. Extracting ReuseViT and baselines for testing

7-1. ReuseViT

cd /workspace
items=(68 85 93 95)
for item in "${items[@]}"; do
    python src/eval.py experiment=nextqa-hard-${item} model.dry_run=false data.test_split=test &
done

7-2. DiffRate

To run DiffRate, one should first obtain the pruning and merging decisions. For now, we provide the precomputed logs here, and plan on sharing the code to train DiffRate.

mkdir -p ${data_dir}/checkpoints/diffrate
tar xvzf nextqa.tar.gz -C ${data_dir}/checkpoints/diffrate

items=('31.0' '34.7' '38.5' '42.3' '46.1')
for item in "${items[@]}"; do
    python -m src.scripts.extract mode=diffrate split=test target_features=o num_gpus=4 num_workers=24 batch_size=16 +diffrate_flops=${item} dry_run=false
done

7-3. Eventful

for k in $(seq 1 15); do
    python src/eval.py experiment=eventful reuse_module.decision.k=$((k*10)) model.dry_run=false &
done

7-4. CMC

for k in $(seq 3 3 45); do
    python src/eval.py experiment=cmc reuse_module.decision.threshold=-$k model.dry_run=false data.test_split=test &
done

8. Running end tasks

First, build and launch the Docker container for the end tasks. The following two commands are run on the host machine.

cd docker/next-gqa
./build.sh
./launch.sh

Make sure the third_parties/NExT-GQA submodule is initialized before this step. We assume the following commands are run inside the Docker container.

8-1. Original

cd /workspace/third_parties/NExT-GQA/code/TempGQA

# Original
CUDA_VISIBLE_DEVICES=0 ./shells/next_test_dual.sh original 2>&1 | tee log_original.txt &

8-2. ReuseViT

items=('68' '85' '93' '95')
for item in "${items[@]}"; do
    ./shells/next_test_dual.sh reuse-$item 2>&1 | tee log_reuse-$item.txt &
done

8-3. Eventful

for item in $(seq 1 15); do
    target=$((item*10))
    ./shells/next_test_dual.sh eventful_${target} 2>&1 | tee log_eventful_${target}.txt &
done

8-4. CMC

for k in $(seq 3 3 45); do
    threshold="-$k"
    ./shells/next_test_dual.sh cmc_${threshold} 2>&1 | tee log_cmc_${threshold}.txt &
done

9. Plotting graph

cd /workspace/scripts
python plot-nextqa.py

After running the script, you should see results similar to the following.