TimeSoccer

TimeSoccer is an end-to-end Multimodal Large Language Model (MLLM) for soccer commentary generation, built upon TimeChat. It extends TimeChat with long-video support, team-aware history modeling, and dense video captioning tailored for soccer match broadcasts.

TimeSoccer: An End-to-End Multimodal Large Language Model for Soccer Commentary Generation
Ling You, Wenxuan Huang, Xinni Xie, Xiangyi Wei, Bangyan Li, Shaohui Lin, Yang Li, Changbo Wang
ACM International Conference on Multimedia (MM), 2025

---

Pipeline Overview

Train → Convert Annotations → Infer → Evaluate

The end-to-end pipeline is implemented in a single script:

bash scripts/train_soccernet.sh

Below we break down each stage.

---

1. Setup

Prerequisites

• Python 3.8+ (conda or virtualenv recommended)
• FFmpeg (for video processing)
• PyTorch 2.0+ with CUDA
• 4+ GPUs recommended for training (single-GPU possible with reduced batch size)

Install Dependencies

conda env create -f environment.yml
conda activate timechat
pip install -r requirements.txt

Download Pretrained Checkpoints

# EVA ViT-G vision encoder
wget -P ckpt/eva-vit-g https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/eva_vit_g.pth

# InstructBLIP Q-Former
wget -P ckpt/instruct-blip https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/InstructBLIP/instruct_blip_vicuna7b_trimmed.pth

# LLaMA-2 7B Chat backbone
huggingface-cli download --resume-download DAMO-NLP-SG/Video-LLaMA-2-7B-Finetuned --local-dir ckpt/Video-LLaMA-2-7B-Finetuned

# TimeChat pretrained checkpoint
huggingface-cli download --resume-download ShuhuaiRen/TimeChat-7b --local-dir ckpt/timechat

Download Dataset

# TimeIT base dataset (from HuggingFace)
bash scripts/download_timeit.sh

The soccer-specific annotation files are included in data/files/. Place your soccer match videos under data/videos/ (or set VIDEO_ROOT to your path).

Configure Paths

Copy and edit the environment template:

cp .env.example .env

Key variables:

Variable	Description
VIDEO_ROOT	Directory containing soccer match videos/frames
CHECKPOINT_PATH	Pretrained model checkpoint
OUTPUT_DIR	Directory for training outputs and predictions
NUM_FRAMES	Number of frames to sample (default: 288)

---

2. Training

Fine-tune TimeChat on soccer commentary data:

torchrun --nproc_per_node=4 train.py \
  --cfg-path train_configs/stage2_finetune_soccer.yaml \
  --options \
    datasets.time_instruct.build_info.anno_dir="data/TimeIT/data/dense_video_captioning/soccernet/video_soccer_qa_v.json" \
    datasets.time_instruct.build_info.videos_dir="$VIDEO_ROOT" \
    datasets.time_instruct.build_info.use_long_video=True \
    datasets.time_instruct.build_info.max_frame_pos=192 \
    datasets.time_instruct.vis_processor.train.n_frms=192 \
    datasets.time_instruct.num_frm=192 \
    model.max_frame_pos=192 \
    model.use_feature=False \
    model.ckpt="$CHECKPOINT_PATH" \
    run.batch_size_train=4 \
    run.init_lr=3e-5 \
    run.output_dir="outputs/train_stage2_soccernet" \
    run.max_epoch=10 \
    run.iters_per_epoch=316 \
    run.warmup_steps=190

The training config train_configs/stage2_finetune_soccer.yaml uses LoRA for parameter-efficient fine-tuning and supports long video via configurable frame position windows.

---

3. Annotation Conversion

The evaluation pipeline uses COCO-format dense caption annotations. Convert the test annotations:

python utils/transfer_to_coco.py \
  --input-file data/files/long_related/video_soccer_qa_v_test_equal_normal_900.json \
  --output-file data/TimeIT/data/dense_video_captioning/soccernet_v/val.caption_coco_format.json \
  --video-prefix "$VIDEO_ROOT" \
  --use_feature False \
  --use_long_video True \
  --range 5

This expands timestamped events into temporal segments (±range seconds) for standard DVC evaluation.

---

4. Inference

Run inference with the fine-tuned checkpoint:

python evaluate.py \
  --anno_path data/TimeIT/data/dense_video_captioning/soccernet_v \
  --video_path "$VIDEO_ROOT" \
  --task sdvc \
  --dataset soccernet \
  --output_dir outputs/prediction \
  --split val \
  --num_frames 192 \
  --batch_size 1 \
  --prompt_file prompts/sdvc_soccer_description_just2_team.txt \
  --timechat_model_path outputs/train_stage2_soccernet/checkpoint_9.pth \
  --use_long_video \
  --long_video_path "$VIDEO_ROOT"

---

5. Evaluation

Compute Dense Video Captioning (DVC) metrics:

cd metrics/dvc
python eval_dvc.py \
  --pred_file outputs/prediction/fmt_soccernet_val_f192_result.json \
  --gt_file data/TimeIT/data/dense_video_captioning/soccernet_v/val.caption_coco_format.json

The metrics directory also includes evaluators for:

• TVG (Temporal Video Grounding): metrics/tvg/
• VHD (Video Highlight Detection): metrics/vhd/

---

End-to-End Script

All five steps are automated in one script:

# Edit paths in scripts/train_soccernet.sh first
bash scripts/train_soccernet.sh

Key parameters (all optional, set via environment variables):

Variable	Default	Description
VIDEO_ROOT	/path/to/soccernet_image/datasets_image	Soccer video directory
BASE_CKPT	ckpt/timechat/timechat_7b.pth	Base TimeChat checkpoint
NUM_FRAMES	288	Frames sampled per video
MAX_FRAME_POS	288	Max frame position embedding
INIT_LR	3e-6	Initial learning rate
BATCH_SIZE_TRAIN	4	Per-GPU batch size

---

Repository Structure

├── train.py                  # Training entry point
├── evaluate.py               # Inference entry point
├── evaluate_stream.py         # Streaming/long-video inference
├── config.py                  # Configuration helper
├── history_team_model.py      # Team-aware history modeling
├── scripts/
│   ├── train_soccernet.sh     # End-to-end pipeline
│   ├── _load_env.sh           # Environment loader
│   └── download_timeit.sh     # Dataset downloader
├── timechat/                  # Core model (TimeChat)
│   ├── models/                # Model definitions
│   ├── datasets/              # Dataset builders
│   ├── processors/            # Video/text processors
│   ├── runners/               # Training loop
│   ├── tasks/                 # Task definitions
│   └── conversation/          # Inference conversation handling
├── train_configs/             # Training YAML configs
├── eval_configs/              # Evaluation YAML configs
├── prompts/                   # Prompt templates
├── metrics/                   # Evaluation metrics (DVC, TVG, VHD)
│   ├── dvc/
│   ├── tvg/
│   └── vhd/
├── utils/                     # Utility scripts
│   ├── transfer_to_coco.py    # Annotation format conversion
│   ├── format_dvc.py          # DVC output formatting
│   ├── format_sdvc.py         # Soccer DVC formatting
│   └── ...
├── dataset_process/           # Dataset preprocessing
├── data/
│   └── files/                 # Soccer annotation files
├── docs/                      # Documentation
├── figs/                      # Figures
├── .env.example               # Environment template
├── environment.yml            # Conda environment
└── requirements.txt           # Pip dependencies

---

Citation

@inproceedings{you2025timesoccer,
  title={Timesoccer: An end-to-end multimodal large language model for soccer commentary generation},
  author={You, Ling and Huang, Wenxuan and Xie, Xinni and Wei, Xiangyi and Li, Bangyan and Lin, Shaohui and Li, Yang and Wang, Changbo},
  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
  pages={3418--3427},
  year={2025}
}

---

License

This project is licensed under BSD-3-Clause (see LICENSE). The code is adapted from:

• LAVIS (BSD-3-Clause) — see LICENSE_Lavis.md
• MiniGPT-4 (BSD-3-Clause) — see LICENSE_Minigpt4.md
• TimeChat (BSD-3-Clause)

The TimeIT dataset is licensed under CC-BY-4.0. Soccer match video data should be obtained separately and may be subject to third-party licenses.