Skip to content

Missing S6 (new→old) config/checkpoint and mismatch between released S5 YAML and checkpoint config #1

Description

@tzcinhust

Hi, thank you for releasing SADA.

I am trying to reproduce the EPIC-Kitchens S6 result in Table 1 (new → old):

  • tIoU 0.1–0.5: 23.94 / 22.95 / 21.47 / 19.16 / 15.24
  • Average mAP: 20.55

I used repository commit:

a96a15b0d0c905a553c7a2b14f607c9485ebe758

However, I could not find an author-released S6 (new_vs_old) configuration or checkpoint. The released checkpoint package appears to contain S1, S2, S4, S5, and CharadesEgo, but not S3 or S6. The repository contains configs/old_vs_new/version_final_ckpt.yaml, but no corresponding new_vs_old configuration.

I also found that the configuration embedded in the official S5 checkpoint:

best_ckpt_old_vs_new/saved_ckpt/best_ckpt/checkpoint-epoch10.pt

differs substantially from the released:

configs/old_vs_new/version_final_ckpt.yaml

Parameter S5 checkpoint config Released YAML
dim_proj_adv 128 128
pl_threshold 0.55 0.3
conditional_adv_drop_rate 0.5 0.1
loss_weights.weight_adv 3 2
loss_weights.weight_downstream 3 2
disc_learning_rate 0.02 0.004
beta 1 0.5
weight_source_cond_adv 0.6 0.75
weight_target_cond_adv 0.6 0.75
Level weights [0.3, 0.3, 0.2, 0.4, 0.3, 0.9] [0.6, 0.1, 1.0, 0.6, 0.2, 0.4]

The tensor shapes in the official S5 checkpoint also confirm that its discriminator width is 128:

  • Standard discriminator: (128, 512) → (128, 128) → (1, 128)
  • Conditional discriminator: (128, 1024) → (128, 128) → (1, 128)

For S6, I reconstructed the direction {EK100 \ EK55} → EK55 and used the supplementary Table A parameters:

  • Level weights: [0.6, 0.1, 1.0, 0.6, 0.2, 0.4]
  • alpha / pl_threshold = 0.3
  • weight_adv = 2.0
  • Seed 6
  • Checkpoint selection based only on source-validation downstream loss

Results so far:

  • Width 128: target-training Avg mAP was 18.30 at epoch 5 and 17.91 at the source-validation-selected epoch 10 checkpoint. Later epochs declined.
  • Width 512 controlled run: target-training Avg mAP was 18.19 at epoch 5.

I also observe differences between the released annotation package and the S6 split counts reported in the paper appendix:

  • Paper source: train 150 / validation 44
  • Released package source: train 157 / validation 37
  • Paper target: train 342 / validation 90
  • Released package target: train 345 / validation 87

Could you please clarify or share:

  1. The exact original S6 new_vs_old YAML/configuration used for the paper.
  2. The official S6 checkpoint, if available.
  3. The annotation/split version used to obtain the S6 train/validation counts reported in the paper.
  4. Whether configs/old_vs_new/version_final_ckpt.yaml is intended to reproduce S5, given that it differs from the configuration embedded in the official S5 checkpoint.
  5. The exact dim_proj_adv used for S6. The paper architecture description mentions width 512, while the official S5 checkpoint uses 128.
  6. Whether additional scenario-specific parameters not listed in Table A are required to reproduce S6.

Any guidance or released artifacts would be greatly appreciated. Thank you!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions