Experimental study conducting various ablations to the LayerNorm present in the ViT architecture.
To reproduce our results, run the following commands to set up the Python environment:
conda create -n DyT python=3.12
conda activate DyT
conda install pytorch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 pytorch-cuda=12.4 -c pytorch -c nvidia
pip install timm==1.0.15 tensorboard
To reproduce our results on ImageNet-100 with ViT-S_16x16_patch , run the following commands:
Dynamic Tanh
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_tanh true --batch_size 128 --model vit_small_patch16_224
Dynamic Sigmoid
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_sigmoid true --batch_size 128 --model vit_small_patch16_224
Dynamic Softsign
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_softsign true --batch_size 128 --model vit_small_patch16_224
RMS Norm
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --rms_norm true --batch_size 128 --model vit_small_patch16_224
Batch Norm
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_softsign true --batch_size 128 --model vit_small_patch16_224
Dynamic Tanh + AdamW
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_tanh true --batch_size 128 --model vit_small_patch16_224
Momentum
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_tanh true --batch_size 128 --model vit_small_patch16_224 --opt momentum
RMSProp
sudo python main.py --data_set IMNET --data_path /teamspace/studios/this_studio/dataset/imagenet-100 --enable_wandb true --project v
it_dys_adamw --dynamic_tanh true --batch_size 128 --model vit_small_patch16_224 --opt rmsprop