CLIP-like-models-text-gradient-ascent

Gradient Ascent for Text Encoders of CLIP-like models, including CLIP, BLIP, SigLIP, LLM2CLIP.

---

Gets a model's raw 'opinion' (sampled text tokens) by optimizing texts for cosine similarity with a given image embedding.

🎯 Features:

• Gradient Ascent: Sampled text tokens that match the image.
• Loss Landscape Visualization: With 2 random directions and/or 2 PCA directions
• Optimization path tracking via smaller 'marbles' + large red sphere (current)
• 3D Stanford PLY export with vertex colors (use e.g. in Blender)
• 🚨 Warning - may unexpectedly produce NSFW output, harmful stereotypes, etc.
• 🚨 Especially LLM (LLM2CLIP) exhibit 'spurious correlation' sampling (SFW below):

---

Model text token 'opinion':

• Auto-saves best + all tokens (.txt)
• Use --dump_embeds to also save embeddings .pt files
• Use --deterministic for reproducible runs
• Use argument --help with any get_*.py file for all options
• Custom models: --plot_distribution to debug --lr, sampling temp. --tau:

Loss Landscape:

• Enable: --plot_loss_landscape
• Loss landscape d1/d2 from PCA of actual trajectory (default)
• Use --random_directions to create 3D views with random directions instead
• Additional overview with 2 random ortho directions (default); use --no_random to skip:

• Use --landscape_steps to define interval for saving landscapes (default 100)
• Use --init_landscape_steps to also save all n initial steps (default 1)
• Use --landscape_grid to control resolution (default 51 → (51²)=2601 evals computed; try 21 for fast)

Specials:

• Use --opt_lion for lucidrains/lion-pytorch Lion optimizer; pip install lion-pytorch
• Lion: Best for LLM; needs smaller learning rate --lr (half or less), maybe adjust --tau
• Needs larger landscape plot --pca_alpha_range due to trajectory; try:

python get_llm2clip_opinion_gradient_ascent.py --deterministic --plot_loss_landscape --opt_lion --lr 2 --pca_alpha_range 60

---