Flickr30k Image Captioning using Vision Transformer and Transformer Decoder

An AI-powered image captioning system that combines Vision Transformers (ViT-B/16) for visual feature extraction with a Transformer-based Decoder for natural language generation. The model is trained on the Flickr30k dataset and generates contextually relevant captions for images.

🚀 Project Highlights

• Extracts image embeddings using Vision Transformer (ViT-B/16).
• Uses a Transformer Decoder with Beam Search for caption generation.
• Trained on the Flickr30k dataset containing 31,000+ images and 158,000+ captions.
• Evaluates model performance using BLEU scores.
• Implements Early Stopping and Cosine Annealing Learning Rate Scheduling.
• Supports feature extraction using both ViT and ResNet-50.

---

🧠 Tech Stack

• Python
• PyTorch
• Vision Transformer (ViT)
• Transformer Architecture
• Hugging Face Transformers
• NumPy
• NLTK
• spaCy
• Flickr30k Dataset

---

📂 Dataset

This project uses the Flickr30k dataset:

• ~31,000 Images
• ~158,000 Human-Written Captions
• Rich Visual and Textual Annotations

Dataset Source: https://www.kaggle.com/datasets/hsankesara/flickr-image-dataset

---

⚙️ Workflow

1. Image Preprocessing
2. Feature Extraction using ViT-B/16
3. Caption Tokenization and Vocabulary Building
4. Transformer Decoder Training
5. Beam Search Caption Generation
6. BLEU Score Evaluation

---

📊 Model Features

Vision Encoder

• ViT-B/16
• ResNet-50 (Optional)

Language Decoder

• Transformer Decoder
• Positional Encoding
• Multi-Head Self-Attention

Training Enhancements

• Early Stopping
• Cosine Annealing Scheduler
• Batch Processing
• Teacher Forcing

---

📁 Project Structure

image-caption-transformer/

├── README.md

├── LICENSE

├── requirements.txt

├── gen.ipynb

├── testing.ipynb

├── preprocess.py

├── model.py

├── dataset.py

├── train.py

├── evaluate.py

├── utils.py

└── main.py

---

📈 Evaluation Metrics

The model performance is measured using:

• BLEU-1
• BLEU-2
• BLEU-3
• BLEU-4

---

🔮 Future Improvements

• CLIP-based Feature Extraction
• Attention Visualization
• Web Application Deployment
• Real-Time Image Caption Generation
• Multilingual Captioning

---

🤝 Contributions

Contributions are welcome.

1. Fork the repository
2. Create a new branch
3. Commit your changes
4. Push the branch
5. Open a Pull Request

---

📜 License

This project is licensed under the MIT License.

---

🙏 Acknowledgments

• Flickr30k Dataset
• PyTorch
• Hugging Face Transformers
• spaCy
• NLTK