This project is an implementation of a transformer from scratch based on the proposed approach in Attention Is All You Need. A transformer is a deep learning model that adopts the mechanism of self-attention, differentially weighting the significance of each part of the input data. It is used primarily in the fields of natural language processing (NLP) and computer vision (CV).
This project is a very simple implementation of the transformer architecture with all of its building blocks. To get a better pictrure; here is the transformer architecture block diagram (Vaswani et al. 2017):
You can find each block of this diagram in the transformer directory.
│ README.md
│ .gitignore
│
└───src
│ └───model
│ └───test
│ └───transformer
| | transformer.py
| └───decoder
| └───encoder
| └───modules
|
└───res <! --- Documentation resources --->
To get a local copy up and running follow these simple example steps.
- Clone the repo
git clone https://github.com/omarxadel/transformer
- Install requirements
pip install -r /path/to/requirements.txt
This transformer has not yet been tested on a real-life use case (still work in progress). To validate that it's working you need to run in the CMD:
python3 -m src.test.test_transformerBuilding a model for a simple NLP task and validating it is still Work in Progress.
Contributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are greatly appreciated.
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Omar Adel - omarxadel21@gmail.com