A natural language processing deep learning model in tensorflow attempting to generate true Shakespearean sonnets with the proper rhyme and meter schemes. We hope that by adding an attention mechanism to Myiamoto & Cho's Gated RLM, the model will be able to pick up on the semantics and morpheme level information to generate solid poems. Our thoughts are expanded in our report and the original publication can be found in this repo as well. Their original source code, in Theano, can be found here.
If anyone would like to collaborate on this, I would appreciate some help. I have no formal training in machine learning or language processing, this is tough stuff!
$ python tokenize_file.py <File Names>
(tensorflow)$ python train.py
- assumes default settings and file hierarchy, bellow, options yet to be set
(tensorflow)$ python sample.py
- NOT YET IMPLEMENTED
- base.py
- basic helper functions & Keras’ Progbar object to visualize training
- data_preprocess.py
- from Miyamoto and Cho’s source code, some functions were slightly modified to accommodate our model and/or TensorFlow
- layers.py
- the layers for the model
- gated_rlm.py
- an implementation of the model using TensorFlows app.flags function to initialize, share, and update variables through the TensorFlow session)
- train.py
- the training script for the gated rlm model,though most of the training logic still resides in gated_rlm.py)
- wordChar_prep.py
- some basic utils
- sample.py
- the sampling script for the gated rlm model to sample output of trained models, not yet implemented
- run_dir
- files generated by the model (checkpoints, stats, temporary files, etc
- recommended to run the model from here with
(tensorflow)$python ../train.pycommand
Gated Recurrent Language Model
.
├── Andromeda_Capstone_Report.pdf
│ # PDF report outlining our research and goals
│
├── Character_Recurrent_Language_Model.pdf
│ # Miyamoto and Cho's original publication on their gated rlm
│
├── README.md
│ # this readme file
│
├── SourceCode
│ │ # this folder holds all of the scripts & files that the model needs to train and execute
│ │
│ ├── gated_rlm.py
│ │ # an implementation of the model using TensorFlows app.flags function to initialize,
│ │ share, and update variables through the TensorFlow session
│ │
│ ├── layers.py
│ │ # implementation of the various layers in the model
│ │
│ ├── train.py
│ │ # the training script for the gated rlm model,
│ │ though most of the training logic still resides in gated_rlm.py
│ │
│ ├── sample.py
│ │ # the sampling script for the gated rlm model to sample output of trained models,
│ │ not yet implemented
│ │
│ ├── wordChar_prep.py
│ │ # some basic utils
│ │
│ ├── base.py
│ │ # basic helper functions & Keras’ Progbar object to visualize training
│ │
│ ├── data_preprocess.py
│ │ # from Miyamoto and Cho’s source code, some functions were slightly modified
│ │ to accommodate our model and/or TensorFlow
│ │
│ ├── run_dir
│ │ # recommended to run th model from here with "(tensorflow)$python ../train.py" command
│ │ # collects files generated by the model (checkpoints, stats, temporary files, etc)
│ │
│ └── data
│ │ # This folder contains files that the model needs to use during training/execution
│ │ # word and char dicts for training
│ │ # GloVe lookup table
│ │ # training, testing, and validation sonnet files
│ │
│ ├── basic_dicts
│ │ │
│ │ ├── char_dict.txt
│ │ │ # simple char vocab without indices, one char per line
│ │ │
│ │ └── word_dict.txt
│ │ # word vocab with the number of occurrences per word, for GloVe
│ │
│ ├── MnC_dicts
│ │ │ # word & character vocab for the model from Miyamoto and Cho’s script
│ │ │
│ │ ├── char_dict.pkl
│ │ └── word_dict.pkl
│ │
│ ├── yoon_dicts
│ │ │ # word & character vocab for the model from Yoon Kim’s script
│ │ │
│ │ ├── char_vocab.pkl
│ │ └── word_vocab.pkl
│ │
│ │
│ ├── GloVe_vectors.trimmed.200d.npz
│ │ # 200-dimensional word embeddings trained on entire corpus,
│ │ trimmed to only include words found in the sonnet files
│ │
│ ├── shakespeare_corpus
│ │ │
│ │ ├── raw
│ │ │ │
│ │ │ ├── citation-n-legal_stuff.txt
│ │ │ │ # legal information from Project Gutenberg
│ │ │ │
│ │ │ ├── shakespeare_complete.txt
│ │ │ │ # the complete works of William Shakespeare
│ │ │ │
│ │ │ └── shakespeare_sonnets.txt
│ │ │ # Shakespeare's sonnets
│ │ │
│ │ └── tokenized
│ │ │
│ │ └── shakespeare_complete_tokenized.txt
│ │ # the complete works of William Shakespeare tokenized
│ │
│ └── sonnets
│ │
│ ├── raw
│ │ │
│ │ ├── sonnets.txt
│ │ │ # all sonnets in a single file, without titles
│ │ │
│ │ ├── test.txt
│ │ │ # test set: 16 sonnnets
│ │ │
│ │ ├── train.txt
│ │ │ # training set: 122 sonnets
│ │ │
│ │ └── valid.txt
│ │ # development/validation set: 16 sonnets
│ │
│ └── tokenized
│ │
│ ├── sonnets_tokenized.txt
│ │ # all sonnets in one file, tokenized
│ │
│ ├── test_tokenized.txt
│ │ # test set, tokenized
│ │
│ ├── train_tokenized.txt
│ │ # training set, tokenized
│ │
│ └── valid_tokenized.txt
│ # development set, tokenized
│
└── tools
│ #tools to help with some of the data preprocessing
│
├── build_dictionary_char.py
│ # build a char dictionary (in cPickle), by Miyamoto & Cho
│
├── build_dictionary_word.py
│ # build a word dictionary (in cPickle), by Miyamoto & Cho
│
└── tokenize_file.py
# preprocess files fed to the model, dictionary scripts, and GloVe