example_video.mp4
This project is a fork of a Pytorch DeepDream implementation by Aleksa Gordić where I have applied the DeepDream algorithm to audio. VGG19 models were trained to classify mel spectrograms from two datasets; UrbanSounds8k and free-music-archive (small). These trained models are then used to "dream" audio features onto input mel spectrograms; like the original DeepDream this creates some interesting outputs!
WARNING! This code can produce some horrible / loud sounds, turn the volume down before listening to an output for the first time.
Note: due to git lfs not supporting public forks, I have a separate repo where the required model files can be found.
examples/ contains some example outputs, including the parameters that were used in each case.
Note: For completeness, I haven't cropped any of these wav files. Some of the ouroboros files end up "overcooked" (bad screeching sound).
Important parameters
- input = the initial input to the model, either "noise" (default), "sine" or "path to wav file".
- layers_to_use = the activation layers to use, gradient ascent is done on the mean of these (see model.py for the list of available layers, "iterate" iterates backwards across the layers for ouroboros output).
- target_class_index = if "output" in layers_to_use, the output class to target / measure loss against.
- create_ouroboros = passes output into input repeatedly to create a sequence of progressively more processed clips.
python dreamsound.py --input="sine" --layers_to_use "relu1_1" "relu5_2" "output" --target_class_index=4
There is a tendency for ouroboros audio to eventually become "overcooked" (bad screeching sound), decreasing the number of iterations per clip can increase the length of ouroboros before this occurs.
Please explore the other parameters and see how they affect the output! I would love to hear clips generated with this project, I need to figure out a good way to receive them.
UrbanSounds8k - 10 classes
- air_conditioner
- car_horn
- children_playing
- dog_bark
- drilling
- engine_idling
- gun_shot
- jackhammer
- siren
- street_music
free-music-archive small - 8 classes
- hip_hop
- pop
- folk
- experimental
- rock
- international
- electronic
- instrumental
I developed this code from https://github.com/gordicaleksa/pytorch-deepdream, many thanks to gordicaleksa for the fantastic DeepDream implementation!
The code I used to train the models was heavily inspired by the following series: https://towardsdatascience.com/urban-sound-classification-part-1-99137c6335f9
UrbandSounds8k dataset: https://urbansounddataset.weebly.com/
Free music archive dataset: https://github.com/mdeff/fma