Hi,
i recently tried this generator, and i was quite impressed about the project. Thanks a lot!
This really addressees the whole toolchain from mp3 to song, which is amazing.
However there are some aspects that were not yet working great, so i took a closer look.
I am considering creating a fork of the project so i can work on it some more and give that back as well.
the problem with the hanging ui became an issue because if the ui doesn't respond within 60 seconds, it is will be forced to a break. so i start the generator in a thread now.
the main issue is that the song generation 'invents' notes, rather than matching a detected beat. due to that the songs felt very off.
This is happening in two places:
-
The BaseRhythmGenerator doesn't take the actual beats, but rather just works with the estimated bpm to create notes at fitting places. This is not really great because the bpm is just an estimate and typically a bit (or a lot) off. for example if the real bpm would be 120 and the estimate 119, then all notes get an increasing distance from their supposed position. at some point the error will become so huge that it rolls over, so occasionally (but rarely) a note matches.
Another problem with this is that it assumes the song to have a constant bpm. This is definitely not true; many artists play with this the tempo to create slow sections or ramping up/down sections. even if they would restrict themselves (maybe a section with exactly 50% bpm, so one could say the notes still match the beat) it creates notes that shouldn't be there.
So my suggestion here would be that the rythm generator only works with real beats and applies its patterns on them.
-
The BeatDetector also creates notes ('regular beats') rather than detecting them reliably. When i visualized the generated 'real' notes, i found that it overall did not find many beats. I am not really sure why it doesn't as the intent of it all wasn't too clear to me. I at least found it strange that during debugging a saw a median filter with a window size of 44100 being applied on an array with ~15k elements, but not sure if that is its only problem.
I created an alternative detector with the intention of identifying all / the most relevant notes, based on some ideas seen in the code but applied differently.
The idea is to first identify the most influential frequency for each time position, which is the one with the greatest variation in strength over a window (of 1 second). This helps isolating the beat to focus on one frequency rather than all of them, which gives a more clear beat. Then beat candidates are found, which are a local maximum in strength, above a certain threshold. (only considering the frequency that is currently most important).
After that, beats are merged that are too close to each other, with the stronger beat consuming the weaker neighbors (important, because averaging would create artificial off beat data!); within a window of some milliseconds ( not sure about its range).
This approach can handle corner cases like varying bpm quite nicely, so that one can feel a difference between song sections. It produces a lot of notes during intense sections, but i think it is better if the rythm generator decides which ones not to utilize.
i attached a diff file with the tweaks i made; it is by no means well written and just to get the idea across but if you want to take a look or try it, feel free to do so.
unified_diff_song_generator.diff.txt
with that change the generator uses the new beat detection to create a song that only contains aligned notes. it can handle varying bpm's during a song as well, so that fast sections typically result in more notes than slow sections. the note generation is currently reduced to just random notes; mainly to verify the detector. (also just expert to debug it faster; not that it matters while the difficulty filter is commented out)
Hi,
i recently tried this generator, and i was quite impressed about the project. Thanks a lot!
This really addressees the whole toolchain from mp3 to song, which is amazing.
However there are some aspects that were not yet working great, so i took a closer look.
I am considering creating a fork of the project so i can work on it some more and give that back as well.
the problem with the hanging ui became an issue because if the ui doesn't respond within 60 seconds, it is will be forced to a break. so i start the generator in a thread now.
the main issue is that the song generation 'invents' notes, rather than matching a detected beat. due to that the songs felt very off.
This is happening in two places:
The BaseRhythmGenerator doesn't take the actual beats, but rather just works with the estimated bpm to create notes at fitting places. This is not really great because the bpm is just an estimate and typically a bit (or a lot) off. for example if the real bpm would be 120 and the estimate 119, then all notes get an increasing distance from their supposed position. at some point the error will become so huge that it rolls over, so occasionally (but rarely) a note matches.
Another problem with this is that it assumes the song to have a constant bpm. This is definitely not true; many artists play with this the tempo to create slow sections or ramping up/down sections. even if they would restrict themselves (maybe a section with exactly 50% bpm, so one could say the notes still match the beat) it creates notes that shouldn't be there.
So my suggestion here would be that the rythm generator only works with real beats and applies its patterns on them.
The BeatDetector also creates notes ('regular beats') rather than detecting them reliably. When i visualized the generated 'real' notes, i found that it overall did not find many beats. I am not really sure why it doesn't as the intent of it all wasn't too clear to me. I at least found it strange that during debugging a saw a median filter with a window size of 44100 being applied on an array with ~15k elements, but not sure if that is its only problem.
I created an alternative detector with the intention of identifying all / the most relevant notes, based on some ideas seen in the code but applied differently.
The idea is to first identify the most influential frequency for each time position, which is the one with the greatest variation in strength over a window (of 1 second). This helps isolating the beat to focus on one frequency rather than all of them, which gives a more clear beat. Then beat candidates are found, which are a local maximum in strength, above a certain threshold. (only considering the frequency that is currently most important).
After that, beats are merged that are too close to each other, with the stronger beat consuming the weaker neighbors (important, because averaging would create artificial off beat data!); within a window of some milliseconds ( not sure about its range).
This approach can handle corner cases like varying bpm quite nicely, so that one can feel a difference between song sections. It produces a lot of notes during intense sections, but i think it is better if the rythm generator decides which ones not to utilize.
i attached a diff file with the tweaks i made; it is by no means well written and just to get the idea across but if you want to take a look or try it, feel free to do so.
unified_diff_song_generator.diff.txt
with that change the generator uses the new beat detection to create a song that only contains aligned notes. it can handle varying bpm's during a song as well, so that fast sections typically result in more notes than slow sections. the note generation is currently reduced to just random notes; mainly to verify the detector. (also just expert to debug it faster; not that it matters while the difficulty filter is commented out)