AI Karaoke – Generate Lyrics based on Karaoke video
2016 EXPERIMENTS AI MUSIC
Maybe you are not familiar with Karaoke machines in Japan. You’ll see not only lyrics but also an accompanying short movie on the screen while singing… and the video tends to have nothing to do with the lyrics of the song! Here comes AI Karaoke: our system generates lyrics on-the-fly based on Karaoke videos and you can sing it instead of the original lyrics.
Here is a short snippet of a test I did at eAT Kanazawa 2016 opening party. You see generated lyrics on the top of the original lyrics on the screen… The generated lyrics felt like a collection of a random text: it has no storyline. But IT WAS HILARIOUS! ;-)
Inspired by Kyle McDonald’s experiments, I got this idea.
I used NeuralTalk to describe the scene in a Karaoke movie and fed the description to an RNN trained with J-POP lyrics as a seed. This test led to a project with Dentsu Lab Tokyo.