Enriching Music with Synchronized Lyrics, Images and Colored Lights

We present a method to synchronize popular music with its lyrics at the stanza level. First we apply an algorithm to segment audio content into harmonically similar and/or contrasting progressions, i.e. the stanzas. We map the stanzas found to a sequence of labels, where stanzas with a similar progression are mapped to the same label. The lyrics are analyzed as well to compute a second sequence of labels. Using dynamic programming, an optimal match is found between the two sequences, resulting in a stanzalevel synchronization of the lyrics and the audio. The synchronized lyrics can be used to compute a synchronized slide show to accompany the music, where the images are retrieved using the lyrics. For an additional enrichment of the experience, colored light effects are synchronized with the music that are computed from the sets of images. The song segmentation can be done reliably, while the mapping of the audio segments and lyrics gives encouraging results.