The launch of Google's music beta service last week introduced us to yet another playlist-building app called Instant Mix. Simply pick a song in your library, and Instant Mix will build a playlist based on similar songs also in your library. If that sounds a lot like Apple's Genius feature, that's because it is.
So Echo Nest director of developer community Paul Lamere put the two to a head-to-head test , along with the music recommendation of the Echo Nest, to see which work better. Now granted he's likely biased towards the Echo Nest's platforms, but his research is extensive enough to be worth a look. And the results are a bit surprising considering Google's already begun to tout its music recommendation feature.
Lamere, who admits evaluating music playlists is hard, has come up with a novel way to quantify how well a program does at building playlists: it's called the "WTF test:"
"It is really quite simple," writes Lamere. "You generate a playlist, and just count the number of head-scratchers in the list….If a playlist is filled with jarring transitions, leaving the listener with iPod whiplash as they are jerked through songs of vastly different styles, it is a bad playlist."
In his five song studies, which ranged from Miles Davis' "Kind of Blue" to Kraftwerk's "Autobahn" to Lady Gaga's "Bad Romance," Google's Instant Mix continually had more WTFs than Apple's Genius feature or the Echo Nest (except when it came to Beatles' songs, for which iTunes was unable to generate any playlist) by a large margin. As is to why Google's Instant Mix would recommend songs by Jack Johnson, Pete Townshend and Sting (amongst many other strange choices) for the minimalist electronic song "Autobahn" is indeed a WTF head-scratcher.
At the end of the study, Lamere concludes that, "The biggest surprise of all in this evaluation is how poorly Google's Instant Mix performed. Nearly half of all songs in Instant Mix playlists were head scratchers - songs that just didn't belong in the playlist. These playlists were not usable. It is a bit of a puzzle as to why the playlists are so bad considering all of the smart people at Google. Google does say that this release is a Beta, so we can give them a little leeway here. And I certainly wouldn't count Google out here. They are data kings, and once the data starts rolling from millions of users, you can bet that their playlists will improve over time, just like Apple's did. Still, when Paul Joyce said that the Music Beta killer feature is 'Instant Mix', I wonder if perhaps what he meant to say was "the feature that kills Google Music is 'Instant Mix'."