Building A Better Algorithm (With A Little Help From Your Friends)


Building A Better Algorithm (With A Little Help From Your Friends)

The Future of Entertainment series by iQ by Intel and PSFK Labs is highlighting the latest in entertainment innovation. Over the course of 10 weeks at iq.intel.com, we are showcasing new products, services and technologies, exploring the changing face of how we consume, share and create content and getting reactions from Intel experts.


Over the course of this week, we’ve been highlighting a number of new services that leverage the knowledge of our friends to help us discover new music, movies and TV shows. These recommendation tools are all part of a trend we’re calling Socially Curated Discovery, which looks at the ways technology is helping to personalize our entertainment experiences with content choices that better match our likes and interests.

To get an expert perspective on the evolution of these services, the growing sophistication of algorithms and how they’ll influence our consumption habits, we spoke with Ted Willke, a Principal Engineer with Intel and the General Manager of the Graph Analytics Operation in Intel Labs. He and his colleagues focus on developing disruptive technologies that will improve the performance and expand the capabilities of large-scale machine learning and data mining.

How are recommendation engines evolving alongside the social web?

At first, recommendation engines were quite rudimentary and made largely ineffective predictions based on similarities inherent in items themselves and the principle of like-mindedness, which says that people with similar historical preferences are likely to share future preferences. But now, the recommendations provided by these engines are becoming more and more personalized, and the social web is playing a big part in this.

Personalization means going beyond basic similarities to account for an individual’s behavior and temporal viewing/purchasing history, as well as how they’re influenced by their trusted social network and the preferences of individuals within it.  Recommendations based on general like-mindedness can be blended with personalization, resulting in more accurate and effective recommendations independent of the degree and recency of an individual’s relevant activity.  

The social web also augments other sources of rapidly-changing contextual information (e.g., time of day, location, current activities, etc.) that can be used to filter the recommendations made by these engines for even more relevance. 

How are these advancements changing the way people discover new content?

It used to be that recommendations were completely un-personalized and the more different or niche your preferences and tastes were, the more off target the engines were. This meant that people on the fringe were on their own to discover new content and the remaining masses were served up mainstream content with a low degree of diverse content, due to the most similar and like-minded content being ranked higher by engines. And, this led to a self-amplifying vicious cycle in which the most sought after content was also the most discoverable, making it the most likely to be selected, making it even more likely to be recommended/discoverable, further narrowing the presentation of content to the mainstream.  

Now, you can get very personalized content recommendations for music, video, news, etc., that accurately reflect your individual tastes and consumption habits, no matter how left field.  The result is that more people trust content recommendation engines with exposing them to the possibilities and are doing less hunting and surfing on their own (without the guidance of a recommendation engine), breaking the vicious cycle of mainstream content amplification and increasing content diversity by capturing more of the fringe. 

Are you noticing any shifts in the way people are consuming their content these days?

People are experiencing higher levels of desirable content diversity and content satisfaction from the recommendations made by engines that are trained to their personal taste and able to explore the content space within and around it. This is leading to increased use of content services (again, music, video, news, etc.) that allow users to take a more passive role in the exploration of content while simultaneously maximizing exposure to new content.  

The fusion of social network information with these content services is fueling the paradox that although most people are consuming/experiencing their personal content in a very private manner in the physical world (i.e., earbuds in and small screens held close), they are exposing more and more of their individual tastes and preferences to social networks with a lower degree of concern for privacy. This fusion is also leading to more trust-based consumption, where like-mindedness now influences consumption in a more personal scope.

As algorithms become more proficient at finding connections between people, what kind of experiences can we expect when using these engines in the future?

We will be more and more surprised by what these engines recommend while being more satisfied with the results. This will result from applying much more data to better algorithms, with an emphasis on more data, which in this case means increasing the number of explicit and implicit connections between people and defining more complex but meaningful pathways.  

Soon enough, these engines may know you and your tastes better than you do, providing an objective viewpoint that is highly accurate and effective. And, they will become the standard for discovering new types of content that we cannot begin to imagine today.

What are the greatest challenges/obstacles encountered when trying to make search/relationship engines better?

People’s tastes and habits can change in subtle and unpredictable ways for many reasons. These shifts are difficult to anticipate and can make it hard for engines to make consistently good recommendations for individuals over long periods of time. We need to power discovery with engines that have the ability to continuously assess their own effectiveness and adapt their approach (e.g., models, filters, etc.) to the circumstance at hand.

Now that we’ve pointed you towards a future where socially-connected algorithms help us get out of the content rut, you’ve got no excuse for watching reruns or listening to your favorite six tracks on continuous repeat. Check some the other examples of Socially Curated Discovery below and tune in next week for a look at multiscreen storytelling: