How to use Collective Intelligence techniques to ensure that your web application can extract valuable data from its usage and deliver

that value right back to the users.

MODULE 1

Hélio TeixeiraNew York, May 2010

Web users are undergoing a transformation… Users are expressing themselves. This expression may be in the

form of: sharing their opinions on a product or a service through reviews or

comments; through sharing and tagging content; through participation in an online community; or by contributing new content.

This increased user interaction and participation gives rise to data that can be converted into intelligence in your application. The use of collective intelligence to personalize a site for a user, to aid him in searching and making decisions, and to make the application more sticky are cherished goals that web applications try to fulfill.

What do these two companies have in common?

They both drew new conclusions and created new business opportunities by using sophisticated algorithms to combine data collected from many different people.

The ability to collect information and the computational power to interpret it has enabled great collaboration opportunities and a better understanding of users and customers.

Netflix

Wisdom of the Crowds… “Under the right circumstances, groups are remarkably intelligent,

and are often smarter than the smartest people in them.” “If the process is sound, the more people you involve in solving a

problem, the better the result will be.” A crowd’s collective intelligence will produce better results than

those of a small group of experts if four basic conditions are met.

That “wise crowds” are effective when they’re composed of individuals who… Have diverse opinions; When the individuals aren’t afraid to express their opinions; When there’s diversity in the crowd; and When there’s a way to aggregate all the information and use it in the

decision-making process.

What is collective intelligence? Collective intelligence is an active field of research that predates

the web. Scientists from the fields of sociology, mass behavior, and computer science have made important contributions to this field.

When a group of individuals collaborate or compete with each other, intelligence or behavior that otherwise didn’t exist suddenly emerges; this is commonly known as collective intelligence.

The actions or influence of a few individuals slowly spread across the community until the actions become the norm for the community.

This forms a self-reinforcing feedback loop, commonly known as a network effect, which enables wider adoption of the service.

What is collective intelligence? Although the expression may bring to mind ideas of group

consciousness or supernatural phenomena, when technologists use this phrase they usually mean the combining of behavior, preferences, or ideas of a group of people to create novel insights.

Collective intelligence was, of course, possible before the Internet. You don’t need the Web to collect data from disparate groups of people, combine it, and analyze it. One of the most basic forms of this is a survey or census. Collecting answers from a large group of people lets you draw statistical conclusions about the group that no individual member would have known by themselves. Building new conclusions from independent contributors is really what collective intelligence is all about.

Collective intelligence on the Web Although methods for collective intelligence existed before the

Internet, the ability to collect information from thousands or even millions of people on the Web has opened up many new possibilities.

At all times, people are using the Internet for making purchases, doing research, seeking out entertainment, and building their own web sites.

All of this behavior can be monitored and used to derive information without ever having to interrupt the user’s intentions by asking him questions.

There are a huge number of ways this information can be processed and interpreted.

Collective intelligence of users in essence is The intelligence that’s extracted out from the collective set of

interactions and contributions made by your users. The use of this intelligence to act as a filter for what’s valuable in

your application for a user—This filter takes into account a user’s preferences and interactions to provide relevant information to the user.

To apply collective intelligence in your application. You need to... 1 - Allow users to interact with your site and with each other,

learning about each user through their interactions and contributions.

2 - Aggregate what you learn about your users and their contributions using some useful models.

3 - Leverage those models to recommend relevant content to a user.

What Is Machine Learning? Machine learning is a subfield of artificial intelligence (AI) concerned

with algorithms that allow computers to learn. What this means, in most cases, is that an algorithm is given a set of

data and infers information about the properties of the data—and that information allows it to make predictions about other data that it might see in the future.

This is possible because almost all nonrandom data contains patterns, and these patterns allow the machine to generalize. In order to generalize, it trains a model with what it determines are the important aspects of the data.

Limits of Machine Learning Machine learning is not without its weaknesses. The algorithms vary

in their ability to generalize over large sets of patterns, and a pattern that is unlike any seen by the algorithm before is quite likely to be misinterpreted.

While humans have a vast amount of cultural knowledge and experience to draw upon, as well as a remarkable ability to recognize similar situations when making decisions about new information, machine-learning methods can only generalize based on the data that has already been seen, and even then in a very limited manner.

Real-Life Examples Google, Amazon, eBay, Last.fm, Netflix, Pandora, e muitos outros Prediction markets are also a form of collective intelligence. One of

the most well known of these is the Hollywood Stock Exchange (http://hsx.com), where people trade stocks on movies and movie stars.

Other Uses for Learning Algorithms: Biotechnology - Advances in sequencing and screening technology

have created massive datasets of many different kinds, such as DNA sequences, protein structures, compound screens, and RNA expression. Machine-learning techniques are applied extensively to all of these kinds of data in an effort to find patterns that can increase understanding of biological processes.

Real-Life Examples Financial fraud detection - Credit card companies are constantly

searching for new ways to detect if transactions are fraudulent. To this end, they have employed such techniques as neural networks and inductive logic to verify transactions and catch improper usage.

Machine vision - Interpreting images from a video camera for military or surveillance purposes is an active area of research. Many machine-learning techniques are used to try to automatically detect intruders, identify vehicles, or recognize faces. Particularly interesting is the use of unsupervised techniques like independent component analysis, which finds interesting features in large datasets.

Product marketing - For a very long time, understanding demographics and trends was more of an art form than a science. Recently, the increased ability to collect data from consumers has opened up opportunities for machine-learning techniques such as clustering to better understand the natural divisions that exist in markets and to make better predictions about future trends.

Real-Life Examples Supply chain optimization - Large organizations can save millions of

dollars by having their supply chains run effectively and accurately predict demand for products in different areas. The number of ways in which a supply chain can be constructed is massive, as is the number of factors that can potentially affect demand. Optimization and learning techniques are frequently used to analyze these datasets.

Stock market analysis - Ever since there has been a stock market, people have tried to use mathematics to make more money. As participants have become ever more sophisticated, it has become necessary to analyze larger sets of data and use advanced techniques to detect patterns.

National security - A huge amount of information is collected by government agencies around the world, and the analysis of this data requires computers to detect patterns and associate them with potential threats.

Collaborative Filtering A collaborative filtering algorithm usually works by searching a large

group of people and finding a smaller set with tastes similar to yours. It looks at other things they like and combines them to create a ranked list of suggestions. There are several different ways of deciding which people are similar and combining their choices to make a list. Collecting Preferences Recommending Items Matching Products Item-Based Filtering

Benefits of collective intelligence Applying collective intelligence to your application impacts it in the

following manner:

Higher retention rates —The more users interact with the application, the stickier it gets for them, and the higher the probability that they’ll become repeat visitors.

Greater opportunities to market to the user —The greater the number of interactions, the greater the number of pages visited by the user, which increases the opportunities to market to or communicate with the user.

Higher probability of a user completing a transaction and finding information of interest —The more contextually relevant information that a user finds, the better the chances that he’ll have the information he needs to complete the transaction or find content of interest. This leads to higher click-through and conversion rates for your advertisements.

Boosting search engine rankings —The more users participate and contribute content, the more content is available in your application and indexed by search engines. This could boost your search engine ranking and make it easier for others to find your application.

Harnessing Collective Intelligence to transform from content-centric to user-centric applications Prior to the user-centric revolution, many applications put little

emphasis on the user. These applications, known as content-centric applications, focused on the best way to present the content and were generally static from user to user and from day to day.

User-centric applications leverage Collective Intelligence to fundamentally change how the user interacts with the web application.

User-centric applications make the user the center of the web experience and dynamically reshuffle the content based on what’s known about the user and what the user explicitly asks for.

User-centric applications are composed of the following four components: Core competency —The main reason why a user comes to the

application. Community — Connecting users with other users of interest, social

networking, finding other users who may provide answers to a user’s questions.

Leveraging user-generated content — Incorporating generated content and interactions of users to provide additional content to users.

Building a marketplace — Monetizing the application by product and/or service placements and showing relevant advertisements.

LinkedIn application leverages the four components of user-centric applications: Core competency — Users come to the site to connect with others and

build their professional profiles. Community — Users create connections with other users; connections

are used while looking up people, responding to jobs, and answering questions asked by other users. Other users are automatically recommended as possible connections by the application.

User-generated content — Most of the content at the site is user-generated. This includes the actual professional profiles, the questions asked, the feed of actions—such as a user updating his profile, uploading his photograph, or connecting to someone new.

Marketplace —The application is monetized by means of advertisements, job postings, and a monthly subscription for the power-users of the system, who often are recruiters. The monetization model used is also commonly known asfreemium9— basic services are free and are used by most users, while there’s a charge for premium services that a small minority of users pay for.

Classifying intelligence The three types of intelligence

Explicit information that the user provides in the application. Implicit information that a user provides either inside or outside the

application and is typically in an unstructured format. Intelligence that’s derived by analyzing the aggregate data collected.

This piece of derived intelligence is shown on the upper half of the triangle, as it is based on the information gathered by the other two parts.

Data comes in two forms: Structured data has a well defined form, something that makes it

easily stored and queried on. User ratings, content articles viewed, and items purchased are all examples of structured data.

Unstructured data is typically in the form of raw text. Reviews, discussion forum posts, blog entries, and chat sessions are all examples of unstructured data.

Explicit intelligence REVIEWS AND RECOMMENDATIONS VOTING TAGGING

Implicit intelligence Information relevant to your application may appear in an

unstructured free-form text format through reviews, messages, blogs, and so forth.

A user may express his opinion online, either within your application or outside the application, by writing in his blog or replying to a question in an online community. Thanks to the power of search engines and blog-tracking engines, this information becomes easily available to others and helps to shape their opinions.

You may want to augment your current application by aggregating and mining external data. For example, if your area is real estate applications, you may want to augment your application with additional data harvested from freely available external sites, for example, public records on housing sales, reviews of schools and neighborhoods, and so on.

Derived intelligence DATA AND TEXT MINING CLUSTERING AND PREDICTIVE ANALYSIS INTELLIGENT SEARCH RECOMMENDATION ENGINE

END MODULE 1

Thanks!

Helio Teixeira is founder of novoDialogo{. He is an analyst, digital communication expert,

public speaker and editor of the Chapa Branca Blog.

@helioteixeira+55 82 9901 5090

heliolteixeira@gmail.comhelioteixeira@novodialogo.com.br

http://comunicacaochapabranca.com.brhttp://novodialogo.com.br