Evaluation Methods for Social Systems

Joan DiMiccoIBM Research

Center for Social Software

IBM’s Center for Social Software

• Cambridge, MA• Within IBM Research– social software– Collaboration– enterprise communication– social visualization

• I manage the Visual Communication Lab

Background

• We build social communities• Measure success of these communities• Our challenge as researchers:

quantifying essentially qualitative experiences• Overview of 5 different analysis approaches

Social Software Community

• We built a social network site for IBM employees

• How do we get people to join and then continue to use?– Reputation, skills, your merit– Photos– Commenting / sharing – Incentive systems (e.g. points)

and honorable roles

Broad summaries from interview data

• Asked of us: What is the ROI on social networking?• We asked: Why are people using our SNS site?• Focused subject interviews on this question (~20 ppl)– Coded each interview statement based on the type of

motivation– Themes emerged: 1. social/personal connecting, 2. career

climbing, and 3. spreading marketing message of pet projects

– Then looked content on site for alignment with these themes

JM DiMicco, DR Millen, W Geyer, C Dugan, B Brownholtz, M Muller. (2008) “Motivations for Social Networking at Work.” Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW 2008). `

Using surveys to measure success• Question: What impact does our SNS have on

IBM? (similar to ROI question)• We used “social capital” as a concept to measure

how employees have more access to information and expertise– Through existing survey instruments, we measured

social capital of employees. – Based on their reported behavior on the SNS, we

correlated (regression analysis) behavior on site with greater social capital.

C Steinfeld, JM DiMicco, N Ellison, C Lampe. (2009) “Bowling Online: Social Networking and Social Capital within the Organization. Fourth International Conference on Communities and Technologies (C&T 2009).

Measuring an intervention

• We deployed as new feature on the SNS (rewarding points for content). How do we measure impact? – Controlled the deployment: only half of people

saw the feature– Compared the two groups– Fully instrumented the system to observe

behavior

R Farzan, JM DiMicco, DR Millen, B Brownholtz, W Geyer, C Dugan. (2008) “Results from Deploying a Participation Incentive Mechanism within the Enterprise.” Proceedings of CHI 2008.

After 3 Months of Points

0

500

1,000

1,500

2,000

2,500

3,000

PHOTOS LISTS COMMENTS

Am

ou

nt

of

Co

nte

nt

Points Group

Control Group

Using a survey to gather ground truth

• Does your behavior on the SNS reflect your feelings about the people?

• We designed a survey “game” asking “How strong is your relationship with these people?”

• Used this ground truth to compare with behavior on site– Regression analysis demonstrated correlations

between closeness and certain behaviors

A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

Using interviews to inform quantitative analysis

• A website for viewing legislation: how are people using it?

• Logs of which features were used and how long each user stayed– Determined few power users, many casual users– Interviewed 6 power users to answer our

questions raised during our log analysis of “why?”

A Wu, JM DiMicco, DR Millen. (2010) “Detecting Professional versus Personal Closeness Using an Enterprise Social Network Site.” Proceedings of CHI 2010.

Where from here?

• What is your question to answer?• How can you measure it? – Can you measure it by proxy? – Do you have the data or do you need to collect? • Surveys• System logging• Interviews• Controlled study