two studies of opportunistic programming: interleaving web foraging, learning, and writing code
DESCRIPTION
Two Studies of Opportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code. Joel Brandt with Philip Guo (S), Joel Leweinstein (S), Mira Dontcheva (A), Scott Klemmer (S), . http:// hci.stanford.edu /opportunistic. - PowerPoint PPT PresentationTRANSCRIPT
Stanford HCI Group Adobe Advanced Technology Labs
Two Studies ofOpportunistic Programming: Interleaving Web Foraging, Learning, and Writing Code
Joel Brandtwith Philip Guo (S), Joel Leweinstein (S), Mira Dontcheva (A), Scott
Klemmer (S),
http://hci.stanford.edu/opportunistic
2
“good grief, I don’t even
remember the syntax for
forms!”
3
4
5
6
7
8
9
10
11
12
13
14
15
The Web has fundamentally changed the cost structure
of information access.
Enables an opportunistic
approach to programming
16
People program opportunistically to prototype, ideate, and discover
Emphasize speed and ease of development over code robustness
and maintainability
17
http://flickr.com/photos/royalsapien/2387707860/
Designers…
18
http://flickr.com/photos/niosh/2492023651/
Scientists…
19
http://www.flickr.com/photos/samthor/3198975900/
Makers…
20
By 2012:13 million who program as part of their job…
…but only 3 million who are
professional programmers
[Scaffidi 2005]
21
How do programmers use online resources?
22
Related WorkWeb Search for Programmers
[Stylos 06, Hoffmann 07]
Code Cloning in Software Engineering[Kim 04, Ducasse 99]
Learning Barriers of Programming[Ko 04]
Learning on the Web[Torrey 09]
23
Study 1: Prototyping a
Web-based chat room
Study 2: Log Analysis of Programmers’
Web Queries
24
Study 1: Web Chat Room 20 participants (9 Graduate, 11
Undergrad) “basic knowledge” of PHP, JS, and
HTML 2.5 hours each use any resources 3 researchers observed,
one asked questions to prompt think aloud
METHOD
25
Chat Room Specifications
1. Set username
2. Post messages
3. Update without page reload
4. Messages have timestamp
5. Limited chat history
METHOD
26
27
Most Subjects Met All Specs
� � � �� � � � �� � �� � � � �� � � � �� � � � �� � � � �� � � �� � � � �� � � � �� � � � �� � � �� � � � �� � � � �� � � �� � � � �� � � � �� � � � �� � � � �� � � � �
user
nam
epo
st m
essa
geAJ
AX u
pdat
etim
esta
mp
hist
ory
� Specification Met
� Specification Attempted
subj
ects
RESULTS
28
Frequent Web Usesu
bjec
ts
time (minutes)
RESULTS
29
Frequent Web Usesu
bjec
ts
time (minutes)
RESULTS
30
Frequent Web Usesu
bjec
ts
time (minutes)
RESULTS
31
Frequent Web Usesu
bjec
ts
time (minutes)
RESULTS
32
The Web plays multiple roles
RESULTS
leng
th (s
econ
ds)
session (sorted by length)
33
The Web plays multiple roles
RESULTS
leng
th (s
econ
ds)
session (sorted by length)
34
The Web plays multiple roles
RESULTS
leng
th (s
econ
ds)
session (sorted by length)
35
Three Intentions Behind Web UseLearning
just-in-time acquisition of new skills
Clarification connecting high-level knowledge to implementation details
Reminder offloading memory to external resources
RESULTS
36
Just-in-Time Learning-by-DoingCopy and modify code before reading prose
RESULTS
37
“There’s some stuff in this code that I don’t really know
what it’s doing, but I’ll just try it and
see what happens.”
38
Just-in-Time Learning-by-DoingCopy and modify code before reading prose
Not concerned with deep learning
RESULTS
39
“I think it’s less expensive for me to just take the first code I find and see how helpful it is at … a very high level … as
opposed to just reading all these
descriptions and text”
40
Just-in-Time Learning-by-DoingCopy and modify code before reading prose
Not concerned with deep learning
Learn new terminology from result snippets
RESULTS
41
Example Query
Terms
TimeNumber of
Refinements
Number of Result Clicks
Types of pages visited
Lines of code
copied
42
Learning
Example Query
Terms
TimeNumber of
Refinements
Number of Result Clicks
Types of pages visited
Lines of code
copied
43
Learning
Example Query ajax tutorial
Terms
TimeNumber of
Refinements
Number of Result Clicks
Types of pages visited
Lines of code
copied
44
Learning
Example Query ajax tutorial
Terms English
TimeNumber of
Refinements
Number of Result Clicks
Types of pages visited
Lines of code
copied
45
Learning
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
sNumber of
Result Clicks
Types of pages visited
Lines of code
copied
46
Learning
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
Types of pages visited
Lines of code
copied
47
Learning
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
Lines of code
copied
48
Learning
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copied
49
Learning
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
50
Clarification of Existing Knowledge
Web search enables translations and “language analogies”
Know exactly what code does when they see it
Copied code often not immediately tested
RESULTS
51
Learning
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
52
Learning Clarification
Example Query ajax tutorial
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
53
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
54
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
55
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
56
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
57
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
Lines of code
copieddozens
58
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens
59
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
60
Reminding about forgotten details Explicitly choose not to learn
things Search serves as “Web-
enabled auto-complete”
RESULTS
61
62
63
Reminding about forgotten details Explicitly choose not to learn
things Search serves as “Web-
enabled auto-complete” Despite utility, can break flow
RESULTS
64
Learning Clarification
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
65
Learning Clarification Reminder
Example Query ajax tutorial javascript
thread
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
66
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
67
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
68
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
less than 1 minute
Number of Refinement
s2-3 0-1
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
69
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
less than 1 minute
Number of Refinement
s2-3 0-1 0
Number of Result Clicks
5+ 1-2Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
70
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
less than 1 minute
Number of Refinement
s2-3 0-1 0
Number of Result Clicks
5+ 1-2 0-1Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
Lines of code
copieddozens several
71
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
less than 1 minute
Number of Refinement
s2-3 0-1 0
Number of Result Clicks
5+ 1-2 0-1Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
API docs,result
snippetsLines of
code copied
dozens several
72
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
less than 1 minute
Number of Refinement
s2-3 0-1 0
Number of Result Clicks
5+ 1-2 0-1Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
API docs,result
snippetsLines of
code copied
dozens several none to several
73
How do these results generalize?
74
Study 2: Query Log Analysis Adobe Community Help logs from July
2008 ~25,000 programmers performing
101,289 queries about Adobe Flex Framework
Grouped queries into 65,955 search sessions
Analyzed three properties of sessions: Format of query strings Process of query refinement Types of Web pages visited
Hand-coded 300 sessions for intent
METHOD
75
Query format predicts types of pages visited
76
Query format predicts types of pages visited
Page type code only English + code English only
Adobe APIs 38% 16% 10%Adobe tutorials 31% 33% 39%
blogs 15% 22% 19%forums 3% 7% 6%
unclassified 13% 22% 27%All inter-row differences significant at p < 0.001
RESULTS
77
Query format predicts types of pages visited
Page type code only English + code English only
Adobe APIs 38% 16% 10%Adobe tutorials 31% 33% 39%
blogs 15% 22% 19%forums 3% 7% 6%
unclassified 13% 22% 27%
RESULTS
All inter-row differences significant at p < 0.001
78
Query format predicts types of pages visited
Page type code only English + code English only
Adobe APIs 38% 16% 10%Adobe tutorials 31% 33% 39%
blogs 15% 22% 19%forums 3% 7% 6%
unclassified 13% 22% 27%
RESULTS
All inter-row differences significant at p < 0.001
79
Query format predicts types of pages visited
Page type code only English + code English only
Adobe APIs 38% 16% 10%Adobe tutorials 31% 33% 39%
blogs 15% 22% 19%forums 3% 7% 6%
unclassified 13% 22% 27%
RESULTS
All inter-row differences significant at p < 0.001
80
Reminder sessions are typically code-only queries
81
Query format indicative of intention
query format reminding learning
code only 56% 21%English + code 10% 29%
English only 34% 50%
RESULTS
All inter-row differences significant at p < 0.05
82
Programmers rarely need to refine queries
83
Programmers refine rarely
RESULTS
num
ber
of q
uerie
s
query number in session
84
…and they start out with either words or code
RESULTS
perc
ent o
f que
ries
query number in session
85
Five Key Insightsfor Tool Design
86
Just-in-time learning is common
Create tutorials that teach “how the code works”
87
Copied code often not immediately tested
Demarcate copied code,highlight what is executed
88
Web search often used as a “translator”
Improve search ranking with synonyms,
perform searches automatically
89
Programmers choose not to learn syntax
Make Web-enabled auto-complete
90
Query refinement is often predictable
Automatically augment queries
91
Preview of Ongoing Work
BlueprintEnabling Example-Centric Programming
92
Stanford HCI Group Adobe Advanced Technology Labs
http://hci.stanford.edu/opportunistic
95
96
Learning Clarification Reminder
Example Query ajax tutorial javascript
threadfetch_array
php
Terms English mix of code and English mostly code
Time 10s of minutes
about 1 minute
less than 1 minute
Number of Refinement
s2-3 0-1 0
Number of Result Clicks
5+ 1-2 0-1Types of
pages visited
tutorials, how-to articles
API docs, blogs, forums
API docs,result
snippetsLines of
code copied
dozens several none to several