an empirical study of the effect of coherent and tailored document delivery as an interface to...
TRANSCRIPT
![Page 1: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/1.jpg)
An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to
Organizational Websites
Cécile Paris
Anne-Marie Vercoustre Stephen Wan PeterWilkins
Ross Wilkinson Mingfang Wu
![Page 2: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/2.jpg)
Outline
• Problem statement• System Description• Experiment• Discussion
![Page 3: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/3.jpg)
Finding Information about a Corporation
• Often done through the web, searching and browsing
• Differences between results from such search-and-browse and printed brochures:– Printed brochures are coherent documents – and include
more than just information about a specific topic
– Printed brochures are typically tailored to an audience
![Page 4: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/4.jpg)
Challenges in accessing an organization’s website
• Browse– know the organization’s structure
– know the web data structure
– otherwise -> users feel lost
• Search– It works for the “right” query or someone with the “right” mental
model how querying/searching works
– Otherwise, it may cause zero results or too many results
– Result lists is disorganized - each result is presented independently, although each result may be a piece of the overall picture
![Page 5: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/5.jpg)
Our research question and approach
Can a tailored hypermedia system offering tailored and coherent information provide a good alternative to search-and-browse?
• Approach: Given a modest user model and a query:– Present information that not only matches the query but also is tailored
to the user.
– Present other information that is relevant to the context of the query (here information about an organization), like printed brochures do (e.g., introduction, contact details, etc.)
– Deliver the result as a concise and readable summary.
– Note that the summary may not cover all information a user needs, but is an informative guide that will lead the user to the needed information.
![Page 6: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/6.jpg)
Our Tailor Hypermedia System: Percy
Dat
a In
terf
ace
User Model
Stereotype
Individual
Virtual Document Planner
DiscourseRules
Content Planner
DataSources
‘Documents’
VD
P D
ata
Inte
rfac
e
Inte
rfac
e S
peci
ficat
ion
Grammar(s)Lexicon(s)TemplatesStylesheets
SurfaceRealiser
PresentationRules
PresentationPlanner
![Page 7: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/7.jpg)
Percy – user model
• Based on a requirements analysis• Contains:
– Stereotypical information• Current job position (e.g., scientist or CEO)
• Industry sector (e.g. science, engineering, finance)
– Individual information• Interests (as represented by a query term, e.g., language
technology)
• Preferred delivery channel (e.g web page, or hand-held device)
![Page 8: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/8.jpg)
Percy – discourse planner
• Discourse rules or Schemas– They indicate which information to
present and how to organize it– They are used by the discourse
planner to build a discourse tree– Each rule extends a branch in the discourse tree– Coherence relations amongst
siblings indicate how the information fits together
![Page 9: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/9.jpg)
Percy – presentation planner
• Discourse tree (which indicates content and organization) is extended with presentation markup
• Uses presentation plans
PPalm
C
PWeb
C
![Page 10: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/10.jpg)
Percy – surface generator
• Traversal of final discourse tree to generate document
• Conversion from tree markup of layout to device dependent markup– Device dependent markup may
include use of further style sheetsSurface Realisation
XSLT templates
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
PWeb
C
![Page 11: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/11.jpg)
PERCY – an example delivery to a scientistIntroduction to CSIRO
Short introduction to the divisions that have the projects
related to the query
Show the expanded query
Short description of the
matched project
![Page 12: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/12.jpg)
PERCY – an example delivery to a scientist Link to the detailed description
Link to the onlinedemonstration
Publications
Contact person
![Page 13: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/13.jpg)
PERCY – an example delivery to a CEO
Short introduction of how to do business with CSIRO
![Page 14: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/14.jpg)
PERCY – an example delivery to a CEO
The project description and business relevance
![Page 15: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/15.jpg)
Preliminary Evaluation
• Question Is this coherent and tailored delivery is useful interface to
organizational websites ?
![Page 16: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/16.jpg)
Possible evaluation criteria
• User’s interactive experience– format
– content
• User’s attitudes towards the received message– intention
• The influence of the delivery message– The further interaction with the corporate
![Page 17: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/17.jpg)
Experimental setting
• Test systems: PERCY vs. search-and-browse mechanism: a website search engine (Panoptic)
• Subjects: 20 university students• Search topics:
– T1: Human computer interaction
– T2: Mathematical modeling
– T3: Image analysis
– T4: Language technology
![Page 18: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/18.jpg)
Evaluation instruments
• Post-system questionnaire (PSQ)– To get users’ opinion on the evaluation criteria
independently on each test system
• Exit questionnaire (EQ)– To get users’ opinion of the second system compared with
the first system
• All questions are evaluated on a 5-point Likert scale
![Page 19: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/19.jpg)
Experiment design
• Latin square design to reduce the sequence effect of systems
Subjects System, topic
No. 1-5 Percy(t1,t2) PSQ Search (t3,t4) PSQ EQ
No. 6-10 Percy (t3,t4) PSQ Search (t1,t2) PSQ EQ
No. 11-15 Search(t1,t2) PSQ Percy (t3,t4) PSQ EQ
No. 16-20 Search (t3,t4) PSQ Percy (t1,t2) PSQ EQ
![Page 20: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/20.jpg)
Post-system questionnaire – Measuring the content of the delivered information
• Q1: The system provides me accurate information
• Q2: The system provides sufficient information
• Q3: The information provided by the system meets my need.
• Q4: The system provides me comprehensive information
Q1 Q2 Q3 Q4
Search 3.50 3.60 3.35 3.70
Percy 3.65 3.30 3.65 3.55
![Page 21: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/21.jpg)
Post-system questionnaire – Measuring the format of the delivered information
• Q5: The structure of the presented information is clear to me.
• Q6: I think the presented information is organized in a useful format.
• Q7: I think the presented information serves well as a useful online brochure.
Q5 Q6 Q7
Search 3.65 3.50 2.75
Percy 4.00 3.80 3.50
![Page 22: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/22.jpg)
Post-system questionnaire – Understanding the intention of the user
• Q8: I would like to get more information on some specific projects presented.
• Q9: I would like to get more information because I have not got the needed information.
Q8 Q9
Search 3.95 3.15
Percy 3.90 2.90
![Page 23: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/23.jpg)
Exit-questionnaire – Evaluating users’ preferences
Q1 Q2 Q3 Q4 Q5 Q6 Q7
Search 2.70 3.00 3.10 2.80 3.30 3.50 2.70
Percy 3.70 3.60 3.50 4.20 4.20 3.40 4.00
P < 0.01 0.03 0.04
focus o
n the s
earch
topic bett
er
make s
earch
task ea
sier
deliver
higher quali
ty info
attrac
t atte
ntion bett
er
provide bett
er ex
planati
on
provide more
compreh
ensiv
e info
prefere
nce
![Page 24: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/24.jpg)
Summary of experimental results
• With Percy, subjects can better focus on their tasks.• With Percy, subjects understand better why a piece
of information is presented.• Subjects preferred Percy.
![Page 25: An Empirical Study of the Effect of Coherent and Tailored Document Delivery as an Interface to Organizational Websites Cécile Paris Anne-Marie VercoustreStephen](https://reader036.vdocuments.us/reader036/viewer/2022062621/551c017f550346a34f8b4c42/html5/thumbnails/25.jpg)
Discussion
• About tailored hypermedia(anything to say about the delivery method itself or limitations?)
– Delivery – who is in charge, user/author? or it is situation dependent? – How to apply AH to a dynamic resource (or even a different domain)
with minimum effort?– Is user’s expectation of AH system different from IR?
• About the evaluation– What to compare – Is it fair to compare the adaptive system with the
non-adaptive system? Or should we compare two adaptive systems? – Will the preference brings the enhancement of user performance? – (How to measure long term benefit?)