defining website comprehensibility from the evaluators perspective

8/14/2019 Defining Website Comprehensibility from the Evaluators Perspective

1/31

Evaluation of Website Comprehensibility

Evaluation of Website Comprehensibility .......................................................... 1

Comprehensibility Project .............................................................................. 4

Defining Comprehensibility .......................................................................... 5

Understanding the Evaluators Expertise .......................................................... 7

Defining the General Public Audience .............................................................. 8

STEM Topics ................................................................................................. 9

Web Site Sources ........................................................................................ 10

Website Pre-qualification for General Public Audience ................................... 11

Site Selection Process ............................................................................... 13

Comprehensibility Rating Process ............................................................... 15

Evaluation of Website Comprehensibility ........................................................ 15

Site Selection .......................................................................................... 15

Review Process ....................................................................................... 15

Review Scaffolding ................................................................................... 16

Review Duration ...................................................................................... 16

Rating Duration ....................................................................................... 16

Training Period ......................................................................................... 16

Comprehensibility Survey Questions ............................................................. 17

Site Classification ..................................................................................... 17

Reading Speed and Linking ....................................................................... 17

Prior Topic or Site Knowledge .................................................................... 17

Concept Recall......................................................................................... 18

Overall Informational or Instructional Rating ............................................... 18

Detailed Rating Method ............................................................................. 18

Information Content Rating .................................................................... 18

Internet Construction Rating ................................................................... 19

Post Evaluation ........................................................................................ 20

Processing Notes ..................................................................................... 20

Rating Matrix ............................................................................................. 21


Ray Garcia, June 2006 Confidential, University of Arizona 1


2/31


Comprehensibility Model Implications ............................................................ 22


Appendix ................................................................................................... 26

Science Taxonomies .................................................................................... 26

Science Content Standards .......................................................................... 26

Science as Inquiry .................................................................................... 26

Physical Science ...................................................................................... 26

Life Science ............................................................................................. 26

Earth and Space Science ........................................................................... 26

Science and Technology ............................................................................ 26

Science in Personal and Social Perspectives ................................................. 26

History and Nature of Science .................................................................... 26

Science Center Exhibits ............................................................................... 27

Bridge of Knowledge ................................................................................ 27

Vivarium ................................................................................................. 27

24-Hour Observatory ................................................................................ 27

Earthworks .............................................................................................. 27

Earth and Planetary Materials .................................................................... 27

Reality Science Exchange .......................................................................... 27

Heath and Wellness .................................................................................. 27

Agriculture and Life Sciences ..................................................................... 27

Library of Congress Science Classification ...................................................... 28

Cybernetics .......................................................................................... 28

Mathematics ......................................................................................... 28

Astronomy ........................................................................................... 28

Physics ................................................................................................ 28

Chemistry ............................................................................................ 28

Geology .............................................................................................. 28

Natural history ...................................................................................... 28

Botany ................................................................................................ 28

Zoology .............................................................................................. 28

Human anatomy ................................................................................... 29



3/31


Physiology ........................................................................................... 29

Microbiology ........................................................................................ 29



4/31


Comprehensibility Project

The University Of Arizona Science Center is conducting a research project with the

UofA Eller School of Management MIS Department to determine whether an

automated process can be used to source and evaluate web site pages for adequate

usage for learning purposes. The learning purposes are focused on self-directed free

choice learners who would use the web as a resource for finding information to

satisfy a learning objective. The automated process will attempt to simulate the

selection process for learning content that has a higher degree of potential

instructional value to the learner due to its ease of access to the main concept of the

information and ease of comprehensibility. The availability of enormous variety of

information on the web presents a challenge to the average person in selecting the

web pages that best suit their learning objectives, skills, and comprehension level.

The research involves the participation of four people who will apply their judgment

in the review and evaluation of web site pages. The evaluation criteria will be tojudge the web pages for comprehensibility of the hypertext information space

presented in the web page. We provide a description of the comprehensibility

criteria to the evaluators prior to conducting the review.

The review process consist of accessing the web site page, finding and reading the

central concept, linking to related pages as necessary to understand the central

concept, and evaluating the web site pages for adequacy for learning or instructional

purposes. The evaluation uses a rating system to rank the web pages relative to the

criteria and relative to other pages reviewed. Approximately eight hundred web site

pages will be reviewed with an average of 10 minutes allocated per web site page.

We are interested in the scanning for the web page site similar to what is typicallydone when browsing the web site when seeking information.

The focus of the website is within the Science, Technology, Engineering, and Math,

(STEM) Domains. The selection of web site pages is based on a typical search

strategy using keywords. This requires the evaluators to have a general interest in

STEM but without any particular expertise in the area. The evaluators would have

the following general profile:

Four people with similar professional profile.

Internet and Media Literate with average daily internet Web Browser usage of

1 - 2 hours or more.

Information literate so that they can discriminate between questionable

information sources and reliable sources.

Adept at selecting internet content for learning activities either for themselves

or for others to use. They should have selected, evaluated, and



5/31


recommended website/pages for others such as the public, parents, friends,

children, peers, etc. that they thought were appropriate.

Active self-directed free choice learners. They have learned something not

taught in school within the last year. They have read a non-fiction book,

visited a museum, watched a documentary, or participated in any free choice

learning activity where they can describe what they learned from the

experience or activity.

Some interest in STEM without any particular expertise in any given area.

They should have internal criteria for what makes a web site suited for

learning that they can express.

We have professional librarians who are not researchers and interface with the

public to select a broad range of appropriate web sites and pages for people to use to

learn about a topic.

The evaluation project was scheduled for July 2006 through August 2006.

The data generated from the project is used to understand what characteristics of a

web page may be more suited for learning insofar as ease of comprehensibility than

other web pages given the same level of interest and motivation in the learning

activity. It may require that we interview the evaluators after the review process to

understand why certain pages where rated higher than others. The specific ratings

provided by each evaluator will not be shared other that to the person providing the

data.

The results will permit the research to more accurately simulate the human

judgment in the selection process and rate pages for comprehensibility moreeffectively. A computational model will be created to simulate the human judgment.

A research paper will be published describing the model and how it was created.

Summary statistics from the rating process and a description of how the data was

generated will be included in the research paper.

Defining Comprehensibility iii

The main interest in comprehensibility is the extraction of the instructional/learning

value of the information content in the web page and the related linked web pages.

The instructional value is a measure of the knowledge contained within the page,

including the degree of application, analysis, synthesis, inquiry or any other narrative

that provokes learning activity. If the information primarily motivates topic interest

in an entertaining manner this is of equal learning value.

Comprehensibility is the degree to which a web page provides direct access to the

substance of the information in the hypertext space without distractions. A

hypertext space is the web page being viewed plus all related linked pages which are

necessary for the reader to visit to understand the information within the related web



6/31


pages. A web page with high comprehensibility will focus on the central concept of

the text narrative and would only use visuals when they are more appropriate for

explaining a concept or audio when it is not spoken language and is relevant to the

concept. A highly comprehensible web page will limit the number of concepts

explained in the text to what can be easily scanned within two page scrolls with each

paragraph building on the concept. The use of the hypertext links are limited to

related concepts that are necessary to understanding the hypertext narrative where

clicking on the link the reader maintains a meaningful context and only optionally is

required to return to the previous page. The reader should easily be able to

determine where they are in the hypertext space relative to where they started.

Comprehensibility encompasses ease of finding and understanding of the concepts

presented assuming that the reading level of the text is equal or less than the

reading level of the evaluator. It does not require that the reader understand all the

text in absolute terms but is able to find meaning and make sense of the text relative

to their framework of understanding or mental map of the topic. If the reader

believes they understood some of the concepts presented this is adequate to

determine the degree of comprehensibility.

Comprehensibility would focus on the main topic with a narrative treatment that is

suited for quick scanning of the web page and uses summary presentations to isolate

the main ideas. The summary may be presented as, the use of short paragraphs

that are easy to scan, use of bullet points, tables, or other presentation that allows

for quick scanning of the information to find the central ideas.

Comprehensibility is evidenced by web page narrative that is suited for short term

memory that allows for ease of linking through and selectively discovering the

meaning relevant to the reader. It is web page narrative that combines both the text

on a single page and the associated pages navigated through links while still

maintaining the context of the related topics.

Comprehensibility of the hypertext space would allow the reader to know how they

navigated through the space and return to any previous page to traverse a different

path. Comprehensibility is enhanced by a site search and site map when a site trail

is not available. Findability of information adds to the comprehensibility.

Comprehensibility is enhanced by consistency and uniformity of the presentation of

the pages on the site. The use of graphical elements only adds to comprehensibility

when it shows information that is more easily presented by the graphical element.The arbitrary use of colors, fonts, background, images distracts from the ease of

reading the text.

The usability and accessibility of the website contribute to comprehensibility. If a

site is not easy to use or cannot be adjusted for accessibility then its

comprehensibility is diminished.



7/31


Comprehensibility puts the readers attention on the most relevant information

quickly and build confidence in the reader that the information has utility for learning

to a degree that the reader will be satisfied that the time spent with the text as

having some value for learning.

Comprehensibility also includes the sense of credibility and trust and access to the

source, author and date of the information being presented. Without trust the

reader is unlikely to try and understand the concept or question it when found.

The opposite of comprehensibility is clutter. Clutter is the presentation of more

than a few related concepts on a single page or the partitioning of the page into

different sections. This is evidenced by Advertising, multiple topics on the same

page, and site navigation links, in areas that present a distraction. Clutter draws the

reader attention away from the central concept of the page or makes it difficult to

find the main idea. Clutter is when the noise on the page is more than 20% of the

page information content. Clutter is when it is difficult to find information and

understand it. Clutter is common on e-commerce and news sites. Clutter includeschatter and ramblings typical of blogs. Clutter includes images that do not add

information to the text or is a concise visual of an idea.

Understanding the Evaluators Expertise

We require a brief professional biography from each evaluator to understand the

level of experience in selecting web content for the general public. The evaluators

should mention any activity that is relevant to the selection of websites such as

participation in social tagging, submitting sites to a virtual library or directory or

search engine, editorial or writing experience, web site development experience, or

any other Internet related experience that would influence their judgment in

selecting websites.

The evaluators will be asked to review this document prior to starting the activity.

They may provide feedback and suggestions to have the document more closely

reflect the criteria that they use in evaluating web site as Librarians.

We may ask the evaluator to optionally participate in a determination of their

Internet competency to establish a comparative level of experience. This may be

through a survey or by an interview process.

The evaluator may optionally participate in determining their preferred cognitive

style using the Cognitive Style Index or simply state whether they believe they are

more analytical or intuitive. The purpose of this information is only to understand

how to interpret the judgment of the ratings.

The ratings are done by the evaluators on a best efforts basis. We would prefer

that the evaluators engage in the rating process in sessions no longer than a few

hours per day to avoid fatigue or skewing of the judgments. If at any time confusion



8/31


over applying the rating criteria occurs we would like to be informed and given the

opportunity to discuss the issue.

The evaluators are allowed to discuss the project with their peers to expand the

professional judgment being applied but only within the first 25 website evaluations.

We want to avoid a shift in the judgment due to outside influences. The first 25

evaluations are considered the training period to practice applying the rating and to

adjust the process or rating if necessary.

Periodic communications during the evaluation process will provide feedback on the

status. Comments provided for particular websites would be discussed to

understand the rating. We are as interested in why a website is disqualified or rated

poorly as we are in why a website is rated appropriate. The rating statistic will not

be discussed with the evaluators during the project to avoid skewing the judgment.

After all the rating is completed we will analyze the data and provide the summarized

results to the evaluators.

We ask the Evaluators to classify the websites, state any prior knowledge, recall the

concepts presented on the website, provide an overall rating, and provide a specific

rating. This should model the selection process that a Librarian uses.

Defining the General Public Audience

The general public is assumed to have an average reading level of the 8th grade.

Reading competency would be sufficient to understand a newspaper or magazine.

The public is assumed to have access to the Internet either at home, work, library,

or public internet kiosk. The content is to be viewed in an Internet Browser and not

a cell phone or other internet enabled device.

Internet and Information Literacyiii is not assumed other than at a very rudimentary

level. The competency level in discriminating selection is assumed to be at a typical

Media Literacy level, which is relatively poor, often biased, and focused on

entertainment. For this audience the Internet may be falsely assumed to be

accurate and not equated to other media outlets. This is the audience that may be

familiar with Yahoo, MSN, AOL and possibly Google. They would use searching

strategies that are limited to single keywords in the search box and may expect that

the results in the first two pages would be satisfactory and valid. They would be

competent enough to use the Internet to find information and extend that activity to

learn about a topic.

The general public is not assumed to have an email account outside of work, buy

from e-commerce sites like Amazon or E-Bay, or use their personal computers for

other purposes. We do not want to bias the study to the top two million frequent

Internet users that are early adopters, know how to create a web page, or work in

STEM, and use computers as a tool. We believe that this population of users may



9/31


also lack appropriate search and information literary skills but they are likely to

overcome any deficiencies through the application of other skills.

Full competency in all the Browser features is not assumed. The typical Internet

user may not be aware of the configuration options for the Browser, parental

controls, history, search options, research, bookmarking, or optional features which

help the user gain control of their browser.

English reading proficiency is presumed for multi-lingual individuals. The websites

selected will be in English and may not have translations or may not be machine

translated without problems. Spanish Language Comprehensibility may be

supported in the future and tools to help English as a Second Language improve

comprehension. These are beyond the scope of this study but we believe that the

findings will be useful for European Languages other than English.

The general public is the audience that is targeted by spamming, phishing, viruses,

chain emails, and other activity that attacks the vulnerable Internet user. These are

the users that may not apply criteria that help them discriminate sources of

information.

Education level or age is not assumed to provide the user with an advantage. Older

professionals who do not need to use the Internet for work may have difficultly in

learning new Internet skills. Younger generations who have a lot of experience with

console games may not be any better at selecting credible and useful content on the

Internet.

Our belief is that the need for Librarians to assist the public in selecting appropriate

sources of information and learning material is even greater with the use of the

Internet across all segments of the population.

STEM Topics

The topics that are included in the evaluation are from Science Technology

Engineering and Mathematics. We have included four taxonomies for consideration,

National Academies Press (NAP), Flaudrau Science Center Exhibit subjects, Library of

Congress, and UNESCO. These taxonomies were chosen for their representative

sampling of STEM topics across a broad range of purposes related to the general

public. NAP publishes books for educators in Science, Engineering, and Medicine.

NAP Science Content Standards is expressed as conceptual phrases for science topics

to cover for each k 12 grade. Flandrau Science Center taxonomy is related to the

types of exhibits that may be designed for the public. The taxonomies from the

Library of Congress and UNESCO represent a view of how STEM information may be

cataloged and used by librarians or researchers. The taxonomies may be used as

sources of keywords used in searching for content. This mimics the expressed

interest by an individual to a librarian who would use the request to determine the



10/31


area to start the search and the associated terms to use. (See appendix for sample

taxonomies.)

Web Site Sources

The sites are selected from Wikipedia, DMOZ.org, and the open web. Wikipediaand DMOZ were chosen due their wide adoption by many search engines as the

encyclopedia and directory sources. While both of these sources are questionable

they reflect the state of the information available on the web and accessed by the

general public. The Wikipedia repository and DMOZ directory are freely available to

any website to reuse therefore much of the content and links appear on many

websites. The repository and directory are managed by volunteers. The volunteers

are not qualified through any certification process. The volunteers are individuals

who have an interest in the topic. They provide text and selections of the websites

within the topic category, accept or reject website submitted by the public, and

summarize a description of the website. The wikipedia entry or directory listing may

have several volunteers working within a topic but generally there are not more than

three volunteer editors and typically only one. The volunteers may post a short

biography of their expertise or interest in the topic but may choose to remain

anonymous. The encyclopedia repository and directory taxonomy of DMOZ have

grown substantially and has many cross links between portions of the websites

organization. The encyclopedia and directory reflects the many ways of organizing

the topics and may appear disorienting at first due its a lack of a consistent audience

perspective and level of competence of the volunteers who created it.

DMOZ represents an organic directory of public websites. It is not as qualified as a

public library directory and therefore would have many more website of questionablevalue. The volunteers do not have consistent criteria for deciding which website is

accepted for inclusion in the directory. DMOZ has enough coverage that most

keyword searches will return a variety of choices of web sites. Similarly, Wikipedia is

organically edited by volunteers without specific editorial control. The quality of

entries varies and the authority of the information cannot be verified. Wikipedia and

DMOZ may be considered the lowest common denominator of web content created

and accessed by the general public. DMOZ has over 100,000 links related to Science

and Wikipedia has over a million pages in its encyclopedia. These are large corpus of

documents that should provide adequate variability in the site content and

construction.

We refer to Wikipedia and DMOZ as sources due to the general public selection of

the content and the links. A public search engine would return many links that

include Wikipedia pages and overlap with DMOZ links as well as content that may be

better or worst than these two sources. The public search engines lack of adequate

filtering of the results is therefore used as the third source of web site pages for

evaluation. A keyword search in a public search engine would return the most



11/31


popular websites based on link structure, not based on information conceptual

relevancy, which results in a lack of adequate filtering necessary to guide the user to

the most qualified source of information. With expert level searching strategy and

persistence it is possible to find quality web sites but this requires competency in

Internet searching and Information Literary neither of which is common within the

general public.

Our site selection goal is to find web site pages that reflect what the general public

would find plus sites that represent the full spectrum of what is available that is

better than the lower common denominator from Wikipedia and DMOZ. We want to

include Web site pages that would be recommended by a Librarian to the general

public interested in a particular topic. We expect that a Librarian may have expert

searching strategies and knowledge of quality sources as well as qualitative

judgment on information that is well beyond the competency of the general public.

We are interested in included web site page that reflect the level of quality that is

representative of a professional Librarian. The Librarians are asked to include web

sites page that may be of poor quality but easy confuse by the general public as a

valid source as well as web site pages that are of the highest information quality.

We prefer to exclude website that may already be included in a virtual library due

to the pre-determine quality of the website. We excluded any sources that are not

freely accessible from any where on the web. Digital libraries and web sites that

require paid subscription are excluded. We assume that the general public could go

to the local library or use the virtual library for their information and instructional

needs. We are interested in helping the general public when they are not accessing

quality directories.

A sample selection of web sites may be presented to the evaluator three times to

test for consistency in the rating judgment. The web sites will be presented within

the normal process and with time delays between the reviews. For example, if a web

site is evaluated on Monday it may appear again on Wednesday. The justification is

to allow enough time in between the reviews, such that the evaluator may not

remember that they already reviewed the site, or recall the ratings they provided

previously.

Website Pre-qualification for General Public Audience

Each selected web site will be pre-qualified for suitability for rating by reviewing the

sites prior to the rating activity and eliminating any sites that are not for the General

Public Audience seeking informational or instructional content. This will allow the

evaluators to focus on only rating web sites that may be of some informational or

instructional value.

The following types of website would be eliminated from the evaluation due to the

purpose of the site not being solely informational or instructional for the general



12/31


13/31


would need to use good sense criteria to evaluate the site owners reputation and

the content contained within the web site.

Collaboration or News site used for communications such as groups, blogs,

forums, polls, open wiki, image galleries, email. These sources are available and

known to many Internet users and would not generally be confused with other

sources of information.

Highly Interactive sites without a clear instructional value, includes flash

introduction pages. These sites are difficult to process and tend to focus on

entertainment media.

Inappropriate web sites, includes all sites that would be blocked from further

review due to safe filtering. These included gaming, pornographic, and other

offensive content.

Unknown due to site error, broken links, slow loading, installation of software or

other problem that hinders access to the information. Any sites that essentiallydo not function are eliminated.

Site Selection Process

The web sites are selected by the research team and the librarians using the

Science Content Taxonomies provided. This is to allow for variety in the site

selections across, prior knowledge, interest, preferences, search strategies, and

sources. The four members of the research team will provide 200 sites. The four

Librarians will provide 600 sites each. This will total 800 sites for evaluation by the

Librarians. Only 25% of the sites being evaluated by any given Librarian would have

been submitted by that Librarian.

Each person may use their own search strategy for the highest quality web sites.

To include variety the researchers may follow a set sequence with the intention of

including both quality and poor site examples for the Evaluators. The search process

is as follows:

1. Go through each second level topic the taxonomy and pick a topic to use as

the keyword and associated concepts in the search.

2. Start with Wikipedia and find any pages that directly relate to the concepts.

3. Navigate through the internal and external links from Wikipedia. (It ispossible for the subject not to be represented in Wikipedia in which case just

go to the next step.)

4. As topically relevant sites are found submit them for rating. Include poor

and quality web sites as they are found.

5. Using DMOZ search for sites related to the topic.



14/31


6. Navigate through the DMOZ directory to find topic relevant sites.

7. As sites are reviewed navigate through any links on the page to find topic

relevant pages

8. Using Google, Gigablast.com, Ask.com or another search engine, with only

the keyword find pages.

9. Use advance searching to find pages and include terms like tutorial, learn

and exclude courses, lesson plans, or other terms as necessary

10.Go to any other known sources like Nasa, AMNH, National Geographic

11.Submit sites as they are found. Include variety in the selection with poor

and quality sites represented. Go to the next topic in the taxonomy.

We do not expect either the researchers or the Librarians to spend much time

searching for web sites for each term. The intention is to make a fast determination

of the website and not the rate them during the selection process. We want a

superficial determination of whether the website should be included for rating based

on the classification of the website previously described.

To store the web sites that are found we are using FURL from Furl.net. This is a

small utility that works with most internet browsers to help bookmark websites of

interest. The reason we selected Furl was due to its export feature which will permit

the researchers to load the data into our rating process. Each researcher should

install and test Furl before the site selection process begins.

All the web site pages will be selected prior to starting the rating process. This is to

allow for adequate representation of the topics and quality in the sites selected. The

site list will be reviewed periodically by the researchers in summary prior to the

evaluation to make recommendations to the Librarians in their choices. We want to

avoid a skewing of data based on any preferences or biases or prior knowledge in a

particular topic.

It is possible that not all topics or levels of quality are represented in the open web.

Web content is often developed to satisfy popular topics therefore science topics that

are of special interest may not be adequately represented.

It may be a common practice for Librarians to use multiple sources for information

to cross check facts as a technique to validate the information source. All the

sources used in this technique should be included for evaluation in the selectionprocess.



15/31


Comprehensibility Rating Process ivvvi

The evaluator will login to the project website and select the URL presented. They

will review the website and evaluate it using the criteria described above. Once they

have read/scan/skim the website to a degree that they are confident in their abilityto make a judgment the evaluator will exit the site and complete a survey form. The

survey will ask some background questions, require that the website be categorized,

and rated for quality. The process is repeated with the next URL.

It is possible for the process to be paused at any point and restarted at a later

time. Once a site is selected the evaluator should complete the survey as a

continuous activity without distraction. The start and end time of the website review

is tracked by the application. The total number of pages visited within the site is

monitored.

Site Selection

The sites are pre-selected and presented in a particular order to each evaluator. No

description of the site is provided prior to it being presented. This removes the web

searching activity and the contextual evaluation of search results. Reviewing the site

in this manner is different from normal behavior since it is not typical for one to be

presented with a web page without some indication of what the web site is about.

The rational for this is to force a complete evaluation of the web site on its own

merits. The evaluator will need to scan the website to find the relevant information

and navigate through the website until they are confident that they understand what

is available and can rate the website content and construction adequately.

Review Process vii

The review process should consider the rating questions but only after the web site

is read/scan/skimmed. It is permissible to scan or skim the contents and not read

the contents if the web site is designed for this type of reading. The evaluator

should use the same process they would use if they had a person ask them to find a

web site on the topic, they found the site through some search process, and they

need to make a quick determination as to the appropriateness of the web site to the

persons goal. The assumption is that the topic is under represented and the website

is not available in a Virtual Public Library. The person may have a goal to either find

information about a topic or have a specific learning objective. The evaluator may

assume that the website being recommended is not the only source they would

provide.



16/31


Review Scaffolding

It is acceptable for the evaluator to use reference sites or dictionaries to

supplement their understanding of the topic. The assumption is that the evaluator is

learning about the Science topics and may need some assistance in specific

vocabulary used within the Science Domain. This activity should only be necessaryfor a few words on the website. If the website has links to help understand specific

concepts this should be positive if the Evaluator needs to find other sources to help

understand the content then this should be accounted for in the rating. If sites are

found that are helpful in the scaffolding process the site should be included for

review.

Review Duration

The evaluation should remain within an average of 10 minutes for each site with

not more than 20 minutes for exceptional sites. If a site is commanding too much

time to comprehend this should be indicated in the rating. It is not necessary tostruggle with poor websites. For websites that are exceptionally interesting it also

not necessary to learn about the topic beyond understanding that the information is

of useful for learning. The websites that appear useful but are unusual or unfamiliar

are the websites that warrant time to understand how to rate them accurately.

Rating Duration

The rating should be done quickly under 5 minutes once the evaluator is familiar

with the rating questions. We are interested in the fast judgment applied in a real

setting where the evaluator is making a recommendation to an individual who is

waiting. This is similar to the behavior that one may have for their own searching

where quick judgments are made based on limited information in the search results

and a review of the first web page seen of the website. The difference is in the

expert judgment being applied using consistent criteria with a focus on adequacy and

appropriateness of the information for learning. The assumption is that web sites

and pages that are comprehensible are easy to read and understand and can be

evaluated quickly.

Training Period

The site selection process will have a training period the validate that content exist

for the range of topics in STEM. The training is also to help the Librarians refine an

effective search strategy and filtering of web sites. The research may review the

initial 25 web sites selected by each Librarian to help establish adequate topic and

quality variety in the selection.

The librarian evaluator will rate 25 to 50 web sites as a trial period to understand

the survey process and level set the criteria. They will pause to allow the



17/31


researchers to review the survey data and make any adjustments to the process.

Depending on the consistency of the rating the data may be discarded.

Comprehensibility Survey Questions

Site Classification

The web site is to be classified before rating. Websites that are not suited to the

general public audience as defined in this document are not to be rated. This step in

the process only validates that the web site belongs in the evaluation data.

Options: dropdown, default is General Public Audience Informational or Instructional Site,

Audience Specific Content, Entity Information Sites, Connector Sites, Content Format non-

HTML, and Sites with Restrictions

Reading Speed and Linking

We are interested in how the site is reviewed. Web sites may be skimmed to

understand the structure, scanned to find specific sections, read to find the main

concept, read for understanding, read for curiosity. Each reading technique is

subject to degrees of comprehensibility and therefore would impact the judgment

during the evaluation. We would like the Librarians to review the web sites on its

own merit and when the web site invokes deeper review to pursue their interest in

the topic within the short time period allotted for each web site. This forces

judgment about what is necessary to review for each web site to make an

evaluation.

The web is not a linear medium therefore clicking on links is common and

necessary to understand the information. The degree to which one links through an

web site page and retraces ones paths is an indicator of the information hypertext

space. The clicking through of links is considered part of the reading process for web

site pages.

Options: dropdown, skimmed to understand the structure, scanned to find specific sections,

read to find the main concept, read for understanding, read for interest

Options: dropdown, did not link through, active browsing of the website

Prior Topic or Site Knowledge

Do you have prior knowledge or interest in the topic of the website with well formed

notions or opinions? Prior knowledge does not include coursework in formal

education or general curiosity in the topic. Stable interest with an intrinsic

motivation to learn the topic where the person actively pursued a learning goal is

what we are interested in identifying.

Options: radio button, True or False; Default: False



18/31


Concept Recall

After reviewing the website read the description and website title and think about

whether this adequately capture the concepts you remember reading. The title and

description should be an adequate representation of what the website or page is

about.

Does the site description and directory category not match the site contents?

Options: radio button, True or False; Default: False

Overall Informational or Instructional Rating

General impression of the website based on what was learned and the presentation

facilitating the finding and reading of information and its suitability for the general

public. Is the information easy to understand, useful, and credible? This is a

judgment using intuition based on the content and appearance of the website.

Would the evaluator recommend this site as a source of informational or instructionalcontent to the public?

Options: radio button

1) 1 Poor, 2 Low, 3 Avg, 4 High, 5 Best

Detailed Rating Method

The rating method uses a five point scale with each point contributing to the value

of the overall rating. The scale is from one to five with one indicating that the

specific criteria do not add value and five indicating that it adds maximum value.

These are relative values based on judgment across the sites reviewed.

The ratings are normalized for the evaluator across all the web sites. The

normalized rating is adjusted using the evaluators weighted criteria then compared

to the other evaluators.

Options: radio button

1 Poor, 2 Low, 3 Avg, 4 High, 5 Best

Information Content Rating

Information Value (text content)

Is it easy to find and understand the main topic and concepts in the text? Is it

readable within the comprehension level of the general public? Are the web pages

informative?

Does the website adequately cover the topic? Does it have sufficient useful

information? Is the background and context for the topic provided?



19/31


Is the audience clear? Is it relevant to the audience? Is the information scope

adequate? Do the links help understanding the information? Are terms defined?

Information Credibility (credibility)

Does the content build confidence in its validity (expertise) and can be verified

(trustworthiness)? Does the content appear to have a editorial control?

Is the Authority of information known? Are the sources credible? Are the sources

listed and obtainable? Is the site owner available? Is the author expertise and

credentials listed? Can they be contacted? Is there a discussion of the topic or

opinions of the quality or a rating shown by peer review? Is there accountability for

the information?

Is the information objective, valid, and accurate? Is the information current? Is

the information free of biases or are the biases easy to determine?

Are there any broken links? Is this a small website that looks amateurish? Does it

link to other websites where the information does not look credible? Are their

typographical errors? Is it hosted by a 3rd party such as Geocities, Yahoo, or AOL?

Media Instructional Value (visual design clarity)

Does the use of graphics, icons, animation, or audio enhance the clarity of the

information? Is it necessary to communicate the concepts?

Is the site free of visual clutter? Is the site free of advertisements and unrelated

information? The site does not include popup ads or techniques that distract the

reader. Are the advertisements placed away from the information so that they are

easy to ignore?

Internet Construction Rating

Affective Attention (overall appeal)

Does the website make it easy to focus attention on the most relevant information?

Does the website look professionally designed?

Are the fonts, sizes, colors, background and sectioning of the site easy to read? Is

the content designed to be printed or have an option to have a printed version of the

same content?Do the format, appearance, and aesthetic of the website enhance the information?

Is the web site organization uniform, consistent, and familiar? Is the page heading

and summary representative of the information on the web site?



20/31


Organization (structure, navigation)

Is the content presented for reading on the web? Does it make appropriate use of

short paragraphs, list, tables, headings, and links, to focus easily on the main ideas?

Is it easy to scan or skim?

Are the links related to the topic and make it easier to understand the information?Are the links clearly organized? Are the navigational links separated from the links

within the narrative? Are the links to external sites relevant to the topic and clearly

indicated? Are links to media files indicated?

Usability (functionality, interactivity)

Do the Web site pages load quickly? Is the site easy to use? Does it have a search

feature and site map or menu that is organized? Is it easy to find the home page?

Does it provide an indicator of where the web page is relative to the site navigation?

Is the back button feature operable?

Does the site adequately support usability and accessibility?

If the website is highly interactive does the activity enhance the informational or

instructional value of the website? Is the interactivity necessary or the easiest

means of communicating the concept?

Post Evaluation

Comment on the evaluation process if necessary to explain an exceptional rating

and characteristic that was not included in the rating questions but was an important

factor in the evaluator criteria.

A calculation of the rating may be done and compared to the initial general

evaluation. If an inconsistency is found a re-evaluation may be done to correct for

the judgment. If some factor accounted for the discrepancy then a comment would

need to be provided to explain the difference.

Processing Notes

The survey time should be less than the time used to review the website. The

evaluation survey facilitates the rating process by displaying criteria questions the

evaluator should think about before providing the rating.

The evaluator is assumed to have an adequate comprehension level to review

general web pages on Science and Technology, and enough interest in the subject

matter to be motivated to find the central concept and attempt to understand it.



21/31


Rating Matrix

The rating matrix established the internal criteria of each factor considered in the

overall rating. It indicates when a factor is dependent on others or if it may be

considered independently. The relative value of each factor is proposed. This matrixis used to weight the ratings against each other to emphasize one factor more than

another.

Each rating indicates the following:

D = Dependent

I= Independent

L = Familiarity may overcome hurdle, can learn to ignore factor

C = Constant Factor no matter how often it is reviewed

< > = Relative importance between the factors

IV IC MV AA SO UA

Information Content Rating

Information Value IV I C < > > < >

Information Credibility IC I C > > > >

Media Instructional Value MV I C >

defining website comprehensibility from the evaluators perspective

Documents