perception and reality:

Perception and Reality:Creating a Baseline for Longitudinal Study of Information Literacy Skills at the Outset of a

High School Laptop Program

Roberta Jordan, PresenterGlickman Library, University of Southern Maine

December 3, 2010

What’s the premise?

PERCEPTION V. REALITY

Perception:

I am able to use the terms AND and OR correctly in my

searching.

Check one:

_____ DO NOT UNDERSTAND _____ NEVER_____ SOMETIMES_____ FREQUENTLY_____ ALWAYS

62% of surveyed students expressed high confidence (i.e. answered “frequently” or “always”) in their ability to use Boolean operators in an information search.

Reality:

Which of the following would retrieve the MOST results in an on-line search?

a. Stars NOT planetsb. Stars OR planetsc. Stars AND planetsd. Stars INSTEAD OF planetse. Don’t know.

Just under 13% of students who took a follow-up information literacy “skills” test chose the correct answer (b) to the multiple choice question on Boolean operators.

Perception v. Reality = BASELINE

The difference between the two numbers (62 versus 13 percent) constitutes a realistic baseline against which to measure gains in student achievement.

PROJECT BACKGROUND

My Interests• Information Literacy/Skills Testing as a means

to inform instruction• Real-world application for capstone project for

my M.L.S.

November 2009: 700 students in grades 9 through 12 at Morse High School received laptops for school and at-home use through the Maine Laptop Technology Initiative (MLTI). Approximately 65 staff members also received laptops the year before.

Commitment by R.S.U. 1: $750,000+ over four years

Goal: Be a successful participant in a pilot program in selected high schools; support attainment of Maine Learning Results, and “help prepare students for a future economy that will rely heavily on technology and innovation.” (Task Force on Maine’s Learning Technology Endowment , 2001, p. vi).

An Opportunity

The Challenge

-Document student skill levels at the program start; -Demonstrate increased student achievement attributable to laptop use (the “holy grail” of research efforts on 1:1 laptop use) to help justify the community investment.

Project Purpose

Develop tools for collecting baseline data on information literacy skills of Morse High School students at the outset of 1:1 laptop initiative.

Why information literacy is a potentially good curricular area for demonstrating achievement:

-a subject that integrates use of technology and other important academic skills (research, synthesis);

-ample evidence suggesting student skills are lacking in this area;

-there are widely accepted standards that guide instruction against which to measure improvement;

-there are cost-effective ways to collect data.

Two Tools Were Developed Over the Course of the Project:

1. A “Perceptions” Survey-intended to provide a snapshot of student and teacher confidence in their ability to integrate technology, research, and information-seeking skills

2. An Information Literacy “Skills” Test-intended to measure how much students’ perceived skills matched up in reality.

PERCEPTIONS SURVEY

Based on the work of Morrisette (2007) and Grimble and Williams (2004), the MHS perceptions survey asks students to express their confidence levels in five areas relating to information-seeking and technology.

Point Access SkillsResearch SkillsDatabase SkillsInternet SkillsTechnology Skills

POINT ACCESSSKILLS

DO NOTUNDERSTAND

NEVER SOMETIMES FREQUENTLY ALWAYS

P1. I am able to access our high school library web site using a computer.

P2. I feel competent using the on-line library catalog to search for books.

P3. I know how to locate books on the shelves after looking them up on the on-line catalog.

P4. I am able to locate and use the databases available on our high school library website using the blue sheet.

P5. I am able to search for books that are in other Maine libraries using MaineCAT.

RESEARCH SKILLS DO NOT UNDERSTAND


R1. I am able to develop questions focusing on my information needs.

R2. I know what a keyword search is, and can think of keywords for a particular information need.

R3. I am able to use several keyword terms to focus my information search.

R4. I know how to cite a source in a bibliography and a footnote (or a parenthetical reference) using the pink sheets.

R5. I understand what plagiarism is and follow copyright principles.

R6. I am able to tell the difference between a primary and secondary source.

DATABASES DO NOT UNDERSTAND


D1. I know when to use MARVEL for my research needs.

D2. I am able to perform a successful search in Gale’s Student Resource Center Gold.

D3. I am able to perform a successful search in Infotrac.

D4. I am able to perform a successful search in SIRS Online.

D5. I know what an advanced search is, and am able to perform one successfully.

D6. I am able to use the terms AND and OR correctly in my searching.

THE INTERNET DO NOT UNDERSTAND


I1. I generally use more than one search engine when doing research.

I2. I know how to narrow a search when using search engines.

I3. I know if a website is valid or might be biased.

I4. I know when to use the Internet and when to use other information sources.

TECHNOLOGY BEGINNER INTERMEDIATE ADVANCED DON’TKNOWHOW

T1. I know how to create a Microsoft PowerPoint or Keynote presentation.

T2. I know how to create a web page.

T3. I know how to make a spreadsheet.

T4. I know how to operate a video camera.

T5. I know how to operate and upload pictures from a digital camera.

T6. I know how to show a computer display through a digital projector (LCD).

T7. I know how to create and insert special graphics, tables, and pictures in my documents.

ACTIVITY REGULARLY OFTEN SOMETIMES NEVER

E-MAIL

FACEBOOK(or similar social networking, such as MySpace, Twitter)

NEWS

WEATHER

DOWNLOADING MUSIC

TO PLAY GAMES

INFORMATION/RESEARCHUSING SEARCH ENGINES

INFORMATION/RESEARCH USING DATABASES

SPORTS (including fantasy leagues and Heal point standings)

SHOPPING

VIDEO (YouTube, Hulu, DVDs, etc.)

VIDEO CHAT

ON-LINE BANKING

Student Participants:

Total Number Completing Survey: 299Male Respondents: 133 Female Respondents: 166Totals by Class:

Grade 9: 68 (36M, 32F)Grade 10: 84 (39M, 45F) Grade 11: 80 (30M, 50F) Grade 12: 67 (28M, 39F )

Faculty Participants:

Total Number of Respondents: 23Male Respondents: 10Female Respondents: 13

STUDENTRESPONSES(out of 299)

FACULTY RESPONSES(out of 23)

POINT ACCESS SKILLS Total Percent Total Percent

P1. I am able to access our high school library web site using a computer. 147 49.2% 13 56.5%

P2. I feel competent using the on-line library catalog to search for books. 123 41.1% 14 60.9%

P3. I know how to locate books on the shelves after looking them up on the on-line catalog. 159 53.2% 17 73.9%

P4. I am able to locate and use the databases on our high school library website using the blue sheet. 127 42.5% 11 47.8%

P5. I am able to search for books that are in other Maine libraries using MaineCAT. 85 28.4% 12 52.2%

AGGREGATE 641 42.9% 67 58.3%

RESEARCH SKILLS

R1. I am able to develop questions focusing on my information needs. 191 63.9% 21 91.3%

R2. I know what a keyword search is, and can think of keywords for a particular information need. 227 75.9% 22 95.7%

R3. I am able to use several keyword terms to focus my information search. 214 71.6% 21 91.3%

R4. I know how to cite a source in a bibliography and a footnote (or a parenthetical reference) using the pink sheets. 175 58.5% 18 78.3%

R5. I understand what plagiarism is and follow copyright principles. 246 82.3% 22 95.7%

R6. I am able to tell the difference between a primary and secondary source. 182 60.9% 21 91.3%

AGGREGATE 1235 68.8% 125 90.6%

Table 3. Frequency of Student and Faculty Survey Responses “Frequently” plus “Always” and Frequency of Student and Faculty Skill Assessments “Intermediate” plus “Advanced”

DATABASESD1. I know when to use MARVEL for my research needs. 91 30.4% 11 47.8%D2. I am able to perform a successful search in Student Resource Center Gold. 109 36.5% 10 43.5%D3. I am able to perform a successful search in Infotrac. 66 22.1% 10 43.5%D4. I am able to perform a successful search in SIRS Online. 84 28.1% 12 52.2%D5. I know what an advanced search is, and am able to perform one successfully. 148 49.5% 13 56.5%D6. I am able to use the terms AND and OR correctly in my searching. 165 55.2% 15 65.2%

AGGREGATE 663 37.0% 71 51.4%

THE INTERNET

I1. I generally use more than one search engine when doing research. 137 45.8% 14 60.9%

I2. I know how to narrow a search when using search engines. 186 62.2% 17 73.9%

I3. I know if a website is valid or might be biased. 174 58.2% 19 82.6%

I4. I know when to use the Internet and when to use other information sources. 201 67.2% 21 91.3%

AGGREGATE 698 58.4% 71 77.2%

TECHNOLOGYT1. I know how to create a Microsoft PowerPoint or Keynote presentation. 246 82.3% 17 73.9%T2. I know how to create a web page. 114 38.1% 6 26.1%T3. I know how to make a spreadsheet. 198 66.2% 11 47.8%T4. I know how to operate a video camera. 247 82.6% 17 73.9%T5. I know how to operate and upload pictures from a digital camera. 255` 85.3% 18 78.3%T6. I know how to show a computer display through a digital projector (LCD). 189 63.2% 18 78.3%T7. I know how to create and insert special graphics, tables, and pictures in my documents 218 72.9% 13 56.5%

AGGREGATE 1,467 70.1% 100 62.1%

RESPONSES STUDENTS (299) FACULTY (23)

(‘Frequently’+ ‘Always’) for all skill set questions (by category)

Total Responses (by category)

Percent of High ConfidenceResponses(by category)

(‘Frequently’+ ‘Always’) for all skill set questions (by category)

Total Responses(bycategory)

Percent of High ConfidenceResponses(by category)

SKILLS SETS

POINT ACCESS(5 questions)

641 1495 42.9% 67 115 58.3%

RESEARCH(6 questions)

1235 1794 68.8% 125 138 90.6%

DATABASES(6 questions)

663 1794 37.0% 71 138 51.4%

THE INTERNET(4 questions)

698 1196 58.4% 71 92 77.2%

TECHNOLOGY(7 skills rated)

1,467 2093 70.1% 100 161 62.1%

RANK(high to low confidence)

STUDENTRANKING OF AGGREGATE HIGH-

CONFIDENCE RESPONSES(by skill category)

FACULTYRANKING OF AGGREGATE HIGH-

CONFIDENCE RESPONSES(by skill category)

1 TECHNOLOGY 70.1% RESEARCH 90.6%

2 RESEARCH 68.8% INTERNET 77.2%

3 INTERNET 58.4% TECHNOLOGY 62.1%

4 POINT ACCESS 42.9% POINT ACCESS 58.3%

5 DATABASES 37.0% DATABASES 51.4%

Table 5. Comparison of Student and Faculty Aggregated “High Confidence” Responses (Ranked from highest to lowest.)

RANK(high to low confidence)

STUDENTRANKING OF AGGREGATE HIGH- CONFIDENCE RESPONSES

(by skill set and grade)

Grade 9 Grade 10 Grade 11 Grade 121 TECHNOLOGY

(69.1%)TECHNOLOGY(78.1%)

RESEARCH(72.1%)

RESEARCH(74.9%)

2 RESEARCH(58.1%)

RESEARCH(72.0%)

TECHNOLOGY(62.0%)

TECHNOLOGY(72.9%)

3 INTERNET(48.5%)

INTERNET(58.0%)

INTERNET(61.3%)

INTERNET(69.0%)

4 POINT ACCESS(39.1%)

POINT ACCESS(50.0%)

POINT ACCESS(37.5%)

POINT ACCESS(46.6%)

5 DATABASES(35.5%)

DATABASES(44.2%)

DATABASES(31.0%)

DATABASES(36.3%)

Table 6. Comparison of Aggregated High-Confidence Responses By Skill Category and Grade Level

Most frequently reported uses of the Internet:

-The most frequently reported uses by students were e-mail, Facebook, search engines, and watching videos (YouTube, Hulu, DVDs.)

-For faculty, e-mail, news, weather, search engines, and on-line banking.

-Both faculty and students indicated more frequent use of search engines over databases when seeking information or doing research.

Developing the Skills Test

A pool of 46 skills test questions was assembled from a variety of existing tests, including those used by the Wooster School Library, Carleton College, Bowdoin College, Wichita State University, TRAILS (Kent State University), and the University of Maryland, Baltimore County (UMBC).

Other Valuable Sources for Test Questions

Question Selection:-School technology committee members (including the school librarian) were asked to select 25 to 30 questions that:

-were of an appropriate level of difficulty;-matched well with items from the perceptions survey;-represented a balance of questions relating to and testing ACRL or AASL information literacy standards.

-School librarian, library aide assisted in final question selection (25 questions total).

- Version one of the skills test was piloted with 37 students (using Survey Monkey); final test reduced to 20 questions.

Student Participant Profile

Maximum Respondent Pool*: 302 students

Grade Levels: Grade 9: 109 Grade 10: 91 Grade 11: 30 Grade 12: 72

Gender: Male: 138 Female: 163 No Response: 1

*A pilot survey of 26 questions was administered to 37 student respondents. Based on the feedback on the pilot test, the skills test was revised and shortened to twenty questions, which was administered in two separate surveys of ten questions each. The tests were administered on-line, through Survey Monkey. 302 students participated in the first part of the skills test; 202 students participated in the second part of the test. The demographic and background information (above) was taken from those who participated in the first part of the skills test.

Students were also asked about:Their Source of Knowledge of Library Resources: A course or project in which a teacher scheduled a class with a librarian: 198 Self-taught: 129 Sought help from librarian on my own: 57 Reference librarian at a public librarian: 47 None of the above: 23

Library Resources Used in the Last Six Months

Electronic Library Catalog 99MARVEL 92

Student Resource Center Gold 76 SIRS Online 37 Infotrac 15 Skipped the question 100

Perception Versus Reality:

Baseline Comparisons

POINT SOURCE SKILLS: PERCEPTION

I feel competent using the on-line library catalog to search for books.

41.1% of students gave high-confidence responses on this statement.

The screen shot, above, is the search page for the Morse High School’s library catalog. If you were to hit the “Search” button using the terms shown, the list of search results would be:

a. A list of magazine and journal articles about both Germany and World War IIb. A list of different web sites about both Germany and World War IIc. A list of databases with information about both Germany and World War IId. A list of books in the Morse High School library about both Germany and World War II.e. Don’t know.

POINT SOURCE SKILLS: REALITY

REALITY 37.4% (113) of students chose the correct response for this question.

26.5% (74) of students answered incorrectly. 15.9% (48) of students said that they did not know the answer. 22.2% (67) of students skipped the question.

Comparison of Perception and Reality: 41.1% were confident about their ability to use the OPAC. 37.4% answered the corresponding skills test question correctly.

RESEARCH SKILLS: PERCEPTION

I am able to tell the difference between a primary and secondary source.


RESEARCH SKILLS: REALITY

Which of these items relating to Walt Whitman is the best example of a primary source?

a. A biography of Walt Whitmanb. A recently published book of letters Whitman wrote to friends and familyc. A Library of Congress collection of Whitman’s personal notebooksd. A book of critical analysis about his poetry.e. Both (b) and (c).

REALITY: 36.1% (109) of students chose the correct response for this question. 28.8% (87) of students answered incorrectly. 11.6% (35) of students identified one of the two primary sources. 23.4% (71) of students skipped the question.

Comparison of Perception and Reality: 60.9% were confident about their ability to identify primary sources. 36.1% answered the corresponding skills test question correctly.

DATABASE SKILLS: PERCEPTION

I am able to perform a successful search in SIRS online.

28.1 % of students gave high-confidence responses on this statement.

DATABASE SKILLS: REALITY

The Morse High School Library database that would be the best source of information for an in-class debate on the Iraq war is:

a. Student Resource Center Goldb. Infotracc. MARVELd. SIRS onlinee. Don’t know

REALITY 5.9% (12) of students chose the correct response for this question. 45.5% (92) of students answered incorrectly. 31.2% (63) of students said they did not know the correct answer. 17.3% (35) of students skipped the question.

Comparison of Perception and Reality: 28.1% were confident about their ability to use SIRS online successfully in a search. 5.9% answered the corresponding skills test question correctly.

INTERNET SKILLS: PERCEPTION

I know if a website is valid or might be biased.

58.2 % of students gave high-confidence responses on this statement.

If you are writing a paper on animal rights, and you use information from a web site produced by the organization People for the Ethical Treatment of Animals (PETA), which of the following considerations is important when you evaluate the information?

a. Is there bias?b. How current is the information?c. Are the sources of factual information easy to verify?d. What related links are provided on the web site?e. All of the above.

INTERNET SKILLS: REALITY


24.2% (73) of students identified at least one aspect of bias, although they answered the question incorrectly. 21.5% (65) of students skipped the question.

Comparison of Perception and Reality: 58.2% were confident about their ability to identify website bias. 54.3% answered the corresponding skills test question correctly.

TECHNOLOGY SKILLS: PERCEPTION

I know how to make a spreadsheet.


TECHNOLOGY SKILLS: REALITY

A B C D1 232 123 364 956 =SUM(C1:C3)

In the spreadsheet below, what is the result in cell C6?a. 80b. 71c. 36d. 57e. Don’t know.


22.2% (67) of students answered the question incorrectly. 38.1% (115) of students said they didn’t know the correct answer. 22.5% (68) of students skipped the question.

Comparison of Perception versus Reality: 66.2% were confident about their ability to make a spreadsheet. 17.2% answered the corresponding skills test question correctly.

SUMMARY TABLES: ALL DATA

PERCEPTIONS SURVEY DATASKILLS TEST

DATA

CATEGORYfollowed by

Question Numberand Topic

Aggregated High Confidence Responses for Category

Question-SpecificHigh-Confidence Response (when applicable)

Correct Response Percentage

Point Access 42.9%

1. url for library website 49.2% 34.3%

2. OPAC screenshot 41.1% 37.4%

3. Source location 25.7-53.5%

4. Source types 27.2-69.5%

Research 68.8%

1. First step in library research 63.9% 41.7%

2. Background research 63.9% 31.7%

3. Keyword selection 75.9% 31.2%

4. Citation identification 58.5% 32.5-50.1%

5. Plagiarism 82.3% 24.2-71.2%

6. Plagiarism 82.3% 26.7%7. Primary source identification 60.9% 36.1%

Database 37.0%

1. MARVEL 40.6%2. General database 31.2%

3. SIRS 28.1% 5.9%4. Database Advanced Search

49.5% 55.3%

5. Narrowing search 55.2% 44.3%

Internet 58.4%

1. Search engine identification 27.2-69.5%

2. Boolean operators 62.2% 12.9%

3. Website bias 58.2% 54.3%

4. Website trustworthiness 58.2% 59.3%

5. Website types 58.2% 57.4-70.3%

Technology 70.1%

1. Spreadsheet operations 66.2% 17.2%

High Confidence Responses % Correct (Perceptions Survey) (Skills Test)

Aggregate Question Specific

QUESTIONS RAISED:

1. What confidence/skill levels are “adequate”, regardless of the gap between perception and reality?

-What are goals for attainment?

2. How meaningful are the gaps?-Are there poor questions? (bad wording, poor correspondence)-Distinguishing true errors from errors that result from test fatigue or disinterest

Additional Observations and Issues:

About the research process:

1. Students and faculty appeared more responsive to perceptions survey.

Possible explanations: -perceptions survey was less threatening -timing of perceptions test versus skills test (at

outset of program versus later in the school year) -survey “fatigue” (due to NEASC accreditation) -complaints that skills test was “too long” or “too

hard”

2. Institutional issues/support affected the overall success of surveying.

Such as:-technical issues (e.g. band width issues);-too short a time frame to develop and pilot the skills test (realities of graduate research)-lack of teacher commitment/teacher overload

3. Other institutions rely on perceptions evaluations or skills evaluations in longitudinal studies, but not necessarily both. For example: - Grimble and Williams repeated perceptions

surveys longitudinally; - Bowdoin is administering only its skills survey

longitudinally;

Instruction Opportunities:

1. Twenty-first century skills. The chance to provide students with skills for college and beyond (particularly skills relating to research and electronic resource access/use);

2. Evidence of achievement relating to 1:1 laptop use. The opportunity to improve (and to measure improvement in) skills that integrate technology and research, and thereby help justify the significant investment by R.S.U. 1 in the laptop program.

3. Professional development. The opportunity to improve high school faculty confidence/skills relating to the access and use of electronic resources available in the MHS and local public library.

Other Evidence of Progress . . One Year Later

1. Commitment by MHS to Continue Longitudinal Study

2. Staff Additions (Technology Integrator)

3. More Library-Based Projects and Instruction?

Morse High School Library Blog

http://mhslibrary.blogs.rsu1.org/

Morrissette, R. (2007). What do they know? A strategy for assessing critical literacy. Knowledge Quest. 35, 15-17.

Sources Cited

Grimble, Bonnie J. & T.D. Williams. (2004). Students’ perceptions of their information literacy skills in the media center. Library Media Connection. 22:4, 26-28.