information technology literacy assessment: a practical approach

INFORMATION TECHNOLOGY LITERACY ASSESSMENT: A PRACTICAL APPROACH AT THE MIDDLE LEVEL

Jill Cohan and Amy Wilcox-Owens

Wright Middle School

Assessment data from all instructional areas is the mandate from the No Child

Left Behind (NCLB) legislation that is currently driving many initiatives in America’s

public schools. The Madison Metropolitan School District (MMSD) is no exception and

is responding to one particular deadline this year. The deadline accommodates the

“Enhancing Education through Technology Act of 2001” embodied in the following goal:

Assist every student in crossing the digital divide by ensuring that every student is technologically literate by the time the student finishes the eighth grade, regardless of the student’s race, ethnicity, gender, family income, geographic location, or disability.

Becoming “technologically literate” is obviously the assessment goal for eighth

graders, but what does this mean and how is the assessment incorporated into the middle

school curriculum? The assumptions behind this requirement are many- staff expertise,

equipment and software, time and individualization. The federal involvement is vague in

the requirement and absent in the financing, but pops back up in the demand for an

assessment that can provide achievement numbers for public accountability.

MMSD Grade 8 Information and Technology Assessment

Given the very general mandate, it becomes clear that the definition of

technological literacy, the method of determining and tracking student technological

literacy, and the ultimate assessment of technological literacy is left up to local districts.

In response, the MMSD gathered a task force in the summer of 2005 and structured the

evaluation as follows:

1. They broadened Technology Literacy into Information Technology Literacy (ITL) as it

is referred to in the Wisconsin Model Academic Standards and defined it as:

The ability to responsibly use appropriate technology to communicate, solve problems, access, manage, integrate, evaluate and to create information to improve learning in all subject areas, and to acquire lifelong knowledge and skills for the 21st century.

2. They determined that the assessment of student ITL would be based upon:

A. The Wisconsin Knowledge and Concepts Examination (WKCE).

Item analysis will focus on questions that assess InfoTech standards and

proficiency will be extrapolated.

B. The MMSD Middle School keyboarding scores.

C. A Content Project that should be aligned with some of the performance standards

and reveal standards accomplishment in a student /teacher checklist.

3. They produced a MMSD/grade 8 Information and Technology Assessment Handbook.

This handbook provides a partial adaptation of the previously established Wisconsin

Model Academic Standards for Information and Technology Literacy. The sub-

categories of Information + Inquiry and Media + Technology are broken down into

performance standards that 8th graders must accomplish to demonstrate proficiency.

From A.8.1 through B.8.8, these 14 performance standards cover a variety of

technology skills and information literacy skills that are general, but are illuminated

by “indicators”. Within the 8th grade curriculum, students are expected to

demonstrate ITL by accomplishing a majority of these indicators, given the time,

appropriate assistance, and effective equipment. (See Appendix A.)

Information Technology Literacy at Wright Middle School

James C. Wright Middle School is the newest middle school in Madison,

Wisconsin. Built in 1997 and highlighted as a “technology charter school,” Wright

opened its doors as an educational beacon for the south side of Madison. The majority of

Wright’s student community is from the neighborhood, creating a unique school in

Madison with nearly 90% students of color. With class size capped at 20, the school’s

small population of 240 students and approximately 30 staff members has a 1997 state-

of-the-art building and technology infrastructure. Over the past 9 years, the school has

slipped away from a technology focus. This shift may be due to staffing cuts, lack of

technology upgrades, or greater need in other areas, but a change has occurred.

Technology is not ignored, but it is no longer the main focus of the school. It is simply

another content area literacy.

The staff leadership team at Wright Middle School identified Content Area

Literacy as a focus for the 2005-2006 school year. The offering of Classroom Action

Research (CAR) on literacy this year at Wright reinforced this commitment. The class

provided the opportunity for two people caught up in the ITL assessment to review our

methods and rubrics and to develop a generic tool that will work for the students and

faculty at Wright to assess 8th grade ITL.

ITL is certainly a content area literacy brought to the center this year with the

timeline for generation of a report of student proficiency, a requirement of NCLB. The

MMSD ITL handbook even delineates the “roles and responsibilities” of school

personnel. Classroom teachers are charged with:

Allowing students to attain ITL proficiency through the completion of a qualifying

project and assess student proficiency using the yes/no checklist provided.

Technology teachers seem to be the major facilitators for this evaluation:

Providing explanation of the standards to staff and students, samples of projects,

keyboarding assessment, and overall support.

Library teachers are to:

Provide instructional and research support for projects and assist in assessment.

Throughout the MMSD, schools vary in the availability of technology teachers

and library teachers on staff to support classroom teachers in this instruction and

assessment. At Wright Middle School, both the technology teacher and the library

teacher are part-time faculty. The ITL handbook is a general guide for the district.

Wright Middle School must decide how best to facilitate this required assessment,

working within our student schedule and faculty framework.

We are Jill Cohan and Amy Wilcox-Owens, the Wright technology and library

teachers providing instructional support for this technology literacy assessment. We have

appreciated the opportunity to work together to come to an understanding of the

standards that seem to be so basic to the assessment, of the students whom we witness

using complicated technology on a daily basis, and of a practical assessment that is

efficient and realistic for students and faculty alike.

The Question

The performance standards use specific performance vocabulary such as send,

view, print, save, but also vocabulary that is vague and even confusing in interpretation

such as, determine, identify, use, judge, and demonstrate. If vocabulary barriers exist that

make it difficult for us to interpret the standards, isn’t it possible that other faculty

members might also have difficulty with interpretation as they are required to assess

student ITL. In addition, how accurately will our students be able to complete the self-

assessment portion of the evaluation.

Technology literacy is not just an assessment of the students’ understanding of the

vocabulary of the performance standards, but their working knowledge of technology.

Eighth graders need to be able to assess accurately their own technology literacy. Our

question: How can 8th grade student self-assessment of Information Technology

Literacy be facilitated to better reflect actual knowledge?

Student Self-Assessment

Students in eighth grade were generally born in 1991. In their lifetime, computers

have been a constant in their world. In their school lives they have had access to

computers in their classrooms, to building computer labs, and to school libraries in which

the library collection and circulation systems have all been computer based. Using

keyword searching combined with Boolean operators, they have found books and

information and used a variety of software programs; they have put together research

projects and reports. Internet resources have been regularly available in the schools and

specialty software programs have enabled them to learn word processing, send emails

and create Power Point presentations.

Of course, some students have benefited from additional technology resources at

home, but this is balanced by students with no reinforcement from home and students

who have come from other school environments with a variety of experiences. We meet

students where we find them as in all other instructional challenges, but since these 8th

graders have been surrounded by a world driven by complex technology, ITL is almost a

given.

We have seen these students in action in the computer lab and the LMC,

producing brochures on world countries and Power Points on African American heroes.

But we need assessment that will establish data, so we turned to the MMSD/grade 8

Handbook for suggestions. Two checklist assessment tools are included in the Handbook

based upon projects assigned elsewhere in the district. The checklists give the standards

met, detail the activity and provide a simple yes/no teacher assessment of the project and

a student self-assessment.

Wright Middle School provides technology instruction for one semester to each

student at the 6th and 7th grade levels. At the eighth grade level, technology is integrated

into most academic classes. At all grade levels, we have projects coming out of a variety

of classes and need a more generic checklist that can be adapted to generate the

assessment data required. In the handbook, the sample checklists are project specific.

But the samples do not apply to our projects and the “design your own project option”

does little to encourage teachers to create high quality projects that encompass a range of

technological and academic skills. The student self-assessment piece is also a concern.

Will students be able to accurately assess their ITL?

The Study

We thought students were more technologically literate than they thought they

were. When asking students about their own ITL, they focused on what they could not

do. They undervalued to the point of disregard, the skills they possessed. One student

even said, “If I know how to download music to my MP3 player, everyone must.” With

the knowledge that the final MMSD ITL assessment would be a self-evaluation, we

feared that students would short change themselves, so we set out to determine several

things. First, could the students interpret the standards? Could they read them,

understand the meaning and then evaluate their own skills against the standards? Second,

we collected data to show that the students did indeed possess the skills set forth in the

standards. Third, we wanted to see if the students could reliably self-assess using a

rubric. Finally, we wanted to see if a modified assessment rubric could provide accurate

data regarding student literacy.

Data and Findings

Data Collection 1

Our first data collection was an activity to determine the degree to which students

could understand the MMSD ITL standards as written. The standards are written in a

table with a list of indicators that could meet the performance criteria. Students were

provided with the standards and the performance criteria. They were then asked to look at

each standard and to identify one indicator from the list that they were able to do and one

indicator that they were unable to do. In a separate document, they were to list the

number of the standard and in their own words state an action they could perform and one

they could not.

For example, for Performance Standard B.8.8-Evaluate the information product

and process, a student may identify themselves as “being able to identify the criteria to be

used in judging the product and the process” and identify themselves as NOT being able

to “determine how well research conclusions and product meet the original information

need or question based on the identified criteria”. In the student’s own words, they might

say, “We use rubrics to grade our projects.” And “I don’t usually remember to go back

and check to see if I am actually answering the question.” That was how we were hoping

students would respond.

Findings 1

We found, however, that all but 3 students were unable to interpret the standards

beyond direct copying of indicators. The majority of the students would copy the

indicators from the electronic version of the worksheet and paste them onto an email

document. They would almost randomly select items they could do and items they could

not do. One student even responded, “I am unable to use a computer and communications

software to access and transmit information.” The student used email software to

compose and send this message to the instructor, completely fulfilling the standard stated.

Clearly, a disconnect existed between the standards and the student’s understanding.

Data Collection 2

Realizing the difficulties students were having with self-assessment in the form of

the standards, we needed to directly observe student abilities within the area of

information literacy. We created a project that incorporated all of the MMSD ITL

standards with an evaluation rubric in the style that the students were comfortable using.

The combination, we hoped, could demonstrate student proficiency in the standards along

with providing an assessment tool that the students could reliably use to self-assess

progress.

Student’s ability to self-assess is a critical milestone of adolescent development.

As middle school teachers, we strive to develop independence in our students. We want

them to be able to function without the daily, hand-holding that is frequently requested.

We try to make students aware that high school, college, and life, in general does not

provide that kind of support. Students need to be able to assess not only their final

projects, but throughout the project process. They need to be able to determine if they are

on the right track, problem solve how to get back on the track if they are off, and

ultimately be able to celebrate their accomplishments.

With our specific project, expectation of the students was to create a webpage

featuring some person, place or event important in the light of African-American History

Month. Students were to use the program Web Dwarf to create the project. Web Dwarf is

a web site creation tool which enables students to set up web pages with text, pictures,

links and animation. Multiple research and technology skills are required to complete the

task thereby giving ample opportunity for broad evaluation. As students completed each

portion of the task, samples were collected and stored for reference.

Findings 2

Nearly all of the students successfully completed an Internet scavenger hunt on

African-American history. Over 80% completed a brainstorming web of potential topics

using Inspiration. Every student had a web site created using Web Dwarf that featured

some aspect of African-American History. By completing the projects, the students had

proven some form of mastery of the standards. The question was whether or not the

students had the ability to identify and report the skills and knowledge they possessed.

Data Collection 3

Realizing that student self-reporting was a key feature of the ITL Assessment we

set out to determine whether students could reliably self-assess. Would everyone give

themselves an A or, just as likely, an F? We decided to use an evaluation rubric in the

same form as the students had experienced in computer class over the past year. The

rubric is a simple grid with three levels of proficiency: Target, Acceptable and

Unacceptable. The aspects of the project being evaluated are listed down the left hand

side of the document and the criteria for achieving the levels are listed in the

corresponding boxes.

Black History Month Website Evaluation Rubric

Target Acceptable Unacceptable Author Name and Citations

Author’s name provided AND Information Sources provided.

Author’s name provided OR Information Sources provided.

Neither Author’s name provided OR Information Sources provided.

Topic African-American History

Contemporary African-American

Not an African-American topic

Content (Spelling and Punctuation)

Accurate. No convention errors.

1 or 2 convention errors or inaccuracies.

More than 2 convention or content errors.

Links Two or more working links.

1 working link. Less than 1 working link.

Information 5 paragraphs/ 25 sentences.

3-4 paragraphs/ 15-20 sentences.

Less than 3 paragraphs/ 15 sentences.

Pictures/Graphics 5 pictures or graphics 3 or 4 pictures or graphics

Less than 3 pictures or graphics

Students received this rubric in an email on the first day of the project. They also

received copies of the rubric in paper form. The teacher observed students referring to

the rubric as they worked on the web site project and the teacher also directed students to

the rubric as they neared the end of the assignment to verify completion. When a student

claimed finality, “I am done,” he or she would get a rubric to self-assess and turn in. Most

students wanted more time after seeing the shortcomings of the project and the ease at

which improvement could be made. They knew how to use a rubric to meet expectations

or evaluation criteria. But, could they use a rubric to reliably self-assess?

At the project deadline, all students were given a clean rubric for self-assessment.

The computer teacher also had the same rubric for evaluation. How close would the

results be?

Findings 3

To ease grade tabulation, we translated the evaluation rubric into a single number.

Each proficiency level was given a value, 3 for target, 2 or acceptable, and 1 for

unacceptable. The total value for the student self-evaluation rubric and the teacher

evaluation rubric were calculated and recorded. The greatest variation for any student

was 3 points. Over 50% of the student and teacher scores matched.

The actual rubrics were also compared. The majority of the rubrics were identical

between teacher and student assessment. Variations that did occur were never more than

one performance level in difference. In other words, a student assessed at the acceptable

level with the teacher assessing at the target level. Overall, the student self-evaluation of

the website project showed that self-assessment could be accurate and reliable. This

supported our original assumption that the students were developing the capabilities to

evaluate their own progress, especially if that assessment format was one they were

familiar with and one they could understand. We want our students to be successful and

sharing the standard for success and the method of measurement with the students should

be a basic starting point.

Data Collection 4

After determining that students possessed the ITL skills and could reliably self-

assess, we saw the need to create a generic assessment tool that was accessible to the

students yet flexible enough for staff members to adapt and use for future technology

projects. This tool would serve to provide a uniform assessment platform for the school

based on the ITL standards and should help teachers create assignments that help students

work toward achieving the standards. Fortunately, we came across a gem from the

Educational Testing Service, the company that creates the SAT and similar tools. The

Educational Testing Service is working on an online technology literacy assessment for

college students. The beauty in this assessment is the clear, performance-based

objectives. Essentially, technology and information literacy is boiled down to seven one-

word objectives: Define, Access, Evaluate, Manage, Integrate, Create, and Communicate.

Evaluators are left to define in what ways the project at hand helps students progress

within the given skill areas. This would be the tool that we decided we could adapt to

meet our ITL assessment objective.

After limited translation of objectives to middle school reading levels and cross-

referencing of the MMSD ITL standards to the ETS objectives, we delineated tasks that

could be evaluated by students and staff to show literacy in the objective area. For

example, how could students realize that they had managed materials in this project and

demonstrate that ability? This was not a specific lesson within the activity, it was

assumed that students already knew how to save, label and retrieve files, how to manage

materials. Past knowledge is not always assessed on each new activity, so many students

forget that it is a valuable skill or it becomes such a part of their skill set that they no

longer know how to label it. Yet, in a comprehensive assessment, these are valuable

skills that often get overlooked.

The adapted new assessment tool was given to the students in hopes that it would

provide a reliable determination of ITL. (See Appendix B.)

Findings 4

Students given the assessment tool did very well. Several students asked if they

had to be truthful. That was a great sign. When the original set of standards was given to

them, they didn’t even know enough to lie. Now they could read the objectives and

clearly see if they didn’t measure up to the requirements. They asked great questions

about to what degree they had to complete an objective to be proficient. This was a very

relevant question showing that they understood the continuum of knowledge and skills

and the difficult nature of drawing a line in the sand of literate or not literate. Students

completed the assessment truthfully and results correlated well with the teacher

assessment using the same assessment tool.

Conclusions

We must admit, our initial reaction to the MMSD ITL assessment was grumbling

and groaning over one more responsibility tacked onto a half-time job. We clearly

KNEW our students possessed skills in the areas of information and technology literacy,

but we didn’t believe those skills would come through in an assessment of the standards

proposed in the ITL handbook. The form was unusable for students and teachers.

Through the process and time of Classroom Action Research we were able to clearly

define the issues, access the information we needed, evaluate that information, manage

the data the we obtained, integrate it and draw some conclusions, create a cohesive,

reliable, and understandable evaluation tool and generate some accurate data on the ITL

skills of the students at JC Wright Middle School.

We have gone beyond our original question to create a tool that can be utilized

across the curriculum for almost any technology-based project. The clear, jargon-free

approach opens use to teachers who are not as technologically savvy or engrained in the

vernacular of the technology realm. It is our hope that if teachers better understand the

standards, they are more likely to recognize when technology activities are helping them

meet those standards.

Teachers may also begin integrating technology rich activities into the academic

curriculum, if they see that the technology objectives: Define, Access, Evaluate, Manage,

Integrate, Create and Communicate are so much like the standards in science, language

arts and math. It is, of course, just another content area.

This embrace is also important, as not all students will have access to computer

class in eighth grade. Schools will have to show literacy even if the child has NEVER

had an official computer class. ITL is a content area literacy to be addressed by all staff

and will be more easily shared among the staff of the school with a usable assessment

model. The evaluation rubric can be a common framework by which the many classroom

projects can be shown to support student ITL. When the day comes that NCLB fades

away, as so many other educational initiatives, students will still be able to self-assess

their learning and be able to identify their areas of knowledge and their challenges for

continued learning.

Appendix A: Cohan & Wilcox-Owens, 2006 Information taken from the MMSD Grade 8 Information & Technology Assessment Handbook.

Appendix B: Cohan & Wilcox-Owens, 2006 Based upon the Educational Testing Service Online Technology Literacy Assessment.

Wright Info & Tech Literacy Assessment Project- Create a Website showcasing a person or aspect of Black History Month.

Define- (B.8.1, B.8.2) Figure out what the assignment is and decide on a topic. What was I asked to do? How did I do?

Access- (B.8.3, A.8.1, A.8.4) Find and retrieve (get) information from a variety of sources. What was I asked to do? How did I do? Search the databases on JCW’s LMC Links.

• You used keywords to search the databases.

• You tried more than one database. • You asked for help from peers or

teachers if needed. Use the Internet to get information.

• You used a search engine like GOOGLE and provided keywords to find information about your topic.

Use the resources in the JCW LMC.

• You added information from the print and other media sources from our library or the public library.

Transfer information to your project.

• You used the information you gathered to create your website.

Define the assignment. • You reviewed the assignment expectations sent to you via email.

• You reviewed the assessment rubric. • You asked questions to make sure

you understood what would be required on your final website project.

Browsed information to determine possible topics.

• You brainstormed topics about famous African-Americans and historical topics concerning Civil Rights Movement

• Using the computer program called Inspiration, you listed and connecting information you already knew.

• You completed the Black History Month Scavenger Hunt using the internet to explore additional people and topics

Student Yes No Teacher Yes No


Student Yes No Teacher Yes NoStudent Yes No Teacher Yes No



Evaluate- (B.8.4, B.8.8, A.8.6- Decide if you have good enough information in terms of quality and quantity. What was I asked to do? How did I do? Use the Internet to get information.

• You reviewed the list of possible web sites and chose ones that you thought would be the best for your topic.

• You looked at each web site and decided whether the information was helpful.

• You turned away from websites that looked questionable in anyway.

Gather enough information to have an informative project.

• You reviewed your resources BEFORE beginning work on web site creation to see if you had enough information.

• You reviewed your resources DURING website creation to see if you had enough information.

• You reviewed your resources AFTER completing work on website to see if additional information was needed.

• If additional resources are needed you went and found them.

Use the evaluation rubric for the assignment to guide your progress.

• You looked at the rubric often to tell if you were gathering the information needed to complete the project.

• You looked at your project and graded it against the rubric to see areas for improvement.

• You evaluated other student’s projects to see where they could make improvements.




Manage- (B.8.5, A.8.2, A.8.3) Organize material for later retrieval. What was I asked to do? How did I do? Gather information from your searches and save to your personal file.

• You created an electronic file to save all of your pictures and information for this project.

• You copied and pasted information (including words and pictures) from various sources to use in your project.

• You kept track of the sources of your information so you could give credit to those sources.

Use the information from your file to create your web site.

• You named your picture files so you could easily identify the picture.

• You named your text files so you could easily identify them.

Integrate- (B.8.6, A.8.2)- Present the information you have gathered in your own way and in your own words. What was I asked to do? How did I do? Find information on your topic and put it in your own words.

• You put your notes into your own words using a Word document.

Use more than one source for your information.

• You used websites from the internet for information.

• You used online sources besides the internet.

• You used non-computer-based information.

List where your information came from.

• You attributed your sources on your website.

• You provided links to additional resources on your website.






Create- (A.8.3, A.8.5)- Generate or adapt online information to express and support a point. What was I asked to do? How did I do? Create a website that features an element of African-American History.

• You used Web Dwarf to plan and design a website.

• You shared pictures and information about your topic.

• You made improvements on your project as you went along.

Communicate- (B.8.7, A.8.4)- Make something to share your information with others. What was I asked to do? How did I do? Publish your web page using Web Dwarf and Internet Explorer.

• You re-indexed your site and published it to the Student Project Folder.

Build an interesting place to feature the importance of African-Americans in history.

• You used strong words. • You selected clear pictures. • You shared important ideas.




information technology literacy assessment: a practical approach

Documents