information technology literacy assessment: a practical approach
Post on 12-Sep-2021
1 Views
Preview:
TRANSCRIPT
INFORMATION TECHNOLOGY LITERACY ASSESSMENT: A PRACTICAL APPROACH AT THE MIDDLE LEVEL
Jill Cohan and Amy Wilcox-Owens
Wright Middle School
Assessment data from all instructional areas is the mandate from the No Child
Left Behind (NCLB) legislation that is currently driving many initiatives in America’s
public schools. The Madison Metropolitan School District (MMSD) is no exception and
is responding to one particular deadline this year. The deadline accommodates the
“Enhancing Education through Technology Act of 2001” embodied in the following goal:
Assist every student in crossing the digital divide by ensuring that every student is technologically literate by the time the student finishes the eighth grade, regardless of the student’s race, ethnicity, gender, family income, geographic location, or disability.
Becoming “technologically literate” is obviously the assessment goal for eighth
graders, but what does this mean and how is the assessment incorporated into the middle
school curriculum? The assumptions behind this requirement are many- staff expertise,
equipment and software, time and individualization. The federal involvement is vague in
the requirement and absent in the financing, but pops back up in the demand for an
assessment that can provide achievement numbers for public accountability.
MMSD Grade 8 Information and Technology Assessment
Given the very general mandate, it becomes clear that the definition of
technological literacy, the method of determining and tracking student technological
literacy, and the ultimate assessment of technological literacy is left up to local districts.
In response, the MMSD gathered a task force in the summer of 2005 and structured the
evaluation as follows:
1. They broadened Technology Literacy into Information Technology Literacy (ITL) as it
is referred to in the Wisconsin Model Academic Standards and defined it as:
The ability to responsibly use appropriate technology to communicate, solve problems, access, manage, integrate, evaluate and to create information to improve learning in all subject areas, and to acquire lifelong knowledge and skills for the 21st century.
2. They determined that the assessment of student ITL would be based upon:
A. The Wisconsin Knowledge and Concepts Examination (WKCE).
Item analysis will focus on questions that assess InfoTech standards and
proficiency will be extrapolated.
B. The MMSD Middle School keyboarding scores.
C. A Content Project that should be aligned with some of the performance standards
and reveal standards accomplishment in a student /teacher checklist.
3. They produced a MMSD/grade 8 Information and Technology Assessment Handbook.
This handbook provides a partial adaptation of the previously established Wisconsin
Model Academic Standards for Information and Technology Literacy. The sub-
categories of Information + Inquiry and Media + Technology are broken down into
performance standards that 8th graders must accomplish to demonstrate proficiency.
From A.8.1 through B.8.8, these 14 performance standards cover a variety of
technology skills and information literacy skills that are general, but are illuminated
by “indicators”. Within the 8th grade curriculum, students are expected to
demonstrate ITL by accomplishing a majority of these indicators, given the time,
appropriate assistance, and effective equipment. (See Appendix A.)
Information Technology Literacy at Wright Middle School
James C. Wright Middle School is the newest middle school in Madison,
Wisconsin. Built in 1997 and highlighted as a “technology charter school,” Wright
opened its doors as an educational beacon for the south side of Madison. The majority of
Wright’s student community is from the neighborhood, creating a unique school in
Madison with nearly 90% students of color. With class size capped at 20, the school’s
small population of 240 students and approximately 30 staff members has a 1997 state-
of-the-art building and technology infrastructure. Over the past 9 years, the school has
slipped away from a technology focus. This shift may be due to staffing cuts, lack of
technology upgrades, or greater need in other areas, but a change has occurred.
Technology is not ignored, but it is no longer the main focus of the school. It is simply
another content area literacy.
The staff leadership team at Wright Middle School identified Content Area
Literacy as a focus for the 2005-2006 school year. The offering of Classroom Action
Research (CAR) on literacy this year at Wright reinforced this commitment. The class
provided the opportunity for two people caught up in the ITL assessment to review our
methods and rubrics and to develop a generic tool that will work for the students and
faculty at Wright to assess 8th grade ITL.
ITL is certainly a content area literacy brought to the center this year with the
timeline for generation of a report of student proficiency, a requirement of NCLB. The
MMSD ITL handbook even delineates the “roles and responsibilities” of school
personnel. Classroom teachers are charged with:
Allowing students to attain ITL proficiency through the completion of a qualifying
project and assess student proficiency using the yes/no checklist provided.
Technology teachers seem to be the major facilitators for this evaluation:
Providing explanation of the standards to staff and students, samples of projects,
keyboarding assessment, and overall support.
Library teachers are to:
Provide instructional and research support for projects and assist in assessment.
Throughout the MMSD, schools vary in the availability of technology teachers
and library teachers on staff to support classroom teachers in this instruction and
assessment. At Wright Middle School, both the technology teacher and the library
teacher are part-time faculty. The ITL handbook is a general guide for the district.
Wright Middle School must decide how best to facilitate this required assessment,
working within our student schedule and faculty framework.
We are Jill Cohan and Amy Wilcox-Owens, the Wright technology and library
teachers providing instructional support for this technology literacy assessment. We have
appreciated the opportunity to work together to come to an understanding of the
standards that seem to be so basic to the assessment, of the students whom we witness
using complicated technology on a daily basis, and of a practical assessment that is
efficient and realistic for students and faculty alike.
The Question
The performance standards use specific performance vocabulary such as send,
view, print, save, but also vocabulary that is vague and even confusing in interpretation
such as, determine, identify, use, judge, and demonstrate. If vocabulary barriers exist that
make it difficult for us to interpret the standards, isn’t it possible that other faculty
members might also have difficulty with interpretation as they are required to assess
student ITL. In addition, how accurately will our students be able to complete the self-
assessment portion of the evaluation.
Technology literacy is not just an assessment of the students’ understanding of the
vocabulary of the performance standards, but their working knowledge of technology.
Eighth graders need to be able to assess accurately their own technology literacy. Our
question: How can 8th grade student self-assessment of Information Technology
Literacy be facilitated to better reflect actual knowledge?
Student Self-Assessment
Students in eighth grade were generally born in 1991. In their lifetime, computers
have been a constant in their world. In their school lives they have had access to
computers in their classrooms, to building computer labs, and to school libraries in which
the library collection and circulation systems have all been computer based. Using
keyword searching combined with Boolean operators, they have found books and
information and used a variety of software programs; they have put together research
projects and reports. Internet resources have been regularly available in the schools and
specialty software programs have enabled them to learn word processing, send emails
and create Power Point presentations.
Of course, some students have benefited from additional technology resources at
home, but this is balanced by students with no reinforcement from home and students
who have come from other school environments with a variety of experiences. We meet
students where we find them as in all other instructional challenges, but since these 8th
graders have been surrounded by a world driven by complex technology, ITL is almost a
given.
We have seen these students in action in the computer lab and the LMC,
producing brochures on world countries and Power Points on African American heroes.
But we need assessment that will establish data, so we turned to the MMSD/grade 8
Handbook for suggestions. Two checklist assessment tools are included in the Handbook
based upon projects assigned elsewhere in the district. The checklists give the standards
met, detail the activity and provide a simple yes/no teacher assessment of the project and
a student self-assessment.
Wright Middle School provides technology instruction for one semester to each
student at the 6th and 7th grade levels. At the eighth grade level, technology is integrated
into most academic classes. At all grade levels, we have projects coming out of a variety
of classes and need a more generic checklist that can be adapted to generate the
assessment data required. In the handbook, the sample checklists are project specific.
But the samples do not apply to our projects and the “design your own project option”
does little to encourage teachers to create high quality projects that encompass a range of
technological and academic skills. The student self-assessment piece is also a concern.
Will students be able to accurately assess their ITL?
The Study
We thought students were more technologically literate than they thought they
were. When asking students about their own ITL, they focused on what they could not
do. They undervalued to the point of disregard, the skills they possessed. One student
even said, “If I know how to download music to my MP3 player, everyone must.” With
the knowledge that the final MMSD ITL assessment would be a self-evaluation, we
feared that students would short change themselves, so we set out to determine several
things. First, could the students interpret the standards? Could they read them,
understand the meaning and then evaluate their own skills against the standards? Second,
we collected data to show that the students did indeed possess the skills set forth in the
standards. Third, we wanted to see if the students could reliably self-assess using a
rubric. Finally, we wanted to see if a modified assessment rubric could provide accurate
data regarding student literacy.
Data and Findings
Data Collection 1
Our first data collection was an activity to determine the degree to which students
could understand the MMSD ITL standards as written. The standards are written in a
table with a list of indicators that could meet the performance criteria. Students were
provided with the standards and the performance criteria. They were then asked to look at
each standard and to identify one indicator from the list that they were able to do and one
indicator that they were unable to do. In a separate document, they were to list the
number of the standard and in their own words state an action they could perform and one
they could not.
For example, for Performance Standard B.8.8-Evaluate the information product
and process, a student may identify themselves as “being able to identify the criteria to be
used in judging the product and the process” and identify themselves as NOT being able
to “determine how well research conclusions and product meet the original information
need or question based on the identified criteria”. In the student’s own words, they might
say, “We use rubrics to grade our projects.” And “I don’t usually remember to go back
and check to see if I am actually answering the question.” That was how we were hoping
students would respond.
Findings 1
We found, however, that all but 3 students were unable to interpret the standards
beyond direct copying of indicators. The majority of the students would copy the
indicators from the electronic version of the worksheet and paste them onto an email
document. They would almost randomly select items they could do and items they could
not do. One student even responded, “I am unable to use a computer and communications
software to access and transmit information.” The student used email software to
compose and send this message to the instructor, completely fulfilling the standard stated.
Clearly, a disconnect existed between the standards and the student’s understanding.
Data Collection 2
Realizing the difficulties students were having with self-assessment in the form of
the standards, we needed to directly observe student abilities within the area of
information literacy. We created a project that incorporated all of the MMSD ITL
standards with an evaluation rubric in the style that the students were comfortable using.
The combination, we hoped, could demonstrate student proficiency in the standards along
with providing an assessment tool that the students could reliably use to self-assess
progress.
Student’s ability to self-assess is a critical milestone of adolescent development.
As middle school teachers, we strive to develop independence in our students. We want
them to be able to function without the daily, hand-holding that is frequently requested.
We try to make students aware that high school, college, and life, in general does not
provide that kind of support. Students need to be able to assess not only their final
projects, but throughout the project process. They need to be able to determine if they are
on the right track, problem solve how to get back on the track if they are off, and
ultimately be able to celebrate their accomplishments.
With our specific project, expectation of the students was to create a webpage
featuring some person, place or event important in the light of African-American History
Month. Students were to use the program Web Dwarf to create the project. Web Dwarf is
a web site creation tool which enables students to set up web pages with text, pictures,
links and animation. Multiple research and technology skills are required to complete the
task thereby giving ample opportunity for broad evaluation. As students completed each
portion of the task, samples were collected and stored for reference.
Findings 2
Nearly all of the students successfully completed an Internet scavenger hunt on
African-American history. Over 80% completed a brainstorming web of potential topics
using Inspiration. Every student had a web site created using Web Dwarf that featured
some aspect of African-American History. By completing the projects, the students had
proven some form of mastery of the standards. The question was whether or not the
students had the ability to identify and report the skills and knowledge they possessed.
Data Collection 3
Realizing that student self-reporting was a key feature of the ITL Assessment we
set out to determine whether students could reliably self-assess. Would everyone give
themselves an A or, just as likely, an F? We decided to use an evaluation rubric in the
same form as the students had experienced in computer class over the past year. The
rubric is a simple grid with three levels of proficiency: Target, Acceptable and
Unacceptable. The aspects of the project being evaluated are listed down the left hand
side of the document and the criteria for achieving the levels are listed in the
corresponding boxes.
Black History Month Website Evaluation Rubric
Target Acceptable Unacceptable Author Name and Citations
Author’s name provided AND Information Sources provided.
Author’s name provided OR Information Sources provided.
Neither Author’s name provided OR Information Sources provided.
Topic African-American History
Contemporary African-American
Not an African-American topic
Content (Spelling and Punctuation)
Accurate. No convention errors.
1 or 2 convention errors or inaccuracies.
More than 2 convention or content errors.
Links Two or more working links.
1 working link. Less than 1 working link.
Information 5 paragraphs/ 25 sentences.
3-4 paragraphs/ 15-20 sentences.
Less than 3 paragraphs/ 15 sentences.
Pictures/Graphics 5 pictures or graphics 3 or 4 pictures or graphics
Less than 3 pictures or graphics
Students received this rubric in an email on the first day of the project. They also
received copies of the rubric in paper form. The teacher observed students referring to
the rubric as they worked on the web site project and the teacher also directed students to
the rubric as they neared the end of the assignment to verify completion. When a student
claimed finality, “I am done,” he or she would get a rubric to self-assess and turn in. Most
students wanted more time after seeing the shortcomings of the project and the ease at
which improvement could be made. They knew how to use a rubric to meet expectations
or evaluation criteria. But, could they use a rubric to reliably self-assess?
At the project deadline, all students were given a clean rubric for self-assessment.
The computer teacher also had the same rubric for evaluation. How close would the
results be?
Findings 3
To ease grade tabulation, we translated the evaluation rubric into a single number.
Each proficiency level was given a value, 3 for target, 2 or acceptable, and 1 for
unacceptable. The total value for the student self-evaluation rubric and the teacher
evaluation rubric were calculated and recorded. The greatest variation for any student
was 3 points. Over 50% of the student and teacher scores matched.
The actual rubrics were also compared. The majority of the rubrics were identical
between teacher and student assessment. Variations that did occur were never more than
one performance level in difference. In other words, a student assessed at the acceptable
level with the teacher assessing at the target level. Overall, the student self-evaluation of
the website project showed that self-assessment could be accurate and reliable. This
supported our original assumption that the students were developing the capabilities to
evaluate their own progress, especially if that assessment format was one they were
familiar with and one they could understand. We want our students to be successful and
sharing the standard for success and the method of measurement with the students should
be a basic starting point.
Data Collection 4
After determining that students possessed the ITL skills and could reliably self-
assess, we saw the need to create a generic assessment tool that was accessible to the
students yet flexible enough for staff members to adapt and use for future technology
projects. This tool would serve to provide a uniform assessment platform for the school
based on the ITL standards and should help teachers create assignments that help students
work toward achieving the standards. Fortunately, we came across a gem from the
Educational Testing Service, the company that creates the SAT and similar tools. The
Educational Testing Service is working on an online technology literacy assessment for
college students. The beauty in this assessment is the clear, performance-based
objectives. Essentially, technology and information literacy is boiled down to seven one-
word objectives: Define, Access, Evaluate, Manage, Integrate, Create, and Communicate.
Evaluators are left to define in what ways the project at hand helps students progress
within the given skill areas. This would be the tool that we decided we could adapt to
meet our ITL assessment objective.
After limited translation of objectives to middle school reading levels and cross-
referencing of the MMSD ITL standards to the ETS objectives, we delineated tasks that
could be evaluated by students and staff to show literacy in the objective area. For
example, how could students realize that they had managed materials in this project and
demonstrate that ability? This was not a specific lesson within the activity, it was
assumed that students already knew how to save, label and retrieve files, how to manage
materials. Past knowledge is not always assessed on each new activity, so many students
forget that it is a valuable skill or it becomes such a part of their skill set that they no
longer know how to label it. Yet, in a comprehensive assessment, these are valuable
skills that often get overlooked.
The adapted new assessment tool was given to the students in hopes that it would
provide a reliable determination of ITL. (See Appendix B.)
Findings 4
Students given the assessment tool did very well. Several students asked if they
had to be truthful. That was a great sign. When the original set of standards was given to
them, they didn’t even know enough to lie. Now they could read the objectives and
clearly see if they didn’t measure up to the requirements. They asked great questions
about to what degree they had to complete an objective to be proficient. This was a very
relevant question showing that they understood the continuum of knowledge and skills
and the difficult nature of drawing a line in the sand of literate or not literate. Students
completed the assessment truthfully and results correlated well with the teacher
assessment using the same assessment tool.
Conclusions
We must admit, our initial reaction to the MMSD ITL assessment was grumbling
and groaning over one more responsibility tacked onto a half-time job. We clearly
KNEW our students possessed skills in the areas of information and technology literacy,
but we didn’t believe those skills would come through in an assessment of the standards
proposed in the ITL handbook. The form was unusable for students and teachers.
Through the process and time of Classroom Action Research we were able to clearly
define the issues, access the information we needed, evaluate that information, manage
the data the we obtained, integrate it and draw some conclusions, create a cohesive,
reliable, and understandable evaluation tool and generate some accurate data on the ITL
skills of the students at JC Wright Middle School.
We have gone beyond our original question to create a tool that can be utilized
across the curriculum for almost any technology-based project. The clear, jargon-free
approach opens use to teachers who are not as technologically savvy or engrained in the
vernacular of the technology realm. It is our hope that if teachers better understand the
standards, they are more likely to recognize when technology activities are helping them
meet those standards.
Teachers may also begin integrating technology rich activities into the academic
curriculum, if they see that the technology objectives: Define, Access, Evaluate, Manage,
Integrate, Create and Communicate are so much like the standards in science, language
arts and math. It is, of course, just another content area.
This embrace is also important, as not all students will have access to computer
class in eighth grade. Schools will have to show literacy even if the child has NEVER
had an official computer class. ITL is a content area literacy to be addressed by all staff
and will be more easily shared among the staff of the school with a usable assessment
model. The evaluation rubric can be a common framework by which the many classroom
projects can be shown to support student ITL. When the day comes that NCLB fades
away, as so many other educational initiatives, students will still be able to self-assess
their learning and be able to identify their areas of knowledge and their challenges for
continued learning.
Appendix A: Cohan & Wilcox-Owens, 2006 Information taken from the MMSD Grade 8 Information & Technology Assessment Handbook.
Appendix B: Cohan & Wilcox-Owens, 2006 Based upon the Educational Testing Service Online Technology Literacy Assessment.
Wright Info & Tech Literacy Assessment Project- Create a Website showcasing a person or aspect of Black History Month.
Define- (B.8.1, B.8.2) Figure out what the assignment is and decide on a topic. What was I asked to do? How did I do?
Access- (B.8.3, A.8.1, A.8.4) Find and retrieve (get) information from a variety of sources. What was I asked to do? How did I do? Search the databases on JCW’s LMC Links.
• You used keywords to search the databases.
• You tried more than one database. • You asked for help from peers or
teachers if needed. Use the Internet to get information.
• You used a search engine like GOOGLE and provided keywords to find information about your topic.
Use the resources in the JCW LMC.
• You added information from the print and other media sources from our library or the public library.
Transfer information to your project.
• You used the information you gathered to create your website.
Define the assignment. • You reviewed the assignment expectations sent to you via email.
• You reviewed the assessment rubric. • You asked questions to make sure
you understood what would be required on your final website project.
Browsed information to determine possible topics.
• You brainstormed topics about famous African-Americans and historical topics concerning Civil Rights Movement
• Using the computer program called Inspiration, you listed and connecting information you already knew.
• You completed the Black History Month Scavenger Hunt using the internet to explore additional people and topics
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes NoStudent Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Evaluate- (B.8.4, B.8.8, A.8.6- Decide if you have good enough information in terms of quality and quantity. What was I asked to do? How did I do? Use the Internet to get information.
• You reviewed the list of possible web sites and chose ones that you thought would be the best for your topic.
• You looked at each web site and decided whether the information was helpful.
• You turned away from websites that looked questionable in anyway.
Gather enough information to have an informative project.
• You reviewed your resources BEFORE beginning work on web site creation to see if you had enough information.
• You reviewed your resources DURING website creation to see if you had enough information.
• You reviewed your resources AFTER completing work on website to see if additional information was needed.
• If additional resources are needed you went and found them.
Use the evaluation rubric for the assignment to guide your progress.
• You looked at the rubric often to tell if you were gathering the information needed to complete the project.
• You looked at your project and graded it against the rubric to see areas for improvement.
• You evaluated other student’s projects to see where they could make improvements.
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Manage- (B.8.5, A.8.2, A.8.3) Organize material for later retrieval. What was I asked to do? How did I do? Gather information from your searches and save to your personal file.
• You created an electronic file to save all of your pictures and information for this project.
• You copied and pasted information (including words and pictures) from various sources to use in your project.
• You kept track of the sources of your information so you could give credit to those sources.
Use the information from your file to create your web site.
• You named your picture files so you could easily identify the picture.
• You named your text files so you could easily identify them.
Integrate- (B.8.6, A.8.2)- Present the information you have gathered in your own way and in your own words. What was I asked to do? How did I do? Find information on your topic and put it in your own words.
• You put your notes into your own words using a Word document.
Use more than one source for your information.
• You used websites from the internet for information.
• You used online sources besides the internet.
• You used non-computer-based information.
List where your information came from.
• You attributed your sources on your website.
• You provided links to additional resources on your website.
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Create- (A.8.3, A.8.5)- Generate or adapt online information to express and support a point. What was I asked to do? How did I do? Create a website that features an element of African-American History.
• You used Web Dwarf to plan and design a website.
• You shared pictures and information about your topic.
• You made improvements on your project as you went along.
Communicate- (B.8.7, A.8.4)- Make something to share your information with others. What was I asked to do? How did I do? Publish your web page using Web Dwarf and Internet Explorer.
• You re-indexed your site and published it to the Student Project Folder.
Build an interesting place to feature the importance of African-Americans in history.
• You used strong words. • You selected clear pictures. • You shared important ideas.
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
Student Yes No Teacher Yes No
top related