does technology integration “work” when key barriers are ... · according to the three...

1

Does Technology Integration “Work” When Key Barriers are Removed?

Deborah Lowther, J. Dan Strahl, Fethi A. Inan, and Steven M. Ross

Center for Research in Educational Policy

The University of Memphis

Paper presented at the annual meeting of the

American Educational Research Association in New York, NY

March 2008

2

Does Technology Integration “Work” When Key Barriers Are Removed?

Abstract

The effectiveness of Tennessee EdTech Launch (TnETL), a statewide technology program designed to meet the NCLB mandate was investigated in this mixed-methods study. The goal of the program was to provide full-time, on-site technology coaches to prepare teachers to create lessons that engage students in critical thinking and use of computers as tools in order to increase learning. The study examined TnETL impact on student achievement, teachers’ skills and attitudes toward technology integration; use of research-based practices; and students’ skills in using technology as a tool. The study was implemented as “Launch” 1 and 2 cohorts that collectively involved 54 schools, 28,735 students and 1,746 teachers. Program effectiveness was measured via direct classroom observations, surveys, student performance assessments, focus groups, and student achievement analysis. A matched treatment-control quasi-experimental mixed-methods research design was used for Launch 1, while a randomized control trial was used for Launch-2. Survey results showed that program teachers had significantly higher confidence to integrate technology and in using technology for learning. Observation results revealed that program as compared to control students more frequently used computers as tools, worked in centers, and engaged in research and project-based learning. Although the TnETL program demonstrated progress in changing school culture to benefit students through the use of technology, student gains on high-stakes tests were mixed. The implications of the results are discussed relative to implementation successes and barriers, sustainability prospects, and the observed impacts of technology integration on teaching and student learning.

Introduction

It is common knowledge that two emergent themes serve as the driving force for integrating technology into K-12 environments: preparing students for the workforce and increasing student knowledge and skills. Yet, after three decades of numerous well-funded technology initiatives, our national efforts are disappointingly meager. For example, a report from the U.S. Department of Commerce revealed that education is ranked as the least technology-intensive enterprise among 55 U.S. industry sectors (ESA, 2003). Further evidence comes from a recent survey of over 400 U.S. employers. Results revealed that our high school graduates are entering today’s workforce deficient in most of the required 21st Century knowledge and skills needed to achieve successful careers (Casner-Lotto & Barrington, 2006). This conclusion is corroborated in a study conducted by the Editorial Projects in Education (EPE), which scored each state’s technology level according to the three following areas: access to technology, use of technology, and capacity to use technology (Education Week, 2006). Based on the study, students in approximately 70% of the states are not yet experiencing the full benefits of effective technology integration (Education Week, 2006).

Similarly, direct observation data from almost 10,000 K-12 classrooms in predominately high- risk schools revealed that computers are infrequently used or used for simple, non-critical thinking activities such as drill and practice or word processing (Ross, Smith, Alberg, & Lowther, 2004). The data also revealed that the most common classroom activities were direct instruction and independent student seatwork, while student-centered activities such as project-based learning and independent inquiry were seen much less frequently. Survey responses from over 90,000 teachers yielded similar results indicating that computers were most often used for low-level learning in teacher-centered environments (Newman, 2002; U. S. DOE, 2003). So, one could ask, what barriers keep K-12 schools from merging the power of technology with classroom instruction to yield high school graduates who are well-educated and ready for the professional responsibilities of future careers?

3

Key Barriers to Technology Integration

Technology initiatives in K-12 schools have been a topic of research interest for the past 30 years. This body of knowledge has identified key barriers that inhibit successful technology integration efforts. Among the list of critical factors are: availability and access to computers (Barron, Kemker, Harmes, & Kalaydjian, 2003; Norris, Sullivan, Poirot, & Soloway, 2003), availability of curriculum materials (Becker & Ravitz, 1999; Butzin, 1992; NCES, 2000b), teachers beliefs (Ertmer, 2005; Lumpe & Chambers, 2001; Van Braak, 2001; Van Braak, Tondeur, & Valcke, 2000; Vannatta & Fordham, 2004; Wozney, Venkatesh, & Abrami, 2006), demographic characteristics of teachers (Bebell, Russell, & O'Dwyer, 2004; Van Braak, 2001), teachers’ technological and content knowledge (Pierson, 2001), and technical, administrative, and peer support (Becker & Ravitz, 1999; NCES, 2000; Ringstaff & Kelly, 2002; Sandholtz & Reilly, 2004; Van Melle, Cimellaro, & Shulha, 2003). Evidence of these barriers influenced the structure and requirements of many technology initiative grants.

The federal government addressed these issues by enacting the Enhancing Teaching Through Technology (ETTT) initiative as Title-II-D of the No Child Left Behind (NCLB) Act of 2001. ETTT mandated active engagement by schools and districts in: (a) implementing proven strategies for integrating technology into curricula and instruction; (b) supporting high-quality professional development activities to facilitate such integration; and (c) examining the conditions under which technology is effective in increasing student achievement and teacher performance (U. S. DOE, 2001). To enact ETTT, state-level grants were made available. This research reports the findings of a study that investigated the overall effectiveness of the Tennessee EdTech Launch (TnETL), an ETTT funded initiative.

The purpose of the TnETL was to integrate the use of technology as a tool into curriculum and instruction to prepare students to meet state academic standards. The specific goals were as follows:

• All students will be educated in learning environments that have access to educational technology used in support of academic achievement.

• All students will demonstrate technology literacy by the end of eighth grade. • All students will be taught by teachers qualified to use technology for instruction.

To accomplish these goals, the Tennessee’s Director of Technology at the time, strategically designed TnETL to address research-identified barriers to technology integration, as seen in Table 1. Foundational to this design was utilization of a full-time, on-site technology coach at each school to provide comprehensive professional development interventions for teachers in their own schools. The Technology Coaches participated in comprehensive, ongoing professional development focused toward preparing teachers to create student-centered environments that engage students in critical thinking and use of computers as tools in order to increase learning and performance and to gain 21st Century skills (ISTE, 2007; McCain, 2005; Morrison & Lowther, 2005).

TABLE 1: TNETL’S SOLUTION TO KEY BARRIERS TO TECHNOLOGY INTEGRATION

Key Barriers TnETL Solution Availability and access to computers Substantial funding for technology purchases was

provided to a limited number of schools rather than funding small grants to multiple schools.

Availability of curriculum materials

Curriculum materials were generated or located by the Technology Coach and lesson plans were cooperatively developed and shared by program teachers.

Teachers beliefs

Technology coach provided one-on-one support and encouragement to show benefits of technology in the specific context of each teacher.

4

Key Barriers TnETL Solution Demographic characteristics of teachers

Technology Coaches were past teachers in the school where they were a coach, thus shared the same general demographic profiles of the teachers they mentored.

Teachers’ technological and content knowledge Teachers obtained technology and content knowledge and skills through one-on-one, just-in-time support from the Technology Coach and other teachers at the school.

Technical, administrative, and peer support TnETL provided funding for part-time technicians at each school to address technical problems; administrators attended training and served as mentors or were mentored by administrators from other schools; peer support was achieved through regular meetings and support received from the Technology Coach.

The overall purpose of this research was to determine the degree to which the TnETL initiative accomplished the program goals. Therefore, the study was structured to address the following research questions:

• Does implementation of the TnETL model raise student achievement in treatment (program) schools as compared to control schools?

• Does participation in TnETL improve teachers’ skill levels in, and attitudes toward integrating technology with curriculum and state standards?

• Does participation in TnETL foster greater use of research-based teaching practices while increasing academically focused instructional time and student attention and engagement?

• Does participation in TnETL improve students’ skill levels in using technology as a tool for enhancing learning?

Method Participants

TnETL was implemented in two cohorts (Launch 1 and Launch 2). Launch 1 consisted of 26 schools (13 program and 13 matched control) that participated in the study for three years (2003-06). The “matched pairs” of schools were formed according to the following criteria: locale, grade levels, number of students, percent qualified for free/reduced lunch, ethnicity, and achievement (elementary = reading and mathematics; middle school = algebra; high school = biology).

The second cohort, Launch 2, included 28 schools (14 program and 14 randomly selected controls) that participated in 2004-05 and 2005-2006. The schools were the top Launch 2 applicants that were first matched according to Launch 1 criteria, and then randomly assigned, by a coin-toss, to be a grant recipient or a control school. Collectively, Launch 1 and Launch 2 schools had 28,735 students (program = 13,856; control = 14,879) and 1,746 teachers (program = 872; control = 874). Specific distribution of schools by grade level is shown in Table 2.

5

TABLE 2: TNETL SCHOOLS BY GRADE LEVELS

Launch 1 Launch 2 Grades Program Control Grades Program Control PK-5 1 0 PK-5 0 1 K-2 1 0 PK-6 1 0 K-5 3 3 K-4 1 0 K-8 2 2 K-5 4 4

K-12 0 2 K-8 3 4 4 -5 0 1 1-5 0 1 4 -6 1 0 3-5 1 0 5 -8 1 1 5-8 1 0 6 -8 2 2 6-8 2 3

6 -12 1 1 9-12 1 1 10 -12 1 1

Total Schools 13 13 Total Schools 14 14

Instrumentation

Program effectiveness was measured using the Formative Evaluation Process for School Improvement: Technology Package (FEPSI/TP) developed by the Center for Research in EducationalPolicy (CREP) (Ross & Lowther, 2003). The FEPSI-TP includes seven components: direct classroom observations, surveys, student performance assessments, interviews, focus groups, school-developed technology benchmarks, and student achievement analysis. The FEPSI-TP instrumentation is directly aligned to national technology and curriculum standards, subjected to reliability and psychometric validation (Sterbinsky & Ross, 2003), and employed in a growing number of peer-reviewed, published studies (Smith et al., 1998; Lowther, Ross, & Morrison, 2003; Ross et al., 2004). In 2006-07, FEPSI instruments were used in over 2,000 schools across the nation, with several states and school districts employing a systemic adoption of FEPSI. This study reports findings from the following FEPSI-TP components: direct classroom observations, surveys, student performance assessments, focus groups, and student achievement analysis.

Direct Classroom Observations

The observations consisted of 3-hour whole school (random visits) to capture routine classroom practices that typically occur on a regular basis. Data were collected with three instruments.

School Observation Measure (SOM©): The SOM examines the frequency with which 24 instructional strategies were used during the observations (Ross, Smith, Alberg, 1999). The frequency is recorded using a five-point rubric that ranges from (0) Not observed to (4) Extensively observed. The target strategies include traditional practices (e.g., direct instruction and independent seatwork) and alternative practices, predominately student-centered methods associated with educational reforms (e.g., cooperative learning, project-based learning, inquiry, discussion, and using technology as a learning tool). These strategies were identified through surveys and discussions involving policy makers, researchers, administrators, and teachers, as those most useful in providing indicators of schools’ instructional philosophies and implementations of commonly used reform designs (Ross, Smith, Alberg, & Lowther, 2004).

Observation of Computer Use (OCU): A companion observation instrument to SOM, the OCU was designed to capture exclusively student access to, ability with, and use of computers rather than teacher use of technology (Lowther & Ross, 2001). Five primary types of data are recorded: (a) computer capacity and currency, (b) student configuration, (c) student computer ability, (d) student computer activities, and (e) overall rubric. The computer activities are grouped by type of software tool: (a) production, (b) Internet/research, (c) educational software, and (d) testing software. Observed student use of these tools is recorded using the same five-point rubric as the SOM. The “Overall Rubric” is used to assess the degree to which each observed activity reflects “meaningful use” of computers as a tool to enhance learning as

6

described by the National Education Technology Standards for Students (NETS-S) (ISTE, 2004). The rubric has four levels of computer use: 1 – Low-level, 2 – Somewhat meaningful, 3 – Meaningful, and 4 - Very meaningful.

Rubric for Student-Centered Activities (RSCA): The RSCA was developed as an extension to SOM and OCU (Lowther & Ross, 2000) to more closely evaluate the degree of learner engagement in seven areas considered fundamental to student-centered learning (cooperative learning, project-based learning, higher-level questioning, experiential/hands-on learning, student independent inquiry/research, student discussion, and students as producers of knowledge using technology). Each item includes a two- part rating scale. The first is a four-point scale, with 1 indicating a very low level of application, and 4 representing a high level of application. The second is a Yes/No option to the question: “Was technology used?” with space provided to write a brief description of the technology use.

To ensure the reliability of observation data, observers complete comprehensive training in a group session and sufficient school-based practice exercises to ensure that his/her data are comparable with those of experienced observers. In addition, each observer is provided with observation manuals that provide definitions of terms, examples and explanations of the target strategies, and a description of procedures for completing each instrument. Results from a 2004 reliability study revealed that whole school observer ratings for the SOM were within one category for 96% of observations; ratings for the OCU were within one category for 97% of observations; and RSCA ratings were also within one category for 97% of the observations (Sterbinsky & Burke, 2004).

Teacher Surveys:

Two surveys were used to collect teacher impressions of the TnETL Launch 1 and 2 programs: the Teacher Technology Questionnaire (TTQ) and the Technology Skills Assessment (TSA). The surveys are described below.

Teacher Technology Questionnaire (TTQ): The TTQ is a two-part instrument used to collect teacher perceptions of computers and technology. In the first section, teachers rate their level of agreement with 20 statements regarding five technology-related areas: impact on classroom instruction, impact on students, teacher readiness to integrate technology, overall support for technology in the school, and technical support. Items are rated with a five-point Likert-type scale that ranges from (1) Strongly Disagree to (5) Strongly Agree. Three primary questions are asked in the second section. The first asks teachers to rate their level of computer ability as very good, good, moderate, poor, or no ability. Next, teachers indicate if they have a home computer, and if they do, if they use the home computer to access instructional materials on the Internet and/or to prepare classroom materials. The final area examines teacher impressions regarding the impact having an on-site Technology Coach.

Technology Skills Assessment (TSA): The TSA is a 57-item survey that includes 50 three-point Likert- type questions designed to assess the perceived technological abilities of the participants (Marvin, Lowther, & Ross, 2002). All of the questions are arranged into seven categories, which are aligned to NETS for Students Grades 6-8 (ISTE, 2004). The categories of the survey are as follows: Computer Basics, Software Basics, Multimedia Basics, Internet Basics, Advanced Skills, Using Technology for Learning, and Policy and Ethics.

Teacher Focus Groups

Focus groups with classroom teachers were used to obtain more in-depth perceptions regarding program implementation. Specifically the Teacher Focus Groups acquires teacher impressions of technology integration efforts regarding their schools’ teachers, students, professional development, parents, and their own roles.

Student Performance-Based Assessment

A technology task was administered to examine the impact of TnETL on student ability to generate computer products that reflect problem-solving solutions. A description of the measure is below.

7

Student Technology Task. The Student Technology Task (Lowther & Marvin, 2004) assesses the degree of proficiency with which 22 basic computer tasks that reflect the NETS for Students in grades 6-8 (ISTE, 2004) are completed. The performance task categories and number of items per category are as follows: spreadsheets (10), presentations (10), and Internet (2). Students are provided with detailed instructions for “what” needs to be done to complete each task, but not given any guidance on “how” to finish the step-by-step procedures. Trained reviewers use a rubric to assess the student products on the degree to which each task is completed: (0) = Did not complete task as described; (1) = Partially completed task as described; and (2) = Completed task as described.

Student Achievement

Student achievement was assessed at the fifth and eighth grade levels to compare student-level mathematics and language performances of program vs. control students on the Tennessee Comprehensive Assessment Program (TCAP), a state-administered standardized test. Initial pre-TnETL analyses were conducted to determine if differences existed in the groups prior to program implementation. To control for ability, students’ preprogram scores were used as a covariate when comparing current year performances.

Procedures

Observers worked with the Technology Coaches at the Program schools and a designated contact person at the Control schools to schedule all data collection events during late spring 2006. Data collection at each Program and Control school included five whole school observations using the SOM, OCU and RSCA, principal interviews, teacher completion of the TTQ and TSA, student performance-based assessment and student achievement. Program schools also participated in one-hour teacher focus groups that were comprised of ten randomly selected teachers from each school. The whole-school visits were specifically scheduled to occur on varied days and times for each school. The student performance-based measures were administered to 32 intact classes of eighth grade students in 16 randomly selected Program (n = 8) and Control (n = 8) schools in May, 2006. Each school had one class complete the Technology Task. All teachers were given student packets of materials and administration instructions.

Data Collection

Table 3 summarizes for each measure, the number completed, administration timeline, and a brief description of the data collection procedure.

TABLE 3: DATA COLLECTION SUMMARY 2005-2006

Number Completed

Measure Timeline Instrument Program Control

Description L-1 L-2 L-1 L-2 Whole School Classroom Observations

Spring 2006

SOM OCU RSCA

65 70 65 69 Three-hour sessions in which about 10 randomly selected classes were observed for 15 minutes each to obtain a perspective on routine teaching practices and technology use.

65 70 65 69

642 679 643 691 Teacher Surveys

May 2006 TTQ TSA

393 418 401 426 Administered during a faculty meeting held during Spring 2006 at each of the 10 schools 394 422 402 429


Spring 2006

Teacher Focus Group Protocol

130 140 NA NA Site researchers conducted on-site, one-hour focus groups with 10 randomly selected teachers from each Program school.

8

Number Completed

Measure Timeline Instrument Program Control

Description L-1 L-2 L-1 L-2

Student Performance- Based Assessment

Spring 2006

Student Technology Task 79 64 54 70

The Technology Task was administered to 8 Program and 8 Control 8th grade classes from randomly selected schools.

Student Achievement

Spring 2006

5th grade Math 177 264 349 363 Student achievement analyses were conducted at the fifth and eighth grade levels to compare the mathematics (math) and reading/language arts (R/LA) performances of Program vs. Control students.

5th grade R/LA 177 264 349 363 Spring 2006

8th grade Math 201 214 207 176 8th grade R/LA 201 214 207 176

Design and Analysis

A matched treatment-control quasi-experimental research design was used for TnETL-1, while a randomized control trial was used for TnETL-2. Both studies were mixed-methods in orientation (Onwuegbuzie & Teddlie, 2003). Teacher-level and school-level quantitative data were analyzed via MANOVA to determine program effects. Qualitative data were examined using a content analysis similar to the analytic procedures developed by Glaser and Strauss (1967) and Strauss and Corbin (1998). Student outcomes on the state mandated tests were analyzed by a one-way MANOVA with 2003 NRT mathematics and reading/language arts (R/LA) scores as the dependent variables. A one-way MANCOVA with 2003 NRT mathematics and R/LA scores as “pretest” covariates was used to compare students’ 2006 CRT mathematics and R/LA performances between program schools and control schools.

Results

A summary of the results are presented by data collection strategy: direct classroom observations, teacher surveys, focus groups, student performance assessments, and student achievement.

Direct Classroom Observations

A total of 269 three-hour whole school observations were conducted for this study. Of these, 135 were completed at Program schools and 134 at Control schools. The 807 hours of observation data were collected with SOMs, OCUs, and RSCAs during 15-minute visits to 2,655 randomly selected classrooms (Program = 1,321; Control = 1,334). Results from each measure are described in the section below.

School Observation Measure

As indicated in the description of SOM, the observation procedure primarily focused on 24 instructional strategies using a five-point rubric (0 = not observed, 1 = rarely, 2 = occasionally, 3 = frequently, and 4 = extensively). Appendix A presents the full, five-category breakdown of the Program and Control whole-school SOM results for L1 and L2. Teachers in both the program and control schools were observed using both traditional direct instruction as well as non-traditional roles of acting as coach/facilitators in 50% or more of the observations. Independent seatwork was also seen occasionally to extensively across program and control classes. As expected, the use of technology as a learning tool and for instructional delivery was seen more frequently during program observations. Independent inquiry was seen more prevalently in program observations, although to a lesser degree than technology use. Higher-level instructional feedback was also observed slightly more often in program schools.

9

Specifically, a series of t-tests, using a Bonferroni adjustment to control for the familywise error rate (FWER), were conducted to compare the Launch 1 (L1) and Launch 2 (L2) Program and Control whole school means on the 26 SOM items. Results showed significantly higher mean scores for L1 Program over Control on 5 items and for L2 Program over Control on 2 items (see Table 4). Effect sizes computed by Cohen’s d formula ranged from +0.55 to +1.15, thus indicating relatively large effects.

Consistent with program goals, the item revealing the greatest difference for L1 and L2 was technology as a learning tool (ES = +1.15). Also noteworthy were differences in the frequency with which computers were used for instructional delivery in Launch 1 (ES = +0.94). There were significant differences in other approaches that are supportive of student-centered learning in a technology-enhanced environment,such as project-based learning, independent inquiry or research on the part of the students, and cooperative or collaborative learning.

TABLE 4: SOM ITEMS SHOWING SIGNIFICANT DIFFERENCES BETWEEN PROGRAM AND CONTROL WHOLE SCHOOL OBSERVATIONS

Differences Reported by Launch

Launch 1 Program Control

(n = 65) (n = 65) SOM Items M SD M SD t (128) p ES

Technology as a learning tool or resource 1.55 1.05 .54 .71 6.48 .000 1.15

Computer for instructional delivery 1.91 1.37 .82 .93 5.32 .000 .94

Project-based learning .97 .92 .32 .56 4.84 .000 .86

Cooperative/collaborative learning 1.55 1.00 .85 .76 4.51 .000 .80

Student independent inquiry/research .82 .79 .35 .65 3.65 .000 .65

* Scale = 0 = Not Observed, 4 = Extensively Observed

Launch 2 Program Control (n =70) (n =70) SOM Items M SD M SD t (138) p ES

Technology as a learning tool or resource 1.76 1.06 .89 .86 5.35 .000 .91 Academically focused class time 3.51 .72 2.99 1.61 3.24 .001 .55


Rubric for Student Centered Activities

One RSCA was completed for each of the 2,655 classrooms observed (L1 Program 642, Control 643; L2 Program 679, Control 691) during whole school visits. Results address the percentage of whole school-sessions in which each RSCA strategy was observed at least once, the quality/strength of strategy applications (1 = limited application to 4 = strong application) (see Appendix B). Inferential analyses (t- test for independent samples) were conducted to compare Program vs. Control rubric ratings. Significant differences were revealed for L1 and L2 cohorts. As seen in Table 5, the L1 results favored the Control group for cooperative learning (ES = - 0.33), student discussion (ES = - 0.53), and higher-level questioning (ES = - 0.21). The L2 results, on the other hand, favored the Program group on two strategies: Independent inquiry/research (ES = +0.34), higher-level questioning (ES = +0.20). However, with the exception of the Control group advantage on student discussion (which is defined on SOM as a traditional whole-class or small-group discussion on a given topic), none of the RSCA effects was strong or statistically significant when a more conservative Bonferroni FWER adjustment was applied.

10

TABLE 5: RSCA ITEMS SHOWING SIGNIFICANT DIFFERENCES BETWEEN L1 AND L2 PROGRAM AND CONTROL WHOLE SCHOOL

Observations by Launch

Launch 1 Program (n = 642) Control (n = 643) RSCA Items n M SD n M SD t p ES Cooperative/collaborative learning 165 2.39 .97 85 2.72 1.10 2.43 .016 -0.33 Higher-level questioning strategies 245 2.56 .92 289 2.76 .99 2.37 .018 -0.21 Student discussion 148 2.28 1.06 202 2.83 1.02 4.86 .000 -0.53

*Rating scale: 1 = limited application; 4 = Strong application.

Launch 2 Program (n = 679) Control (n = 691) RSCA Items n M SD n M SD t n M Higher-level questioning strategies 255 2.60 .87 256 2.42 .95 2.31 .021 +0.20 Independent inquiry/research 73 2.71 .89 79 2.41 .95 2.05 .042 +0.34 *Rating scale: 1 = limited application; 4 = Strong application.

Observation of Computer Use

A summary of the OCU whole-school inferential results for L1 and L2 Program and Control schools is provided by types of software tools used by the students: production, Internet/research, educational software, and testing. OCU frequency distributions are seen in Appendix C.

Production tools used by students. When examining L1 results, three of the production tools were used significantly more in Program vs. Control classrooms: word processing (ES = +1.06), presentation (ES = +0.81), and concept mapping (ES = +0.63) (see Table 6). Even though the Effect Sizes indicate that the differences were educationally important, the extent of use was limited in that the mean scores for Program results only ranged from M = 0.74 to M = 0.26, on a scale where 1 = rarely observed. L2 results revealed one significant difference between student use of concept mapping (ES = +0.54). Again, however, the overall use of presentation software by Program students was very limited. Internet/research tools used by students. L1 Program students used Internet browsers more frequently than any other software application and significantly more than the Control students (ES = +01.59). There were no significant differences between L2 Program and Control student use of Internet/research tools.

Educational software use by students. There were no significant differences between Program and Control student use of educational software.

Testing software use by students. Testing software was infrequently observed in both Program and Control classes and there were no significant differences between the L1 and L2 groups.

11

TABLE 6: OCU ITEMS SHOWING SIGNIFICANT DIFFERENCES BETWEEN PROGRAM AND CONTROL WHOLE SCHOOL COMPUTER ACTIVITIES

Differences Reported by Launch

Launch 1 Program (n = 65) Control (n = 65) OCU Items M SD M SD t(128) p ES

Production Tools Word Processing Presentation Concept Mapping

Internet/Research Tools Internet Browser


Launch 2 Program (n = 70) Control (n = 69) OCU Items M SD M SD t(128) p ES Production Tools

Concept Mapping * Scale = 0 = Not Observed, 4 = Extensively Observed

Overall meaningful use of computers. The culminating assessment on the OCU was the observer’s evaluation of the meaningfulness of the way in which technology was integrated with teaching and learning. To do this, they were asked to indicate how often they observed computer activities at each level of the rubric; e.g., how often did the observers see very meaningful use of computers (not observed to extensively). As can be seen in Table 7, significant differences between L1 Program and Control observations were found on three levels of the rating scale, with fairly strong associated Effect Sizes that ranged from of +0.40 to +1.06. Furthermore, significant differences between L2 Program and Control observations were found on two levels of the rating scale, with an Effect Size range of .49 to .58.

TABLE 7: WHOLE SCHOOL OCU ITEMS SHOWING SIGNIFICANT DIFFERENCES BETWEEN PROGRAM AND CONTROL MEANINGFUL USE OF COMPUTERS

Differences by Launch

Launch 1 Program (n = 65) Control (n = 65)

OCU Items M SD M SD t (128) p ES

Meaningfulness

Low level use of computers .74 1.05 .37 .78 2.27 .025 +0.40

Somewhat meaningful use of computers .85 1.09 .28 .63 3.65 .000 +065

Meaningful use of computers 1.40 1.17 .38 .70 6.00 .000 +1.06

Very meaningful use of computers .66 1.07 .20 .59 3.06 .003 .54

Rating scale: 0 = Not observed; 4 = Extensively

Launch 2 Program (n =70) Control (n =69)

OCU Items M SD M SD t (138) p ES

Meaningfulness

Meaningful use of computers 1.54 1.28 .96 1.15 2.85 .005 +0.49

Very meaningful use of computers .84 1.13 .29 .80 3.38 .001 +0.58

Rating scale: 0 = Not observed; 4 = Extensively

12

Teacher Surveys

The two surveys (TTQ, and TSA) were examined for Program vs. Control differences. The results for each are described below.

Teacher Technology Questionnaire

The Teacher Technology Questionnaire (TTQ) was completed by a total of 1,638 Program (L1 n = 393; L2 n = 418) and Control (L1 n = 401; L2 n = 426) teachers. TTQ responses from Program teachers were more positive than Control responses on all items (see Appendix D). For example, between 68.7% and 93.6% of the Program teachers agreed or strongly agreed with all TTQ items as compared to 42.1% to 86.4% of the Control teachers. Specifically, the Program teachers had the highest overall agreement that their students could “capably use computers at an age-appropriate level” (L1 Program = 93.6%; L2 Program = 93.1%). When teachers were asked if they “routinely integrate technology into [his/her] instruction,” over three-fourths of the Program teachers as compared to approximately 60% of the Control teachers were in agreement.

Responses to the final items on the TTQ indicate that more Program (L1 Program = 74.9%; L2 Program = 75.8%) than Control (L1 Control = 55.9%; L2 Control = 59.6%) teachers rated their computer ability as very good to good. However, there were no notable differences in the percentages of teachers who had home computers, accessed instructional material on the web, or used them to prepare classroom materials.

The MANOVA, treating the five survey categories (impact on classroom instruction; impact on students; teacher readiness; overall support; and technical support) as dependent measures, was highly significant for Launch 1 (F(5, 716) = 43.89, p < .001) and Launch 2 (F(5. 768) = 39.29, p < .001). As seen in Table 8 follow-up univariate analyses yielded significance on all five categories with each yielding educationally important effect sizes ranging from +0.62 to +0.78 for Launch 1 and from +0.37 to+0.77 for Launch 2. Most notable, Program teachers had more confidence (L1 ES = +0.78; L2 ES = +0.58) than Control teachers that they were ready to integrate technology and that use of technology positively impacts students (L1 ES = +0.76; L2 ES = +0.48).

TABLE 8: TTQ ITEMS SHOWING SIGNIFICANT DIFFERENCES BETWEEN PROGRAM AND CONTROL TEACHER RESPONSES

Differences by Launch


TTQ Items M SD M SD F p ES

Readiness to Integrate Technology 4.25 .59 3.71 .78 35.30 .000 .78

Overall Support 4.09 .62 3.63 .68 18.59 .000 .71

Impact on Classroom Instruction 4.02 .71 3.50 .76 27.34 .000 .71

Impact on Students 4.20 .59 3.73 .65 29.17 .000 .76

Technical Support 4.05 .74 3.56 .83 15.62 .001 .62


TTQ Items M SD M SD F p ES

Readiness to Integrate Technology 4.29 .63 3.90 .72 18.02 .000 .58

Impact on Students 4.26 .61 3.97 .60 14.00 .001 .48

Impact on Classroom Instruction 4.08 .73 3.81 .73 10.93 .003 .37

Technical Support 4.24 .66 3.66 .83 17.36 .000 .77

Overall Support 4.23 .60 3.87 .59 12.99 .001 .61

13

Technology Skills Assessment

The primary purpose of the Technology Skills Assessment (TSA) was to assess teacher perceptions of their technology ability as indicated in the NETS-S grades 6-8. Teachers rate “How easily…” (Not at all, Somewhat, Very easily) they could use software features to complete 47 tasks related to computer basics, software basics, multimedia basics, Internet basics, advanced skills, and using technology for learning. A descriptive summary of the results from 1,647 (L1 Program n = 394, L1 Control n = 402; L2 Program n = 422, L2 Control n = 429) TSA surveys is presented in Appendix E.

Both Program and Control teacher confidence was highest for computer and software basics For the remainder of the categories, approximately 60% or less of both Program and Control teachers indicated that they could easily do tasks related to: Internet basics; multimedia basics; using technology for learning; and advanced skills. A MANOVA comparing the Program and Control means on the seven TSA categories yielded a highly significant difference for both Launch 1 (F(7, 679) = 16.26, p < .001) and Launch 2 (F(7, 755) = 5.54, p < .001). Univariate analysis of variance (ANOVA) with Bonferroni adjustment (alpha was set to be .007) was then separately performed on each category (see Table 9). Significant differences, with effect sizes ranging from +0.40 to +0.69 for Launch 1, were found for the six of the seven areas. The strongest difference occurred for the category “using technology for learning” (L1 ES = +0.69). Less dramatic differences were seen for computer basics (L1 ES = +0.40) and Internet basics (L1 ES = +0.41). These findings are not surprising considering the widespread use of computers and the Internet among the general population. Although L2 schools have higher means scores in all categories, none of these differences was found to be significant.

TABLE 9: TSA ITEMS SHOWING SIGNIFICANT DIFFERENCES BETWEEN PROGRAM AND CONTROL TEACHER RESPONSES


TSA Items M SD M SD F p ES

Multimedia Basics 2.34 .57 2.03 .61 24.81 .000 .53

Using Technology for Learning 2.23 .60 1.81 .62 27.99 .000 .69

Software Basics 2.70 .40 2.48 .54 21.74 .000 .46

Computer Basics 2.85 .26 2.71 .42 14.40 .001 .40

Advanced Skills 2.13 .61 1.81 .63 18.03 .000 .52

Internet Basics 2.44 .54 2.20 .62 17.28 .000 .41


Responses from 270 Program teachers (L1 n = 130; L2 n = 140) who participated in teacher focus groups at their schools reported that although there are still a few teachers reluctant to use technology, overall attitudes were positive and have improved greatly over the life of the program. There was a general consensus that students were enthusiastic about the use of technology and frequently requested more technology-based lessons. Teachers stated that they believed that technology integration gave their students a competitive edge, improved the school’s image and improved student performance. Teachers also felt most students were more “focused” and behavior problems diminished in technology-driven classrooms. While teachers’ attitudes reflected positively towards their respective Technology Coaches, they also said they would have benefited from even more professional development. Principals were generally seen as supportive of their schools technology integration efforts.

The greatest disappointments for L1 and L2 teachers were related to sustainability of the program, lack of updated hardware, insufficient student access to computers, being overwhelmed by the technology, and maintaining hardware. Final teacher comments included: “It has been a big boost to our staff to feel confident using technology and to make our teaching more effective.” “It has enriched my teaching and made me a better teacher” “It has enhanced every student’s education.” “Will the students be at a disadvantage because we do not have continued support?”

14

Student Performance-Based Assessment

A total of 258 eighth-grade students completed the technology task (L1 Program n = 79, Control n = 54; L2 Program n = 64, Control n = 61). Program students generally exhibited slight advantages as compared to Control students across the three types of software (see Appendix F). However, the degree of proficiency varied somewhat within groups. For example, the mean scores for L1 Program students were directionally higher on 60% of the spreadsheet and presentation tasks and 100% on the Internet tasks. The mean scores for L2 Program students varied from this trend in that the Control students had directionally higher mean scores on 70% of the spreadsheet tasks, however the Program student’s means scores were higher on 100% of the presentation and Internet tasks. Overall, on a scale that ranges from 0

= No to Yes = 2, L1 Program students’ mean scores ranged from M = 0.79 to 1.80, while L1 Control student scores ranged from M = 0.85 to a high of M = 1.77. The greatest advantage was seen for the L2 Program students in that the mean scores ranged from M = 1.20 to 1.93, whereas the range for the L2 Control was M = 0.68 to 1.78.

Inferential results: Program vs. Control A MANOVA comparing the L1 Program and Control student technology task scores did not reveal any significant differences. However, a MANOVA comparing the L2 Program and Control student technology task scores yielded a highly significant difference (F (3, 128) = 6.10, p = 0.001). Follow-up analyses showed significant advantages for the Program group overall with regard to completing the presentation tasks (see Table 10). The effect size was ES = +0.49, indicating that the significant difference was strong and educationally meaningful.

TABLE 10: STUDENT TECHNOLOGY TASK: SIGNIFICANT DIFFERENCES BETWEEN PROGRAM AND CONTROL

Launch 2 Program (n= 64) Control (n= 70) Technology Tasks M SD M SD F (1, 130) p ES Presentation 1.59 .40 1.31 .69 7.67 .006 .49 p < .01

Student Achievement

Student-level achievement analyses at the 5th and 8th grade levels revealed mixed results in L1 and L2 Program and Control schools with regard to student performance in mathematics or reading/language arts on the Tennessee Comprehensive Assessment Program (TCAP). Details are provided below by Launch cohort.

Launch 1 For Launch 1, the MANCOVA analysis showed that there were significant differences in 5th grade students’ 2006 CRT mathematics and reading/language arts (R/LA scores between program and control schools (F(2,521) = 4.88, p = .008). Follow up ANOVA showed that only 2006 CRT mathematics was significantly different, favoring program schools (F(1,522) = 8.62, p = .003). There were no differences between L1 8th grade Program and Control students’ performance in mathematics or language arts.

Launch 2 The Launch 2 results were also mixed in that the 8th grade Program students out-performed the 8th grade Control students in mathematics and language arts, but the reverse occurred for the 5th grade students. Specifically, the MANCOVA analysis showed that there was a significant difference in 5th grade students’ 2006 CRT Mathematics and Language scores between Program and Control schools (F(2,622) = 17.43, p = .001). Follow up ANOVA analysis showed that both 2006 mathematics (F(1,623) = 17.19, p = .001) and CRT language scores (F(1,623) = 32.00, p = .001) were significantly different, favoring Control schools.

The MANCOVA analysis for the 8th grade showed significant differences in students’ 2006 CRT Mathematics and Language scores between Program and Control schools (F(2,385) = 7.66, p = .001).

15

Follow up ANOVAs showed that both 2006 CRT Mathematics scores (F(1,386) = 13.33, p = .000) and Language scores (F(1,386) = 8.07, p = .005) were significantly different, favoring Program schools.

Achievement Summary As seen, student-level achievement analyses at the 5th and 8th grade levels revealed mixed results in L1 and L2 Program and Control schools with regard to student performance in mathematics or language arts (see Table 11). In summary, L1 5th grade students achieved significantly higher mathematics scores than their Control counterparts; yet, there were no differences with regards to language arts scores. There were no differences between L1 8th grade Program and Control students’ performance in mathematics or language arts. The L2 results were also mixed in that the 8th grade Program students out-performed the 8th grade Control students in mathematics and language arts, but the reverse occurred for the 8th grade students.

TABLE 11: SUMMARY OF SIGNIFICANT DIFFERENCES IN TCAP ACHIEVEMENT

Launch 1 Launch 2

5th Grade Mathematics

Language Arts

8th Grade Mathematics

Language Arts

Legend

Program Significantly Higher

Control Significantly Higher

No Significant Difference

Discussion

A review and discussion of the key findings by research question is presented below.

Does implementation of the TnETL model raise student achievement in program schools as compared to control schools?

When examining the student achievement findings, a promising trend emerges as the Program students out-performed or performed as well as Control students in all instances except with regard to Launch 2 5th grade mathematics and language arts. Yet program students also emerge with more experience using technology as a learning tool in meaningful computer activities. Specifically, students in the Program classrooms were significantly more engaged in student-centered learning activities such as experiential, hands- on learning, independent inquiry/research, and cooperative learning. In other words, the Program students were better able than the Control students to demonstrate the application of critical thinking skills, which for some students resulted in superior or comparable TCAP mathematics and language arts performance. However, it seems fair to question the degree to which the TnETL model, which focused on preparing students to use technology for learning and teachers to integrate technology in higher-level and more engaging lessons, could be fairly expected to raise achievement within only a few years on high-stakes objective-type tests (Baker, 2007). Surely, a dividend of improved instructional practices and higher-level learning in schools should be in the long run higher achievement on diverse levels of learning outcomes and types of assessments. But such effects would likely take much more time to be evidenced than afforded by the present three-year study.

16

Does TnETL participation improve teachers’ skill levels in, and attitudes toward, integrating technology with curriculum and state standards?

Teachers who participated in the Launch1 and 2 Programs revealed more positive attitudes toward technology integration, and teachers who participated in the L1 Program reported significantly more confidence to complete computer tasks than the Control teachers. For example, Program teachers had higher agreement that they knew how to meaningfully integrate technology into lessons, that their computer skills were adequate to conduct classes that have students using technology, and that integration of technology positively impacted student learning. Yet, more importantly, data from the classroom observations suggest positive program effects on improving teachers’ skill levels in, and attitudes toward, integrating technology with curriculum and state standards. The Program teachers as compared to Control teachers integrated more intensive and meaningful student use of technology in student-centered environments. However, the scope or variety of software used in Program classes was rather limited, which implies that although Program teachers demonstrated greater skills and attitudes, the need still exists for continued professional development focused on effective use of technology as a learning tool.

Does TnETL participation foster greater use of research-based teaching practices while increasing academically focused instructional time and student attention and engagement?

Overall, the randomly conducted whole school observations revealed that the instructional strategies implemented in TnETL Program schools were more reflective of research-based practices that accommodate technology integration than those observed in Control classes. These practices included greater use of student-centered strategies such as project-based learning, cooperative learning, and independent inquiry and research on the part of students (Grant, Ross, Wang, & Potter, 2005; Kozma, 2003, Rakes, Flowers, Casey, & Santana, 1999; Roschelle, Pea, Hoadley, Gordin, & Means, 2000; Ross, Smith, Alberg,& Lowther, 2004). Of critical importance to this study, the program teachers were better able to integrate greater and higher-quality use of computers as a learning tool and for instructional delivery, as compared to Control teachers. Further, Program teachers reported a significantly higher agreement that the use of technology positively influenced student learning and their use of student-centered practices. The Program classes were more frequently focused on academics than Control classes; however, the difference was only found to be significant for the L2 classes. Similarly, a high level of student attention and interest was more frequently observed in Program classes vs. Control classes. Although these results are positive and reflective of the TnETL goals, continued professional development is needed to better prepare teachers to increase the frequency and intensity of implementation, which could perhaps yield greater and more consistent improvement in student learning.

Does TnETL participation improve students’ skill levels in using technology as a tool for enhancing learning?

Data from the Student Technology Task Performance Assessment, which is directly aligned with the ISTE NETS for Students revealed that in general the program students demonstrated slightly more proficiency in using technology research tools to process data and report results” (ISTE, 2007, Research and Information Fluency #3) as compared to control students; except with regard to L2 program students who performed significantly better than control counterparts in using presentation software. This is consistent with observation data that indicated students in program classes used technology as a tool significantly more often than those in control classrooms. Although student attitudes were not directly measured, there was a consensus among teachers that students “loved” using computers, but wanted more computers available for classroom use.

Implications

17

If one examines TnETL results according to the NCLB mandates and other research-supported guidelines for increasing student learning through effective use of technology, most requirements would be addressed (U.S. DOE, 2001). These elements include adequate computer access; well-implemented, on-site professional development; teachers who believe in and regularly integrate student use of technology, and administrator support in a positive school climate (Barron, Kemker, Harmes, & Kalaydjian, 2003; ISTE, 2007; CEO Forum, 2001). Yet, student gains on high-stakes tests were mixed. Current research offers some possible reasons for this conundrum.

• Even though teachers believed in and engaged students in higher-order use of technology, students also used computers for low-level tasks – this mixed approach may have limited the overall impact on student learning (Ertmer, Gopalakrishnan, & Ross, 2001).

• Teachers may have primarily used computers due to “subjective norms” or expectations placed on them as TnETL grant recipients – possibly resulting in less rigorous efforts due to lack of personal choice (Marcinkiewicz & Regstad, 1996).

• The intensity, focus, or amount of intervention may not have adequately prepared the teachers to effectively develop and implement lessons that fully support achievement of curriculum standards (Kleiman, 2000) or to model 21st Century skills (21st Century Education System Task Force, 2007)

However, despite the above constraints, it remains educationally significant that in school contexts where technology usage was formerly minimal, clear strides were evidenced within only three years in changing school culture by integrating technology with student-centered teaching methods. In this way, a foundation was being established at TnETL schools for increasing student engagement, higher-order learning, and importantly, preparation for using technology skillfully as a tool in both educational settings and the workplace. However, further research is needed to refine and address intervention methods, sustainability of the program, alignment of technology with academic standards and testing, and interventions to change teacher beliefs about and practices with technology so student learning can be better ensured.

18

References

Baker, E. (2007). The end(s) of testing. Educational Researcher, 36(6), 309-317.

Barron, A. E., Kemker, K., Harmes, C., & Kalaydjian, K. (2003). Large-scale research study on technology in K-12 schools: Technology integration as it relates to the National Technology Standards. Journal of Research on Technology in Education, 35, 489-507.

Bebell, D., Russell, M., & O'Dwyer, L. (2004). Measuring teachers' technology uses: Why multiple-measures are more revealing. Journal of Research on Technology in Education, 37(1), 45-63.

Becker, H. J. (1994). How exemplary computer-using teachers differ from other teachers: Implications for realizing the potential of computers in schools. Journal of Research on Computing in Education, 26, 291-321.

Becker, H. J., & Ravitz, J. (1999). The influence of computer and internet use on teachers' pedagogical practices and perceptions. Journal of Research on Computing in Education, 31(4), 356-384.

Butzin, S. M. (1992). Integrating technology into the classroom: Lessons from the project CHILD experience. Phi Delta Kappan, 330-333.

Casner-Lotto, J. & Barrington, L. (2006). Are they really ready to work: Employers perspectives on the knowledge and applied skills of new entrants to the 21st Century U.S. workforce. The Conference Board, Inc., the Partnership for 21st Century Skills, Corporate Voices for Working Families, and the Society for Human Resource Management. Retrieved January 21, 2008 from http://www.21stcenturyskills.org/index.php?option=com_content&task=view&id=82&Ite mid=40.

CEO Forum (2001, June). The CEO forum on school technology year 4 report: Key building blocks for student achievement in the 21st century. Retrieved December 12, 2007 from http://www.ceoforum.org/downloads/report4.pdf.

Ertmer, P. A. (2005). Teacher pedagogical beliefs: The final frontier in our quest for technology integration? Educational Technology Research and Development, 53(4), 25-39

Ertmer, P. A., Gopalakrishnan, S., & Ross, E. M. (2001). Technology-using teachers: Comparing perceptions of exemplary technology use to best practice. Journal of Research on Technology in Education, 33(5). Retrieved January 4, 2008 from http://www.iste.org/jrte/33/5/ertmer.html

Glaser, B.G., & Strauss, A.L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago, IL: Aldine.

Grant, M. M., Ross, S. M., Wang, W., & Potter, A. (2005). Computers on wheels: An alternative to 'each one has one'. British Journal of Educational Technology, 36(6), 1017-1034.

International Society for Technology in Education (ISTE) (2007). National Educational Technology Standards for Students: Technology Foundation Standards for all Students. Retrieved January 21, 2008 from http://cnets.iste.org/students/s_stands.html

Jonassen, D. H. (1994). Thinking technology: Toward a constructivist design model. Educational Technology 34(4), 34-37.

Kleiman, G. M. (2000, April -June). Myths and realities about technology in K-12 schools. Leadership and New Technologies, 14. Retrieved September 12, 2007 from http://www2.edc.org/LNT/news/Issue14/feature1.htm.

Kozma, R. B. (2003). Technology and classroom practices: An international study. Journal of Research on Technology in Education, 36(1), 1-14.

http://www.21stcenturyskills.org/index.php?option=com_content&task=view&id=82&Ite

http://www.ceoforum.org/downloads/report4.pdf

http://www.iste.org/jrte/33/5/ertmer.html

http://cnets.iste.org/students/s_stands.html

http://www2.edc.org/LNT/news/Issue14/feature1.htm

19

Lowther, D. L., & Marvin, E. D. (2004). Student Technology Task. Memphis, TN: Center for Research in Educational Policy, The University of Memphis.

Lowther, D. L., Ross, S. M., & Morrison, G. R. (2003). When each one has one: The influences on teaching strategies and student achievement of using laptops in the classroom. Educational Technology Research and Development, 51(03), 23-44.

Lowther, D. L. & Ross, S. M., (2003). Formative evaluation process for school improvement: Technology instrument suite. Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Lowther, D. L., Ross, S. M., Walter, J. W., McDonald, A. J., & Wang, L. W. (2002). Tennessee Technology Literacy Challenge Fund: Evaluation Report. Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Lowther, D. L., Ross, S. M., Wang, L. W., Strahl, J. D. & McDonald, A. J. (2004). Tennessee Department of Education EdTech Launch 1 2003-2004 Evaluation Report. Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Lumpe, A. T., & Chambers, E. (2001). Assessing teachers' context beliefs about technology use. Journal of Research on Technology in Education, 34(1), 93-107.

Marcinkiewicz, H. R., & Regstad, N. G. (1996). Using subjective norms to predict teachers’ computer use. Journal of Computing in Teacher Education, 13(1), 27-33.

Marvin, E. D., Lowther, D. L., & Ross, S. M. (2002). Technology Skills Assessment. Memphis, TN: Center for Research in Educational Policy, The University of Memphis.

McCain, T. (2005). Teaching for tomorrow: Teaching content and problem-solving. Thousand Oaks, CA: Corwin Press.

Morrison, G. R. & Lowther, D. L. (2005). Integrating computer technology into the classroom (3rd ed.). Englewood Cliffs, NJ: Pearson/Merrill/Prentice Hall.

National Center for Education Statistics (NCES). (2000). Teachers' tools for the 21st century: A report on teachers' use of technology. Washington, DC: U.S. Government Printing Office.

Newman, H. (2002, February 26). Computers used more to learn than teach. Detroit Free Press. Retrieved, June 7, 2007 from http://www.freepress.com/news/education/newman26_20020226.htm

Norris, C., Sullivan, T., Poirot, J., & Soloway, E. (2003). No access, no use, no impact: Snapshot surveys of educational technology in K-12. Journal of Research on Technology in Education, 36(1), 15-27.

North Central Regional Educational Laboratory (NCREL). (2000). enGauge: A framework for effective technology use in schools [Brochure]. Oak Brook, IL:NCREL.

Onwuegbuzie, A., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tashakkori and C. Teddlie (Eds.). Handbook of mixed methods in social and behavioral research (pp. 351-383). Thousand Oaks, Sage. Partnership for 21st Century Skills (2007). Partnership for 21st Century skills. Retrieved January 7, 2008 from http://www.21stcenturyskills.org

Pierson, M. E. (2001). Technology integration practice as a function of pedagogical expertise. Journal of Research on Computing in Education, 33(4), 413-430.

Rakes, G. C., Flowers, B. F., Casey, H. B., & Santana, R. (1999). An analysis of instructional technology use and constructivist behaviors in K-12 teachers. International Journal of Educational Technology, 1(2), 1-18

Ringstaff, C., & Kelly, L. (2002). The learning return on our educational technology investment: A review of findings from research. San Francisco, CA: WestEd RTEC.

http://www.freepress.com/news/education/newman26_20020226.htm

http://www.21stcenturyskills.org/

20

Roschelle, J. M., Pea, R. D., Hoadley, C. M., Gordin, D. N., & Means, B. M. (2000). Changing how and what children learn in school with computer-based technologies. The Future of Children, 10(2), 76-101.

Ross, S. M., & Lowther, D. L. (2003). Formative evaluation process for school improvement: Technology instrument suite. Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Ross, S. M., Lowther, D. L., Wang, L. W., Strahl, J. D. & McDonald, A. J. (2004). Tennessee Department of Education EdTech Launch 1 2003-2004 Evaluation Report. Memphis, TN: The University of Memphis, Center for Research in Educational Policy.

Ross, S. M., Smith, L., Alberg, M., & Lowther, D. (2004) Using classroom observations as a research and formative evaluation tool in educational reform: The school observation measure. In S. Hilberg and H. Waxman (Eds.) New Directions for Observational Research in Culturally and Linguistically Diverse Classrooms (pp. 144-173). Santa Cruz, CA: Center for Research on Education, Diversity & Excellence.

Ross, S. M., Smith, L. J., & Alberg, M. (1999). The School Observation Measure (SOM©). Memphis, TN: Center for Research in Educational Policy, The University of Memphis.

Sandholtz, J. H., & Reilly, B. (2004). Teachers, not technicians: Rethinking technical expectations for teachers. Teachers College Record, 106(3), 487-512.

Smith, L., Ross, S., McNelis, M., Squires, M., Wasson, R., Maxwell, S., Weddle, K., Nath, L., Grehan, A., & Buggey, T. (1998). The Memphis restructuring initiative: Analysis of activities and outcomes that impact implementation success. Education and Urban Society 30(3), 296-325.

Sterbinsky, A., & Ross, S. M. (2003). Summary of CSRTQ Reliability Studies. Technical Report. Memphis, TN: Center for Research in Educational Policy, The University of Memphis.

Strauss, A., & Corbin, J. (1998). Basics of qualitative research: Techniques and procedures for developing grounded theory (2nd ed.). Newbury Park, CA: Sage.

Swanson, C. B. (2006, May 4). Tracking U.S. Trends: How Education Week Graded the States. Education Week: Technology Counts, 25(25), 56.

Tennessee Department of Education (TN DOE) (2003) Enhancing Education Through Technology: EdTech LAUNCH (Leading All Users to New Challenging Heights) (TnETL). Retrieved August 2, 2007 from http://tennessee.gov/education/news/nr/2004/nr040506.php

21st Century Education System Task Force (2007). Maximizing the impact: The pivotal role of technology in a 21st Century education system. International Society for Technology in Education, Partnership for 21st Century Skills, and State Education Technology Directors Association. Retrieved January 27, 2008 from http://www.setda.org/web/guest/maximizingimpactreport

U. S. Department of Commerce, Economics and Statistics Administration (ESA) (2003). Digital Economy 2003. Washington, DC: Author. Retrieved January 17, 2008 from https://www.esa.doc.gov/reports/DE-Chap4.pdf.

U. S. Department of Education (DOE) (2001). No child left behind Act of 2001: Enhancing Education Through Technology (II-D-1&2). Retrieved August 2, 2007 from http://www.ed.gov/admins/lead/account/nclbreference/page_pg28.html#iid1

U. S. Department of Education (DOE) (2003). Federal funding for educational technology and how it is used in the classroom: A summary of findings from the Integrated Studies of Educational Technology. Washington, D.C: Office of the Under Secretary, Policy and Program Studies Service. Retrieved October 11, 2007 from http://www.ed.gov/about/offices/list/os/technology/evaluation.html

Vannatta, R. A., & Fordham, N. (2004). Teacher dispositions as predictors of classroom technology use. Journal of Research on Technology in Education, 36(3), 253-271.

http://tennessee.gov/education/news/nr/2004/nr040506.php

http://www.setda.org/web/guest/maximizingimpactreport

https://www.esa.doc.gov/reports/DE-Chap4.pdf

https://www.esa.doc.gov/reports/DE-Chap4.pdf

http://www.ed.gov/admins/lead/account/nclbreference/page_pg28.html#iid1

http://www.ed.gov/about/offices/list/os/technology/evaluation.html

21

Van Braak, J. (2001). Individual characteristics influencing teachers' class use of computers. Journal of Educational Computing Research, 25(2), 141-157.

Van Braak, J., Tondeur, J., & Valcke, M. (2000). Explaining different types of computer use among primary school teachers. European Journal of Psychology of Education, 19(4), 407-422.

Van Melle, E., Cimellaro, L., & Shulha, L. (2003). A dynamic framework to guide the implementation and evaluation of educational technologies. Education and Information Technologies, 8(3), 267-285.

Wallace, C. & Steptoe, S. (December 10, 2006). How to bring our schools out of the 20th century. Time Magazine, 168(25). Retrieved January 21, 2008 from http://www.time.com/time/magazine/article/0,9171,1568480,00.html

Wozney, L., Venkatesh, V., & Abrami, P. (2006). Implementing computer technologies: Teachers' perceptions and practices. Journal of Technology and Teacher Education, 14(1), 173-207.

http://www.time.com/time/magazine/article/0%2C9171%2C1568480%2C00.html

22

Appendix A Whole School SOM© Data Summary

L1 Program n = 65 (642 classrooms), L 1 Control n = 65 (643classrooms) L2 Program n = 70 (679 classrooms), L 2 Control n = 69 (691 classrooms)

The extent to which each of the following was used or present in the classroom

Percent Observed

Launch Group None Rarely Occasionally Frequently Extensively

Instructional Orientation

Direct Instruction (lecture)

L1 Program 0.0 6.2 27.7 33.8 32.3 L1 Control 4.6 1.5 12.3 40.0 41.5


Team teaching



Cooperative/collaborative learning



Individual tutoring (teacher, peer, aide, adult volunteer)



Classroom Organization

Ability groups



Multi-age grouping


L2 Program 88.6 4.3 2.9 4.3 0 L2 Control 84.3 4.3 1.4 1.4 8.6

Work centers (for individuals or groups)



Instructional Strategies

Higher level instructional feedback (written or verbal) to enhance student learning



Integration of subject areas (interdisciplinary/thematic units)



23


Percent Observed


Project-based learning



Use of higher-level questioning strategies



Teacher as a coach/facilitator

L1 Program 3.1 16.9 36.9 26.2 16.9 L1 Control 12.3 15.4 40 18.5 13.8


Parent/community involvement in learning activities



Student Activities

Independent seatwork (self-paced worksheets, individual assignments)



Experiential, hands-on learning



Systematic individual instruction (differential assignments geared to individual needs)



Sustained writing / composition (self-selected or teacher-generated topics)



Sustained reading



Independent inquiry / research on the part of students


L2 Program 45.7 44.3 7.1 2.9 0.0 L2 Control 54.3 32.9 10 2.9 0.0

Student discussion



24


Percent Observed


Technology Use

Computer for instructional delivery (e.g. CAI, drill & practice)



Technology as a learning tool or resource (e.g. Internet research, spreadsheet or database creation, multi-media, CD ROM)

L1 Program 16.9 33.8 27.7 20 1.5 L1 Control 56.9 33.8 7.7 1.5 0.0


Assessment

Performance assessment strategies



Student self-assessment (portfolios, individual record books)



Summary Items

Academically focused class time



Level of student attention / interest / engagement



25

Appendix B Whole-School RSCA: Percent of Observed Strategies by Application Strength and Technology Use

L1 Program n = 642, L 1 Control n = 643 L2 Program n = 679, L 2 Control n = 691

% Observed Rating 1 Rating 2 Rating 3 Rating 4 Technology Used

Item Launch Group n % N % n % n % n % Mean n % Cooperative learning

L1 Program 165 25.7 37 15.1 48 19.6 59 24.1 21 8.6 2.39 80 48.4 L1 Control 85 13.2 17 5.9 15 5.2 28 9.7 25 8.7 2.72 21 24.7

L2 Program 162 23.8 18 7 45 17.5 77 30 22 8.6 2.63 73 45 L2 Control 136 19.6 18 7 46 18 58 22.7 14 5.5 2.5 40 29.4

Project based learning

L1 Program 100 15.5 4 1.6 17 16.9 62 25.3 17 6.9 2.92 76 76 L1 Control 29 4 5 1.7 7 2.4 9 3.1 8 2.8 2.68 8 27.6


Higher-level questions

L1 Program 245 38.1 47 19.2 39 15.9 133 54.3 26 10.6 2.56 78 31.8 L1 Control 289 44.9 49 17 38 13.1 135 46.7 67 23.2 2.76 41 14.1

L2 Program 255 37.6 32 12.5 70 27.2 120 46.7 33 12.8 2.6 66 25.8 L2 Control 256 37 51 19.9 80 31.3 92 35.9 33 12.9 2.42 32 12.5

Experiential hands-on


L2 Program 130 19.1 9 3.5 38 14.8 61 23.7 22 8.6 2.73 52 40 L2 Control 165 23.8 22 8.6 50 19.5 72 28.1 21 8.2 2.55 56 33.9

Independent inquiry


L2 Program 73 10.7 9 3.5 15 5.8 37 14.4 12 4.7 2.71 60 82.2 L2 Control 79 11.4 17 6.6 22 8.6 31 12.1 9 3.5 2.41 53 67

Student discussion

L1 Program 148 23 45 18.4 38 15.5 43 17.6 22 9 2.28 37 25 L1 Control 202 31.4 32 11.1 28 9.7 84 29.1 58 20.1 2.83 20 9.9

L2 Program 167 24.6 24 9.3 51 19.8 68 26.5 24 9.3 2.55 61 36.5 L2 Control 173 25 39 15.2 55 21.5 58 22.7 21 8.2 2.35 17 68

Students as producers

L1 Program 93 14.4 20 8.2 12 4.9 42 17.1 19 7.8 2.64 NA NA L1 Control 28 04.3 15 5.2 3 1.0 8 2.8 2 0.7 1.89 NA NA

L2 Program 111 16.3 15 5.8 22 8.6 52 20.2 22 8.6 2.72 NA NA L2 Control 68 09.8 18 7.0 20 7.8 20 7.8 10 3.9 2.32 NA NA

*Rating scale: 1 = limited application; 4 = Strong application.

26

Appendix C Whole School OCU Data Summary

L1 Program n = 70 (642 classrooms), L 1 Control n = 65 (643 classrooms) L2 Program n = 70 (679 classrooms), L 2 Control n = 69 (691 classrooms)

The extent to which each of the following was used or present in the classroom Percent Observed

Launch Group None Rarely Occasionally Frequently Extensively Production Tools

Word Processing

L1 Program 49.2 29.2 20 1.5 0 L1 Control 93.8 4.6 1.5 0 0

L2 Program 60 22.9 14.3 2.9 0 L2 Control 80 14.3 4.3 1.4 0

Database

L1 Program 98.5 1.5 0 0 0 L1 Control 100 0 0 0 0

L2 Program 98.6 0 0 0 0 L2 Control 97.1 2.9 0 0 0

Spreadsheet

L1 Program 92.3 6.2 1.5 0 0 L1 Control 100 0 0 0 0

L2 Program 90 2.9 4.3 0 0 L2 Control 100 0 0 0 0

Draw/Paint/Graphics

L1 Program 93.8 1.5 1.5 3.1 0 L1 Control 100 0 0 0 0

L2 Program 88.6 5.7 2.9 0 0 L2 Control 97.1 2.9 0 0 0

Presentation (e.g., MS PowerPoint)

L1 Program 80 15.4 3.1 1.5 0 L1 Control 100 0 0 0 0


Authoring (e.g., HyperStudio)

L1 Program 96.9 1.5 0 0 0 L1 Control 98.5 0 1.5 0 0


Concept mapping (e.g., Inspiration)

L1 Program 80 15.4 3.1 1.5 0 L1 Control 100 0 0 0 0


Planning (e.g., MS Project)



Other

L1 Program 84.6 12.3 0 0 0 L1 Control 93.8 6.2 0 0 0


Internet/Research Tools

Internet Browser (e.g., Netscape)

L1 Program 26.2 32.3 32.3 7.7 1.5 L1 Control 89.2 10.8 0 0 0

L2 Program 37.1 25.7 22.9 8.6 5.7 L2 Control 62.9 20 10 2.9 4.3

27

The extent to which each of the following was used or present in the classroom Percent Observed


CD Reference (encyclopedias, etc.)

L1 Program 100 0 0 0 0 L1 Control 100 0 0 0 0


Communications

L1 Program 100 0 0 0 0 L1 Control 100 0 0 0 0


Other



Educational Software

Drill / Practice / Tutorial

L1 Program 47.7 26.2 15.4 6.2 4.6 L1 Control 64.6 23.1 10.8 1.5 0


Problem Solving (e.g., SimCity)


L2 Program 85.7 7.1 4.3 2.9 0 L2 Control 92.9 7.1 0 0 0

Process Tools (Geometer’s Sketchpad, etc.)


L2 Program 90 0 7.1 1.4 0 L2 Control 97.1 2.9 0 0 0

Other


L2 Program 87.1 5.7 4.3 0 0 L2 Control 92.9 5.7 0 0 1.4

Testing Software

Individualized / Tracked (e.g., Accelerated Reader)

L1 Program 80 6.2 7.7 6.2 0 L1 Control 73.8 15.4 9.2 0 1.5

L2 Program 64.3 20 8.6 4.3 1.4 L2 Control 58.6 22.9 12.9 0 5.7

Generic


L2 Program 98.6 0 1.4 0 0 L2 Control 98.6 1.4 0 0 0

Other

L1 Program 80 6.2 7.7 6.2 0 L1 Control 98.5 1.5 0 0 0

L2 Program 95.7 0 1.4 1.4 0 L2 Control 100 0 0 0 0

28

Overall Rubric The extent to which each of the following was used or present in the classroom Percent Observed

Level of Meaningful Use Launch Group None Rarely Occasionally Frequently Extensively

Low level use of computers

L1 Program 60 16.9 12.3 10.8 0 L1 Control 76.9 12.3 9.2 0 1.5

L2 Program 52.9 28.6 11.4 2.9 1.4 L2 Control 60 20 18.6 1.4 0

Somewhat meaningful use of computers

L1 Program 52.3 23.1 15.4 6.2 3.1 L1 Control 80 13.8 4.6 1.5 0


Meaningful use of computers


L2 Program 28.6 22.9 20 22.9 5.7 L2 Control 47.1 27.1 11.4 11.4 2.9

Very meaningful use of computers

L1 Program 63.1 20 7.7 6.2 3.1 L1 Control 87.7 6.2 4.6 1.5 0

L2 Program 54.3 21.4 12.9 8.6 2.9 L2 Control 84.3 8.6 4.3 0 2.9

29

Appendix D Teacher Technology Questionnaire (TTQ)

L1 Program N = 393, L 1 Control N = 401 L2 Program N = 418, L 2 Control N = 426

Items by Category % Strongly Agree + Agree Launch Program Control

Impact on Classroom Instruction

My teaching is more student-centered when technology is integrated into the lessons 1 68.7 45.1 2 71.1 66.2

I routinely integrate the use of technology into my instruction. 1 79.4 51.4 2 81.1 68.8

Technology integration efforts have changed classroom learning activities in a very positive way. 1 84.7 66.3 2 86.8 79.1

My teaching is more interactive when technology is integrated into the lessons. 1 74.3 51.6 2 72.7 68.5

Overall Impact on Classroom Instruction 1 4.02 3.49 2 4.08 3.81

Impact on Students

The use of computers has increased the level of student interaction and/or collaboration. 1 87.3 63.1 2 90.0 78.6

The integration of technology has positively impacted student learning and achievement. 1 88.3 69.6 2 90.7 84.0

Most of my students can capably use computers at an age-appropriate level. 1 93.6 79.8 2 93.1 85.4

The use of technology has improved the quality of student work. 1 72.0 53.4 2 71.3 65.5

Overall Impact on Students 1 4.2 3.73 2 4.26 3.97

Teacher Readiness to Integrate Technology

I know how to meaningfully integrate technology into lessons. 1 93.1 73.1 2 89.7 78.4

I am able to align technology use with my district's standards-based curriculum. 1 86.8 63.3 2 88.5 75.8

I have received adequate training to incorporate technology into my instruction. 1 88.8 61.3 2 87.8 70.2

My computer skills are adequate to conduct classes that have students using technology. 1 89.8 73.1 2 89.0 80.8

Overall Teacher Readiness to Integrate Technology 1 4.25 3.71 2 4.29 3.9

Support for Technology in the School

Parents and community members support our school's emphasis on technology. 1 75.6 56.9 2 83.3 73.5

Teachers receive adequate administrative support to integrate technology into classroom practices. 1 86.8 65.3 2 88.5 78.9

Our school has a well-developed technology plan that guides all technology integration efforts. 1 76.8 42.1 2 82.8 53.3

Teachers in this school are generally supportive of technology integration efforts. 1 89.1 74.6 2 92.1 86.4

Overall Support for Technology in the School 1 4.09 3.63 2 4.23 3.87

Technical Support

Most of our school computers are kept in good working condition. 1 81.2 73.3 2 91.6 71.6

I can readily obtain answers to technology-related questions. 1 82.4 73.1 2 88.8 75.4

My students have adequate access to up-to-date technology resources. 1 85.5 55.9 2 89.7 63.4

30

Items by Category % Strongly Agree + Agree Launch Program Control

Materials (e.g., software, printer supplies) for classroom use of computers are readily available. 1 78.1 50.9 2 82.5 61.0

Overall Technical Support 1 4.1 3.6 2 4.2 3.7

Technology Coach Effectiveness I have frequently participated in professional development that was planned by or provided by my Technology Coach.

1 84.7 NA 2 82.8 NA

I more frequently integrate technology into my instruction as a result of participating in professional development planned or provided by my Technology Coach.

1 78.9 NA 2 79.4 NA

The quality of my technology integration lessons has improved as a result of participating in professional development planned or provided by my Technology Coach.

1 80.4 NA 2 82.8 NA

Overall, my Technology Coach has been a valuable asset to our school’s technology integration program.

1 89.1 NA 2 90.0 NA

Technology Coach Effectiveness Overall 1 4.20 NA 2 4.28 NA

*Note: Item percentages may not total 100% because of missing input from some respondents.

Section 2: Percent of Response by Rating Launch 1 Launch 2 Item Ratings Program Control Program Control

How would you rate your level of computer ability?

Very Good 32.1 19.2 33.0 18.3

Good 42.8 36.7 42.8 41.3

Moderate 22.1 34.9 22.7 32.9

Poor 1.8 7.5 1.2 5.9

No Ability 0.0 0.5 0.0 0.2

Do you own a home computer? Yes 93.9 92.3 92.3 93.0

No 5.9 7.0 6.9 5.4

If yes, do you use your home computer to access instructional materials on the Internet?

Yes 86.4 82.2 89.9 83.6

No 11.1 14.3 9.1 12.6

If yes, do you use your home computer to prepare instructional materials?

Yes 83.2 77.6 86.5 80.1

No 14.1 17.0 12.7 14.4

31

Appendix E Technology Skills Assessment (TSA) Data Summary

L1 Program Teachers N = 394, L 1 Control Teachers N = 402 L2 Program Teachers N = 422, L 2 Control Teachers N = 429

TSA Item by Category Not at All Somewhat Very Easily

Group L1 L2 L1 L2 L1 L2 Computer Basics ~ How easily can you…

Use a spell check tool. Program 1.8 1.9 7.1 4.5 91.1 93.4 Control 5.0 4.2 11.9 9.8 82.8 85.8

Create basic computer documents (word processed) in a timely manner. Program 0.8 1.9 9.6 7.1 89.3 91.0 Control 5.5 3.5 12.7 10.7 81.6 85.5

Use help menus for software programs. Program 1.8 2.6 20.3 21.3 77.7 75.8 Control 6.2 4.4 27.1 27.3 65.9 67.4

Use basic computer terms like mouse, keyboard, hard drive, CD-ROM, and monitor.

Program 0.8 0.5 4.3 4.0 94.4 95.0 Control 0.7 0.5 9.2 9.6 89.8 89.7

Save documents so they can be opened on both a Macintosh and PC. Program 8.9 14.2 23.6 23.5 67.3 62.1 Control 14.4 12.1 24.4 24.5 60.4 62.9

Create folders on a hard drive or disk. Program 1.8 4.5 14.2 19.9 84.0 75.4 Control 10.2 9.6 25.6 24.9 63.4 65.0

Save files to specific folders. Program 1.0 3.8 10.4 14.9 88.6 80.8 Control 1.8 1.9 7.1 4.5 91.1 93.4

Locate and delete unwanted files. Program 1.0 3.1 12.7 13.7 85.8 83.2 Control 7.2 6.1 18.7 17.9 72.9 75.8

Use keyboard commands to cut, copy, or delete text. Program 2.3 2.8 15.2 14.5 82.5 82.7 Control 8.5 6.3 20.4 19.3 70.6 73.9

Proficiently use a mouse and keyboard. Program 0.3 0.2 4.8 3.8 94.7 96.0 Control 1.2 0.2 9.5 6.3 88.6 93.0

Print a document using “Print” from the File menu and/or the toolbar icon.

Program 0.3 0.5 3.8 2.4 95.9 97.2 Control 1.2 1.2 5.2 3.3 93.3 95.3

Computer Basics: Overall Program 1.89 3.27 11.45 11.78 86.48 84.37 Control 6.25 4.94 17.16 16.03 76.01 78.56

Software Basics ~ How easily can you…

Use software preview features to check work. Program 2.3 6.2 22.3 24.4 72.3 69.4 Control 10.5 12.6 29.1 25.6 59.7 60.4

Open and use software programs that are installed on your computer. Program 0.8 0.9 10.7 13.0 88.6 86.0 Control 2.0 2.6 19.2 16.6 78.6 80.7

Work with and move between two open programs (e.g., Internet and database) to create a product.

Program 3.0 5.0 17.3 18.2 79.7 76.5 Control 14.9 13.5 25.1 22.4 59.2 63.9

Describe the difference between downloading and installing software. Program 4.8 5.9 22.3 26.5 72.8 67.5 Control 8.7 8.2 30.1 30.1 60.0 61.1

Save documents so they can be opened in a different program (e.g., from Word to Word Perfect).

Program 9.6 14.0 30.5 34.4 59.9 51.7 Control 25.1 21.4 30.6 33.1 43.5 45.2

Install software. Program 3.6 9.2 23.6 30.8 72.6 59.7 Control 10.7 10.3 33.8 31.0 54.5 58.3

Software Basics: Overall Program 4.01 6.86 21.11 24.55 74.31 67.46 Control 11.98 11.93 27.98 26.46 59.25 61.60

Multimedia Basics ~ How easily can you…

Import digital video from a camera to a computer. Program 22.8 28.0 32.5 33.2 44.4 38.9 Control 40.5 36.8 29.4 30.3 29.9 32.6

Record and save your voice onto a computer. Program 37.3 48.3 32.5 29.9 29.9 21.8 Control 62.7 54.3 21.6 28.2 15.2 17.2

Use a scanner to import a photo or document into a computer. Program 16.2 21.3 31.5 35.5 52.0 43.1 Control 32.8 31.7 30.3 32.9 36.1 35.2

32


Group L1 L2 L1 L2 L1 L2

Play a music CD on the computer. Program 3.6 5.0 9.4 15.2 86.3 79.6 Control 7.0 7.7 18.2 16.6 74.4 75.1

Multimedia Basics: Overall Program 19.97 29.90 26.47 28.45 53.15 45.85 Control 35.75 32.62 24.87 27.00 38.90 40.25

Internet Basics ~ How easily can you…

Connect to the Internet with a modem (phone, cable). Program 5.8 8.1 13.2 17.3 81.0 74.6 Control 12.2 10.7 12.7 13.5 74.4 75.5

Use Boolean strategies for Internet searches. Program 27.4 37.4 24.4 21.8 46.7 39.1 Control 46.0 42.7 23.1 20.0 29.4 36.4

Use appropriate software and the Internet to find audio, video, and graphics for lesson plans.

Program 6.6 6.4 27.4 28.2 65.7 64.9 Control 15.7 15.9 37.3 32.9 46.5 50.8

Use the Internet to find help when you have a computer program. Program 17.0 19.9 32.7 38.6 50.0 41.5 Control 28.1 27.7 34.3 33.3 37.1 38.7

Determine if information you find on the Internet is accurate and valid. Program 10.2 10.9 36.3 35.8 53.3 53.3 Control 14.4 20.0 42.0 37.1 40.3 42.7

Evaluate Internet search strategies to determine those that are most efficient.

Program 9.4 10.9 36.5 34.8 53.8 54.3 Control 18.4 20.0 39.6 38.0 41.5 41.7

Determine the usefulness and appropriateness of digital information Program 15.0 18.0 33.8 36.7 51.0 45.3 Control 26.9 23.3 37.8 41.3 34.1 34.7

Internet Basics: Overall Program 13.05 15.94 29.18 30.45 57.35 53.28 Control 22.92 22.90 32.40 30.87 43.32 45.78

Advanced Skills ~ How easily can you…

Use more advanced computer terms like megahertz, gigabytes, and RAM. Program 20.8 29.6 55.6 48.1 23.4 45.3 Control 38.1 36.1 41.3 41.0 19.7 22.6

Access information on local area networks (LANs) and wide area networks (WANs).

Program 30.2 36.5 39.1 36.7 29.4 26.3 Control 40.5 42.9 35.6 28.2 22.9 28.4

Use appropriate digital layout and design to meet the needs of defined audiences.

Program 28.7 34.8 38.8 36.7 32.2 27.3 Control 52.0 47.1 29.9 27.0 17.2 25.2

Use appropriate digital layout and design for the selected media (e.g., multimedia, web, print).

Program 27.2 31.3 39.3 37.0 33.2 30.8 Control 48.0 43.8 34.6 30.1 16.4 25.2

Publish information in a variety of media (e.g., printed, monitor display, web-based, video).

Program 22.3 23.5 40.6 39.8 36.3 36.0 Control 44.3 38.0 33.8 36.6 20.9 24.5

Connect a computer to a local server to share files. Program 34.0 37.0 34.8 29.9 30.5 32.7 Control 53.0 45.5 28.4 28.2 17.4 25.4

Determine if a software program works with an operating system. Program 27.4 32.7 38.1 36.7 34.0 29.4 Control 41.0 41.7 36.8 31.9 20.9 25.6

Print to a specific printer when connected to a network that has more than one printer.

Program 14.2 15.9 21.6 22.7 63.7 60.7 Control 23.4 26.6 30.3 25.4 45.0 47.8

Use presentation software to share information with specific audiences. Program 14.2 17.8 30.5 32.2 54.6 49.5 Control 36.8 32.6 31.3 29.4 30.6 37.5

Advanced Skills: Overall Program 24.33 37.01 37.60 39.07 37.47 37.55 Control 41.90 39.36 33.55 30.86 23.44 29.13

Using Technology for Learning ~ How easily can you…

Use multimedia software to enhance learning experiences. Program 6.9 7.6 31.5 33.6 61.7 58.1 Control 16.9 14.5 46.3 43.6 35.3 41.0

Use appropriate software (e.g., word processing, graphics, databases, spreadsheets, simulations, and multimedia) to express ideas and solve problems.

Program 7.9 9.5 33.0 34.4 58.9 55.7

Control 18.4 16.1 41.3 41.7 38.6 41.5

Use text and graphics to create and modify solutions to problems. Program 13.7 17.8 39.1 37.9 47.2 43.4 Control 33.3 29.6 38.6 40.1 26.9 29.6

Use digital audio and video to create and modify solutions to problems. Program 26.1 30.3 42.1 41.0 31.0 27.7 Control 47.3 42.4 33.8 33.1 17.7 23.5

Use communication tools to participate in group projects. Program 17.0 18.5 35.5 37.4 46.7 42.7 Control 37.6 31.2 37.3 37.8 22.9 30.3

33


Group L1 L2 L1 L2 L1 L2 Manipulate information in interactive digital environments (e.g., simulations, virtual labs, field trips).

Program 27.9 32.5 37.3 36.7 34.0 30.1 Control 49.8 45.5 32.3 34.5 16.4 19.3

Participate in a listserv, chat, and bulletin board session. Program 28.9 32.9 36.5 34.8 34.0 31.3 Control 45.3 48.5 30.6 29.8 21.4 20.5

Create an electronic teaching portfolio to evaluate your work. Program 31.5 46.4 37.6 30.8 30.2 21.6 Control 61.7 56.4 25.6 26.1 11.2 16.8

Evaluate electronic portfolio products. Program 33.2 46.9 35.8 31.0 29.9 21.1 Control 63.2 59.2 24.1 24.2 10.7 15.6

Create technology tools to assess student work (e.g., checklists, timelines, rubrics).

Program 13.7 16.1 30.5 32.2 55.3 50.9 Control 37.6 30.3 33.3 33.3 27.6 35.7

Using Technology for Learning: Overall Program 20.68 25.85 35.89 34.98 42.89 38.26 Control 41.11 37.37 34.32 34.42 22.87 55.13

Policy and Ethics - I understand…

My school’s acceptable use policy. Program 1.3 2.1 16.2 17.1 81.5 79.1 Control 2.7 3.5 25.9 25.9 68.9 69.7

The concept of a school site license for software. Program 4.1 2.6 16.8 22.0 78.4 73.9 Control 7.2 7.7 26.9 26.1 62.9 65.0

How to determine if it is legal to copy a software program or another individual’s electronic work.

Program 6.9 7.8 32.5 35.3 59.9 55.2 Control 14.9 14.5 36.3 37.5 46.3 47.1

Policy and Ethics: Overall Program 4.10 4.16 21.83 24.80 73.26 69.40

Control 8.26 8.56 29.70 29.83 59.36 60.60

34

Appendix F Student* Technology Task Frequencies, Percentages, and Means

L1 Program Students n = 79, L 1 Control Students n = 54 L2 Program Students n = 64, L 2 Control Students n = 70

Completion of Technology Task Total 0 = No 1 = Somewhat 2 = Yes Overall

n n % n % n % M SD

SPREADSHEETS

Enter numerical data into spreadsheet cells? L1 Program 79 20 25.3 1 1.3 58 73.4 1.48 0.87 L1 Control 54 4 7.4 5 9.3 45 83.3 1.75 0.58

L2 Program 64 18 28.1 4 6.3 42 65.6 1.37 0.89 L2 Control 70 7 10.0 1 1.4 62 88.6 1.78 0.61 Place column names into correct cells? L1 Program 79 21 26.6 0 0 58 73.4 1.46 0.88 L1 Control 54 5 9.3 6 11.1 43 79.6 1.70 0.63

L2 Program 64 18 28.1 12 18.8 34 53.1 1.25 0.87 L2 Control 70 9 12.9 1 1.4 60 85.7 1.72 0.67 Place row names in correct cells? L1 Program 79 20 25.3 1 1.3 58 73.4 1.48 0.87 L1 Control 54 5 9.3 2 3.7 47 87.0 1.77 0.60

L2 Program 64 18 28.1 3 4.7 43 67.2 1.39 0.90 L2 Control 70 7 10.0 1 1.4 62 88.6 1.78 0.61 Use a spreadsheet formula to calculate the average of a column of numbers? L1 Program 79 29 36.7 3 3.8 47 59.5 1.22 0.96 L1 Control 54 15 27.8 2 3.7 37 68.5 1.40 0.90

L2 Program 64 25 39.1 0 0 39 60.9 1.21 0.98 L2 Control 70 18 25.7 2 2.9 50 71.4 1.45 0.87 Create a chart? L1 Program 79 15 19.0 6 7.6 58 73.4 1.54 0.79 L1 Control 54 14 25.9 1 1.9 39 72.2 1.46 0.88

L2 Program 64 11 17.2 4 6.3 49 76.6 1.59 0.77 L2 Control 70 8 11.4 4 5.7 58 82.9 1.71 0.66 Change the color of columns in a column chart? L1 Program 79 17 21.5 0 0 62 78.5 1.56 0.82 L1 Control 54 14 25.9 1 1.9 39 72.2 1.46 0.88

L2 Program 64 13 20.3 1 1.6 50 78.1 1.57 0.81 L2 Control 70 11 15.7 0 0 59 84.3 1.68 0.73 Add a title to a chart? L1 Program 78 14 17.7 1 1.3 63 79.7 1.62 0.77 L1 Control 54 18 33.3 0 0 36 66.7 1.33 0.95

L2 Program 64 7 10.9 0 0 57 89.1 1.78 0.62 L2 Control 70 15 21.4 4 5.7 51 72.9 1.51 0.82 Add a title to a chart axis? L1 Program 79 22 27.8 2 2.5 55 69.6 1.41 0.90 L1 Control 54 19 35.2 1 1.9 34 63.0 1.27 0.95

L2 Program 64 19 29.7 1 1.6 44 68.8 1.39 0.91 L2 Control 70 31 44.3 0 0 39 55.7 1.11 1.00 Change the range of the Y-axis scale? L1 Program 79 29 36.7 0 0 50 63.3 1.26 0.97 L1 Control 54 20 37.0 2 3.7 32 59.3 1.22 0.96

L2 Program 64 19 29.7 1 1.6 44 68.8 1.39 0.91 L2 Control 70 17 24.3 0 0 53 75.7 1.51 0.86

35


n n % n % n % M SD Change the location of the legend? L1 Program 79 21 26.6 4 5.1 54 68.4 1.41 0.88 L1 Control 54 17 31.5 0 0 37 68.5 1.37 0.93

L2 Program 64 13 20.3 6 9.4 45 70.3 1.50 0.81 L2 Control 70 20 28.6 3 4.3 47 67.1 1.38 0.90

PRESENTATION

Add a title to a slide? L1 Program 79 10 12.7 0 0 69 87.3 1.74 0.66 L1 Control 54 7 13.0 0 0 47 87.0 1.74 0.67

L2 Program 62 2 3.10 0 0 60 93.8 1.93 0.35 L2 Control 70 13 18.6 0 0 57 81.4 1.62 0.78 Add a slide to a presentation? L1 Program 79 8 10.1 0 0 71 89.9 1.79 0.60 L1 Control 54 16 29.6 0 0 38 70.4 1.40 0.92

L2 Program 62 4 6.3 0 0 58 90.6 1.87 0.49 L2 Control 70 14 20.0 0 0 56 80 1.60 0.80 Insert a clipart image or photograph to a slide? L1 Program 79 14 17.7 0 0 65 82.3 1.64 0.76 L1 Control 54 9 16.7 0 0 45 83.3 1.66 0.75

L2 Program 62 7 10.9 0 0 55 85.9 1.77 0.63 L2 Control 70 15 21.4 0 0 55 78.6 1.57 0.82 Change the font of a text within a presentation? L1 Program 79 22 27.8 0 0 57 72.2 1.44 0.90 L1 Control 54 27 50.0 0 0 27 50.0 1.00 1.00

L2 Program 62 21 32.8 0 0 41 64.1 1.32 0.95 L2 Control 70 37 52.9 0 0 33 47.1 0.94 1.00 Change the size of text within a presentation? L1 Program 79 28 35.4 0 0 51 64.6 1.29 0.96 L1 Control 54 26 48.1 0 0 28 51.9 1.03 1.00

L2 Program 62 24 37.5 0 0 38 59.4 1.22 0.98 L2 Control 70 35 50.0 0 0 35 50.0 1.00 1.00 Bold text within a presentation? L1 Program 79 17 21.5 0 0 62 78.5 1.56 0.82 L1 Control 54 31 57.4 0 0 23 42.6 0.85 0.99

L2 Program 62 11 17.2 0 0 51 79.7 1.64 0.77 L2 Control 70 19 27.1 0 0 51 72.9 1.45 0.89 Insert a Microsoft Excel spreadsheet chart onto a slide? L1 Program 79 22 27.8 3 3.8 54 68.4 1.40 0.89 L1 Control 54 22 40.7 0 0 32 59.3 1.18 0.99

L2 Program 62 8 12.5 9 14.1 45 70.3 1.59 0.71 L2 Control 70 18 25.7 5 7.1 47 67.1 1.41 0.87 Arrange content on a slide as a bullet list? L1 Program 79 13 16.5 11 13.9 55 69.6 1.53 0.76 L1 Control 54 17 31.5 7 13.0 30 55.6 1.24 0.90

L2 Program 62 7 10.9 12 18.8 43 67.2 1.58 0.69 L2 Control 70 18 25.7 17 24.3 35 50.0 1.24 0.84 Add a design template to a presentation? L1 Program 79 47 59.5 1 1.3 31 39.2 0.79 0.97 L1 Control 54 27 50.0 0 0 27 50.0 1.00 1.00

L2 Program 62 24 37.5 1 1.6 37 57.8 1.20 0.97 L2 Control 70 46 65.7 0 0 24 34.3 0.68 0.95 Select and use relevant images? L1 Program 79 10 12.7 1 1.3 68 86.1 1.73 0.67 L1 Control 54 15 27.8 1 1.9 38 70.4 1.42 0.90

L2 Program 62 8 12.5 54 84.4 62 96.9 1.74 0.67 L2 Control 70 14 20.0 1 1.4 54 77.1 1.57 0.81

36


n n % n % n % M SD

INTERNET

Navigate to a web site given a specific web address (URL)? L1 Program 79 10 12.7 0 0 69 87.3 1.74 0.66 L1 Control 54 17 31.5 0 0 37 68.5 1.37 0.93

L2 Program 62 6 9.4 0 0 56 87.5 1.80 0.59 L2 Control 70 14 20.0 0 0 56 80.0 1.60 0.80 Obtain an image from a website and use it in a document? L1 Program 79 8 10.1 0 0 71 89.9 1.79 0.60 L1 Control 54 13 24.1 0 0 41 75.9 1.51 0.86

L2 Program 62 6 9.4 0 0 56 87.5 1.80 0.59 L2 Control 70 14 20.0 0 0 56 80.0 1.60 0.80

* 8th Grade Students

does technology integration “work” when key barriers are ... · according to the three...

Documents