Outcomes of a Statewide Outcomes of a Statewide Educational Interactive Educational Interactive

Video Program EvaluationVideo Program Evaluation

Mark HawkesMark HawkesDakota State UniversityDakota State University

Annual Conference of the America Evaluation AssociationAnnual Conference of the America Evaluation AssociationNovember 5, 2003November 5, 2003

The Proliferation of Statewide The Proliferation of Statewide Interactive Video SystemsInteractive Video Systems

IllinoisIllinois Illinois Video Education Illinois Video Education Network (IVEN)Network (IVEN)

MissouriMissouri Missouri Research and Missouri Research and Education Network (MOREnet)Education Network (MOREnet)

MontanaMontana Montana Educational Montana Educational Telecommunications Network Telecommunications Network METNET)METNET)

New New HampshireHampshire

Granite State Distance Granite State Distance Learning NetworkLearning Network

New JerseyNew Jersey Garden State Distance Garden State Distance Learning ConsortiumLearning Consortium

I-TV Proliferation . . .I-TV Proliferation . . .

OhioOhio Ohio Telecommunity InitiativeOhio Telecommunity Initiative

OklahomaOklahoma OneNetOneNet

OregonOregon State Distance Education State Distance Education ProgramProgram

TexasTexas Texas Education Texas Education Telecommunications Network Telecommunications Network (TETN)(TETN)

West West VirginiaVirginia

West Virginia Education West Virginia Education NetworkNetwork

DDN Room—Dakota Valley

DDN Room—Sturgis

DDN Room—Madison, H.S.

DDN Room—Lemmon H.S.

DDN Room—Redfield H.S.

DDN Room—Pollock H.S.

DDN Room—Watertown H.S.

DDN Room—Watertown M.S.

DDN Room—Canton H.S.

DDN Room—Shannon Co. H.S.

DDN Room—DeSmet H.S.

DDN Room—Faith H.S.

Study ProcessStudy Process

Funded by the South Dakota Funded by the South Dakota Department of Education through Department of Education through SDADE, which is South Dakota's SDADE, which is South Dakota's Federally Funded Star Schools Federally Funded Star Schools Project.Project.

Major focus of evaluation centers on Major focus of evaluation centers on use and application of the Digital use and application of the Digital Dakota Network.Dakota Network.

Evaluation MethodsEvaluation Methods

32 high schools and elementary 32 high schools and elementary schools visited statewide by schools visited statewide by evaluator pairs.evaluator pairs.

Data collection commenced in Data collection commenced in February and ended in May 2002, February and ended in May 2002, with site visit summaries constructed with site visit summaries constructed August through December 2002.August through December 2002.

Qualitative methods, including Qualitative methods, including interview, focus groups, observation, interview, focus groups, observation, and hardware audit were used.and hardware audit were used.

Summary Statistics of Study SitesSummary Statistics of Study Sites

Average Range State Ave.

Community Size 7,278 339-59,607 1,833

Student Population 896 131-4,000 713.7

Certified Teachers 64 12-249 51.6

Student to computer ratio 2.41:1 .90-4.79:1 3:1

Available seating for DDN 20 8-75 NA

DDN Instructional UsesDDN Instructional Uses

Collaboration activities (research Collaboration activities (research data comparisons, foreign language data comparisons, foreign language exchanges)exchanges)

Enrichment activities (Dr. Seuss Day, Enrichment activities (Dr. Seuss Day, Mystery Reader, performances, Mystery Reader, performances, Buffalo Round-up)Buffalo Round-up)

Expert access (Smithsonian, EROS)Expert access (Smithsonian, EROS) High School course deliveryHigh School course delivery College course deliveryCollege course delivery

DDN Logistical UsesDDN Logistical Uses Teacher professional development Teacher professional development

activitiesactivities Meetings between Educational service Meetings between Educational service

providers (LOFTI) providers (LOFTI) Regional consortia meetings between Regional consortia meetings between

administrators or teachersadministrators or teachers Community “Cracker Barrel” sessionsCommunity “Cracker Barrel” sessions Athletic seeding and bracketingAthletic seeding and bracketing Community organization use (mental Community organization use (mental

health services)health services)

Visual map of DDN use on instructional Visual map of DDN use on instructional and logistical dimensions and logistical dimensions

Over-arching Evaluation FindingsOver-arching Evaluation Findings

South Dakota schools are filled with South Dakota schools are filled with educational technologies and folks educational technologies and folks who can robustly use them who can robustly use them

The DDN system is an underutilized The DDN system is an underutilized resource in most schools resource in most schools

Conditions for optimal I-TV are Conditions for optimal I-TV are consistent from site to siteconsistent from site to site

Barriers to use are generally local Barriers to use are generally local and not system-relatedand not system-related

Lack of accessLack of access Inconsistent bell scheduleInconsistent bell schedule Attitude the interactive video is just for Attitude the interactive video is just for

small schoolssmall schools Lack of incentives for teacher/instructorLack of incentives for teacher/instructor Difficulty of distance delivery coursesDifficulty of distance delivery courses Suspicion that new state administration Suspicion that new state administration

will not support the DDNwill not support the DDN Belief that system replaces teaching peers Belief that system replaces teaching peers

and postionsand postions

Barriers to be AddressedBarriers to be Addressed

Conditions for Exceptional UseConditions for Exceptional Use

Exceptional planningExceptional planning Remote site supervisionRemote site supervision Teacher incentivesTeacher incentives Motivated studentsMotivated students Principals and superintendents with Principals and superintendents with

foresightforesight Activating and leveraging local Activating and leveraging local

partnershipspartnerships

State Delivery ModelState Delivery Model

Regional Delivery ModelRegional Delivery Model

Regional or State Delivery:Regional or State Delivery:Criteria for AnalysisCriteria for Analysis

Bell and school schedule compatibilityBell and school schedule compatibility Flexibility of interaction technologies usedFlexibility of interaction technologies used Duration of consortia affiliationsDuration of consortia affiliations Activity level of current consortiaActivity level of current consortia Access to remote instructional sites Access to remote instructional sites Teacher availability Teacher availability DemographicsDemographics Distribution of operation costs (technical Distribution of operation costs (technical

support, bridges, etc.) support, bridges, etc.)

Criteria . . .Criteria . . .

Nature and prevalence of professional peer Nature and prevalence of professional peer group collaborations (professional group collaborations (professional development, special education, development, special education, conferences, etc.)conferences, etc.)

Coordination of extra-curricular activities Coordination of extra-curricular activities (athletics, administrative, etc.)(athletics, administrative, etc.)

Availability of I-TV operation expertise Availability of I-TV operation expertise Participation and support of community Participation and support of community

partners—or capacity to attract the supportpartners—or capacity to attract the support

Effects of Regional Delivery Effects of Regional Delivery ModelModel

Community viabilityCommunity viability Remote site contactsRemote site contacts Distributed expertiseDistributed expertise Re-energized cooperativesRe-energized cooperatives Tailored I-TV curriculumTailored I-TV curriculum Better interactions with remote Better interactions with remote

supervisorssupervisors

Role for a Statewide Course Role for a Statewide Course VendorVendor

Course coordination and brokerageCourse coordination and brokerage Delivery of specialized coursesDelivery of specialized courses

E. Rogers Stages of Innovation E. Rogers Stages of Innovation Adoption ModelAdoption Model

AwarenessInterestEvaluationTrialAdoption

Report Prepared by Joanne Ustad in completion of the M.S. in Educational Technology, Dakota State University

Elementary Stages of Adoption

Aw areness24%

Interest32%

Evaluation32%

Trial12%

Adoption0%

Middle School Stages of Adoption

Aw areness61%

Interest24%

Evaluation10%

Trial5%

Adoption0%

High School Stages of Adoption

Aw areness31%

Interest31%

Evaluation26%

Trial7%

Adoption5%

Limitations of the StudyLimitations of the Study The self-selection process may bias and The self-selection process may bias and

results toward the more progressive results toward the more progressive districts.districts.

Schools for site visits were selected from Schools for site visits were selected from among volunteers. The lack of among volunteers. The lack of randomization may skew toward randomization may skew toward technology-users.technology-users.

Site selection could tend to result in the Site selection could tend to result in the perception that schools are further along perception that schools are further along in the Interactive Video (DDN) adoption in the Interactive Video (DDN) adoption process than they really are. process than they really are.

Applying the Lens of EvaluationApplying the Lens of Evaluation

Delivery modelsDelivery models System utility System utility

– Dominated by course deliveryDominated by course delivery– Applies traditional pedagogiesApplies traditional pedagogies– Abandons constructivist approachesAbandons constructivist approaches

Contextual factorsContextual factors Staying true to the notion of Staying true to the notion of

meaningful learning (theory-driven)?meaningful learning (theory-driven)?

More Information at:More Information at:www.homepages.dsu.edu/hawkesmwww.homepages.dsu.edu/hawkesm

Email: Email: Mark.hawkes@dsu.eduMark.hawkes@dsu.edu