diana cano, ets tom murdock, moodle rooms heidi larson, edc moving on-line assessments forward using...
TRANSCRIPT
DIANA CANO, ETSTOM MURDOCK, MOODLE ROOMS
HEIDI LARSON, EDC
Moving On-Line Assessments Forward using an Open
Source Technology Platform
Findings from a study conducted byEducation Development Center
& Grunwald Associates LLC
Sponsored by ETS
Purpose of Study
• To find out: – Experiences with & attitudes toward
• What:– Internet-based testing and the potential for
an open source delivery platform• To help:
– Further the conversation on how technology can enhance and support education goals.
Study Participants
• Interviews with state assessment and/or ed tech leaders from 27 states
• 38 interviews with national education opinion leaders representing both public and private organizations
“Open source delivery platform” means, for this talk:
• A technology platform for the delivery of online assessments, accessible by signing a free licensing agreement, with public collaboration on software development and improvement. Vendor support may be available for purchase.
• It does not include:– Open content or Curriculum– Teacher-developed tests– Computers or other devices– Freeware
5
C States (11)9 Assessment 7 Technology
C States (11)9 Assessment 7 Technology
B States (11)10 Assessment
10 Technology
B States (11)10 Assessment
10 Technology
A States (5)5 Assessment Leaders
2 Technology
A States (5)5 Assessment Leaders
2 Technology
Defining Experience Distinctions: A, B, C States
Paper & PencilAccountability Testing
Statewide Online Accountability
Testing
Piloting and Experimenting with Online Accountability
Testing
Open Source Platform
Accountability Testing
Overall, a positive reaction
• Both State and Opinion Leaders are interested in learning more about an internet-based, open source assessment delivery platform.– Generally more favorable responses from:
• States with greater experience with online assessment (A & B States)
• Technology and Opinion leaders
• However, moving to an open source platform would be a large step forward: – Experience and knowledge of open source is
limited. (Overall, although C states in particular)– Many states have yet to adopt any significant
online testing, for various reasons.
7
Perceived benefits of an open source delivery platform
• Flexibility to customize to state requirements
• Adaptability of the platform (mostly Opinion and State Technology Directors)
• Community: Ability to share, build upon and transfer innovations across states
8
“It’s an effective use of resources in the sense that if there is an open source large scale assessment platform, then states could pool their resources together similarly as districts have to improve software around Moodle and some of the other open source technologies...” [Opinion Leader]
“It’s an effective use of resources in the sense that if there is an open source large scale assessment platform, then states could pool their resources together similarly as districts have to improve software around Moodle and some of the other open source technologies...” [Opinion Leader]
Benefits, and potential benefits,of an Internet-based platform
Nearly all of the key benefits mentioned were related to IBT:
9
Faster score reporting For constructed response items in particular
Logistics relief No more tractor-trailer trucks unloading boxes of testing materials
Fewer errors, greater integration Demographic data can be pulled in from a district database
Potential for interactive and multimedia items
To gain a clearer picture of what students know and can do, and where they are getting off track
Accommodating students with special needs
Adaptability of platform would enable more accessible assessment for all students
Students like the computer tests No report of students wanting to return to paper and pencil.
Other “potential benefits” of open source platform not embraced
• Cost savings: Respondents liked the potential of cost savings, but anticipated a greater burden on staff, infrastructure, and other resources.
• Vendor Independence: Often vendors were seen as a partner or at least as necessary and reliable support.
10
Primary concerns about anopen source delivery platform include:
11
Security Respondents not comfortable with perception of open source and Internet safety, at least not for high-stakes tests.
Infrastructure Availability of computers and servers, Internet reliability, bandwidth capacity
Expertise Lack of enough expertise at the state, district,and school level to support development and delivery
Going solo Generally, collaborative relationships are seen as desirable, if not necessary, by some for such an initiative to be successful.
Potentials of Collaboration?Participants Unsure
• Many respondents, in particular those familiar with open source programs and with consortiums, could see opportunities for collaboration:– Sharing the development and maintenance burden– Sharing quick-turnaround improvements from one state to
the next• Others were wary:
– No experience, or negative experience, with multi-state collaboration
– Concerned with having to compromise standards• Leadership and/or structured governance
– Considered key by many respondents
12
A StatesPerceptions of Feasibility
•Perceive concrete benefits of internet based testing
•Some respondents perceive open source solutions as a strong contribution to Common Core standards.
B StatesPerceptions of Feasibility
•Many are interested in potential benefits of an open source platform for high stakes testing, but would want to learn more or see results in other states before adopting an open source platform in their own state.
•Some saw ways that open source could be used in developing and sharing test items, particularly in relation to the development of Common Core Standards across states.
C StatesPerceptions of Feasibility
• Using open source would require major infrastructure improvement; many C state respondents do not see their state in a position to make this move.
Feasibility of an Open Source, I-Based, Delivery Platform: A, B, C States
Desired features in an open source platform
• Able to handle more complex item types to take full advantage of the technology
• User-friendly, intuitive interface with easy end user support tools
• Item development standards to allow greater sharing across formative tests, states and vendors
• Enhanced data aggregation and analysis tools
• Flexible reporting options
14
Strategies for SuccessFrom the public report: State officials experienced with internet-based assessments offered the following strategies to ensure successful implementations:
Start small: Stagger implementation of online assessments1.Start simple: Begin with multiple choice test items before venturing on to more complex items2.Stretch the testing window to allow all students access to the test3.Collaborate: The states that reported the most success with their online initiatives also reported strong working relationships among both assessment and technology experts.4.Safeguard: Include redundancies, checks, and safeguards to prevent data loss or corruption
To read more about this study: http://www.grunwald.com/pdfs/
Grunwald_Open_Source_Public_Report_v3.pdf
Or contact:
Heidi Larson ([email protected]) and Bob Spielvogel ([email protected])
of Education Development Center
Or Peter Grunwald ([email protected])of Grunwald Associates LLC
16
1
How open-source LMS became adoptedby classroom teachers
• Easier room logistics• Easier time logistics• Rich reporting on student activity• Rich reporting on instructor activity• Collaborative events• “Things you can’t do in a brick-and-mortar classroom”
1
How open-source LMS became adoptedby institutions
• Use of a common platform• Functionality built across the platform• Interoperability between upstream and downstream
systems• Collecting and acting upon longitudinal data• Understanding transitions• The transparency of open-source systems• Adoption Levels• Open-source and closed-source hybrids
2
Use of a common platform
Higher Ed Market
Economics
Legacy System Attrition
Growth in for-profit online
schools
Open-sourceAdoption &
Choice
Pluggable Architecture with
vendors
Market Seismic Shift
3
Functionality built across the platform
• Provide a platform that delivers:• Flexible deployment• Comprehensive features• Configurable Implementations• Industry-leading reliability• Security through transparency• High-quality, community-based development
4
Interoperability between upstream and downstream systems
• Open standards• IMS LTI• SCORM
• Pluggable architecture• More than 620 non-core modules
• Open API• Web-services
7
Secondary to Secondary to Post SecondaryPost Secondary
Post Secondary Post Secondary to Workforceto Workforce
2- 4 Year 2- 4 Year Higher EdHigher Ed
Academic Academic LifecycleLifecycle
Collecting and Acting upon Longitudinal Data
A growing number of states have policies to ensure High School Students graduate College and are workforce ready
Achieve.org Survey/Research 2007
Understanding transitions by tracking standards
Many States Are Aligning College- and Work-Ready High School Standards
9
The transparency of open source systems
• Core code is available to anyone• No vendor lock• Schools do not lose IP• Multiple vendors can write to open standards, open code
5
Adoption Levels
K.C. Green, Campus Computing, 2009
8
Open-source and closed-source system hybrids
Increase Capabilities
Increase Capabilities
• Focus internal resources on value-
added, strategic activities
• Transition to SLA-driven 24x7x365
end-to-end support models
• Leverage cloud modularity to enable rapid introduction of
new SaaS applications
• Focus internal resources on value-
added, strategic activities
• Transition to SLA-driven 24x7x365
end-to-end support models
• Leverage cloud modularity to enable rapid introduction of
new SaaS applications
Mitigate RisksMitigate Risks
• Eliminate technology and vendor lock-in
• Ensure solution scalability from pilot
projects through enterprise
deployments
• Enable cost effective disaster recovery
and business continuity solutions
• SaaS vendors can provide a wall that
protects proprietary elements of a
solution
• Eliminate technology and vendor lock-in
• Ensure solution scalability from pilot
projects through enterprise
deployments
• Enable cost effective disaster recovery
and business continuity solutions
• SaaS vendors can provide a wall that
protects proprietary elements of a
solution
10
Watching for the adoption of open source for assessment
Open Source is now mainstream.
“Adoption of open-source software (OSS) is becoming
pervasive, with 85% of companies surveyed currently using
OSS in their enterprises and the remaining 15% expecting to in
the next 12 months.” Gartner, Inc., “User Survey Analysis: Open-Source Software, Worldwide, 2008”
®
Using Open Source Software for Innovative Assessments
June 2010
®
What are “innovative” assessments?
• Exercise types that allow us to expand measurement beyond constructs possible with largely multiple-choice testing
– These include open-ended and performance testing
– They also include using technology to expand constructs• To test things we wished we could test on paper• To test skills and knowledge that do not exist absent the
technology itself
• Use of these items in high-stakes assessments
32
®
Delivering by computer has operational and measurement advantages
Item level computer adaptive test• Reduced testing time required for examinees• Greatly increased the number of items that must be authored, pretested,
scaled, etc.• Unique aspects of the automated item selection algorithm can be exploited
Item types are becoming more complex and technology rich• Test development and software development are becoming more closely tied• Opportunity for IT to directly impact assessment
33
®
Challenge #1: Cost
• Some of these sorts of items can be expensive and time-consuming to develop
– Particularly a problem for exercises for which we lack a set of “operational norms and practices” (e.g. simulations)
• Many of these exercises require human scoring, which adds cost
– Unlike development or analysis costs, these go up on a per-student basis
– Also add schedule time
– There are electronic scoring options, although these carry limitations as well
• Computer-based testing can add hidden costs (machines, development of larger item pools)even if tests use traditional items
– School computer labs need to keep pace and offer access for test takers
34
GN
U F
ree
Do
cum
enta
tio
n L
ice
nse
fro
mW
ikim
ed
ia C
om
mo
ns
®
Challenge #2: Test development know-how
• For some item types, little “operational knowledge” or templates for development exist in the industry
– We understand how to produce large numbers of multiple-choice items with known performance characteristics
– This knowledge also exists for some types of constructed-response items, although the cost of a mistake in development is far higher
• Need improved cognitive models since we cannot rely on the brute force of multiple-choice tests
35
Th
e T
hin
ker
, by
Au
gu
ste
Ro
din
th
e C
alif
orn
ia P
ala
ce
of
the
Le
gio
n o
f H
on
or
by
Ka
rora
; p
ub
lic d
om
ain
®
36
• Some types of items need computers and possibly longer windows
• We need to ensure that interfaces and activities are not so complex that it takes weeks for someone to learn to take the test
• Advanced item types may yield different group patterns, which may be an equity issue depending on construct definition
• We must make sure we remember it is an assessment
Challenge #3: Technology
®
Working with open source software for delivering assessments
• ETS’ Cognitively-Based Assessments of, for, and as Learning (CBAL) research initiative will use Moodle this year
– handling the tasks and activities associated with that research project
– integrating a proprietary testing engine for the assessment delivery
• OECD’s Programme for the International Assessment of Adult Competencies (PIAAC) uses Testing Assisté par Ordinateur (TAO)
– interviewing adults aged 16-65 years in their homes – 5,000 in each participating country
– assessing their literacy and numeracy skills and their ability to solve problems in technology-rich environments
37
Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved.
FOR MORE INFORMATION ON THIS OPEN SOURCE STUDY: HTTP://GO.EDC.ORG/OPENSOURCEPLATFORMREPORT
38
HEIDI LARSON ([email protected])BOB SPIELVOGEL ([email protected]) PETER
GRUNWALD ([email protected])TOM MURDOCK ([email protected])
DIANA CANO ([email protected])