issues with online testing in the k to 12 school system and proposed remediation

7/30/2019 Issues With Online Testing in the K to 12 School System and Proposed Remediation

1/47

Final Paper

for

Master of Education in Educational Practice

Simon Fraser University Cohort VII

Issues with Online Testing in the K to 12School System and Proposed Remediation

December 13, 2008

John Mohr

973014655


2/47

2

Introduction

Every year teachers are dazzled by the latest technologies which promise to make

their jobs easier or help their students learn more effectively. In fact, many times, the

more bells and whistles these new technologies have, the more teachers seem to be

impressed with the innovation. When new technologies enter schools they are often

placed in a box for intended use at another time. Unfortunately, the more that time passes,

the less likely the technology will be removed from the box to be used by teachers in

their classrooms.

One variety of technology that seems to be making its appearance in education is

the computer system that is used for assessment of students. Computer assisted

assessment, or CAA, has been an easy sell for many teachers because it promises to save

them time. CAA also ties in nicely with the popular assessment for learning movement

as it offers teachers the opportunity to provide their students with individualized and

immediate feedback on written tests. In addition, these systems often come with powerful

analytical tools for teachers to use on student data that has been collected.

To showcase these CAA tools, vendors often provide trial copies of software or

they send their sales staff to amaze teachers with the latest product. As a result, teachers

are often persuaded when delivered an optimistic perspective about these systems.

Frequently, the pitch is delivered from a sales representative with a financial vested

interest, making inaccurate assessments about the marriage of their product with

educational practice. For this, among other reasons, I embarked on an inquiry that aimed

to observe these technologies from a critical perspective to balance out the arguments

for the use of CAA and to make an assessment about the true value of its promise.


3/47

3

Initially, however, I joined the many teachers who were convinced these new

technologies could remove mundane tasks from their day and, at the same time, help their

students learn more effectively. From the beginning, I had noticed how teachers still used

optical mark readers (OMRs). These systems have been able to save teachers time in

assessing students (Bull & McKenna, 2004), but paper cards required by these machines

are often costly and their analytical tools limited. Since I became a secondary teacher, I

have questioned our persistent use of OMRs. I found it odd that this technology, which

originates in the 1930s, continued to be a popular device for teachers who wish to assess

their students. I then found this pattern even more puzzling when I saw that certain

software packages available today could easily replace OMRs if they were used with the

computers in our schools. In addition to benefits for the teacher, these assessment

software packages could offer immediate feedback for incorrectly answered questions.

With an initial look at CAA software package, I became excited about the advantages this

software offered me and my students. Consequently, I attempted to implement the CAA

system into my practice. However, I soon came across a number of obstacles in

spearheading this assessment system for my staff. It was through this experience that I

began to see that other practitioners worldwide seemed to be encountering similar issues

when trying to implement these new CAA systems in their schools. I began to wonder

about the precise obstructions or hindrances that stood in the way of this technology.


4/47

4

Origins of the Inquiry

Bentz and Shapiro (1998) argue that for the postmodernist, a mindful inquiry

will include an awareness of self and reality and their interaction is a positive value

itself and should be present in research processes (p.6). First, as a researcher, the conflict

I experienced was an inherent technological stasis that seemed to exist for summative

assessment methods in my school. Desiring to be an agent for change, I wanted to know

why new, more powerful technologies do not seem to be embraced by many educators

and if there could be a way to change the current situation. I wondered if online testing

could help teachers work more efficiently with the countless tasks they are now

responsible for. I wanted to be an advocate for teachers in promoting tools that might

help them with marking and record keeping to hopefully lessen the burden for teachers

so that they might be free to spend more time teaching. Secondly, I wished to pursue the

admonishments of Black & Wiliam when they suggest we improve student achievement

standards by providing meaningful and individualized feedback to students each time we

assess them (1998). Students would then be given the opportunity to correct their

misunderstandings in areas that are revealed through regular assessment. Because

immediate feedback would be possible with computer-assisted assessment, I was

motivated to bring this tool into the fore. I felt that online testing might be able to

effectively help secondary teachers provide this useful feedback to many more students.


5/47

5

The Inquiry

As I read the literature about the benefits of online testing and as I acquired

experience through my fruitless attempts to use the assessment software I was trialing, I

began to sense a conflict in my practice. I knew that districts were continuing to promote

technology in schools to keep up with world trends, but there seemed to be no movement

in implementing this sensible use for technology. I began to wonder about the particular

factors that held back progress for this powerful tool. And throughout the study, I wanted

to perform a critical assessment of available CAA systems to identify the disparity

between CAA products and the needs of education. Therefore, I set out to explore and

identify any issues that seemed to prevent the success or feasibility of online testing in

my school. Or stated differently, I wanted to discover the factors that prevented or

hindered the use of online testing in K to 12 schools. Following this phase of my inquiry I

also strove to shed light on factors that mightpromote or facilitate the use of online

testing in the K to 12 school system. It was these two pivotal questions that formed the

basis of my inquiry.

Context

The research context for this inquiry took place in Surrey, British Columbia,

Canada at a secondary school established in 1994 that enrolled approximately 1300

students and employed about sixty staff members in 2008. Due to the age of the school

and limited funding, the existing computer network continued to remain considerably

outdated and in need of hardware upgrades. Through the summer and fall of 2008, there

were new wiring upgrades and the installation of industrial-grade wireless routers to


6/47

6

make improvements on speed and accessibility of desktop or laptop computers with the

local network. However, my colleagues and I observed that in spite of the wiring

upgrade, the processing speed of our network had only marginally improved. It was

speculated that contributing issues for our slow speed might have been attributed to old

equipment still in use and the common T1 connection shared throughout our district.

I, as the sole researcher for this inquiry, regularly teach junior science and

physics. I have also taken part in establishing a distributed learning support block for

students cross-enrolled in online courses. As an avid user of technology, I also draw on

experience with website creation, online conference and forum administration, course

augmentation with WebCT, and exposure to wide range of computer applications.

Throughout our district, many teachers seem to be unfamiliar with online testing.

However, some teachers have had minimal exposure through province-wide standardized

testing administered to grades 4, 7, and 10. Online provincial examinations in British

Columbia made their first appearance in 2007 as an alternative to paper exams.

Government e-Exams delivered via the internet still seem to be working efficiently,

however, teachers also tend to disapprove of the imposed testing. Mainly, teachers are

opposed to the publishing of test result data for public scrutiny because when data is

revealed to the public, connections are not made to cultural and socio-economic factors,

which arguably, can play a critical role in student achievement for certain communities.

As published data is often interesting to parents when choosing a school to enroll their

child in, educators often feel the process is an unfair appraisal of their schools. Teachers

have also noticed harmful effects on student learning when underprivileged students

prepare for and write standardized exams, especially if the level of questioning is beyond


7/47

7

their ability. However, at the same time, ministry officials seem to be interested in new

solutions which lower costs for paper and pencil examinations.

Because of these events, it has seemed that many teachers have had a negative

first-impression of online testing as it first appeared in conjunction with standardized

testing. When the benefits of online testing are discussed with teachers in British

Columbia, one is often met with skeptical reactions about its value in light of the political

climate and the issues that surround standardized testing.

Definition of CAA and its Forms

Computer-assisted assessment (CAA) is a term that can be generally applied to

any system that utilizes computers for delivery, marking, and analysis of student

assessment. In addition, most systems allow the instructor to group class data and to

redistribute analyzed data in a number of ways. Occasionally online testing can be used

as a synonym for CAA. As these systems have developed through the years, vendors

have taken a number of approaches in using the power of computers to accomplish tasks

that relate with assessment. Some systems gather data by scanning cards (optical mark

readers), some systems provide each student with a special input device that can be held

in the hand (student response systems), and other systems have students situated at a

computer terminal for questioning and response to test items. After data is collected, it

can then be analyzed by the teacher and sent back to students immediately or at another

time set by the instructor. And when students receive their results, the advantage of CAA

is that often students receive helpful feedback, on how they might improve their

understanding of concepts. Also, depending on the system, prepared feedback might be


8/47

8

provided to individuals or an entire class so that overall trends can become teaching

opportunities. In any case, CAA can help teachers with both formative assessment and

summative assessment.

Although I explored CAA in a general sense throughout this inquiry, it became

evident to me that identifying the forms of CAA and their advantages or disadvantages

could offer insight about the best products for CAA. Some products are suited to

formative assessment and others are could be used for more summative assessment. It

will often be helpful to refer to certain forms of CAA systems. Therefore, we will cover a

brief description of each type of system here:

Optical Mark Reading Systems (or Scanning Systems).

As mentioned, these systems have been in existence for a number of years. The

system involves students entering their answers on special cards that can be read by an

optical mark reader (OMR). When the teacher feeds the cards through the OMR, the data

is marked and answers and feedback can be given back to the student in a few different

ways. Older systems provide a number of options for teachers when student results are

displayed on scanning cards. Selectable functions include: revealed correct answers,

wrong answers merely identified, total score identified as total or percentage, or no

feedback shown at all. After the key and student cards are scanned, teachers can insert

another special card for simple analysis of their class. More complex OMR systems

involve the installation of software on computers for the collection and advanced analysis

of class data. One advantage of this second-generation system is that tests, keys, and

student data can be stored for later use on a computer hard drive. Also, these systems can

help teachers analyze data for individual students or for the entire class. As well,


9/47

9

feedback to students can be printed on the cards or reports using regular paper can be

printed off for each student. Because these systems involve a program to interact with,

often software designers have added features that allow teachers to dispense with

scanning cards and utilize a computer lab for test deployment. This novelty occurred

close to the birth of LAN-based assessment systems.

Local Area Network Testing (LAN-Based Assessment Systems).

As OMR systems developed, a different form of computer applications offered

teachers the ability to construct tests with the use of test banks. Through these

applications, paper tests could be created by selecting predesigned question items from an

inventory of questions. Often, these tools were popular with mathematics and the

sciences and would involve multiple-choice questioning. Close to the turn of the century,

software suites began to emerge for the construction of paper tests but had an added

feature for testing over a local area networks (or LANs). Online tests were developed

the same way as paper tests, but teachers would publish an online test and store it on a

local server within the school. If a teacher chose this option, a LAN-based test could be

run.

Today, with most software programs that offer this feature, there are various

options available for teachers on how a test is to be deployed. Some of these options

include: test availability for particular days or times, limitations on test-writing time,

number of attempts, need for student identity and password entry, and settings for

displaying feedback. Because tests can be issued over computers, LAN-based systems

improved on OMR systems in being able to pose a wider variety of questioning styles:

multiple response, questioning with audio-video content, fill in the blank, matching, and


10/47

10

textual entry (for short answer or essay format). Often these software suites also have

teacher-related software which functions to manage and analyze student data. However,

to run a LAN test, certain arrangements must be made to ensure the test will run properly.

Smaller programs used for the purpose of playing tests must be installed on all

computers used for testing. Also, test files must be stored on a shared portion of the

network that allows students read and write to files. As a result, the teacher must learn

how to use the management software for various tasks: building class lists, setting

passwords, mapping test data to grade book files, storing animations or audio files in the

proper location, and various other tasks. Often these systems require the need for a

teacher or network administrator to be knowledgeable enough to troubleshoot problems

that may arise with the delivery of the test. And, as with OMR systems, there are high-

end products for LAN-based systems with increased functionality for larger

organizations such as universities or colleges. In these cases, teams of technical support

are required for the proper functioning of the system.

Wide Area Network Testing (WAN-Based Assessment Systems).

Some systems utilize the internetfor delivery of an online test. Another descriptor

for the internet is a wide area network (or WAN). A WAN can involve the entire

internet or merely a portion of it. As internet sites require a server to store files for web

pages, WAN-based testing involves the storage of test files on a publicly accessible

server. This may be carried out in a number of different ways. The simplest form of

internet testing uses simple programs available from the internet that produce an HTML

version of the test to be suitably viewed on the internet. Many LAN-based software

applications also offer this option. When the file(s) are generated, they are stored on a


11/47

11

server to be displayed on a website. Teachers supply students with a URL address and the

test is written by the students when they visit the address. Sometimes these tests are

intended for practice where results are not provided to the teacher. Other times,

however, teachers can make arrangements for tests to require passwords and send test

data to an email address of their choosing. More advanced WAN-based testing systems

may involve consultation with IT specialists to build tests and manage student data.

Following the test, teachers are supplied with the results from the test from the vendor.

Student Response Systems (SRS Systems). Often there is a need for portability and

reduced cost of systems in schools. To meet this need, various vendors have developed

assessment systems which utilize a special hand-held device (or clicker) for entry of

student data. These systems are known as student response systems (SRS), voting

systems, or polling systems. After logging in, students use these impact-resistant devices

to send data by infrared or radio frequency to a receiving station that is connected to a

teachers computer in the room. The computer that has received the data will often have

software that is designed to manage, analyze and present student data in visually telling

ways. These visual reports display results that may or may not include a students

identity. Accordingly, teachers can modify their lessons to accommodate the progress of

their whole class and provide students with the opportunity to learn from their classmates.

However, this more formative use of SRS is not the only approach used. Teachers may

also issue paper tests and have students enter their answers on their response devices. In

this case, teachers gather and analyze the data on their own without showing the analysis

to the class.


12/47

12

Theoretical Framework

In recent years, there have been some leading thinkers that have made some initial

attempts at publishing books for this new area of assessment. As a primary source, I drew

from Bull and McKennas Blueprint for Computer-assisted Assessment which defines

CAA with its varied forms and locates it within current thinking about assessment and

online learning (2004, p. 10). It also covers best practices on how to create questions and

construct tests that provide feedback for students. Then, Bull and McKenna touch on

apparent issues that seem to occur with the use of CAA and ways to possibly mitigate

these issues. Finally, the authors close by highlighting areas of CAA that require further

evaluation by future researchers. To supplement Bull and McKenna, I also drew upon

many individual articles on the subject of CAA. Particular journals that seem to publish

articles relevant to my inquiry are the Journal of Computer-Based Instruction and Journal

of Computer Assisted Learning.

CAA has gained some attention as it bodes well for Gardners spatial and

verbal-linguistic learners (1986). Because it uses questions involving animations or

video clips, these innovative ways of questioning are able to reach a wider variety of

learners than for conventional pen and paper methods.

Bull and McKenna (2004), in conclusion of their book, call for more qualitative

and quantitative research to assess the methods and strategies with which CAA is used:

Often the process of introduction CAA prompts a reconsideration of existing assessment methods

and strategies, raising questions about purpose, scope, process, delivery, and roles and

responsibilities of both staff and students. Heightened awareness of such issues can provide the

opportunity to debate the role of assessment in higher education and to explore the appropriateness

of both existing and emerging methods. Furthermore, questions should continue to be asked

following the adoption of a new method, and the importance of assessment to student learning is

such that it necessitates a consideration of what impact CAA will have for students (for example,


13/47

13

in terms of learning, control, autonomy, identity), academics, assessment strategies, working

practices, educational resources etc. Additionally, we feel that the overall body of educational

research into CAA should be expanded and strengthened. (p.144)

It is through my qualitative inquiry that I intend to contribute to the body of research that

currently exists. In line with Bull and McKennas call for expansion and strengthening of

research, I striven to aid progress of CAA system implementation for the K to 12 school

system.

Methodology

The methodology I used for this inquiry is the action research approach first

developed by Kurt Lewin in the 1940s (Oxford Reference Online, 2005). This approach

is often suitable for qualitative research in education because it can be employed within a

short duration of time and lends well to study of a small community as in the case of a

school. Because my research took place over approximately two months and was situated

in a distinct public secondary school, this type of research worked well with my inquiry.

Additionally, action research was a suitable methodology because it offers a

flexible process of inquiry for this promising yet experimental field of study. The

iterative and cyclical problem solving process allowed adjustments to be made to the

direction of my inquiry as I engaged in transformative planning and action stages to

accommodate the emergent results. Once I looked at the data that was being acquired, I

was able to narrow my focus on more fruitful areas of interest that seemed to reveal

deeper-seated issues.

Lastly, I feel action research as a qualitative research method allows educational

practitioners to explore their questions in situ while they are teaching and, in turn,


14/47

14

allowing them to be agents for change. Dewey calls educators to secure social changes

without introducing disorder so that democracy in education can be maintained (1916, p.

11). Also, Stringer notes that action research has strong associations with political

activism (2007). As teachers study the conflicts they encounter, they are empowered to

take justifiable action toward mitigating issues they see in their particular situations. By

combining the ideas of Lewin, Dewey, and Stringer, teachers can achieve emancipation

and responsibly find answers to the situated problems they are so intimately aware of;

problems that are often born out of policy that does not accommodate for the educators

profession.

In addition to action research methodology, I used methods associated with online

ethnography, a recent methodological approach to qualitative research (Oxford Reference

Online, 2005). Besides accessing baseline data from online sources such as forums and

email correspondence, I was able to set up an online survey that called for willing

participants to offer feedback on questions about the issues of CAA. This feedback would

then be sent to an email account where data could be printed.

Methods

As I set out to obtain data which would inform me about the issues with online

testing in K to 12 schools, it made sense to interview the individuals that comprised

education: teachers, students, administration, and ministry officials. Also, I had made

plans to observe students during an online test, distribute a student survey, obtain related

articles from the internet, record personal reflections in a journal, and conduct an

asynchronous online interview with participants from a support forum.


15/47

15

However, due to time constraints and transformations of my intended research

trajectory toward more prolific data, I ended up interviewing six teachers, two

administrators, and two experts in the field of CAA: a teacher who was an OCR system

specialist and the proprietor for a company that specialized in WAN-based CAA testing.

For interviewing, I followed guidance provided by Lichtman (2006) as I prepared

interview questions, recorded responses on an audio recorder, and had the interviews

professionally transcribed afterward. For these interviews I posed questions similar to

those that follow:

Questions Posed to Interview Participants

1. What do you like about the idea of online assessment? What do you dislike aboutit?

2. Online tests versus paper-based tests. Do you have any preference for one methodof assessment over the other?

3. What problems do you feel would prevent online testing from becoming a reality:a. At schools?b. At the district level?c. At the provincial level?

4. What things do you feel could solve the problems you just mentioned?5.

Do you have any issues or fears concerning assessment with computers?6. If such a system was in place, how would this affect the student workload?I also was able to keep a continuous journal of reflections on my own encounters

with issues that I faced as I was trying to implement a LAN-based CAA system in my

school. In a slightly different fashion and in accordance with online ethnography

methodology, I conducted an asynchronous, online interviewing (Lichtman, 2006;

Markham, 2006) by posing similar interview questions on a support forum. However,

with this online survey, I seemed to have difficulty with attracting participants as I only

received responses for one willing teacher during the data collection period. At the start

of the research period, I was able to obtain some baseline data from this same forum by


16/47

16

making printouts of some forum discussion. In addition to this baseline data, I was able to

draw upon a list of journal entries that related to a colleagues experimentation with a

student response system. More baseline data was obtained in accessing news articles that

related to online testing I had retrieved from the internet. From one of these articles I was

able to engage in regular email correspondence with a project leader for a WAN-based

system being implemented in Australia.

The timeframe in which the data was gathered began at the start of September and

finished by the end of October. Through these varied lenses, I it was my aim to achieve

rigor by triangulation (Denzin & Lincoln, 2005; Lichtman, 2006). I also made a goal to

provide a thick description for all journal entries I made (Geertz, as cited in Lichtman,

2006).

Primary Analysis Phase

In analyzing my data, I drew from Lichtman (2006) and Schensul & LeCompte

(1999). In general, I followed the four steps Lichtman outlines in her chapter on data

analysis: description, sense making, interpretation, and implication drawing. However,

even before I began my formal analysis of the data, I noted patterns and features that

became evident as I continued to gather evidence.

The first pattern I noticed was that the participants which provided comments

were varied in their knowledge and experience of online testing. Some participants had

first-hand experiences use, implementation of, and experimentation with CAA. Other

participants appeared to make presumptions that were not rooted in experience or

knowledge. It seemed that an individuals experience with this topic might be arranged


17/47

17

along a spectrum ranging from participants that had little experience with CAA to those

that had first-hand experience with CAA.

I thought a spectrum was an appropriate device to represent experience because I

reasoned that there ought to be a number of places along the continuum where various

individuals could be positioned. As I interviewed the ten participants that were involved

in my action research, I noticed that seven of the participants lacked direct experience

with CAA. I reasoned that this was mostly due to the fact that the field is relatively new

to education and, therefore, experimental in many instances. In relation to experiential

background, later in my research, I indentified my sources for their lens so that I might

distinguish between experiential statements and speculative statements.

The second feature of my data that became evident was that many of the

participant responses were repeated across the range of sources I drew from. Themes that

repeated were: shortage of money, unstable or aged technology, teacher preference

issues, computers or network problems, and educational incompatibility issues.

After approximately six weeks of data collection, I began my formal analysis of

all data I had collected. To begin, I made arrangements to obtain paper copies of all data I

was able to sequester: interviews, forum records, emails, news articles, survey data, and

journal entries. Next, I read through the data in search of any reference made to a

problems, shortcomings, or issues that related to online testing. If relevant references

were found, I attempted to summarize longer portions of text or multiple references into

Continuum

Practitioners withLittle or No

Experience withCAA

(SpeculatingPartici ants

Practitioners

HavingExperience with

CAA


18/47

18

abbreviated phrases that seemed to capture the sense of the reference(s). I would then

highlight these statements in yellow to signify issues with online testing. In most cases,

issues were overtly revealed by the data, however, in other cases, I interpreted situations

or attitudes of individuals as issues, which involved my interpretation of the data as the

researcher. Concurrently, with the reading of the data, I also demarked any reference

made to solutions for issues highlighting these in orange.

Following this initial reading, I then typed out all of the statements highlighted in

yellow organizing them into a continuous table. In this table, I included three headings:

Issue, Source, and Page Reference. During this process, I combined any duplicated

statements if they were found to occur within the same source and I also revised any

statement I had initially made for clarity being cautious to avoid inclusion of any

erroneous deductions in my interpretation of the sources. Apart from this list, I also drew

up a short list that noted the Lens of the participants involved. Initially, I drew up this list

to keep track of the codes I used to identifying sources, but then I noticed that

participants could also be grouped according to their familiarity with CAA or experiential

background within the area of CAA. I called these labels participant lenses:

Clicker Trialing ExamView LAN Issues ExamView Software Issues General Issues in the Media LAN Trialing OMR User Presuming Participant Regular LAN User WAN Researcher WAN Specialist WAN Trialing


19/47

Although I created these labels, I did not attempt to use these lens titles in the analysis that would

later follow. I chose to overlook the origin of the comment due to the complexity of the analysis

I would be faced with if I was to overlay the issue categories with their sources. I also surmised

that although some participants lacked experiential background from which to draw upon, these

same participants revealed evidence of teacher preference, teacher attitudes and concerns, and

lack of training that existed in the participant sample I was drawing from. I felt that these

elements and could easily stand alone as issues in their own right.

After I had completed this continuous table, I scanned through the list of approximately

240 comments to look for overall trends. Approximately, seven themes seemed to emerge:

Technical Support Issues: latent response, lack of knowledge, various sources Financial Issues: funding for professional development, technology, software, and the issues

surrounding dispensation of funds

School-based Issues: restrictions of space, requirement for teachers to share technology, thepersistent need for paper despite the use of computers.

Teacher Perception Issues: Concern over requirements to learn the system, investment oftime, faulty equipment, cheating, share technology, concern over teacher jobs being

eliminated, and preference for assessment of students

Educational Issues: cheating, the occurrence of student frustration, data loss, limitations ofsoftware to assess students or allow them to process their thinking, and mismatch withchanging views on assessment of students in an arcane system.

System Design Issues: Wide-scale problems or requirements associated with the differentmodels for CAA (Computer-based, LAN-based, WAN-based, Student Response Systems,

and Optical Mark Reading Systems)

Technical Issues: platform incompatibility, data loss, overloaded networks or computers, lackof suitability of software.

After listing these themes, I set aside any deductions for groupings that I had made earlier

and chose to approach the data from an alternative vantage point. I separated each issue

statement by physically cutting them apart. I then attempted to sort statements into piles that

signified categories. However, soon after I had begun to use this approach, I experienced

difficulties in making decisions about new categories to start and the placement of statements in


20/47

20

existing categories I had already created. I found this method of analysis problematic due to the

fact that much of the data appeared to be interrelated in ever-increasing ways. Therefore, I

dispensed with this approach for analysis.

In a subsequent attempt to sort the data for emerging themes I revisited the list of seven

themes I had devised earlier and broke elements of these issue categories into one or two-word

ideas.

Through mind-mapping, I tried to organize these ideas using paper and a whiteboard with

multiple revisions. Through this method, I sought to gain insight into ways that the issues might

be interconnected and grouped. I found this method of analysis to be quite successful as I had

found that maintaining an overall view of the issues facilitated self-questioning, spatial

arrangement, linking, and discovery of trends. It also seemed to allow for determination of the


21/47

21

cause and effect relationships that existed; it seemed that some issues were dependant on more

deeply-seated issues. After my fifth revision and much pondering, I arrived at the model depicted

below.

This organizational diagram seemed to fit with overarching questions that could be asked of

CAA relating to my inquiry:

What spending would be required by CAA systems to enable it to work properly? What entity might be responsible for limitations of spending on CAA systems?First, it seemed that a schools budgetary limitation was the main reason why CAA systems

could not be implemented properly as there seemed to be a need to spend money on the purchase

of the system, training, and technical support of the system. Therefore, because many issues for

CAA seemed to arise from a shortage of financial backing, the root reason could ultimately be

traced back to the governments financial allotment to districts and, in turn, schools. It would

then follow that if schools were given an endless budget, all CAA requirements with a requisite

$CAA

S stemsIssues

Related toRequirements

For CAA

Suitability IssuesBetween CAA and

Education


22/47

22

for funding could be purchased, which would, in turn, eliminate many of the problems I found.

For example, if money were unlimited, issues with sluggish computers would be eliminated as

new computers could be purchased to replace old computers. Or, if funding was not a problem,

issues related to lengthy response times of technical support staff would be rectified with the

possibility of having expert staff on hand for continual consultation when teachers wished to use

the system.

I also reasoned that governments had two main roles in this schema: first, the responsible

provision of money to fund government programs and, second, provision of an adequate

education program to the people. Therefore, the general interests of governments would be to

ensure funding was spent well and that educational ideals were upheld in schools. Following this

rationale, we move our attention to the lower portion of the diagram.

The second set of over-arching questions could then be asked when considering the

implementation of any innovation that helped to attain to the goal of providing a high-quality

education program:

Will CAA systems prove to be a suitable match for educational ideals (i.e. as they used touphold acceptable assessment practices)?

If CAA does not match will all educational ideals, could it still be used?These questions address issues of suitability between CAA and Education as a whole. Many

issues seemed to focus upon mismatches with CAA and education. An example of one such

mismatch of many systems is the prevalent and erroneous assumption is that teachers would have

the time and expertise to implement, maintain, and troubleshoot CAA systems on their own with

little or no assistance.


23/47

23

Lastly, we must turn our attention to the upper right portion of the schema: CAA systems.

Here considerations might be made about the different systems that are available today.

Therefore, at this point it might be asked:

Are all CAA systems created equally? If not, how expensive is each CAA system and how well is each one suited to educational

ideals?

These considerations did not overtly reveal themselves in the data. However, these reflections

are valuable in determining a system that will best fit with the available funding and with

educational norms and ideals. For instance, if some systems allowed for more variety in

questioning students, this would be more valuable to teachers. Also, when all costs are tallied for

the system, training, and support, it would be useful to find the system that is the most cost

effective for what is offered. It is interesting to note that different system vendors have very

different aims for their product. Some CAA systems focus on allowing teachers to perform a

myriad of functions through their software, others aim to make the software intuitive for

teachers, and still others take the burden off of teachers entirely and offer full internet testing

services.

Secondary Analysis Phase

After the overall schema was formed, more attention was given to the clarification of the

one or two-word labels. I arranged these labels into a table and made attempts to expand on the

meaning of the labels by fleshing out descriptions of each label. In doing so, I realized that the

one or two-word labels became issue-categories that might feasibly be used to order issue-


24/47

24

instances which came from the individual comments made by my participants. The issue-

category labels I had elaborated on were:

Suitability Issues When CAA and

Education are Compared

CAA System Requirement Issues

Simplicity Hardware

Teacher Preference Hardware Capability

System Ability to Assess Software

Student Design Resources

Time Efficiency Space

Cheating Technical Support

Management Network Configuration

Training

SecurityManagement

As can be seen by the headings of the table, the two groups of labels would fit into the schema I

had devised earlier. I wanted to offer some structure within the two overarching themes or

issue-themes as I called them. Therefore the levels of organization that were materializing

moving from the macro to the micro were: issue-schema, issue-themes, issue-categories, and

issue-instances.

I then made large print copies of the tables and I gathered my collection of 240 issue-

instances that I had cut out. I reasoned that if I were to try placing these issues into piles

alongside the issue-categories, I could verify that the data fit into my pre-determined perception

of the issue-themes. Therefore, I began ordering the data into the seventeen issue-categories that

I had created. However, as I proceeded through the data, I noted a number of problems: some

categories were almost empty, some categories had large piles, and some data posed difficulties

with the placement into only one category.


25/47

25

One such example of an issue-instance that could have been placed into two categories is

given below:

Comment Source Page

Old computers may perform slowly and, in turn, frustratestudents. MK 2

System Ability to Assess Hardware Capability

When interpreted, this issue-instance might be seen as belonging to two different issue-

categories. But when this arose, I tried to look back or remember the main point the speaker was

trying to convey during the interview. I this particular case, I felt the participant was trying to

shed light on the fact that students might be affected during their assessment if technology were

to operate slowly; therefore, I decided that the statement related more to the problems that CAA

systems have when they are used in the assessment of students.

After all sorting had been completed, I began combining, omitting, creating, renaming,

and even moving issue-categories between issue-themes. In conjunction with this, I needed to

revise my issue-category descriptions. The final arrangement is shown below.

Suitability Issues WhenCAA and Education are Compared

CAA SystemRequirement Issues

Category Issue Sub-Category Issue Category

Teacher-Oriented Limitation Issues Issues with System Cost

Teacher-Oriented Preference Issues Lack of Teacher Training

Epistemological Mismatches and

General Issues Involving the

Delivery of Education

Issues with ExistentSystems

System Deficiency

or Mismatch Issues

Student-0riented MismatchesSpace Limitations inSchools

Technical Support

Security IssuesIssues of Perception or Endorsement from Staff Who Lack

Experience Miscellaneous and

Unexpected Issues


26/47

26

In looking back over the course of the research period, there seemed to be cyclical and

iterative patterns to my inquiry and analysis which followed with my intended methodological

approach. I did not feel bound by the particular steps delineating the action research model that

Dana & Yendol-Silva outline (2003), yet I felt it natural to allow my research to increasingly

focus on sources that provided more fruitful insight into my inquiry. I am also aware that the

ways in which I have arranged and grouped the data is not the only way in which they could be

grouped. Due to the situated nature of my data sources and time limitations, the possibility exists

that more data might be gathered (both temporally and spatially) and more time for analysis

could be allotted. Also, due to the fact that CAA is a topic area that is rapidly advancing, there is

a possibility that some pertinent details will not be included at the time of this papers

presentation. Because of these limiting factors, analysis will need to be modified to suit different

scenarios that exist throughout the world and the emergent data and research that follow this

paper.

Results: A Description of the Issue-Categories

The pieces that we shall now focus attention on are the two main issue-themes described above

(the two tables) as well as to elaborate on issue-categories within these themes.

Issue Categories for CAA System Requirements

For the first issue-theme, I noticed the presence of many quandaries and criticisms from

practitioners that pointed toward a lack of funding for the purchase of new technologies and the

sense that schools were presently ill-equipped for the implementation of any new technology. As

stated earlier, issues were grouped into the following issue categories:


27/47

27

Depending on the CAA system, many issues seemed to stem back to short-term and long-term

costs that would be incurred if online testing were to be successfully implemented in schools.

This deduction is based on the notion that many issues end up costing money for a school or

district. Costs are incurred when time of employed staff (wage costs) or goods must be devoted

to solving of the issue or meeting of system requirements in order to ensure the system works as

it should. Therefore, because governments carefully allocate limited funding to districts and

schools, and because public institutions must budget money responsibly, CAA systems are often

out of reach due to cost. As a result, vendors that provide products which are more expensive are

less likely to be successful in selling their product to interested buyers in education. It is for this

reason that CAA systems must be reasonably priced if they are to take root in schools.

Issues with System Cost Provision of Teacher Training Issues with Existent Systems Space Limitations in Schools Technical Support Issues Security Issues Miscellaneous and Unexpected

Issues


28/47

28

The main cost issues are:

1. Systems typically involve the need for hardware, whether they are computer terminals oradditional components such as scanning devices (for OMR systems) or hand-held units (for

student response systems).

2. Site licenses are often needed if the software needs to be installed on numerous machines forteacher and student use of the system.

3. Once the system is in place, in-service training for teachers would typically be needed forteachers to learn how to utilize the system. Otherwise, the system is brought into schools, but

teachers do not use it. Also, there would be a need to build in class time to orient students

with the CAA system.

4. Often the existing hardware in schools can be used for the different kinds of CAA systems,but frequently the technology in schools does not operate well with the system. This might

include network incompatibility issues, slow processing speeds, and vulnerability for data

loss when computers crash. If these problems arise and persist, there would be a need for

system upgrades if the CAA system is to work as intended.

5. If testing is to occur in a controlled way, classrooms that are devoted to online assessmentwould be useful arrangement for teacher convenience and prevention of academic

dishonesty. Computers could be ready to use with no need for set-up time and terminals

could be oriented strategically in the room with privacy filters to prevent students from

seeing the work of another student on an adjacent monitor. However, many schools are

limited with space and rooms dedicated for this purpose may not be possible.

6. Once the system is running, there would undoubtedly be a need for consistent technicalsupport. Although, many companies offer free support with their system, the response time is


29/47

29

often slow and support personnel may require time to address issues that come up. If schools

were to seriously consider implementation of CAA, there would likely be a need for funded,

continual, high-quality technical support so that IT personnel would be ready to respond to

teacher questions both quickly and aptly.

7. Although there may not be a direct connection to cost, CAA systems often require specialsettings for data collection when students write tests at many terminals. These settings are

often conflict with the security settings of school networks if students are given the required

permissions necessary for proper functionality of the CAA system. Information management

personnel may be required to devote extra time to rectifying these issues or there may be a

need to purchase additional software to preserve the security of the network.

8. Unforeseen events such as fires or power outages may cause expenditures of time or moneyif they interrupt the running of the CAA system.

These issues and additional explanation of the cost issues that were obtained are summarized in

the table below.

Description of Issue Categories that Directly Impinge on CAA System Requirements

Issue Category Issue Category Description

Issues with System

Cost

Issues relating to monetary costs on limiting budgets for acquisition and maintenance of hardware

(new computers and/or peripherals), software, and any other system resource that is required (e.g.

OMR forms).

Lack of Teacher

Training

Cost and organizational issues that deal with the provision of in-service for teachers to learn how

to use CAA systems. Also, costs associated with built-in school time to orient students on CAA

systems.

Issues with Existent

Systems

Issues that relate to current school technology systems: shortage of computers (unavailability of

computers), software incompatibility (e.g. cross platforms), instable workstations or networks (e.g.

crashes, data loss), overloaded systems (data transfer), and low performance of systems (slow

processing speeds).

Space Limitations in

Schools

Issues in schools that deal with the reservation of space for computer labs designed for online

testing.

Technical Support Issues surrounding the identity of technical support, timeliness of service, and level of expertise

available for teachers when problems or challenges arise.

Security Issues Issues related to the actual or perceived security risks caused by certain CAA systems and their

need for additional networking permissions (e.g. students requiring read/write permission in

shared portion of network)

Miscellaneous and

Unexpected Issues

Varied issues of human error (memory of passwords), acts of God (power outages, fire drills), and

unexplained errors that occur.


30/47

30

SystemDeficiency

andMismatch

Issues

Teacher-Oriented Limitation Issues Teacher-Oriented Preference Issues Epistemological Mismatches and

General Issues Involving the Deliveryof Education

Student-0riented Mismatches Issues of Perception or Endorsement from Staff Who

Lack Experience

Issue Categories Relating to Suitability of CAA System for Education

The second issue-theme in the data that seemed to emerge involved conflicts on how

CAA systems were designed to meet the needs of education and the interested parties within it. It

occurred to me, that with many issues, there was an underlying mismatch that existed with the

way CAA systems were designed to assist educators and assess students. Therefore, many of the

issues tended to reflect a gap in the suitability of CAA systems for education. Generally, it

could be concluded that a large part of the problem and the solution seemed to reside with the

vendor and the way they had designed their products. Of course development in technology

continually changes to make improvements for particular applications. However, for some CAA

systems this gap is too wide to warrant their frequent use in schools.

A note of worthy mention should be made about this issue-theme. Due to the fact that

many of the issue-instances within theme were greatly interconnected, the categories developed

did not have clearly defined borders. I found these issues difficult to categorize because

education tends to be very complex in nature and issues within education are multifaceted.


31/47

31

The issue-categories that were discovered are:

1. CAA systems tended to reveal a mismatch with two overall parts of education: the peoplethat might use these systems and problems involving time-honored philosophical principles

of education. To begin, we will look at suitability issues for teachers.

a. When teachers were presented with the idea that CAA systems could be used forassessment of their students, often they there was a feeling that learning and using the

system was an impossibility due to lack of time. Also, many teachers felt that learning

CAA systems would be difficult. Moreover, there seemed to be an interesting conflict

that existed with the assumptions vendors had for the teachers ability to set up CAA

systems with little or no help. System designers seemed to expect that teachers should

have the network clearance and ability to set-up, learn, and troubleshoot the system upon

implementation. Of course, many teachers tend to shy away from technology and,

therefore, these systems tend to remain unused.

b. Many teachers are used to performing their job in certain ways. As a result, teachers tendto have preferences for reading student answers, marking, assessing, offering feedback,

and teaching after assessment. Many of these preferences stem from the desire for more

convenient and comfortable ways of working. Teachers want to move through material

efficiently and quickly. As a result, they choose to work with paper where pages can be

flipped, scanned quickly, and written on easily. Also, teachers often want to assess

different attributes of a students ability. One example in science or math would be with

steps that might set up the calculation for a solution. If a student arrives at the wrong

answer, the teacher still wants to see if the correct steps are present or not. If they are,


32/47

32

teachers may award some marks for these steps. On the other hand, if a student arrives at

the right answer, but the steps conflict with the answer, full marks may not be given.

c. Often there were issues that seemed to conflict with best practices for assessing students;these conflicts might then run counter to epistemological principles of assessment or they

may conflict with mandates governments might have for education. Issues seemed to

reveal that assessment methods could be compromised in their rigor, or, that problems

existed with the way in which information was gathered. First, due to limitations with

computers, CAA systems cannot easily reveal the processes of student knowledge or

insight into their reasoning. Though many systems boast that a variety of question types

can be issued and marked, most of the styles of questioning are objective in nature with

either a right or wrong answer. Another limitation of computers related to the way that

students might set up mental scaffolding with rough work, outlining, or simple mind

maps. If this preliminary step is not permitted through computers, students are at a

disadvantage in the way that they are to arrive at their answer. Second, there was

evidence of student data being lost or irretrievable. Not only is this risk at fault for

improperly measuring all students equally, it can cause frustration or anxiety in students

which may lead to effects in their answers during testing. Third, CAA systems seemed to

be particularly vulnerable to academic dishonesty if students were able to store answers

on networks, search the internet for answers, look onto adjacent computer displays, or

communicate with each other electronically. Lastly, governments may want to align their

educational mandates with certain societal goals such as the environmental agenda or

with cost cutting measures in government spending. With this in mind, some issues arose

for some systems that still required the use of paper for student feedback or data entry via


33/47

33

scanned cards. Also, when all time investments are considered for test bank development,

teacher training, and system set-up, there may not be a savings of time for teachers; this

reality would defeat of the purposes of CAA if it is to save tax dollars for efficient use

teacher time.

d. Issues exist for students who would use CAA systems. For essay style questioning,answers must be typed. Therefore, along with knowledge assessment, students are

inherently assessed for their typing ability because not all students are able to keyboard

with the same proficiency. Also, as students are able to interact with a paper test that is

marked by their teacher in order to identify and correct their misunderstandings, they

would not likely be able to enjoy the same benefits with CAA systems. In addition to this,

there are also the aforementioned issues that address the need for workspace and the

student anxiety or frustration caused by technology. Either students prefer to process their

thinking alternatively or they may be frustrated with computer speed, data loss, or with

the unfamiliar mode of delivery.

2. Throughout the data acquisition phase of this inquiry, I noticed that many of the issuesstemmed from concerns or fears that teachers held. I reasoned that these emotionally affected

perspectives might be generally grouped as issues of CAA system perception. Fears revolved

around a shortage of money, job security, loss of assessment data, and unfamiliarity with

electronic cheating methods. Many of these issues related back to former experiences that

teachers had with the use of unstable systems or the cost of new technological tools. Though

some of the perceived issues may not exist, the data revealed the need for convincing

teachers that CAA could be useful. Some data showed that buy-in was a problem when a

few teachers were willing to try CAA within their schools. However, concerns did not stop


34/47

34

with teachers. Network administrative personnel were equally reluctant if teachers requested

read/write access privileges for their students to write online tests. School district

administration could also be reluctant if systems required unhealthy reliance on commercial

vendors resulting in long-term payments for required supplies or services.

These issues of suitability are summarized in the table below.

Description of Issue Categories for System Compatibility with Secondary Education

Category Issue Sub-

Category

Issue Category Description

Teacher-

Oriented

Limitation Issues

Systems lacking simplicity and ease of use designed for teachers who have busy

schedules and varied aptitude range for technology. Apparent problems about the

teachers role: erroneous expectations and assumptions on the part of the CAA system

developer that teachers should be equipped with the time, knowledge, ability, andnetwork clearance to establish, maintain, and troubleshoot CAA systems with little or no

assistance.

Teacher-

Oriented

Preference

Issues

The system doesnt always suit how teachers want to use it; teachers want to use or

modify it in particular ways. Teachers seem to find it easier to assess with paper than to

organize for online testing. Also, limitations of systems for offering time-saving

advantages for other questioning types (i.e. written work cannot be marked) and

inherent limitations that do not allow teachers work as they prefer (i.e. reading off paper

rather than on a screen), how they want to assess students for process (i.e. seeing steps

in calculation problems), and how they want to teach through their feedback by having

students interact with it.

Epistemological

Mismatches and

General Issues

Involving the

Delivery of

Education

Issues on how systems assess of students: limitations on question types (i.e. mostly

objective style), student data loss when computers crash, computer glitches causing an

erroneous result, student frustration of systems that can affect student performance

during assessment, system vulnerability of CAA systems to academic dishonesty, studenttyping ability skewing student performance, and system inability to distinguish between

good test writing and presence of knowledge. Failure for systems to save teachers t ime

over longer terms. Not necessarily decreasing paper usage for environmental reasons

(scanning systems). Minimal research about online testing as an assessment method.

System

Deficiency

or

MismatchIssues

Student-0riented

Mismatches

Program design issues associated with how students process or communicate

information: the need for workspace, preference for reading media, failure to offer

interaction with written feedback, unfamiliarity with technology, and inherent affect of

typing ability on test scoring. Also, issues that deal with anxiety or frustration of students

with alternate mode of test delivery, slow computers, data loss, and technical difficulties

during test writing.

Issues of Perception or

Endorsement from Staff Who

Lack Experience

Issues that involve value conflicts where governments and taxpayers favor cost saving

with CAA while teachers may fear a reduction of jobs or disruption to educational

assessment. Issues that stem from negative teacher emotion or perception: fear,

reluctance, frustration, and general concern over the prospect of CAA systems in light ofprevious experience (unstable networks, limited budgets, required time needed to learn

the system, unfamiliarity with electronic cheating, and skepticism about assessment

efficacy). Teacher disapproval with spending limited money on CAA; similarly, small scale

systems being dependant on buy-in for successful implementation. Issues associated with

fewer teachers know about CAA and are working alone. Reluctance of Teachers, schools,

and districts not wanting to continually depend on third-party vendors. Management

issues that arise from network managers being wary of added risks that CAA systems can

impose on networks.


35/47

35

Conclusions

When one looks at the illustrated patterns, a number of conclusions might be drawn about the

issues that face CAA in the K to 12 school system. In many cases CAA might seem to be

feasible. However, when conclusions are made about the issues we might see more clearly how

viable CAA is for the near future. Either the issues should have viable solutions or there will be

issues that persist for one reason or another. The issues outlined will present a number of hurdles

for teachers to overcome if they are to make CAA work in their schools. However, because a

solution may be possible for one teacher and not another, assessments about the viability will

certainly vary from place to place. Teachers may be working in teams where they have the

support of their administration and governments to make CAA a reality. Conversely, they may

be working as lone advocates for a tool that is still experimental by many means. Therefore, as

an aid to teachers, districts, and governments I present a number of conclusions about the present

issues to help these parties determine whether or not the possibility exists for CAA to become an

option for assessment in your locale.

Intra-Theme Conclusions

An aspect that can be seen with the review of the issue-categories is the cause and effect

relationship between to two overall themes. If money is absent, teachers will be limited with the

time they can commit. If governments do not regard CAA as worthwhile prospect, time and

attention will not be given to modifying the curriculum to include student training. If schools are

outfitted with up-to-date computer networks, staff may not be so hesitant to use CAA as an

alternative to assessment. These relationships reveal that another part of the problem and solution

will ultimately exist with how governments regard CAA. If assessment systems are deemed to be


36/47

36

a valuable solution for educational goals and financial backing is given to their implementation,

many of the issues will diminish. Because of this, teachers should evaluate CAA for its

usefulness. If CAA proves to be useful for them, a sensible effort would be to try to convince

governments of CAAs advantages. Then, when governments are persuaded of CAAs merits,

they could allot the necessary funding to districts for system costs.

Magnitude of Successful CAA Projects for Public Education

Being that access to funding is a critical issue, I have noticed that successful efforts have

typically resided at the government or district level where technical support among other costs

seemed to be lower and more efficiently allocated. In both cases, decision makers for budget

allocation on projects exist at these levels. School-based projects tend to involve only a handful

of teachers and gradually lose momentum as fewer teachers choose to participate. Also, teachers

usually are required to solve many of the technical problems on their own. This can often be

problematic when teachers do not have the technical expertise, time, or security clearance to

solve problems that arise. However, if CAA is implemented over a district, expert technical

support with the proper clearance might be devoted to maintenance of the system and

organization of files on a full-time basis. As well, there exists a possibility for collaboration on

test bank development with district-wide initiatives. It could be arranged that test banks might be

stored on a secure server that is accessible by many teachers. And as initiatives progress,

teachers might be able to contribute questions to test banks for joint use throughout the district.

With these factors in mind, it would seem that there was a critical mass for CAA because

implementation tended to be more successful as a district initiative.


37/47

37

The Most Successful Form of CAA for K to 12 Education

As a precursor to future work, the types of CAA could easily be assessed for the presence

or absence of issues. For this simple comparative analysis, I have combined the issue categories

for both themes and I used a checkmark to resemble the presence of significant issues related to a

particular issue-category.

Presence of Issues for CAA System Cost Requirements and Suitability with Education

Issue Categories OMR SRS LAN WANIssues with System Cost Lack of Teacher Training

Issues with Existent Systems Space Limitations in Schools Technical Support Security Issues Miscellaneous and Unexpected Issues

Teacher-Oriented

Limitation Issues

Teacher-OrientedPreference Issues

EpistemologicalMismatches and GeneralIssues Involving the

Delivery of Education

System

Deficiency

orMismatch

Issues

Student-0rientedMismatches

Issues of Perception or Endorsement

from Staff Who Lack Experience


38/47

38

The result may also be depicted in the modified schematic below which shows a reduced set of

issues that exist for WAN-based testing.

Reduction of Issue-instances Revealed for WAN-Based Testing.

Because this analysis may be somewhat subjective to my interpretation of the data, a need for

future research is revealed for comparing CAA product types. It might be advisable to arrange

for an equal representation of participants with experience that spans the different forms of CAA

systems. In this way, issues instances might be identified for their frequency to establish more

than just a binary label of just presence or absence.

Having stated this, we might be able to note the approach that seems to work best for

CAA when considering the issues. It could be that many problems are eliminated with the use of

WAN-based systems especially for district-scale systems with ongoing professional technical

support. First, cost for additional hardware and software is reduced to a large degree; there is no

$Issues

Related toRequirements

For CAA

Suitability IssuesBetween CAA and

Education

WAN-Based

Systems

OtherSystems


39/47

39

need for software installation on school machines and no peripheral equipment needed (no

clickers, receivers, scanners, or scanning cards). Second, costs are reduced for teacher training

because of the simplicity of the systems use and the fact that much of the back-end expertise is

handled by support staff. Third, there may be some space limitations if desktop computers are

used; however, because no special program is needed to interface with the test, any computer

that can access the internet is suitable. Fourth, security issues are minimized as permissions need

not be granted inside the school network and teachers can be less concerned about storage of test

answers within shared folders. Fifth, if there is an unexpected power failure or emergency, the

probability that data is lost is minimized. Sixth, it is not necessary for teachers to have

knowledge in network administration, management of data between files, and special clearance

for read and writing ability on the school network as these tasks are handled by the technical

support staff. These and other advantages exist for WAN-based testing and although some issues

have not been addressed, further solutions will follow for the remaining issues.

However, in cases where smaller systems are implemented, it seems that a LAN-based

format is the next best choice. These do it yourself systems require a willing and tech-savvy

teacher to help configure the system appropriately and install the needed software onto machines

used for testing, but they can offer similar advantages to internet-based testing. If one plans to

organize this system in their school, it is advisable to ensure students have a common network

profile which is exclusively used for testing. This profile can then be tweaked to add the

necessary permissions, as well as, to restrict access to applications or other parts of the network.

This strategy also avoids situations where students might forget individualized login information

whether by mistake or on purpose. It is also advisable to ensure good relations exist with


40/47

40

network administrators as lead teachers often need to work closely with network management

personnel to ensure the appropriate settings can be established. This leads to the next conclusion.

Role Structure for Optimal Success of CAA Systems

Niall Sclator and Karen Howie (2001) outline 21 possible roles for the efficient operation

of the ultimate online assessment engine at the post-secondary level. These roles include:

question author, test designer, marker, and system administrator among others. Though multiple

roles might be filled by one person, these systems reveal the need for specialized expertise in

higher systems. In the K to 12 school system, the need for such a complex system is not

necessary; however, a similar need for role identification would likely exist. Because there is

often a need for technical support and because teachers are restricted by time and training for

tasks such as programming and data management, it becomes more important to allow teachers

to merely use the system for their means and to leave technical problems with IT support staff. A

simplified version of Sclator and Howies roles is shown below:

Problems seem to begin to arise when teachers take on roles that technical support would fulfill

and technical staff cannot be expected to understand how CAA might be integrated with

education. Therefore, it seems sensible that key roles exist to liaise between teachers and

programming experts. Also, the model would seem to work best if in place on a continual basis

so that response time for support remains at a minimum and teachers always have a person to

contact when needs arise.

Educators Education

Liaison

Technical

Support

Liaison

Programmers


41/47

41

Augmentation of Summative Assessment with CAA

For many teachers who engage in summative assessment of their students, it seems that

there is an effort to try to issue an array of questioning styles to make measurement more

reliable. Examples of some questioning styles might include: essay, short answer, multiple

choice, true or false, and matching. However, many of these styles involve the choice or only one

or more pre-defined answer. This kind of questioning has been identified by Bull & McKenna as

objective questioning (2004, p.1). Often CAA is used to pose objective questions and has the

ability to issue results for students because answers are easily identified by the computer.

However, there are difficulties when CAA is used to mark written answers. Even the simplest

forms of written response pose difficulties for test designers when spelling errors are taken into

account. Therefore, as the number of words increase, the probability for potential variability in

answers greatly increases. As a result, it seems that CAA offers nothing more than a convenient

way for teachers to gather written-response answers from their students.

Another piece my evidence reveals that teachers sometimes wish to assessprocess over

product. One example might involve a teachers interest in a students thinking for a calculation

question involving steps; often students may receive marks for showing these steps. As written

out steps cannot easily be entered into computers, teachers are left with conventional methods

using paper.

In a related, but different way, students may need to form scaffolding as they approach a

problem. It is often practical to have a piece of paper in these situations so that a rough sketch or

mind map may be drawn. If students are not presented with the opportunity to form this

scaffolding, they may encounter difficulties with questions that require numerous stages of

thinking.


42/47

42

In addition to this argument, interviewees raised concerns over the varied typing ability

that may be found among students. Because this would be inherently true, some students may be

at a disadvantage when trying to type their textual answers into the computer. Of course, to

eliminate this problem, governments could make amendments to the curriculum to place more

emphasis on the acquisition of typing skill and if keyboarding was a mandatory course for

students entering high school, other benefits may be seen as they enter a society that is quickly

developing technologically. However, it is likely that variability in typing ability would still exist

despite mandatory training in schools. Also, students at the beginning of their training would not

be able to take part in online exams if written question were included. For these reasons, answers

that demand lengthy keyboarding do not seem to be a suitable match for CAA.

In conclusion, because teachers will continue to be motivated to question their students in

a varied fashion, it would follow that CAA would be impractical for some questioning styles.

Therefore, CAA could very easily be implemented for portions of an examination that involve

objective questioning styles. Then teachers will have an opportunity to utilize additional benefits

offered by CAA such as questioning with the use of video clips. However, it would be logical to

employ a pencil and paper approach for other methods of testing because due to the evident

restrictions involving computers. In conclusion, it is proposed that CAA be used to augment

current approaches of assessment and not to replace them.

A Call for CAA System Development

As for most technological innovations, improvements can be made as issues and

resolutions are identified. The gap that remains between educational needs and CAA systems on

the market seems to be narrowing gradually. Through this inquiry, the issue-themes suggest the


43/47

43

need for affordability and suitability. We will discuss two possible development veins that

technology could follow to meet the needs of education.

Student response systems have the added advantages of portability, durability, along with

the similar benefits of computer network systems. However, many SRS systems have software

that is tailored to other markets besides education. Often SRS is used to poll opinions of an

audience or to gather data during a corporate meeting. The questions that are posed are often

issued spontaneously. Because of this, software for these systems normally does not offer

capability for the acquisition and storage of question banks and student response data. Also, the

LCD graphic interface on the hand-held response units is often simple and lacking colour. The

reason for this becomes evident with the challenges that are faced when 30 or more students are

responding over the airwaves to a solitary receiving station. For these systems, data must be

transferred in both directions resulting in heavy data traffic in a wireless environment. As this

is the case, graphics rendered on response units have to be simple and generally lengthy portions

of text are not issued to these units. In the future, as improvements in data transmission come

about, response units may be able to offer similar features to screen-based CAA. This way,

questions might be issued to response units with improved displays, online testing rooms would

not be necessary with the portability advantages, and costs may be reduced if hand-held units

were durable and designed for long-term use.

Another possible direction of development of CAA systems could involve the

incorporation of accessible test bank building for WAN-based systems. As suggested earlier,

internet-based systems seem to be a preferred avenue for test delivery. However, one drawback

with these systems might be that teachers would be reliant on technical support each time they

wished to build a test. The teacher may choose the questions but the technician is needed to


44/47

44

convert the tests to a form that can be displayed on the internet. Essentially, the advantage

teachers would enjoy in being able to choose questions from a test bank is lost. Therefore, it

would be of value to pursue development of WAN-based systems so that password-protected test

banks could be built up over time as teachers issued online tests. To make the point, a well

known Chinese proverb may be fitting, Give a man a fish and you feed him for a day. Teach a

man to fish and you feed him for a lifetime. For this reason, advancements in online course

management systems (or CMSs), such as WebCT, could be a vehicle for restoring this

functionality to teachers. These systems offer a secure place for teachers to access tools and

functionality suited for teachers while students are restricted to viewing course material as

desired by the teacher. At the same time, student scores can be stored in digital grade books.

Although tests are already being delivered with these systems, considerable skill is required to

integrate these tests into the CMS software. However, for this stage, expert technical support

could be involved. The online system might work as follows:

1. Programmers could begin by taking teacher tests and converting them to a digital format.2. At the same time the questions within these tests could be stored in a central test bank on the

system for many teachers to use.

3. The technical support would then upload the test to the teachers account within the systemso that students could write the test.

4. Once the test had been written the students, grades would be automatically sent to theteachers grade book where teachers could access and analyze data at any time.

5. Then, for a future test, a different teacher at another school who wishes to issue a test on asimilar subject might log into their account on the same system, access the newly updated

test bank, quickly design a test, and initiate a similar but modified test in their own account.


45/47

45

This plan illustrates the possibility of integrating two ideas that currently exist as separate

entities: test creation software (with developed test banks) and online course management

systems (with secure individualized access and data management). Of course, this application

mash up would have an increased need for expert technical support to constantly maintain the

system. However, this demonstrates the possibility for improvements made to services and

systems offered by vendors for WAN-based CAA systems.

Review

This paper begins by illustrating that many new technologies are offered to

issues with online testing in the k to 12 school system and proposed remediation

Documents