digital forms of assessment extra report on analysis of 2008 … · 2011-12-11 · digital forms of...
TRANSCRIPT
Digital Forms of Assessment
Extra Report on Analysis of 2008 Data Applied Information Technology
A research project supported by a grant from the Australian Research Council Linkage Scheme
[Project Code: LP0882482]
For the Curriculum Council of Western Australia
June 2009
Centre for Schooling and Learning Technologies (CSaLT)
Edith Cowan University
6/
23/2
009
2 of
52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Investigating the feasibility of using digital representations of work for authentic and reliable performance assessment in senior secondary school courses. Research conducted by a team from Edith Cowan University under the leadership of Associate Professor Paul Newhouse and Associate Professor John Williams. AIT Research Team: Associate Professor Prof. Paul Newhouse , Mr. David Miller, Dr. Alistair
Campbell, Dr. Martin Cooper, Mr. Ian Gaynor, Ms. Lyn Sampson, Ms. Christine Woods (and the seven teachers whose classes were involved in the project).
This report was written by Paul Newhouse, David Miller, Alistair Campbell and Martin Cooper.
© 2009 School of Education, Edith Cowan University, Western Australia.
This is an addition to the first report to the Curriculum Council of Western Australia of research exploring the potential of using digital technologies to support assessment tasks for summative high-stakes purposes in the Applied Information Technology, Engineering Studies, Italian and Physical Education Studies courses. This report provides an analysis of all the data collected from February to December 2008 from the sample of classes involved in the Applied Information Technology course.
Contents
1 - EXECUTIVE SUMMARY...................................................................................................... 2 1.1 Summary of Results...................................................................................................... 2 1.2 Specific Recommendations for Universal Implementation in the Course....................... 3
2 – STATEMENT OF PROBLEM AND REVIEW OF LITERATURE ................................................ 6 2.1 Statement of Problem for AIT....................................................................................... 6 2.2 Review of Literature ..................................................................................................... 7
3 - METHOD ......................................................................................................................... 13 4 - AIT ASSESSMENT TASK IMPLEMENTATION ................................................................... 14
4.1 AIT Implementation and Technologies ....................................................................... 14 4.2 AIT Case Studies ........................................................................................................ 15 4.3 Online Repository....................................................................................................... 17 4.4 Analytical Marking and Analysis ................................................................................ 17 4.5 Comparative Pairs Marking and Analysis.................................................................... 20 4.6 Conclusions About Marking Process........................................................................... 22 4.7 Conclusions About Portfolio versus Exam .................................................................. 23 4.8 Applying a Rasch Model to the Exam Marks .............................................................. 25 4.9 Applying a Rasch Model to the Portfolio Marks.......................................................... 26 4.10 Conclusions From Student and Teacher Data ............................................................ 30
5 - CONCLUSIONS FROM ALL AIT FINDINGS........................................................................ 33 5.1 Conclusions About the Assessment Task .................................................................... 33 5.2 Conclusions About Methods of Marking..................................................................... 33 5.3 Summary of Findings.................................................................................................. 34 5.4 Summary of AIT Findings .......................................................................................... 37 5.5 Recommendations for Universal Implementation........................................................ 38
REFERENCES........................................................................................................................ 41 APPENDIX – AIT ASSESSMENT TASK 2008 .......................................................................... 43
Digital Reflective Process Portfolio .................................................................................. 43 Performance Tasks Exam ................................................................................................. 45 Criteria for Marking: AIT Assessment Task (ECU) .......................................................... 47 Performance Tasks and Reflective Exam (Marking Criteria)............................................. 49
2 6/
23/2
009
2 of
52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
1 - Executive Summary This report is in addition to the project report on the first year of a three-year study conducted at the Centre for Schooling and Learning Technologies (CSaLT) at Edith Cowan University (ECU) in collaboration with the Curriculum Council of Western Australia and supported by an Australian Research Council (ARC) Linkage research grant. The study commenced in January 2008, will be completed by December 2010, and concerns the potential to use digital technologies to represent the output from assessment tasks in four senior secondary courses, Applied Information Technology (AIT), Engineering Studies, Italian and Physical Education Studies (PES).
This report discusses the results concerning the Applied Information Technology course in more depth to provide recommendations to the Council.
The study involved seven AIT teachers and one class of senior-secondary students for each, with a total of 115 students participating. Both the portfolio and exam were implemented for all seven classes, however, for one class the portfolio was not submitted and the extent to which individual students completed all components of the portfolio varied considerably. The exam was completed by all students with almost no technical difficulties evident apart from the recording of sound for three classes. The portfolio was facilitated by the teacher, while the exam was facilitated by a researcher and the teacher.
1.1 Summary of Results Almost all students indicated a preference for the assessment of practical performance at a computer. Most indicated that both the portfolio and the exam provided a good assessment of practical performance. They commented on the ease of working on the computer compared to working on paper … correcting errors, speed of writing, amount of writing, speed of action and physical comfort. When pressed, their major concern was about malfunctions of systems during exam. However, for the few students for whom this occurred they were able to continue at an alternative workstation.
The manner in which the portfolio was implemented varied somewhat between teachers/classes although all teachers attempted to adhere to the requirements. Success in this endeavour tended to be connected with the extent to which this was included as a part of the school-based assessment rather than as an additional task. In general, students provided a more complete portfolio where it was included as part of their school-based assessment. The product component of the portfolio tended to be included in school assessment with four of the teachers using the default design brief that led to the development of a web site for a store. Generally the product requirements provided adequate scope for students to demonstrate their capability. The process document varied considerably in quality with some lack of understanding of technology process. Students needed to select the best information to include within page limit and many appeared to have difficulty with this. The two extra digital artefacts provided scope for presentation of a broader range of skills. Unfortunately many students did not make use of this with the submission of two websites. In the majority of cases students didn’t demonstrate a breadth of skill. However, some students did and included the half-page information sheet explaining their artefacts.
The reflective question component of the exam did highlight some discrepancies on portfolio products and process document for some students. Typing into the Word document was efficient but one-hour was too long and the results were of limited value. It is therefore recommended that the reflective questions component not be included in 2009. The performance tasks component of the exam provided scope for demonstration of capability. All students completed most requirements (some ran out of time and some lacked the skill). Generally this was implemented without technical difficulty apart from sound recording. Many students didn’t understand what a logo was and the inclusion of graphs was generally poor (basic graphs with little editing evident). Less than 20 students designed a logo and used drawing tools to create it. However, overall the brochures were well done by many students. The sound recording was of limited value as most students just read what they had typed for the reflection.
The results of marking using the analytical rubric-based approach provided a good spread of scores for both the portfolio and exam with very high correlations between the two markers (around r = 0.9). There were also significant moderate correlations between the scores on the portfolio and exam (around r = 0.5). There was little correlation with any of these marks and those provided by the teachers except for the two schools that most rigorously implemented the portfolio. Most teachers did not provide a set of marks for the assessment task but rather a semester mark and grade. The comparative pairs approach to marking
3
provided a reliable set of scores (SI = 0.93) that was significantly correlated to the analytical marking scores (around r = 0.7). There were similar outcomes for rankings created by the two marking approaches.
For the analytical marking of the performance tasks component of the exam Rasch analysis was completed that generated a reliable set of scores (SI=0.85). This analysis found that for one of the criteria (creation of the logo) there was inconsistency in the use of the two highest levels of performance that was fixed by combining them.
As a result of the success in 2008, it was recommended that for 2009 a very similar assessment task be used with the only major changes being the removal of the reflective exam component and the oral reflection in the performance tasks exam. This exam should follow the same structure but involve a new scenario and set of tasks, in line with the syllabus. Once again teachers will be strongly encouraged to include all components of the assessment task as part of school-based assessment for Semester Two.
1.2 Specific Recommendations for Universal Implementation in the Course This section makes recommendations based on an analysis of the 2008 data for the universal implementation of an electronic portfolio and/or a computer-based exam in the AIT course in WA. These two options were tried because internationally they are the most likely to be used to assess students’ capability in using ICT and WA teachers are familiar with these options. From the literature and the pilot study it was understood that each would have different strengths and weaknesses and thus a choice between them depends on the balance and the options for addressing weaknesses.
The syllabus states that the AIT course “provides opportunities for students to develop knowledge and skills relevant to the use of ICT to meet everyday challenges”. As such in the course students should “consider a variety of computer applications for use in their own lives, business and the wider community”. In the course students spend the majority of their time in class using digital technologies to develop information solutions. It should therefore be surprising that currently the external assessment consists of a three-hour paper-based exam. This is despite the fact that the syllabus stipulates that around 50% of the weighting of assessment should be on production. There is a strong rationale for quick change given that most students in Stage 2 and 3 units now have to submit to the final examination.
In general from the 2008 data it could be concluded that both options were able to be implemented successfully although the computer-based exam was the easiest to implement in a consistent fashion between schools. It appeared that only two of the seven teachers implemented the portfolio adequately according to the parameters agreed upon and this was reflected in the results, with students in these schools doing significantly better as a group. However, the portfolio was more reliably marked using the analytical method. The only implementation issues for the exam were the failure of audio recording in three schools and the handful of students who had to move workstations mid-exam due to technical failure at a cost of no more than five minutes.
While the exam was technically implemented well the analytical marking was not as reliable as the portfolio (correlation between markers were statistically significant both for the performance exam, r=0.43, and for the portfolio r=0.79) although the comparative pairs method was highly reliable (SI=0.93). However, Rasch analysis showed that with a minor modification to one criterion reliable scores (SI=0.85) were generated for the exam even from the analytical marking. The main reason for lower reliability for the exam appeared to be because on a few criteria very few top marks were given (e.g. file formats, logo and brochure – explanation or justification).
Rasch analysis of the portfolio analytical marks (Product, Process Document and Extra Artefacts analysed separately) gave a reliable set of scores for all three components (SI=0.96, 0.96 and 0.94 respectively). No modifications were required although for the Extra Artefacts the thresholds for three of the criteria did not work very well. The analytical marks generated by the three components of the Portfolio were not highly correlated probably indicating that they were addressing different types of performance.
A major weakness of the exam was that a relatively low level set of tasks were required (logo, brochure, and spreadsheet graph) so that students in all classes could attempt them using a typically standard set of software that they would all have available. The types of practical tasks teachers give students varies tremendously (e.g. many teachers do not give database type of tasks or even spreadsheets while others will not do animations or movie editing). Therefore it is very difficult to set tasks that all students would
4 6/
23/2
009
4 of
52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
have the background to attempt and would also allow adequate scope for the more capable students. This is not a problem with the portfolio as the design brief could vary between classes and typically allowed tasks to be relatively open-ended.
The following table summarises the comparison of the two forms of assessment, Portfolio and Exam, using the feasibility dimensions.
Portfolio Computer-Based Exam
Man
agea
bilit
y Readily implemented by teachers. Needs strict invigilation procedures. Submission via disc – online system would be preferable.
Implemented by teacher and researcher in collaboration – would require external invigilator and teacher. Facilitated using USB Flash Drive – online system preferable but may lead to network difficulties. Difficult to ensure students can’t view each others screens
Tech
nica
l
No major technical issues to implement. Some difficulty marking with range of file types and some large files. Files have to be checked by assistant. Teachers set tasks appropriate to the technology available.
Needs to be tested on site to ensure all required technologies work. At most three workstations failed in a class with students moved to spare workstations. In some cases there were no failures.
Valid
ity
Inconsistency of implementation with only two teachers doing so adequately. Good discrimination between students. The three components were not highly correlated indicating they added information.
Difficulty in setting tasks that all students could attempt and would extend more capable students. Quite good discrimination between students. Limited context and set of skills upon which to judge overall capability.
Func
tiona
l
Rel
iabi
lity
Highly reliable analytical marking (SI=0.96). Each of the three components was reliably marked but not highly correlated. Not all students submitted all components. Would be more difficult to mark using pairs.
Less reliable analytical marking (SI=0.85). Highly reliable comparative pairs marking (SI=0.93).
Peda
gogi
c Readily aligns with teachers’ typical pedagogical practices. All teachers already had a similar portfolio. Students liked the form of assessment and were familiar with it.
Was a new experience for two teachers and almost half the students. Students liked the form of assessment but were not as familiar with it as the portfolio.
Recommendation Overall when comparing the Portfolio and the Exam there was no compelling reason to choose one over the other, each had strengths and weaknesses. Therefore it is recommended that the Council make its decision on the basis of how well each could be implemented in the manner outlined in the next two sections. The choice is a limited portfolio, a computer-based exam or perhaps a combination.
Implementing the Portfolio If the Portfolio was implemented the structure used in the study is recommended. This allows students to adequately demonstrate their capability with some scope for tailoring to the context for the student. However, ideally it would need an online portfolio management system and would need a well-structured system for verification that would probably include some type of signed affidavit with spot checks on a sample of students to ensure all teachers implemented the portfolio according to the required conditions.
(1) Digital product (a 15-hour project)
(2) Process document (a 5-hour task to collate evidence)
(3) Two previously created digital artefacts (a 1-hour task to select and present the artefacts)
A set of parameters needs to be set for the Portfolio including the following.
• Time limits for development of digital product (15 hours within 4 weeks), process document (5 hours), and artefact preparation (1 hour).
5
• A choice of design briefs is needed, ideally teachers would be allowed to set an appropriate design brief within the parameters of the portfolio (e.g. aim, purpose, included components, file sizes and types). However, many teachers will not feel confident to do this and thus three or four example design briefs should be supplied for them to use or modify.
• The nature of the process document (nine pages total with suggested limits for each of the four sections: Research, Design, Production and Evaluation) as a selection and collation of material resulting from the development of the digital product that best presents the process of development employed.
• The nature of the extra artefacts as demonstrations of skills in areas other than that demonstrated in the main digital product (short support document – half a page table).
• The components of the Portfolio need to be marked separately using different criteria as they represent different types of performance (e.g. the Process Document provides an opportunity to demonstrate an understanding of the Technology Process and capability in planning, analysing, organising, managing and evaluating).
Implementing the Computer-Based Performance Exam If the performance tasks exam was implemented then the structure of the exam used in the study is recommended with some minor modifications (e.g. removal of audio response). However, the study has highlighted two areas in which decisions would need to be made: technical implementation; and performance tasks specification.
Technical Implementation
The project used USB flash drives that worked on school computers for all students. This would be cumbersome but not unrealistic to scale up state-wide but in the long-term an online exam management system should be used. However, this introduces the school network as a variable that our project found was a confounding variable in many schools.
There is the question of whether access to the Internet should be precluded. In 2008 this was not a requirement although at least two schools did this of their own accord. There was a requirement that students could only use the media (e.g. photos) that were provided on the USB flash drive. In one class at least one student used an online text design tool.
In 2008 no attempt was made to limit access to software that was normally available to the students. The low-end nature of the tasks meant that this provided little, if any, advantage to any students with most using Microsoft software (e.g. Publisher) and a relatively basic graphics package. The choice would seem to be to specify the software allowed OR allow all available OR provide software on the USB (probably not practical for AIT).
The option of paper-based or digital design phase appeared to be appreciated by students with most opting for paper-based design. These then had to be scanned for marking by project assistants.
Performance Tasks Specification
In the study the performance tasks were selected to be relatively low-level and easy to ensure all students could engage. Further these tasks were defined fairly explicitly with little or no choice for students. Clearly if this were done for the final external assessment then it would limit the opportunity for high ability students and would tend to stifle context variety in the course. Therefore it is likely that some degree of choice of context, type of solution and/or tools used is needed but this makes the development of appropriate tasks, description of those tasks and the statement of assessment criteria more difficult.
Given that currently many teachers do not appear to be addressing all the content, with some focusing almost exclusively on interactive multimedia development and others on business software, the exam would have to follow a pre-defined structure so that teachers could adjust their programmes accordingly. For example, it may need to be known that students will have to develop a graphic, a spreadsheet, access a database, and hyperlink media elements etc. What will then be unknown is the specific scenario to be addressed and the associated contextual media provided.
In the second year we will trial a more complex task that allows for more choice.
6 6/
23/2
009
6 of
52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
2 – Statement of Problem and Review of Literature The aim of the research project concerns the feasibility of using ICT to support performance assessment.
In contrast to the other three courses in the project for the Applied Information Technology course digital technologies provide the content for study as well as pedagogical support. Therefore performance relates to using the technologies to demonstrate capability in using the technologies.
2.1 Statement of Problem for AIT The syllabus states that the AIT course “provides opportunities for students to develop knowledge and skills relevant to the use of ICT to meet everyday challenges”. As such in the course students should “consider a variety of computer applications for use in their own lives, business and the wider community”. In the course students spend the majority of their time in class using digital technologies to develop information solutions. It should therefore be surprising that currently the external assessment consists of a three-hour paper-based exam. This is despite the fact that the syllabus stipulates that around 50% of the weighting of assessment should be on production.
Towards the end of 2006 an external assessment brief had been developed that provided for an electronic portfolio and a written exam with equal weighting, however, at that stage the details of what would go in the portfolio and the format of the written exam had not been finalised with some disagreement over the example that had been created. The main reasons why a portfolio had been recommended were that it was assumed to be the easiest to implement in the short term, would allow for the variety of contexts, would support the practical performance nature of the course, and was already familiar to teachers in most of the subjects the course was replacing. It was not assumed that this would necessarily be the best long-term solution but that more time and research was required.
Then early in 2008 courses like AIT were changed with the decision that all students in Stage 2 and 3 units were to submit to the external examination process. The ramifications of this decision were likely to be widespread including that the ‘exam’ would have to be appropriate for lower achieving students, it would dominate the course delivery more and would involve a lot more students, increasing the cost considerably. Originally it had been assumed that because only higher achieving students were likely to be involved the extra time needed to collate a portfolio was reasonable and would only include higher quality work that would be easier to mark.
Another confounding change was the requirement for the course to be packaged in a syllabus format with details of specific content for each unit rather than what had been a definition of the boundaries of the content with the opportunity to address the content to varying depths across the units and depending on relevant contexts for the students and teacher. This also led to a shift of focus away from outcomes towards content that immediately highlighted the issue of the variety of relevant contexts that could be involved in the course and the issue of the rapidly changing content of these areas of technology. This had not been such an issue with the focus on outcomes because they could be applied to the range of contexts and did not specify particular content that could quickly date. This has since led to the focus for assessment being on assessment type rather than outcomes.
While students can include study in AIT towards University entry this would be of no value if the external assessment propels the course towards becoming mainly ‘book work’ rather than creative digital work. We are living in a society where almost every avenue of work and life requires the use of digital tools and resources. Whether a senior student is aiming to be a mechanic, doctor, accountant or travel agent study in AIT could begin to give them the skills, attitudes and understanding that will support them in being more successful in work and life.
There are a number of ways in which students could be assessed on their use of digital technologies. Principally these are either forms of portfolio, forms of computer-based exam or combinations. Among educational leaders and local teachers there are differences in opinion over which method of assessment of practical performance would be best. Each has strengths and weaknesses for different situations.
Therefore the research problem for the AIT course becomes that to align with the aims, rationale, outcomes, content and preferred pedagogy assessment must include students using digital technologies but there are a number of ways in which that may be achieved. The research question therefore becomes,
7
which method of assessment, portfolio or computer-based exam or combination, is most feasible for the course at this time?
2.2 Review of Literature The study connects with two main fields of research: performance assessment, and computer-supported assessment. However, clearly these are subsumed within the general field of assessment. While it will be assumed that the basic constructs within the field of assessment are known and apply perhaps it is useful to be reminded of this through a statement of the three pillars that Barrett (2005) suggests provide the foundation for every assessment rests.
1. A model of how students represent knowledge and develop competence in a content domain. 2. Tasks or situations that allow one to observe students’ performance. 3. An interpretation method for drawing inferences from performance evidence.
2.2.1 Computer-Supported Assessment Computer-Supported Assessment, sometimes referred to as Computer-Assisted Assessment, is a broad term encompassing a range of applications from the use of computers to conduct the whole assessment process such as with on-screen testing, to only assisting in one aspect of the task assessment process (eg. recording performance or marking) (J Bull & D Sharp, 2000). The first area of the task assessment process that took advantage of computer-support was objective type assessments that automated the marking process (eliminating the marker) and allowed the results to be instantly available. Bull and Sharp (2000) found that the use of computers to support assessment has many advantages for the assessment process, assessors and students.
Much of the published research in the field of computer-supported assessment relates to higher education, particularly in university settings (e.g. Brewer, 2004), with little specific to school-based education. However, in the school sector assessment of student creative work in the arts has been addressed for some time with, for example, Madeja (2004) arguing the case for alternatives to paper-and-pencil testing for the arts. Further, there has been some research into the use of portfolios for assessment but most often this is for physical, not digital, portfolios. There has been a limited amount of research in the area in Australia, typically these have been small-scale trials in the use of IT to support assessment processes (e.g. Newhouse, 2005). There have also been reports on the use of online testing in Australia, such as by MacCann (2006), but these usually do not involve assessing practical performance and merely replicate paper-and-pen tests in an online environment.
While there has been only limited empirical research into many areas of computer-supported assessment there are many useful theoretical discussions of the issues such as Spector’s (2006) outline of a method for assessing learning in “complex and ill-structured task domains”. While providing useful ideas and rationales these ideas remain largely untested in the reality of classrooms. What is known is that any use of ICT involves school change (Lim & Hung, 2003; Newhouse, Clarkson, & Trinidad, 2005) and will require training of teachers, changes in thinking, and pedagogical understandings that are difficult to take on, even for younger teachers (Newhouse, Williams, & Pearson, 2006).
There has been increasing interest internationally in the application of computer support to improve assessment as indicated in the focus of a recent keynote address by McGaw (2006). The University of Cambridge Local Examinations Syndicate is conducting over 20 projects to explore the impact of new technologies on assessment including using online simulations in assessing secondary science investigation skills (Harding, 2006). Other organisations (e.g. Becta, 2006) or groups of researchers (e.g. Ridgway, McCusker, & Pead, 2006) have reported on exploratory projects, particularly the increasing use of online testing, although rarely for high-stakes assessment and not without some difficulty (Horkay, Bennett, Allen, Kaplan, & Yan, 2006).
The British Psychological Society has produced a set of guidelines for Computer-Based Assessment. While they mainly focus on online testing they provide a conceptual model that includes Assessment Generation, Assessment Delivery, Assessment Scoring and Interpretation, and Storage, Retrieval and Transmission. The latter two were relevant to the present study with the guidelines for developers and users.
8 6/
23/2
009
8 of
52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Recently the Joint Research Centre for the European Commission (Scheuermann & Bojornsson, 2009) brought out a major report titled, The Transition to Computer-Based Assessment. Kozma (2009) lays out the rationale in terms of a mismatch between what is needed in modern society and what is addressed and thus assessed at school. In particular he draws attention to the differences between standardized pen-and-paper assessment and “Tasks in the Outside World”. In the latter he explains how tasks: require cross-discipline knowledge; relate to complex ill-structured problems; and are completed collaboratively using a wide range of technological tools to meet needs and standards. These characteristics are at odds with traditional approaches to assessment. While he does not see assessment reform only requiring the use of ICT he outlines a number of significant advantages including: reduced costs; increased adaptability to individuals; opportunity to collect process data on student performance; the provision to tools integral to modern practice; and better feedback data. Kozma does introduce a number of challenges to using ICT to support assessment including: start-up costs for systems; the need to choose between standardized and ‘native’ applications; the need to integrate applications and systems; the need to choose between ‘stand-alone’ and online implementation; the need for security of data; the need for tools to make the design of tasks easy and efficient; and the lack of knowledge and examples of high-quality assessments supported by ICT. He also highlights methodological challenges including: the extent of equivalence with pen-and- paper; the design of appropriate complex tasks; making efficient and reliable high-level professional judgements; scoring students’ processes and strategies; and distinguishing individual contributions to collaborative work.
A recent research initiative of Cisco, Intel and Microsoft (Cisco, 2009) is the Assessment and Teaching of 21st Century Skills project. The paper that was a call to action clearly argues that changes are required to high stakes assessments before needed change will occur in schools.
Reform is particularly needed in education assessment - how it is that education and society more generally measure the competencies and skills that are needed for productive, creative workers and citizens. Accountability is an important component of education reform. But more often than not, accountability efforts have measured what is easiest to measure, rather than what is most important. Existing models of assessment typically fail to measure the skills, knowledge, attitudes and characteristics of self-directed and collaborative learning that are increasingly important for our global economy and fast changing world. New assessments are required that measure these skills and provide information needed by students, teachers, parents, administrators, and policymakers to improve learning and support systemic education reform. To measure these skills and provide the needed information, assessments should engage students in the use of technological tools and digital resources and the application of a deep understanding of subject knowledge to solve complex, real world tasks and create new ideas, content, and knowledge. (Cisco, 2009, p. 1)
Ripley (2009) defines e-assessment as “the use of technology to digitise, make more efficient, redesign or transform assessments and tests; assessment includes the requirements of school, higher education and professional examinations, qualifications, certifications and school tests, classroom assessment and assessment for learning; the focus of e-assessment might be any of the participants with the assessment processes – the learners, the teachers and tutors, managers, assessment and test providers and examiners. He highlights presents two ‘drivers’ of e-assessment; business efficiency and educational transformation. The former leads to migratory strategies (i.e. replicating traditional assessment in digital form) while the latter leads to transformational strategies that change the form and design of assessment. An example of the latter is the recent ICT skills test conducted with 14-year olds in the UK in which students completed authentic tasks within a simulated ICT environment. He raises issues that need to be addressed including: providing accessibility to all students; the need to maintain standards over time; the use of robust, comprehensible and publicly acceptable means of scoring student’s work; describing the new skill domains; overcoming technological perceptions of stakeholders (e.g. unreliability of IT systems); and responding to the conceptions of stakeholders about assessment.
Lesgold (2009) calls into question the existence of a shared understanding among the American public on what is wanted out of schools and how this may have changed with changes in society. He argues that this must go with changes to assessment to include 21st century skills and this will not be served by the traditional standard approach to testing based on responses to small items that minimises the need for human judgement in marking. Instead students will need to respond to tasks representing complex performances, supported by appropriate tools with the results needing to be judged by experts. He recognises the issues that this would throw up and provides ‘stealth assessment’ as an example solution.
9
In this example students complete a portfolio of performance at school over time and supervised by the teacher. The testing system then selects one or two “additional performances” to be externally supervised “as a confirmation that the original set was not done with inappropriate coaching” (p. 20). This is more amenable to ‘learning by doing’ and project-based learning where bigger, more realistic tasks can be accomplished that develop attributes such as persistence.
2.2.2 Assessment of Practical Performance Research in, and the call to investigate “performance-and-product assessment” is not new as pointed out by Messick (1994, p. p. 14), tracing back at least to the 1960s. However, Messick claims that mainstream schooling showed little interest in this in until an “upsurge of renewed interest” in the 1990s with “positive consequences for teaching and learning” (p. 13). While Messick does not specifically address digital forms of performance assessment, his arguments for the need to address “issues of validity, reliability, comparability and fairness” apply, particularly to a range of validity criteria. He argues they are social values that require close attention to the intended and unintended consequences of the assessment through considerations of the purposes of the assessment, the nature of the assessed domain, and “construct theories of pertinent skills and knowledge” (p.14). For example, he outlines situations under which product assessment should be considered rather than performance assessment. The issue is their relationship to replicability and generalisability requirements because these are important when performance is the “vehicle” of assessment.
Lane (2004) claims that in the USA there has been a decline in the use of performance assessments due to increased accountability requirements and resource constraints. She outlines how this has led to a lack of alignment between assessment, curriculum standards, and instructional practices; particularly with regard to eliciting complex cognitive thinking. At the same time Pollitt (2004) argues that current methods of summative assessment that focus on summing scores on “micro-judgements” is “dangerous and that several harmful consequences are likely to follow” (p. 5). Further, he argues that it is unlikely that such a process will accurately measure a student’s “performance or ability” (p. 5), and more holistic judgements of performance are required. Koretz (1998) analysed the outcomes of four large-scale portfolio assessment systems in the USA school systems and concluded that overall the programmes varied in reliability and were resource intensive with “problematic” (p.309) manageability. This body of literature clearly presents the assessment of student performance as critically important but fundamentally difficult with many unanswered questions requiring research.
Globally interest in performance assessment has increased with the increasing use of standards-referenced curricula. Standards-referenced curricula have evolved over the past 20 years particularly from the UK and more recently in Australian states since the early 1990s. The key concept in these curricula was that student achievement was defined in terms of statements describing what students understood, believed or could do. The term standards-referenced has tended to be used recently to indicate that student achievement is measured against defined standards.
Koretz (1998), who defines portfolio assessment as the evaluation of performance by means of a cumulative collection of student work, has figured prominently in USA debate about education reform. He analysed the outcomes of four large-scale portfolio assessment systems in the USA school systems, in particular, in terms of their reliability. Each example involved marking student portfolios for the purpose of comparing students and/or schools across a state, mainly in English and Mathematics. All of the examples occurred in the 1990s and none involved digital representations of performance. Koretz concluded that overall the programmes were resource intensive and did not produce “evidence that the resulting scores provide a valid basis for the specific inferences users base on them…” (p.332). Even though he noted that significant improvements in the implementation and reliable marking of portfolios had been achieved, at that time he saw portfolio-based assessment as “problematic” (p.309). Findings such as this provide a rationale for considering digital solutions to performance assessment.
2.2.3 Design and Development of Digital Assessments As in most areas of education, and particularly for assessment, authorities and/or researchers in many localities have developed guidelines for the use of digital technologies with assessment processes. For example, the British Psychological Society published a set of general guidelines for the use of “Computer-Based Assessments” through its Psychological Testing Centre (The British Psychological Society, 2002). These guidelines include the use of digital technologies in Assessment Generation,
10 6/
23/2
009
10 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Assessment Delivery, Assessment Scoring and Interpretation, Storage, Retrieval and Transmission. These guidelines are defined from a developer and user perspective. Similarly, The Council of the International Test Commission developed international guidelines for good practice in computer-based and Internet delivered testing (The Council of the International Test Commission, 2005). These were focussed on four issues: the technology, the quality of the testing, the control of the test environment, and the security of the testing. The contexts considered all involved students sitting at a computer to complete a test.
All assessment items are required to be valid, educative, explicit, fair and comprehensive, and should allow for reliable marking. The descriptions of the digital assessment tasks below assume this but focus on any areas that are of a particular challenge to that assessment type.
2.2.4 Guidelines Specific to Computer-Based Exams Computer-based exams involve students sitting at computer workstations completing tasks, including typing answers to questions. They may be required to use various pieces of software to create digital products or may simply use a browser to complete response type assessment. In AIT while both types of assessment activities may be involved it is likely that the focus would be on the former.
The International Test Commission has provided detailed guidelines for computer-based exams (The Council of the International Test Commission, 2005). These guidelines were specific to test developers, test publishers and users and mainly related to response type assessment. An array of specific guidelines was presented according to the following structure.
(1) Give due regard to technological issues in Computer Based Testing (CBT) and Internet testing
a. Give consideration to hardware and software requirements
b. Take account of the robustness of the CBT/Internet test
c. Consider human factor issues in the presentation of material via computer or the Internet
d. Consider reasonable adjustments to the technical features of the test for those with disabilities
e. Provide help, information, and practice items within the CBT/Internet test
(2) Attend to quality issues in CBT and Internet testing
a. Ensure knowledge, competence and appropriate use of CBT/Internet testing
b. Consider the psychometric qualities of the CBT/Internet test
c. Where the CBT/Internet test has been developed from a paper and pencil version, ensure that there is evidence of equivalence
d. Score and analyse CBT/Internet testing results accurately
e. Interpret results appropriately and provide appropriate feedback
f. Consider equality of access for all groups
(3) Provide appropriate levels of control over CBT and Internet testing
a. Detail the level of control over the test conditions
b. Detail the appropriate control over the supervision of the testing
c. Give due consideration to controlling prior practice and item exposure
d. Give consideration to control over test-taker’s authenticity and cheating
(4) Make appropriate provision for security and safeguarding privacy in CBT and Internet testing
a. Take account of the security of test materials
b. Consider the security of test-taker’s data transferred over the Internet
c. Maintain the confidentiality of test-taker results
Clearly many of the guidelines apply generally to any test-taking context (e.g. 2d, 2e and 2f), whether on computer, or not. Many of the other guidelines were not applicable to the current project (e.g. 4a, b and c) because only single classes and their teachers in particular schools were involved. However, many of the
11
guidelines in the first three areas were relevant to one or more of the cases in the project. For example, some of the guidelines associated with 1a, 1b, 2a and 2b were relevant, and to some extent some guidelines associated with 3a, 3b and 3d were relevant. Even so they were mainly relevant to the implementation of large scale online testing.
More recently there has been increased international interest in computer-based testing to assess ICT capability that is more relevant to the AIT course. For example, over the past year an international research project, the Assessment and Teaching of 21st Century Skills project, has commenced supported by of Cisco, Intel and Microsoft. There have also been trials of such tests in a number of countries including the UK, Norway, Denmark, USA and Australia (MCEETYA., 2007). In Australia the ACER used a computer-based test to assess the ICT literacy of Year 6 and 10 students. They developed the test around a simulated ICT environment and implemented the test using sets of networked laptop computers. While they successfully implemented the test with over 7000 students this was over a long period of time and would not be scalable for an AIT examination. Also the use of a simulated environment would be expensive and not scalable to provide a great enough variety of activities each year. The trial in the UK also involved a multi-million pound simulated system but was accessed by students through their school computers. In the Norwegian example students used their own government-provided notebook computers. In the USA a decision has been made to include an ICT literacy test in national testing in 2012 but in a number of states there are already such tests.
2.2.5 Guidelines Specific to Digital Portfolios The main concerns with the use of digital portfolios for assessment are:
• The authentication of student work given the period of time within which work is completed
• Ensuring that they are fair to all students in terms of access to information, materials and tools
• That they can be marked reliably given the usually varied types of student work output.
Therefore it is often recommended that the portfolio require a particular structure and limit the contents in type and size, the time available be controlled, and the work be authenticated by a teacher and the students. In a review of e-assessment it was suggested that a digital portfolio may involve three sections: student self-awareness; student interaction; and thinking about futures and informed decisions (Ridgway, McCusker, & Pead, 2006). In British Columbia, Canada, students complete a graduation portfolio. They are provided with a number of guides as Word documents that act as templates to construct their portfolios.
Carney (2004) developed a set of critical dimensions of variation for digital portfolios:
(1) Purpose(s) of the portfolio;
(2) Control (who determines what goes into the portfolio and the degree to which this is specified);
(3) Mode of presentation (portfolio organisation and format; the technology chosen for authoring);
(4) Social Interaction (the nature and quality of the social interaction throughout the portfolio process);
(5) Involvement (Zeichner & Wray identify degree of involvement by the cooperative teacher important for preservice portfolios; when considered more broadly, other important portfolio participants might include university teachers, p-12 students and parents, and others); and
(6) Use (can range from low-stakes celebration to high-stakes assessment).
Barrett (2005) suggests the following.
Identify tasks or situations that allow one to assess students’ knowledge and skills through both products and performance. Create rubrics that clearly differentiate levels of proficiency. Create a record keeping system to keep track of the rubric/evaluation data based on multiple measures/methods. Provide opportunities for students to learn and resubmit, maximizing diagnosis and remediation. Model the power of assessment as learning. (p. 10)
She goes on to suggest that for “Portfolios used for Assessment of Learning” that is for summative assessment the following are defining characteristics.
• Purpose of portfolio prescribed by institution
12 6/
23/2
009
12 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
• Artefacts mandated by institution to determine outcomes of instruction
• Portfolio usually developed at the end of a class, term or program - time limited
• Portfolio and/or artefacts usually "scored" based on a rubric and quantitative data is collected for external audiences
• Portfolio is usually structured around a set of outcomes, goals or standards
• Requires extrinsic motivation
• Audience: external - little choice
Beetham (n.d.) finds that e-portfolios are “less intimidating for some learners than a traditional examination” and provide evidence that gives a “much richer picture of learners’ strengths and achievements than, for example, a test score” (p. 4). She points to the need for web-based relational database systems to implement portfolios. While she points out that in the past e-portfolios have been found to take longer to moderate and mark this has become more streamlined where this is part of a “integrated assessment facility” – she provided five commercial examples of such systems. She provides a list of “issues relating to the use of e-portfolios for summative assessment” (p. 5). Seven of the nine issues are technical and most are addressed by the use of a good assessment management system. The remaining issues are:
• Acceptability and credibility of data authenticated by Awarding Bodies
• Designing assessment strategies to make effective use of the new tools and systems
• Ensuring enhanced outcomes for learners, e.g. higher motivation, greater choice over evidence, assessment around capabilities and strengths
She also raises some issues for teachers and learners (p. 16).
• Fit with existing practices and expectations.
• Access and ICT capability of teachers and learners.
• Acceptability and appropriateness of e-portfolio use.
On most of these issues it is easy to argue that for the AIT they are not issues as it has become normal practice over many years for school-based assessment. Provided there is a good assessment management system the only issue that may not be addressed currently for AIT is the “Acceptability and credibility of data authenticated by Awarding Bodies”.
13
3 - Method The general aim of this study was to explore the potential of various digitally based forms for external assessment for senior secondary courses in Western Australia. Specifically there was a need to determine the cost effectiveness of each format in terms of the need to support an outcomes-based curriculum framework for students in schools across the state. The problem being addressed was the need to provide students with assessment opportunities in new courses that were on the one hand authentic, where many outcomes do not lend themselves to being assessed using pen and paper over a three hour period, while on the other hand being able to be reliably and manageably assessed by external examiners. That is, the external assessment for a course needs to accurately and reliably assess the outcomes without a huge increase in the cost of assessment.
Therefore the specific aims of the research were:
1. To determine the feasibility of each digitally based form to support authentic summative assessment of student practical performance in different types of courses.
2. To establish ways of applying the comparative pairs marking procedure to digital forms of summative assessment to achieve high reliability in a cost-effective manner.
3. To extend existing conceptual models for digital forms of performance assessment. The first year of the study was a ‘proof of concept’ project to explore the feasibility of particular digitally-based formats for external assessment for four courses. The feasibility was investigated within a framework consisting of the four dimensions: technological, pedagogic, manageability, and functionality. This built on the work by Kimbell and Wheeler (2005) from the United Kingdom.
The study was evaluative in nature set within an ethnographic framework in that activity was considered to occur within learning environments where the characteristics of teachers and students and the culture created are critical to an understanding of all aspects of the curriculum and pedagogy, including assessment. Therefore, this project employed an ethnographic action research evaluation methodology using interpretive techniques involving the collection of both qualitative and quantitative data. The study drew on the traditions of interpretive research but also sought to employ, where appropriate, the quantitative methods of more traditional positivist research. That is, quantitative measures concerning achievement, costs and resource use were used but need to be interpreted within the context of the learning environment and in which the measures occurred.
The main research question became:
How are digitally based representations of student work output on authentic tasks most effectively used to support highly reliable summative assessments of student performances for courses with a substantial practical component?
The research design can be described as participative action research with participants contributing to development through evaluative cycles. As such this required an analysis of the perspectives of the key groups of participants (teachers, assessors, students) with data collected from each group. These data were compiled into case studies within a multi-case approach (Burns, 1996) in which each case is defined by one digital form of assessment in one class for one course. This approach allowed for refinement and further development of findings based on multiple instances of the same phenomenon under different conditions (Willig, 2001). Therefore, this study largely employed an ethnographic action research evaluation methodology using interpretive techniques involving the collection of both qualitative and quantitative data.
The study seeks to involve as many schools, teachers and students as possible. The AIT course was selected because at least two of its outcomes were directly related to the production of digital materials and students and teachers were more likely to have adequate competence in the use of ICT to be able to readily implement a range of digitally-based formats of assessment. The Engineering course was selected because it is a completely new course and its outcomes include processes and practical performance. The Italian course was selected because it required the assessment of oral language skills. The Physical Education Studies course was selected because it involved human movement performance. However, application to many other courses was considered when drawing up the conclusions.
14 6/
23/2
009
14 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
4 - AIT Assessment Task Implementation The results of the analysis of data related to the implementation of the AIT assessment task are presented in this chapter starting with a discussion of the implementation of the task and the technologies employed. This is followed by results of an analysis of the data collected from marking the students’ work and from surveys and/or interviews of students, teachers and assessors. Results for each school are not discussed separately as these were presented as case studies in the previously submitted first report.
4.1 AIT Implementation and Technologies The AIT assessment task was implemented at seven schools with a total of seven classes (one at each school) of Year 11 or 12 students studying the AIT Unit 2B. Each of the seven classes incorporated the three components of the portfolio and two components of the computer-based exam within their second semester programme. Each teacher was permitted to set their own design brief for the portfolio product although four used the example, The Miss Shoppe website, provided with the project documentation. This section describes in general how the task was implemented and the technologies used in the implementation.
4.1.1 Task Implementation Although there were some differences in the manner in which the assessment task was implemented for the six classes of students in most ways they were similar. Each class was visited at least four times to complete the following five sections. All of the sessions were conducted by the teacher in a computer laboratory at the school. For the exam session(s) at least one researcher assisted the teacher in invigilation.
Portfolio Product Development
Each teacher was permitted to set their own design brief for the portfolio product although four used the example, The Miss Shoppe website, provided with the project documentation. Students had 15 hours of class time over 4 weeks to develop a prototype. All work was supposed to be completed in class but teachers varied in the extent to which this was policed.
Hardware and software were restricted to those available at the school. On completion, evidence of the investigation, design, production and evaluation processes undertaken was to be collated into a Design Process Document. The focus of the activity was the application of the whole technology process to a real-world context, as set out in the scenario contained in the design brief. The product was required to:
• suit the intended purpose and audience/users; • meet the requirements of the design brief and/or client specifications; • illustrate creative application of information design principles and technologies; • make use of appropriate information structures, forms, layouts and symbols; • employ relevant standards and conventions to create multi-tiered information solutions; and • use appropriate methods and techniques to represent the design of information solutions.
Portfolio Design Process Document
Students had 5 hours of class time to collate a design process document using material from their product development. This was in four sections (Investigation, design, produce, evaluate) with prompting questions and page limits for each. Some teachers permitted students to complete this at home and to collaborate although this was not the intention.
Portfolio Extra Artefacts
Students submitted two additional digital artefacts they had created in the course along with a half-page form explaining the artefact. It was intended that these should be demonstrate ICT skills and knowledge that was different from the product development but this was generally not emphasised by teachers.
Exam Part A: Reflective Response Questions
The first part of the examination was allocated one hour and typically was done with the second part, but was not required to be so. Students were asked to respond to a series of reflective questions by typing
15
their answers into a Microsoft Word document provided on the USB drive. The questions asked them to reflect on the product development component of the portfolio.
Exam Part B: Production
The second part of the examination was allocated two hours and typically was done with the first part, but was not required to be so. Students were given a real-world design brief and prompted to follow the technology process to create a digital product. With the exception of design sketches, which had the option of being paper or computer based, the entire examination was done on computer, students’ responses being saved as digital files in various formats.
Students were given a paper copy of the examination, a 4GB USB mass storage device and an audio headset with microphone. There was 10 minutes reading time prior to the commencement of the three-hour paper which was completed under examination conditions with the teacher and researcher invigilating. The USB device contained 18 digital photographs, a text file of data, design templates in MSWord and PowerPoint and a template for preparation of an audio reflection in MSWord. A hard copy of the design template in MSWord was also supplied to give students the option of designing on paper. Students were permitted to use any software available on their desktop computer saving their work to the mass storage device. Based on previous experience, the written and practical sections were reversed with the two-hour practical test preceding the reflective questions. Students were not permitted to continue with the practical once the two hours allocated to it had expired.
4.1.2 Technologies All student work, apart from the initial design section of Exam Part B, had to be completed using computer systems. For the production component of the portfolio students may have also used other peripheral devices such as cameras and other software depending on the nature of the design brief.
Technologies for Exam
The implementation of the exam components of the assessment task required the use of a computer workstation, a USB flash-drive, a microphone and headphones and appropriate software (office and graphic production). The teacher at the school was responsible for setting up the workstations while the researcher provided the USB flash-drive and microphone/headphone.
Collecting Student Work
For the computer-based exam all student digital work was saved to the USB flash-drive and typically a copy was also saved to the school’s server. Student design work that was done on paper was collected and either scanned or photographed to add to their digital work. All digital work was then transferred to a folder, named using the student’s ID, on a University server, so that the work could be accessed by markers.
Typically student portfolio work was provided by the teacher on a disc and organised by student folders. These files were transferred to the student folders on the University server with files named in a consistent fashion.
4.2 AIT Case Studies There were seven schools, seven teachers and seven classes of Year 11 or 12 students involved in the project associated with the AIT course. Seven case studies were constructed and are reported upon in this section. All students involved in the study were enrolled in AIT Unit 2B.
The full range of research data were sought for each case study, however, for each case there were data missing. For example, no teacher provided a full set of marks although all but one provided a set of semester marks. The survey and interviews were conducted for all case studies. For one case study no portfolios were submitted.
Surveys, Interviews and Observation For each case the survey of students was conducted immediately on completion of the performance examination. Broadly, it sought students’ opinions on the examination itself (questions q1a to q2k) and the portfolio (p2c to p2k), the portfolio of work completed during the term (questions p1a to p2b), use of computers and other digital devices (questions 5 to 10e), attitudes to using computers (questions 11a to
16 6/
23/2
009
16 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
12f) and facility with computer applications (questions 13a to 13k). The questionnaire consisted of 58 closed response items and two open-response items.
For each case study a summary of responses to the student questionnaires was collated into a table using the column headings shown below.
(N=?) Abbreviated Questions Min Max Pop
Mean Mean SD Data from observations were collated into a table, illustrated below, showing notes on activities. A photograph was taken of the classroom laboratory being used.
Visit Observations
15/08/08, 9:05-10:09 Product Development 21 studs, Comp lab
Started over a week ago .... Teacher started by going through …. etc. etc.
Marking of Student Work The two external assessors marked all of work submitted for each student comprising the design document for the product, the product itself, two further digital artefacts, a theory section and a practical section of a three-hour examination. The marks for these were totalled. No allowance was made for missing work. Teachers were requested to mark the portfolio and examination to include within their own school-based assessment.
For each case study the results of marking were compiled into a table, using the headings illustrated below, showing the results for each individual student. The rank for the average of the external assessors marks (Ass Ave) was for all 115 students whereas the teacher’s semester mark (Tch Sem) rank was just for the class. The rank is the position out of the whole sample of 115 students spread over six schools. For example an average assessor rank of 86.5 means that this students total ranked 86th equal out of 115. The teacher’s semester rank is the position of the mark awarded by the teacher compared to the marks awarded by all other teachers in the sample.
Assessors marking (Total) Assessors Teacher Rank
St ID M1 (%) M2 (%) Ave (%) Pfolio (70) Exam (30) Sem (%) Ave* Tch Sem
ca101 37 30 33.5 23.0 10.5 70 63.0 3.0 *Average of the two assessors, M1 and M2.
A summary of means and standard deviations is provided in Figure 4.1. Detailed reports on each case study were included with the main report. In this report only the combined results from the case studies is provided.
Assessors marking (Total) Assessors Case
M1 (%) M2 (%) Ave (%) Pfolio (70) Exam (30) Teacher Sem (%)
Rank Ave*
CA 33.0 (12.6) 31.4 (14.9) 32.2 (13.4) 21.7 (11.6) 12.2 (3.5) 42.4 (19.4) 67.2 (26.4)LA 37.2 (10.1) 43.9 (13.9) 40.6 (10.6) 27.0 (8.8) 15.1 (4.7) 69.7 (5.8) 48.5 (21.5)MA 49.0 (12.8) 45.8 (16.2) 47.4 (13.6) 32.3 (10.5) 15.1 (4.0) 54.1 (12.9) 37.7 (24.7)RA 30.8 (13.8) 31.6 (16.6) 31.2 (14.8) 18.7 (11.8) 13.8 (3.6) 64.0 (15.9) 69.0 (30.1)WA 12.7 (3.1) 12.4 (3.2) N/A N/A 12.6 (2.9) N/A 76.2 (20.5) XA 38.9 (12.8) 40.7 (14.8) 39.8 (13.5) 24.3 (11.0) 16.4 (3.6) 54.3 (13.4) 51.1 (27.1)ZA 54.2 (16.1) 53.0 (16.0) 53.6 (15.2) 35.8 (11.5) 17.8 (4.4) 67.7 (10.1) 27.5 (24.6)
*Ranking of average of external assessors is for all 115 students involved in the study. Figure 4.1: Tabulation of results from marking as means (standard deviations) for each case.
17
4.3 Online Repository The assessment outputs for all students were uploaded to the online repository. The structure of the repository for AIT was all files stored on a University server in a unique folder for each student. The folder contained a folder for each of the artefacts, one for the portfolio product and one for the exam. The portfolio process document was a PDF file and was placed within the main folder on its own. Within each folder there was an index.htm page that was used by the marking tools to display the contents of the folder, this page contained links to the other files. This file and the structure of folders was set up by a research assistant at the University. The first artefact folder contained a PDF file of the student’s descriptions of the two artefacts. The exam folder contained all the files copied of the exam USB flash drives and PDF versions of their brochure, reflections, plans, Section A responses and one file with all the Section B PDF files combined. The structure of a student’s folder is shown in the graphic on the right hand side.
4.4 Analytical Marking and Analysis The students’ assessment outputs were marked analytically by two external assessors using an online marking tool. The tool displayed the students’ work on-screen and allowed the judgements of the assessors to be digitally recorded. The tool incorporated a marking key (rubrics) based on the assessment criteria developed for the task. Both assessors were experienced computing teachers.
4.4.1 The Analytical Marking Tool The analytical marking tool was developed from syllabus of the AIT Unit 2B Course of Study as related to the assessment task (AIT Assessment Task criteria, Appendix A) and created using FileMaker Pro. The marking tool was design based on a 20” screen with the assessment criteria displayed on the left side and the student work on the right. The development environment of FileMaker Pro allowed the tool to be deployed on the Internet with minor modifications, and to be password protected. The marking tool consisted of five marking screens, each with different criteria and student work. Marks were recorded by clicking on buttons, as was movement between marking windows, and comments could be recorded when required. The tool was designed to do all the clerical functions, such as totalling the marks.
4.4.2 Analysis of Results from the Analytical Marking All the students’ work was marked by the two external assessors using the analytical marking tool. At the same time the teacher marked the students’ work using his/her own method. The two external assessors marked the student work on the criteria developed for the assessment task using analytical ‘marks’. Summaries of some of the descriptive statistics on the results of this marking are provided in Figures 4.2 and 4.3. Correlations between the markers are also shown in Figure 4.4.
The two external markers scored the same student work entirely independently and solely by reference to the student work and the marking rubrics. The aim of the study was to have teachers mark the same content using their own assessment methods and include this in the semester mark awarded. However, there was widespread misunderstanding of this intention and the content actually marked by the teachers, and the proportion included in the final semester mark, varied according to school and was not always obvious to the researchers. Of the 115 students, only 58 final semester and 26 assessment task marks were received from teachers. Further, student portfolios were submitted in various stages of completeness. With these qualifications in mind, the following conclusions may be drawn.
Both external assessors gave a very similar range of scores with a resulting similar mean and standard deviation. There was no significant difference between their means on the overall scoring of the
18 6/
23/2
009
18 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
assessment task (Portfolio and Exam combined). The range of scores, the means and standard deviations varied considerably between individual classes on separate components of the assessment task and for the task as a whole (Figure 4.3, All Tasks). For example, Class ZA had a mean of 53.6 while Class RA had a mean of 31.2 (Class WA is overlooked as portfolios were not submitted).
There was a strong and significant correlation between the assessment of the two markers for the overall assessment task with correlation coefficients of 0.89 (p<0.01) for the scores and 0.91 (p<0.01) for the ranking of students (Figure 4.4). Correlation between the average mark of the external assessors and the semester mark awarded by the teacher was moderately strong and significant (r=0.62, p<0.01). There was no significant correlation between the average mark of the external assessors and the examination mark awarded by the teachers. Correlations of ranking, (the order in which the marks awarded placed the student in the population) were similarly correlated. This suggests that, in spite of the differences in content assessed, student competence was recognised consistently by both assessors and teachers.
N Minimum Maximum Mean Std. Deviation M1_Tot 115 7.0 83.0 36.7 17.0 M2_Tot 115 4.0 80.0 37.0 18.3 Average 115 6.5 74.5 36.9 17.2 Tch% 26 28.1 91.8 57.9 17.1 Sem% 58 9.0 84.0 56.5 18.3
Figure 4.2: Descriptive statistics on marking for all students and individual schools.
All Tasks Portfolio Exam Exam (Sect B) Product Class N Range Mean SD Range Mean SD Range Mean SD Range Mean SD Range Mean SD CA 20 14-67 32.2 13.7 0-53 20.6 12.3 0-18 11.6 4.3 0-12 7.8 3.4 0-15 7.3 4.6 LA 10 18-56 40.6 11.2 12-40 27.0 8.8 0-23 13.6 6.5 0-16 8.8 4.6 3-14 8.6 3.2 MA 12 25-69 47.4 13.6 13-47 32.3 10.5 10-23 15.1 4.0 6-15 11.3 2.4 5-17 10.4 3.6 RA 14 13-63 31.2 14.8 0-43 17.4 12.4 9-21 13.8 3.6 6-15 9.9 2.5 0-14 7.4 4.7 WA 14 6-17 12.6 2.9 0-0 0 0 6-17 12.6 2.9 3-13 9.0 2.5 0-0 0 0 XA 29 17-66 39.8 13.5 0-47 23.4 11.7 8-24 16.4 3.7 5-16 11.5 2.7 0-17 6.6 6.5 ZA 16 26-75 53.6 15.1 14-53 35.8 11.5 9-25 17.8 4.4 7-17 11.9 2.4 0-17 11.1 4.5 All 115 6-75 36.9 17.2 0-53 22.3 14.6 0-25 14.6 4.6 0-17 10.1 3.2 0-17 7.2 5.6
Figure 4.3: Descriptive statistics on marking for all students and individual schools based on average of the two external assessor’s marks.
M1_Tot M2_Tot Average Tch% Sem% M1_Tot 1 .892** .970** .247 .601** M2_Tot .892** 1 .975** .362 .580** Average .970** .975** 1 .319 .619** Tch% .247 .362 .319 1 .933** Sem% .601** .580** .619** .933** 1 Rank of M1_Tot M2_Tot Average Sem Tch M1_Tot 1 .905** .971** .578** .239 M2_Tot .905** 1 .978** .565** .347 Average .971** .978** 1 .606** .305 Sem .578** .565** .606** 1 .936** Rank of Tch .239 .347 .305 .936** 1 **. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
Figure 4.4: Correlations for scores and rankings from marking for all students.
The assessment task had two major components, the portfolio and the exam. The marking of these two major components was analysed separately. Summary statistics and correlations are shown in Figures 4.5 and 4.6. In each the average of the two markers is used (i.e. Avg_Exam, Avg_Port, Avg_Tot). The mean score for the exam was around 50% but for the portfolio only around 37%. This is because many students did not submit all components of the portfolio whereas almost all students completed all of the exam. Of the exams marked by the teachers the mean was quite similar at 52.5%.
19
There was a moderate but significant correlation between marks awarded by the external assessors for the examination and marks awarded for the portfolio (Figure 4.6). This was in spite of the fact that many submissions of student portfolio work were incomplete. Correlations, between average marks awarded by external assessors and marks awarded by the teacher, were very weak. As mentioned above, the exact content assessed by the teacher and the method of assessment were not made available to the researcher. A similar analysis on rankings rather than scores delivered similar results.
N Possible Minimum Maximum Mean Std. Dev Mean (%) Avg_Exam 113 30 6.0 24.5 14.8 4.2 49.3 Avg_Port 98 70 4.0 52.5 26.2 12.2 37.4 Avg_Tot 96 100 13.5 74.5 41.2 15.2 41.2 T_Exam 74 100 13.0 90.0 52.5 20.0 52.5 T_Port 51 100 17.1 94.3 59.5 17.5 59.5 T_Tot 26 100 18.9 93.0 50.0 20.6 50.0
Figure 4.5: Descriptive statistics for scores from marking for all students.
Avg_Exam Avg_Port Avg_Tot T_Exam T_Port T_Tot Avg_Exam 1.000 .578** .750** .157 .402** .229 Avg_Port .578** 1.000 .973** -.140 .360** .048 Avg_Tot .750** .973** 1.000 -.072 .390** .098 T_Exam .157 -.140 -.072 1.000 .335 .617** T_Port .402** .360** .390** .335 1.000 .912** T_Tot .229 .048 .098 .617** .912** 1.000 **. Correlation is significant at the 0.01 level (2-tailed). *. Correlation is significant at the 0.05 level (2-tailed).
Figure 4.6: Correlations for scores from marking for all students.
An Analysis of Variance was completed to investigate variance across the classes of students. The results are shown in Figure 4.7. As expected the marks awarded by the external markers varied significantly by class for the Portfolio and Exam (and components of each) indicating differences in the capability of students between classes. The differences were consistent with the same classes having higher means for both the Portfolio and Exam. The scatter diagrams in Figure 4.8 show correlations between Portfolio and Exam marks for two classes with fairly similar means. The ZA class marks were more highly correlated (r=0.55, p<0.05) than for XA (r=0.31, p>0.05), probably explained by the more stringent implementation of the portfolio requirements for the former class.
SSquares df Mean Square F Sig. (Combined) 11725.810 6 1954.302 16.700 .000 Within Groups 12638.590 108 117.024
Port * Class Between Groups
Total 24364.400 114 (Combined) 523.093 6 87.182 5.095 .000 Within Groups 1848.198 108 17.113
Exam * Class Between Groups
Total 2371.291 114 (Combined) 274.508 6 45.751 5.340 .000 Within Groups 925.266 108 8.567
Average_5 * Class
Between Groups
Total 1199.774 114 (Combined) 1121.286 6 186.881 8.398 .000 Within Groups 2403.245 108 22.252
Average1 * Class Between Groups
Total 3524.530 114
Figure 4.7: ANOVA for external marks for Portfolio (Port), Exam, only the Performance Tasks component of the exam (Average_5) and the Portfolio Product (Average_1) against Class.
20 6/
23/2
009
20 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Figure 4.8: Scatter diagrams for two classes with similar means but variation in correlation. For the ZA
class the portfolio was more accurately implemented.
4.5 Comparative Pairs Marking and Analysis Comparative pairs marking involved assessment of the practical examination (Section B) only, for a reduced population of 60 students. These students were chosen because their practical work samples were equivalent in the degree of completeness and had no missing sections. In particular they all had an audio response file for the exam. Five assessors each completed the same pre-determined set of comparisons between students using a digital marking tool. All five were computing teacher experts, two being involved in the standards-referenced marking. One holistic and three specific assessment criteria were developed for the comparative pairs marking from the criteria previously developed for the task. These criteria were:
Holistic Criteria: Brochure is effective for target customers through developed planning to incorporate all the required features and information, appropriate use of aesthetic effects on a theme, consistent and balanced layout, and professional look. [Evidenced across all components including evaluation]
Specific Criteria 1. Design Process
Product originates from planned design showing development of ideas and justification in reflection. [Focus on planning sheets at beginning of PDF, reflection at the end and the MP3 sound file]
Specific Criteria 2. Technical Proficiency
Demonstrable capability and facility with the range of required software (spreadsheet, logo, brochure). [Focus on features of graphs, logo and layout in brochure]
Specific Criteria 3. Design Principles
Creative application of appropriate design principles and elements such as alignment, balance, contrast, emphasis, harmony, proportion, proximity, repetition, unity, and white space. [Focus on brochure and logo].
4.5.1 The Comparative-Pairs Marking Tool The comparative-pairs marking tool was designed to display the two students work side-by-side, with the recording of the marker’s choices located between them. The tool was developed using FileMaker Pro and deployed on the Internet with minor modifications. The marker was required to make four choices, one holistic and three criteria choices, by clicking on large green arrows pointing to the student they wanted to select. A short description of what to consider for each of the choices was given. A field was added for each student so markers could record their comments on the students’ work and would not have to scroll through pages or view complete videos of the student work each time. When completed, they clicked on a button to go to the next pair.
4.5.2 Analysis of Results from the Comparative-Pairs Marking The RUMM software provided a summary of results that included the exemplar ID. The exemplars were ranked from best (most number of times preferred) to worst (least number of times preferred). The ‘Preferred’ is the actual number of times the exemplar was preferred in all the ‘Involved’ comparisons.
21
The ‘Estimate’ is the exemplar location in logits (logarithmic units of measurement) and is the determinant of the rank order of the exemplars. The ‘Std Err’ is the standard error of measurement. The ‘Outfit’ is an index of whether the pattern was more or less Guttmann like. It is expected to have a value of about 1.00.
A Separation Index was calculated as an indicator as to whether or not the exemplars were sufficiently diverse in quality to assure a broad enough range for the purposes of comparison. It is given as a number from 0 to 1. Values closer to 1.00 are more desirable. If the value is close to 0.00 (up to about 0.3 or 0.4) the range is too narrow. If it is above about 0.7 the separation is reasonable and if it is above 0.8, the separation is good. The Separation Index for the Holistic criterion was 0.958 indicating a very good spread of quality in the exemplars (Figure 4.9).
Intra-rater reliability analysis was done in order to assess individual judge’s consistency with the judgments of the other judges in the group. The “Outfit” statistic, in this instance, should be between 0.5 and 1.5. The group reliability is defined as the average of the individual rater reliability indices. For Holistic comparisons, the group reliability was 1.01 (Figure 4.9).
Intra-rater Reliability Coefficient Type of Judgement
Separation Index
Rater 1 Rater 2 Rater 3 Rater 4 Rater 5 Overall
Criteria 1 0.940 1.057 0.800 1.259 1.145 1.007 1.050
Criteria 2 0.946 0.884 0.685 1.022 1.426 1.092 1.020
Criteria 3 0.951 1.342 0.831 0.903 2.517 0.972 1.310
Holistic 0.958 0.910 0.602 1.1016 1.016 1.367 1.010
Figure 4.9: Separation indices and intra-rater reliability coefficients for AIT in the first year.
Figure 4.10 shows the correlations between the pairs-comparison marking of the examination and the analytical marking of the examination by the two markers: Assessor A and Assessor B. AssAv Exam is the average of the two assessors marks for the examination which consisted of both theory and practical. A similar analysis using rankings rather than scores gave similar results. Pairs Holistic is the overall mark awarded to the student’s examination practical work based on the comparisons it underwent by the five judges in the pairs marking. Pairs C1 is the mark awarded for criterion 1 of the student’s practical. Tch Exam is the mark awarded by the teacher for the examination theory and practical. Tch Sem% is the teachers mark for the student for the whole semester’s work and Average practical is the average of the assessors marks for the practical component only.
There was a strong and significant correlation (0.733 p<0.01) between the mark generated by comparative pairs marking and the mark determined by analytical marking. As might be expected, the criteria (pairs marking) are also strongly correlated with the average practical mark (analytical marking). There was no significant correlation between the teacher’s examination mark and the pairs marking, with the exception of criterion 2 (0.461 p<0.05), and the absence of a relationship once again highlights the fact that the exact content and method of marking by the teacher was not known to the researcher. However, the teacher’s semester mark appears to be weakly correlated with all criteria in the pairs marking.
Pairs_Holistic Pairs_C1 Pairs_C2 Pairs_C3 T_Exam Sem% Average_5 Pairs_Holistic 1 .838** .920** .972** .334 .472* .733** Pairs_C1 .838** 1 .739** .848** .177 .419* .622** Pairs_C2 .920** .739** 1 .904** .461* .432* .726** Pairs_C3 .972** .848** .904** 1 .330 .461* .696** T_Exam .334 .177 .461* .330 1 .096 .165 Sem% .472* .419* .432* .461* .096 1 .359** Average_5 .733** .622** .726** .696** .165 .359** 1
** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.10: Correlations for scores and rankings from marking for all students.
A similar analysis was performed using the rankings produced by each of the marking methods. The rank of the average practical score (analytic marking) was strongly and significantly correlated with all criteria of the pairs marking. The strongest correlation is between the average score and the holistic criterion
22 6/
23/2
009
22 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
(r=0.71, p<0.01). There was again no significant correlation between the rank of teacher’s examination mark and the pairs marking, with the exception of criterion two r=(0.43, p<0.05). The teacher’s semester mark appears to be weakly correlated with all criteria in the pairs marking with the holistic criterion being strongest (r=0.53, p<0.01).
4.6 Conclusions About Marking Process This section makes conclusion about the marking processors largely using data collected from assessors.
Time Taken for Marking
The amount of time taken by the expert assessors in the analytical marking varied from an average of about 5 minutes to 25 minutes per student. The shorter times corresponded to students whose submissions of the five components were incomplete. The longer times were associated with student work which comprised large files (for example some animations were several MB) and these took time to download to the assessor’s computer prior to marking particularly where this marking was done outside Australia. Apart from delays in downloading, longer time was spent on higher quality work, particularly where evidence of performance had to be gathered from several parts of the submission.
The comparative pairs marking focussed only on one part of the students’ work, the practical examination, so the time per student was correspondingly reduced. The time required to make a comparison was initially several minutes, mainly where the samples were of similar quality. However, as familiarity with the criteria increased the time per pair became less. Because the comparisons were pre-determined and not dynamically generated, several were very one sided and for these the marking time was seconds rather than minutes. For the comparative pairs marking assessors took about 3 minutes per comparison.
Comments of Assessors
The assessors were asked for feedback on the suitability of the assessment tasks, the marking process of and the quality of student work.
The examination was a mixture of design, practical and reflection, and was acknowledged to be faithful to the course outline for stage 2 AIT which states that, “the focus is on information and communication technologies in business” and further that, “students design information solutions for problems encountered in these contexts and understand the social issues inherent in work practices”. The course makes direct reference to “commonly used applications software in organisations for productivity, planning and communication e.g. word processing, publishing, presentation and financial data management” and to “input, validation and manipulation of data (e.g. text-, numerical- and image-based), integration and presentation of these data”. With this in mind, the assessment task, marketing a hotel by designing and creating a logo and a brochure including photographic, numerical, textual and graphical information, was certainly suitable.
The portfolio tasks were also faithful to the course outline. Although the results were generally of a low standard this was not due to unsuitability. Students who were well versed in the subject were able to perform well whereas many failed to understand the intention of the tasks, particularly the written sections, or submitted work that was incomplete. However, the language of the instructions and questions could have been simplified and examples given for clarification.
The marking process was simplified by the fact that all submissions were in digital form, allowing anytime anywhere access, and by the use of the online marking tool. Some drawbacks of the marking system were the inevitable delays in opening large files and scrolling between the mark key and work sample. Changing a mark already entered was a little clumsy requiring a post back of the marking form. The running score of the mark also didn’t update until the marking form was submitted and this was confusing at first.
The quality of student work was wide ranging and this supported the suitability of the tasks as discriminators of student ability. There was common misunderstanding, in the examination, of what was meant by a logo, and perhaps some examples might have aided clarification. Students were engaged with the activities, particularly the examination practical section and frequently asked to be allowed to go back to complete or edit their work after completion of the theory section.
23
4.7 Conclusions About Portfolio versus Exam This section considers in more depth a comparison between the results of marking for the Portfolio and for the Exam. Only the practical component of the Exam (Component 5) was marked with both methods of marking so this is considered first. Figure 4.11 shows correlations for this component, a similar analysis of rankings rather than marks gave similar results. There was a relatively low correlation (r=0.43, p<0.01) between to the two external markers although their average was relatively highly correlated to the results of the pairs marking. An analysis of correlations between the marking of Component 5 compared with the other components resulted in only moderate to low correlations.
Average_5 M1_Comp5 M2_Comp5 Pairs_Holistic T_Exam Sem% Average_5 Pearson Correlation 1.000 .860** .833** .733** .166 .393* M1_Comp5 Pearson Correlation .860** 1.000 .434** .687** .107 .421*
Pearson Correlation .833** .434** 1.000 .549** .183 .196 M2_Comp5 N 60 60 60 60 29 27 Pearson Correlation .733** .687** .549** 1.000 .334 .472* Pairs_Holistic N 60 60 60 60 29 27 Pearson Correlation .166 .107 .183 .334 1.000 .819* T_Exam N 29 29 29 29 29 8 Pearson Correlation .393* .421* .196 .472* .819* 1.000 Sem% N 27 27 27 27 8 27
** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.11: Correlations for scores from marking for all students.
Secondly the portfolio scores are considered. Only two schools (MA and ZA) implemented all aspects of the portfolio (Components 1 to 3) largely in line with the stated requirements. Figure 4.12 shows graphs of the spread and correlation of marks for these two classes combined. Correlation coefficients are provided in Figure 4.13. Similar analyses were completed for each component of the assessment and for rankings, with similar results. Correlations between the two markers on the Portfolio (r=0.79, p<0.01) was relatively high when compared with that for the exam.
Figure 4.12: Spread and correlation of marks for students in classes MA and ZA (combined).
M1_Port M2_Port Port Exam T_Port Sem% M1_Port 1.000 .786** .938** .718** .696** .592** M2_Port .786** 1.000 .952** .682** .636** .517** Port .938** .952** 1.000 .739** .702** .584** Exam .718** .682** .739** 1.000 .682** .517** T_Port .696** .636** .702** .682** 1.000 .904** Sem% .592** .517** .584** .517** .904** 1.000 ** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.13: Correlations for portfolio marks for students in classes MA and ZA (N=28).
Compare this with when all schools are included but removing all students who either did not submit a portfolio or did not sit the exam. Results of analysis are shown in Figures 4.14 and 4.15. Results are very similar to those for just the two schools, almost all of whose students would have been included. The
24 6/
23/2
009
24 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
correlations between the two external markers are slightly higher (Portfolio - r=0.85; Exam – r=0.54) and both significant at the 0.01 level. However, this measure of reliability is still much more acceptable for the Portfolio than for the Exam.
Figure 4.14: Scatter plot of Portfolio and Exam marks for students completing both.
M1_Port M2_Port Port T_Port Sem% Pearson Correlation 1.000 .846** .958** .299* .563** M1_Port N 96 96 96 50 55 Pearson Correlation .846** 1.000 .964** .374** .592** M2_Port N 96 96 96 50 55 Pearson Correlation .958** .964** 1.000 .355* .609** Port N 96 96 96 50 55 Pearson Correlation .299* .374** .355* 1.000 .906** T_Port N 50 50 50 50 37 Pearson Correlation .563** .592** .609** .906** 1.000 Sem% N 55 55 55 37 55
M1_Comp5 M2_Comp5 Average_5 Pairs_Holistic T_Exam Sem%
Pearson Correlation 1.000 .537** .896** .630** .038 .279* M1_Comp5 N 96 96 96 47 71 55 Pearson Correlation .537** 1.000 .856** .374** .074 .350** M2_Comp5 N 96 96 96 47 71 55 Pearson Correlation .896** .856** 1.000 .640** .060 .361** Average_5 N 96 96 96 47 71 55 Pearson Correlation .630** .374** .640** 1.000 .330 .472* Pairs_Holistic N 47 47 47 47 28 27 Pearson Correlation .038 .074 .060 .330 1.000 -.012 T_Exam N 71 71 71 28 71 30 Pearson Correlation .279* .350** .361** .472* -.012 1.000 Sem% N 55 55 55 27 30 55
** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.15: Correlations for Portfolio and Exam marks for students completing both.
Consider the operation of the marking criteria for Exam Component 5, the performance tasks exam. Using t tests to test for differences in means between the markers found a significant difference for the overall score but only a significant difference (p < 0.05) on two of the separate criteria, C5b and C4a.
The C5b criterion concerns “Applies appropriate file formats, compression and encryption techniques, conversion, size and storage requirements.” Markers to allocate 0, 1 or 2. Generally no 2s were allocated, just 1s and 0s. The C4a criteria was associated with the production of the brochure, “Used specific styles, forms and techniques to create brochure represent the particular effect of the design on the audience and achieve defined standards of quality.” A summary of the marks given by the two markers on this criterion are given in Figure 4.16. Once again the top mark of 4 was sparingly used by both markers.
25
M1 M2 Mark Frequency Percent Frequency Percent
0 4 3.6 4 3.6 1 25 22.3 35 31.2 2 42 37.5 43 38.4 3 32 28.6 29 25.9 4 9 8.0 1 .9
Figure 4.16: Summary of marks given on criteria C4a.
4.8 Applying a Rasch Model to the Exam Marks A Rasch model was applied to the Exam marks using the responses of both markers to generate a combined score for each student. This resulted in a mean person location of 0.230, fit residual of -0.349 and standard deviation of 1.219. The Separation Index (SI) was 0.85213. There were few extreme outliers with a frequency distribution relatively well spread. Unless otherwise stated the thresholds on all items worked adequately.
Figure 4.17: Some output from RUMM2020 from the Rasch analysis of the marks allocated by the two
markers on the exam (component 5 only).
26 6/
23/2
009
26 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
The reversed threshold on c2a was between a response of 3 and 4. As a result the analysis was repeated with these responses (3 and 4) for c2a and d2a both scored as 3. This increased the SI marginally to 0.85298 and removed the reversed threshold. The correlation coefficients between the location scores and raw marks were high for both markers and for the mean of their marks (Figure 4.18). The distribution of all scores, showing standard error bars, is provided in Figure 4.19.
Location M1 M2 Mean Location 1.000 .895** .874** .996** M1 .895** 1.000 .576** .906** M2 .874** .576** 1.000 .869** Mean .996** .906** .869** 1.000 ** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.18: Correlations between location values and raw marks from the two markers for the exam (N=110).
Figure 4.19: Distribution of location scores for the exam, including standard error bars.
4.9 Applying a Rasch Model to the Portfolio Marks A Rasch model was applied to the Portfolio analytical marks using the responses of both markers to generate a combined score for each student for each of the three components of the portfolio (Product, Process Document and Extra Artefacts analysed separately). For each component if a student did not submit work they were removed and thus not all the 115 students were included in the analysis. The analysis gave a reliable set of scores for all three components (SI=0.96, 0.96 and 0.94 respectively). There were a few extreme outliers particularly for the first component, the product. These tended to be students scoring 0 on all or almost all of the criteria. The frequency distributions tended to be well spread, with high standard deviations and not very ‘normal’ in structure. Unless otherwise stated the thresholds on all items worked adequately.
27
Component 1 – Product
Figure 4.20: Some output from RUMM2020 from the Rasch analysis of the marks allocated by the two
markers on the Product component of the Portfolio.
Location M1 M2 Mean Location 1.000 .865** .884** .957** M1 .865** 1.000 .670** .916** M2 .884** .670** 1.000 .911** Mean .957** .916** .911** 1.000 ** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.21: Correlations between the Location scores and raw marks from the two markers for the Product component of the Portfolio (N=83).
No modifications were required.
28 6/
23/2
009
28 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Component 2 – Process Document
Figure 4.22: Some output from RUMM2020 from the Rasch analysis of the marks allocated by the two
markers on the Process Document component of the Portfolio.
Location M1 M2 Mean Location 1.000 .932** .940** .982** M1 .932** 1.000 .819** .946** M2 .940** .819** 1.000 .961** Mean .982** .946** .961** 1.000 ** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.23: Correlations between the Location scores and raw marks from the two markers for the Process Document component of the Portfolio (N=81).
No modifications were required.
29
Component 3 – Extra Artefacts
Figure 4.24: Some output from RUMM2020 from the Rasch analysis of the marks allocated by the two
markers on the Extra Artefacts component of the Portfolio.
Location M1 M2 Mean Location 1.000 .915** .886** .989** M1 .915** 1.000 .659** .912** M2 .886** .659** 1.000 .910** Mean .989** .912** .910** 1.000 ** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.25: Correlations between the Location scores and raw marks from the two markers for the Extra Artefacts component of the Portfolio (N=78).
No modifications were required although the thresholds for three of the criteria did not work very well (Criteria 1 – c1 and d1 – the response 2 was rarely used, Criteria 3 – c3 and d3 - the responses of 1, 2 and 3 and Criteria 4 the response 2 for one marker – d4).
30 6/
23/2
009
30 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Conclusions In general the analytical marking of the portfolio components gave reliable scores when taken separately. However, some improvements in the marking criteria could be made. As can be seen from the correlation results in Figure 4.26 the three components of the portfolio were not highly correlated with each other with the best being between components 1 and 2 (r=0.54).
Location_P1 Location_P2 Location_P3 P1_Mean P2_Mean P3_Mean Location_P1 1.000 .537** .200 .962** .564** .211 Location_P2 .537** 1.000 .217 .585** .984** .234 Location_P3 .200 .217 1.000 .206 .212 .990** P1_Mean .962** .585** .206 1.000 .607** .216 P2_Mean .564** .984** .212 .607** 1.000 .232 P3_Mean .211 .234 .990** .216 .232 1.000 ** Correlation is significant at the 0.01 level (2-tailed) * Correlation is significant at the 0.05 level (2-tailed).
Figure 4.26: Correlations between the Location scores and mean on the raw marks from the two markers for the three components of the portfolio (N=53).
4.10 Conclusions From Student and Teacher Data This section reports on the data collected from students and teachers for all the cases combined.
Survey of Students 110 students completed a questionnaire consisting of 57 closed response items and two open-response items. The minimum, maximum, mean and standard deviations were calculated for each closed response item using SPSS. Responses to the open-response items were tabulated to assist in drawing out themes.
The best things … The worst things … On the positive side, students overwhelmingly commented on the ease of working on the computer compared to working on paper. The exact aspect of easiness cited varied, but frequent mention was made of correcting errors, speed of writing, amount of writing, speed of action and physical comfort.
On the negative side, by far the most prevalent comments referred to the fear, rational or otherwise, that it could all go horribly wrong. These fears were often mentioned together with a critical appraisal of the hardware provided, and clearly for many schools, the software employed by students and the manner in which it was deployed strained system resources.
A number of scales were derived from combining items from the questionnaire. Results are shown in the table and graphs below. Table 4.1 Descriptions and descriptive statistics for the scales based on items from the student questionnaire.
N Min Max Mean Std. D Description eAssess 110 1.4 4.0 3.2 0.4 Ease of completion of the exam. Score between 1 and 4 eAssessP 108 1.2 4.0 3.2 0.4 Ease of completion of the portfolio. Score between 1 and 4. Apply 105 1.4 3.0 2.4 0.4 Application of computer use. Score between 1 and 3. Attitude 105 1.4 3.0 2.6 0.3 Attitude towards using computers. Score between 1 and 3. Confidence 105 1.0 3.0 2.7 0.4 Confidence in using computers. Score between 1 and 3. Skills 105 1. 4.0 3.3 0.5 Self assessment of ICT skills. Score between 1 and 4. SCUse 105 0.0 334 95.9 62.1 Estimate of time in mins/day using computers at school.
31
Figure 4.27: Graphs for the distribution of scores for the scales on the student questionnaire.
Though many students indicated that they had little experience in doing examinations on computers (42% indicated no experience) about half felt they would need little time to get used to the process. Most of the rest felt they would need some time to do so. Almost all students indicated that doing the examination on the computer was quick, easy and preferable to the traditional pen and paper examination. In general it could be said they were comfortable and positive about both the Portfolio and Exam.
Nearly all the students had home access to the technologies listed in the questionnaire with mobile phone and MP3 player ownership both more than 90%. Two thirds of the students owned their own laptop computer and 95% had a broadband Internet connection. At school they used computers for an average of 95 minutes per day. All students felt confident with computers and liked using them. Of the types of computer software listed students felt least confident about web authoring and databases. Overall they
32 6/
23/2
009
32 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
indicated a high self-assessment of their computer skills with a mean of 3.3 on a four-point scale (in comparison this mean was 2.9 for PES and Italian, and 3.1 for Engineering).
Comments of Teachers
In general the teachers were very positive towards assessment by portfolio and by practical examination and felt that these complemented their own aims, principles and methods of instruction. As one teacher observed “If the external marking of a portfolio does away with the moderation process, I’m all for it”. All teachers said they would like to see a greater emphasis on the practical aspects of the course. One teacher commented, “If we are asking our students to complete the majority of their assessments using these tools throughout the year, then surely we should in the final exam”. Some were cautious about the potential of the examination because of the possibility of technical problems. Quite a few of these were encountered in spite of advanced planning and testing. One teacher suggested running the whole examination from a bootable mass storage (USB) memory device containing not only the examination files but also the application software and commented, “In this way there is more control over the whole environment”.
33
5 - Conclusions from all AIT Findings This section brings together the main conclusions that may be drawn from an analysis of all the data for the AIT case studies.
5.1 Conclusions About the Assessment Task All seven classes attempted the five components of the assessment tasks to varying degrees of completion. The three-hour examination, consisting of a one-hour theory paper and a two hour practical test, was attempted by all students.
The theory section consisted of several reflective questions. Students’ responses suggest that either the intention of the questions was unclear or that they did not know how to answer. One student commented “The wording in the second part…a bit confusing. I had to guess at what it meant”, and another, “It took me a while to work out what was meant by it”. There was widespread confusion over the stages of the technology process and the distinction between these with many responses repeated. As one student noted, “it just seemed like you asked the same questions four times…I got four words out of a thesaurus and copied and pasted those in three or four times”.
From observation, marking and comments made by students, the intention of the practical test was clearly understood by all participants namely to design and create a logo for a business and incorporate that into a tri-fold advertising brochure. However, there was widespread variation in students’ interpretation of what constituted a logo. More than half the students simply added a caption to one of the photographs supplied. Less than 20 students designed a logo and used drawing tools to create it and even here, many used themes or cues from the photographs supplied. It was as if the photographs closed off creative avenues and constricted students’ imagination.
Only five students did not use a software template for the tri-fold brochure and of these four used a word-processed document with three columns. In general, students selected appropriate photographs for the brochure showing that they understood the intention of the task was to market a resort as luxurious but having a low environmental impact. Student audio reflections, where available, supported this understanding.
The portfolio, consisting of a product, process document and two other digital artefacts, was intended to form part of the assessment structure for the course. However, there was misunderstanding among the teachers of this and five of the seven teachers ran the portfolio in whole or in part as an additional task, not counting towards a students’ final semester mark. It is therefore not surprising that students did not give the portfolio their best efforts and many portfolio submissions were incomplete.
5.2 Conclusions About Methods of Marking The collation of student work was in digital form had several obvious advantages for assessment such as ease of storage, backup, transmission, access and sharing. The web based database, which held the student work was responsive and easy to use provided that adequate Internet bandwidth was available to the marker. However, there were difficulties accessing some files if the marker was within the Curriculum Council network and also the marking tools operated slowly and sometimes crashed or logged out. Marking was possible, and indeed took place, from countries outside Australia though opening of large files presented delays. With regard to the analytic marking, the ability to view both the work sample with the marking rubric alongside it was convenient and ensured focus was maintained. Switching rapidly between different aspects of student work was easy. The database recorded and summed the scores and this was obviously quick and accurate. The analysis of the marking, with the strong correlations between assessors, indicates that the method was reliable. After a little practice with the system and a brief familiarisation with the criteria, the comparative pairs marking was also quick and convenient. For many of the comparisons, it was immediately obvious which piece of work was superior. Only occasionally were the two samples so close that it took time to arrive at a decision. The reliability of the results of marking was high for both methods with correlations between markers using analytical marking over 0.8 and the Separation Index for the comparative pairs marking above 0.9. However, on more in depth analysis it was found that the correlation between markers for the analytical marking was more acceptable for the marking of the Portfolios (r=0.9) than for the Exam (r=0.5).
34 6/
23/2
009
34 o
f 52
CSa
LT D
igita
l Ass
essm
ent R
epor
t
Table 8.1 Correlations between assessors and between assessors and teachers for the AIT case studies.
Correlations Between Markers or Methods of Marking
Analytic Assessors Average and Teacher Analytic and ‘Pairs’
Case N Marks Rankings Marks Rankings Marks Rankings
CA 20 0.88** 0.86** 0.64** 0.65** n/a n/a LA 10 0.55 0.51 0.36 0.37 0.99 1.00 MA 12 0.75** 0.80** 0.87** 0.83** 0.54 0.51 RA 14 0.90** 0.88** 0.46 0.46 0.89* 0.86 WA 14 0.66* 0.57* N/A N/A 0.77** 0.84** XA 29 0.91** 0.92** 0.66** 0.73** 0.71** 0.74** ZA 16 0.78** 0.86** 0.36 0.28 0.69** 0.64**
All 115 0.89** 0.91** 0.32 0.30 0.73** 0.71**
** p<0.01 (2-tailed) * p<0.05 (2-tailed) Note: LA, MA and RA only had 3, 8, and 5 students included respectively in the pairs marking.
5.3 Summary of Findings A summary of findings from the AIT case studies was compiled based on the Feasibility Framework and including a summary of the constraints and benefits of the form of assessment used. The Functional dimension was divided into findings regarding Validity and those regarding Reliability. Validity was analysed by considering: (1) how well the performance of students matches the curriculum outcomes; (2) the extent to which the method of representing performance was authentic; and, (3) whether the task and context were meaningful and relevant to students and community practice.
Feasibility Framew
ork for Applied Inform
ation Technology Case Studies
Functional
ID
Manageability
Technical
Validity* R
eliability
Pedagogic C
onstraints B
enefits
CA
Portfolio and exam
m
anageable though class sizes m
ade for cram
ped and close seating in exam
. Teacher w
as happy w
ith the resources provided.
Several technical
problems: exam
- audio w
idespread failure of sound test. N
ot responding w
ith multiple
apps open. Scans of
designs hard to read.
Portfolio activities based
on course outline and in correct context. Task (e com
merce w
ebsite) authentic though difficult – front end only attem
pted S
tudents found exam
‘appropriate’
Strong and significant
correlations between
assessors A
ssessors/Teacher sem
ester marks
moderately but significantly
correlated.
Teacher and students indicated that assessm
ent matched
typical pedagogy. D
esign process docum
ent didn’t exactly m
atch teacher docum
ent requirements.
All students able to
complete portfolio and
exam. H
ardware,
software, tim
e all sufficient S
ome students
concerned about exam
– malfunction, linear,
disturbance.
Engagem
ent/Positive
student response and opportunity to dem
onstrate creative solutions. S
tudents found exam
‘convenient’.
LA
P
ortfolio and exam
easily managed w
ith sm
all class size in purpose built lab. A
rtefacts – not part of school assessm
ent – som
e not done – large video files?
Up to date system
s presented few
problems.
Som
e user errors with
sound recording- easily fixed. S
tudent concerns re reliability of IC
T.
Student panel indicated
that exam w
as a fair and appropriate assessm
ent. P
ortfolio Task was w
ebsite –business context as course outline. Teacher felt w
ork reflected student capability.
Only m
oderate correlation betw
een assessors sample
size small (10).
Teacher and students indicated that assessm
ent matched
typical pedagogy. Teacher strongly in favour of practical com
ponent. N
ot sure how m
uch to allow
at home.
All students able to
complete portfolio and
exam. H
ardware,
software, tim
e all sufficient.
Motivated students felt
tasks were ‘w
orthy’ of additional effort. P
ositive attitude tow
ards exam, prefer
keyboarding.
MA
Sm
all group. Class
worked exactly as
suggested. 5 m
ins too short for audio. A
ll students able to com
plete portfolio and exam
. Hardw
are, softw
are, time all
sufficient.
Teacher planned for possible problem
s. E
xam- 3 photo files
corrupt – had spare U
SB
.
Teacher followed portfolio
requirements to the letter
and these were based on
course description. Teacher felt w
ork reflected student capability but m
arking guide not adequate. Full com
pliance.
Strong and significant
correlations between
assessors. Good
correlation between
assessors and teachers assessm
ent. Only here
was the sam
e work
marked for the portfolio
and the exam.
Teacher and students positive tow
ards portfolio and exam
w
hich matched typical
classroom practice and
assessment. D
ifficult covering content of syllabus.
None evident.
Positive student
response – easier, faster, m
ore accurate and creative.
RA
Year 12 students
hence reduced time to
complete portfolio.
Exam
– need to swap
sections to do timed
section first.
Server delivered
applications led to a few
hangs and restarts during exam
. Students
able to move to spare
machines.
Sound cards disabled –
fixed for exam.
Teacher followed portfolio
requirements and these
were based on course
description. Students
positive about “doing it on the com
puter” although inexperienced.
Strong and significant
correlations between
assessors and with pairs
marking but no correlation
with teacher m
arking.
Teacher and students indicated that assessm
ent matched
typical pedagogy. S
tudents strongly favoured practical w
ork.
Language used in instruction and theory questions challenged com
prehension of some
students.
Positive students’
response. Exam
fully engaged class. S
ome students w
ell engaged w
ith portfolio.
36
WA
Easy to m
anage small
groups in spacious facilities. S
ome com
ponents not included in school-based assessm
ent. Teacher concern for cheating in exam
.
Som
e technical problem
s related to netw
ork rights, sound drivers – this surprised the teacher.
Authentic task developed
by teacher with real client
who provided task
parameters.
Teacher didn’t like ‘paper-w
ork’ process com
ponents. Felt student w
ork was poor.
Strong and significant
correlations between
assessors and with pairs
marking but no teacher
marks available.
Teacher very positive about m
aking assessm
ent match
classwork- liked
practical exam but not
reflective questions.
Porfolio product
became too com
plex for som
e students who
opted to use advanced softw
are and then becam
e bogged down.
Students very positive
about assessment w
ith a practical bias.
XA
No difficulties in
delivery of portfolio or exam
ination though portfolio task led to som
e file managem
ent problem
s.
Four computers froze
during the exam w
hen opening P
hotoshop and had to be restarted tw
o students m
oved to spare m
achines.
Student portfolio task w
as an anim
ated safety feature m
atching course objectives. Teacher felt exam
was
appropriate but students thought w
as too easy.
Strong and significant
correlations between
assessors and teacher exam
marks, and w
ith pairs m
arking.
Exam
matched
classroom practice.
Portfolio not fully part of
school-based assessm
ent.
Language in reflective questions challenged students understanding. P
ortfolio was not
managed according to
required parameters.
Teacher and students all preferred practical exam
to theory paper. P
ortfolio gave scope for student skills and creativity.
ZA
Tim
e pressure- study ran in parallel w
ith full course so students did portfolio as an extra. Teacher tried to com
ply fully.
No technical difficulties
encountered in im
plementation. S
erver fault led to loss of backup copies by teacher.
Portfolio and exam
com
pleted – course objectives m
et. S
tudents well equipped
and prepared for exam
theory and practical.
Strong and significant
correlations between
assessors and with pairs
marking. N
o correlation w
ith teacher semester
mark.
Exam
and portfolio extended classroom
practice but done as an extra.
None evident other than
time due to non-
inclusion with school-
based assessment.
Students positive
towards com
puter based assessm
ent. Teacher also but w
ith qualifications and suggestions for im
provement.
37
5.4 Summary of AIT Findings Manageability
• No major issues with implementation of portfolio or examination.
• Even larger groups accommodated though proximity meant students work easily visible to neighbours.
• Portfolio submissions presented some difficulties with file sizes and types.
• Audio recording in examination disturbs flow and makes secure invigilation a problem.
Technical
• Sound recording encountered widespread technical difficulties even though this should have been simple. Causes were mainly due to network restrictions put in place to stop students downloading music files.
• Choice of software impacted on processing speed particularly images files. Many schools have high end software but systems barely run this particularly when multitasking
Functional
• In all cases students readily perceived the assessment task was authentic and meaningful for their course.
• The tasks allowed discrimination between candidates of widely varying ability. Pairs marking separation index pointed to wide spread of performance.
• In all cases, the marks and rankings of the two external assessors were significantly and strongly correlated. Marks and rankings of the external assessors and the teacher showed weak or negligible correlation.
Pedagogic
• Typically students liked the idea of practical assessment task.
• Teachers overwhelmingly felt that computer based assessment matched the typical pedagogy for the course.
• Most students believed they could demonstrate their full ability on the computer.
Portfolio
• The requirements of the portfolio and implications of the study on lesson time were not fully appreciated by some of the teachers.
• Where students’ portfolios formed an extra to the course assessment, quality and completion of the requirements were limited.
Computer-Based Exam
• There were some minor technical difficulties in almost every school but these were easily overcome.
• The audio recording caused a lot of problems and probably didn’t add much information as most students simply read their written reflection. The audio reflection disturbed the peace and quiet of the examination.
• Some students were not happy with the rigid structure of the examination and many suggested an unconstrained time format and more creative freedom.
38
5.5 Recommendations for Universal Implementation This section makes recommendations based on an analysis of the 2008 data for the universal implementation of an electronic portfolio and/or a computer-based exam in the AIT course in WA. These two options were tried because internationally they are the most likely to be used to assess students’ capability in using ICT and WA teachers are familiar with these options. From the literature and the pilot study it was understood that each would have different strengths and weaknesses and thus a choice between them depends on the balance and the options for addressing weaknesses.
The syllabus states that the AIT course “provides opportunities for students to develop knowledge and skills relevant to the use of ICT to meet everyday challenges”. As such in the course students should “consider a variety of computer applications for use in their own lives, business and the wider community”. In the course students spend the majority of their time in class using digital technologies to develop information solutions. It should therefore be surprising that currently the external assessment consists of a three-hour paper-based exam. This is despite the fact that the syllabus stipulates that around 50% of the weighting of assessment should be on production. There is a strong rationale for quick change given that most students in Stage 2 and 3 units now have to submit to the final examination.
In general from the 2008 data it could be concluded that both options were able to be implemented successfully although the computer-based exam was the easiest to implement in a consistent fashion between schools. It appeared that only two of the seven teachers implemented the portfolio adequately according to the parameters agreed upon and this was reflected in the results, with students in these schools doing significantly better as a group. However, the portfolio was more reliably marked using the analytical method. The only implementation issues for the exam were the failure of audio recording in three schools and the handful of students who had to move workstations mid-exam due to technical failure at a cost of no more than five minutes.
While the exam was technically implemented well the analytical marking was not as reliable as the portfolio (correlation between markers were statistically significant both for the performance exam, r=0.43, and for the portfolio r=0.79) although the comparative pairs method was highly reliable (SI=0.93). However, Rasch analysis showed that with a minor modification to one criterion reliable scores (SI=0.85) were generated for the exam even from the analytical marking. The main reason for lower reliability for the exam appeared to be because on a few criteria very few top marks were given (e.g. file formats, logo and brochure – explanation or justification).
Rasch analysis of the portfolio analytical marks (Product, Process Document and Extra Artefacts analysed separately) gave a reliable set of scores for all three components (SI=0.96, 0.96 and 0.94 respectively). No modifications were required although for the Extra Artefacts the thresholds for three of the criteria did not work very well. The analytical marks generated by the three components of the Portfolio were not highly correlated probably indicating that they were addressing different types of performance.
A major weakness of the exam was that a relatively low level set of tasks were required (logo, brochure, and spreadsheet graph) so that students in all classes could attempt them using a typically standard set of software that they would all have available. The types of practical tasks teachers give students varies tremendously (e.g. many teachers do not give database type of tasks or even spreadsheets while others will not do animations or movie editing). Therefore it is very difficult to set tasks that all students would have the background to attempt and would also allow adequate scope for the more capable students. This is not a problem with the portfolio as the design brief could vary between classes and typically allowed tasks to be relatively open-ended.
39
The following table summarises the comparison of the two forms of assessment, Portfolio and Exam, using the feasibility dimensions.
Portfolio Computer-Based Exam M
anag
eabi
lity Readily implemented by teachers.
Needs strict invigilation procedures. Submission via disc – online system would be preferable.
Implemented by teacher and researcher in collaboration – would require external invigilator and teacher. Facilitated using USB Flash Drive – online system preferable but may lead to network difficulties. Difficult to ensure students can’t view each others screens
Tech
nica
l
No major technical issues to implement. Some difficulty marking with range of file types and some large files. Files have to be checked by assistant. Teachers set tasks appropriate to the technology available.
Needs to be tested on site to ensure all required technologies work. At most three workstations failed in a class with students moved to spare workstations. In some cases there were no failures.
Valid
ity
Inconsistency of implementation with only two teachers doing so adequately. Good discrimination between students. The three components were not highly correlated indicating they added information.
Difficulty in setting tasks that all students could attempt and would extend more capable students. Quite good discrimination between students. Limited context and set of skills upon which to judge overall capability.
Func
tiona
l
Rel
iabi
lity
Highly reliable analytical marking (SI=0.96). Each of the three components was reliably marked but not highly correlated. Not all students submitted all components. Would be more difficult to mark using pairs.
Less reliable analytical marking (SI=0.85). Highly reliable comparative pairs marking (SI=0.93).
Peda
gogi
c Readily aligns with teachers’ typical pedagogical practices. All teachers already had a similar portfolio. Students liked the form of assessment and were familiar with it.
Was a new experience for two teachers and almost half the students. Students liked the form of assessment but were not as familiar with it as the portfolio.
Recommendation Overall when comparing the Portfolio and the Exam there was no compelling reason to choose one over the other, each had strengths and weaknesses. Therefore it is recommended that the Council make its decision on the basis of how well each could be implemented in the manner outlined in the next two sections. The choice is a limited portfolio, a computer-based exam or perhaps a combination.
Implementing the Portfolio If the Portfolio was implemented the structure used in the study is recommended. This allows students to adequately demonstrate their capability with some scope for tailoring to the context for the student. However, ideally it would need an online portfolio management system and would need a well-structured system for verification that would probably include some type of signed affidavit with spot checks on a sample of students to ensure all teachers implemented the portfolio according to the required conditions.
(1) Digital product (a 15-hour project)
(2) Process document (a 5-hour task to collate evidence)
(3) Two previously created digital artefacts (a 1-hour task to select and present the artefacts)
A set of parameters needs to be set for the Portfolio including the following.
• Time limits for development of digital product (15 hours within 4 weeks), process document (5 hours), and artefact preparation (1 hour).
• A choice of design briefs is needed, ideally teachers would be allowed to set an appropriate design brief within the parameters of the portfolio (e.g. aim, purpose, included components, file sizes and
40
types). However, many teachers will not feel confident to do this and thus three or four example design briefs should be supplied for them to use or modify.
• The nature of the process document (nine pages total with suggested limits for each of the four sections: Research, Design, Production and Evaluation) as a selection and collation of material resulting from the development of the digital product that best presents the process of development employed.
• The nature of the extra artefacts as demonstrations of skills in areas other than that demonstrated in the main digital product (short support document – half a page table).
• The components of the Portfolio need to be marked separately using different criteria as they represent different types of performance (e.g. the Process Document provides an opportunity to demonstrate an understanding of the Technology Process and capability in planning, analysing, organising, managing and evaluating).
Implementing the Computer-Based Performance Exam If the performance tasks exam was implemented then the structure of the exam used in the study is recommended with some minor modifications (e.g. removal of audio response). However, the study has highlighted two areas in which decisions would need to be made: technical implementation; and performance tasks specification.
Technical Implementation
The project used USB flash drives that worked on school computers for all students. This would be cumbersome but not unrealistic to scale up state-wide but in the long-term an online exam management system should be used. However, this introduces the school network as a variable that our project found was a confounding variable in many schools.
There is the question of whether access to the Internet should be precluded. In 2008 this was not a requirement although at least two schools did this of their own accord. There was a requirement that students could only use the media (e.g. photos) that were provided on the USB flash drive. In one class at least one student used an online text design tool.
In 2008 no attempt was made to limit access to software that was normally available to the students. The low-end nature of the tasks meant that this provided little, if any, advantage to any students with most using Microsoft software (e.g. Publisher) and a relatively basic graphics package. The choice would seem to be to specify the software allowed OR allow all available OR provide software on the USB (probably not practical for AIT).
The option of paper-based or digital design phase appeared to be appreciated by students with most opting for paper-based design. These then had to be scanned for marking by project assistants.
Performance Tasks Specification
In the study the performance tasks were selected to be relatively low-level and easy to ensure all students could engage. Further these tasks were defined fairly explicitly with little or no choice for students. Clearly if this were done for the final external assessment then it would limit the opportunity for high ability students and would tend to stifle context variety in the course. Therefore it is likely that some degree of choice of context, type of solution and/or tools used is needed but this makes the development of appropriate tasks, description of those tasks and the statement of assessment criteria more difficult.
Given that currently many teachers do not appear to be addressing all the content, with some focusing almost exclusively on interactive multimedia development and others on business software, the exam would have to follow a pre-defined structure so that teachers could adjust their programmes accordingly. For example, it may need to be known that students will have to develop a graphic, a spreadsheet, access a database, and hyperlink media elements etc. What will then be unknown is the specific scenario to be addressed and the associated contextual media provided.
In the second year we will trial a more complex task that allows for more choice.
41
References Barrett, H. C. (2005). Researching Electronic Portfolios and Learner Engagement: The REFLECT
Initiative. Becta. (2006). Becta's View: E-assessment and e-portfolios: Becta ICT Research. Beetham, H. (n.d.). e-portfolios in post-16 learning in the UK: developments, issues and opportunities.:
JISC. Brewer, C. A. (2004). Near Real-Time Assessment of Student Learning and Understanding in Biology
Courses. Bioscience, 54(11), 1034. Bull, J., & Sharp, D. (2000). Developments in computer-Assisted Assessment in UK Higher Education.
Paper presented at the Learning to Choose: Choosing to learn, QLD, AUS. Bull, J., & Sharp, D. (2000). Developments in computer-Assisted Assessment in UK Higher Education.
Paper presented at the Learning to Choose: Choosing to Learn., Queensland, Australia. Burns, R. B. (1996). Introduction to research methods. South Melbourne, Australia: Addison Wesley
Longman Australia Pty. Limited. Carney, J. (2004, April 14, 2004). Setting an agenda for electronic portfolio research: a framework for
evaluating portfolio literature. Paper presented at the American Educational Research Association Conference.
Cisco, I. a. M. (2009). Transforming Education: Assessing and Teaching 21st Century Skills. Retrieved 8/6/2009, 2009
Harding, R. (2006). What have examinations got to do with computers in education? Journal of Computer Assisted Learning, 17(3), 322–328.
Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., & Yan, F. (2006). Does it Matter if I take My Writing Test on Computer? An Empirical Study of Mode Effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2).
Kimbell, R., & Wheeler, T. (2005). Project e-scape: Phase 1 Report. London: Technology Education Research Unit, Goldsmiths College.
Koretz, D. (1998). Large-scale portfolio assessments in the US: Evidence pertaining to the quality of measurement. Assessment in Education, 5(3), 309-334.
Kozma, R. B. (2009). Transforming Education: Assessing and Teaching 21st Century Skills. In F. Scheuermann & J. Bojornsson (Eds.), The Transition to Computer-Based Assessment (pp. 13-23). Ispra, Italy: European Commission. Joint Research Centre.
Lane, S. (2004). Validity of High-Stakes Assessment: Are Students Engaged in Complex Thinking? Educational Measurement, Issues and Practice, 23(3), 6-14.
Lesgold, A. (2009). Better schools for the 21st Century. Retrieved 9/6/2009, 2009, from http://atc21s.basecamphq.com/clients
Lim, C. P., & Hung, W. L. (2003). An activity theory approach to research of ICT integration in Singapore schools. Computers and Education, 41(1), 49-63.
MacCann, R. (2006). The equivalence of online and traditional testing for different subpopulations and item types. British Journal of Educational Technology,, 37(1), 79-91.
Madeja, S. S. (2004). Alternative assessment strategies for schools. Education Policy Review, 105(5), 3-13.
MCEETYA. (2007). National Assessment Program – ICT Literacy Years 6 & 10 Report 2005. Carlton South, Australia: Curriculum Corporation.
McGaw, B. (2006). Assessment to fit for purpose. Paper presented at the 32nd Annual Conference of the International Association for Educational Assessment., Singapore.
Messick, S. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23(2), 13-23.
Newhouse, C. P. (2005). Assessment of students by a teacher with a hand held device and a networkable database. In B. L. Mann (Ed.), Selected Styles in Web-Based Educational Research. Hershey, PA: Information Science Publishing.
Newhouse, C. P., Clarkson, B., & Trinidad, S. (2005). A framework for leading school change in using ICT. In S. Trinidad & J. Pearson. (Eds.), Using Information and Communication Technologies in Education. Singapore: Prentice-Hall, Pearson Education South Asia.
Newhouse, C. P., Williams, P. J., & Pearson, J. (2006). Supporting Mobile Education for Pre-Service Teachers. Australasian Journal of Educational Technology,, 22(3), 289-311.
42
Pollitt, A. (2004, June 2004). Let’s stop marking exams. Paper presented at the The International Association for Educational Assessment Conference, Philadelphia.
Ridgway, J., McCusker, S., & Pead, D. (2006). Report 10: Literature Review of E-assessment.: Futurelab. Ripley, M. (2009). Transformational Computer-based Testing. In F. Scheuermann & J. Bojornsson
(Eds.), The Transition to Computer-Based Assessment (pp. 92-98). Ispra, Italy: European Commission. Joint Research Centre.
Scheuermann, F., & Bojornsson, J. (Eds.). (2009). The Transition to Computer-Based Assessment. Ispra, Italy: European Commission. Joint Research Centre.
Spector, J. M. (2006). A methodology for assessing learning in complex and ill-structured task domains. Innovations in Education and Teaching International, 43(2), 109-120.
The British Psychological Society. (2002). Guidelines for the Development and use of Computer-Based Assessments. Leicester, UK: The British Psychological Society.
The Council of the International Test Commission. (2005). International Guidelines on Comptuer-Based and Internet Delivered Testing. Granada, Spain: The Council of the International Test Commission.
Willig, C. (2001). Introducing qualitative research in psychology adventures in theory and method. Buckingham: Open University Press.
43
Appendix – AIT Assessment Task 2008 This was a hybrid assessment of a reflective process portfolio and a performance tasks exam to allow a comparison between them. The aim has been to be as open-ended as possible to allow variety of contexts, but structured to support valid and reliable marking.
Digital Reflective Process Portfolio A digital portfolio to contain: (1) A digital product the student has designed as a prototype of an information solution. (2) The design process document for that digital product. (3) Two other digital artefacts that illustrate skills in two areas from a specified list. Portfolio (Component 1) - The Digital Product Create a prototype of an information solution in the form of a digital product relevant to a business context and using applications software commonly used in organizations for productivity, planning and communication (e.g. word processing, publishing, presentation and financial data management). A technology process should be employed in the investigation, design, production and evaluation of the product. Output from these processes will be required for the Design Process Document and therefore the requirements of this document should be used to guide the technology process. The digital product should:
• suit the intended purpose and audience/users; • meet the requirements of the design brief and/or client specifications; • illustrate creative application of information design principles and technologies; • make use of appropriate information structures, forms, layouts and symbols; • employ relevant standards and conventions to create multi-tiered information solutions; and • use appropriate methods and techniques to represent the design of information solutions.
The digital product will be delivered in a single digital file with one of the following formats: PDF, AVI, JPG, GIF, SWF, FLA, HTML or ZIP (must be a collection of files with the permitted formats e.g. zipped folder of a website of HTML and FLA files). The file will not exceed 20MB. The product must have been produced at school using hardware and software provided by the school and represent no more than 15 hours work from receiving the design brief to completion of the prototype over a period of no more than 4 weeks. [Notes: Teachers may select a design brief context for students or allow them to select their own (see example below).] Example Design Brief Miss Shoppe is the manager at a local retail clothes outlet. She is very concerned with the increasing number of people shopping online and the declining number of consumers venturing into her shop to purchase her products. The shops target market is teens (12 – 20 years). She has approached you to create her own online shop front. She would like the website to include general information regarding the shop (Open hours, Products, Location), contact details (Location, Telephone number, Email address) and an online catalogue (List of products, Bulletin Board, Mailing List, Current News). Her corporate colours are Green, White and Black. Using this information, design the online presence for Miss Shoppe. Miss Shoppe has requested that you present your designs as detailed storyboards and provide a summary of recommendations that you have made. Miss Shoppe has also requested that a detailed production plan be developed. Select your best design and develop a website that will allow her shop to have an online presence as a means of contacting her target audience, promoting her business and potentially selling more products. Use any suitable software to create the website and any suitable media, taking care to appropriately acknowledge the source of any media you use.
44
Portfolio (Component 2) – The Design Process Document [Notes: Many pages may include diagrams, photos, screenshots etc. – it is not all text.] Over a period of five hours students will collate a document with a maximum of NINE pages as a single PDF file that comprises four sections: Research, Design, Production and Evaluation. Relevant language and terminology should be used throughout. (1) RESEARCH The document will present, in no more than TWO pages, the results of their investigation of solutions to the information design problem to include:
• An outline of the human need or opportunity that was addressed. • The main objectives of the information solution. • Brief descriptions of existing solutions and what aspects needed improving and thereby the criteria
that could be used to evaluate the success of their own solution. • A summary of the strategies that were used to find and analyse relevant information to generate ideas
including methods such as brainstorming and mind-mapping. (2) DESIGN The document will present, in no more than THREE pages, the final design and design processes to include:
• Adequate information that would allow another skilled person to complete the production such as descriptions, storyboarding and concept development processes such as thumbnail sketches, annotations, photographs, drawings, flowcharts and schematics developed to represent the design.
• Examples of early attempts which were subsequently improved with explanation of the improvements.
• An explanation of how they applied technologies in creative and original ways to meet the need, considering purpose, meaning, target audience and client specifications.
(3) PRODUCTION The document will present, in no more than THREE pages, a plan of project management, activities, sequencing and logistics, to include:
• The production plan for the prototype solution including the key decisions, acknowledging contextual influences, the use of design elements, standards and conventions and justification of tools used.
• A list of the hardware, software, materials and personnel resources employed. • Descriptions of the skills and understanding that were needed to apply the hardware and software.
(4) EVALUATION The document will present, in no more than ONE page, the evaluation of the prototype information solution and technology processes employed, to include:
• An explanation of how the information solution was evaluated. • A summary of the results of the evaluation reflecting on the strengths and weaknesses of the
solution. • An evaluation of the technology process employed.
Portfolio (Component 3) - Two Extra Digital Artefacts Two digital artefacts should be submitted that illustrate the student’s skills in applying design principles in any two of the following domains … graphics, databases, spreadsheets, web-publishing etc. The digital artefacts must have been created by the student, at school, under supervision from the teacher. Any assistance from the teacher or others must be explained. The digital artefacts must include a document of no more than ONE a page in length (combined) describing for each artefact what hardware, software, techniques and skills were needed to create the artefact. [Note: The rationale is that in the ‘project’ part of the portfolio they will only have been able to illustrate skills in one domain. This section allows them to illustrate skills in other domains without having to explain a technology process even though the artefacts may have resulted from other projects.]
45
Performance Tasks Exam The performance tasks exam will be completed using a computer workstation and will include a set of reflective questions (on-screen) concerning the digital portfolio and a set of short performance tasks.
Exam (Component 4) - Reflective Questions [Notes: (1) Should be marked in combination with the portfolio. (2) Students could be permitted to view their portfolio product (not process document) while responding to the questions. (4) Could provide audio recording of oral responses rather than text response. (5) Could include a section that either asked them to compare their work in terms of design principles with a high quality product or critique the latter.]
Students complete on-screen a series of reflective response question (1 hour). 1. For the project you presented in your portfolio think about how you investigated and evaluated potential
solutions to the information design brief. (a) Describe the technology challenge or problem you were responding to. (b) Define the objectives you developed for the project. (c) Explain the research you undertook and how this influenced TWO of your design decisions.
2. For the project think about how you devised and communicated design plans. (a) Justify the TWO most critical factors in the selection of your final design. (b) Explain the forms and techniques you used to communicate your design/s (e.g. diagrams, descriptions,
charts). (c) Explain how you evaluated your proposals and design plans.
3. For the project think about how you went about making or producing a prototype or final product from your designs. (d) Describe the main production processes you used for your information solution. (e) Explain and justify the most important change or modification to your design you had to make as you
were producing your information solution. (f) Explain the TWO most important measures you used to ensure you used time efficiently and resources
safely and effectively. 4. For your project, consider the evaluation of your information solution and your production processes.
(g) Describe how you evaluated your final information solution. (h) Explain and justify the standards and conventions that were appropriate for your information solution. (i) Describe ONE change you would have liked to make to your information solutions explaining how
this would improve the quality of your solution.
Exam (Component 5) - Performance Task(s) Students complete a set of scripted tasks at a computer workstation over a short period of time (2 hours) under ‘exam’ conditions.
Context: Business Stationery
In two hours respond to the following technology challenge. [Initial explanation of scripting of exam]
Items supplied: 1 x 1GB mass storage device containing: 20 digital photographs (photo1.jpeg to photo20.jpeg), 1 text file with information about the resort (data.txt); 1 x audio headset and microphone
Ningaloo Reef Resort is opening a 200-room hotel and marina on the coast of Western Australia. The resort is in a sensitive conservation area and there was a lot of opposition to the development. The hotel itself offers fine dining, luxurious rooms, swimming pools as well as swimming, diving and snorkelling on the reef. The hotel manager wants you to apply your information technology skills to a series of tasks to help with the marketing of the resort.
Task 1: Planning Logo and Brochure (required time 15 minutes)
For this task you may either work on the paper provided or using the Word document plan_template.doc. You will not be permitted to continue into Task 2 until the 15 minutes has elapsed.
46
Develop three ideas for a resort logo and two ideas for a brochure to advertise the resort. The logo and brochure have to suggest that the development has a low environmental impact and is ‘clean and green'. Remember there was opposition to the development. Add notes to your designs to explain the creative processes involved in developing the logo and brochure. To help you, think about your ideas, themes, design principles, colours, shapes and symbolism. Make any notes clearly as the designs will be scanned for assessment. Put your candidate number at the top of each page.
Task 2: Logo (suggested time 30 minutes)
Take one of your design ideas and using any software available to you, create, A4 size, a logo for the resort. Save your logo as logo.pdf on the mass storage device provided.
Task 3: Graphs (suggested time 15 minutes)
In the file data.txt there is some climatic data. In the next task you will be creating a brochure into which you will be asked to present the climatic data graphically. Import the data into a spreadsheet and create at least two different graphs using the climatic data. Save the spreadsheet file with the graphs included as graphs.xls on the mass storage device provided.
Task 4: Brochure (suggested time 45 minutes)
Using some of the digital photographs supplied, the data found on the file data.txt, and your own ideas develop a tri-fold brochure to promote the hotel. The brochure is designed for travel agents and tourist offices. You may manipulate the photographs in any way you wish but only these files may be used. The brochure must include
• your logo, the address and contact details of the resort
• financial information about the cost of staying at the resort
• climatic information (temperature and rainfall) at the resort presented graphically
• the text that introduces and sells the resort, explains or captions the images you have chosen and holds the brochure together. How much you write is up to you.
Save the brochure as brochure.pdf on the mass storage device provided.
Task 5: Prepare Reflection (suggested time 10 minutes)
Prepare a 30 second (half a minute) audio reflection by listing headings or points in the document reflection.doc to
• explain the creative processes involved in developing the logo and brochure
• appraise your finished product
• other uses and other products that could be developed
Think about your ideas, themes, the images chosen, design principles, colours, shapes and symbolism of your logo and brochure as well as what you achieved and would have liked to achieve. Open and make ready a suitable sound recording application but do not begin recording until the invigilator gives permission.
Task 6: Audio recording (required time 5 minutes)
The invigilator will announce that audio recording may begin. Record the audio commentary. Save the audio file as reflection.mp3 or reflection.wav
47
Criteria for Marking: AIT Assessment Task (ECU)
Digital Portfolio (Marking Criteria)
Quality of Production (20%) 6: A range of consistent application to achieve particular effects/impacts. 5: Consistent application. 4: Some consistent application to achieve at least one particular effect/impact. 3: A range of application. 2: At least one significant application appropriate to the purpose of the solution.
Application of design principles through styles/structures/codes/conventions relevant to the form of information solution (e.g. Digital Graphics - spacing, colour, size, position, fonts, consistency, variations, ...)
1: One significant application relevant to the form but not necessarily appropriate to the purpose of the solution.
6: Application of a range to achieve industry entry standards of quality. 5: Some application to achieve industry entry standards of quality. 4: Application to create particular appropriate effects. 3: Some consistent application. 2: Some appropriate application of techniques to address the task.
Application of technical skills and techniques (e.g. Digital Graphics - layers, tools, file format, editing)
1: Some appropriate application of technical skills to address the task. 4: Creative and innovative attributes to achieve a particular purpose and impact
consistent with a desired meaning and/or effect. 3: Individuality and experimentation consistent for a particular effect. 2: Experimentation with design ideas.
Creativity and innovation in design to produce functional and aesthetic features
1: Some individuality. 4: Consistent and effective consideration of target audience/user and/or client. 3: Inter-related consideration of purpose, audience/user and/or client. 2: Considers some characteristics of the audience and/or users.
Prod
uct
Appropriateness of product design for the likely target audience and/or client
1: Communicates meaning according to some of the characteristics of the purpose. Design Processes (10%)
3: Considers ideas of others, the appropriateness of existing technologies and the needs of environments.
2: Considers the use by individuals and communities to meet particular needs.
Investigation of a technology challenge
1: Uses information from more than one source and selects resources in response to identified needs and research criteria.
3: Includes an examination of a range of options and reasons for choices. 2: Develops a design providing problem analysis, and identification of input/output
needed and constraints.
Devises designs
1: Develops a design plan although inadequate. 4: Uses graphics and technical languages associated with the relevant fields of ICT
including the use of visual imaging techniques; and using consistent technical terms and conventions.
3: Communicates adequately using a range of drawings, diagrams, notes, visual and graphical representations and models; annotating and completing the designs; and using simple technical terms and conventions.
2: Communicates completely using at least two of a range of drawings, diagrams, notes, visual and graphical representations and models.
Communicates designs
1: Communicates plans inadequately or incompletely using any of drawings, diagrams, notes, visual/graphical representations, or models.
Management of Production Processes (10%) 4: Considers the efficiency and effectiveness.
3: Manages and organises showing some personal style.
2: Employs a given framework to manage production processes.
Management processes
1: Limited management.
3: Detailed production plans.
2: Adequate plans.
Production plans
1: Insufficient plans.
3: Consistent use.
2: Uses suitable technical language and conventions.
Use of technical language and conventions in production plans
1: Some relevant technical language and conventions.
Evaluation Processes (10%) 3: Consistent with original design, function, form and purpose that may be an adaptation of a standard set. 2: Processes linked to original design, function, form and purpose.
Development of evaluation processes
1: Some limited processes.
Proc
ess
Doc
umen
t
4:.Fully implements own evaluation processes.
48
3: Applies appropriate standard/provided evaluation processes. 2: Applies limited standard/provided evaluation processes.
Implementation of evaluation processes
1: Applies standard/provided processes inadequately. 3: Consistent consideration of target audience/user and/or client. 2: Inter-related consideration of purpose and audience/user.
Consideration of the likely target audience, purpose and/or client
1: Considers some characteristics of the audience and/or users. Use of technologies, skills and processes (20%)
4: Complex solutions. 3: Reasonably complete solutions. 2: Applications are very different.
Breadth of types of software/hardware, skills and processes
1: Some differences in software applications used but general similar types. 4: Used software (and hardware where relevant) in an unusual but effective manner. 3: Consistent personal style. 2: Relevant personal style.
Creative use of software and hardware
1: Shows some personal style. 4: Consistent selection of appropriate software and hardware for type of information solution. 3: Mainly appropriate 2: Appropriate selection
Selection of software and hardware
1: Some selection evident 4: Typically selects most appropriate techniques to apply to particular tasks. 3: Selects appropriate techniques for tasks in one of the products. 2: Selection of techniques tends to be adequate but not always the most appropriate.
Selection of techniques
1: Selects techniques but they are inadequate for the tasks. 4: High proficiency across both types of information solutions. 3: Efficiently and showing some personal style. 2: Application of skills is typically adequate but not efficient.
Extr
as(2
)
Application of skills
1: Inadequate application of skills.
49
Performance Tasks and Reflective Exam (Marking Criteria) Technology processes in the workplace (5%)
3: Justifies investigation processes employed with a variety of methods relevant to workplaces. This will include reference to problem-solving methodology incorporating ICT, research, and strategies for efficient search, retrieval, and referencing.
2: Describes research.
Explanation of investigation of information solutions (Question 1)
1: Describes challenge and objectives. 2: Explains with some justification a set of processes with a variety of procedures,
methods and rules relevant to typical workplaces and compares with alternatives. This will include reference to planning and organisational skills, techniques to communicate plans and design (e.g. graphic overviews and flowcharts), and the use of ICT to support enterprise, individual and collaborative interactions.
Explanation of design and planning processes (Question 2)
1: Describes a set of processes employed that would be appropriate for a relevant workplace with some reference to planning, organisational skills, and techniques to communicate plans and designs.
Processing and management of data (5%) 3: Justifies the implementation and adjustment of production processes, considering the
efficiency and effectiveness of these processes to achieve accepted standards of quality and effective strategies for the management of data including archiving practices and procedures.
2: Explains modifications using suitable technical languages and conventions.
Explanation of production processes and management of data (Question 3)
1: Describes, using some relevant ICT-related concepts, standards and terminology, simple techniques and procedures to select, use and control hardware and software
2: Describes and justifies, using suitable technical languages and conventions, the evaluation and adjustment of production processes, considering the efficiency and effectiveness of these processes to achieve accepted standards of quality.
Ref
lect
ive
Que
stio
ns
Explanation of evaluation of production processes (Question 4)
1: to develop an information solution that includes the forms, structures and conventions, including ergonomic principles that are appropriate for the solution.
Applications and systems software (10%) 2: Applies appropriate features of spreadsheet to create an adequate graph (e.g. scales,
labels, colours). Spreadsheet graphs (Task 3)
1: Creates an appropriate graph using the supplied data. 4: Justifies the selection of software to achieve accepted standards of quality. Used
specific styles, forms and techniques to create or modify information to represent the particular effect of the design and achieve defined standards of quality.
3: Applies appropriate features of software to create an adequate logo (e.g. colour, line, transparency).
2: Effective logo.
Logo (Tasks 2, 5 & 6)
1: Creates a logo.
4: Used specific styles, forms and techniques in Word to create brochure represent the particular effect of the design on the audience and achieve defined standards of quality.
3: Applies appropriate features of Word to create an adequate brochure (e.g. columns, tables, format styles).
2: Completed brochure including all required information in useable form.
Brochure (Task 4, 5 & 6)
1: Almost complete brochure.
The nature, form and transfer of digital data (5%) 3: Application to represent the particular effect of the design on the audience and
achieve defined standards of quality for specific print formats. 2: Application suitable to the nature and form of the data.
Styles, forms, structures, conventions and techniques used with digital data
1: Application for a purpose. 2: Applies appropriate file formats, compression and encryption techniques, conversion,
size and storage requirements. File formats (Submission)
1: Provides required file formats. Creative application of information design principles (5%)
3: Creates particular effects to achieve a particular purpose and impact, creatively experimenting with design options considering the purpose, audience and client specifications.
2: Justifies appropriately the selection of design options.
Creative exploration of design options of the relevant forms, structures and conventions. (Task 1)
1: Purposefully explored one or more relevant alternative design option for Logo or Brochure.
2: Planning shows consideration of legal requirements and the requirements of clients and audience to ensure solutions are acceptable.
Perf
orm
ance
Tas
ks
Consideration of individuals and communities. (Task 1)
1: Planning shows consideration of characteristics and requirements of clients and audience.