v. turau, d. fahrenholtz, and m. venzke 8 th international caa conference 05+06/07/2005 online...

V. Turau, D. Fahrenholtz, and M. Venzke

8th International CAA Conference 05+06/07/2005

Online Assessment for University Admission: Goals, Problems, and

Experience

Dietrich FahrenholtzDietrich FahrenholtzTelematics Group, Hamburg University of Technology, GermanyTelematics Group, Hamburg University of Technology, Germany

email: [email protected]



OverviewOverview

• Motivation for Online Student Recruitment

Goals, Issues

• System Features and Implementation

• System Evaluation

• Conclusion and Outlook



Motivation for Online Student Recruitment (1)Motivation for Online Student Recruitment (1)

Environment as enabler

• Change in German legislation Now German universities have more choices in selecting

students

• Traditional selection procedures are costly (time, effort, money, other resources)

may involve external trusted party

invitation not feasible (many students apply from abroad)

Web-based technology offers promising solution



Motivation for Online Student Recruitment Motivation for Online Student Recruitment (2)(2)

Hamburg University of Technology wants to ... select students who

• closely fit to desired course(s)

• examine course contents thoroughly before applying

• will most likely meet University standards

support applicants in finding out whether they are qualified or not deter those who show low prospects increase (decrease) number of good candidates (dropouts) assess where a student stands in terms of his/her knowledge

• identify lack of necessary learning prerequisites

high degree of automation• reliable assessment results with little human intervention

figure out provenience of good to outstanding candidates• name/type of high school, bias towards one, etc.



Motivation for Online Student Recruitment (3)Motivation for Online Student Recruitment (3)

Get hands-on experience whether or not online student recruitment lives up to expectations Web technology shows its suitability workload for administration reduces/increases/remains the

same

Main advantages online student recruiting independent of location and time

fewer financial resources necessary

potential for high automation of applicant performance assessment



Issues with Online Student Recruitment (1)Issues with Online Student Recruitment (1)

General familiarity with computer and Internet use authenticity and identity of participants balance between protection of applicant data and meaningful

statistics inability of any Online Assessment (OA) to measure

• student efficacy• stress resistance• motivation• creativity

Sociological participant honesty (who actually did the test?) Q&A of tests appear on some Web site

→ highly parameterized questions→ continuous revisions and additions

“pranksters”



Issues with Online Student Recruitment (2)Issues with Online Student Recruitment (2)

Technical server outages hacker attacks

Legal aspects appeal against results of test admission enforceable by law?



OverviewOverview

• Motivation for Online Assessment

Goals, Issues






System Features and Implementation (1)System Features and Implementation (1)

Features of Web questionnaire

Categories reading comprehension English/German numeracy knowledge about university and course general knowledge

Questions cover many engineering disciplines have different degrees of difficulty emphasis is on mathematics drawn uniformly from a pool of many questions representatives of professors jointly identify well-chosen questions technical support checks if a question fits to system model

Answer types multiple-choice single/multiple select single-choice comment

• adjustable weight of category (# of questions in pers’ed questionnaires)• points only for complete and correct answer, no negative scoring




Guiding principles (Blandford et al.) ease and effectiveness

Registration first/last name, date of birth, gender,

name of school, country, valid email address, etc.

obtain unique identifier

Online Test personalized copy of self-test solution submission within 60 minutes

Analysis document contains both answers and

questions student only learns percentile s/he is in,

no disclosure of correctness of answer

Essay why decision for particular course other activities




Technical details• browser independence

little use of JavaScript (clock)

• user interface adapts to different languages

• import and export of questions conforming IMS



OverviewOverview

• Motivation for Online Assessment

Goals, Issues






System Evaluation (1)System Evaluation (1)

Aspects suitability and quality of questions

• evaluation by sample groups

• meta data derived from Online tests

• analysis of additional post factum data

user acceptance real admission procedure vs. conceived (condition of OA software)

• well known techniques from Software Engineering

Goal: Rejection of a highly talented applicant should not happen!




Evaluation of suitability and quality of questions

Sample groups professors, post graduates, 2nd term students

Procedureexperimentee was observed and asked for his/her opinion on comprehensibility of introductory instructions and questions in general time allowed for completing the Web test part overall assessment and the online part in particular

indicates that varying degrees of difficulty are well chosenleads to a catalogue of questions for the initial OA




Online assessment results evaluation period: winter term 2004/2005 (6 months) number of participants from target group: >1000 >45% of candidates answered more than 50% of questions correctly




Question quality/usefulness derived from meta data determine degree of difficulty

• number of correct/wrong answers• number of times a question remains unanswered• total points distribution skewed• facilityfacility: ratio of correct to total number of answers

discriminate stronger to weaker candidates• correlation between performance of a sub-group of candidates on a

particular question and their overall score

continuous revision of questions and sample answers




Appropriateness of test duration• number of questions answered in time

• number of late submissions

~70% of applicants answer all questions

~34% did not submit answers

Spotting “pranksters”• caveat: missing paper application not a valid

indicator

• senseless personal data (name, date of

birth, etc.)

• unexpected way of taking test answers submitted too early (within, e.g., 6

minutes)

only a few questions answered




Verification of objectives interview applicants about impact of assessment result on

their decision

redo test with second year students and compare results with

applying ones

same as above but compare results with examination results

(e.g., grades, overall progress in course)



Conclusion and OutlookConclusion and Outlook

Designing a system to support student recruiting requires some effort to find suitable procedures and questions

viable alternative if student recruitment needs to be carried out with

limited personal and financial resources.

Operation of our OA system In winter term 2004/2005 optional part of an application (test phase)

mandatory from now on



DiscussionDiscussion

Thank you!

Further questions?



Web Test in ActionWeb Test in Action

remaining time

questions

answers



Web Test Administration GUI (1)Web Test Administration GUI (1)



Web Test Administration GUI (2)Web Test Administration GUI (2)

personal dataessay

v. turau, d. fahrenholtz, and m. venzke 8 th international caa conference 05+06/07/2005 online...

Documents

international caa conference

student stands

online assessment oa

outlook slide

promising solution slide

overview motivation

university admission

expectations web technology