creating a multiple measures placement system an exercise with ron gordon & armand brunhoeber
TRANSCRIPT
![Page 1: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/1.jpg)
Creating A Multiple Measures Placement System
An Exercise
With
Ron Gordon & Armand Brunhoeber
![Page 2: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/2.jpg)
The Concept
• Although test scores may predict failure, they do not necessarily predict success.
• Research shows that students’ backgrounds, environments, and personal habits may have more influence on their potential academic success than their residual academic skills.
• Find a way to factor that information into the placement decision at testing time.
![Page 3: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/3.jpg)
Decision Process
• Which Courses• Which Tests• Back Ground Questions
– How many– Variety
• Select Weight Values– Use negative weights?– How much possible total weight
![Page 4: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/4.jpg)
Planning Flow
• Who– Discipline faculty– Testing staff– Counseling– IT– Others – Research?
![Page 5: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/5.jpg)
Planning Flow
• What– Which courses in which disciplines– What information can students provide at the
time of testing?– Which questions will be used in each
discipline?– How much total value should additional
measures have?
![Page 6: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/6.jpg)
Planning Flow
• Why– State mandate?– Literature– Improve placement accuracy– Give students better opportunities to learn
![Page 7: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/7.jpg)
How Does it Work?• Students answer locally developed
background questions during the test session
• ACCUPLACER placement rules compute a weight value for the student’s answers based on locally developed weighting rules
• The test score, plus a percentage based on the background information is used for placement
![Page 8: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/8.jpg)
The Process• Educate faculty and others on how the
system functions• Find examples of background questions
and edit to fit• Assign weight values to each answer
choice.• Write placement rules using the multiple
measures editing function
![Page 9: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/9.jpg)
Select Questions
• Questions must, in some way, relate to student success.
• Must be information the student will have at the time of testing
• Must be multiple choice• Should solicit behavioral,
historical/experiential, and environmental information
![Page 10: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/10.jpg)
Limits
• Limit total weight so that background information does not allow students to skip a course level
• Limit number of questions to a manageable number– More questions adds to testing time– Placement rules can become unmanageable
• Answer choices must be mutually exclusive and all inclusive
![Page 11: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/11.jpg)
Example
• (If Arithmetic, plus all weighted choices is >= 75 OR
Algebra, plus all weighted choices is >= 48)AND
(Algebra, plus all weighted choices is < 65 OR Algebra Not Taken) AND
(CLM, plus all weighted choices is <62 OR
CLM Not Taken)
Then Placement is Elementary Algebra
![Page 12: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/12.jpg)
Example, Continued
• If this rule had 5 questions with 4 weighted answer choices each, there would be 80 lines just for weights.
• With too many questions, or too many choices per question, rules can become unmanageable
![Page 13: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/13.jpg)
Conditional Weights
• High School Accomplishments Have Limited Shelf Life
• How Much Does it Matter That a 25-Year-old Student Had 2 years of High School Algebra?
• Does it Matter That The Same 25-Year-old Student Works for a Surveyor and Uses Algebra Daily?
![Page 14: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/14.jpg)
Example
• How long has it been since you were enrolled in high school or other formal educational process?– Less than 2 years or still enrolled– 2 to 5 years– More than 5 but less than 7 years– 7 years or more
• Use high school data for up to 5 years, experience for more than 5 years.
![Page 15: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/15.jpg)
Assigning Weights
• Total possible weight should not move student more than one level in either direction
• Set maximum possible weight so a student who scores at or above the midpoint of a placement range could move up, but one who scores below the midpoint could not.
• Use faculty to select BGQ and assign weight• Guide them
![Page 16: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/16.jpg)
Multiple Measures Influence on Placement
Range 60 To 71 To 81 To 93
Class Engl 110 Engl 105 Engl 51
Midpoint 65.5 76 87
Movement ? ? 8.4%
? ? 6.6%
? ? 6.9%
Multiple Measures Movement Model
![Page 17: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/17.jpg)
Sample Question With Weights
• Which choice below best describes you when you read textbooks or other complex information?– I usually need to read material several times before I
understand it well -.01– Sometimes I can understand what I read the first time,
but often I must reread it .00– I usually understand what I read if I take notes or
highlight passages. +.01– I always understand what I read the first time
through +.02
![Page 18: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/18.jpg)
Think About Uploading to SIS
• Multiple measures may change placements, but not scores.
• SIS has no place to store multiple measures
• To upload multiple measures, the placement (course name) must be uploaded
• In ACCUPLACER, course name can be numeric
![Page 19: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/19.jpg)
Preparing to Build the System
• Assign numeric codes to course names• Determine which tests will be used for
each course in each discipline• Create cut score Table• Create a BGQ weight Matrix
![Page 20: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/20.jpg)
Building the System• Create Background questions• Assign BGQ to groups• Create branching profiles• Create course groups• Create courses and assign to groups List• Create majors if used • Create placement rules Edit1 Edit2• placement rule sample.pdf
![Page 21: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/21.jpg)
Verify
• Write most complex rule first• Run verify function in branching profile• Use several BGQ and score combinations
to test the placement rule• Compute weighted score for each run• Try to hit cut scores to test for bad weight
or answer choice selections
![Page 22: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/22.jpg)
Computing the Weight 1
Question Response # Weight
High English A -.02
Grade English C 0
Goal Import. B .02
Understand Read D .03
Study Time A .01
Total Weight +.04
![Page 23: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/23.jpg)
Computing the Weight 2
Question Response # Weight
High English A -.02
Grade English F -.02
Goal Import. A -.01
Understand Read D .03
Study Time C -.01
Total Weight -.03
![Page 24: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/24.jpg)
Computation
• Score is multiplied by 1 plus the accumulated weight.– 85 * (1+.04) = 88.4– Placement will be based on a score of 88.
• Example 2– 85 * (1+ [-.03]) = 82.45– Placement will be based on a score of 82
![Page 25: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/25.jpg)
Common Errors
• Unequal weights between rules– E.G. A response has .01 weight in one rule and -.01
weight in the next rule• Misplaced Parentheses
– The multiple measures weights make the rule larger and more difficult to visualize
• Misuse of AND/OR• Misuse of arithmetic operators• Wrong answer choice in rule line
![Page 26: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/26.jpg)
Troubleshooting
• From the score report, determine what the student’s weight should be from the BGQ responses
• Using the weight, compute the weighted score
• Determine what the placement should be• Examine the appropriate rule for errors
![Page 27: Creating A Multiple Measures Placement System An Exercise With Ron Gordon & Armand Brunhoeber](https://reader038.vdocuments.us/reader038/viewer/2022103112/551aa1355503466b3a8b5704/html5/thumbnails/27.jpg)
Creating a Multiple Measures Placement System
An exercise with
Ron Gordon & Armand Brunhoeber
Thank you for not throwing things at the presenters