Cognitive and Academic AssessmentDr. K. A. KorbUniversity of Jos

OutlineClassroom AssessmentAchievement TestingIntelligence TestingGeneral Issues in Psychological TestingReliabilityValidityDr. K. A. KorbUniversity of Jos

Classroom AssessmentPurpose of classroom assessmentProvide information to students, parents, teachers, and othersProvide information to adapt instructional practicesIncrease learningEnhance motivationGrades should measure achievement in the specific classGrades should NOT measure:Achievement in the specific classEffortAbilityInterestsAttitudeDegree of corruptionDr. K. A. KorbUniversity of Jos

Classroom AssessmentGrading system should be:Clear and understandableDesigned to support learning and provide frequent feedbackBased on hard dataFair to all studentsDefendable to students, parents, and administrators

Dr. K. A. KorbUniversity of Jos

Classroom AssessmentComponents of grading systemFormative Evaluation: Evaluation before or during instruction to provide feedback to the teacher/studentSummative: Evaluation after instruction for grading purposesDr. K. A. KorbUniversity of Jos

Tips for Writing ExamsThe purpose of an exam is assess how well the student understands the course contentCritically read questions from the students perspective to determine if the question is understandable.Rotate exam questions between terms so students cannot get access to the questions prior to the exam.Students have to study more of the material and think deeper about the contentAsk questions that test Blooms higher level of thinking Assess understanding, not memorizationCheating becomes more difficult

Dr. K. A. KorbUniversity of Jos

Multiple Choice ItemsStrengthsAble to assess different levels of thinkingReliable scoringEasy to gradeWeaknessesUnable to give credit for partial knowledgeDecontextualized

Dr. K. A. KorbUniversity of Jos

Multiple Choice ItemsMartha talks to the person sitting next to her while the teacher is giving instruction. To make sure that Martha does not talk in class next time, the teacher makes Martha write an essay on why paying attention to the teacher is important. This is an example of:Positive ReinforcementNegative ReinforcementPositive PunishmentNegative PunishmentDr. K. A. KorbUniversity of Jos

Multiple Choice ItemsComponentsStem: Question/ProblemDetermines level of knowledge assessedCorrect answerDistracters: Wrong answersContain likely misconceptionsDr. K. A. KorbUniversity of Jos

Multiple Choice ItemsGuidelines for preparing Multiple Choice ItemsPresent one clear problem in the stemMake all distracters plausibleAvoid similar wording in the stem and correct choiceKeep the correct answer and distracters similar in lengthAvoid absolute terms (always, never) in incorrect choicesKeep stem and distracters grammatically consistentAvoid using two distracters with the same meaningEmphasize NEGATIVE wordingUse None of the above with care and avoid All of the aboveDr. K. A. KorbUniversity of Jos

Essay ItemsStrengthsAssess creative and critical thinkingStudents more likely to meaningfully organize information when studyingWeaknessesScoring takes timeScoring can be unreliableDr. K. A. KorbUniversity of Jos

Essay ItemsRubric: Scoring scale that describes criteria for gradingEstablish criteria for credit based on critical element of essay According to the Theory of planned Behavior, what are the four major factor that influence the relationship between attitudes and behavior?2 points apiece, 1 points the name and 1 points for the explanationBehavioral intentions: Cognitive representation of the readiness to perform a behaviorSpecific attitude towards behavior: Whether like or dislikeSubject norms: Belief about how significant others view behaviorPerceived behavior control: Perception about the ability to perform the behaviorDr. K. A. KorbUniversity of Jos

Essay ItemsScoring essay itemsRequire students to answer each itemPrepare rubric in advanceWrite a model answer for each item and compare a few students responses to model to determine if score adjustments are needed Score all students answers to one essay question before moving to the next essay questionScore all responses to single item in one sittingScore answers without knowing the identity of the studentDr. K. A. KorbUniversity of Jos

Intelligence TestingAchievement Test: Instrument created to assess developed skills or knowledge in a specific domainPurpose of standardized achievement testing is to place students in appropriate education environmentIntelligence Test: Ability to perform on cognitive tasksSample performance on a variety of cognitive tasks and then compare performance to others of a similar developmental levelPurpose of Intelligence TestingDiagnose students with special needsLearning DisabilitiesTalented and GiftedPlace students in appropriate educational environmentsEducational Research

Dr. K. A. KorbUniversity of Jos

Intelligence TestingExample: Stanford Binet Intelligence Scale: Fourth EditionVerbal ReasoningVisual ReasoningQuantitative ReasoningShort-Term MemorySubtest Score and General ScoreDr. K. A. KorbUniversity of Jos

Reliability: Consistency of resultsReliableReliableUnreliableDr. K. A. KorbUniversity of Jos

Reliability TheoryActual score on test = True score + ErrorTrue Score: Hypothetical actual score on testThe reliability coefficient indicates the ratio between the true score variance on the test and the total varianceIn other words, as the error in testing decreases, the reliability increasesDr. K. A. KorbUniversity of Jos

Reliability: Sources of ErrorError in Test constructionError in Item Sampling: Results from items that measure more than one construct in the same testError in Test AdministrationTest environment: Room temperature, amount of light, noiseTest-taker variables: Illness, amount of sleep, test anxiety, exam malpracticeExaminer-related variables: Absence of examiner, examiners demeanorError in Test ScoringScorer: With subjectively marked assessments, different scorers may give different scores to the same responsesDr. K. A. KorbUniversity of Jos

Reliability: Error due to Test ConstructionSplit-Half Reliability: Determines how consistently the measure assesses the construct of interest.A low split-half reliability indicates poor test construction.An instrument with a low split-half reliability is probably measuring more constructs than it was designed to measureCalculate split-half reliability with Coefficient AlphaDr. K. A. KorbUniversity of Jos

Reliability: Error due to Test AdministrationTest-Retest Reliability: Determines how much error in a test score is due to problems with test administration.Administer the same test to the same participants on two different occasions. Correlate the test scores of the two administrations using Pearsons Product Moment Correlation.Dr. K. A. KorbUniversity of Jos

Reliability: Error due to Test Construction with Two Forms of the Same MeasureParallel Forms Reliability: Determines the similarity of two different versions of the same measure.Administer the two tests to the same participants within a short period of time. Correlate the test scores of the two tests using Pearsons Product Moment Correlation.

Dr. K. A. KorbUniversity of Jos

Reliability: Error due to Test ScoringInter-Rater Reliability: Determines how closely two different raters mark the assessment.Give the exact same test results from one test administration to two different raters.Correlate the two markings from the different raters using Pearsons Product Moment CorrelationDr. K. A. KorbUniversity of Jos

Validity: Measuring what is supposed to be measuredValidInvalidInvalidDr. K. A. KorbUniversity of Jos

ValidityThree types of validity:Construct validity: Measure the appropriate psychological constructCriterion validity: Predict appropriate outcomesContent validity: Adequate sample of contentEach type of validity should be established for all psychological tests.Dr. K. A. KorbUniversity of Jos

Construct ValidityConstruct Validity: Appropriateness of inferences drawn from test scores regarding an individuals status of the psychological construct of interestTwo considerations:Construct underrepresentationConstruct irrelevant variance Dr. K. A. KorbUniversity of Jos

Construct ValidityConstruct underrepresentation: A test does not measure all of the important aspects of the construct.Content ValidityConstruct-irrelevant variance: Test scores are affected by other unrelated processesDr. K. A. KorbUniversity of Jos

Sources of Construct Validity EvidenceHomogeneity: The test measures a single constructEvidence: High internal consistencyConvergence: Test is related to other measures of the same construct and related constructs Evidence: Criterion ValidityTheory: The test behaves according to theoretical propositions about the constructEvidence by changes in test scores according to age: Scores on the measure should change by age as predicted by theory. Evidence from treatments: Scores on the measure change as predicted by theory from a treatment between pretest and posttest. Dr. K. A. KorbUniversity of Jos

Criterion ValidityCriterion Validity: Correlation between the measure and a criterion.Criterion: Other accepted measures of the construct or measures of other constructs similar in nature.A criterion can consist of any standard with which your test should be relatedExamples:BehaviorOther test scoresRatingsPsychiatric diagnosis

Dr. K. A. KorbUniversity of Jos

Criterion ValidityThree types: Convergent validity: High correlations with measures of similar constructs taken at the same time.Divergent validity: Low correlations with measures of different constructs taken at the same time.Predictive validity: High correlation with a criterion in the futureDr. K. A. KorbUniversity of Jos

Criterion Validity Example: An essay test of science reasoning was developed to admit students into the science program at the university.Convergent Validity: High correlations with other science tests, particularly well established science tests.Divergent Validity: Low correlations with measures of writing ability because the test should only measure science reasoning, not writing ability.Predictive Validity: High correlations with future grades in science courses because the purpose of the test is to determine who will do well in the science program at the university.Dr. K. A. KorbUniversity of Jos

Criterion Validity ExampleHigh correlations with other measures of science ability indicates good criterion validity.Low correlations with measures unrelated to science ability indicates good criterion validity.High correlation with future measures of science ability indicates good criterion validity.Dr. K. A. KorbUniversity of Jos

Criterion Validity Evidence for New Science Reasoning Test:Correlations between Science Reasoning and Other MeasuresNew Science Reasoning TestWAEC Science Scores.83School Science Marks.75WAEC Writing scores.34WAEC Reading Scores.24Future marks in university science courses.65

Content ValidityContent Validity: Sampling the entire domain of the construct it was designed to measureDr. K. A. KorbUniversity of Jos

Chart1

50

30

19

1

Test Coverage

Sheet1

AdditionSubtractionMultiplicationDivision

5030191

Chart1

25

25

25

25

Class Coverage

Sheet1

AdditionSubtractionMultiplicationDivision

25252525

Content ValidityDr. K. A. KorbUniversity of Jos

Content ValidityTo assess:Gather a panel of judgesGive the judges a table of specifications of the amount of content covered in the domainGive the judges the measureJudges draw a conclusion as to whether the proportion of content covered on the test matches the proportion of content in the domain.Dr. K. A. KorbUniversity of Jos

Face ValidityFace validity: Addresses whether the test appears to measure what it purports to measure.To assess: Ask test users and test takers to evaluate whether the test appears to measure the construct of interest.Face validity is rarely of interest to test developers and test users.The only instance where face validity is of interest is to instill confidence in test takers that the test is worthwhile.Face validity CANNOT be used to determine the actual interpretive validity of a test.Dr. K. A. KorbUniversity of Jos

Concluding AdviceThe best way to determine that the measures you use are both reliable and valid is to use a measure that another researcher has developed and validatedThis will assist you in three ways:You can confidently report that you have accurately measured the variables in the study.By using a measure that has been used before, your study is intimately tied to previous research that has been conducted in your field, an important consideration in determining the importance of your study. It saves you time and energy in developing your measureDr. K. A. KorbUniversity of Jos

RevisionWhat are the purposes of classroom assessment, achievement testing, and intelligence testing?Compare and contrast the strengths and weaknesses of multiple choice and essay testsDescribe the three sources of error that contribute to lowering the reliability of an instrument. How can the reliability coefficient be calculated for each source of error?What are the three types of validity evidence required for psychological measurement? How can each type be assessed?Dr. K. A. KorbUniversity of Jos

*******What sources of error?**Criterion: DL test, ACT/SAT