Assessing the Assessing the Development of Critical Development of Critical

Thinking Skills in Thinking Skills in Undergraduate Undergraduate

Psychology StudentsPsychology Students

Phil Dunwoody & Mark McKellop

Partially funded by grants from the Teagle Foundation, the Council of Independent Colleges/Collegiate Learning Assessment Consortium, and the James J. Lakso Center for the

Scholarship of Teaching & Learning at Juniata College.

What is Critical Thinking?What is Critical Thinking?

What is Critical Thinking?What is Critical Thinking?

American Philosophical Association’s “Delphi Report”

(Facione, 1990)

CT is “purposeful, self-regulatory judgment which results in interpretation, analysis, evaluation, and inference, as well as explanation of the evidential, conceptual, methodological, criteriological, or contextual considerations upon which that judgment is based.”

What is Critical Thinking?What is Critical Thinking?

CT is multidimensional construct – it is both a disposition & a learned skill

(Halpern, 1998, 2003)

CT has general & discipline-specific aspects with limited generalizability

(Lehman & Nisbett, 1990; Toplak & Stanovich, 2002; Williams et al., 2004)

Psychological Critical Thinking (PCT)Psychological Critical Thinking (PCT)

American Psychological Association“Guidelines for the Undergraduate

Psychology Major” (APA, 2007)

◦Goal 3: Critical Thinking Skills in Psychology. Students will respect and use critical and creative

thinking, skeptical inquiry, and, when possible, the scientific approach to solve problems related to behavior and mental processes.

Documenting & Assessing our Teaching◦Departmental Assessment Plan◦Mapping our POE

Psychological Critical ThinkingPsychological Critical Thinking

PCT is “the ability to judge the plausibility of specific assertions, to weigh evidence, to assess the logical soundness of inferences, to construct counter-arguments and alternative hypotheses.” (Nickerson et al, 1985)

CT skills in psychology include:◦ Argument analysis and evaluation◦ Methodological reasoning◦ Statistical reasoning◦ Causal reasoning◦ Focusing and clarifying questions (Bensley & Murtaugh, 2012)

Psychological Critical Thinking Psychological Critical Thinking Inventory (PCTI)Inventory (PCTI)

Our assessment/measure of PCT will◦be relatively brief◦be (relatively) easily scored◦evaluate “applied” critical thinking skills (can one “think like a behavioral scientist?”)

PCTI focuses on following aspects of PCT: Appropriate evaluation of expert behavioral research

(i.e., peer-reviewed journal articles) Appropriate evaluation of non-expert behavioral claims

(i.e., media reports) Identifying reasoning faults in faux-novice designed

behavioral research

Validating the PCTIValidating the PCTI

California Critical Thinking Skills TestWatson-Glaser Critical Thinking AppraisalLawson’s Psychological Critical Thinking

ExamCollegiate Learning AssessmentPsychology Area Concentration

Achievement Test (PACAT)National Survey of Student Engagement

(NSSE)

ParticipantsParticipants

Participants◦Intro Psych: 96 (Spring, 2011)◦Senior Psych Capstone: 39 (Spring & Fall, 2011

and Spring, 2012)◦Pre and Post Research Methods: 29 (Fall, 2011

and Spring, 2012) Pre and Post with CAT currently

Descriptive Stats PCTIDescriptive Stats PCTI

Popular Press◦Possible range: 0-10 points◦Observed range=2-8, Mean=5.9, SD=1.7

Critique Faux-Novice Study◦Possible range: 0-7 points◦Observed range= 0-6, Mean=2.9, SD=1.3

Peer-review Articles◦Possible range: 0-7 points◦Observed range= 0-6, Mean=3.28, SD=1.3

Total PCTI◦Possible range: 0-24◦Observed range= 6-20.5, Mean=12.1, SD=2.7

Hypothesis 1Hypothesis 1

Graduating psychology seniors will score higher on PCTI than freshman in Introductory Psychology.

Popular Press◦ Intro: M=5.19, SD=1.48◦ Capstone: M=6.55, SD=1.99◦ Significant: t(91)=-3.72, p<0.001, d=0.78

Critique Faux-Novice Study◦ Intro: M=2.48, SD=1.19◦ Capstone: M=3.88, SD=1.26◦ Significant: t(91)=-5.27, p<0.001, d=1.14

Peer-review Articles◦ Intro: M=3.03, SD=1.37◦ Capstone: M=3.34, SD=1.45◦ Significant: t(91)=-1.02, p=0.31

Total PCTI◦ Intro: M=10.70, SD=2.34◦ Capstone: M=13.77, SD=2.76◦ Significant: t(91)=-5.64, p<0.001, d=1.20

Hypothesis 2Hypothesis 2

Students in Research Methods should score higher at the end of the semester than at the beginning of the semester (alternate form).

Popular Press◦ Pre: M=6.52, SD=1.46◦ Post: M=7.30, SD=1.32◦ Significant: t(26)=-2.09, p=0.05, d=0.56

Critique Faux-Novice Study◦ Pre: M=3.26, SD=1.10◦ Post: M=3.81, SD=1.27◦ Significant: t(26)=-1.86, p=0.07, d=0.46

Peer-review Articles◦ Pre: M=3.67, SD=1.30◦ Post: M=3.93, SD=1.75◦ Significant: t(26)=-0.55, p=0.59

Total PCTI◦ Pre: M=13.44, SD=2.68◦ Post: M=15.04, SD=2.49◦ Significant: t(26)=-2.39, p=0.02, d=0.62

PCTI Convergent ValidityPCTI Convergent Validity

PCTI with measures of generic PCTI with measures of generic (abstract) critical thinking(abstract) critical thinking

PCTI with Lawson’s Psychological PCTI with Lawson’s Psychological Critical Thinking ExamCritical Thinking Exam

PCTI with measures of educational PCTI with measures of educational achievementachievement

PCTI with measures of Psychology PCTI with measures of Psychology Knowledge (PACAT)Knowledge (PACAT)

ConclusionConclusion

Our PCTI has good face validity and shows clear differences not only between Freshman and Seniors, but also within one course.

The PCTI does not correlate much with more traditional abstract measures of CT.

The PCTI clearly correlates with the psychology knowledge-based PACAT.

The PCTI also clearly correlates with general measures of academic achievement such as GPA, SAT V, and SAT M.

Intelligence tests also correlate with these same measures.

So…

Is critical thinking just another Is critical thinking just another term for intelligence? term for intelligence?

"[Intelligence is] the aggregate, or global capacity to act purposefully, think rationally, and deal effectively with the environment. It is global because it characterizes the individual's behavior as a whole; it is an aggregate because it is composed of elements or abilities which, though not entirely independent, are qualitatively differentiable.”

◦David Wechsler, 1938

ReferencesReferences

American Psychological Association. (2007). APA guidelines for the undergraduate psychology major. Washington, DC: Author. Retrieved from www.apa.org/ed/resources.html

Bensley, D. A., & Murtagh, M. P. (2012). Guidelines for a scientific approach to critical thinking assessment. Teaching of Psychology, 39, 5-16.

Facione, P. A. (1990). Critical thinking: A statement of expert consensus for purposes of educational assessment and instruction. Millbrae, CA: The California Academic Press.

Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: Dispositions, skills, structure training, and metacognitive monitoring. American Psychologist, 53, 449–455.

Halpern, D. F. (2003). Thought and knowledge: An introduction to critical thinking. Mahwah, NJ: Erlbaum.

Lehman, D. R., & Nisbett, R. E. (1990). A longitudinal study of the effects of undergraduate training on reasoning. Developmental Psychology, 26, 952-960.

Nickerson, R. S., Perkins, D. N., & Smith, E. E. (1985). The teaching of thinking. Hillsdale, NJ: Lawrence Erlbaum.

Toplak, M. E., & Stanovich, K. E. (2002). The domain specificity and generality of disjunctive reasoning: Searching for a generalizable critical thinking skill. Journal of Educational Psychology, 94, 197-209.

Williams, R. L., Oliver, R, & Stockdale, S. (2004). Psychological versus generic critical thinking as predictors and outcome measures in a large undergraduate human development course. The Journal of General Education, 53, 37-58.