assessing professional and personal development in contemporary graduate education graham d. rowles,...
TRANSCRIPT
Assessing Professional Assessing Professional and Personal and Personal Development in Development in Contemporary Graduate Contemporary Graduate EducationEducation Graham D. Rowles, Monica M. Underwood, John Graham D. Rowles, Monica M. Underwood, John F. Watkins and Heidi H. EwenF. Watkins and Heidi H. EwenUniversity of Kentucky Graduate Center for GerontologyUniversity of Kentucky Graduate Center for Gerontology
Mary T. JohnsonMary T. JohnsonIndiana University School of MedicineIndiana University School of Medicine
Graham D. Rowles, Monica M. Underwood, John Graham D. Rowles, Monica M. Underwood, John F. Watkins and Heidi H. EwenF. Watkins and Heidi H. EwenUniversity of Kentucky Graduate Center for GerontologyUniversity of Kentucky Graduate Center for Gerontology
Mary T. JohnsonMary T. JohnsonIndiana University School of MedicineIndiana University School of Medicine
• ““Business as Usual” doesn’t satisfy Business as Usual” doesn’t satisfy the public expectation for the public expectation for compassionate professionals any compassionate professionals any longerlonger
• Learners are challenged to Learners are challenged to incorporate defined guiding principlesincorporate defined guiding principles
• Graduate programs are codifying Graduate programs are codifying these principles within formal student these principles within formal student assessment processesassessment processes
Our Sector of Education Our Sector of Education is Undergoing Major is Undergoing Major
ChangeChange
Old Social ContractOld Social Contract• Professionals were perceived as Professionals were perceived as
motivated by their service to othersmotivated by their service to others
• Society relied upon professional Society relied upon professional associations to set ethical and associations to set ethical and performance standards performance standards
• Colleagues can effectively Colleagues can effectively monitormonitor compliance to standards for practicing compliance to standards for practicing professionalsprofessionals
New PerceptionsNew Perceptions• The public is suspicious of professionals The public is suspicious of professionals
of all typesof all types
• Society relies on formal administrative Society relies on formal administrative rules and government regulations for rules and government regulations for control over professional behaviorscontrol over professional behaviors
• Litigation, lobbying, and attention to Litigation, lobbying, and attention to special interest groups have altered special interest groups have altered relationships with professionalsrelationships with professionals
Emerging ParadigmEmerging Paradigm• Peer awareness & assessment of Peer awareness & assessment of
self in relationship to othersself in relationship to others
• Continuing development at all levels Continuing development at all levels to enhance professional viability to enhance professional viability
• Programs that strive to inculcate Programs that strive to inculcate learners with professional standards learners with professional standards during trainingduring training
Professional Professional CharacteristicsCharacteristics
IU School of MedicineIU School of Medicine•5 Competencies Defined by 5 Competencies Defined by Medical Expertise Medical Expertise• 4 Competencies Related to4 Competencies Related to Personal Growth & Role Recognition Personal Growth & Role Recognition
UK Graduate Center for GerontologyUK Graduate Center for Gerontology•7 Dimensions of Intellectual 7 Dimensions of Intellectual & Professional Growth & Professional Growth•4 Dimensions of Personal Growth4 Dimensions of Personal Growth
Graduate Center forGerontology
Graduate Center forGerontology
From Admission to From Admission to Graduation and Graduation and BeyondBeyondAssessing Outcomes in Assessing Outcomes in Gerontology Doctoral Gerontology Doctoral EducationEducation
Graham D. Rowles, Ph.D.
Monica M. Underwood, B.S.
John F. Watkins, Ph.D.
Heidi H. Ewen, Ph.D.
University of Kentucky
Supported by University of Kentucky Quality Enhancement Program grant
CIP 30.1101; PCS 046
Graham D. Rowles, Ph.D.
Monica M. Underwood, B.S.
John F. Watkins, Ph.D.
Heidi H. Ewen, Ph.D.
University of Kentucky
Supported by University of Kentucky Quality Enhancement Program grant
CIP 30.1101; PCS 046 Graduate Center forGerontology
Doctoral Program DevelopmentDoctoral Program Development
19891989 University of Southern CaliforniaUniversity of Southern California19911991 University of Massachusetts, BostonUniversity of Massachusetts, Boston19941994 University of South FloridaUniversity of South Florida19971997 University of KentuckyUniversity of Kentucky19981998 University of KansasUniversity of Kansas20032003 University of Maryland Baltimore/ University of Maryland Baltimore/
University of Maryland Baltimore CountyUniversity of Maryland Baltimore County20042004 University of North TexasUniversity of North Texas20052005 Miami University of OhioMiami University of Ohio20052005 Purdue University (dual discipline Purdue University (dual discipline
degree) degree)
Graduate Center forGerontology
Diversity of EmphasisDiversity of Emphasis Interdisciplinary: theoretical and appliedInterdisciplinary: theoretical and applied
((University of Southern California)University of Southern California)Public Policy
((University of Massachusetts, Boston)University of Massachusetts, Boston)Health/Mental Health/Cognitive Aging/Public Health/Mental Health/Cognitive Aging/Public Policy Policy
((University of South Florida)University of South Florida) Interdisciplinary: Aging and HealthInterdisciplinary: Aging and Health
((University of Kentucky)University of Kentucky) Applied GerontologyApplied Gerontology
((University of North Texas)University of North Texas)Social and Behavioral GerontologySocial and Behavioral Gerontology
((University of Kansas)University of Kansas)(Miami University of Ohio)(Miami University of Ohio)
Epidemiology/Policy/PsychosocialEpidemiology/Policy/Psychosocial((University of Maryland Baltimore/ University of Maryland Baltimore/ University of Maryland Baltimore County)University of Maryland Baltimore County)
Dual Degree with DisciplineDual Degree with Discipline((Purdue UniversityPurdue University))
Graduate Center forGerontology
ContextContext
1. 1. Lack of evaluation strategies and tools Lack of evaluation strategies and tools for the for the dynamic assessment of student dynamic assessment of student progress progress through doctoral programs in through doctoral programs in gerontologygerontology
2. 2. Lack of information on gerontology Lack of information on gerontology doctoral doctoral program outcomesprogram outcomes
3. 3. Lack of mechanisms for assessing Lack of mechanisms for assessing overall overall doctoral program effectivenessdoctoral program effectiveness
Graduate Center for
Gerontology
The Quality Enhancement The Quality Enhancement ProjectProject
Objective
To develop a comprehensive tracking To develop a comprehensive tracking system and evaluation tool to provide system and evaluation tool to provide ongoing assessment of student progress and ongoing assessment of student progress and doctoral program effectiveness in moving doctoral program effectiveness in moving toward specified programmatic and personal toward specified programmatic and personal goals.goals.
Graduate Center forGerontology
The Quality Enhancement The Quality Enhancement ProjectProject
Elements
1. Comprehensive program data base and resource system.
2. Self rating assessment tool for students to dynamically monitor their own progress toward accomplishing specific professional and personal goals.
3. Protocol for faculty assessment of students.
4. System of interactive feedback among students and faculty. Graduate Center for
Gerontology
The Data SystemThe Data System Characteristics Characteristics
1.1. On-line system 24 hour accessOn-line system 24 hour access
2.2. Student profiles and portfolio Student profiles and portfolio (controlled by student)(controlled by student)
3.3. Assessment tool (completed by Assessment tool (completed by students with selective students with selective faculty access)faculty access)
4.4. Controlled Accessibility Controlled Accessibility and privacyand privacy
Graduate Center forGerontology
Graduate Center for
Gerontology
ScreenScreen
Sample:Sample:
Profile and Profile and Portfolio Portfolio PagesPages
ScreenScreen
Sample:Sample:
Profile and Profile and Portfolio Portfolio PagesPages
Dimensions of Dimensions of AssessmentAssessment
Professional Growth
Scholarly and Professional Independence
Critical and Integrative Thinking
Methodological Competence
Intellectual Risk Taking/Creativity
Gerontological Knowledge Base
Scholarly Communication
Identification with Gerontology
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment (Professional Growth)(Professional Growth)
1) Scholarly and Professional Independence
• I am able to express my own thoughts and ideas.
• I am able to conduct myself effectively in a professional environment.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree,
9=Don’t KnowGraduate Center for
Gerontology
Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)
2) Critical and Integrative Thinking
• I am confident in my ability to critically assess research design and methods.
• I feel confident integrating insights and ideas from diverse areas of research and scholarship.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know Graduate Center for
Gerontology
Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)
3) Methodological Competence
• I am confident in my ability to formulate useful research questions.
• I am competent working in a clinical environment (i.e. patient contact).
• I am confident in my ability to analyze and interpret data.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)
4) Intellectual Risk Taking/Creativity
• I am confident expressing alternative points of view in scholarly and professional settings.
• I am confident in extending theory through my own research activities.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)
5) Gerontological Knowledge Base
• I feel well-grounded in theory.
• I am well-versed in the current issues and problems of aging.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)
6) Scholarly Communication
• I am confident in my ability to orally present my research ideas and findings.
• I am confident in my ability to express my research ideas or findings in writing.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor
Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment(Professional Growth)(Professional Growth)
7) Identification with Gerontology
On the following scale, indicate the extent to which you identify yourself as a gerontologist.
10 – Identify myself as a gerontologist
9
8
7
6
5 – Not sure of my scholarly identity
44
33
22
11
00 – I identify myself as a disciplinarian – I identify myself as a disciplinarian
(biologist, sociologist, psychologist)(biologist, sociologist, psychologist) Graduate Center forGerontology
Dimensions of Dimensions of AssessmentAssessment
Personal Growth
Involvement/Commitment
Emotional Wellbeing/Stress Management
Physical Health
Time ManagementGraduate Center for
Gerontology
Dimensions of AssessmentDimensions of Assessment(Personal Growth)(Personal Growth)
1) Involvement/Commitment
• Engagement/involvement in departmental seminars, colloquiums, meetings and events is important to my development.
• I have opportunities to participate in the development of the program.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment(Personal Growth)(Personal Growth)
2) Emotional Well-Being/Stress Management
• I can adjust to the rigors of the doctoral program.
• I can maintain a sense of self as a participant in this program.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Dimensions of AssessmentDimensions of Assessment(Personal Growth)(Personal Growth)
3) Physical Health
• How is participating in the doctoral program affecting your health, if at all?
(1=very negatively, 2=negatively, 3=no effect, 4=positively5=very positively, 9=don’t know)
• How often do you experience symptoms of depression or anxiety?
(1=always, 2=often/frequently, 3=occasionally/infrequently, 4=never)
Graduate Center for
Dimensions of AssessmentDimensions of Assessment(Personal Growth)(Personal Growth)
4) Time Management
• I maintain adequate levels of leisure and recreation.
• I effectively manage the many responsibilities and competing demands of the program.
Scale: 1=Strongly Disagree, 2=Disagree, 3=Neither Agree Nor Disagree, 4=Agree, 5=Strongly Agree, 9=Don’t Know
Graduate Center forGerontology
Self Assessment Self Assessment ScheduleSchedule
1. Entry into program
2. End of first semester (completion of initial core)
3. End of first year
4. Completion of Integrative Seminar (usually following fourth semester)
5. Following qualifying examinations
6. Following doctoral dissertation defense
7. Two years post graduation Graduate Center forGerontology
Current StatusCurrent Status
1. Pilot tested with four students (a Sigma Phi Omega Initiative)
2. Students excited about potential for file sharing and collaborative work on projects
3. Students enthusiastic about vita building potential and flexibility.
Graduate Center forGerontology
Current StatusCurrent Status (cont.d) (cont.d)
4. Concerns about open faculty access to personal growth measures (particularly emotional well- being/stress management). Resolution: only advisor and Director of Graduate Studies (on a “need to know” basis) have access to personal growth measures.
5. System now fully on line as of September, 2006
6. Research using the system has begunGraduate Center for
Gerontology
Future DirectionsFuture Directions
1. Continued research on student progress, productivity and outcomes (IRB approved, initial analyses completed), and curriculum evaluation and refinement of program
2. Evaluation and refinement of system
3. Sharing of system with other doctoral programs to develop comprehensive
system of evaluation of progress in gerontological education.
Graduate Center forGerontology
Assessing Student Progress Assessing Student Progress in a New Medical Curriculum in a New Medical Curriculum Using Electronic PortfoliosUsing Electronic Portfolios
Mary T. Johnson, Ph.D. Mary T. Johnson, Ph.D. Indiana University School of MedicineIndiana University School of MedicineDirector of Problem Solving Competency Director of Problem Solving Competency
Medical CompetenciesMedical Competencies•Problem SolvingProblem Solving•Lifelong learningLifelong learning•Basic Clinical SkillsBasic Clinical Skills•Effective CommunicationEffective Communication•Scientific Basis of MedicineScientific Basis of Medicine
•Social and Community Contexts of Health CareSocial and Community Contexts of Health Care•Self-Care, Self-Awareness, & Personal GrowthSelf-Care, Self-Awareness, & Personal Growth•Moral Reasoning and Ethical JudgmentMoral Reasoning and Ethical Judgment•Professionalism & Role RecognitionProfessionalism & Role Recognition
• The time is ripe for Electronic The time is ripe for Electronic Portfolios in U.S. Medical EducationPortfolios in U.S. Medical Education
• European medical portfolio modelEuropean medical portfolio model
• ACGME Toolbox – Documentation of ACGME Toolbox – Documentation of learning for Residency Programslearning for Residency Programs
• US medical schools: educational needs US medical schools: educational needs similar to other professional disciplinessimilar to other professional disciplines
Portfolios in Portfolios in Medical EducationMedical Education
• Medical schools should share responsibility Medical schools should share responsibility with medical students for their learning with medical students for their learning outcomesoutcomes
• A better, more granular picture of how A better, more granular picture of how emerging professionals understand and emerging professionals understand and assume accountability for their own learning assume accountability for their own learning is neededis needed
• A vision should develop to facilitate student A vision should develop to facilitate student planning of learning choices and assembly planning of learning choices and assembly of evidence for their learning of evidence for their learning
School-Wide ConcernsSchool-Wide Concerns
The Indiana Statewide System for Medical Education
Electronic portfolios will be used soon to assess student
competency accomplishments
at all sitesSTATEWIDESTANDARDS
Integration of separate electronic systems for
competency assessment, course grading, and
verification of student clinical encounters
Student Perception:Student Perception:Where Is Competency Where Is Competency
Content?Content?
Medical Student Medical Student
Formative Learning Experiences Embedded in the Curriculum
Students Select Best Artifacts from Across Their Medical
Learning Experiences
Students Select Best Artifacts from Across Their Medical
Learning Experiences
Validated Artifacts
Validated Artifacts
Drivenby
CurriculumSelf-Selected
Examples
• Faculty Comments Imported• Grading Imported
• Cannot be ChangedAfter Selected
Medical Student Competency Portfolio
Medical Student Competency Portfolio
THE WORKING PORTFOLIO:An Archive of
Learner Artifacts
THE WORKING PORTFOLIO:An Archive of
Learner Artifacts
Student Competency Reflections
SCIENTIFIC BASIS OF MEDICINE – Feb 27, 2006 @ 2:45pm
“The ELISA Assay is a very common and important laboratory test. I have found the
knowledge that I gained from researching this test to be invaluable. I also appreciate
opportunities to present to a group so that I can practice my communication skills.”
Competency Content:Competency Content:Self Selected ExamplesSelf Selected Examples
What are tensions between What are tensions between portfolio uses, and how can they portfolio uses, and how can they be resolved?be resolved?
Faculty Must Engage Faculty Must Engage in Discussion About in Discussion About
Best PracticesBest Practices
– MetaphorMetaphor: : Narrative medicine - “our stories” Narrative medicine - “our stories” vsvsTraditional medicine - “just the Traditional medicine - “just the facts”facts”
– Locus of controlLocus of control: : Curriculum-Driven vs Curriculum-Driven vs Learner-Defined ContentLearner-Defined Content
– PurposePurpose: accountability vs wonder: accountability vs wonder
• Course Level Student AssessmentCourse Level Student Assessment
• Clerkship Level Student AssessmentClerkship Level Student Assessment
• Competency Level Student AssessmentCompetency Level Student Assessment
• Program EvaluationProgram Evaluation
• Individual Residency Advisement & Individual Residency Advisement & Dean’s Letter PreparationDean’s Letter Preparation
• Liason Committee on Medical Education Liason Committee on Medical Education (LCME) Accreditation Documentation(LCME) Accreditation Documentation
Potential Portfolio Potential Portfolio ApplicationsApplications
• AssurancesAssurances – licensure examinations, traditional – licensure examinations, traditional checkpoints for specific accountability (USMLE: checkpoints for specific accountability (USMLE: Step 1 before Year 3, Step 2 prior to graduation) Step 1 before Year 3, Step 2 prior to graduation)
• CertificationsCertifications – BLS, ACLS, CME, Match to – BLS, ACLS, CME, Match to Residency Program & USMLE Step 3Residency Program & USMLE Step 3
• BalanceBalance – Accomplishment at Three Levels of – Accomplishment at Three Levels of Medical Competency; Integration across DisciplinesMedical Competency; Integration across Disciplines
• Reflection ConnectionsReflection Connections – an additional – an additional demonstration of the compassionate, caring demonstration of the compassionate, caring dimension of medical training dimension of medical training
Graduating Competent Graduating Competent PhysiciansPhysicians
Concluding ObservationsConcluding Observations
Graduate Center forGerontology
•Evolving Central Assessment Evolving Central Assessment GoalGoal
•Balancing Benefits and Balancing Benefits and BurdensBurdens
•Compliance IssuesCompliance Issues
•Confidentiality and Control Confidentiality and Control IssuesIssues•Relating Academic and Relating Academic and Professional Growth to Life Professional Growth to Life as a Wholeas a Whole
•Evolving Central Assessment Evolving Central Assessment GoalGoal
•Balancing Benefits and Balancing Benefits and BurdensBurdens
•Compliance IssuesCompliance Issues
•Confidentiality and Control Confidentiality and Control IssuesIssues•Relating Academic and Relating Academic and Professional Growth to Life Professional Growth to Life as a Wholeas a Whole
For further information For further information contact:contact:
• Mary T. Johnson, Ph.D.Mary T. Johnson, Ph.D.
Associate Professor of Microbiology Associate Professor of Microbiology and Immunologyand Immunology
Indiana University School of Indiana University School of MedicineMedicine
Statewide Competency Director for Statewide Competency Director for Problem SolvingProblem Solving
[email protected]@iupui.edu
• Graham D. Rowles, Ph.D.Graham D. Rowles, Ph.D.
Director and ProfessorDirector and Professor
Graduate Center for GerontologyGraduate Center for Gerontology
University of KentuckyUniversity of Kentucky
[email protected]@uky.edu
• John F. Watkins, Ph.D.John F. Watkins, Ph.D.
Associate ProfessorAssociate Professor
Graduate Center for Graduate Center for GerontologyGerontology
University of KentuckyUniversity of Kentucky
[email protected]@uky.edu
• Heidi H. Ewen, Ph.D.Heidi H. Ewen, Ph.D.
Post-doctoral ScholarPost-doctoral Scholar
Graduate Center for Graduate Center for GerontologyGerontology
University of KentuckyUniversity of Kentucky
[email protected]@uky.edu