2014  PTLC  Research  Projects    Index  of  2014  PTLC  Research  Projects    Krista Estes University of Colorado, Anschutz Medical Campus

Shaun Ellen Gleason University of Colorado, Anschutz Medical Campus Jeannette Guerrasio University of Colorado, Anschutz Medical Campus Joel Marrs University of Colorado, Anschutz Medical Campus David Rickels University of Colorado, Boulder Darcy Solanyk University of Colorado, Anschutz Medical Campus Darlene Tad-y University of Colorado, Anschutz Medical Campus

Krista Estes, DNP, FNP-C Assistant Professor University of Colorado Anschutz Medical Campus College of Nursing Division of Women, Children and Family Health 13120 E. 19th Ave. Mail Stop C288-18, ED2N, Room 4310 Aurora, CO 80045 Krista.Estes@UCDenver.edu

Breaking Down the Silos: Fostering an Interdisciplinary Approach to Learning Advanced Physical Assessment

A. What is the central question, issue or problem you plan to explore in your proposed work? PICO Questions: Do graduate nursing students in an interdisciplinary Advanced Physical Assessment class with first year medical students compared to students in a traditional Advanced Physical Assessment class score a higher mean on the Standardized Physical Assessment Teaching Associates Final? What are the students’ perceptions and attitudes of interdisciplinary education before and after an integrated Advanced Physical Assessment course?

The Anschutz Medical Campus (AMC) was built with the inspiration and ambition of providing a rich interdisciplinary environment. In the past years, this has come to fruition through the implementation of multiple interprofessional educational (IPE) programs. While these programs have greatly enriched the students’ education, there are minimal IPE classroom experiences with college of nursing (CON) graduate students. Students in various disciplines are taking many of the same courses, but in silos. An example is Advanced Physical Assessment. This course is taken by both the graduate nursing students and the undergraduate medical students with the same Standardized Physical Exam Teaching Associates (SPETAs), in the same Center for Advancing Professional Excellence (CAPE) lab, with many of the same teaching standards. By providing a shared IPE classroom environment among these two disciplines, there is the potential to enhance the students’ education through learning mutual planning, decision making and goal setting. Therefore, the silos can begin to be broken down.

B. Why is this central question, issue or problem important to you and to others who might benefit from or build on your findings?

The delivery of healthcare in American has become complex and as a result, there is role differentiation and specialization. While there are multiple disciplines in healthcare, there is one commonly shared goal: improvement of patient outcomes. To meet this goal, it is important for there to be mutual respect and understanding among all disciplines. There should not be competition, but collaboration. This begins with a shared educational experience.

In the Fall 2012 semester, Dr. Marylou Robinson implemented a pilot project integrating 1 CON graduate student with 3 students from the School of Medicine (SOM) in the CAPE. These students used teamwork to learn advanced physical assessment skills on SPETAs. Using the Readiness for Inter-Professional Learning (RIPL) survey, attitudes of student’s about IPE pre and post event were measured. It was through Dr. Robinson’s vision of transforming teaching and learning in discipline specific silos to a more integrated and shared experience that Dr. Estes became very interested in IPE education.

The idea of IPE education is not new. For years programs have tried to implement IPE experiences, but many fail or continued implementation is given up due to its daunting nature. The American Association of Colleges of Nursing recommends the development of curricula that incorporate opportunities for graduate nursing students to work together with other disciplines and recognize shared content and clinical experiences. The role of the educational process in IPE in comparison to traditional methods is not well studied; therefore, continued implementation of this IPE experience on a larger scale while evaluating methods to maximize the educational experience and educational outcomes is essential. Our goal is to build upon Dr. Robinsons’ PTLC 2012-2013 grant by implementing a similar study, but on a much larger scale and with more comparison of traditional learning versus the IPE experience.

C. How do you plan to conduct your investigation? What sources of evidence do you plan to examine? What methods might you employ to gather and make sense of this evidence? Dr. Robinsons’ 2012-2013 study involved 11 volunteer graduate RN students of NURS 6761 Advanced Physical Assessment with 33 newly admitted SOM students in learning IPE Advanced Physical Assessment. For 2013-2014, the goal is to incorporate entire student cohorts of 60 graduate nursing students with the estimated 166 SOM students. Groups will consist of 2 graduate nursing students with 2 SOM students for a more discipline balanced IPE experience. Since the SOM begins their academic year 10 days prior to the CON academic schedule, the first CAPE lab

session will be used to orient the medical students to basic vital signs. The graduate nursing students will join medical students for the second day of class and remain with them for the remaining seven weeks of SPETA sessions. After the completion of their sessions, both student cohorts will take the same SPETA physical assessment final. CON student grades will be compared retrospectively to a semester with the traditional learning format.

Prior to and after the IPE experience, the RIPL survey will be distributed to the CON students. At the end of the IPE experience, the PI and SOM faculty will conduct focus groups to gain additional insight. The focus groups will get qualitative analysis using thematic analysis. Data from the surveys will be processed using SPSS software with descriptive and inferential statistics for analysis. Data results will be reviewed by the CAPE director, SOM and CON faculty to identify areas of strengths and weakness, as well as ways to improve the IPE learning experience for students in the future. Dr. Robinsons’ initial pilot went through expedited COMIRB so that information gathered within the pilot grant is accessible in this second study. Expedited COMIRB is planned for continuation of this project in the future.

D. How might you make your work valuable to others in ways that facilitate scholarly critique and review, and that contribute to thought and practice beyond the local? The University of Colorado is known to be a pioneer not only through research, but also education. This IPE experience would extend IPE education at the AMC. Its outcomes will provide a valuable framework and example of a trial model of IPE education for various healthcare disciplines and programs of education to evaluate for possible development and implementation. The process, outcomes, limitations, and potential educational benefits related to IPE education can be further explored. This study would be made available to others through dissemination of findings at the university, regional, and national level. Poster and podium presentations as well as publication in a respected, peer-reviewed journal will facilitate this.

E. Include a literature review of the theory and effective teaching practice of the subject of your inquiry in order to locate your research in the literature preceding it. The Institute of Medicine (2010), calls for nurses to be full partners with other health care professionals in redesigning health care in the United States. It emphasizes IPE learning to reduce errors and encourages pre-graduation learning to foster communication and develop respect for the

strengths of each discipline. A 2001 literature review examining IPE in healthcare identified that most of the existing literature evaluated academic activities and failed to compare IPE with traditional methods of learning (Hall & Weaver, 2001). Since, there continues to be minimal literature comparing these learning methods in the health care field. Dillon, Noble, and Kaplan (2009) evaluated IPE collaboration with nursing and medical students using clinical simulation and studied the students’ perceptions of the learning experience. As a result of the IPE simulation exercise, a significant difference was found in the medical students' posttest scores for collaboration and nursing autonomy. The nurses also viewed the nurse-physician relationship became more collaborative after the experience. Both cohorts felt that the experience should be continued.

F. What is your record of innovations in teaching and/or the assessment of learning? Dr. Estes has always been an advocate for innovative teaching and learning. Six months after graduation from her family nurse practitioner program, she started a program of student integration for clinical rotations in the emergency department. Although she had limited clinical experience, it was important for students to observe how a practicing NP used resources to answer difficult clinical questions. Student evaluations of this clinical experience were excellent.

Upon becoming a faculty member a little over a year ago, Dr. Estes has served as a fresh viewpoint on old practices. In the courses she has been assigned to teach, she has made changes to improve the educational and learning experience of students. For example, in teaching Advanced Pharmacology and talking with faculty who have taught the course in the past, she identified that students have a difficult time learning to write prescriptions. As a result, she and another faculty member are in the process of developing an innovative educational module based on commonly identified prescribing errors to assist graduate students in learning to write prescriptions. The literature review and research study plan has been presented at the 24th Annual Rocky Mountain Interdisciplinary Research & Evidence-Based Practice Symposium, Collaborating to Improve Practice through Research & Evidence-Based Practice.

Dr. Robinson has held two PTLC grants 2011-2012 and 2012-2013 at the AMC. As noted before, she has pioneered this effort of IPE with the SOM. She has enlisted the endorsement of this idea from the Director and Assistant Director of the CAPE, and the Foundations of Doctoring faculty. Her role will be to mentor Dr. Estes in expanding the pilot work as

well as mentoring her in the PTLC process.

G. Are you able to attend the required meetings? Both investigators are able to attend the “required” meetings, although attendance to these meetings is seen more as being a privilege than a requirement. The benefits of these meetings are numerous. We really look forward to learning about the research projects of other PTLC scholars as well as presenting our research and receiving feedback from peers.

H. Name and contact information for a mentor/Coach for your project Dr. Marylou Robinson, PhD, FNP-C, Assistant Professor at the University Of Colorado College Of Nursing, will serve as mentor. She is a 2011 and 2012 PTLC scholar and has a real heart for implementing IPE education. Address: University of Colorado Anschutz Medical Campus, College of Nursing, Division of Adult and Elder Health, 13120 E. 19th Ave Room 4117, Mail Stop C288-19, Aurora, CO 80045; Phone: 303-724-8564, Email: Marylou.robinson@ucdenver.edu

I. If your project is selected, will you serve as a coach in the PTLC in a future year? Yes, both would be honored to serve as a coach in the future. References

Dillon, P., Noble, K. & Kaplan, L. (2009). Simulation as a Means to Foster Collaborative Interdisciplinary Education. Nursing Education Perspectives, 30(2), 87-90.

Hall, P. & Weaver, L. (2001). Interdisciplinary education and teamwork: a long and winding Road. Medical Education, 35(9), 867-75.

Institute of Medicine (2010). The Future of Nursing: Leading Change, Advancing Health. Retrieved from http://www.iom.edu/Reports/2010/The-Future-of-Nursing-Leading-Change-Advancing-Health.aspx

Robinson, M. Pilot for Inter-Professional Learning of Physical Assessment Skills. Podium presentation at the Western Institute of Nursing Conference in Anaheim, CA, April 13, 2013.

Shaun Ellen Gleason, PharmD, MGS Director, Distance Degrees and Programs and Assistant Professor University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences Department of Clinical Pharmacy 12850 E. Montview Blvd. Room V20-1116R Mailstop C-238 Aurora, CO 80045 303-724-3548 shaun.gleason@ucdenver.edu

Implementation of Faculty-Guided Self-Directed Learning (SDL) Using Innovative Technology as a Single Lesson Within a Multi-Instructor Traditional-Delivery Course

a. What is the central question, issue, or problem you plan to explore in your proposed work? What is the impact on student learning and reaction to learning when faculty-guided self-directed learning (SDL) is implemented in a single lecture of a multi-instructor, traditionally-delivered course? In particular, how do specific SDL implementation factors, such as technology, and student and instructor involvement impact outcomes when SDL is implemented in this setting, where it is an outlier in educational methodology used throughout the remainder of the course?

b. Why is your central question, issue, or problem important to you and to others who might benefit from or build on your findings? Many educational programs, including those within our school, may need to redesign their educational methodology for various reasons, such as reduced budgets leading to reduced classroom time, and the growing need to implement new technology to meet the expectation of today’s students. Additionally, many healthcare professions, including pharmacy, are expecting practitioners to direct their own continuing professional development.3 One method of addressing these needs is through SDL, with expert guidance of the faculty member when used within a curriculum. A concern when implementing new methodologies, such as SDL, within a curriculum, is the effect on students when a different or unexpected method is used, especially as the sole offering within a course where all other lectures are delivered in a traditional or expected manner.

By providing evidence on the impact to student learning and reaction to learning with varying methods of SDL implementation, ranging from complete SDL to traditional delivery, my proposal will provide my school and other educators with information on factors to consider when implementing SDL and if it is reasonable to do so within a traditionally delivered course.

c. How do you plan to conduct your investigation? What sources of evidence do you plan to examine? What methods might you employ to gather and make sense of this evidence? What literature have you reviewed on your topic?: I will assess the varying implementation methods of faculty-guided SDL in a single lecture of a traditionally-delivered course, ranging from complete SDL with low technology to traditional delivery. The methods of SDL delivery have varied across the years, with each year providing components of the methodology, or none at all, to contrast and compare. Four of the varying methods of SDL delivery have already taken place (2009 – 2012), on an annual basis over the past four years of the same class. The fifth variation is upcoming this fall 2013 semester, in the same class, on the same content. The constants throughout these varying methods of SDL delivery have been that 1) this lecture has been the only lecture taught by the investigator, 2) this lecture has been the only lecture delivered in a manner unlike the remaining 34 lectures taught in a traditional manner (slide-based live lectures) by nine other lecturers, and 3) the content has remained largely unchanged. Details of the methods of SDL delivery for each year are: 2009: SDL required, low-technology: MS Word document as learning guide. Self-assessment questions. No classroom time of instructor. 2010: SDL required, high-technology: Interactive, multi-media, web-based authoring tool (SoftChalk™). Self-assessment questions. Live classroom introductory information. Live Q&A and case review. Live session captured via PanOpto. 2011: SDL optional, high-technology: SoftChalk™ as optional replacement or as supplement to a live lecture using web-based tool. Live Q&A and case-review. Live session captured via PanOpto. 2012: No SDL. Low-technology: Traditional, slide-delivered lecture. Live Q&A and case-review. Live session captured via PanOpto. Planned 2013: SDL required, high-technology: SoftChalk™ Live

introductory session regarding: SoftChalk™ and SDL as methodology, introduction to an optional, anonymous survey regarding student choices in: 1) method of formative feedback (pre/post self-assessment questions, or skills activities in SoftChalk™ lesson requiring/not requiring mastery before progression, or post-lesson case-based questions with answers), and 2) percent coverage of learning outcome categories, with guidance on applicability of each. Live Q&A and case-review will be provided in a post-lesson review period. A table indicating the investigator-assigned rankings of individual SDL components of each of the years of implementation is provided below:

Year Level of SDL [1=low (not

SDL), 2=medium (optional SDL),

3= high (required SDL), 4 = very high (required,

with many components of

SDL)]

Level of technology

[1=low (traditional; MS Word or PowerPoint),

2= high(SoftChal

kTM)]

Level of instructor

involvement6 [1= very low (no classroom time), 2= low-moderate, 3=moderate-high

4=high(full classroom time)]

Level of student involvement6

[1=low (none), 2= moderate (some; optional SDL or not, addressing identification of resources), 3= (considerable; identification of resources and

learning objectives)]

2009 3 1 1 1 2010 3 2 2 1 2011 2 2 2 2 2012 1 1 4 1 2013

(planned) 4 2 3 3

Assessment of student learning: Student learning will be assessed by analyzing the class mean midterm exam scores on topic –specific questions. Acknowledging a call for information on the value of Knowles’ individual components of successful SDL implementation identified in the table above, I will evaluate exam scores by two of those components, level of instructor as facilitator and level of student involvement (in developing learning objectives, and identifying appropriate resources).6 Likewise, I will address the call for information on how to best use technology, by assessing the impact of educational technology, with or without student involvement, on exam scores.7 A Kruskal Wallis test will be used to evaluate exam scores across the five years in relation to level of overall SDL, and to each of the investigator-assigned rankings of

individual SDL implementation components shown in the table above.

Assessment of reaction to learning: Student reaction to learning will be assessed by analyzing the results of instructor-specific course evaluation questions regarding teaching methods and communication/presentation skills. All questions use a Likert scale of one to five, with 1= negative to 5=to positive reactions. A Kruskal Wallis test will be used to evaluate scores assessed across the five years in relation to the degree of overall SDL, and to each of the investigator-assigned rankings of individual SDL implementation components.

Two voluntary student focus groups will be conducted by a faculty member unbiased and external to the project. One will occur prior to the 2013 course to evaluate preferred methodologies in SoftChalk™ lessons (using an unrelated topic), and another post-course to provide feedback on the 2013 delivery, with questions relating to SDL as a methodology, the use of SoftChalk™, and student and instructor involvement. These questions will occur as a survey using a Likert scale of one to five, with 1= negative to 5=to positive reactions, and as open-ended questions to elicit additional information. The analysis of these findings will be descriptive.

d. How might you make your work available to others in ways that facilitate scholarly critique and review, and that contribute to thought and practice beyond the local? I am actively involved in several groups who can provide scholarly critique and review:

• I am a faculty member at the Skaggs School of Pharmacy and Pharmaceutical Sciences (SSPPS), where curricular renewal is underway in the on-campus and distance-based programs, leading to renewal of methodologies in many classes. In addition to my mentor, many faculty members will be available to review and comment on my project.

• As a member of the Colorado Extended Studies Deans and Directors group, I will be able to share my findings with colleagues across the state who will be able to provide the perspective of extended studies programs, live and online.

• As a member of the American Association of Colleges of Pharmacy (AACP), I will have the opportunity to submit my work for review by pharmacy faculty, including those with a special interest in curricular and technology issues.

e. Include a literature review of the theory and effective teaching practice of the subject:

I conducted a literature search, with the assistance of PTLC librarian Lisa Traditi, in PubMed and ERIC using the terms “self” OR “student” AND directed learning” AND “implementation” OR “efficacy” OR “educational technology”, and “multi-instructor”, and “single lecture”. Selected articles were placed in Google Scholar for additional findings. Articles from prior related searches were included.

Literature on SDL notes its widespread acceptance as an effective learning methodology for adult learners for approximately forty years.4,6 In this context, it has been used largely for continuous professional development (CPD) of many professions, but due to reduced budgets and limited classroom and faculty time, it is also being explored within curricula. 6 Its use within the curricula of health professions is attractive, as it is known to lead to enhancing important future professional skills such as autonomy and accountability.4 The use of SDL within pharmacy curricula meets a recent call by the AACP for more innovation in pharmacy education. In this report, Blouin states that pharmacy education must move beyond the use of classroom time for the transfer of education, and instead focus on the how and why of gained knowledge.2 This call fits well with the steps involved in my implementation of SDL being assessed in my proposed project.

My project is centered on my assessment of the implementation of SDL in a single lecture of a traditionally-delivered course. Malcolm Knowles described seven key components to the successful implementation of SDL, with three being the need to involve: 1) involve the educator as a facilitator, 2) the student in developing learning objectives, and 3) the student in identifying appropriate resources.4,6 The literature, however, has not identified the importance of single components of Knowles’ seven components of successful SDL implementation. This may be due to the fact that while introductory time and student and instructor preparation for SDL can occur with its use in the healthcare profession setting, accreditation constraints may prevent the use of a true student-driven learning contract. 4,6 The literature notes that faculty involvement may be key the key to successful SDL implementation in the healthcare setting, especially in multi-instructor courses, where factors associated with successful single lectures in a multi-instructor course were introduction / organization and student involvement.1,3 As described earlier, my report will attempt to address the importance of both faculty and student involvement in SDL coursework; it will also provide insight into the implementation of unique methodologies in a single lecture of a multi-instructor, traditional-delivery course.

SDL literature describes that the assessment of SDL focuses largely on its acceptance by students, and less on its impact on learning, especially in comparison to traditional methods.4,6 When SDL data on student learning are provided, they usually demonstrate matching efficacy to traditionally-delivered lessons, but rarely note use of SDL exceeding traditional methods. Few reports in comparison to traditional methods provide negative findings on SDL implementation, but are called for to better understand best practices of SDL implementation.6,8 My project will compare factors using SDL to traditional-delivery and may identify negative findings.

The impact of technology in SDL is limited in the literature. Authors share its use in SDL can enhance independence in learning, but for healthcare professions, live faculty time is still needed to optimize learner attitudes toward the methodology and technology. 3-5 Blouin notes that innovation in the form of technology, and in particular the use of multimedia offerings, can assist in the goal of bringing innovation to pharmacy education.2 Others go further to discuss the importance of assessing how to use technology versus assessing if it produces a significant difference in learning.7 My report describing different methods of implementing technology in SDL as a single lecture in a multi-instructor, traditional-delivery course, will contribute to this literature. Chosen references are provided below:

• Albanese MA. An observational study of the lecture delivery style characteristics of high and low rated lectures. 70th Ann Meeting of the Am Educ Research Assoc, San Francisco, CA, April 16-20, 1986.

• Blouin RA, Riffee WH, Robinson ET et al. AACP curricular change summit supplement: Roles of innovation in education delivery. Am J Pharm Educ 2009;73(8): Article 154.

• Brydges R, Dubrowski A and Regehr G. A new concept of unsupervised learning: directed self-guided learing in the health professions. Acad Med 2010; 85: S49-S55.

• Levett-Jones TL. Self-directed learning: implications and limitations for undergraduate nursing education. Nurse Educ Today 2005; 25: 363-368.

• Manochehri NN amd Sharif K. A model-based investigation of learner attitude towards recently introduced classroom technology. J Information Tech Educ 2010; 9:31-52.

• Murad HM and Varkey P. Self-directed learning in health professions education. Ann Adac Med Singapore 2008; 37:580-90.

• Oblinger DG and Hawkins DL. The myth about no significant difference: “using technology produces no significant difference”. Educause

Review 2006; 41(6):14-16. • Petty J. Interactive, technology-enhanced self-regulated learning tools in

healthcare: a literature review. Nurse Educ Today 2013; 33: 53-59. f. What is your record of innovation in teaching and/or the assessment of learning? I am a 2011 recipient of the President’s Teaching and Learning Collaborative, finishing my project this summer. As director of the school’s Distance Degrees and Programs (DDP) office, I have applied innovative educational methods in both DDP and on-campus courses. As a result, I have presented and will present the following (selected):

• Poster presentation: Gleason SE, Nuffer M, Grogan M et al. Franson KL. Evaluation of the innovative use of teleconferencing for problem- and team-based learning (PBL, TBL) in an online post- baccalaureate (BS) PharmD program for globally based pharmacists. FIP World Congress, to be presented Aug. – Sept. 2013.

• Oral presentation: Gleason SE, Nuffer M, Grogan M et al. Teleconferencing for team- and case-based learning (TBL, CBL) in online PharmD pharmacotherapy courses. Monash Pharmacy Education Symposium; to be presented July 2013.

• Ongoing PTLC project: Berning, SE and Nuffer, M. Evaluation of foundational team- and problem-based learning in an online post-baccalaureate Doctor of Pharmacy program; 2011 – 2013.

• Oral presentation: Berning SE, Nuffer M, Wang S. Teleconferencing: A bridge to addressing multiple needs; Cite 2011; April 2011; Denver, CO.

• Poster Presentation: Berning SE, Wang S, Franson KL. Self-directed learning (SDL) pilot in a pharmacotherapy course with and without a web-based learning tool. AACP Annual Meeting; July 2011.

g. Are you able to attend the required meetings? Yes, I will be able to attend the meetings.

h. Please provide the name and email address for your coach/mentor. Are you willing to set each coach/mentor meeting twice each semester? Christopher Turner B.Pharm., Ph.D., Professor and Director of Experiential Programs, University of Colorado Skaggs School of Pharmacy and Pharmaceutical Sciences, Christopher.Turner@ucdenver.edu; yes.

If your project is selected, are you willing to serve as a coach/mentor in a future year? Yes, gladly.

Jeannette Guerrasio, MD Associate Professor of Medicine University of Colorado School of Medicine, General Internal Medicine

Anschutz Medical Campus Internal Medicine – HMG Mail Stop F782 12401 East 17th Ave. Aurora, Colorado, 80220 jeannette.guerrasio@ucdenver.edu

Graduate Medical Trainees Placed on Probation: Descriptive Statistics and Outcomes

Please consider my proposal for the President’s Teaching and Learning Collaborative, entitled, “Graduate Medical Trainees (GMTs) Placed on Probation: Descriptive Statistics and Outcomes.” Following the successful completion of medical school, physicians are required to complete additional graduate medical training prior to independent practice, also know as residency and fellowship. While it has been shown that medical students with professionalism lapses are more likely to be cited by their State Medical Boards when in independent practice 1, little is known about GMTs who underperform and are placed on academic probation due to either poor professional behavior and/or insufficient clinical skills 2,3.My proposal specifically asks, does being placed on probation as a GMT for poor professional behavior or insufficient clinical skills predict future negative professional outcomes such as disciplinary actions by State Medical Boards, lack of specialty board examination certification, or cessation of clinical practice?

In 2006, under the direct mentorship of Eva Aagaard, MD, I created a pioneering remediation program for internal medicine GMTs. The program provided a systematic approach for the identification of struggling learners, diagnosis of learner deficits, creation and implementation of a remediation plan tailored to the learners’ deficits and unbiased reassessment of subsequent performance. With much success, the program expanded to include medical students and most recently GMTs across all specialties. In 2011, there was a national call for an evidence-based approach to remediation. Since then, I have written a book on remediation of struggling medical learners, and a description of the methodologies used and our

outcomes were accepted for publication in the journal Academic Medicine 5,6,7.

Placing underperforming learners on probation and providing remedial teaching is both costly and high-stakes for the learner and institution. Many undergraduate and graduate education programs struggle to balance their professional obligations as teachers with obligation to their professions and to the public. While our prior research has demonstrated that for every 1 hour of face to face faculty time with a struggling learner the odds of having a negative academic outcome such as delayed graduation, probation, transfers to other specialties or programs, withdrawal and dismissal decreased by 3%, having long-term outcome data will help further inform and guide educational institutions as they determine how much to invest in academic probation actions and remedial teaching.

The central research question of my proposal represents the first step in our understanding of the long-term trajectory of underperforming GMTs. This 10 year retrospective review will examine the academic files of all learners placed on probation prior to the implementation of the described remediation program. We will extract data about personal demographics, program type, year of training, learner deficit(s), whether mental health evaluation was mandated, program completion data, and prior licensing exam scores. For the same subjects, information will be collected from the Federation of State Medical Boards, including disciplinary actions by State Medical Boards, number of states granting licenses, lack of specialty board examination certification, and cessation of clinical practice. After descriptive statistics are analyzed, the outcomes data will be compared to the average data for all GMTs who remained in good academic standing throughout their graduate medical training. Logistic regression will be used to identify predictors of future negative outcomes.

This study will serve to provide comparison data for a second study, separate from this proposal that will be conducted in 5-10 years to compare the outcomes between learners placed on probation before and after the implementation of the remediation program.

In order to make my work available to others, I hope to present the data and conclusions at poster sessions locally and at one or more national medical education conferences, and to then publish the results in an academic medical journal. The data can then be subsequently published in a higher education journal comparing outcomes for underperforming GMTs with underperforming learners in other graduate and undergraduate programs, including PhD, PA, DNP, and JD programs.

I am able to and look forward to attending the required meetings as specified in sections titles, “What are the Benefits?” and “What commitments are expected of participants?” Given my long-standing and productive mentoring relationship with Eva Aagaard, MD, founder and director of the Academy of Medical Educators, she will serve in the collaboration as my coach. Her email address is eva.aagaard@ucdenver.edu. For years, we have met twice monthly for my professional development, program development and research advising and will continue in this role. Eva is also available for the video conference meeting on June 27, 2013 from 9:15 to 11:00am. I also agree, if selected, to serve as a future coach in PTLC.

Thank you for considering my proposal. Several of the references from my literature search were written by me along with colleagues. Because the study of underperforming learners is such a new area of medical education research, there is much to be learned from other disciplines in higher education. I see the President’s Teaching and Learning Collaborative as a bridge between our campuses that will help provide connections for future educational research on struggling learners, how to maximize remediation efforts and how best to define successful remediation.

Resources:

1 Papadakis MA. Teherani A. Banach MA. Knettler TR. Rattner SL. Stern DT. Veloski JJ. Hodgson CS. Disciplinary action by medical boards and prior behavior in medical school. NEJM. 2005;353(25):2673-2682.

2 Teherani A. Hodgson CS. Banach M, Papadakis MA. Domains of unprofessional behavior during medical school associated with future disciplinary action by a state medical board. Acad Med. 2005;80(10S):S17-S20.

3 Zbieranowski I, Takahashi SG, Verma S, Spadafora SM. Remediation of Residents in Difficulty: A retrospective 10-year review of the experience of a postgraduate board of examiners. Acad Med. 2013;88:111-116.

4 Hauer KE. Ciccone A. Henzel TR. Katsufrakis P. Miller SH. Norcross WA. Papadakis MA. Irby DM. Remediation of the deficiences of physicians across the continuum from medical school to practice: a thematic review of the literature. Acad Med. 2009;84(12):1822-32.

5 Guerrasio J. Teaching Those Who Need Us Most: Remediation of the Struggling Medical Learner. Irwin, PA: Association for Hospital Medical Education; 2013.

6 Guerrasio J. Garrity MJ. Aagaard EM. Medical students and physicians referred for remediation: Demographics and outcomes. Acad Med. Accepted. 4/2013

7 Guerrasio J. Aagaard EM. Methods and outcomes for the remediation of poor clinical reasoning. Acad Med. Pending revision.

Joel C. Marrs, PharmD, BCPS (AQ Cardiology), CLS

Assistant Professor University of Colorado Anschutz Medical Campus Skaggs School of Pharmacy and Pharmaceutical Sciences (SSPPS) Department of Clinical Pharmacy Mail Stop C238 12850 E. Montview Blvd., V20-2128 Aurora, CO 80045 303-724-5780 Joel.marrs@ucdenver.edu

Quality Assessment of the Colorado Pharmacy Residency Teaching Certificate (CPRTC) Program

a. What is the central question, issue, or problem you plan to explore in your proposed work? An increasing number of graduates from colleges and schools of pharmacy are completing post-graduate residency training. Across the United States there are a number of teaching certificate programs at various schools, but there is no set standard for curriculum components from American Society of Health-system Pharmacists (ASHP ), the pharmacy residency accrediting body. ASHP states that one of the practice skills that should be taught during residency is how to deliver effective education. The University of Colorado Skaggs School of Pharmacy as offered the Colorado Pharmacy Residency Teaching Certificate (CPRTC) Program for the past 3 years. Up to this point there has only been qualitative assessment of the program as it relates to residents satisfaction with the overall focus and activities of the program. There is a need to quantitatively assess the CPRTC program in the areas of perceived benefit, confidence in teaching abilities, and career planning as it relates to seeking faculty positions post-completion of the program. A pre and post survey in the core areas of perceived benefit, confidence in teaching abilities and interest in seeking faculty positions will allow for better assessment of the quality and impact of the CPRTC program.

b. Why is your central question, issue, or problem important to you and to others who might benefit from or build on your findings? There is an increasing shortage of faculty members at schools and colleges of pharmacy across the United States. In 2008, the American Association of Colleges of Pharmacy (AACP) published the findings of a task for on pharmacy faculty workforce and estimated that 1200 new faculty will be needed over the next 10 years at colleges/schools of

pharmacy in the United States. 1 McNatty, et al. surveyed pharmacy residents who completed residencies from 2003-2006 and reported that 2/3 of pharmacy residents were interested in getting more teaching experience during residency.2 Teaching certificate programs are a means to increase the amount of teaching activities pharmacy residents are exposed to during residency training. In addition they are a potential means to increase pharmacy resident’s confidence in their teaching abilities and interest in pursuing a career in academia.

The actual benefit of teaching certificate programs has been under assessed in the literature and therefore a need to evaluate the impact and quality of the CPRTC program is needed. If the program is able to show an improvement in the perceived benefit of the program, confidence in teaching abilities, and interest in seeking faculty positions then this could impact other residency programs in the United States by encouraging them to develop similar programs.

c. How do you plan to conduct your investigation? What sources of evidence do you plan to examine? What methods might you employ to gather and make sense of this evidence? What literature have you reviewed on your topic? The objective of this prospective study is to evaluate the impact of the CPRTC program on pharmacy residents as it pertains to their perceived benefit of the program, confidence in their teaching abilities, and there future career plans relative to academia in pre and post assessment.

Previous studies have evaluated the impact of residency teaching certificate programs on residents abilities within a single residency program and have shown improvements in residents confidence in their teaching abilities.3-4 In addition one study has evaluated pharmacy residents perceived value of a teaching certificate program and showed that 90% of individuals felt the experience helped them in their current position.5 No published evaluation has specifically looked at a state wide program like CPRTC to determine the impact and value a teaching certificate across numerous residency program participants. The CPRTC program includes 8 two hour workshops throughout the academic year on a variety of topics (e.g., developing teaching outcomes and objectives, active learning techniques, teaching philosophy development). In addition residents are required to complete 8 hours of facilitation in the PharmD curriculum and give a 1 hour presentation within the PharmD curriculum or at their residency practice site.

Lastly, residents are required to submit a teaching portfolio which includes

a teaching philosophy statement, list of teaching activities they are involved in during the year, peer and self-evaluations on didactic and facilitating activities and a self-reflection statement on their teaching experience during the academic year.

The pre and post survey assessment will consistent of numerous questions in the three core areas of perceived benefit of the CPRTC program, confidence in their teaching abilities, and future career plans relative to academia. For questions on perceived benefit and confidence in teaching abilities a Likert scale (agree, partially agree, unsure, partially disagree, or disagree) will be used. For future career plans a scale of yes, no, unsure will be utilized. An assessment of the type of residency (first or second year) and/or fellowship will be collected as well.

Descriptive analyses (mean values and standard deviations) of perceived benefit and confidence in teaching ability parameters will be calculated. Nonparametric, ordinal data will be analyzed with the Wilcoxon Rank Sum Test comparing responses pre and post program. Categorical data will be analyzed with the Chi Square Test. Resident survey data will be de-identified to include no resident or fellow names.

Recruitment: Residents or fellows will be sent via email (SurveyMonkey) the baseline survey 1 week prior to the first teaching workshop in August 2013 and give the opportunity to participate in the survey if they so choose. Then all participants in the CPRTC program will be sent via email (SurveyMonkey) the post program survey in June 2014 and be given the opportunity to complete the survey if they so choose.

Subject Population: In 2012-2013 47 residents or fellows completed the CPRTC program. It is estimated that 40-50 residents will complete the program in 2013-2014.

d. How might you make your work available to others in ways that facilitate scholarly critique and review, and that contribute to thought and practice beyond the local? The study protocol will be submitted to COMIRB for expedited approval. Results of the study will be submitted as a poster presentation at the American Association of Colleges of Pharmacy Annual meeting where I would have the opportunity to share the results with the national academic pharmacy community. A final manuscript will be prepared and submitted for publication in an education-based, peer-reviewed journal such as the American Journal of Pharmaceutical Education.

e. Include a literature review of the theory and effective teaching

practice of the subject of your inquiry in order to locate your research in the literature preceding it.

1. Beardsley R, Matzke GR, Rospond R, et al. Factors influencing the pharmacy faculty workforce. Am J Pharm Educ. 2008 April 15; 72(2): 34.

2. McNatty D, Cox CD, Seifert CF. Assessment of teaching experience completed during accredited pharmacy residency programs. Am J Pharm Educ. 2007 October 15; 71(5): 88.

3. Romanelli F, Smith KM, Brandt B. Teaching residents how to teach: A scholarship of teaching and learning certificate program (STLCP) for pharmacy residents. Am J Pharm Educ 2005; 69(2) article 20.

4. Castellani V, Haber SL, Ellis SC. Evaluation of a teaching certificate program for pharmacy residents. Am J Health-Syst Pharm 2003;60:1037-1041.

5. Gettig JP, Sheehan AH. Perceived value of a pharmacy resident teaching certificate program. Am J Pharm Educ. 2008 October 15;72(5):104.

6. Oregon Pharmacy Teaching Certificate (OPTC) Program: A Two Year Experience. Marrs JC, Bearden DT, Cawley P, Marcus K. American Association of Colleges of Pharmacy Annual Meeting. July 18-22, 2009, Boston, MA

7. Portland Pharmacy Residency Teaching Certificate (PPRTC) Program. Marrs JC, Fugisaki B, Marcus K, Singh H, Stein SM. Abst #S124 American Association of Colleges of Pharmacy Annual Meeting. July 20-23, 2008, Chicago, IL

f. What is your record of innovation in teaching and/or the assessment of learning? I have been a faculty member for 6 years and have taught, directed, coordinated, developed and revised several professional pharmacy courses. I have directed two pharmacy residency teaching certificate programs which include the Oregon Pharmacy Teaching Certificate (OPTC) program from 2006-2009 and the Colorado Pharmacy Residency Teaching Certificate (CPRTC) program from 2010 to present.

I have previously evaluated and presented on the impact of residents perceived benefit, confidence in teaching abilities and interest in pursuing academic positions while directing the OPTC program.6-7 My previous

experience with evaluating the OPTC program in regards to residents perceived benefits and confidence in their teaching abilities has prepared me to evaluate the CPRTC program. I am involved in teaching numerous courses in the University of Colorado SSPPS curriculum as well as precepting both first and second year residents from the University of Colorado and Denver Health.

I served on the curriculum committee for 3 years while on faculty at Oregon State University College of Pharmacy from 2006-2009. I have been a member of the curriculum committee here at University of Colorado since joining the faculty 3 years ago and have been able to contribute to the committee’s charge of curricular renewal.

g. Are you able to attend the required meetings as specified in the sections titled, “What are the Benefits?” and “What commitments are expected of participants?” Yes.

h. Please provide the name and email address for your coach/mentor. Are you willing to set each coach/mentor meeting twice each semester?

Yes. Coach/mentor: Joseph Saseen, PharmD, BCPS (joseph.saseen@ucdenver.edu)

i. If your project is selected, are you willing to serve as a coach/mentor in PTLC in a future year? Yes

David A. Rickels Assistant Professor of Music Education University of Colorado

Boulder College of Music, Music Education IMIG Music Building, 301 UCB Boulder, CO 80309-0301 david.rickels@colorado.edu

Evaluating Synchronized Video Feedback in Pre-Service Teacher Education

In the current era of high-stakes assessment of teachers and increased focus on teacher quality, there are many critical issues in the preparation and assessment of preservice teachers (students enrolled in teacher preparation programs). This study will address the problem of how to deliver effective feedback on preservice teacher performance at executing lessons, given constraints such as time and feedback mode. When preservice teachers conduct a mock or real lesson, it can be disruptive for an instructor to stop their teaching to make corrections and offer immediate feedback based on assessment of their performance. However, if instructor feedback is delayed until after the lesson, it may not seem as relevant or authentic to the preservice teacher and may be less likely to lead to change in practice. This lack of authenticity may be compounded by the difficulty of capturing key moments of a lesson in a written mode, where the preservice teacher may not fully understand or recall what the instructor is referring to in the written comments.

This issue of feedback is critical to all those involved in teacher preparation, regardless of specific discipline. This study will explore a novel approach to delivering authentic feedback in the context of an undergraduate introductory music education course. This approach, referred to as synchronized video feedback (SVF), consists of a video recording of a lesson taught by a student overlaid with assessment feedback from the course instructor in the form of an audio track and graphical annotations. Using the SVF approach for preservice teachers offers the potential for more authentic feedback as the student is able to re-experience the lesson through audiovisual media while receiving relevant feedback on what she/he immediately sees and hears. This study holds the possibility of demonstrating a new and more effective teaching tool for all teacher educators.

The purpose of this study will be to examine the effect of the SVF format on the teaching performance of preservice teachers enrolled in an undergraduate teacher preparation program. This study will use an experimental design to compare SVF with traditional written-only feedback delivery. Data will be gathered in junior- or senior-level teaching methods courses. The primary evidence and dependent variable for quantitative

analysis employed in this study will be a measure of teaching effectiveness, such as the Survey of Teaching Effectiveness (Hamann & Baker, 1995). If funded by the PTLC, the funding will allow the recruitment of additional data collection sites beyond CU Boulder by supporting a graduate assistant who will conduct training and coordinate data between site leaders. The additional sites will increase the sample size and thus support greater generalizability of the findings. Faculty members at several universities have already expressed interest in this project. The funding will also support compensation of external scorers on the primary data collection instrument, which will increase the reliability of the data.

This experimental study of the SVF approach will follow two previous exploratory studies I conducted during the 2012-2013 school year. These precursor studies were designed to demonstrate the feasibility of the technology needed to combine audio feedback with videos of the preservice teacher lessons. These proof-of-concept studies also collected data on the attitudes of the preservice teachers toward the SVF approach. Findings indicated that the participants generally felt the SVF approach was superior to written feedback in allowing them to specifically understand corrective instructions. Findings also pointed to inconsistency or unreliability of the technology as the largest negative influence on participant attitudes.

There has been significant interest in feedback models from disciplines across teacher education, particularly as technology affordances have evolved to allow possibilities beyond written or direct verbal communication. Two prior feedback models in particular are relevant to the study of SVF. The first, referred to as video-elicited reflection or VER (Sewall, 2009), involves a preservice teacher and a mentor such as a field supervisor conducting a joint debriefing after a lesson, using the video recording of the lesson as a prompt for cooperative reflection. Sewall reported that this technique resulted in greater depth and breadth of reflection by the preservice teacher compared to traditional cooperative debriefing without a video artifact as prompt. The second feedback model has made use of “bug-in-the-ear” or BIE technology—a small one-way earpiece receiver that allows a preservice teacher to privately receive live feedback from an observer during a lesson. Multiple investigations in different teacher education disciplines have reported positively on the use of this device (Giebelhaus, 1994; Kahan, 2002; Rock, Gregg, Gable, & Zigmond, 2009; Scheeler, Congdon, & Stansbery, 2010). While these studies generally found the preservice teacher was able to make use of the feedback without suffering undue distraction, the distraction factor could be of additional concern in a more aurally intensive environment such as a music

classroom. The SVF approach is designed to maximize the affordances of both the VER and BIE approaches, while minimizing the negative aspects of each.

Although the ultimate goal for sharing this work is peer-reviewed journal publication, the nature of the technology also means that there are several other adjunct outlets for information in addition to journal publication. As this project utilizes video media, I plan to offer examples from the results via the internet to share with other practitioners worldwide. This will be done in conjunction with presentations at conferences for teacher educators (such as the Society for Music Teacher Education or the American Educational Research Association) in order to open a dialogue with as wide an audience as possible. As an example, the video report with example media from my recently completed precursor study is available on the internet at this address: http://youtu.be/dfI-43w1JAI

The education of future teachers is a primary area of my research agenda. Much of my research has focused on the use of technology in teacher education. During my first year at CU Boulder, I applied and was accepted to the fall 2012 cohort of the Teaching With Technology Faculty Seminar. Prior to coming to CU Boulder, I was part of the first and second cohorts (2010-2011 and 2011-2012) of the m-Learning Scholars Program at Boise State University—an interdisciplinary campus initiative that was aimed at investigating innovative uses of mobile technology in the learning environment. Both of these programs have supported the research projects that led to the reviewed presentations noted in the vita above. In all cases, the projects have explored novel uses for existing technologies in previously untried ways to improve transfer of learning among students as well as deepen the potential for authentic assessment of learning.

If accepted to the 2013-2014 PTLC, I agree to attend the meetings specified in the call for proposals. I have spoken to Dr. James Austin, College of Music, about serving as my coach for this project, and he has expressed his willingness to participate. Dr. Austin previously served as a PTLC coach for another faculty member in a prior year. If I am accepted to the PTLC, I will gladly serve as a Coach in the future, as I believe strongly in this method of sharing peer knowledge about best practices in teaching and learning.

Darcy Solanyk, MS, PA-C University of Colorado Denver

School of Medicine Department of Pediatrics Child Health Associate/Physician Assistant Program Mail Stop F543 13001 East 17th Place Aurora, CO 80045 Email: darcy.solanyk@ucdenver.edu

Development of a Valid and Reliable Evidence-Based Practice Curriculum Assessment Tool 3a. What is the central question, issue, or problem you plan to explore in your proposed work? Do the attitudes, knowledge and skills of PA students in the area of evidence-based practice (EBP) progress after taking a one semester Evidence Based Medicine (EBM) Course? Do the students also progress in these areas from the point of matriculation to completion of the three years of the Child Health Associate/Physician Assistant (CHA/PA) program?

b. Why is your central question, issue, or problem important to you and to others who might benefit from or build on your findings? As core faculty within the CHA/PA program and co-course director of its Evidence Based Medicine course, it is important to know if the information conveyed results in effective learning and retention of key concepts and skills in the area of evidence-based practice. Not only is it important to know if the class is effective, but being proficient at EBP is essential to being a high quality healthcare provider. The CHA/PA program needs to know if our students are competent healthcare providers and if the curriculum we teach is equipping them adequately to practice in the current healthcare system. Evidence, or Practice-Based Medicine is one of the key competencies in medical training, and therefore an important program outcome measure.

c. How do you plan to conduct your investigation? What sources of evidence do you plan to examine? What methods might you employ to gather and make sense of this evidence? What literature have you reviewed on your topic? In order to ascertain whether or not the EBM course and the CHA/PA curriculum improve attitudes, knowledge and skills around evidence-based practice, I plan to take the following approach: I will consult with an expert in developing a reliable and valid questionnaire. There are currently two well-tested questionnaires in use - one questionnaire is used to measure thoughts and attitudes about EBP. The second questionnaire, the Fresno Test for Evidence Based Medicine, has

been found to be valid and reliable when measuring knowledge and skills around EBP . By effectively combining the two questionnaires, I can get at an overall picture of the attitudes, knowledge and skills in EBP of the PA students. There is currently no single valid tool in use that measures each of these important components. There have been several instruments developed to evaluate the attitudes around EBP. There are two under consideration for merger with the Fresno Test. The Knowledge, Attitudes, Access and Confidence Evaluation (KACE) and the Evidence-Based Practice Attitude Scale (EBPAS-50) have demonstrated validity in assessing self-efficacy (student’s judgments regarding their ability to perform a certain EBP activity) and attitudes around EBP. KACE has been used with dental students and faculty. EBPAS-50 has been used with mental healthcare workers and social service providers. My goal is to identify which components of the current tools might be combined to produce a valid new tool that would be best suited for use in measuring attitudes, knowledge and skills of PA students... The Fresno Test is one of only two questionnaires that fully, reliably and validly evaluates the knowledge and skills of all areas of EBP. Rarely has this test been utilized to evaluate the effectiveness of a longitudinal curriculum. The few times it has been utilized, it has been found to be valid and reliable. Although it adequately assesses the knowledge and skills of students, it does not evaluate the attitudes surrounding EBP, and as the literature shows, attitude frequently drives behavior. Once I have created a questionnaire that can evaluate the attitudes, knowledge and skills of PA students in regard to EBP, I plan to administer it at four different times throughout the three year curriculum. I will administer the questionnaire at matriculation, as a pre-test to the EBM course, as a post-test for the EBM course and at the end of the three years prior to graduation. By establishing multiple points of evaluation within the curriculum, I can more adequately evaluate the impact of the EBM course itself and the impact of the PA curriculum as a whole in teaching our students how to conduct EBP. If I can see a progression of the attitude, knowledge and skills of the PA students in the utilization of EBP while in our program, then they are more apt to retain those behaviors and skills once they are out in clinical practice.2 Retaining such skill sets will make them more effective clinicians that will better serve their patients and communities.2 d. How might you make your work available to others in ways that facilitate scholarly critique and review, and that contribute to thought and practice beyond the local? (Keep in mind that coaching will be available to invite you to develop these aspects of your proposal, so you need not feel you must present a finished project design at this time.) Once I have tested the newly developed questionnaire and found it

to be valid and reliable, I would be interested in sharing it with other programs on the Anschutz Medical Campus who utilize EBP in a longitudinal fashion within their curriculum. I would also like to present the data from the questionnaire to the national PA Education Association for other PA programs to evaluate and critique. Hopefully other programs would be willing to further evaluate the questionnaire’s validity and reliability within their own settings. There is currently no tool in use within other PA programs. Because Practice-Based Medicine is a competency measured by most health care training programs, a valid tool could be useful across multiple disciplines. e. Include a literature review of the theory and effective teaching practice of the subject of your inquiry in order to locate your research in the literature preceding it. (The website http://www.colorado.edu/ptsp/ptlc/libraryresources.html offers expert advice on how to conduct a relevant literature review.) Evidence-based practice may be defined as the integration of the best research evidence with patients’ values and clinical circumstances in clinical decision-making.5 EBP is an essential quality in a good provider not to mention it involves concepts that many accrediting bodies require being taught to health professions students. There are over 104 unique assessment strategies for evaluating EBP in learners.5 Most of these assessment tools are utilized in medical students (undergraduate and postgraduate trainees) or practicing physicians. Few (13 out of 104) are utilized in non-physician professions5 and only a small number (27 out of 104) evaluate the attitudes around EBP.5 Only two effectively and objectively evaluate the knowledge and skills involved in EBP—the Fresno Test and the Berlin Questionnaire.5 No questionnaire or survey has been found to assess the attitudes, knowledge and skills of future healthcare providers, especially non-physician trainees. f. What is your record of innovation in teaching and/or the assessment of learning? My record of innovation in teaching over the past three years has been mostly in regard to the Evidence Based Medicine course. I have expanded the course curriculum to not only teach EBP concepts but to also include sessions on Information Mastery that help students learn to navigate the literature effectively at the point of care. I have also implemented interactive sessions with students within the course including case-studies from the Centers for Disease Control and Team Based Learning sessions to enhance concepts. g. Are you able to attend the required meetings as specified in the sections titled, “What are the Benefits?” and “What commitments are expected of participants?” Yes, I am able and excited to attend the required meetings and complete the expectations as specified in the application. I have the full support of the CHA/PA Program Director as well

as an identified mentor. Name and contact information for a mentor/Coach for your project? My mentor will be Cathy Ruff, MS, PA-C. Ms Ruff is the Associate Director of Curriculum within the CHA/PA Program and has over 10 years of experience in PA education. Her expertise is in curriculum development and assessment. Email: cathy.ruff @ucdenver.edu; Phone: 303-724-1345.

If you are selected, we ask you to agree to serve as a Coach in PTLC in a future year. I would be happy to serve in this capacity in the future.

1. Ramos KD, Schafer S, Tracz SM. (2003). Validation of the Fresno Test of competence in evidence based medicine. BMJ. 326 (7384), 319-321.

2. Tilson JM, Kaplan SL, Harris, JL, Hutchinson A, Ilic D, Niederman R, ….&Zwolsman SE. (2011). Sicily statement on classification and development of evidence –based practice learning assessment tools. BMC education, 11(1), 78.

3. Hendricson WD, Rugh JD, Hatch JP, Stark DL, Deahl T, Wallmann ER. (2011) Validation of an instrument to assess evidence-based practice knowledge, attitudes, access and confidence in the dental environment. JDentalEduc, 75 (2): 131-144.

4. Aarons GA, Cafri G, Lugo L, Sawitzky A. (2010) Expanding the Domains of Attitudes Towards Evidence-Based Practice: the Evidence Based Practice Attitude Scale-50. Adm Policy Ment Health 1-10.

5. Shaneyfelt T, Baum, KD, Bell D, Feldstein D, Houston T…& Green M. (2006) Instruments for Evaluating Education in Evidence-Based Practice: A Systematic Review. JAMA 296 (9), 1116-1127.

6. West CP, Jaeger TM, McDonald FS. (2011) Extended Evaluation of a Longitudinal Medical School Evidence-Based Medicine Curriculum. J Gen Intern Med 26 (6): 611-15.

Darlene Tad-y, MD Assistant Professor

University of Colorado School of Medicine Department of Medicine Division of General Internal Medicine – Section of Hospital Medicine 12401 E. 17th Avenue, Mailstop F782 Aurora, CO 80045 (720) 848- 4289 darlene.tad-y@ucdenver.edu Evaluating Internal Medicine Residents' Competence and Readiness for Unsupervised Practice

Background Since the introduction of Tomorrow’s Doctors in 1993, medical education has embraced competency-based medical education (CBME). This paradigm shift focuses on the attainment of knowledge, attitudes and skills relevant to the profession, rather than the amount of time or process of training. 1-5 Graduate medical education has also adopted the tenets of CBME, fundamentally changing the way residents should be considered competent to practice medicine in their specialties without supervision.6 Importantly, this change requires residents to demonstrate competence in the professional activities of our specialty, which has critical implications for the assessment and evaluative process of trainees. In 2012, the Accreditation Council for Graduate Medical Education (ACGME) announced that the Next Accreditation System (NAS) would be implemented 7, requiring internal medicine residency training programs (IMRPs) to implement the NAS by July 2013.The American Board of Internal Medicine (ABIM) and the ACGME convened task forces which defined the Entrustable Professional Activities (EPAs) of an Internal medicine physician and also the developmental milestones to guide progression to competence and ultimately unsupervised practice8,9.

My work will focus on the development of a comprehensive assessment framework for determining trainee competence and readiness for graduation from residency training, and will meet requirements for the Reporting Milestones to the ACGME. The framework, in its entirety, will allow residency program leadership and faculty to successfully assess residents’ developmental progress through training and ultimately provide the information needed to determine trainees’ readiness for unsupervised practice.

Because the training activities of residents are diverse and not always data driven, I anticipate that a comprehensive framework will require a variety of assessment tools, such that the framework could include a “toolbox” that

will include valid and reliable direct observation tools for clinical settings that will simultaneously encourage relevant formative feedback to the learners. Additionally, educational methods such as portfolios, resident written reflection, scholarly products, all require a standard method of assessment, which currently do not exist. The framework I propose will include these assessment tools and a described method for implementation. I anticipate that as I undertake this project, I will encounter many more assessment tools that will help our program accurately describe our residents’ progression through the milestones and finally achievement of competence in the professional activities of our specialty. Inevitably, implementation of this framework will require an accompanying faculty development program, which I intend to create as a companion to the framework itself.

Methods In addition to the review of the literature as briefly described above, I also plan to incorporate these methods:

• Review of all published direct observational tools for clinical and non-clinical activities related to graduate medical training

• A needs assessment survey of our faculty to determine their gaps in knowledge about CBME, the NAS, milestones and EPAs

• Pilot assessment tools and elicit feedback from faculty and trainees, revise as needed for improvement and applicability

• Collaborate with other IMRPs to share and define best practices Dissemination and Scholarship Opportunities exist for peer-reviewed dissemination of the tools, including Med Ed Portal. The American Association of Medical Colleges also hosts iCollaborative which is an online clearinghouse for educational works-in-progress that would allow me to vet my work along the way. I would also hope to share this framework with fellow program directors around the country by presenting at national program directors’ meetings, and hosting workshops. With regards to publication of outcomes, I hope this would be eligible for publication in a journal such as Medical Teacher or the Journal of Graduate Medical Education. About the Author My fellowship training provided me with training in curriculum development and teaching skills. In the 3 years since joining CU’s SOM, I have developed small-scale programs, such as the Phase I Quality Improvement Elective (9 learners per yer), and large scale programs such as the Hospitalist Training Program Didactic Curriculum (24 total learners per year) and the IMRP’s Quality and Safety Curriculum (~160 learners per

year). As an Associate Program Director of CU’s IMRP, I will be able to implement and study the proposed framework directly. I am committed to attending all required meetings for participants in the PTLC.

References

1. Snell LS, Frank JR. Competencies, the tea bag model, and the end of time. Med Teach. 2010;32(8):629–630. doi:10.3109/0142159X.2010.500707.

2. Taber S, Frank JR, Harris KA, Glasgow NJ, Iobst W, Talbot M. Identifying the policy implications of competency-based education. Med Teach. 2010;32(8):687–691. doi:10.3109/0142159X.2010.500706.

3. Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676–682. doi:10.3109/0142159X.2010.500704.

4. Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638–645. doi:10.3109/0142159X.2010.501190.

5. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631–637. doi:10.3109/0142159X.2010.500898.

6. Iobst WF, Sherbino J, Cate OT, et al. Competency-based medical education in postgraduate medical education. Med Teach. 2010;32(8):651–656. doi:10.3109/0142159X.2010.500709.

7. Nasca T, Philibert I, Brigham T, Flynn T. The Next GME Accreditation System - Rational and Benefits. New England Journal of Medicine. 366(11):1051–1056.

8. Caverzagie KJ, Iobst WF, Aagaard EM, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158(7):557–559. doi:10.7326/0003-4819-158-7-201304020-00593.

9. Green ML, Aagaard EM, Caverzagie KJ, et al. Charting the road to competence: developmental milestones for internal medicine residency training. Journal of Graduate Medical Education. 2009;1(1):5–20.