ExaminerNational Board of Medical Examiners3750 Market StreetPhiladelphia, PA 19104-3102

www.nbme.org

Fall/Winter 2015 Volume 62, Number 2

In its early years, the NBME and itssmall staff worked from various leasedoffice spaces around Philadelphia. In1966, the NBME moved to its ownnewly designed and constructed head-quarters on the 3900 block of ChestnutStreet, adjacent to the University ofPennsylvania and other major educa-tional and medical institutions, i.e., thearea known as University City. The build-ing also housed offices for two otherorganizations involved in medical edu-cation and assessment: the AmericanBoard of Internal Medicine and theEducational Council (now Commission)for Foreign Medical Graduates. Thisarrangement permitted these organiza-tions ready access to the NBME’s techni-cal and professional staff and special-ized computer system.

With the growth of NBME services andstaff, the building built in 1966 was nolonger sufficient to house the staff, hostcommittee meetings, and other impor-tant activities. In 1991, NBME gover-nance took action to approve construc-tion of new headquarters a few blocksfrom the Chestnut Street location. In1994, the NBME moved to its currentlocation on Market Street, also in WestPhiladelphia’s University City neighbor-hood. The new building was designedwith a conference center and centrallibrary to support the many test commit-tees that meet throughout the year in

In This Issue:

1 NBME In Situ: Creating Community

1 The NBME Welcomes Craig N. Mills

4 News from International Programs

5 Trilateral Board Meeting: July 2015

Philadelphia. Over the years, two newwings were added to accommodate thegrowing staff and growing services.

While focused on mission-driven workto protect the public through state ofthe art assessment of health profes-

The NBME is pleased to welcome Craig N. Mills, EdD,as senior vice president for professional services.Craig joined the staff in August 2015, following theretirement of Ron Nungester. The ProfessionalServices unit provides test development, scoring,measurement consulting, examinee support, andoperations research support.

(continued on page 5)

(continued on page 2)

© 2015 NBME

NBME In Situ: Creating Community

The NBME Welcomes Craig N. Mills

PROFILE

Former NBME Headquarters on Chestnut Street

2

sionals, the NBME has maintained astrong commitment to be a goodneighbor and support the UniversityCity/West Philadelphia community.Over the years, the University City sec-tion of Philadelphia has become avibrant neighborhood, teeming witheducational, medical, research, andtechnology activities, while also servingas a residential neighborhood for long-time residents, as well as students andfaculty from local universities andother institutions.

The University City District (UCD) is apartnership among the various entitiesin the community, from world-renownedinstitutions like the University ofPennsylvania, Drexel University, andChildren’s Hospital of Philadelphia tosmall businesses and residents. UCD hasa mission to revitalize the neighborhood,encourage economic vitality, and buildlocal workforce capacity by connectingmembers of the community to skill-building opportunities and jobs.

The NBME is a longtime member of theUCD and has contributed to and partici-pated in many UCD-sponsored activities.Membership in the UCD supports manyimportant services to the community,including: public safety patrols, night-time walking escorts for students andresidents to get safely to their homes,investment in public spaces and publictransportation, and more. NBME presi-dent, Dr. Donald E. Melnick, sits onUCD’s Board of Directors.

The NBME is particularly proud to partic-ipate in UCD’s West Philadelphia SkillsInitiative (WPSI), a program launched in2010 to connect local employers withWest Philadelphians seeking job skillsand employment. According to the WestPhiladelphia Skills Initiative ImpactReport (2014), the following numbersshow the opportunities and challengesthis unique community faces:

Challenge• 31% of West Philadelphians live

below the poverty level.• 45% of households have incomes

below $25,000/year.• Only 21% of residents age 25 or

older have a bachelor’s degree.

Opportunity• University City contains more than

10% of the entire employment basein the city of Philadelphia.

• University City institutions invest $1 billion/year in research anddevelopment.

• New or recently completed construction projects total about $4 billion.

Taking a jobs-first approach, WPSI’s tal-ent-development programs are intendedto meet two goals: (1) to provideemployers with individuals from thecommunity who can add value to theirorganizations, and (2) to provide jobseekers with the necessary skills to gainand retain employment. From the onsetof this program in 2010 through 2014,NBME hosted 11 participants from theWPSI program, and one participant com-pleted two summers at NBME. WPSI par-ticipants gained valuable work experi-ence in NBME’s Facilities Services,Desktop Services, and Legal Services.

Transforming Public Spaces is anotherUCD initiative supported in part byNBME. The recently opened InnovationPlaza, an outdoor gathering spot (some-times called a “pocket park”) featuringfree WiFi, game tables, lighting, seating,

(NBME In Situ: Creating Community, continued from page 1)

Current NBME Headquarters on Market Street

3

and greenery, runs along 37th Streetbetween Market and Chestnut Streets.The public space can be used for variousevents, such as concerts, movies, farm-ers’ markets, and street festivals, and isintended to further vitalize the growingUCD neighborhood. Innovation Plazaalso features an Innovators Walk ofFame that showcases “groundbreakingcontributions made to the scientific andentrepreneurial communities that haverevolutionized the local, regional andglobal landscape.”

NBME’s former Chestnut Street head-quarters is situated directly across thestreet from Philadelphia’s RonaldMcDonald House (PRMH), and NBMEhas been a longtime supporter. Eachyear, NBME staff members donate toysand make monetary contributions toPRMH to aid its mission to support thefamilies of seriously ill children.

Collection jars are placed around thebuilding, and staff fills them with pull-tabs from aluminum cans. PRMH derivesneeded funds by selling the tabs to arecycler. In 2014, NBME was pleased tohelp sponsor PRMH’s 40th anniversarycelebration. Through NBME’s donation,several current and past PRMH familieswere able to attend the event and enjoythe celebration.

NBME has been a power partner of thePhiladelphia Reads program since 2008.Philadelphia Reads works with individu-als, schools, community- and faith-basedorganizations, and businesses to providementors, resources, and advocacy tostrengthen the literacy skills of the city’syoungest and neediest children inkindergarten through the third grade.Since 2008, NBME staff literacy coacheshave volunteered their lunch hours eachWednesday from October through May

to read to the students at NBME. Staffmembers have donated several shelvesworth of books in the NBME library toPhiladelphia Reads, so the children areexposed to a variety of reading experi-ences.

In May 2015, NBME completed 100years of service to the public and thehealth professions. Included among themany celebratory activities that markedthis milestone was the launch of NBME’sLend a Hand Volunteer Program inwhich staff members are given up toseven hours to help out in our WestPhiladelphia community or the commu-nity in which they live. Staff memberscan complete their volunteer service atthe non-profit organization of theirchoosing.

So far in 2015, staff has volunteered formore than 230 hours of service at 18different organizations.

NBME remains committed to enhancingits West Philadelphia neighborhood andto helping meet the human, educational,environmental, and public safety needsof the community where we work.

NBME senior staff showing their support for Philadelphia Ronald McDonald House.

4

China

NBME has an exciting opportunity toinfluence the health of the Chinese pub-lic with a pilot project to develop a certi-fication examination for a new health-care professional in China, i.e., a “healthcoach.” The Chinese government issueda request for proposals for an assess-ment to assure the competency ofhealthcare professionals and to addressthe knowledge and skills needed tomotivate people to take responsibility fortheir improved health and wellness.Success of this pilot project will be a firststep to establishing NBME as a leader inassessment in China.

NBME staff has been working for thepast year to develop the ProfessionalExamination of Health Coach, or PEHC.健康指导师职业考评 is the officialname for the China health coach certifi-cation examination. Though there arehealth coaches in the United States, theresponsibilities of the health coach inChina will differ somewhat, and devel-opment of the examination blueprintincluded ethical and cultural topicsunique to China.

Staff conducted three item writing work-shops (IWWs) with medical school facul-ty, including physicians, nurses, UShealth coaches, and several Chinesenationals to develop items for the exam-ination. Individuals involved in the IWWswere highly engaged and helped toshape the definition for the health coachin China and how this new health pro-fession will work within the currenthealthcare system in China. It is evidentfrom conversations with Chinese nation-als who participated in the IWWs, aswell as the excitement of the Chinesepeople we meet on visits to China, thatthis new health profession will have amajor impact on the health of theChinese public.

The examination is part of a larger ini-tiative of the Smart Healthy CitiesAlliance (SHCA). The goal of the SHCAis to combine world class capabilities toprovide integrated solutions that focuson the elderly and aging population inChina, on chronic diseases, and on pro-moting health and wellness in theChinese population. The SHCA wasorganized through Orchestrall, Inc. andbrings together ministries, associations,and organizations that can help inaccomplishing the goal. The secondannual SHCA conference was held InHangzhou, China on September 23-25,2015 and both M. Brownell Andersonand Dr. Donald Melnick participated inthe conference.

Kazakhstan

In September, 2014 the NBME and theMinistry of Health (MOH) of theRepublic of Kazakhstan executed anagreement for consulting services toimprove the medical education system,establish a licensure/certification frame-work for health specialists, and advancethe healthcare of the population ofKazakhstan. Putting on their warmestouterwear, NBME staff traveled toAstana, Kazakhstan in December 2014(where it was -5° C) to work with staffof the newly established Republic ofKazakhstan Center for Knowledge andSkills Assessment (RCKSA). The agendafor the meeting included presentationson assessments and blueprint design,standards for health specialists accord-ing to the International Organization forStandardization, National Council forMeasurement in Education, andAmerican Psychological Association, andscore reporting and equating. NBMEstaff also conducted workshops on itemwriting and standard setting. In a reportsubmitted to the MOH, staff recom-mended that RCKSA send representa-

News from International Programs

INSIDE THE NBME

(continued on page 6)

5

Prior to joining the NBME, Craig wasvice president, research at McGraw HillEducation/CTB working in the K-12assessment space. From 1998 to 2014Craig worked in several positions at theAmerican Institute of Certified PublicAccountants (AICPA), where he led thedevelopment and implementation of thecomputer-based version of the UniformCPA Examination in 2004. The projectto computerize the CPA Examinationwas notable in its groundbreaking workwith regard to disciplined item develop-ment, introduction of computerizedmulti-stage adaptive testing, incorpora-tion of novel, technology-based assess-ments, and a number of other contentand psychometric advances.

Craig was employed by EducationalTesting Service (ETS) for almost 15 yearsbefore he joined the AICPA. He beganhis employment at ETS as a measure-ment statistician. As executive directorof the Graduate Record ExaminationsProgram he directed the development ofthe computer adaptive GRE GeneralTest, the first high-stakes admissions testto be computerized. After leaving theGRE Program, Craig worked with seniormanagement of ETS to design a corpo-rate infrastructure to support CBT.

Craig received his doctorate from theUniversity of Massachusetts, Amherst in1982. He is the author of over 75 publi-cations and professional presentationson standard setting for decision tests,computerized testing, and other meas-urement topics. He was a co-author ofone of the first sets of guidelines for

computer adaptive assessment, pub-lished by the American Council onEducation. He is co-editor of Computer-Based Testing: Building the Foundationfor Future Assessments. In 2008 Craigwas named one of the ten most distin-guished graduates in the first 100 yearsof the University of MassachusettsSchool of Education.

When asked what sparked his interest injoining the NBME, Craig cited the organ-ization itself, the high quality and dedi-cation of the staff, and excitement aboutNBME’s future. Craig stated, “Who wouldn’t want to join an organization thathas 100 years of success behind it and yetboth recognizes the need to change and iscommitted to changing in order to be successful well into the future?”

(Craig N. Mills, continued from page 1)

On July 11, 2015, members of NBME’s Executive Board met with members of theEducational Commission for Foreign Medical Graduates (ECFMG®) Board of Trusteesand members of the Federation of State Medical Boards (FSMB) Board of Directors for aseries of meetings related to topics of interest to the three organizations. In plenary andsmall group break-out sessions, participants discussed topics such as: health workforce,interstate medical licensure compact, competency-based assessment, and internationalmedical education. The meetings were facilitated by senior staff and board members ofall three organizations.

The trilateral board meetings are held every few years and provide an opportunity forthe leadership of these collaborative organizations to meet and discuss issues importantto their missions.

Trilateral Board Meeting: July 2015

INSIDE THE NBME

ECFMG Board of Trustees, FSMB Board of Directors, and NBME Executive Board members at the July 2015 trilater-al board meeting in Middleburg, Virginia.

tives to Philadelphia to spend additionaltime in workshops to learn about itemwriting, standard setting, and scoring. InJune 2015, 18 representatives fromthroughout Kazakhstan spent one weekwith NBME staff in a variety of work-shops. In August, 2015, staff made asecond visit to Astana (it was warmerthan December). The agenda focusedon such topics as conducting a jobanalysis and the impact of that on anexamination blueprint, the importance ofdeveloping a coding schema for items,and building an item bank. The teamprepared a detailed report of their find-ings and provided recommendations fornext steps. Braving the -10° C weather,M. Brownell Anderson returned toAstana in November to present at aconference that focused on healthcare inKazakhstan. While there, she met withthe newly appointed vice-minister ofhealth, who expressed his hope that

NBME would be able to continue work-ing with the RCKSA to develop a licen-sure examination in Kazakhstan forphysicians and nurses.

BrazilAvaliacão de Desempenho deEstudantes de Medicina(ADEM+) [English translation:Performance Evaluation ofMedical Students]

The ADEM+ examination was introducedin Brazil in 2013. A test committee com-prising faculty from medical schoolsthroughout Brazil has worked withNBME staff to contribute more than 50new questions to the exam. The lengthof the exam has grown over the years,from 80 items the first year to 100 itemsin 2014; it now has 120 items. Theexamination is performing well, is reli-able, and demonstrates a progression ofperformance between years of studentsand across the years of the examination.The goal is to have the examinationused throughout Brazil to measureprogress of medical students before theygraduate from medical school. NBME is engaged in a joint project com-mittee with the Hospital Sírio Libanês(HSL) in São Paulo. HSL has repurposedthe ADEM+ examination and used it asa residency selection examination.

6

(International Programs, continued from page 4)