an assessment tool translation study

Policymakers hoped to substitute a new,multi-purpose, functional assessment instru-ment, the minimum data set post-acute care(MDS-PAC), into the planned prospectivepayment system (PPS) for inpatient reha-bilitation hospitals. PPS design requires alarge database linking treatment costs withmeasures of the need for care, so the PPSwas designed using the functional indepen-dence measure (FIM™) database linked toMedicare hospital claims. An accuratetranslation from the MDS-PAC items toFIM™-like items was needed to ensure pay-ment equity under the substitution. Thisarticle describes the translation ef forts andsome of the problems that led policymakersto abandon the ef fort.

INTRODUCTION

Over the past 20 years, functional statusmeasurement has become a regular compo-nent of national health surveys, clinical caremanagement, and evaluation and researchstudies of elderly persons. As attention hasshifted from acute to long-term care, policy-makers and providers have become increas-ingly interested in including functional sta-tus measures in payment, monitoring, andoutcomes management systems. Providers,payers, and consumers would all benefitfrom comparable measures of functionalstatus and rehabilitation outcomes across

multiple care settings to facilitate equitablepayment and to monitor the quality and effi-ciency of care delivery. The numerousassessment tools that are currently in use,particularly those used to group patients bylevels of function, were designed for use ina single setting and have limited utilityacross different treatment settings (Morriset al., 1990; Granger et al., 1986; Fries et al.,1994; Stineman et al., 1994; and Hittle et al.,2002).

The field lacks a standardized rigorousapproach to functional content and assess-ment techniques. Nearly all functionalassessment measurements include someform of activities of daily living (ADL), thatis, the ability to perform basic tasks suchas eating, dressing, grooming, transfer-ring, walking and bathing, and often alsoinclude instrumental ADLs, (IADLs), suchas shopping, telephone use, laundry, med-ication use, managing finances, mealpreparation and housework (Katz et al.,1963; Lawton and Brody, 1969; Applegate,Blass, and Franklin, 1990; McDowell andNewell, 1996; Branch and Meyers, 1987;Teresi et al., 1997). However, which sub-sets of tasks, how assistance is measured,and who is providing the assessment oftendiffer (Jette, 1994). Thus, to assess eitherthe population being treated or the qualityof care rendered either across tools withina setting, or across settings when differentmeasures are used, requires that we beable to convert items from one measure-ment tool to another.

This article describes some lessonslearned from a large scale effort to substi-tute a new multi-purpose assessment tool,

HEALTH CARE FINANCING REVIEW/Spring 2003/Volume 24, Number 3 45

Joan L. Buchanan and Alan M. Zaslavsky are with HarvardMedical School. Patricia L. Andres and Stephen M. Haley arewith Boston University. Susan M. Paddock is with RAND. Theresearch presented in this article was sponsored by the Centersfor Medicare & Medicaid Services (CMS) under ContractNumber 500-95-0056 to RAND with a subcontract to HarvardUniversity for this study. The views expressed in this article arethose of the authors and do not necessarily reflect the views ofHarvard Medical School, Boston University, RAND, or CMS.

An Assessment Tool Translation StudyJoan L. Buchanan, Ph.D., Patricia L. Andres, M.S., P.T., Stephen M. Haley, Ph.D., P.T.,

Susan M. Paddock, Ph.D., and Alan M. Zaslavsky, Ph.D.

the MDS-PAC, (Federal Register, 2000) intoa patient classification and payment systemdesigned around another functionalassessment tool, the FIM™ (Granger et al.,1986). Effective payment system designrequires large amounts of data relatingresource needs (functional and cognitivestatus) to actual resource use (costs oftreatment). To this end, FIM™ data werelinked to Medicare rehabilitation hospitalclaims enabling the design of a compre-hensive patient classification system (PCS)and the calculation of a set of paymentweights. Because the MDS-PAC was anew instrument, the data needed to devel-op a PCS and an associated set of paymentweights for MDS-PAC assessments did notexist. Instead, policymakers hoped that asufficiently accurate translation from theMDS-PAC to the FIM™ could be devel-oped so that the MDS-PAC could be usedin the payment system that had beendesigned around the FIM™. Without anaccurate translation, payments for sometypes of patients would no longer reflecttheir resource needs and could thus lead,to access difficulties and/or paymentinequities to facilities. For the substitutionto be successful, ADL and cognitive itemsin the MDS-PAC needed to be translatedinto similar FIM™-like items to createFIM™-like motor and cognitive scales.Ultimately, this effort to substitute theMDS-PAC for the FIM™ was not sufficient-ly accurate to ensure payment equity.Industry objections to the administrativeburden of the new longer assessment toolin combination with our findings led poli-cymakers to opt to continue using theFIM™. It may be that the level of transla-tion accuracy needed for payment isnotably greater than what is needed formonitoring performance, for outcomesmanagement, or for other purposes.Nonetheless, the challenges posed in thistranslation effort provide important insights

on the need for standardization and defini-tional clarity in all aspects of functional sta-tus assessment.

BACKGROUND AND PURPOSE

PPSs provide a fixed payment per casethat is adjusted for differences in patienttype, but is independent of the amount ofservice provided. Consequently, they arebelieved to provide an incentive for costcontainment and efficient care delivery.Inpatient rehabilitation was exempted fromthe Medicare PPS for acute-care hospitalpayment when it was introduced in 1984.Rehabilitation hospitals were exemptedbecause research at the time demonstratedthat diagnoses, the basis of the MedicarePPS, were not adequate to predict resourceneeds in the inpatient rehabilitation popu-lation and that measures of functional sta-tus were needed (Hosek et al., 1986). Atthat time, there was no agreement on whatmeasures of functional status should beused, nor were these data routinely collect-ed. Since then, rehabilitation professionalshave developed a parsimonious 18 itemmeasure, the FIM™ (Granger et al., 1986).Further, more than one-half of all inpatientrehabilitation providers use the FIM™ andvoluntarily submit these data to a central-ized repository (Hamilton et al., 1987).Stineman and colleagues (1994) used theFIM™ data to develop a PCS for medicalrehabilitation, called the FIM™–functionrelated groups (FIM™-FRGs). Building onthe basic FIM™-FRG design, but usinglarger and more recent data sets, a RANDteam refined and expanded the classifica-tion system to cover all rehabilitation hos-pital discharges and provided the designfor a PPS for rehabilitation hospitals (Carteret al., 2002).

In the 1980s and 1990s, research inanother segment of the provider communi-ty, nursing facilities, was evolving along a

46 HEALTH CARE FINANCING REVIEW/Spring 2003/Volume 24, Number 3

separate path. In response to a 1986Institute of Medicine Study of the qualityof care in nursing homes that called forimprovements in nursing home quality andmore patient-centered care, researchers inthis community developed a comprehen-sive, multi-purpose instrument, the residentassessment instrument—MDS. (Morris etal., 1990). This instrument was mandatedfor use in all nursing facilities and is nowused for care planning, patient classifica-tion for prospective payment, and qualityassurance. A MDS was also developed forhome health care though the payment sys-tem for home care uses an alternativeassessment tool, the standardized Outcomeand Assessment Information Set (OASIS)for Home Health Care (Hittle et al., 2002).

Since the introduction of the hospitalPPS, hospital length of stay has fallen dra-matically while discharges to all types ofPAC providers (rehabilitation hospitals,nursing facilities, and home health agen-cies (HHAs) have increased markedly. Inan effort to control costs in the PAC area,the Balanced Budget Act of 1997 mandatedthe introduction of PPSs for nursing facili-ties, rehabilitation hospitals, and HHAs. In1998, the nursing home PPS, which usesper diem payments and a MDS-basedpatient classification system, resource uti-lization groups, version III, went into effect(Fries et al., 1994). This was followedshortly by a PPS for home health based onthe OASIS with episode-based payments.With the growth in the use of PAC cameincreased recognition of the considerableoverlap in populations being treated ineach setting. Many nursing facilities nowspecialize in sub-acute and rehabilitationcare or have special units within them toattract these patients. Thus, policymakerscalled for a more integrated approach topatient assessment that would cross post-acute settings. The MDS-PAC was devel-oped as a response to this need for integra-

tion across settings. Policymakers believedthat this new tool could be substituted intothe proposed inpatient rehabilitation PPSthat had been designed around the FIM™.A study by Williams and colleagues (1997)concluded that MDS items could be usedto predict FIM™ subscale scores with rea-sonable accuracy, which lent credence tothe proposed plan. While we know of noplans to substitute the MDS-PAC into nurs-ing homes or HHAs (the other post-acutesettings), its adoption in rehabilitation hos-pitals theoretically should have enabledmore direct comparisons of the popula-tions being treated in these treatment set-tings because ADLs items were fairly simi-lar in the nursing home MDS and theMDS-PAC and the MDS-PAC includedIADL items thought to be important inhome health.

INSTRUMENTS

The FIM™ is an 18-item measure thatwas constructed to evaluate and monitorfunctional and cognitive status in inpatientrehabilitation settings. Each of the 18 itemsis rated on a seven-point scale from com-plete dependence (1) to complete indepen-dence (7). The FIM™ is often described ashaving two domains, a motor score domain(13 items) and a cognitive score domain (5items). The FIM™ motor scale was creat-ed by summing the 13 individual motoritem scores and the FIM™ cognitive scalescore and by summing the 5 individualcognitive items. Item scoring is actuallyfairly complex and although the sameseven standard response categories areused for all items, scoring rules differsomewhat by item. For example, the loco-motion item has an explicit distancerequirement and the use of modified dietsfor swallowing affects scoring on the eat-ing item. Safety and the time required tocomplete an activity also influence scoring.


The Uniform Data System for MedicalRehabilitation (UDSMR) developed trainingmaterials, runs a training and certificationprogram, routinely collects FIM™ datafrom participating hospitals, and providesbenchmarking information back to itsmember facilities (Hamilton et al., 1987).As part of the FIM™ training, UDSMR pro-vides a detailed training manual with deci-sion tree-like scoring instructions for thedifferent levels of each item. Additionaltraining materials, called FIM™ Lessons,are also available to help therapists learnthe scoring nuances.

The FIM™ is a measure of disability andburden of care. It was designed for mea-surement by trained clinicians, but wasintended to be discipline free. All 18 itemsmust be completed so any activity that can-not be completed is scored as level 1, totalassistance. Admission scores must becompleted within the first 72 hours afteradmission, but generally refer to perfor-mance over the past 24 hours. Scoringinstructions indicate that the best availableinformation should be used and that directobservation of subject performance is pre-ferred. At the time of this study, roughly60 percent of the industry voluntarily usedthe FIM™ and submitted their data toUDSMR.

Several studies have looked at the validi-ty of the FIM™. Rasch analysis (1980) wasused to compare the scaled measuresacross impairment groups and the analysisprovided support for the two fundamentalconstructs, the motor domain and the cog-nitive domain (Heinemann et al., 1993).Multi-trait scaling and factor analysis wereused to evaluate the FIM™ and providedsupported for the cognitive and motordomains in all 20 impairment categories(Stineman et al., 1996, 1997). Others com-pared FIM™ scores for individuals living atthree different levels of assistance in a con-tinuing care retirement community and

found that as a measure of disability, boththe cognitive and motor scores discrimi-nated across the three care levels in waysthat were consistent with differences inburden of care (Pollak et al., 1996.)Another study used factor analysis onFIM™ scores for a sample of 127 consecu-tive admissions to a French rehabilitationhospital and found support for consideringthree domains within the motor score: self-care, overall body mobility, and sphinctercontrol (Ravaud, Delcey, and Yelnik, 1999.)Construct validity in the FIM™ was evalu-ated by confirming that FIM™ scores var-ied by age, comorbidity, discharge destina-tion, and impairment severity for patientswith stroke and spinal cord injuries (Doddset al., 1993). For specific subgroups suchas patients with multiple sclerosis, trau-matic brain injury, and spinal cord injury,FIM™ scores have been validated againstdisease-specific instruments (Sharrack etal., 1999; Corrigan, Smith-Knapp, andGranger, 1997).

The inter-rater reliability of the FIM™has been assessed in several studies. In anearly study of 89 facilities, unweighteditem-level kappa coefficients ranged from0.53 (moderate agreement) to 0.66 (goodagreement). For the subset of facilities thathad passed a competence exam, scoreswere notably higher ranging from 0.69(good agreement) to 0.84 (excellent agree-ment). Intra-class correlation coefficients(ICC) for the motor domain were 0.96 andthe 0.91 for the cognitive domain(Hamilton et al., 1994). Test-retest reliabil-ity was assessed on 45 cases yielding amotor score ICC=0.9 and cognitive scoreICC=0.8 (Pollak et al. 1996). Inter-rateragreement varied with kappa coefficientsranging from 0.26 (poor agreement) to0.88 (excellent agreement); ICC rangedfrom 0.56 to 0.99 (Sharrack et al., 1999).Another study found that while the totalreliability score was good (0.83), reliability


coefficients across individual items variedmarkedly from 0.02 (poor agreement) to0.77 (very good agreement) (Segal, Ditunno,Staas, 1993).

Several studies have looked at the inter-nal consistency of FIM™ scales. Onefound that the FIM™ had high overallinternal consistency (Dodds et al., 1993).Another found that when viewed across 20diverse impairment categories, the motorand cognitive subscales exceeded mini-mum criteria for item internal consistencyin 97 percent of the tests (Stineman et al.,1996.)

The MDS-PAC is a newer and morecomprehensive instrument than the FIM™.It is intended to measure comparablepatients across a variety of treatment set-tings and to serve as a care planning toolfor each of these groups. Content areas onthe MDS-PAC include demographic admis-sion history, cognitive patterns, communi-cation/vision patterns, mood and behaviorpatterns, functional status, bladder/bowelmanagement, diagnoses, medical complex-ities, pain status, oral/nutritional status,procedures/services used, functional prog-nosis, and resources for discharge.

With its origins in the nursing homeMDS and building on experience with aMDS home care, the MDS-PAC differedsubstantially from the FIM™ in both thebreadth of coverage and in its approach toassessment. The MDS-PAC was viewed asa multi-purpose information-gathering tooland data collectors were instructed to con-sult the patient, the patient’s family, and allcaregivers from all shifts during the first 3days of the patient’s hospital stay, as well asto review the chart. Another differencebetween the instruments was that theFIM™ often instructed scorers to use themost dependent episode, while the MDS-PAC scorers were instructed to collect dataover this longer timeframe and to use amore comprehensive consultation list, but

to allow one or two more dependent episodesbefore scoring patients to a more depen-dent level. The MDS-PAC is scored on aneight-point scale, but scoring is from 0 to 6going in reverse order from independent tototal assistance and allowing for the activi-ty did not occur (score 8).

As a relatively long instrument, theMDS-PAC relies more on written instruc-tions and multiple items for completing theform. An example of this is the treatmentof physical assistance in the performanceof self-care activities. In the FIM™, theamount of physical assistance providedinfluences the level of dependence scored.In contrast, the MDS-PAC first scores thelevel of self-performance and then recordsthe amount of physical assistance receivedin another item. Thus, in order to use theMDS-PAC information to create FIM™motor and cognitive scale scores, rules forcombining MDS-PAC elements into eachof the 18 FIM™ items were needed.

A pilot study of the time to complete theMDS-PAC in rehabilitation hospitalsreported 105 minutes for the first fewassessments, dropping to 85 minutes after10 or more cases. This contrasts with 20-25minutes to complete the FIM™. A pilotinter-rater reliability study of 171 casesfound that average reliability of 315 MDS-PAC items on draft 9 was 0.78 with a rangeof 0.51 to 1.00 (Federal Register, 2000).

The MDS-PAC was developed from theMDS for nursing facility residents, whichhas been translated and used in 15 othercountries and has undergone reliabilitytesting in 6 countries (Hawes et al., 1997;Sgadari et al., 1997). In a multi-State evalu-ation of the MDS, researchers found thatitems in key areas of functional status (cog-nition, ADLs, continence, and diagnoses)had intra-class correlations of 0.7 or high-er, and that 89 percent of all items hadintra-class correlations of 0.4 or higher(Hawes et al., 1995). The construct validity


of the MDS cognitive, ADL, and behaviordomains was examined by comparingthem to the Folstein Mini-Mental StatusExam, the Dementia Rating Scale scores,and the Alzheimers Disease PatientRegistry physician behavior checklist. Thestudy concluded that the MDS datademonstrated reasonable criterion validityfor research purposes (Snowden et al.,1999). A confirmatory factor analysis wasused on MDS data to evaluate five domainswithin the MDS: cognition, ADLs, timeuse, social quality, depression, and prob-lem behaviors. For cognitively intact indi-viduals and all residents together, thedomain clusters except social quality wereconfirmed. For individuals with seriouscognitive impairment, none of the domainswere confirmed (Casten et al., 1998).Construct validity in the MDS was evaluat-ed by testing the confirmed MDS domains(ADLs, cognition, time use, depression,and problem behaviors) against estab-lished clinical research measures. Thatstudy found the majority of their hypothe-ses were confirmed, but the validity coeffi-cients were modest and performance fordepression and problem behaviors was notas good as for ADLs, cognition and timeuse (Lawton et al., 1998).

INFORMATION FOR PATIENT CLASSIFICATION IN PPS

In order to classify patients for paymentin the planned PPS for inpatient rehabilita-tion, one needs to know (1) the rehabilita-tion impairment category (reason for theinpatient rehabilitation admission, e.g.stroke, traumatic brain injury, lowerextremity joint replacement), (2) patientage, (3) the FIM™ motor scale score (thesum of 121 motor items each scored fromtotal assistance (1) to complete indepen-dence (7) and (4) the FIM™ cognitive scale

score (the sum of the 5 cognitive itemseach scored from total assistance (1) tocomplete independence (7)). The first twoitems are recorded using the same formaton both instruments, so we focus on thelatter two elements.

DEVELOPMENT OF THE INITIALTRANSLATION

Knowing the planned substitution of theMDS-PAC for the FIM™ in the inpatientrehabilitation PPS, policymakers took sev-eral steps to facilitate and improve transla-tion from the MDS-PAC to the FIM™.Telephone conferences between the twoinstrument development teams identifiedpotential problem translation areas, lead-ing to both item and scoring refinementsfor the functional status items and to theinclusion of supplemental items.

Either as part of the original MDS-PACdevelopment process or as a result of thetelephone conferences, a number of itemswere changed or refined from their MDScounterparts. Item translation was chal-lenging both because the differences inthe underlying approach to scoring in thetwo instruments and because of the desireto retain comparability with the MDS. Anexample of an item refinement was thedressing item. In the MDS, this is a singleitem. In the FIM™, dressing is two sepa-rate items, one for dressing upper bodyand the other for dressing lower body. TheMDS-PAC uses two dressing items to par-allel the FIM™.

Scoring refinements converted the six-point MDS scores independent (0), super-vision (1), limited assistance (2), extensiveassistance (3), total dependence (4), activi-ty did not occur (8) to an eight-point scale


1 One of the 13 motor items, tub transfer, was dropped from thescale used in the payment system because its relationship tocost was not consistent with other scale items.

by adding a setup help only, and maximalassistance (between extensive assistanceand total dependence). Labels on responselevels helped to establish or maintain com-parability across the instruments.

The scoring for ADL assist codes wasalso changed. On the MDS, the ADL assistcodes are scored—for no personal assis-tance (0)—for one person assistance (1),and—for two or more person assistance(2). On the MDS-PAC this scoring waschanged to weight bearing assistance withone limb (1), two or more person physicalassistance (2) and neither code applies (0).The latter then included both persons withno assistance and those receiving one per-son weight bearing assistance with thetorso or with more than one limb.

Supplemental items added to improvecomparability with the FIM™ included newitems such as distance walked, stair climb-ing in last 24-hours, bladder appliance sup-port, and bowel appliance support.

Policymakers also asked the MDS-PACdevelopers to provide an initial item-by-itemtranslation. For the motor items, obviouscounterparts existed in the two instruments.For these items, the translation reversed theorientation of the MDS-PAC’s scoringscale—independent (0) to total assistance(6)—and mapped it into the correspondingFIM™ numerical values from total assis-tance (1) to complete independent (7) (e.g.MDS-PAC 0 became FIM™ 7). Further, itwas generally agreed that the scoring level 8activity did not occur, was used when indi-viduals were unable to perform a task, so theMDS-PAC 8 was rescored to a FIM™ totalassistance. The physical assistance codeswere only used in the translation for caseswhere the MDS-PAC score was scored max-imal assistance (5) and (1) two or more per-sons were needed for physical assistance.These cases were rescored to 1—total assis-tance (1) on the FIM™ scale.

For the five FIM™ cognitive items (com-prehension, expression, social interaction,problem solving, and memory), there wereno analogous individual items. The MDS-PAC contained a cognitive section withfour items (comatose, memory/recall abil-ity with four subcomponents, cognitiveskills for daily decisionmaking, and indica-tors of delirium) and a communication sec-tion with six items (hearing, models ofcommunication, making self understood,speech clarity, ability to understand others,and vision). A fairly complex multi-item,empirically derived translation that usedboth these and other items was providedby the MDS-PAC development team. Thiscognitive translation was used throughoutthe evaluation study.

EVALUATION STUDY DESIGN

Fifty FIM™-certified rehabilitation facili-ties (out of 180 volunteering hospitals),representing rehabilitation hospitals andunits throughout the country, participatedin the study. These facilities were purpo-sively sampled to represent regions, size,rural-urban, unit-freestanding, and clus-tered geographically to facilitate trainingand data collection. Participating facilitiesranged in size from 13 to 150 beds. Sixteenpercent were rural and 28 percent werefreestanding facilities. All facilities werepreviously certified in FIM™ and were par-ticipants in the UDSMR system. Data col-lectors were teams of one to four clinicians(physical and occupational therapists,nurses, speech language pathologists, etc.)from each site who attended a 2-day MDS-PAC training session and successfully com-pleted a certification exam before data col-lection began. Each facility was asked tocomplete both the FIM™ and the MDS-PAC on all new Medicare admissions withstays beyond 3 days for an 8-week period.


This resulted in over 3,200 FIM™ andMDS-PAC pairs. One or more of threehighly trained calibration teams visitedeach participating hospitals and rescoredboth the FIM™ and the MDS-PAC forthree-eight current cases. Thus for approxi-mately 200 cases we had two FIM™ andtwo MDS-PAC ratings.

Because we needed to train more than250 rehabilitation professionals fromacross the country in a 2-week period, weused a train the trainers model. All studytrainers were trained and certified on theMDS-PAC by the MDS-PAC developmentteam trainers. Study trainers were rehabil-itation professionals who were also FIM™instructors and most had participated inpilot projects on the MDS-PAC. Thus, thiswas their second training session on theMDS-PAC and each had completedapproximately 30 MDS-PAC cases in theearlier studies. All intended to becomeMDS-PAC instructors when the newinstrument became official. The functionalassessment portion of the trainers’ trainingincluded scoring videos and written casestudies and a visit to a rehabilitation facilityfor onsite scoring of actual patients. Eachof these activities was followed by adebriefing session and discussion of therationale for the case scoring.

In nursing homes, the MDS is complet-ed by the nurses. However, in rehabilita-tion hospitals, the FIM™ was completed byrehabilitation professionals, sometimes asingle individual, but often an interdiscipli-nary team with physical therapists com-pleting the mobility items, occupationaltherapists completing the self-care items,rehabilitation nurses completing the boweland bladder items, and speech languagepathologists completing the communica-tion and language items. We did not wantto assume that nurses (the MDS data col-lection model) would replace the rehabili-tation specialists so we asked hospitals to

send four-person data collection teams fortraining. The study data collection teams,each with one-four members2, includedpracticing rehabilitation professionals(physical and occupational therapists,nurses, speech language pathologists,recreation therapists) from each of the 50hospitals in the study. Each team attendedthe 2-day training, completed a post-train-ing assignment, and went through a tele-phone certification process conducted bythe full-time study team field coordinator.The functional assessment training includ-ed scoring videos and written case studieseach followed by a debriefing to discussthe rationale for each score.

The field coordinator maintained regularcontact with the study hospitals. In addi-tion, an 800 telephone number was provid-ed so the scoring teams could call in ques-tions to the field coordinator, who was sup-ported by the MDS-PAC and FIM™ devel-opment team trainers. A document of fre-quently asked questions and answers wasmaintained on a study Web page and peri-odically distributed to data collectors.Regular newsletters with information onstudy progress, procedural updates, andscoring clarifications were sent to all datacollectors.

The three calibration teams spent amonth training in Boston. During the firstweek, they were trained and certified onthe MDS-PAC by the MDS-PAC develop-ment staff trainers. They were also re-trained and re-certified on the FIM™ bythe FIM™ development group trainers.The functional assessment portion of theirtraining mimicked that of the trainers withscoring videos, written case studies, and asite-visit to score actual patients each fol-lowed by a debriefing session. Calibration


2 Each hospital was asked to send one or more four-person teamsto train, including a physical and an occupational therapist, anurse, a speech language pathologist or other rehabilitationprovider. Some hospitals sent only three-person teams. Actualdata collection teams varied from one to four people.

team members spent the next 3 weeksworking together in different cross-disciplinary combinations and rotatingthrough four rehabilitation hospitals in thegreater Boston area practicing both theMDS-PAC and the FIM™. These rotationswere used to standardize scoring across allcalibration team members and also provid-ed necessary experience entering unfamil-iar institutions and establishing proce-dures for the assessments. Final teamassignments were made near the end ofthe training.

Refining the Translation

The first study task was to compare theactual FIM™ motor scale and item scoreswith those obtained from the MDS-PACtranslations and summated scales. Themean FIM™ cognitive scale score wasquite close to the mean PAC translation,28.50 compared to 28.51. However, themean FIM™ motor scale score differedfrom the mean PAC motor scale translationby nearly 5 points, 45.46 compared to50.26. Individual motor items with thelargest mean scoring differences were thelocomotion item with a mean difference ofmore than 1.5 points on the seven-pointscale, and grooming and toileting withmean differences of more than 0.5 points.

The evaluation team undertook aprocess to review and refine the transla-tion. We began with a review of the instru-ments’ scoring sheet instructions and thescoring manuals for both the FIM™ andthe MDS-PAC. We benefited from our par-ticipation in the training courses, the certi-fication process, and a review of the fre-quently asked questions. These all high-lighted areas where the approach to scor-ing differed in the two instruments. Afterdiscrepant areas and possible refinementswere identified, they were tested empirical-ly to confirm that they led to better scoring

agreement. Scoring agreement was mea-sured with Pearson correlations3, andweighted and unweighted kappa statistics.Occasionally, scoring changes improveditem level agreement on kappas and corre-lations, but not on item level means. Aslong as these also improved scale levelmean comparisons, they were retained.

There were both fundamental and item-specific differences between the twoassessment tools that we knew could notbe overcome in any translation. First, thereference timeframe in the MDS-PAC is a3-day lookback conducted on day 4, but theFIM™ scoring takes place anytime in thefirst 72 hours and references only the last24 hours. Second, the FIM™ generallydirects assessors to score the most depen-dent episode during this 24-hour period,while the PAC assessors are instructed toallow one or two more dependent episodeswithout scoring the more dependent level.Third, the MDS-PAC definition of totalassistance (full staff performance of activi-ty during entire period) was much morerestrictive that the FIM™ definition(patient performs less than 25 percent ofthe effort). Fourth, the MDS-PAC includ-ed transfers on and off the bedpan as partof the toilet transfer item, but the FIM™does not. (Buchanan et al. [2003] providemore complete listing and discussion.)

Realigning the Seven Scoring Levels

The study team review found that theseven scoring levels of the FIM™ and thetranslated MDS-PAC did not align properly(Table 1). The FIM™ scoring levels differ-entiated between complete independence(7) and modified independence (6). A FIM™level 7 indicated that the activity was per-


3 Pearson correlations quantify the linear association betweentwo measures and thus indicate how accurately one measurecan be predicted from another. They are not, however, truemeasures of agreement as two items or scales can be perfectlycorrelated, but have few, if any agreed values.

formed safely and completely independentand without assistive devices. Modifiedindependence (level 6) was used whenthere were safety concerns, or the patientrequired extra time (three times normal),or the patient used an assistive device inorder to perform the activity completelyindependent. The MDS-PAC motor itemshave a single score for independence (0),regardless of the equipment used or themanner in which the activity was per-formed. For some motor items, the MDS-PAC devices/aids items could be used todetermine if an item scored as independentshould be scored as modified indepen-dence. For example, if the patient usedadaptive eating utensils and was scoredindependent (0), then the revised transla-tion converted the MDS-PAC score to mod-ified independence (6) on the FIM™ scale.

When the device item was not sufficientto separate cases that should be scored asmodified independence from those that aretruly independent, then both groups werescored at the most likely FIM™ level (6 or7) at admission based on the current sam-ple and confirmed against historical FIM™data. Thus, if in the FIM™ more caseswere scored as modified independence (6)than as complete independence (7) at

admission, then we revised the translationto rescore the group as a 6.

Our review also found that the FIM™included both set up and supervision in thesame score while the MDS-PAC used dif-ferent scores for set up (1) and supervision(2). Thus, in the revised translation, MDS-PAC scores of 1 and 2 were mapped to aFIM™ score 5 (Table 1).

Incorporating ADL Assist Codes

After realigning the scoring categories,our review also concluded that the use ofphysical assistance had not been fully cap-tured in the original translation. TheFIM™ incorporates the use of physicalassistance into the actual item scoring lev-els. In contrast, the MDS-PAC scores self-performance of an activity and the use ofphysical assistance in the activity separate-ly. The ADL assist code section, particu-larly the one limb assist code, does not cor-respond precisely to the FIM™ assistanceconcepts and is therefore, much more dif-ficult to incorporate into the translation.Since one limb assist is weight-bearingassistance, the translation adopted the rulethat the maximum FIM™ score a patientcould have with an ADL assist code of 1


Table 1

Revised Scoring Correspondence for Converting the MDS-PAC Self-Performance Motor Items toFIM™ Like Scores

MDS-PAC Item Scoring Level FIM™ Item Scoring Level

0 Independent 7 Complete Independence (Timely, Safe) 6 Modified Independence (With Device)

1 Set Up Help Only 5 Supervision 2 Supervision

3 Minimal Assistance (Limited Assistance) 4 Minimal Assistance

4 Moderate Assistance (Extensive Assistance) 3 Moderate Assistance

5 Maximal Assistance 2 Maximum Assistance

6 Total Assistance (Total Dependence) 1 Total Assistance 8 Activity Did Not Occur

NOTES: MDS-PAC is minimum data-set post-acute care. FIM™ is functional independence measure.

SOURCES: Buchanan, J. L. and Zaslavsky, A. M., Harvard Medical School; Andres, P. L., and Haley, S. M., Boston University; and Paddock, S. M., RAND.

was a FIM™ 4, minimum assistance. Thus,if a functional status item was scored 0, 1,or 2 which would translate to FIM™ 7, 6, or5, but the ADL assist code was 1, then therevised translation rescored the item to aFIM™ 4. For more dependent scores, PAC3-6, an ADL assist code of 1 did not affectthe scoring.

Because the FIM™ is a burden of careinstrument, any activity needing the assis-tance of two persons is always scored 1(total assistance). The revised translationrescored any item where two person phys-ical assistance was used, regardless of thePAC self-performance score, to total assis-tance.

Item-Specific Translation Revisions

As previously noted the locomotion itemhad the largest mean discrepancy, 1.5points between the mean FIM™ score andthe mean MDS-PAC score, so we carefullyreviewed the scoring instructions on thisitem. FIM™ scoring rules instruct ratersto score patients for locomotion perfor-mance at admission, by using the mode(walk or wheelchair) expected to be usedmost frequently at discharge. It requiresthat the patient move at least 150 feet fornearly all ratings above maximum assis-tance. The MDS-PAC scores locomotionusing the most common mode at admis-sion and has no distance criterion. Wecompared the MDS-PAC locomotion withthe walk in facility item and found that thelatter had substantially better agreementwith FIM™ scores and could be used incombination with the distance item.However, by using the walk in facility itemin the revised translation, we could not dif-ferentiate those expected to use a wheel-chair at discharge from ambulators in thetranslation. The former comprised lessthan 15 percent of the patient population.

FIM™ scoring rules for the bladder andbowel management items are quite com-plex. Raters are asked to consider howthey would score the level of assistancewith bladder (bowel) management on thestandard 1-7 scale and then to also considerhow they would score the frequency of acci-dents on the same scale, but they recordonly the minimum of the two scores. TheMDS-PAC on the other hand used threeitems, but with differing timeframes forbladder and three for bowel and recordsthe scoring for each. The original transla-tion reversed the orientation to FIM™ scor-ing order and took the minimum of the twoscores. Our revised translation incorporat-ed the information on the specific appli-ances being used to aid in rescoring.

Another problem was the MDS-PACscoring for medication use in the bladderand bowel appliance support section.When a nurse passes the medication to thepatient, PAC scoring instructions were toscore this as maximal assistance. Sincemedications are routinely controlled bynursing staff in the acute rehabilitation set-ting, regardless of the patient’s ability toparticipate in this activity, all patientsreceiving medications got scored as maxi-mal assistance. Further, during training,assessors were directed to treat fiber sup-plements, which virtually all patientsreceive, as medications. The revised trans-lation attempted to rescore these cases.

Evaluating the Translation

We used factor analysis on the combinedset of motor items from both the FIM™and the MDS-PAC. We repeated the factoranalysis, first replacing the motor itemsfrom the MDS-PAC with their translationcounterparts and then again with therevised translation items. We did this toassess whether the translation and its revi-


sion improved the conceptual agreementbetween analogous items in the two instru-ments. We found that neither the rawitems nor those from the original transla-tion all loaded onto the same factors as thecorresponding FIM™ items, while itemsfrom the revised translation did.

The revised translation reduced the meandifference in motor scores between theFIM™ and the MDS-PAC by 50 percent fromthe original translation (Table 2). Despitethe improvement, we found that the agree-ment between the instruments for institu-tionally-based scoring teams was only mod-erate and absolute agreement was worse.However, when the calibration teams scoredpatients using both instruments, we foundnotably higher levels of agreement.

We used regression analysis to analyzescoring differences and found that after con-trolling for administrative factors andpatient, and hospital characteristics, that arandom effect for hospitals was significant.This implies that scoring differences variedby hospital and this variation was notexplained by any of the independent vari-ables. The effect was substantial enough tobe of concern for the comparability of scor-ing procedures across facilities and suggeststhat more training is needed to adequatelystandardize the assessment process.

The ultimate test to determine how wellthis instrument substitution would workmapped each case into its payment cell,first using the FIM™ motor and cognitivescale scores and then using the translated


Table 2

FIM™ and PAC-FIM™ Mean Scores, by Type of Assessment Team1,2,3

Assessment TeamsInstitutional Calibration

FIM™ PAC-FIM™ FIM™ PAC-FIM™Mean Mean Mean Mean

Item Scores Scores Scores Scores

MotorEating 5.51 5.54 5.73 ***5.77Grooming 4.73 ***4.88 4.61 4.56Dressing-Upper Body 3.24 *3.30 3.27 ***3.03Dressing-Lower Body 4.25 ***4.35 3.92 3.95Toileting 2.99 ***3.21 2.80 2.77Bladder Management 3.37 ***3.71 3.66 **3.49Bowel Management 4.29 4.27 4.61 ***4.15Transfer-Bed/Chair 4.70 ***5.20 5.33 *5.30Transfer-Toilet 3.58 ***3.70 3.56 ***3.32Transfer-Tub/Shower 3.28 ***3.67 3.60 **3.44Locomotion-Walk/Wheelchair 1.96 1.98 1.86 **2.05Stairs 2.22 2.20 2.51 2.38Motor Scale 45.46 ***47.82 46.80 ***45.76

CognitiveComprehension 5.87 ***5.93 5.88 5.86Expression 5.97 5.99 5.93 5.87Social Interaction 5.91 ***5.63 6.04 ***5.54Problem Solving 5.32 5.34 5.21 5.26Memory 5.37 ***5.56 5.34 5.44Cognitive Scale 28.50 28.51 28.53 ***28.07

***p<=0.001.

**p<=0.01.

*p<=0.05.1 Greater score equals greater level of independence.2 Uses revised motor item translation and the original cognitive item translation.3 Statistically significant difference between the FIM™ and the MDS-PAC translated mean scores.

NOTES: FIM™ is functional independence measure. PAC-FIM™ is minimum data set-post acute care translated into FIM™-like items.

SOURCES: Buchanan, J. L. and Zaslavsky, A. M., Harvard Medical School; Andres, P. L., and Haley, S. M., Boston University; and Paddock, S. M., RAND.

MDS-PAC motor and cognitive scalescores. We tried several different empiri-cal adjustments to improve the matchbetween the mappings. Under all of theseadjustments, the level of classificationagreement was low and clearly not ade-quate for payment purposes. Further, wefound that a substantial proportion of thefacilities would experience potentiallyimportant shifts in revenue. As a conse-quence, policymakers opted to retain theFIM™.

DISCUSSION

This translation effort failed to achievesufficient accuracy for use in the plannedpayment system despite several importantadvantages: (1) the issue of translationfrom the MDS-PAC to FIM™ scales wasaddressed before finalizing the instrument,(2) formal communication aimed at facili-tating the translation was establishedbetween the two instrument developmentgroups, (3) the study team and its trainersbenefited from training from the instru-ment development group trainers, and (4)data collection teams and the study teamhad multidisciplinary input from rehabilita-tion specialists. While it may be that trans-lations needed for applications such asquality monitoring and outcomes manage-ment will not require such stringent levelsof accuracy, these findings suggest weshould be cautious regarding our ability tomake such substitutions.

Much of the prior research on instru-ment performance has been in researchsettings or undertaken by research staff.This study aimed to test what might hap-pen under national clinical implementation.Clinicians practicing in their regular envi-ronment performed one set of assess-ments and centrally trained calibrationteams visited institutions for repeat assess-ments.

Both instruments had strengths and lim-itations, and their differences often provid-ed important insights. The FIM™ was con-cise, which facilitated its widespread volun-tary adoption and allowed a greater focuson standardization. Rehabilitation profes-sionals told us that FIM™ language wasroutinely used to describe patients as pro-fessionals communicated with one anotherthroughout the field. Documentationincluded both a training manual and sever-al sets of FIM™ lessons, training tools thatwere designed to illustrate scoring rules.

The major limitation of the FIM™ was itsscoring complexity, which was masked bydeceptively simple data collection forms.The scoring rules often required implicitintegration of several concepts with no wayof diagnosing what aspects of the integra-tion created scoring difficulties. The loco-motion item illustrates this complexity.Assessors must score patients at admissionusing the mode of locomotion that isexpected at discharge. Further, distancetraveled and the amount of physical assis-tance must be incorporated into the scoringlevels. Similarly, in the eating item, whenswallowing problems requiring diet modifi-cations are present, patients who are other-wise independent in eating should bescored as modified independent. While theFIM™ scoring rules were clearly stated inthe training manuals and well illustrated inthe FIM™ lessons, the proprietary natureof these documents precluded open dis-semination and discussion. This probablycontributed to our findings of scoring prac-tices that differed from one institution tothe next even after controlling for observ-able patient and facility characteristics.

The MDS-PAC form contained moreexplicit instructions for scoring each itemand tended to use multiple items to buildcomplex constructs. While this made uscombine the items to achieve FIM™-likescoring, it enabled the study team to diag-


nose some problem areas. For example,when we looked at scoring reliability forthe translation, we were able to look at thereliabilities of the translated item and ofeach of its component parts. For the func-tional status items, the physical assistanceitems had much lower scoring reliabilitiesthan the self-performance items.

Having the explicit components in theMDS-PAC also enabled us to determinethat FIM™ scorers did not always incorpo-rate concepts such as distance and modi-fied diets into their scoring. This led us torecommend that these items be added tothe officially adopted assessment tool toremind scorers to consider them. Atworst, their inclusion allows us to explicitlyidentify this as a scoring problem withpotential for correction.

Other strengths of the MDS-PACinclude: (1) its greater breadth of coverageof substantive areas, (2) its potential foruse in care planning and monitoring quali-ty of care, and (3) its potential for applica-tion across multiple settings.

Its major limitations are its length and itslack of standardization in response formatsand timeframes. Much of the training ses-sion was devoted to the mechanics of theform, that is, where to put checks versusnumerical scores and where blanks wereallowed, leaving inadequate time to addressdefinitional clarity and standardization inassessment. For example, the rehabilita-tion professionals clearly wanted additionalguidance on selecting functional status per-formance levels (especially distinguishingminimal, moderate, maximum, and totalassistance). They also wanted clearer def-initions for terms like partial versus fullloss of voluntary motor control. An exam-ple of the lack of standardization on time-frame is in the bowel management section.Bowel continence is coded over the last 7-14 days even though on the admission

assessment, bowel appliances are codedover the last 3 days and bowel appliancesupport is coded over the last 24 hours.

In conclusion, this evaluation clearlypoints to the need for a unified commonconceptual framework and a rigorous stan-dardized approach to the content of func-tional assessment measures and to theassessment techniques used. Standardiza-tion may be facilitated when the item set isrelatively small and scoring rules are sim-plified or when complex scoring rules canbe broken down into simpler, explicitsteps. These findings should be kept inmind as we move forward with the devel-opment, refinement, and evaluation of theInternational Classification of Functioning.

REFERENCES

Applegate, W.B., Blass, J.P., and Williams F.T.:Instruments for the Functional Assessment ofOlder Patients. New England Journal of Medicine322(17):1207-1214, 1990.Branch, L.G., and Meyers, A.R.: Assessing PhysicalFunction in the Elderly. Clinics in GeriatricMedicine 3(1):29-51, 1987.Buchanan, J.L., Andres, P. L., Haley, S.M., et al.:Final Report on the Instruments for PPS. MR-1501-CMS. RAND. Santa Monica, CA. 2002.Carter, G.M., Buntin, M.B., Hayden, O., et al.:Analyses for the Implementation of Medicare’sInpatient Rehabilitation Prospective PaymentSystem. MR-1501-CMS. RAND. Santa Monica, CA.2002.Casten, R., Lawton, M.P., Parmelee, P.A., andKleban, M.H.: Psychometric Characteristics of theMinimum Data Set I: Confirmatory Factor Analysis.Journal of the American Geriatric Society 46(6):726-735, June 1998.Corrigan, J.D., Smith-Knapp, K., and Grange, C.V.:Validity of the Functional Independence Measurefor Persons With Traumatic Brain Injury. Archivesof Physical Medicine and Rehabilitation 78(8):828-834, August 1997.Dodds, T.A., Martin, D.P., Stolov, W.C., and Deyo, R.A.: A Validation of the Functional IndependenceMeasurement and Its Performance AmongRehabilitation Inpatients. Arch Phys MedRehabilitation 74(5):531-536, May 1993.


Federal Register: Medicare Program: ProspectivePayment System for Inpatient RehabilitationFacilities; Proposed Rule. 65 FR 26281, May 5,2000. Fries, B.E., Schneider, D.P., Foley, W.J., et al.:Refining a Case-Mix Measure for Nursing Homes:Resource Utilization Groups (RUG III). MedicalCare 32(7):668-685, July 1994.Granger, C.V., Hamilton, B.B., Keith, R.A., et al.:Advances in Functional Assessment for MedicalRehabilitation. In Lewis, C.B. (ed.): Topics inGeriatric Rehabilitation 1:59-74, 1986.Hamilton, B.B., Granger, C.V., Sherwin, F.S., et al.:A Uniform National Data System for MedicalRehabilitation. In Fuhrer, M.J. (ed.): RehabilitationOutcomes: Analysis and Measurement. Baltimore,MD. Brookes Publishing Co. 1987.Hamilton, B.B., Laughlin, J.A., Fiedler, R.C., andGranger, C.V. Inter-rater Reliability of the SevenLevel Functional Independence Measure (FIM™)Scandinavian Journal of Rehabilitation Medicine26:115-119, 1994.Hawes, C., Morris, J. N., Phillips, C. D., et al.:Development of the Nursing Home ResidentAssessment Instrument in the U.S.A. Age andAging 26(Suppl 2):19-25, September 1997.Hawes, C., Morris, J. N., Phillips, C. D., et al.:Reliability Estimates for the Minimum Data Sets forNursing Home Resident Assessment and CareScreening (MDS). The Gerontologist 35(2):172-178,April 1995.Heinemann, A.W., Linacre, J.M., Wright, B.D., et al.:Relationships Between Impairment and PhysicalDisability as Measured by the FunctionalIndependence Measure. Archives of PhysicalMedicine and Rehabilitation 74(6):566-573, June 1993.Hittle, D. F., Crisler, K. S., Beaudry, J. M. et al.:OASIS and Outcome-Based Quality Improvement inHome Health Care: Research and DemonstrationFindings, Policy Implications, and Considerations forFuture Change. University of Colorado HealthSciences Center. March 2002. Internet address:www.cms.hhs.gov/providers/hha.(Accessed 2003.)Hosek, S., Kane, R., Carney, M., et al.: Charges andOutcomes for Rehabilitative Care: Implications forthe Prospective Payment System. R-3424-HCFA.RAND. Santa Monica, CA. 1986. Institute of Medicine Study: Improving the Qualityof Care in Nursing Homes. National Academy ofSciences Press. Washington, DC. 1986.Jette, A.M.: How Measurement TechniquesInfluence Estimates of Disability in OlderPopulations. Social Science and Medicine 38(7):937-942, July 1994.

Katz, S., Ford, A.B., Moskowitz, R.W., et al.: Studiesof Illness in the Aged. The Index of the ADL: AStandardized Measure of Biological andPsychological Functioning. Journal of the AmericanMedical Association 185(12):914-919, September1963. Lawton, M.P., and Brody, E.M.: Assessment ofOlder People: Self-Maintaining and InstrumentalActivities of Daily Living. Gerontologist 9(3):179-186,September 1969.Lawton, M.P., Casten, R., Parmelee, P.A., et al.:Psychometric Characteristics of the Minimum DataSet II: Validity. Journal of the American GeriatricSociety 46(6):736-744, June 1998. McDowell, I. and Newell, C.: Measuring Health: AGuide to Rating Scales and Questionnaires. OxfordUniversity Press, New York, NY. 1996. Morris, J.N., Hawes, C., Fries, B.E., et al.:Designing the National Resident AssessmentInstrument for Nursing Homes. The Gerontologist30(3):293-307, June 1990.Pollak, N., Rheault, W., and Stoecker, J.L.:Reliability and Validity of the FIM™ for PersonsAged 80 Years and Above From a MultilevelContinuing Care Retirement Community. Archivesof Physical Medicine and Rehabilitation 77(10):1056-1061, October 1996.Rasch, G.: Probabilistic Models for Some Intelligenceand Attainment. University of Chicago Press.Chicago, IL. 1980. Ravaud, J.F., Delcey, M., and Yelnik, A.: ConstructValidity of the Functional Independence Measure(FIM™): Questioning the Unidimensionality of theScale and the Value of FIM™ Scores. ScandinavianJournal of Rehabilitation Medicine 31(1):31-41,March 1999.Segal, M.D., Ditunno, J.F., and Staas, W.E.:Interinstitutional Agreement of IndividualFunctional Independence Measure (FIM™) ItemsMeasured at Two Sites on One Sample of SCIPatients. Paraplegia 31:622-631, 1993.Sgadari, A., Morris, J.N., Fries, B.E., et al.: Effortsto Establish the Reliability of the ResidentAssessment Instrument. Age and Aging 26(Suppl2):27-30, September 1997.Sharrack, B., Hughes, R.A.C., Soudain, S., andDunn, G.: The Psychometric Properties of ClinicalRating Scales Used in Multiple Sclerosis. Brain122(1):141-159, January 1999.Snowden, M., McCormick, W., Russo, J., et al.:Validity and Responsiveness of the Minimum DataSet. Journal of the American Geriatric Society 47(8):1000-1004, August 1999.


Stineman, M.G., Goin, J.E., Hamilton, B.B., andGranger, C.V.: A Case Mix Classification Systemfor Medical Rehabilitation. Medical Care 32(4):366-79, April 1994.Stineman, M.G., Shea, J.A., Jette, A., et al.: TheFunctional Independence Measure: Tests of ScalingAssumptions, Structure, and Reliability Across 20Diverse Impairment Categories. Archives ofPhysical Medicine and Rehabilitation 77(11):1101-1108, November 1996.Stineman, M.G., Jette, A.J., Fiedler, R.C., andGranger, C.V.: Impairment-Specific DimensionsWithin the Functional Independence Measure.Archives of Physical Medicine and Rehabilitation78(6):636-643, June 1997.

Teresi, J.A., Lawton, M.P., Holmes, D., and Ory, M.(eds.): Measurement in Elderly Chronic CarePopulations. Springer Publishing, New York, NY.1997.Williams, B.C., Li, Y., Fries, B.E., and Warren, R.L.:Predicting Patient Sores Between the FunctionalIndependence Measure and the Minimum Data Set:Development and Performance of a FIM™-MDS“Crosswalk”. Archives of Physical Medicine andRehabilitation 78(1):48-54, January, 1997.

Reprint Requests: Joan L. Buchanan, Ph.D., Department ofHealth Care Policy, Harvard Medical School, 180 LongwoodAvenue, Boston, MA 02115 E-mail: [email protected]


an assessment tool translation study

Documents