This may be the author’s version of a work that was submitted/acceptedfor publication in the following source:

Ash, Susan, O’Connor, Jackie, Anderson, Sarah, Ridgewell, Emily, &Clarke, Leigh(2015)A mixed-methods research approach to the review of competency stan-dards for orthotist/prosthetists in Australia.International Journal of Evidence-Based Healthcare, 13(2), pp. 93-103.

This file was downloaded from: https://eprints.qut.edu.au/91967/

c© Consult author(s) regarding copyright matters

This work is covered by copyright. Unless the document is being made available under aCreative Commons Licence, you must assume that re-use is limited to personal use andthat permission from the copyright owner must be obtained for all other uses. If the docu-ment is available under a Creative Commons License (or other specified license) then referto the Licence for details of permitted re-use. It is a condition of access that users recog-nise and abide by the legal requirements associated with these rights. If you believe thatthis work infringes copyright please provide details by email to qut.copyright@qut.edu.au

Notice: Please note that this document may not be the Version of Record(i.e. published version) of the work. Author manuscript versions (as Sub-mitted for peer review or as Accepted for publication after peer review) canbe identified by an absence of publisher branding and/or typeset appear-ance. If there is any doubt, please refer to the published source.

https://doi.org/10.1097/XEB.0000000000000038

1

AMixedMethodsResearchApproachtotheReviewofCompetencyStandardsfor1

Orthotist/ProsthetistsinAustralia.2

34SusanAsh,Ph.D,M.H.P.,Dip.Nut.Diet.,B.Sc.,Consultant,QueenslandUniversityof5

Technology,Brisbane,Queensland,Australia.6

7

JackieO’Connor,B.P.O.(Hons),ProjectOfficer,AustralianOrthoticProsthetic8

Association;POBox1219Greythorn,Vic3104,Australia.9

10

SarahAnderson,M.P.H.,B.P.O.,Lecturer,NationalCentreforProstheticsand11

Orthotics,LaTrobeUniversity,KingsburyDrive,Bundoora,Vic3086,Australia.12

13

EmilyRidgewell,Ph.D.,B.P.O.(Hons),Registrar,AustralianOrthoticProsthetic14

Association;POBox1219Greythorn,Vic3104,Australia.15

16

LeighClarke,M.P.H.,B.P.O.(Hons),ExecutiveOfficer,AustralianOrthotic17

ProstheticAssociation;POBox1219Greythorn,Vic3104,Australia.18

19

20

21

Acknowledgements:Theworkundertakeninthismanuscriptwasfundedbya22

ProfessionalServicesDevelopmentProgramgrantfromtheAustralian23

Government.ThanksareextendedtoShaneGrant,projectofficerforStage1of24

theproject.25

2

AMixedMethodsResearchApproachtotheReviewofCompetencyStandardsfor26

Orthotist/ProsthetistsinAustralia.27

Aim:Therequirementforanalliedhealthworkforceisexpandingastheglobal28

burdenofdiseaseincreasesinternationally.Tosafelymeetthedemandforan29

expandedworkforceoforthotist/prosthetistsinAustralia,competencybased30

standards,whichareup‐to‐dateandevidencebased,arerequired.Theaimsof31

thisstudyweretodeterminetheminimumlevelforentryintothe32

orthotic/prostheticprofession;todevelopentrylevelcompetencystandardsfor33

theprofessionand;tovalidatethedevelopedentrylevelcompetencystandards34

withintheprofessionnationally,usinganevidencebasedapproach.35

Methods: A mixed methods research design was applied, using a three step36

sequentialexploratorydesign,wheresteponeinvolvedcollectingandanalyzing37

qualitative data from two focus groups; step two involved exploratory38

instrument development and testing, developing the draft competency39

standards; and step three involved quantitative data collection and analysis, a40

Delphisurvey.InStage1(steps1and2),thetwofocusgroups,anexpertanda41

recent graduate group of Australian orthotist/prosthetists, were led by an42

experiencedfacilitator,toidentifygapsinthecurrentcompetencystandardsand43

then to outline a key purpose, work roles and tasks for the profession. The44

resulting domains and activities of the first draft of the competency standards45

were synthesised using thematic analysis. In Stage 2 (step 3), the draft46

competencystandardswerecirculatedtoapurposivesampleofthemembership47

of theAustralianOrthotic Prosthetic Association, using three rounds of Delphi48

survey.Aprojectreferencegroupoforthotist/prosthetistsreviewedtheresults49

ofbothstages.50

3

Results: In Stage 1, the expert (n=10) and the new graduate (n=8) groups51

separatelyidentifiedworkrolesandtasks,whichformedtheinitialdraftofthe52

competency standards. Further drafts were refined and performance criteria53

added by the project reference group resulting in the final draft competency54

standards.InStage2,thefinaldraftcompetencystandardswerecirculatedto5655

members(n=44finalround)oftheAssociation,whoagreedonthekeypurpose,56

6 domains, 18 activities and 68 performance criteria of the final competency57

standards.58

Conclusion: This study outlines a rigorous and evidence‐basedmixedmethods59

approach for developing and endorsing professional competency standards,60

whichisrepresentativeoftheviewsoftheprofessionoforthotist/prosthetists.61

4

Background:62

TheWorldHealthOrganisation(WHO)WorldReportonDisabilityrecommends63

that rehabilitation services are multidisciplinary, effective and accessible to64

those with disabilities. Approximately 15% of the world’s population65

experiences some form of disability. The disabling barriers include lack of66

servicesforhealthcareandrehabilitationbutalsopoorcoordinationofservices,67

inadequatestaffingandweakstaffcompetencies(1).68

Orthotist/prosthetists are tertiary qualified allied health professionals who69

clinically assess the physical and functional attributes of individuals with70

mobility and functional limitations. These limitations may result from illness,71

injuryand/ordisability,includinglimbamputations.Orthotist/prosthetistsmay72

then prescribe and facilitate the provision of orthoses and prostheses to73

minimize the effect of these limitations. Australia currently has an74

orthotist/prosthetist workforce of approximately 400 practicing clinicians,75

whichrepresentsalowratiopercapita (1/56,552ofpopulation)(2)compared76

totheUKbenchmarkfororthotistsalone,of1/30,555ofpopulation(3).Health77

workforce shortages have been recognized around the globe, however simply78

trainingmoreof the same typeofhealthworkforcewillnotmeet theneedsof79

complex health environments. The WHO has recommended cooperation80

between educational institutions and health employing agencies to match81

professionaleducationtohealthservicedelivery(4).Oneofthekeydeliverables82

fromtheAustralianHealthWorkforceStrategicReformforActionistocreatean83

adaptablehealthworkforce that is equippedwith the competencies toprovide84

teambasedandcollaborativemodelsofcare(5).85

5

TheAustralianOrthoticProstheticAssociation(AOPA)developedtheiroriginal86

competencystandardsin1999,withsmallchangesamendedin2003(6),87

howeverneitherprocesshadinvolvedanendorsementprocessbythe88

membership.Trainingfortheprofessionisfocusedinoneuniversityandthe89

numberofgraduatesfromthissourceisnotlikelytomeetthecurrentorfuture90

workforcedemandscreatedbyanincreasinglyageingpopulationandincreased91

ratesofchronicdisease.Up‐to‐datecompetencystandardsarethusrequiredto92

describe21stcenturypracticeofAustralianorthotist/prosthetistsandtoensure93

anyinternationallytrainedorthotist/prosthetists,wishingtofillworkforcegaps94

andpracticeinAustralia,havemettransparentstandards.95

Mixed methods research can address exploratory and confirmatory questions96

usingbothqualitativeandquantitativeapproaches(7).Narrativeinterviewsand97

guided focus groups have been used in health professions to explore themes98

around professional competencies (7‐10). Thematic analysis of interview data99

canprovidearigorousbutflexiblemethodforundertakingqualitativeresearch100

thatismethodologicallysound(11).BraunandClarkesuggestthatusingwritten101

transcripts from initial data collection to assignmeanings or codes; clustering102

thesemeanings into themes; integrating the results into an exhaustivemap of103

theanalysis;andthenvalidatingtheresultswithextractsandanalysisfromthe104

relevantliterature;providessucharigorousapproach(11).105

TheDelphiprocessisusedtodeterminethelevelofconsensusonanissueand106

uses a quantitative approach. It attempts to overcome the disadvantage of107

domination of the group by individuals or those with a vested interest. The108

extent to which each respondent agrees with the issue is measured on a109

numerical or categorical scale. Surveys are generated to explore the issue110

6

involved over numerous rounds, usually three,with participants anonymously111

ratingtheirlevelofagreement(12).ThefirstroundoftheDelphiofteninvolves112

expertsprovidingtheiropinionsonaspecificmatterandthesubsequentrounds113

providingthelevelofagreement.Criticsofthismethodpointoutthatagreement114

doesnotnecessarilymeanthatthecorrectinterpretationhasoccurred.Despite115

these reservations, the Delphi process has been used globally in health116

professionalcompetencydevelopment,includingnursing(13)andnutritionand117

dietetics (14, 15). The reactive Delphi is a variation of the traditional Delphi118

whereinformationispresenteddirectlyintothefirstroundofthesurvey.Whilst119

the reactive Delphi may constrain initial opinion (16), this technique is often120

utilisedsubsequenttopreviousresearch(13,16,17).121

Theaimof thispaper is todescribe themixedmethods researchmethodology122

used by AOPA to determine the minimum level for entry into the123

orthotic/prosthetic profession in Australia; to develop entry level competency124

standards for the profession; and to validate the developed entry level125

competencystandardswithintheprofessionnationally. 126

Methods:127

Amixedmethodapproach,usingathreestepsequentialexploratorydesignwas128

used,wheresteponeinvolvedcollectingandanalyzingqualitativedatafromtwo129

focus groups; step two involved exploratory instrument development and130

testing, developing the draft Competency Standards; and step three involved131

quantitative data collection and analysis, a Delphi survey. The study was132

conductedintwostagesandwasoverseenbyaprojectreferencegroup,which133

included the representatives of the professional association executive, those134

involvedinresearchanduniversitytraining,aprojectofficerandanexperienced135

7

facilitator.Stage1(February–May,2013)involvedtheidentificationofgapsin136

the 2003 Competency Standards and development of the key purpose of the137

profession and the domains, activities and performance indicators of the draft138

competency standards. Stage 2 (November, 2013 – April 2014) involved the139

confirmationorvalidationofthesecompetencystandardsandtheirfinalization140

(seeFigure1).Ethicsapprovalwasgrantedbythe<deidentified>HumanEthics141

Committee.142

143

Figure1nearhere.144

145

In Stage 1, semi‐structured focus groupswere used to elicit themes about the146

gaps,keypurpose,workrolesandtasksofAustralianorthotist/prosthetistsfrom147

twogroups,anexpertgroupandarecentgraduategroup.Inthispaper,wehave148

defineddomainasaworkrole,sometimesreferredtointheliteratureasaunit149

of competencyor standard,activityasa taskperformedwithin thatworkrole,150

sometimes referred to as a core competency or element and performance151

indicatorasastatementofhowtheactivityortaskwouldbemeasured.152

A series of questions, shown in Table 1, were presented to both groups and153

discussion facilitated by the experienced facilitator with expertise in154

competency standardsdevelopment andapractitionerof anotheralliedhealth155

profession (SuA). The questions have been used previously in competency156

standardsdevelopment(9).157

158

Table1nearhere.159

8

Focus group participants were all members of AOPA or teaching into the160

academic course and actively employed in the profession. Expert group161

participants had aminimumof 10 years’ experience in the orthotic/prosthetic162

fieldand/orwereaknownexpert.Recruitmentoftheexpertgroupwasinitially163

through direct approach via email. The recent graduate group graduated in164

Australiabetween2009‐2011andwererecruitedthroughanadvertisementon165

theAOPAwebsite.Purposivesamplingwasusedtoensurerepresentationfrom166

different workplace settings within both groups, with a maximum number of167

twelveineachgroup. Focusgroupswereconductedface‐to‐facefortheexpert168

group and via teleconference/videoconference where available for the recent169

graduategroupandwererecordedandtranscribedverbatim.170

Tworesearchers,theprojectofficerandexperiencedfacilitator,undertook171

thematicanalysisofthetranscriptsindependently.Textinthetranscriptswas172

readtoidentifyinitialcodes,whichwereunderlinedandannotatedinmargins. 173

Majorthemeswereidentifiedfromthesecodesandcomparedforcongruence174

beforebeingcategorisedintomajorworkrolesandcounted.Relationships175

betweenworkrolesandthesubthemeofworktasks,wereestablished.These176

themeswerethentabulatedasworkrolesandtasks.177

Themesrelatingtothekeypurposeoftheprofessionweregroupedtoformfour178

mainstatements.Gapsinthecurrentcompetencystandardsweretabulated179

separatelytoinformthedomains,activitiesandperformanceindicators.180

Performance indicators were added to the domains and activities after the181

project reference group agreed on the domains and activities. The project182

reference group generated six iterations of this draft before the expert group183

9

met face‐to‐face to agree on the final draft, again facilitated by the same184

experiencedfacilitator(SuA).185

BetweenStage1and2,theprojectreferencegroupdefinedtermsandcontexts186

forassessment,viarangestatements.Rangestatements,sometimesreferredto187

as range variables, are defined as ‘the part of a unit of competency which188

specifiestherangeofcontextsandconditionstowhichtheperformancecriteria189

apply’ (18).Definingtherangestatementswasthefinalstageofdevelopmentof190

thedraftcompetencystandards,whichwereusedinStage2.191

Stage 2 involved the use of the reactiveDelphi technique to validate the draft192

competency standards. Participants were required to be full or part time193

financial members of AOPA who had consented to receive AOPA emails. The194

AOPA database was searched to determine membership demographics in195

relationtoage,location,genderandworktype.Fromthe271eligiblemembers,196

184 were identified as meeting the following purposive sample criteria for197

utilizingand/orbeinginterestedincompetencystandards;beinginternationally198

trained;teachingwithintheacademiccourse;beingfacilitymanagerspotentially199

employinggraduatepractitionersorsupervisingstudentsonpracticeplacement.200

Invitations to participatewere sent via email to a subset ofmembers (n=107)201

whosedemographicsreflectedthebroadermembership.202

Surveys were constructed using SurveyMonkey®(19) and were distributed203

alongwith supporting documentation via the AOPAwebsite. A personal email204

with completion instructions was sent to each participant in each round.205

Demographic and professional informationwas collected in Round 1. Prior to206

disseminationall surveyswerepilotedby theproject referencegroupand five207

otherswhowerenoteligibletoparticipate inthesurveybuthadknowledgeof208

10

andinterestintheproject.Surveyswerecompletedoveratwoweekperiodwith209

reminder emails sent at oneweek, 72hours and24hoursprior to closing the210

surveyinordertoreduceattritionrates.211

InRound1,thedomainsandactivitieswerepresentedwithouttheperformance212

indicatorsandrangestatements.Participantsexpressedtheirlevelofagreement213

thateachactivitywasrequiredofanentrylevelorthotist/prosthetistinorderto214

achieve safepractice andpositive client outcomes, using a5point Likert scale215

(strongly disagree, disagree, undecided, agree, strongly agree)with the option216

for open‐ended comment on all aspects of the document. The percentage of217

participantsnominatingeachoftheLikertcategorieswascalculated.Agreement218

wasdefinedas≥75%ofparticipantsnominatingeitherstronglyagreeoragree. 219

The competency standardswere adjusted in light of feedbackbetween rounds220

andparticipantswereprovidedwithasummaryofgroupresultsandanoutline221

ofanychanges.222

InRound2participantswereasked torate theiragreementwithanyactivities223

thatdidnotachieve75%agreementaswellasallperformanceindicators.Range224

statementswereincludedinRound2inordertoprovidefurthercontext.225

In Round 3, participants were asked to rate their agreement with any226

performanceindicators,whichdidnotachieveagreementinRound2aswellas227

their agreement with the key purpose of the profession. In this round228

participantswerealsoprovidedwithanindividualsummaryofRound2results.229

230

Results:231

SampleCharacteristics232

11

InStage1,10expertsand8graduatesparticipatedinthefocusgroups.InStage233

2,52%(n=56)of those invited toparticipate consented todo so. Seventynine234

percent(n=44)ofthosewhoconsentedcompletedallthreerounds,resultingin235

anattritionrateof21%.Ultimately,16%ofthemembershipparticipatedinthe236

threeroundsoftheDelphiprocess.Thedemographicdetailsoftheparticipants237

in Stage 1 and 2 are shown in Table 2. Demographics were similar in both238

samplesforStage1andStage2.239

240

Table2nearhere.241

242

DevelopmentofCompetencyStandards243

Stage1:244

Keythemesidentifiedasgapsinthecurrentcompetencystandardsweresimilar245

intheexpertandgraduatefocusgroups.Theseincluded;anemphasison246

evidence‐basedandethicalpracticewithaclient/patientfocus;theneedfor247

continuingprofessionaldevelopment;andmentoringtoremainprofessionally248

currentandidentificationofscopeofpractice.Graduatesfeltverystronglythat249

theirtechnicalskillswereessentialtotheirrole.250

Statementsaboutgapsincluded:251

Evidence:252

‘youshouldhavetheskillstoclinicallyjustifywhyyouareprescribingit,and253

yes, you should have the clinical skills to review it and see what the254

functionaloutcomesofyourdecisionsareforapatient.’ExpertB255

Continuingprofessionaldevelopment:256

12

‘anessential skillwouldbe to identify thegaps inyourknowledgeand in257

yourpractice,maybeitwouldbeworthincludingsomethinginhavingthat258

skill. And also secondly following on from that, being able to access259

research and have the methods to be able to gain knowledge of best260

practice.’GraduateB261

Graduateswereabletoclearlydefineakeypurpose,whichinvolvedtheclinical262

decisionmakingandtechnicalskillsrequiredtoperformtheroleofthe263

orthotist/prosthetist.264

Keypurposestatementsincluded:265

‘ItellpeopleImakeartificiallimbs’GraduateA266

‘weassessapatient,andthenwecomeupwithaprescriptionfortheirdevice,267

evenifit’srelatedtoaprosthesisandit’sprettyclearthattheyneedone268

belowtheknee…there’slotsmoreinvolvedwiththat,andperhapssayinga269

prescriptionisagoodwordtoinclude.Alsotoincludethefactthatwe270

clinicallyassessthepatientusingourclinicalknowledge.’GraduateB271

Theexpertgrouphadamorediverserangeofviewsaboutthewordingofthe272

keypurpose,perhapsreflectingtheirgreaterdiversityofemploymentroles,273

howevertheoverallthemesofclinicalcareandtechnicalskillweresimilar.274

‘weareabletoconductfullbio‐mechanicalassessmentsandlinkthosebio‐275

mechanical assessmentwith intimate knowledge ofmaterials, technology276

and gait goals and functional goal, tomatchmaterials, technology and277

patient presentation with that equipment, or thatmaterial tomeet the278

patientoutcomes.’ExpertA279

Fourkeypurposestatementsweredevelopedfromtheanalysisoftheinitial280

focusgroupsfordiscussionduringthesecondexpertfocusgroup.281

13

Thethemesaroundworkrolesandtasksfrombothgroupsweresimilarandare282

showninTable3.283

284

Table3nearhere.285

286

Thereweresomeareaswhichrecentgraduatesthoughtshouldbecoveredinthe287

competencystandardsandwhichtheexpertsfeltwerebeyondentrylevel.288

Intheservicemanagementarea,graduatesgaveexamplesofbeinginsituations289

wheretheymayhavebeenthesolepractitionerwithinmonthsofgraduation.290

‘theseniorclinicianhere….hadtobeawayforaboutnineweeksforahealth291

issue,soIwasleftwiththewholedepartmentwithoutanytechnicians,andby292

myself,managing…doingthetechworkandtheclinicalworkhere.’Graduate293

C.294

‘Ithinkinthefuture…we’llallbeinvolvedwithsettingupmoreoutreach295

clinics,anddevelopingservicedeliverytothebroaderpopulation.’Graduate296

B297

Expertsontheotherhand,focusedmoreonskillsthatwereminimumandhow298

youcouldconsistentlyassessthese.299

‘theyneedtohavetimemanagementskillsandprojectmanagementskills’300

ExpertB.301

‘those key kind of graduate management skills, self‐management, time302

managementarethecoreones.’ExpertC.303

The initial draft domains and activities were developed from Table 3, with304

reference to the 2003 Competency Standards and presented to the project305

referencegroup.Atthispoint,referencewasmadetointernationalorthoticand306

14

prosthetic andotherAustralianhealthprofessions’ competency standards.The307

processofaddingrangestatementsresultedinamendmentstothecompetency308

standards to avoid repetition and increase clarity. The final draft was further309

refinedtoinclude6domains,20activitiesand69performancecriteria.Figure2310

showsthe6domainsofpractice.311

312

Figure2nearhere313

314

Stage2:ValidationoftheCompetencyStandards315

Atotalof56participantscompletedthefirstroundoftheDelphisurvey.316

Participantsagreedthateighteen(18,or90%)ofthe20activitieswere317

appropriateforanentry‐levelorthotist/prosthetist(seeFig3).Agreement318

rangedfrom87.5to100%with16activitiesreceiving>90%agreement.319

320

Figure3nearhere.321

15

Twoactivities,bothwithinDomain4(ServiceManagementandImprovement),producedan

undecidedresult:

4.3 Managesclientfundingallocation(53.6%agree,33.9%undecided,12.5%

disagree)and

4.4 Participatesinfacilityresourcemanagement(73.2%agree,17.9%undecided,

8.9%disagree).

Commentsfromparticipantssuggestedthatwhiletheconceptwascorrect,thewording

indicatedactivitiesappropriateforaskilllevelhigherthanentrylevel.AsActivities4.3and

4.4hadanundecidedresultratherthandisagreement,thereferencegrouprevisedthe

activitieswithinthisdomaintoallowtheintendedconceptstoremain.Thisprocessresulted

intheconceptsofActivities4.3and4.4beingincorporatedintoperformancecriteria,rather

thanactivities.

DuetothehighlevelofagreementinRound1,Round2progressedtoratingtheperformance

indicators. A specific question asking for comments in relation to Domain 4 activitieswas

includedtoallowforfurtherfeedbackonthechangesmadefromRound1.

Forty‐nine(49)participants(87.5%)completedthesecondroundofthesurvey.

Participantswereaskedtoratewhethertheperformanceindicatorsdescribedan

observableand/orassessableactionwhichisexpectedoftheworkforcewhen

performingtherelevantactivity.Participantsagreedthat68outofthe69performance

indicatorswereassessabletasksoftherelevantactivity.Agreementlevelsrangedfrom

79.6to100%,with66performanceindicatorshavinganagreementlevel>90%(see

Table5).Participantsdidnotagreeonperformanceindicator5.3.3‐acquiresfurther

16

qualificationstopracticebeyondprofessionalscopeofpractice(54.2%agreement)‐in

Domain5(ProfessionalValuesandBehaviours).

Table5nearhere.

As inRound1, theproject reference group revised theperformance indicatorswith<75%

agreement,basedonthequalitativecommentsprovidedbyparticipants.Althoughonlyone

performance indicator did not achieve agreement, all three performance indicatorswithin

this activity (Activity5.3)werepresented to participants for re‐assessment inRound3. In

Round3,participants(n=44)agreedthatthesethreerevisedperformanceindicators(listed

below)wereallassessabletasksoftherelevantactivity:

5.3.1Workswithinprofessionalscopeofpracticeandauthorityprovidedbytheclient

andemployer(95.4%agreement)

5.3.2 Seeks assistance or refers on when beyond own level of competence (97.7%

agreement)

5.3.3 Recognises where further training is required to conduct independent practice.

(97.7agreement)

DISCUSSION:

Thisstudyisoneoffewoutliningthemixedmethodsresearchmethodology(13,16,20)used

todevelopor reviewcompetencystandards inhealthprofessions.Other studieshaveused

similar methodology however few have combined rigorous qualitative and quantitative

methodologiestodevelopandthenvalidateprofessionalstandards.

17

Sherbino et al (20) conducted a mixed methods study to define the key roles and

competenciesofClinician‐Educators.Expertsparticipatedinfivefocusgroups(finalsample,

n=22), to define attributes, domains of competence and core competencies. One national

surveyofkeyeducator stakeholders (n=1110)validated the results,usingamixof4‐point

categorical responses, such as agree/disagree and important/not important. Mixing

categorical responseswithouta secondroundof surveyingpotentiallyweakens theoverall

result.O’ConnellandGardner(8)usedcriterionsamplingofemergencypracticenurses(n=5)

to develop a competency framework for emergency nursing practice. The draft specialist

competenciesandperformanceindicatorsweresenttoanexpertpanel(n=12)viaaseriesof

Delphi surveys for agreement.Thenumberswere small andpossiblynot representativeof

thenursingprofession,howeverboththesestudiesusedsimilarmethodologyasthecurrent

study.

Oneof theunique featuresofour studywas theuseof separateexpertandgraduate focus

groups. The experts were chosen to represent key stakeholders, especially officers of the

professionalassociation,previousauthorsofthecompetencystandards,universityeducators

andmanagersof largedepartmentsemployingnewgraduateorthotist/prosthetists.This is

similar to other studies in allied health(21). The graduate focus group was chosen to

represent relatively recent graduates whomay be working in emerging areas of practice.

Interviewsof recentgraduates indieteticshaveshownthat theyoftendescribeworking in

emergingsituations,whichseniorpractitionerswouldnotconsiderentrylevel(9,22). Asa

stakeholder group, graduates are rarely approached as a separate group in the review of

competency standards in other allied health professions. Much discussion occurred in the

expert group about whether the standards were minimum on graduation or aspirational,

18

describingentry levelpractice in the first6 ‐12months.Theverydefinitionofcompetency

havingtobeassessedintheworkplacemakestheformer,ieminimumongraduation,difficult

to achieve, however for professional registration or recognition purposes, this is the

norm(21, 23). Despite this, the resulting domains or work roles, shown in Figure 2, were

grouped similarly to other health professions; collaboration/sharing/communicating;

therapeutic role; prioritizing workload; ethical care; promoting best practice;

professionalism(24) or Clinical Expert, Communicator, Collaborator, Manager, Health

Advocate,Scholar,andProfessional(25).

Several iterations of the initial draftwere considered first by the project reference group,

priortopresentationtothesecondexpertfocusgroupfordiscussion.Theprojectreference

groupagainrefinedthedraftcompetencystandardspriortothefinaldraftbeingusedinthe

Delphiprocess.Thisprocess,step2intheexploratorysequentialmodeloutlinedabove,and

outlined in Figure 1, allowed reflection and integration of views but also review of

appropriateliteratureandotherprofessionalcompetencystandards.

Thequantitativestep,step3orDelphiprocessallowedarankingofagreementfromawider

group of participants. Jones and Hunter(12) describe consensus methods as needing

anonymity, iteration, controlled feedback and statistical group response (only if scores are

sequentialandnotcategorical).OurresultsfromthefirstDelphiroundshowedremarkable

agreement, with 16 activities achieving >90% consensus. Revision of the Service

ManagementandImprovementdomain, theonlydomainwhereactivitiesshowed less than

90%agreement,resultedinsomeactivitiesbeingre‐writtenasperformanceindicators.This

was consistent with both focus groups perceiving management differently, at entry level.

Commentsfromparticipantsindicatedthatmuchofthelackofagreementinvolvedwording

19

of the activity rather than the activity itself. Therefore, in the second round of Delphi, all

revised activities were shown but participants rated their level of agreement with the 69

performance indicators,within those activities. ThismodifiedDelphimethod is similar to

that usedbypharmacists inprimaryhealth care (16)where anonline surveywasused to

agree on competency descriptions, organized into domains, elements and sub‐elements.

Participantswerethenaskedtorankona6pointLikertscalehowoftentheyperformedthe

sub‐element(performance indicator)andhowcritical theythoughtthesub‐elementwasto

achieve patient outcomes. The results expressed as importance rankings were seen as

validatingtheoriginalcompetencystatements.

Other studies have used subsequent rounds ofDelphi to improve agreement.Hughes et al

used two rounds of Delphi surveys to improve agreement on 143 competency elements

(activities) inpublic healthnutrition,howeveronly reported the shift in agreementon the

109elements forwhich therewas>67%agreement(14). Others inadvanceddieteticsand

nursing, have reported changed median (17) or mean score(8, 13) respectively between

rounds.

Thehigh levelof agreement inour study forall roundsof theDelphiwasnotunlikeother

studies, although other professional competency studies used differentmeanings for their

ratingscalesanddifferentmethods forcomputing therankings.Generallyconsensus levels

are high. Hughes et al only reported 33 (out of a possible 143) competencies rated as

essential by 100% of participants (14). O’Connell and Gardner reported mean agreement

scoresrangingfrom4.6‐4.9outofamaximumscoreof5(8).

Thestrengthofthisstudyisthatitusedastructuredexploratoryandconfirmatoryapproach

withacceptedqualitativeandquantitativemethods todevelopand thenvalidateminimum

20

entry level competency standards for orthotist/prosthetists in Australia. The level of

agreementwasveryhigh(>87%for18outof20activities,seeFigure3).Wheretherewasno

agreement in the first rounds of the Delphi survey re‐wording resulted in very high

agreementonsubsequentrounds(>80%forall18activitiesand68performancecriteria,see

Table3).

Limitations include thesmallnumbers,withonly16%of the totalmembership involved in

the validation, however these numbers and the attrition rate of 21% are consistent with

other studies of competency validation in other professions (8, 14, 16). Most participants

graduatedfromtheonlyprograminAustralia,whichcouldsuggestthatorthotist/prosthetist

experience is constrained by training in only one institution in Victoria. Every attempt

howeverwas used to ensure representativeness of the profession in Australia. Due to the

purposive sampling criteria, only184outof271memberswereeligible toparticipateand

only 107members invited to participate to reflect themembership profile. Every attempt

was made to include orthotist/prosthetists who were members of AOPA but had trained

outside Australia and participants were selected to be representative of the AOPA

membership,includingthosewhoworkedoutsidethemetropolitanarea.Table2showsthat

despite a large percentage of respondents being Victorian this is representative of the

membership and where the majority practice. Further research could apply these

competency standards to assess entry‐level practice for graduates exiting an education

program,oralternatively,orthotist/prosthetistsfromothercountries.

Overall this study has outlined an evidence‐based approach to the review of competency

standardsforahealthprofession,suchasorthotist/prosthetists.Themethodologyisrobust

andcouldbeappliedtootherdisciplines.

21

Table1.Focusgroupquestions1. WhatdoyouthinkarethegapsinthecurrentCompetencyStandards?2. Whatisthekeypurposeoftheprofession?3. Whatischangingorlikelytochangeintheprofessionthatmightaffect

thispurpose?4. Whatmusthappenforthekeypurposetobeachieved?5. Whydoestheprofessiondoit?6. Whatmajorthingswouldyouhavetodotoperformthatrole?7. Returningtothegapsidentifiedearlier,doyouwishtochangeoradd

anything?

22

Table2Mean(±SD)ageandproportion(%)byfemalegenderandworklocationofoverallOrthotist/Prosthetistprofession,focusgroupsandDelphiparticipants. Membership

Profilen=271

ExpertFocusGroupn=10Stage1

GraduateFocusGroupn=8Stage1

DelphiParticipants(3rounds)n=44Stage2

Mean(±SD)Age(yrs)

39(±11.6) 41(±9) 27(±5) 38(±11.8)

Female(%) 39

50 78 43

*Victoria(%) 52

100 56 48

ThesoletraininginstitutionforOrthotist/ProsthetistsinAustraliaisinVictoriaandthemajorityofgraduatesareemployedinthatstate.

23

Table3InitialthemesdevelopedfromfocusgroupsWorkroles TasksCollaborativeclientcare Managespatientassessmentand

treatmentaspartofahealthcareteamAssesses,prescribes,treats,plansandrefers,liaiseswithotherprofessionals

EthicalandSafePractice Usesevidence,justifiesinterventionFollowslegislation,occupationalhealthandsafetyregulationsKnowsscopeofpractice

Communicatesusinggoodwritten,oralandinterpersonalskills

Canusearangeofmodalitiesincludinge‐healthAdvocatesforpatientinavarietyofforumsAdvocatingforotherstaff

Managesresources ManagesselfManagesbudgets,fundingsystemsandadministrationofthese.

PrescribesmaterialsandtechnologywithinOrthotist/Prosthetistscopeofpractice

Manufacturesoroverseemanufactureofcustomdevicesandmodifications

Continuingprofessionaldevelopment Engagesinselfdevelopment

24Table4:PercentageAgreementforPerformanceIndicatorsinRound2and3DelphiSurveysPerformanceIndicator–finalwording Round

2Round3

1.1.1Ensuresallinteractionswiththeclientand/orcarerdemonstraterespect,honesty,empathyanddignity,andareconductedinaculturallyappropriatemanner

95.9

1.1.2Ensurestheclientisthefocusofthecarepathway 91.8 1.1.3Ensurestheclientand/orcarerisawareoftheirrightsandresponsibilities 93.9 1.1.4Obtainsinformedconsentfromtheclientand/orcarerpriortotheprovisionofcare 95.9 1.1.5Listenseffectivelytotheclientand/orcarer 100 1.1.6Encouragestheclientand/orcarertoparticipateandprovidefeedback 98 1.1.7Providesprompt,accurateandcomprehensiveinformationincleartermstoenableclientsand/orcarerstomakeinformeddecisions

95.9

1.2.1Receivesandformulatesclientreferrals,professionalhandovers,healthcareteamreportsandothertreatmentplans

98

1.2.2Respects,acknowledgesandutilisestheexpertiseofotherhealthprofessionals 100 1.2.3Establishesandmaintainseffectiveworkingrelationshipswithotherhealthprofessionalstoenhancecollaborativepracticeandaccesstocare

100

1.2.4Activelyparticipatesinhealthcareteamsandseeksopportunitiestodemonstrateprofessionalexcellence

89.8

1.3.1Providesclinicaljustificationandevidenceforprescribedorthotic/prostheticclientcare 93.9 1.3.2Providesrelevantinformationinordertofacilitateclientaccesstocare 79.6 2.1.1Identifiessubjectiveandobjectiveinformationtoenabledevelopmentofanappropriateorthotic/prostheticmanagementplan

98

2.1.2Selectsassessmenttechniques,outcomemeasuresandothertools/instruments,basedonevidencewhicharerelevanttotheclient’spresentation

95.9

2.1.3Performsassessmentprofessionally,safelyandeffectively 100 2.2.1Accessesandutilisesthebestavailableevidencetoguideclinicaldecisions 93.8 2.3.1Facilitateclientand/orcarertoestablishpersonalgoals 93.9 2.3.2Considerstheinformationobtained,theclientand/orcarer’sgoalsandavailableevidencewhenformulatingtreatmentoptions

100

2.3.3Discussestreatmentoptionswiththeclientand/orcarertosupportclientcentredcareandinformedchoice

98

2.3.4Discussesshortandlongtermtreatmentgoalswiththeclientand/orcarer 91.8

252.3.5Identifiesclientswhorequirecollaborativecareandliaiseswiththehealthprofessionalteamtoensureintegratedcareplanning

98

2.3.6Determinesandjustifiesthedesigndetailsoftheorthosisand/orprosthesisprescription 95.9 2.3.7Includesclient,carerand/orhealthprofessionalteameducationandfollow‐upwhenplanningtreatment

95.9

2.3.8Selectsappropriateoutcomemeasures 81.7 2.4.1Considersallrelevantcharacteristicsoftheclientduringorthosis/prosthesisfittingandreviewprocesses

100

2.4.2Usesappropriatetechniquestoensureoptimalfitandfunctionoftheorthosis/prosthesis 93.9 2.4.3Reviewstheclientatappropriateintervalstoevaluatefit,function,qualityandsafetyoftheorthosis/prosthesis

100

2.4.4Evaluatesandmonitorstreatmentoutcomesusingpatientfeedbackand/oroutcomemeasures 98 2.4.5Modifiestreatmenttoensurebestpossibleoutcomesaremaintained 95.9 2.4.6Discussesprogresstowardgoalswiththeclientand/orcarer 93.9 2.5.1Adherestolegislativeandorganisationalrequirementsforalldocumentation 98 2.5.2Maintainslegible,conciseandaccuratedocumentationusingcontemporarymethods 98 2.5.3Safelyandsecurelystoresinformationandactstomaintainconfidentialitywhilstensuringavailabilityofinformationtootherhealthprofessionalsinvolvedinthecarepathway

93.8

3.1.1Utilisesappropriatecasting,measuringand/orcastmodificationtechniquestofacilitatefabrication 100 3.1.2Fabricatesand/orcoordinatestheoptimalfabricationoforthoses/prostheses 96 3.1.3Performsand/orcoordinatesrequiredmodificationsoforthoses/prostheses 93.8 3.2.1Assessestheorthosis/prosthesisforstructuralsafetyatappropriateintervals 93.9 3.2.2Ensurestheorthosis/prosthesisiscompliantwithmanufacturerguidelinesandstandards 95.9 4.1.1Facilitatesappropriatecompletionofallsupportiveactivities 87.8 4.1.2Facilitatesappropriatecompletionoftreatmentprovision 93.9 4.1.3Demonstratesanabilitytotriageindividualclientcaseloadwithinbroaderfacilityworkload 95.9 4.2.1Determinesavailablefundingforprescribedcareplan 95.8 4.2.2Preparesand/orcoordinatessubmissionofdocumentationforclientfundingsupportasrequired 95.9 4.2.3Prescribesanddesignsorthosis/prosthesistoachieveoptimaloutcomeswithintheapprovedbudgetforclientcare

91.7

4.2.4Understandsandconformstofundingarrangements,budgetallocations,statisticalreportingandfinancialtransactionrequirementsrelevanttotheworkplace

91.6

4.3.1Strivestocontinuallyimproveefficiency 89.6

264.3.2Recognisesservicegapsorinefficienciesandworkscollaborativelytoidentifysolutions 91.7 4.3.3Participatesinauditprocessesandqualityimprovementinitiatives 95.8 5.1.1Adherestolegislationandworkplaceguidelinesrelatingtosafety 100 5.1.2Identifiesworkplacehazardsandactstoeliminateorreducerisks 100 5.2.1Complieswithrelevantlegallaws,regulations,policiesandguidelines 100 5.2.2Abidesbyapplicablecodesofethicsandconduct 100 5.2.3Recognisestheresponsibilitytodonoharm 97.9 5.2.4Recognisesandrespondsappropriatelyifclientand/orcarerisatrisk 100 5.3.1Workswithinprofessionalandpersonalscopeofpracticeandauthorityprovidedbytheclientandemployer(Round2)

97.9

5.3.1Workswithinprofessionalscopeofpracticeandauthorityprovidedbytheclientandemployer(Round3)

95.4

5.3.2Acquiresfurthertrainingandassessmenttodeveloppersonalscopeofpractice 91.7 5.3.2Seeksassistanceorrefersonwhenbeyondownlevelofcompetence 97.75.3.3Acquiresfurtherqualificationstopracticebeyondprofessionalscopeofpractice(Round2) 54.2 5.3.3Recogniseswherefurthertrainingisrequiredtoconductindependentpractice(Round3) 97.76.1.1Undertakesindependentlearningtofurtherownknowledgeandskillsonacontinuousbasis 95.8 6.1.2Sharesskillsandknowledgewithhealthprofessionalcolleaguesandstudents 100 6.1.3Participatesinhealthprofessionaltrainingandresearchasopportunitiesarise 89.6 6.1.4Seeksoutleadersintheprofessionforadviceandmentoring 97.9 6.1.5Offersconstructivefeedbackandassistancetootherhealthprofessionals 95.8 6.2.1Assessesandcriticallyanalysesresearchliteratureandothersourcesofevidencetoimprovepractice

91.6

6.2.2Demonstratesasystematicapproachtoanalysisanddecisionmaking 91.7 6.2.3Integratesthebestavailableevidenceandnewlearningintopracticetoimprovehealthoutcomesforclients

97.9

6.2.4Demonstratesknowledgeofnewtechniquesandtechnologyrelevanttoorthotics/prosthetics 81.3 6.2.5Criticallyandcontinuouslyevaluatespractice 93.7

27

1 2

28

FigureLegends3Figure1MixedMethodsApproachtoDevelopingCompetencyStandard4Figure2:DomainsofOrthotic/ProstheticCompetencyStandards 5Figure3DelphiRound1AnalysisofDomainsandActivities. 6

29

References7

1. WorldHealthOrganisation.SummaryWorldReportonDisability.:World8Health Organisation and World Bank.; 2011 [cited 2014 30 April, 2014].9Available from:10http://whqlibdoc.who.int/hq/2011/WHO_NMH_VIP_11.01_eng.pdf?ua=1.112. AustralianBureauofStatistics.Customisedreport:Censusofpopulation12andhousing;customizeddatareportcode251912;orthotistorprosthetist.2014.133. National Health Service Scotland. Scottish Orthotic Services Review14Edinburgh: National Health Service, Scotland; 2005. Available from:15http://www.sehd.scot.nhs.uk/publications/dc20050614orthotics.pdf.164. World Health Organisation. Transformative scale up of health17professional education. Geneva, Switzerland:World Health Organisation; 201118[cited 2014 30 April,]. Available from:19http://whqlibdoc.who.int/publications/2011/9789240685215_eng.pdf.205. HealthWorkforceAustralia.HealthWorkforceAustralia: 2011.National21HealthWorkforceInnovationandReformStrategicFrameworkforAction2011‐222015 Adelaide, South Australia: Health Workforce Australia; 2011. Available23from: https://http://www.hwa.gov.au/sites/uploads/hwa‐wir‐strategic‐24framework‐for‐action‐201110.pdf.256. The Australian Orthotist Prosthetist Association Inc. Entry Level26CompetencyStandardsforAustralianOrthotist/ProsthetistsMelbourne,Victoria272014.3rd:[Availablefrom:http://www.aopa.org.au/documents/item/27.287. Larkin PM, Begley CM, Devane D. Breaking from binaries‐‐using a29sequentialmixedmethodsdesign.NurseResearcher.2014;21(4):8‐12.308. O'Connell J, Gardner G. Development of clinical competencies for31emergency nurse practitioners: a pilot study. Australasian Emergercy Nursing32Journal.2012Nov;15(4):195‐201.eng.339. AshS,GoncziA,HagerP.Combiningresearchmethodologies todevelop34competency‐based standards for dietitians: a case study for the professions.35National Office of Overseas Skills Recognition Research paper no. 6. DEET.36Canberra:AustralianGovernmentPublishingService;1992.3710. Phillips S, Ash S, Tapsell L. Relevance of the competency standards to38entry level dietetic practice. Australian Journal of Nutrition & Dietetics.392000;57(4):198‐207. PubMed PMID: 2002151039. Language: English. Entry40Date:20021101.RevisionDate:20091218.PublicationType:journalarticle.4111. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative42ResearchinPsychology.20062006/01/01;3(2):77‐101.4312. Jones J, Hunter D. Consensus methods for medical and health services44research. BMJ (Clinical Research Ed). 1995;311(7001):376‐80. PubMed PMID:457640549.4613. ChangAM,GardnerGE,DuffieldC,RamisM‐A.ADelphistudytovalidate47an advanced practice nursing tool. Journal Of Advanced Nursing.482010;66(10):2320‐30.4914. HughesR,BegleyA,YeatmanH.Aspirationalcompetencyexpectationsfor50publichealthnutritionistsinAustralia:Aconsensusstudy.Nutrition&Dietetics.512013.Epub13November,2013.52

30

15. Davies J, Hughes R, Margetts B. Towards an international system of53professionalrecognitionforpublichealthnutritionists:afeasibilitystudywithin54theEuropeanUnion.PublicHealthNutrition.2012;15(11):2005‐11.5516. Kennie‐KaulbachN,FarrellB,WardN,JohnstonS,GubbelsA,EgualeT,et56al.Pharmacistprovisionofprimaryhealthcare:amodifiedDelphivalidationof57pharmacists'competencies.BMCfamilypractice.2012;13:27.Epub2012/03/30.58eng.5917. BrodyRA,Byham‐Gray L, Touger‐DeckerR, PassannanteMR,O'Sullivan60Maillet J. IdentifyingComponentsofAdvanced‐LevelClinicalNutritionPractice:61A Delphi Study. Journal of the Academy of Nutrition and Dietetics.622012;112(6):859‐69.6318. Naidu R, Stanwick, J., Frazer, K. Glossary of Terms Adelaide, South64Australia: National Centre for Vocational Education Research; 2013. Available65from:http://www.voced.edu.au/content/glossary‐vet.6619. SurveyMonkeyInc.SurveyMonkey®.PaloAlto,California,USA2013.6720. SherbinoJ,FrankJR,SnellL.DefiningtheKeyRolesandCompetenciesof68the Clinician‐Educator of the 21st Century: A National Mixed‐Methods Study.69Academicmedicine : journal of the Association of American Medical Colleges.702014May89(5):783‐9.Epub2014/03/29.Eng.7121. RodgerS,ClarkM,BanksR,O'BrienM,MartinezK.Anationalevaluation72of the Australian Occupational Therapy Competency Standards (1994): a73multistakeholder perspective. Australian Occupational Therapy Journal.742009;56(6):384‐92.7522. DowdingK,AshS,Shakespeare‐FinchJ.Usingcriticalincidentinterviews76to identify the mental health knowledge, skills and attitudes of entry‐level77dietitians.Nutrition&Dietetics.2011;68(4):297‐304.7823. Australian New Zealand Podiatry Accreditation Council. Podiatry79Competency Standards for Australia and New Zealand 2012 [cited 2014 1280August]. Available from: http://www.anzpac.org.au/files/Podiatry Competency81StandardsforAustraliaandNewZealandV1.1211212(Final).pdf.8224. Dewing J, Traynor V. Admiral nursing competency project: practice83development and action research. Journal of Clinical Nursing. 2005;14(6):695‐84703.8525. FrankJR,DanoffD.TheCanMEDSinitiative: implementinganoutcomes‐86basedframeworkofphysiciancompetencies.MedicalTeacher.2007;29(7):642‐877.888990

91

92

31

93