validity and reliability of qualitative assessments based on self-reported statements

Validity and Reliability of Qualitative Assessments Based on Self-reported Statements

Validity and Reliability of Qualitative AssessmentsBased on Self-reported Statements

Alexander Haarmann

QMM2015

2 October 2015

Alexander Haarmann 2 October 2015 1 / 21


Background

PhD thesis on emergence of formal collective patient and publicinvolvement (PPI) in (health-)care

the Netherlands (all in-patient care) & England (Foundation TrustHospitals) have established almost identical means of collective PPI

user councils as additional tier in governanceadvisory character, few veto rightsvirtually identical rights



Theory & Reality



Theory & Reality

Healthcare Types

Typology according to. . .Neubauer andBirkner (1984)

Wendt and Rothgang (2007) Moran (2000) Esping-Andersen (1990)

Pay on yourown

(Predominantly Character-ised by) Private Healthcare

- Supply Liberal

CharityProvision (Predominantly) State-run/

National Healthcare SystemEntrenched/ Command& control

Social-democratic

Insurance (Predominant) Logic of So-cial Insurance

Corporatist Conservative

postulated relation between funding, provision, & governancenevertheless same type of PPI in England & the Netherlands



Resulting Questions

What factors can be identified to have contributed to or hamperedthe implementation of collective patient involvement in hospitals?

Have the main goals pursued with the respective legislationbeen achieved according to key actors in the field?



Resulting Questions

What factors can be identified to have contributed to or hamperedthe implementation of collective patient involvement in hospitals?Have the main goals pursued with the respective legislationbeen achieved according to key actors in the field?



Methodology & Sourcesexplorative, qualitative, comparative study with the main cases England& the Netherlands, which are compared to Sweden & Germany

Sources

all kinds of literature, including grey literaturegreen & white papersparliamentary minuteslegal textsexpert interviews (nine to 15 interviews per country; on averagemore than 50 minutes recorded length)

politicians of ruling & opposition parties at time of enactment(health-)care managers (acute, mental, ambulance; long-term)members of user councilspatient organisationsresearchers

participant observation




Sourcesall kinds of literature, including grey literaturegreen & white papersparliamentary minuteslegal texts

expert interviews (nine to 15 interviews per country; on averagemore than 50 minutes recorded length)






Sourcesall kinds of literature, including grey literaturegreen & white papersparliamentary minuteslegal textsexpert interviews (nine to 15 interviews per country; on averagemore than 50 minutes recorded length)





Question of the Presentation

How valid & reliable is the collected data?How suitable is it for evaluating changes, policies,implementations?



Data to Compare With

internalexpert interviews vs. participant observationcontrasting statements of actors/ professional groups

externalNL: a number of reports, surveys, & broader range of scientificcontributions (e. g. Savornin Lohman 2000; Hoogerwerf, Nieversand Scholten 2004; LSR 2007; Lammerts et al. 2008b; Lammertset al. 2008d; Lammerts et al. 2008c; Lammerts et al. 2008a;Oudenampsen et al. 2008; Trappenburg 2008)England: few anecdotal reports, one survey (Ham and Hunt 2008;Ipsos MORI 2008; Bojke and Goddard 2010; House of CommonsHealth Committee 2008a; House of Commons Health Committee2008b)



Results (NL)

Sectoral Differencesworks best in institutions of long-term caresticking point hospitalsdifferent traditions

Frequency of Informing Other Patients/ Residents in Percent

Mental Care Rehab Facility Home Care Hospitalsonce a year 12 n. a. 39 41

once half a year 9 n. a. 13 13once a quarter 30 25 13 6once a month 33 n. a. 3 2

other 16 n. a. 32 37

n 43 11 31 46

Sources: Lammerts et al. 2008a, p. 28; Lammerts et al. 2008b, pp. 2829; Lammerts et al. 2008c,

pp. 2627; Lammerts et al. 2008d, p. 28



Results (NL)


I think, that [. . . ] they [the client councils] do not have a strong positionin hospitals [. . . ]. [S]uch a council of clients is for most people a realitythat is far away. [. . . ] And I think, to be honest, if I were responsible inthe Hague as minister nowadays I would rethink about a role of theclient councils. The function is taken over by others. [. . . ] That issomething different for an old-age home or a nursing home, or for amental care institution, that is substantially different. Even for aprimary care institution in a village or a city with a dentist and a GP, andsocial work etc. (interview 11; authors translation)




other 16 n. a. 32 37

n 43 11 31 46





Results (NL)





other 16 n. a. 32 37

n 43 11 31 46





Results (NL)


Involving Grassroots in Advice in Percent

Mental Care Rehab Facility Home Care HospitalsVisiting clients 70 36 30 4

Surveys 56 54 60 58Contact person 48 36 33 31

Meetings 46 27 18 31Ideas Boxes 32 63 9

Website 23 54 39 40Telephone 23 27 21 11

Other 32 54 42 51

n 43 11 30 46

Sources: Lammerts et al. 2008a, p. 30; Lammerts et al. 2008b, p. 29; Lammerts et al. 2008c, p. 27;

Lammerts et al. 2008d, p. 29



Results (NL)

Sectoral Differencesworks best in institutions of long-term caresticking point hospitalsdifferent traditionsdifferent topics

rehab & hospitals: budget & organisational changeshome care: quality of care & complaint procedurespsychiatric care: closure/ extension of wards, quality of services,food & budget

Lammerts et al. 2008b; Lammerts et al. 2008d; Lammerts et al. 2008c;Lammerts et al. 2008a



Results II (NL)

Learning Curvechange over time, learning curve of 1015 years

process from What are our rights? to What issues would we like toraise?find a common languagedifferent lifeworlds

Look, such a client council, they do not want to talk a full hour about thetemperature of the tea. However, if it feels it is not heard that keepscoming back. And the member of the executive board keeps thinking:Why do I sit here? and tries to suck the client council with him into hisworld of the system, of control, of long-term perspective, whereas theclient council has the perspective on the short-term. (interview 1;authors translation)



Results II (NL)

Learning Curvechange over time, learning curve of 1015 yearsprocess from What are our rights? to What issues would we like toraise?find a common languagedifferent lifeworlds

Look, such a client council, they do not want to talk a full hour about thetemperature of the tea. However, if it feels it is not heard that keepscoming back. And the member of the executive board keeps thinking:Why do I sit here? and tries to suck the client council with him into hisworld of the system, of control, of long-term perspective, whereas theclient council has the perspective on the short-term. (interview 1;authors translation)



Results III (NL)

Different Perspectivesheterogeneous statements of different actors/ groups

virtually everyone agrees that hospital sector is a specific case

scientists & politicians rather positiveusers umbrella organisations: 1

3positive, a few bad examples, &

many in between extremes

members of user councils & management agree: cooperation and, thereby, user involvement works fairly well

perceived influence: in general between 4 & 5 (on 5 point scale)not all boards give requested advice, only few unrequestedSources: Lammerts 2008; Lammerts 2008a; Lammerts 2008b; Lammerts 2008c



Results III (NL)



scientists & politicians rather positive

Thus, they have actually very much influence on the policies,in case they claim it and do not let the management do withthem. (interview 10; authors translation)

users umbrella organisations: 13

positive, a few bad examples, &many in between extremes





Results III (NL)






The problem is always that the client council needs to collaboratewith the executive board. However, if you come to a controversy witheach other it does not get better for anyone! [. . . ] It is difficult to [say]:We demand this and we demand that, since then such an executiveboard says: There is no money for that! You may demand whatever youwant! [. . . ] That will not happen! (interview 7; authors translation)How do you deal with the flow of information from the board ofdirectors? They dump it, let the council of clients keep busy. [. . . ] Youcan flood the councils with information. And boards of directors havea knack for it. (interview 3; authors translation)





Results III (NL)










Results (England)

Englandsimilar picture as in the Netherlands, similar problemsPPI liaison managers & politicians more critical than in NLdifficulties to recruit a sufficient number of members andgovernors, in particular in mental caregovernors roles are ill-defined

results are mirrored in survey and reports



Summary of ComparisonComparison with survey results, reports show strengths and weaknesses

=objective information about size & composition of council, frequency of meetingsetc.differences between actorsdevelopmental aspectsthe overall picture

representativityprone to sampling error

+information & more subtle differences going beyond standardised items; e. g.:

learning curve of 1015 years to understand each otherdifferent lifeworldsfollowing the rules can mean infringing the original intention

Comparison seems proof for high reliability & validity ofexpert interviews



Comparison with Internal Data

Comparison of results from expert interviews with content ofparticipant observation during internal council meeting

Expert interview Participant observationgood director & goodrelation to management

lack of support byadministration

management is supportive relevant papers arerepeatedly distributed toolate

council is taken seriously& can influence quite a lot

council is not takenseriously & needs to stepup



Potential Reasons for Discrepancies

potential reasons might consist of a range of social-psychologicalmechanisms such as:

social desirabilityself-praise

managers profit from portraying themselves as listening &understandingusers profit from portraying themselves as active & successful

presenting PPI as a success story to the foreign researchermaybe actual improvement over situation some 25 years ago. . .



Further Steps!?

age-old question about validity

the problem is not to be found in a specific method, but in aspecific combination of topic and/ or actor

mixed methods design does not increase validity per sebetter understanding of underlying processes is neededlong-term perspective necessary? one-off shot of reality sufficient?

potential ways to handle problem:

posing much more specific questionsmixed methods designs that raise the validity of the datadifferent cultures of access in various countries



Further Steps!?

age-old question about validitythe problem is not to be found in a specific method, but in aspecific combination of topic and/ or actor

mixed methods design does not increase validity per se

better understanding of underlying processes is neededlong-term perspective necessary? one-off shot of reality sufficient?

potential ways to handle problem:

posing much more specific questionsmixed methods designs that raise the validity of the datadifferent cultures of access in various countries



Further Steps!?

age-old question about validitythe problem is not to be found in a specific method, but in aspecific combination of topic and/ or actor

mixed methods design does not increase validity per sebetter understanding of underlying processes is neededlong-term perspective necessary? one-off shot of reality sufficient?

potential ways to handle problem:posing much more specific questionsmixed methods designs that raise the validity of the datadifferent cultures of access in various countries



Thankyou

foryou

r attention!

Contact details:Alexander Haarmann

University of Bremen, [email protected]

[email protected]



Table of Contents1 Starting Point

BackgroundContradiction between Theory & RealityResulting QuestionsMethodology

2 Question of the PresentationData to Compare With

3 ResultsResults (NL)Summary of ResultsComparison with Internal Data

4 ConclusionsPotential Reasons for DiscrepanciesFurther Steps?

References



Bibliography I

Bojke, Chris and Maria Goddard (Sept. 2010). Foundation Trusts: A Retrospective Review. Centre for Health Economics,University of York. York.

Esping-Andersen, Gsta (1990). The Three Worlds of Welfare Capitalism. Princton: University Press.

Ham, Chris and Peter Hunt (July 2008).Members Governance in NHS Foundation Trusts: A Review for the Department of Health.

Hoogerwerf, Remco, Eline Nievers and Christel Scholten (July 2004).De invloed van clintenraden Het verzwaard adviesrecht in de dagelijkse praktijk vanuit clintenraadperspectief. researchvoor beleid bv. Leiden.

House of Commons Health Committee, ed. (17th Oct. 2008a). Foundation trusts and Monitor. Sixth Report of Session 200708Volume I: Report, together with formal minutes.

ed. (17th Oct. 2008b). Foundation trusts and Monitor. Sixth Report of Session 200708 Volume II: Oral and written evidence.

Ipsos MORI (Jan. 2008).Survey of Foundation Trust Governors Research Study Conducted for Monitor Independent Regulator of NHS Foundation Trusts.London.

Lammerts, R., L. M. Verwijs, J. Homberg, K. D. Hekkert, S. J. W. de Kroon and H. C. M. Kamphuis (2008a).Patinten-, gehandicapten- en ouderenorganisaties in Nederland Clintenraden in beeld Clintenraden van GGz-instellingen in 2008.Utrecht: LSR.

(2008b).Patinten-, gehandicapten- en ouderenorganisaties in Nederland Clintenraden in beeld Clintenraden van revalidatiecentra in 2008.Utrecht: LSR.



Bibliography II

Lammerts, R., L. M. Verwijs, J. Homberg, K. D. Hekkert, S. J. W. de Kroon and H. C. M. Kamphuis (2008c).Patinten-, gehandicapten- en ouderenorganisaties in Nederland Clintenraden in beeld Clintenraden van thuiszorgorganisaties in 2008.Utrecht: LSR.

(2008d).Patinten-, gehandicapten- en ouderenorganisaties in Nederland Clintenraden in beeld Clintenraden van ziekenhuizen in 2008.Utrecht: LSR.

LSR (Mar. 2007). een onderzoek naar de praktijk van de wmcz in ziekenhuizen en revalidatieinstellingen. Utrecht: LSR.

Moran, Michael (June 2000). Understanding the welfare state: the case of health care. In:British Journal of Politics and International Relations 2.2, pp. 135160.

Neubauer, Gnter and Barbara Birkner (1984). Alternativen der Organisation und Steuerung des Gesundheitswesens. In:Alternativen der Steuerung des Gesundheitswesens im Rahmen einer Sozialen Marktwirtschaft. Ed. by Gnter Neubauer.Gerlingen: Robert Bosch Stiftung, pp. 1370.

Oudenampsen, Dick, Helen Kamphuis, Rob Lammerts, Jeroen Homberg and Esmy Kromontono (Dec. 2008).Patinten- en Consumenbeweging in Beeld. Utrecht: Nederlandse Patinten Consumenten Federatie (NPCF).

Savornin Lohman, J. de (2000). Evaluatie Wet medezeggenschap clinten zorgsector. In:Evaluatie Wet medezeggenschap clinten zorginstellingen. Ed. by J. de Savornin Lohman. Utrecht: Verwey-Jonker Instituut.

Trappenburg, Margo (2008). Genoeg is genoeg. Amsterdam: Amsterdam University Press, p. 272.

Wendt, Claus and Heinz Rothgang (2007).

Gesundheitssystemtypen im Vergleich Konzeptionelle berlegungen zur vergleichenden Analyse von Gesundheitssystemen.61. Universitt Bremen, Sonderforschungsbereich 597 (Staatlichkeit im Wandel).


Starting PointBackgroundContradiction between Theory & RealityResulting QuestionsMethodology

Question of the PresentationData to Compare With

ResultsResults (NL)Summary of ResultsComparison with Internal Data

ConclusionsPotential Reasons for DiscrepanciesFurther Steps?

validity and reliability of qualitative assessments based on self-reported statements

Science