Leadership of your PhD

Anders Dysvik, Ph.D. Professor of Organizational BehaviorBI Norwegian Business School

Spend 3 minutes discussing facets of research quality

• How rigorous is the study and the report?

• How original and new is the contribution?

• How relevant is it within the field?

• How usefulness is it in society?

• Can we pick and choose among these requirements - or are they interdependent?

Requirements I

• Prepared for replication• Empirical support or lack of rejection• Precision regarding constructs, level of analysis,

form of relationships, and generalizability• A unifying conceptual framework• Falsifiable statements and freedom from prejudice

”…when properly performed,

the rain dance will make rain fall

from the skies”.

”…when properly implemented,

incentive motivators are effective

mechanisms for enhancing

individual performance”.

Peterson, S. J., & Luthans, F. (2006). The impact of financial and

nonfinancial incentives on business-unit outcomes over time. Journal of Applied Psychology, 91(1), 156-165.

Requirements II

• Generalizability• theoretical statements are by definition about something more than

single cases• the less situation specific, object specific and time specific, the

better

• Generalizability in terms of• the number of different persons and objects• how well a phenomenon is explained• how many phenomena that are explained

The research problem/question• More specific than the research topic • Not given, they are discovered, found or constructed• Something we don’t already know and that is possible to explore,

investigate or answer with the means of scientific methodology• Should solve a problem or gain relevant insight that develops

theory or practice• Explicitly related to relevant research literature(s)• Evolving during the early phases• Expressed in terms of relationships and/or questions

Positioning the research problem

• Novelty• something we don’t already know• typical, a lack of knowledge, i.e. “few studies have….”

• Contribution to theory and/or practice• the reason why we should know something we don’t already know –

lack of knowledge is not sufficient!• “Accordingly, the purpose of this study was to test the relationship

between …….. with the objective of contributing to …. by way of ….”

Research problem check list • Is it clearly formulated? (formulation vs. content)• Is it specific enough to be dealt with within existing time

and resource constraints? • Is it positioned – is it explained why it is interesting (cf.

research quality)?• Is it related to existing literature(s)?• Are the hypotheses related to it?

What does it take to make a scientific contribution?

• Contribution in terms of• theory• empirical• or both

• That something has not been investigated (well enough) yet is a necessary, but not a sufficient condition• as there may be a number of good reasons for lack of research on

a particular phenomenon

Some pitfalls you should avoid

• Not explicitly stating the contribution you set out to make• Stating it, but not explicitly enough

• exactly how will the study increase our understanding of the phenomenon

• Stating it, but too late in paper• Stating it in the beginning, but not in the discussion section• Ending up with a “collection of variables” with several (too)

minor contributions• minor contributions can mentioned in the discussion section

Some pitfalls you should avoid• Several (too) small contributions

• ending up with a collection of variables• adding every variable that may explain variance• pick one or two main contributions, leave minor ones to the discussion section

• Making a (too) small empirical contribution• replications in contexts where there are no compelling reasons that you will not

end up with the same findings/observations as in the original study• need to explicitly state why and how the findings/observations should be

different

The importance of active reading and writing• Don’t buy everything you read

• tactics in the business of publish or perish

• Don’t use rules of thumbs as rules without understanding them• statistical significance and α = 0.7 or 0.69

• choices of statistical methods and analyses (e.g. second order models, EFA/CFA, lack of discriminant validity)

• Don’t write something you don’t understand as lack of clarity won’t disappear by itself

• Be an active reader and writer

Normative research problems

• How to conduct performance appraisal that increases employee performance?• not suited for empirical testing• an interesting study contains much more than a discussion of factors (different Xs)

that may influence Y • e.g. in order for performance appraisal to increase employee performance it is

important to

• Reframe the research problem to contain something we don’t already know (enough about)• e.g. whether specific factors of performance appraisal are related to employee

performance

Research problems without variance in X when we ask whether X can explain Y

• Does an organization’s CSR practice influence employees’ affective commitment?• impossible to investigate with only one organization because X is a constant and

not a variable

• Look for differences between groups or units or significantly increase the number of organizations• Or, investigate employee perceptions of X, knowledge about X, use

of X etc.• Does employee perceptions of an organization’s CSR practice influence their

affective commitment?

Research problems where X does not exist yet

• Will implementing, increasing or changing X increase Y? • Will increasing pay differences among employees lead to perceptions of

injustice?

• It is better to investigate perceptions of the existing X (perceptions of or tolerance for pay differences)• Or, investigate antecedents to X (then X becomes Y), which may be

important to know before an organization implements, increases or otherwise changes X (e.g. differences between groups of employees)

Purely descriptive research problems

• Do managers and employees vary in their perceptions of an organization’s HR practices?• not very interesting

• Extend the research problem and include factors that may explain a potential variance in perceptions of X

Quality of research publications

• The Harzing-list provides several rankings• http://www.harzing.com/

• Australian Business Deans’ Council Journal List• http://www.abdc.edu.au/• Four levels: A*, A, B, C• 11 Disciplines: Business and Management

A*-Journals (28/424 = 6.60%)

Best or leading journal in its field Publishes outstanding, original and rigorous

research that will shape the field Acceptance rates are typically low and the editorial

board is dominated by leading scholars in the field or subfield, including from top institutions in the world

Where relevant to the field or subfield, the journal has the highest impact factors or other indices of high reputation

Acceptance rates usually less than 10%

A selection of A*-JournalsAcademy of Management JournalAcademy of Management Review Administrative Science QuarterlyBritish Journal of Industrial RelationsCalifornia Management Review Decision SciencesEntrepreneurship: Theory and PracticeHarvard Business ReviewHuman Resource Management (US)Industrial RelationsJournal of Applied PsychologyJournal of Business VenturingJournal of Int. Business Studies Journal of Management

Journal of Management StudiesJournal of Operations ManagementJournal of Organizational BehaviorJournal of Personality and Social PsychProduct Innovation ManagementJournal of Vocational BehaviorLeadership QuarterlyManagement Science Operations ResearchOrganization ScienceOrganization StudiesOrg. Behavior & Human Decision Proc.Personnel PsychologyStrategic Management Journal

A-Journals (68/424 = 16.04%)• Highly regarded journal in the field or subfield• Publishes excellent research in terms of originality,

significance and rigour, has competitive submission and acceptance rates, excellent refereeing process and where relevant to the field or subfield, has higher than average impact factors• Not all highly regarded journals have high impact factors,

especially those in niche areas• Acceptance rates 10% - 20%

A selection of A-journalsAcademy of Management PerspectivesAcademy of Management Learning and EducationAnnals of Operations ResearchApplied Psychology: An Int. ReviewBritish Journal of Social PsychologyBusiness Ethics QuarterlyEuropean Journal of Industrial RelationsFamily Business ReviewGroup & Organization ManagementHuman PerformanceHuman RelationsHuman Resource Mgmt Journal (UK)

Journal of Industrial RelationsJournal of Occupational Health PsychologyJournal of Occupational and Org. Psych.Journal of Small Business ManagementJournal of World BusinessLong Range PlanningManagement International ReviewManagement LearningOrganizationOrganizational Dynamics Organizational Research MethodsSupply Chain ManagementWork and Occupations

B-Journals (97/424 = 22.88%)• Well regarded journal in the field or subfield• Publishes research of a good standard in terms of

originality, significance and rigor and papers are fully refereed according to good standards and practices but acceptance rates are higher than for Tiers A* and A• Depending on the field or sub-field, will have a modest

impact factor and will be ISI listed• Acceptance rates 20%-50%

A selection of B-journalsCanadian Journal of Adm. SciencesEmployee RelationsEuropean Business ReviewEuropean Journal of Work and Organizational PsychologyHuman Resource Development Int.Human Resource Development Quart.Human Resource Development Review Human Resource Management Review Industrial Relations JournalIntegrated Manufacturing SystemsInternational Journal of Conflict Mngt.International Journal of Cross-Cultural Management

Journal of Applied Behavioral Science Journal of Business and PsychologyJournal of General ManagementJournal of Labour ResearchJournal of Management EducationJournal of Management InquiryJournal of Organizational Behavior Mngt.Journal of Org. Change ManagementLabor Studies JournalLabour and IndustryLeadership and Org. Dev. JournalManagement Communication QuarterlyNonprofit Management & LeadershipOrganization Development Journal

C-Journals (231/424 = 54.48%)

• A recognized journal

• Publishes research that is of a modest standard and/or is yet to establish its reputation because of its newness

• This tier is more inclusive than the others but only includes refereed journals

• Acceptance rates typically > 50%

• Remember that most journals are not listed!

A selection of C-journals

Advances in Developing Human Res.Advancing Women in LeadershipAsia-Pacific Journal of Human Res.Career Development InternationalCulture and OrganizationDevelopment and Learning in Organizations: An International Journal Employment Relations RecordEqual Opportunities InternationalEuropean Business JournalEuropean Management JournalEuropean Management ReviewHuman Systems ManagementIntern. Employment Relations Rev.

International Journal of ManagementIntern. J. of Mngt. and Decision MakingIntern. Journal of Management ReviewsIntern. Journal of Organization BehaviourIntern. J. of Training and DevelopmentIntern. J. of Work Org. and EmotionIntern. Rev. of Industrial & OrganisationalIntern. Review of Women and LeadershipJournal of Change ManagementScandinavian Journal of Management

Relevance versus Rigor

Top journals = top research?• Dissensus among reviewers

• interrater reliability typically less than 0.30

• Low acceptance rates• “Highly prestigious journals publish quite a few low-value articles, low prestige

journals publish some excellent articles, and excellent manuscripts may receive successive rejections from several journals”

• Variation in article impact • 12% of articles published in top journals are not cited• less than 4% generate more than 100 citations• reviewers are not able to predict the future impact of an article (r = .14)

Glick, W. H., Miller, C. C., & Cardinal, L. B. (2007). Making a life in the field of organization science. Journal of Organizational Behavior, 28(7), 817-835.

Conclusions

• Don’t equate top journals with top research• although usually more rigor (multiple data sources, better

measures, longitudinal designs etc.)

• Don’t buy everything you read• good journals also publish low quality research• top scholars sometimes publish low quality research