FROM THE EDITORS

PUBLISHING IN AMJ—PART 2: RESEARCH DESIGN

Editor’s Note:

This editorial continues a seven-part series, “Publishing in AMJ,” in which the editors give suggestions and advice forimproving the quality of submissions to the Journal. The series offers “bumper to bumper” coverage, with installmentsranging from topic choice to crafting a Discussion section. The series will continue in October with “Part 3: Setting theHook.”- J.A.C.

Most scholars, as part of their doctoral education,take a research methodology course in which theylearn the basics of good research design, includingthat design should be driven by the questions beingasked and that threats to validity should be avoided.For this reason, there is little novelty in our discus-sion of research design. Rather, we focus on commondesign issues that lead to rejected manuscripts atAMJ. The practical problem confronting researchersas they design studies is that (a) there are no hard andfast rules to apply; matching research design to re-search questions is as much art as science; and (b)external factors sometimes constrain researchers’ability to carry out optimal designs (McGrath, 1981).

Access to organizations, the people in them, andrich data about them present a significant challengefor management scholars, but if such constraintsbecome the central driver of design decisions, theoutcome is a manuscript with many plausible al-ternative explanations for the results, which leadsultimately to rejection and the waste of consider-able time, effort, and money. Choosing the appro-priate design is critical to the success of a manu-script at AMJ, in part because the fundamentaldesign of a study cannot be altered during the re-vision process. Decisions made during the researchdesign process ultimately impact the degree of con-fidence readers can place in the conclusions drawnfrom a study, the degree to which the results pro-vide a strong test of the researcher’s arguments, andthe degree to which alternative explanations can bediscounted. In reviewing articles that have beenrejected by AMJ during the past year, we identifiedthree broad design problems that were commonsources of rejection: (a) mismatch between researchquestion and design, (b) measurement and opera-tional issues (i.e., construct validity), and (c) inap-propriate or incomplete model specification.

Matching Research Question and Design

Cross-sectional data. Use of cross-sectional datais a common cause of rejection at AMJ, of both micro

and macro research. Rejection does not happen be-cause such data are inherently flawed or becausereviewers or editors are biased against such data. Ithappens because many (perhaps most) research ques-tions in management implicitly—even if not framedas such—address issues of change. The problem withcross-sectional data is that they are mismatched withresearch questions that implicitly or explicitly dealwith causality or change, strong tests of which requireeither measurement of some variable more than once,or manipulation of one variable that is subsequentlylinked to another. For example, research addressingsuch topics as the effects of changes in organizationalleadership on a firm’s investment patterns, the effectsof CEO or TMT stock options on a firm’s actions, orthe effects of changes in industry structure on behav-ior implicitly addresses causality and change. Simi-larly, when researchers posit that managerial behav-ior affects employee motivation, that HR practicesreduce turnover, or that gender stereotypes constrainthe advancement of women managers, they are alsoimplicitly testing change and thus cannot conductadequate tests with cross-sectional data, regardless ofwhether that data was drawn from a pre-existing database or collected via an employee survey. Researcherssimply cannot develop strong causal attributionswith cross-sectional data, nor can they establishchange, regardless of which analytical tools they use.Instead, longitudinal, panel, or experimental data areneeded to make inferences about change or to estab-lish strong causal inferences. For example, Nyberg,Fulmer, Gerhart, and Carpenter (2010) created a panelset of data and used fixed-effects regression to modelthe degree to which CEO-shareholder financial align-ment influences future shareholder returns. This datastructure allowed the researchers to control for cross-firm heterogeneity and appropriately model howchanges in alignment within firms influenced share-holder returns.

Our point is not to denigrate the potential use-fulness of cross-sectional data. Rather, we point outthe importance of carefully matching research designto research question, so that a study or set of studies

� Academy of Management Journal2011, Vol. 54, No. 4, 657–660.

657

Copyright of the Academy of Management, all rights reserved. Contents may not be copied, emailed, posted to a listserv, or otherwise transmitted without the copyright holder’s expresswritten permission. Users may print, download or email articles for individual use only.

is capable of testing the question of interest. Research-ers should ask themselves during the design stagewhether their underlying question can actually beanswered with their chosen design. If the questioninvolves change or causal associations between vari-ables (any mediation study implies causal associa-tions), cross-sectional data are a poor choice.

Inappropriate samples and procedures. Much or-ganizational research, including that published inAMJ, uses convenience samples, simulated businesssituations, or artificial tasks. From a design stand-point, the issue is whether the sample and proceduresare appropriate for the research question. Asking stu-dents with limited work experience to participate inexperimental research in which they make executiveselection decisions may not be an appropriate way totest the effects of gender stereotypes on reactionsto male and female managers. But asking these samestudents to participate in a scenario-based experi-ment in which they select the manager they wouldprefer to work for may present a good fit betweensample and research question. Illustrating this notionof matching research question with sample is a studyon the valuation of equity-based pay in which Devers,Wiseman, and Holmes (2007) used a sample of exec-utive MBA students, nearly all of whom had experi-ence with contingent pay. The same care used inchoosing a sample needs to be taken in matchingprocedures to research question. If a study involvesan unfolding scenario wherein a subject makes a se-ries of decisions over time, responding to feedbackabout these decisions, researchers will be well servedby collecting data over time, rather than having aseries of decision and feedback points contained in asingle 45 minute laboratory session.

Our point is not to suggest that certain samples(e.g., executives or students) or procedures are inher-ently better than others. Indeed, at AMJ we explicitlyencourage experimental research because it is an ex-cellent way to address questions of causality, and werecognize that important questions—especially thosethat deal with psychological process—can often beanswered equally well with university students ororganizational employees (see AMJ’s August 2008From the Editors [vol. 51: 616–620]). What we ask ofauthors—whether their research occurs in the lab orthe field—is that they match their sample and proce-dures to their research question and clearly make thecase in their manuscript for why these sample orprocedures are appropriate.

Measurement and Operationalization

Researchers often think of validity once they be-gin operationalizing constructs, but this may be toolate. Prior to making operational decisions, an au-

thor developing a new construct must clearly artic-ulate the definition and boundaries of the new con-struct, map its association with existing constructs,and avoid assumptions that scales with the samename reflect the same construct and that scales withdifferent names reflect different constructs (i.e., jinglejangle fallacies [Block, 1995]). Failure to define thecore construct often leads to inconsistency in a man-uscript. For example, in writing a paper, authors mayinitially focus on one construct, such as organization-al legitimacy, but later couch the discussion in termsof a different but related construct, such as reputationor status. In such cases, reviewers are left without aclear understanding of the intended construct or itstheoretical meaning. Although developing theory isnot a specific component of research design, readersand reviewers of a manuscript should be able toclearly understand the conceptual meaning of a con-struct and see evidence that it has been appropriatelymeasured.

Inappropriate adaptation of existing measures.A key challenge for researchers who collect fielddata is getting organizations and managers to com-ply, and survey length is frequently a point of con-cern. An easy way to reduce survey length is toeliminate items. Problems arise, however, whenresearchers pick and choose items from existingscales (or rewrite them to better reflect their uniquecontext) without providing supporting validity ev-idence. There are several ways to address this prob-lem. First, if a manuscript includes new (or sub-stantially altered measures), all the items should beincluded in the manuscript, typically in an appen-dix. This allows reviewers to examine the face va-lidity of the new measures. Second, authors mightinclude both measures (the original and the short-ened versions) in a subsample or in an entirelydifferent sample as a way of demonstrating highconvergent validity between them. Even betterwould be including several other key variables inthe nomological network, to demonstrate that thenew or altered measure is related to other similarand dissimilar constructs.

Inappropriate application of existing mea-sures. Another way to raise red flags with review-ers is to use existing measures to assess completelydifferent constructs. We see this problem occurringparticularly among users of large databases. Forexample, if prior studies have used an action suchas change in format (e.g., by a restaurant) as a mea-sure of strategic change, and a submitted paper usesthis same action (change in format) as a measure oforganizational search, we are left with little confi-dence that the authors have measured their in-tended construct. Given the cumulative and incre-mental nature of the research process, it is critical

658 AugustAcademy of Management Journal

that authors establish both the uniqueness of theirnew construct, how it relates to existing constructs,and the validity of their operationalization.

Common method variance. We see many rejectedAMJ manuscripts in which data are not only cross-sectional, but are also assessed via a common method(e.g., a survey will have multiple predictor and crite-rion variables completed by a single individual).Common method variance presents a serious threat tointerpretation of observed correlations, because suchcorrelations may be the result of systematic errorvariance due to measurement methods, includingrater effects, item effects, or context effects. Podsa-koff, MacKenzie, Lee, and Podsakoff (2003) dis-cussed common method variance in detail and alsosuggested ways to reduce its biasing effects (seealso Conway & Lance, 2010).

Problems of measurement and operationalizationof key variables in AMJ manuscripts have implica-tions well beyond psychometrics. At a conceptuallevel, sloppy and imprecise definition and opera-tionalization of key variables threaten the infer-ences that can be drawn from the research. If thenature and measurement of underlying constructsare not well established, a reader is left with littleconfidence that the authors have actually tested themodel they propose, and reasonable reviewers canfind multiple plausible interpretations for the re-sults. As a practical matter, imprecise operationaland conceptual definitions also make it difficult toquantitatively aggregate research findings acrossstudies (i.e., to do meta-analysis).

Model Specification

One of the challenges of specifying a theoreticalmodel is that it is practically not feasible to includeevery possible control variable and mediating pro-cess, because the relevant variables may not exist inthe database being used, or because organizationsconstrain the length of surveys. Yet careful atten-tion to the inclusion of key controls and mediatingprocesses during the design stage can provide sub-stantial payback during the review process.

Proper inclusion of control variables. The in-clusion of appropriate controls allows researchersto draw more definitive conclusions from theirstudies. Research can err on the side of too few ortoo many controls. Control variables should meetthree conditions for inclusion in a study (Becker,2005; James, 1980). First, there is a strong expecta-tion that the variable be correlated with the depen-dent variable owing to a clear theoretical tie orprior empirical research. Second, there is a strongexpectation that the control variable be correlatedwith the hypothesized independent variable(s).

Third, there is a logical reason that the controlvariable is not a more central variable in the study,either a hypothesized one or a mediator. If a vari-able meeting these three conditions is excludedfrom the study, the results may suffer from omittedvariable bias. However, if control variables are in-cluded that don’t meet these three tests, they mayhamper the study by unnecessarily soaking up de-grees of freedom or bias the findings related to thehypothesized variables (increasing either type I ortype II error) (Becker, 2005). Thus, researchersshould think carefully about the controls they in-clude—being sure to include proper controls butexcluding superfluous ones.

Operationalizing mediators. A unique charac-teristic of articles in AMJ is that they are expectedto test, build, or extend theory, which often takesthe form of explaining why a set of variables arerelated. But theory alone isn’t enough; it is alsoimportant that mediating processes be tested em-pirically. The question of when mediators shouldbe included in a model (and which mediators)needs to be addressed in the design stage. When anarea of inquiry is new, the focus may be on estab-lishing a causal link between two variables. But,once an association has been established, it be-comes critical for researchers to clearly describeand measure the process by which variable A af-fects variable B. As an area of inquiry becomesmore mature, multiple mediators may need to beincluded. For example, one strength of the trans-formational leadership literature is that many me-diating processes have been studied (e.g., LMX[Kark, Shamir, & Chen, 2003; Pillai, Schriesheim, &Williams, 1999; Wang, Law, Hackett, Wang, &Chen, 2005]), but a weakness of this literature isthat most of these mediators, even when they areconceptually related to each other, are studied inisolation. Typically, each is treated as if it is theunique process by which managerial actions influ-ence employee attitudes and behavior, and otherknown mediators are not considered. Failing toassess known, and conceptually related mediators,makes it difficult for authors to convince reviewersthat their contribution is a novel one.

Conclusion

Although research methodologies evolve overtime, there has been little change in the fundamen-tal principles of good research design: match yourdesign to your question, match construct definitionwith operationalization, carefully specify yourmodel, use measures with established construct va-lidity or provide such evidence, choose samplesand procedures that are appropriate to your unique

2011 659Bono and McNamara

research question. The core problem with AMJ sub-missions rejected for design problems is not thatthey were well-designed studies that ran into prob-lems during execution (though this undoubtedlyhappens); it is that the researchers made too manycompromises at the design stage. Whether a re-searcher depends on existing databases, activelycollects data in organizations, or conducts experi-mental research, compromises are a reality of theresearch process. The challenge is to not compro-mise too much (Kulka, 1981).

A pragmatic approach to research design startswith the assumption that most single-study designsare flawed in some way (with respect to validity).The best approach, then, to a strong research designmay not lie in eliminating threats to validity(though they can certainly be reduced during thedesign process), but rather in conducting a series ofstudies. Each study in a series will have its ownflaws, but together the studies may allow for stron-ger inferences and more generalizable results thanwould any single study on its own. In our view,multiple study and multiple sample designs arevastly underutilized in the organizational sciencesand in AMJ submissions. We encourage researchersto consider the use of multiple studies or samples,each addressing flaws in the other. This can bedone by combining field studies with laboratoryexperiments (e.g., Grant & Berry, 2011), or by test-ing multiple industry data sets to assess the robust-ness of findings (e.g., Beck, Bruderl, & Woywode,2008). As noted in AMJ’s “Information for Contrib-utors,” it is acceptable for multiple study manu-scripts to exceed the 40-page guideline.

A large percentage of manuscripts submitted toAMJ that are either never sent out for review or thatfare poorly in the review process (i.e., all three re-viewers recommend rejection) have flawed designs,but manuscripts published in AMJ are not perfect.They sometimes have designs that cannot fully an-swer their underlying questions, sometimes usepoorly validated measures, and sometimes have mis-specified models. Addressing all possible threats tovalidity in each and every study would be impossiblycomplicated, and empirical research might never getconducted (Kulka, 1981). But honestly assessingthreats to validity during the design stage of a re-search effort and taking steps to minimize them—either via improving a single study or conductingmultiple studies—will substantially improve the po-tential for an ultimately positive outcome.

Joyce E. BonoUniversity of Florida

Gerry McNamaraMichigan State University

REFERENCES

Beck, N., Bruderl, J., & Woywode, M. 2008. Momentumor deceleration? Theoretical and methodological re-flections on the analysis of organizational change.Academy of Management Journal, 51: 413–435.

Becker, T. E. 2005. Potential problems in the statisticalcontrol of variables in organizational research: Aqualitative analysis with recommendations. Organi-zational Research Methods, 8: 274–289.

Block, J. 1995. A contrarian view of the five-factor ap-proach to personality description. PsychologicalBulletin, 117: 187–215.

Conway, J. M., & Lance, C. E. 2010. What reviewersshould expect from authors regarding commonmethod bias in organizational research. Journal ofBusiness and Psychology, 25: 325–334.

Devers, C. E., Wiseman, R. M., & Holmes, R. M. 2007. Theeffects of endowment and loss aversion in manage-rial stock option valuation. Academy of Manage-ment Journal, 50: 191–208.

Grant, A. M., & Berry, J. W. 2011. The necessity of othersis the mother of invention: Intrinsic and prosocialmotivations, perspective taking, and creativity.Academy of Management Journal, 54: 73–96.

James, L. R. 1980. The unmeasured variables problem inpath analysis. Journal of Applied Psychology, 65:415–421.

Kark, R., Shamir, B., & Chen, G. 2003. The two faces oftransformational leadership: Empowerment and de-pendency. Journal of Applied Psychology, 88: 246–255.

Kulka, R. A. 1981. Idiosyncrasy and circumstance. Amer-ican Behavioral Scientist, 25: 153–178.

McGrath, J. E. 1981. Introduction. American BehavioralScientist, 25: 127–130.

Nyberg, A. J., Fulmer, I. S., Gerhart, B., & Carpenter, M. A.2010. Agency theory revisited: CEO return andshareholder interest alignment. Academy of Man-agement Journal, 53: 1029–1049.

Pillai, R., Schriesheim, C. A., & Williams, E. S. 1999. Fair-ness perceptions and trust as mediators for transforma-tional and transactional leadership: A two-samplestudy. Journal of Management, 25: 897–933.

Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. 2003.Common method biases in behavioral research: Acritical review of the literature and recommendedremedies. Journal of Applied Psychology, 25: 879–903.

Wang, H., Law, K. S., Hackett, R. D., Wang, D., & Chen,Z. X. 2005. Leader-member exchange as a mediatorof the relationship between transformational leader-ship and followers’ performance and organizationalcitizenship behavior. Academy of ManagementJournal, 48: 420–432.

660 AugustAcademy of Management Journal

Copyright of Academy of Management Journal is the property of Academy of Management and its content may

not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written

permission. However, users may print, download, or email articles for individual use.