relevance in academic research dan remenyi phd visiting professor school of systems and data studies...

Relevance in academic researchDan Remenyi PhDVisiting ProfessorSchool of Systems and Data StudiesTrinity College Dublindan.remenyi@tcd.ie

Establishing the context•The issues of relevance and rigor are complex

concepts and need careful understanding.•Some words have a large semantic footprint

and relevance, rigor, validity, reliability, generalisability and integrity are good examples of these.

•No one can proceed with any understanding of research without having a clear agreement with regards the meaning of terms.

April 21, 2023

2

The effectiveness and efficiency parallel •A quick rule of thumb understanding of

relevance is doing the right thing or looking at the right issues

•A quick rule of thumb understanding of rigor is doing the thing right

•These are the colloquialisms coined by Peter Drucker to explain effectiveness and efficiency

•Thus relevance equates to effectiveness and rigor relates to ……… well… not so much efficiency but something more like adequacy

April 21, 2023

3

Relevance and Rigor•Relevance is a high level issue in that you

can say that the research project is relevant or not relevant as a whole …………it could also be said that the use of case studies was not relevant to the objectives of the study

•Rigor is an issue which permeates the whole project in that you can say that the literature review was not rigorously approached or the sample was not rigorously chosen, or the conclusions were not rigorously written up.

5

21 April 2023

Academic Research is a Challenge

Academic research

•What is the nature of academic research?▫It is an intellectual activity▫It is an attempt to answer a question▫It is a process of question, data and analysis

•Academic research has two primary attributes:-▫It adds something of value to the body of

theoretical knowledge…….and practice!▫It demonstrate a high level of scholarship

April 21, 2023

6

Something of value to the body of theoretical knowledge…….and practice•Something of value is reflected in the

concept of relevance.•Relevance needs to be perceived by both

the non-academic stakeholders and also the academic community.

•Without some contribution to theory the research will not be much valued by academics.

•Mode 2 brings both together.

Scholarship and rigor•Being well read and being able to

demonstrate that in conversation•Being able to craft an argument in writing•Scholarship is connected to rigor•An argument is a series of logical steps by

which someone is persuaded. In this sense is not confrontational.

•Academic work i.e. either a dissertation or a peer reviewed paper needs to argue for relevance, rigor, validity, reliability, generalisability.

Quality academic research• Dimensions of quality in academic research

addresses a pertinent question uses academic language with care - remember the

fog factor uses appropriate data is written up in intelligible English and correct

format is presented in an attractive way - structure invoking authority finds an acceptable/useful answer

• Being parsimonious quality academic research needs to be relevant and rigorous

April 21, 2023

9

Double hurdle

•Academic research needs to be relevant i.e. of interest and of use to a wide range of management at different levels and rigorous i.e. display academic excellence.

•Being both relevant and rigorous has been described as a synthesis strategy.

Research question

•The research question has been reached through a careful examination of and understanding of :1. The field of study2. The topic3. The specific questions and sub-questions

April 21, 2023

11

Some critical issues

• Something of value As judged by the academic community

• Body of theoretical knowledge Add something to established domains

• Parsimonious Expressed in a precise and economical way

• Authority Ideas/concepts/theories which have through the

peer review process• Relevance

Has value to a group of stakeholders• Rigor

Complies with the rules of the research paradigm

April 21, 2023

12

Three more issues•Validity ……………..Has the research

addressed the intended issues i.e. addressed the issues which are required to answer the research question?

•Reliability…………..Has the research been conducted sufficiently well (rigorously) to provide results in which we can have confidence?

•Generalisabilty……..To what extent can the research be applied else where?

April 21, 2023

13

Rationalism versus Empiricism• Rationalism (theory)

The mind is superior to the senses – dating from the Greeks……at least Plato…..thought experiments

Reason Logic Secondary data

• Empiricism If it cannot be held, felt or shaken etc it does not

exist! In its pure form it fails due to particle physics etc Positivism versus interpretivism Primary data

April 21, 2023

14

Assumptions - preliminary concepts• What are the assumptions under which academic research

is conducted?▫ The world is understandable.▫ We are interested in understanding it. ▫ Researchers will be open minded.▫ Researchers are capable of objective reasoning.▫ We can find data/evidence to explore the phenomena and

therefore understand them.▫ The nature of the understanding can be both broad and also

detailed.▫ Occam’s razor points out that too much complexity can

obscure what is happening.▫ We always have to use ceteris paribus – all other things being

equal.▫ That some degree of cause and effect can be

identified/assumed. ▫ We all operate in terms of our cognitive capacity which

changes, and hopefully develops/increases over time.

April 21, 2023

15

21 April 2023

16

The Issue of Integrity – Fleischmann and Pons

21 April 2023

17

Integrity and Professor Woo Suk Hwang And Professor Sir Cyril Burt

Sir Cyril BurtReported nonexistent studies on I.Q. tests in the 20’s and 30’s, and added fictitious collaborators to his papers

Some ideas to examine • Relevance ………..doing the right thing• Relevance ………..to whom or to what?

theory/academics/practitioners......Is it significant?

• Rigor ……………….doing the thing right• Is rigor a dimension of validity?• If the research is not valid then do not worry

about reliability? • If the research is not reliable can it be valid?• If the research is neither valid or reliable then is

it worthless?• What can go wrong question, data, analysis,

finding conclusions etc……threats to the research

April 21, 2023

18

Evaluation of relevance and rigor• Relevance is normally evaluated holistically i.e. is

the research important or is it not. Sometimes some aspects of the research may be more useful than others but this question is essentially a holistic one.

• Rigor is normally evaluated on an issue by issue basis. This requires comment of individual aspects of the research. But there is also a holistic view. There may be some tension between the individual evaluations and a holistic view of the research. The sum of the parts may be different to the whole.

April 21, 2023

19

Relevance• In information science “Relevance is the extent to

which some information is pertinent, connected, or applicable to the matter at hand. It represents a key concept in the fields of documentation, information science, and information retrieval”. Pehcevski and Larsen

• Although academic relevance is today of essential importance to research, surprising little has been written about it.

• It used to have been assumed that if the research is “good” then it is academically relevant and this attitude has only recently been challenged.

• The main drivers of this challenge to relevance may be seen in the debate surrounding Mode 1 and Mode 2 research.

April 21, 2023

20

A stumbling block issue•Previously relevance was not a visible issue.

If the research and his/her supervisor thought that the question was important then that was all that was needed.

•Now the research has to convince the examiners or the reviewers that it is adequately relevant. An argument has to be made.

•A research dissertation/paper could be refused/rejected on the grounds that the research is not adequately relevant

April 21, 2023

21

Weak and strong relevance•According to Blake Ives (US IS academic)

there are two forms of relevance - weak relevance and strong relevance - http://www.misq.org/archivist/vol/no20/issue3/edstat.html

•Weak relevance occurs when a research question addresses an issue which is known and documented

•Strong relevance requires an action research approach to answering the research question which produces concrete findings

A closer look at relevance•The statement that a research question is

relevant begs the question of “relevant to whom?” and “in which way?”

•A research question is relevant if there is:- A distinct interest among the stakeholders – but

who are the stakeholders? A material stakeholder group

The research question will provide interesting results and/or useful guidelines to professionals

Improvements to practice i.e. there is clear application for the findins

April 21, 2023

23

How are relevant research questions to be identified?• Relevant research questions may be found in:-

▫ the extant literature▫work place experiences▫special interest groups▫Business/professional/practitioner groups▫government requests▫general business press

• Remember that a relevant question leads to a solution for someone

• Is there “pure” or “basic” research in B&M Studies?

April 21, 2023

24

What do we need to find a relevant research question?• Imagination!• …….. But also needed is imagination to create from

these hints the great generalizations – to guess at the wonderful, simple, but very strange patterns beneath them all, and then to experiment to check again whether we made the right guess. Richard Feynman, Six Easy Pieces, p2, Penguin Books, London (1995)

• Finding a suitable research question should not be rushed.

• Activities which can help to find relevant research questions include:-

Talking to knowledgeable informants Focus groups Qualitative questionnaires Blue sky and scenario sessions Talking to your supervisor/s Literature………academic and popular press

21 April 2023

25

A caveat!• The relevance will be judged by both

academics and by practitioners but academics will/may have a greater say than practitioners.

• The degree of relevance is often tested by the “So What?” or “What’s new ....... and who cares?” question. This question needs to be directly answered in a research dissertation and in a peer reviewed academic paper.

• Relevance or the lack thereof will always be known post hoc or a posteriori but you have to address this issue on an a priori basis

April 21, 2023

26

The time dimension

•Relevance has a time dimension.•What is relevant today may not be

relevant tomorrow.•The time value of a research question is a

function of the field of study and the topic and directly influences the relevance of the research.

April 21, 2023

27

The argument

•Relevance may not be taken for granted.•The researcher needs to argue that the

research is relevant because it:- answers a well known /important question/s addressed a non-obscure question will produce a useful result usually a paradigm will allows several other questions to be

explored it will attract people and funding to the

university or business school etc

April 21, 2023

28

The argument is one of the key issues for establishing relevance and rigor

April 21, 2023

29

Assessing relevance

•Feedback from informants and peers is the most effective way of ensuring that the research is relevant.

•It is useful to have in mind the need to test this relevance occasionally through out the duration of the research

•Also test the findings by asking the stakeholders of the results of the research are likely to be useful to them.

April 21, 2023

30

Journal titles......Relevant?• Stark, S., Chernyshenko, O., & Guenole, N.

(Accepted).  Can subject matter expert ratings of statement extremity be used to streamline the development of unidimensional pairwise preference scales. (18 no “?”)  Organizational Research Methods

• Chernyshenko, O., Stark, S., & Guenole, N. (2007). Can the discretionary nature of certain performance criteria lead to differential prediction across cultural groups? (15 yes or no answer) International Journal of Selection and Assessment, 15, 175-184

• Be careful of papers with very long titles and many authors.

April 21, 2023

31