![Page 1: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/1.jpg)
Producing Good Research: Behavioral and Otherwise
American Accounting Association Auditing Doctoral Consortium
Clearwater, Florida.January 14, 2004
Jane Jollineau KennedyUniversity of Washington
![Page 2: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/2.jpg)
Today’s Outline
I. What is good research (experimental and otherwise)?
II. Why do experiments?
III. Finding a topic in Behavioral (auditing) research
IV. Common Pitfalls
![Page 3: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/3.jpg)
I. What is good research?
A. Good topic
B. Good design
![Page 4: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/4.jpg)
A. What is a good research topic?
• Problem is described clearly, comprehensively• Important constituents care about it!• Problem is challenging but not impossible• Possible to evaluate solutions or interpret the
direction of results• The results will add to our knowledge about the
problem and have theoretical and/or practical ramifications
![Page 5: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/5.jpg)
For every research idea/paper write 3 paragraphs answering the following:
1. What problem or issue will be addressed?
1. Why is the problem or issue important?
1. How you will address the problem (and what do you expect to find)?
Tip: Kinney’s 3 Paragraphs (footnote 18 in his 1986 TAR article)
![Page 6: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/6.jpg)
My Example Paper for Today
Frank Hodge, TAR, October 2001:
“Hyperlinking unaudited information to audited financial statements: effects on investor judgments”
Test of Kinney’s 3 questions
![Page 7: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/7.jpg)
Q1: Hodge 2001
“This study investigates whether firms can influence investors’ perceptions of their financial reports by hyperlinking unaudited information to information in the audited financial statements.”
![Page 8: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/8.jpg)
Hodge 2001 Q1 continued
Specifically, does hyperlinking optimistic, unaudited information about a firm’s prospects to audited financial statements lead investors to:– Misclassify unaudited info. as audited, – Inflate the credibility of unaudited info.,– Judge the firm’s earnings potential to be higher,
relative to viewing the same info. in hardcopy format?
Will a warning help?
![Page 9: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/9.jpg)
Q2: Who cares about this problem?
• Management
• SEC
• FASB
• AICPA
• Accounting/auditing researchers
• System designers(?)
![Page 10: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/10.jpg)
Q3: Method and results?
• Method: – Used an experiment. Why? Data not observable
from investors’ investment decisions. Can control info. and extraneous factors re. people, tasks, and environment.
• Results:– Hyperlinked info was perceived as more credible– Firm judged to have greater earnings potential– Simple labels “(Un)audited” reduced the effects
![Page 11: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/11.jpg)
B. What is good Research design?
• Theory answers “why?”
• Construct Validity
• Internal Validity
• External Validity
• Giving yourself a decent chance of testing your theory and finding results
![Page 12: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/12.jpg)
Y = f ( X, Vs, Zs )
Y = phenomenon to be explained
X = your (new) theory about a cause of Y
Vs = prior causes of Y
Zs = contemporaneous causes of Y
Model & Definitions (Kinney ’86)
![Page 13: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/13.jpg)
A simple model
{V-3, V -2, V -1 } Y1
X0
Z0
?
![Page 14: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/14.jpg)
Building a Research Model
Control Other potentially influential variables: Vs and Zs
OperationalX
OperationalY
Operational
TheoreticalX
TheoreticalY
Conceptual
Independent Dependent
![Page 15: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/15.jpg)
For X to cause Y:
• X and Y are correlated• Alternative explanations are ruled out by
design, including– Y causes X– X and Y caused by an omitted V or Z
• Reason to believe that operational X and Y represent X and Y
• Reason to believe that X : Y relation generalizes to other persons, times, and settings.
![Page 16: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/16.jpg)
. . .threats to Predictive Validity
• Statistical Conclusion Validity (5)
• Internal Validity (4)
• Construct Validity (2 and 3)
• External Validity (1)
![Page 17: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/17.jpg)
Threats to Validity (cont’d)
5: Stat. Val.
4: Internal Validity
2:Construct Validity 3: Construct Validity
1: Ext. Val.
Control Other potentially influential variables: Vs and Zs
OperationalX
OperationalY
Operational
TheoreticalX
TheoreticalY
Conceptual
Independent Dependent
![Page 18: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/18.jpg)
Research Model for Hodge 2001
FinancialDecisions
Assoc. of unaudited Information with
Audited F/S
1
Hyperlinked Letter to F/S
Credibility and Earnings
Judgments
2 3
5
Accounting & finance courses; investing experience
4
Conceptual
Operational
Control
Independent Dependent
![Page 19: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/19.jpg)
To achieve Internal Validity:
“account” for Vs and Zs by:
• random assignment of participants to treatment/control
• estimate statistically and remove (covariate analysis, regression)
• match on Vs
• hold constant by design/selection
(special case of matching)
![Page 20: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/20.jpg)
The Researcher’s Problem (Kinney 1986)
or Giving Yourself a Chance to find Results
= risk that data incorrectly “accepts” new theory = risk that data incorrectly “rejects” new theory = “true” effect of X on Y
= residual variation given research design (i.e., after effects of Vs and Zs)
n = available sample size
All five are related through a single, simple formula
![Page 21: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/21.jpg)
Researcher’s Problem (continued)
is fixed at .05 or .10 by convention
is the risk of failing to detect ‘true’ phenomenon (you want to minimize )
= f(X)
= f(Vs, Zs)n is semi-fixed by data availability or cost
= f ( n ) - - + -
![Page 22: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/22.jpg)
Graphically . . . (small , small okay)
_Y| H0, n,
_Y| HA, n,
0
Accept H0 Reject H0
Y
![Page 23: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/23.jpg)
Graphically (large , large okay)
_Y| H0, n,
_Y| HA, n,
0
Accept H0 Reject H0
Y
![Page 24: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/24.jpg)
Graphically… (small , large
_Y| H0, n,
_Y| HA, n,
0
Accept H0 Reject H0
Y
yikes!)
![Page 25: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/25.jpg)
What to do? Need Power!
• Increase sample size (not always possible)
• Improve research design– Theory– Have you identified and measured all the major
V’s and Z’s?– Have you manipulated the X’s at appropriate
levels that allow you to find effects?– Manipulation checks?
![Page 26: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/26.jpg)
II. Why use experiments?
• Causal theory can be tested (not just association)
• Manipulate the variables of interest and control or randomize on others
• Can get at “process” not just outcomes• Ex ante research is possible• Some conditions that do not exist in natural
settings can be created in the lab
![Page 27: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/27.jpg)
(Difficult but) Necessary Choices in Experiments
1. Professional participants?
2. Monetary incentives?
3. Between- or within-subjects design (see Schepanski, Tubbs, and Grimlund 1992)?
4. Level of analysis, individual or market (see Libby, Bloomfield & Nelson 2002)?
![Page 28: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/28.jpg)
1. Choice of participants?
• Theory should dictate this choice• Negative externalities of using professionals
when not needed: limited resource (Libby, Bloomfield & Nelson 2002)
• Selection bias is stronger with professionals than with more available groups like students (Peecher & Solomon 2001)
• Hodge 2001 used MBA students as investors• Elliott, Hodge and Kennedy (2004)
![Page 29: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/29.jpg)
2. Monetary incentives?
• Dictated by theory• Want participants to take the task seriously• Many results driven more by ability than
incentives (running a 4-minute mile)• Incentives do not necessarily mitigate
judgment bias– in some cases reduce (see Camerer & Hogarth 1999)
• Incentives are not always possible or practical in real life either!
• Hodge 2001 paid a flat rate for participation
![Page 30: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/30.jpg)
3. Within- vs. Between-?
• Enhanced statistical power, subjects are their own control– less variance
• Increases salience of treatment effects• Vulnerable to carry-over effects• Can help demonstrate the (un)intentional
nature of cognition• Requires proper analysis• Hodge 2001 used a between-subject design
![Page 31: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/31.jpg)
Hodge 2001 Experimental Design
• 3 x 1 between-subjects• Control: Hardcopy of unaudited letter
and audited financial statements• Treatment 1: Audited financial
statements on computer with hyperlinked unaudited letter
• Treatment 2: same as 1. Above, except that when the hyperlink was activated a notice popped up saying “(Un)audited.”
![Page 32: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/32.jpg)
4. Individuals or Markets?
• Theory dictates• Markets are expensive• Markets can have less observations, therefore
less power (unless repeated measures)• Research shows that biases are not driven out
by markets (may be reduced though)• Hodge 2001 studies individual judgments but
ultimately wants to say something about behavior in the market – systematic individual results aggregate to market behavior?
![Page 33: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/33.jpg)
III. Finding a topic in behavioral auditing research?
• Think broadly! Is Hodge 2001 auditing research?
• You want to get the “big potatoes!”
• Audits have a supply and a demand side
• Many financial reporting issues are of concern to auditors and investors
• Audit environment has changed greatly
![Page 34: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/34.jpg)
Finding a topic in behavioral audit research? (cont’d)
• Enron and other accounting scandals have emphasized many problems such as: – lack of independence – engagement risk – incentives to report aggressively – incentives to please the client
• Sarbanes-Oxley Act• Read, read, read…
![Page 35: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/35.jpg)
IV. Common Pitfalls• Topic fails “Yawn Test”• Predicting the null hypothesis• Accepting the null hypothesis• Low power tests• Poor motivation; small incremental contribution• Running experiments too early• Not getting adequate feedback before
submitting
![Page 36: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/36.jpg)
Top 5 Takeaways
1. Have fun!
2. Seek lots of input at each stage
3. Use a well-thought out design
4. Choose the research method that best informs the question
5. Choose an interesting topic
![Page 37: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/37.jpg)
Helpful Papers on Research
• Bonner, S., Accounting Horizons, Dec. 1999• Camerer, C, & R. Hogarth. JRU, 1999.• Kinney, W., The Accounting Review, Apr. 1986.• Libby, R. & M. Lipe, JAR, 30, 1992. • Libby, R., Bloomfield, R., & M. Nelson. AOS,
Nov., 2002. • Maines, L., BRIA, Supplement 1994.• Peecher, M. & I. Solomon, IJA, 2001.
![Page 38: Producing Good Research: Behavioral and Otherwise American Accounting Association Auditing Doctoral Consortium Clearwater, Florida. January 14, 2004 Jane](https://reader036.vdocuments.us/reader036/viewer/2022062618/5514c2af550346b0338b497f/html5/thumbnails/38.jpg)
Q & A?