Statistics and Research Methodology II

Ashoka University

Spring 2018

Instructor: Assistant Professor Kai Qin Chan |

Co-instructor: Aakanksha Mehta | Peer tutor: Dhairyya Singh

Course description

How would you go about to find out whether women talk more than men, whether Indians are more interdependent than Americans, or whether nutrition improves IQ? Quantitative data is merely a bunch of structured numbers; they cannot be interpreted meaningfully until one understands and appreciates how the data came about. In this course, we focus on methodological issues in sampling, designing experiments, correlation studies, and ethics. This course is intended for students who already know the basics of statistical analyses. The aim is to develop an eye for sound methodology, so that they can better evaluate other scientist’s research, as well as their own.

Prerequisites

Pass in Statistics and Research Methodology I, or equivalent modules in math, economics, or political science.

Course goals

By the end of this course, you will be able to:

Compose a psychometrically valid questionnaire; Formulate hypotheses, test them, analyze the results, interpret them, and communicate

your results; Become a literate consumer of scientific information; Appreciate why meaningful interpretation of data relies on both statistical and

methodological understanding.

Required

Zechmeister, J. S., Zechmeister, E. B., & Shaughnessy J. J. (2009). Essentials of research methods in psychology. New Delhi: Tata McGrawhill. [Price: ~ Rs 650]

RStudio and R JASP (download the Dec 2017 version)

Optional book Teetor, P. (2011). R Cookbook. New Delhi: O’Reilly. [Price: ~ Rs 420]

1

Evaluation for students taking this as a 4-credit course:1

[Individual] Quiz on syllabus: 5% [Group] Questionnaire Development Report: 15% [Individual] Attendance: 8% (0.5% for each class)2

[Individual] Final Exam: 36% [Group] Final Project Report: 36%

Evaluation for students taking this as an audited course:

Students need to attend at least 80% of the classes, pass the quiz (50%) and the exam (50%) to get the AU on their transcript. Audit students do not need to do the two reports.

Office hours

Not available on Mon/Wed 10 am – 1 pm; Tues/Thurs 8 am to 1 pm. My office is at Office 608 (old block). I prefer you to just drop by rather than email for appointment, unless you really need to see me at a specific date/time. If you are afraid of making a wasted trip, you can also call my office to see if I am in: 130-230-0394.

Seminar schedule

Week Date 1 Session 1 Date 2 Session 21 23/1 Introduction (Chap 1-2, & p.

116-119)

Psychometrics: Dorst, E. A. (2011). Validity and reliability in social science research. Education Research and Perspectives, 38, 105-123.

In addition, we will take some time in class to form up your project groups for Questionnaire

25/1 Ethical issues in psychology research (Chap 3)

Remember to install R and RStudio, and also download Dec 2017 version of JASP before the next class!

1 Grades may be moderated. See subsection Explanatory Notes on Moderation on p. 9.2 Excluding Week 1 and 2, there are 20 in-class sessions. For each class that you attend, you earn 0.5%, up to a maximum of 8%. This means you can miss 4 in-class sessions without penalty. There will be no differences between “excused” and “unexcused” absences. This implies that if you willfully decide to be absent 4 times early on in the semester, you will be penalized from the 5th time onwards even if you are genuinely ill, need to participate in sports, have emergencies, have looming deadlines, etc. Bereavement and long-term medical or psychological conditions will be considered separately. Inform the instructor in Week 1 if you need special considerations.

2

Development Report.

Complete an online questionnaire, which will be used in the demo session of Week 2.1: https://docs.google.com/forms/d/e/1FAIpQLSdgpqpqNNUFLVImmM7jUpT-qHdM9FNWgelVEz-CsXWQ6W84JA/viewform?usp=sf_link

2 30/1 Demonstration session [bring laptop]

Install R and RStudio, and also download Dec 2017 version of JASP before class!

Analyzing the psychometric properties (validity and reliability) of an established questionnaire using R and/or JASP.

Form up groups for both your Questionnaire Development Project and Final Project.

Advice: You should start working on refining your chosen construct and writing questions for the Questionnaire Development Report.

[Attendance counted: 1]

1/2 Discuss: Kramer, A. D. I., Guillory, J. E., & Hancock, J. T. (2014). Experimental evidence of massive-scale emotional contagion through social networks. Proceedings of the National Academy of Sciences, 111, 8788-8790.

Discuss: Loftus, E. F., & Pickrell, J. E. (1995). The formation of false memories. Psychiatric Annals, 25, 720-725.

Discuss: Martens, A., Kosloff, S., Greenberg, J., Landau, M., & Schmader, T. (2007). Killing begets killing: Evidence from a bug-killing paradigm that initial killing fuels subsequent killing. Personality and Social Psychology Bulletin, 33, 1251-1264.

Advice 1: Focus on the methods, not the introduction, statistics, or discussion.

Advice 2: You should be collecting data for your QDR.

[Attendance counted: 2]3 6/2 Quiz on syllabus (5 mins)

Introduction to experimental research

(Chap 6)

Advice: You should be analyzing data and preparing for the group presentation in this week.

8/2 Control problems in research

(Chap 6)

[Attendance counted: 4]

Reminder: Send list of priorities of your project by 15 Feb!

3

[Attendance counted: 3]4 13/2 Stress-free student presentation

on Questionnaire Development (ungraded)

Advice: Use feedback from your classmates to improve your report.

Due to time limits and the large class size, extra session for 10:10 am class will be arranged.

[Attendance counted: 5]

15/2 Assignment: Analyze a research design I

Discuss: Baker, L., & Lombardi, B. R. (1985). Students' lecture notes and their relation to test performance. Teaching of Psychology, 12(1), 28-32.

Discuss: Chenneville, T., & Jordan, C. (2008). Impact of attendance policies on course attendance among college students. Journal of the Scholarship of Teaching and Learning, 8, 29-35.

Discuss: Mehl, M. R., Vazire, S., Ramírez-Esparza, N., Slatcher, R. B., & Pennebaker, J. W. (2007). Are women really more talkative than men? Science, 317(5834), 82-82.

Use the Template for analyzing research designs.docx to help you deconstruct the methods.

Advice: Treat your readings seriously. The skills you acquire are precisely the ones tested in your exam.

Advice: You should be finishing up your Questionnaire Development Report.

[Attendance counted: 6]5 20/2 Single-factor designs

(Chap 6)

[Attendance counted: 7]

22/2 Assignment: Analyze a research design II

Discuss: Ackerman, R., & Lauterman, T. (2012). Taking reading comprehension exams on screen or on paper? A metacognitive analysis of learning texts under time pressure. Computers in Human Behavior, 28, 1816–1828.

Discuss: Wookey, M. L., Graves, N. A., & Butler, J. C. (2009). Effects of

4

a sexy appearance on perceived competence of women. Journal of Social Psychology, 149, 116-118.

Chen, P., Chavez, O., Ong, D. C., Gunderson, B. (2017). Strategic resource use for learning: A self-administered intervention that guides self-reflection on effective resource use enhances academic performance. Psychological Science, 28, 774-785.

Use the Template for analyzing research designs.docx to help you deconstruct the methods.

Advice: Treat your readings seriously. The skills you acquire are precisely the ones tested in your exam. Focus on the methods section.

[Attendance counted: 8]6 27/2 Factorial design I: Fully

independent, fully dependent (Chap 7)

[Attendance counted: 9]

Supplementary readingFully between-subjects: https://www.discoveringstatistics.com/repository/twoway.pdf

Fully within-subjects:https://www.discoveringstatistics.com/repository/repeatedmeasures.pdf

1/3 Power analysis[bring laptop]

Install G*Powerhttp://www.gpower.hhu.de/en.html

Discuss: Sherman, G. D., Haidt, J., & Coan, J. A. (2009). Viewing cute images increases behavioral carefulness. Emotion, 9, 282-286.

Use the Template for analyzing research designs.docx to help you deconstruct the methods.

Advice: Treat your readings seriously. The skills you acquire are precisely the ones tested in your exam.

Data analysis:[bring laptop]

In this tutorial, we will also analyze a dependent groups design with more than two levels

Dataset & R codes:dataset_earthquake.csv & dataset_earthquake.r

5

[Attendance counted: 10]

7 6/3 Factorial design II: Factorial ANOVA – Between- and within-subjects designs

Data analysis: In this tutorial, we will analyze fully between-subjects and fully dependent-subjects factorial designs. [bring

laptop]

Dataset & R codes:dataset_haloeffect.csv & dataset_haloeffect.rdataset_memory.csv &dataset_memory.rdataset_emotionVideo.csv (if time permits)

[Attendance counted: 11]

8/3 Factorial design III: Factorial ANOVA – Mixed designs (Chap 7)

[bring laptop]

Data analysis: In this tutorial, we will analyze mixed factorial designs, as well as clear up any remain issues about factorial ANOVAs.

Dataset & R codes:dataset_stroke.csv & dataset_stroke.rdataset_bassin.csv & dataset_bassin.r

[Attendance counted: 12]

Supplementary readingMixed ANOVA: http://www.stat.cmu.edu/~hseltman/309/Book/chapter11.pdf[Attendance counted: 13]

8 Mid-semester break (10/3 – 18/3)

9 20/3 Correlational research, surveys (Chap 5), observational (Chap 4)

22/3 Data analysis: We will analyze two more datasets, but this time, the focus is on learning how to write a results section that is clear, complete, and unambiguous.[bring

laptop]

dataset_potato.csvdataset_pizza.csv

Advice: This activity prepares you for Q3 of your exam.

[Attendance counted: 14]10 27/3 Data analysis: We will analyze

more datasets. The focus is on consolidating your knowledge about factorial design analyses. [bring laptop]

Datasets will be uploaded later; R codes won’t be.

29/3 Stress-free Research Proposal Presentation (ungraded)

Advice: Use feedback from your classmates to improve your research.

Due to time limits and the large class size, extra session for 10:10 am class will be arranged.

6

11 3/4 Quasi-experimental designs (Chap 9), small N designs (Chap 8)

[Attendance counted: 15]

5/4 Activity A (45 mins)SRM II exam review of key concepts

Activity B (45 mins)I will hand out a sample exam paper. We will try a few questions and discuss answers in class

[Attendance counted: 16]

12 10/4 Activity AYou will learn how to write a methods section that is clear, complete, and unambiguous.

We will watch a video of an experiment on frogs and “worms”. You will describe what was done in the experiment.

Advice: This activity prepares you for Q1 of your exam and Methods section of your Final Project Report.

Activity BMock Exam: You will have 25 mins to write out the design of an experiment to answer a research question. We will then spend 20 mins discussing your answers. The research question will mimic Q1 of your exam.

[Attendance counted: 17]

12/4 Exam: All content up to and including Week 11.

[Attendance counted: 18]

13 17/4 Data analysis & presentation preparation (no classes)

Advice: Read Appendix B on “Communication in Psychology”

19/4 Make-up weekFinale session: What is an average? (45 mins)

Release of Final Exam results +Open consultation (45 mins)

Read: Chap 10, and Molenaar & Campbell (2009). The new person-specific paradigm in psychology. Current Directions in Psychological Science, 18, 112-117.

14 24/4 Stress-free Final Project 26/4 Stress-free Final Project

7

Presentation: Groups 1, 2, 3

Due to time limits and the large class size, extra session for 10:10 am class will be arranged.

[Attendance counted: 19]

Presentation: Group 4, 5

Open consultation

[Attendance counted: 20]

15Make-

up week

1/5 [Make-up for SRM I]We will spend about 30 mins on power analysis since this was not covered in SRM I.

Install G*Power before coming to class and bring your laptop.

If you want, we can also practice analyzing more datasets.

3/5 No class, unless any class above is cancelled.

Other issues

Academic dishonestyDo not plagiarize or cheat in exams and assignments. I do not grade plagiarized work, or give a chance for resubmission. Ashoka University expects you to fulfill your academic obligations through honest and independent effort. It is your responsibility to familiarize yourself with Ashoka’s policy about academic dishonesty in your Student Handbook. And as a final warning: Merely citing sources without paraphrasing is still plagiarism. Consult the CWC if in doubt what plagiarism is. In the last 3 semesters, I have caught at least 5 students plagiarizing (one was suspended). Do not become the next student.

Contingency plansIn the past few years, classes have been disrupted because of smog and social unrests in the region. If classes are cancelled and make-up sessions are impossible to schedule, the remaining attendance percentage will be distributed equally.

How to do well in this course?This is the course where everything you learnt in SRM I falls into place, where you start to appreciate why statistics and research methodology are both important in psychology. The expected hours you need to put in is 10 hours per week. This 10 hours is divided approximately as such, but this can vary week to week:

3 hrs – in-class 2 hrs – revising Session 1 1 hr – preparing for Session 2 4 hrs – working on your projects

8

You decide what you want to get out from the course, and work towards it. There are no shortcuts in this course.

The math behind the statisticsIn SRM I and II, we covered the math behind the statistics at a surface level. For those curious about the math, you can read Cardinal (2004) here: https://egret.psychol.cam.ac.uk/psychology/graduate/Guide_to_ANOVA.pdf. This is what you will learn more in graduate school in psychology. If you were to do the same content in a statistics department, the math behind it is even more complex, often involving matrix algebra and triple summations.

How much of math do you really need to know? Not much – if you can understand conceptually what variance partitioning is, that’s good enough for this level; but if you struggle to understand why the formula for standard deviation makes sense, then I worry for you. My personal take is that you only need to understand the super advanced math if you want to specialize in quantitative psychology. Check out some of the world’s best statistical psychologists such as David McKinnon, Andrew Hayes, Kristopher Preacher, Leona Aiken, Steven West, Mike Cheung, etc. (By the way, quantitative psychologists are among the highest paid and most employable psychologists.)

Am I shortchanged if I don’t know the math? Most carpenters don’t know ionic and covalent bonds. Are they less competent as carpenters? Probably not. (You can turn the question around: Would a scientist fully competent in ionic and covalent binds be a good carpenter?)

9

Detailed components5-

min

Qui

z

The first step to doing well in this course is to take charge of your own learning; start by knowing what is expected in this course. Knowing the syllabus of any course you take is important because it helps keep yourself on track; you know what is expected of you, when, and why. Unfortunately, many students do not read syllabus.

This is an open-paper quiz. There will be 10 factual questions, and answers can be found in the syllabus.

Dis

cuss

ion

artic

les

The discussion articles equip you with a variety of skills to answer Q2 and Q3 of your exam. But if you simply read passively, you won’t get much out of it. What is useful is to generate questions and try answer to answer them (or get your fellow classmates to answer them!). Better still: Generate tricky questions that your classmates find it difficult to answer.

Essentially, this is also how I set questions in Q2 and Q3: I pick an article, and generate methodological and statistical questions. This is exactly the same way that will help you if you are conscientious in reading the discussion articles.

Exa

m

There will be three questions.a) Q1 will ask you to devise an experiment.b) Q2 and Q3 will ask you to answer statistical and methodological questions. You

will not need to run any analysis. The mock exam will give you the style of my questions.

c) Q3 will also ask you to write out the results of a statistical output.

Advice: Treat (c) seriously. It looks like a no-brainer, but it is challenging.

I emphasize answers that are clear, complete, and unambiguous.

Exam is open-book; internet connection is strictly prohibited.

10

Que

stio

nnai

re D

evel

opm

ent

Questionnaire DevelopmentQuestionnaires are often used in social sciences. How do you compose a set of

questionnaires that is valid and reliable – and how would you know if they are? We will learn about validity and reliability in class. Then you will work in groups to devise a set of questionnaires and administer these questionnaires to your fellow classmates and friends. Aim to collect as many responses as you can. After that, you will analyze the validity and reliability of your questionnaire, present them in class (15 mins – ungraded), and write a short group report (graded).

ConstructsChoose one of the following constructs. First, attempt to define the construct.

Think of whether there is a hierarchical structure involved (a hierarchy may not be necessary). Drawing diagrams usually help at this stage, especially when there is a hierarchy. Word the items, along with their numerical and verbal anchors. Aim for no more than 10 items in total. Have your classmates complete them (and you should also complete your classmates’ questionnaires). Collect additional data from other friends, if necessary. Students who are curious about hierarchical structures may consult Google or YouTube on how to do a factor analysis.

Philosophical Political correctness

Slacker Entrepreneurial spirit

Perfectionism Sense of justice Grade anxiety Hypocrites

Sense of insecurity Distrust in science Environmentalism Social loafing

Adaptability Ingrate Back-stabber Foodie

Group sizes8:30 am: 5 groups max10:10 am: 10 groups max

Questionnaire Development Report (max. 1000 words)Your report needs to explain your constructs. Draw diagrams if you have to.

After collecting and analyzing the data, discuss the psychometric properties of your questionnaire. Attach your questionnaire items as an Annex, along with the appropriate correlation matrix.

The aim is to produce a questionnaire that has good psychometric properties. However, your grade is not determined on the outcome (i.e., the psychometric properties), but rather the process. In other words, it is possible to get excellent grades even if the psychometric properties are suboptimal.

Deadline - Questionnaire Development Report19 Feb, 1800 hrs.

11

Fina

l Res

earc

h Pr

ojec

t

Final Research ProjectStatistical methods, particularly as they are applied in research, are best learned

through practice. In SRM I, you did secondary data analysis – analyzing data collected by others. In SRM II, you will experience the whole research process from start to end. The main intention is to practice formulating sound research questions, testing your hypothesis, and communicating what you found via a presentations (ungraded) and a report (graded).

Group Presentation – Research Proposal (ungraded)Scientists often gather feedback from peers before launching an investigation.

As budding scientists, you will do the same. Your presentation will last 15 mins – you will present for 10 minutes, allocating an additional 5 mins to gather feedback from your classmates. You may decide on the number of speakers presenting. You will need to present the following: Intro; Hypothesis; Design; Expected results (please graph it); Analytic strategy.

Advice: Keep your presentation to 5 slides (excluding title slide)

Group Presentation – Research Results (ungraded)After concluding a research, scientists often want to communicate their findings

to their peers. As budding scientists, you will do the same. Your presentation can last up to 25 mins. Allocate some time to solicit feedback from your classmates. You may decide on the number of speakers presenting. You will need to present the following: Intro; Hypothesis; Design; Results (please graph it); Limitations.

Advice: Treat your presentation seriously. In order to solicit effective feedback, you need to communicate clearly your content.

Group sizes8:30 am: 5 groups max10:10 am: 10 groups max

Deadline – Research Report1. Each group submits ONE group report.2. To prevent social loafing, each individual of the group submits a confidential

assessment of each member’s contribution, including the individual him/herself. The idea is not to “outcompete” others but to ensure everyone doe his/her fair share of work. Thus, your grade may be moderated by others’ assessment of your contribution. More details will follow as the semester goes on, but briefly, the assessment includes items such as: Keeping abreast of group progress Sharing ideas Completing tasks on time Attending meetings Demonstrating respect for others Contributing to group discussions

7 May, 1800 hrs

12

Explanatory Notes on Moderation

There are two families of grading scheme: absolute grading scheme (AGS) and relative grading scheme (RGS). Simply put, absolute grading defines letter grades based on predefined cut-offs scores, whereas relative grading defines letter grades based on a student’s rank-ordered scores vis-à-vis other students. In reality, each grading scheme exist is a family of grading techniques. For example, there are at least three ways to implement absolute grading3, and many more ways to implement relative grading.

I use the relative grading scheme, specifically relative cluster grading, because it fits my education philosophy and understanding of statistics. This means I rank order scores and look for “precipitous” drops. These drops form my cutoff scores. Technically A and A- signal categorical differences, but under an absolute grading system, this categorical system is artificially created — a difference of 0.1 point can mean the difference between A and A-. With a relative cluster grading, the categorical differences between letter grades is naturally reflected in the data as the “precipitous drops”.

Because relative cluster grading is a data-driven technique, there is no fixed rule how large a cluster can be, or how many grade bins there must be. Occasionally, I might skip a grade bin if the deviation between two adjacent clusters is too large. How you measure “large” is debatable. One can go for “≥ x point difference = large” or one can calculate average range per cluster and use that range to define how large is large. For example, if the average range is 3.5, and at the border of two clusters you have scores that differ by 8, then you could drop 2 grades down. My personal definition is that a cluster must have adjacent scores ≥ 1.0. Here is a real example from SRM I (2015):

Raw score RGS (cluster) AGS57.98 D+ D+58.26 D+ D+58.36 D+ D+62.25 C C-63.12 C+ C-65.44 B- C+69.73 B C+72.26 B B-72.56 B+ B-73.10 B+ B-73.65 B B-73.70 B+ B-74.18 B+ B-77.02 A- B77.63 A- B80.53 A- B+84.98 A B+85.04 A A-

3 Walvoord, B. E. (1998). Effective grading: A tool for learning and assessment in college. Jossey-Bass.

13

87.57 A A-95.13 A A

Here are some frequently asked questions about RGS:

Q: What are some advantages of relative cluster grading?

A: RGS offers some protection such that regardless of the course’s difficulty, there will be A’s. Also, RGS fosters cooperative learning: If all scores fall into a tight cluster, all will get the same score. TEAMwork: Together Everyone Achieves More.

Q: Are there situations where AGS is better over RGS suited?

A: Yes, when there are clearly right and wrong answers (e.g., MCQs, fill-in-the-blanks).

Q: Doesn’t relative grading system induce cut-throat competition?

A: This is true only if relative grading is inflexible e.g., there must be 10% A’s, 30% B’s, 30% C’s, etc. This is sometimes known as “curving”. I do not use curving.

Q: Isn’t it impossible for everyone to get an A under relative grading?

A: If the distribution follows a uniform distribution, everyone gets an A. If the distribution follows a near uniform distribution, everyone likely gets an A. The questions you need to ask are: What is the distribution? How will anyone know the distribution before seeing the data?

Q: Does absolute grading creates certainty whereas relative grading creates uncertainties?

A: Absolute grading does create certainty if and only if the predefined score is attainable (e.g., MCQs); relative grading creates uncertainties; that is true because you are unlikely to know how well others are doing. How do you manage uncertainty? Do your best, don’t compare.

Q: Isn’t relative grading is statistically illogical in small samples?

A: It is true that when sample size, k ∞, scores x ~ N (𝜇 = 0, 𝜎 = 1). But RGS ≠ grading using normal distribution. RGS is a family of techniques, some of which (e.g., cluster grading) do not rely on assumptions of normality.

14

Q: Isn’t RGS’ cut-off points are arbitrary?

A: This argument applies to both RGS and AGS.

Q: Grades are continuous variables, but RGS creates artificial categorical outcomes?

A: This argument applies to both RGS and AGS. This is the result of the letter-grade system, not of RGS or AGS. Any downward change in the levels of measurement (continuous categorical) loses sensitivity. (This is why you should never dichotomize continuous variables in an ANOVA but should instead use regression techniques to prevent Type I or Type II errors.)

Q: Is AGS bad, whereas RGS is superior?

A: Each has its advantages and disadvantages. Neither is superior over the other. It depends on the subject, instructor’s education philosophy, instructor’s understanding of statistical distributions.

15