The Integrity of Data Management
Responsible Conduct of Research Workshop
Howard University
November 2, 2010
Peggy Carr, Ph.D.
Associate Commissioner
National Center for Education Statistics
1
Overview of Statistical and Research Standards
2
OMB Guidelines for Federal Statistical Agencies
16 Federal Statistical Agencies adhere to Office of Management and Budget Guidelines to maintain high standards of performance Development of concepts and methods
Planning and design of surveys and other means of collecting data
Collection of data
Processing and editing of data
OMB, Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, February 22, 2002, (67 FR 8752-8460)
3
Best Practices for the Conduct of Research
4
Best Practices for the Conduct of Research
1. Conceptualize Research Problem
2. Operationalize Study Design
3. Engage in Peer Review and Networking
Activities
4. Publish and Publish Early/Often
5. Be Ethical
5
Oprah Video Clip
1. Conceptualize Research Problem
7
Literature ReviewLiterature Review
Statement of ProblemStatement of Problem
HypothesesHypotheses Research QuestionResearch Question
Definition of Terms, Concepts, and
Constructs
Definition of Terms, Concepts, and
Constructs
Conceptual
Operational
2a. Operationalize Study Design
The Concept of Study Design
Hypothesis 1Hypothesis 1
Explained VarianceExplained Variance
Unexplained VarianceUnexplained Variance
Sampling Error
Sampling Error
Measurement Error
Measurement Error
Validity
Reliability
2b. Operationalize Study Design
Validation of Constructs Face – Content validation across judges Criterion – Predictive and concurrent Construct – Methodological approaches to validation
9
Jealousy Envy
2c. Operationalize Study Design
Reliability of Measures
Test and Retest – stability over time
Equivalency – consistency across forms
Internal – internal consistency
10UNRELIABLE RELIABLE
2c. Operationalize Study Design
Research Design
Internal Validity – refuting rival plausible hypotheses
External Validity – generalizability of findings
Choice of Designs – experimental, quasi, survey,
correlational
11
2c. Operationalize Study Design
Quality Assurances of Data Data Checks (e.g., out of range variance, criterion reference
checks, replication)
Thoughtful and Appropriate Data Analysis The nature of your data Limitations of statistical technique
Discuss Findings and Conclusions with Colleagues Relate back to literature review Implications for future research Flaws and limitations
12
NEXT
2c. Operationalize Study Design
Trend in fourth-grade NAEP mathematics average scores
13
2c. Operationalize Study Design
14
3. Engage in Peer Review and Networking Activities
Find a specialty and stick with it
Attend professional meetings and conferences – share your research
Keep an open mind and learn from others
15
4. Publish and Publish Early/Often
Start your publication record early in peer reviewed journals As first, second, or third author
Publish and/or present often Regardless of professional focus
Annually
16
5. Be Ethical
Be committed and follow through
Adhere to Subjects’ Rights Institutional Review Board Confidentiality and Privacy Rights
17
Questions?
18