managing data responsibly to enable research interity
TRANSCRIPT
Managing data responsibly to enable research integrity
Heather Coates | Digital Scholarship & Data Management Librarianhttp://ulib.iupui.edu/digitalscholarship/datasupport/
Introduction to Research EthicsQuaid G504 (Fall 2016)
What is research integrity?
security
privacy
trust
honesty
accuracy
efficiency
objectivity
personal responsibility
ownership
stewardship
governance
Why does RDM matter?
RDM as a component of RCR
Roles & Responsibilities
Practical RDM
WHY DOES RDM MATTER?
The value of data increases with their use.
-Paul Uhlir
Source: John Gantz, IDC Corporation: The Expanding Digital Universe
0
100,000
200,000
300,000
400,000
500,000
600,000
700,000
800,000
900,000
1,000,000
2005 2006 2007 2008 2009 2010
The World of Data Around Us
Transient information
or unfilled demand for
storage
Information
Available Storage
Pe
tab
yte
s W
orld
wid
e
• Natural disaster • Facilities infrastructure failure • Storage failure • Server hardware/software failure• Application software failure• External dependencies (e.g. PKI failure)• Format obsolescence• Legal encumbrance • Human error• Malicious attack by human or automated agents• Loss of staffing competencies• Loss of institutional commitment • Loss of financial stability • Changes in user expectations and requirements
The World of Data Around Us: Data Loss
CC
im
age b
y S
hary
n M
orr
ow
on F
lickr
CC
im
age b
y m
om
bole
um
on F
lickr
Poor Data Management Affects Everyone
“MEDICARE PAYMENT ERRORS NEAR $20B” (CNN) December 2004
Miscoding and Billing Errors from Doctors and Hospitals totaled $20,000,000,000 in FY2003 (9.3% error rate) . The error rate measured claims that were paid despite being medically unnecessary, inadequately documented or improperly coded. In some instances, Medicare asked health care providers for medical records to back up their claims and got no response. The survey did not document instances of alleged fraud. This error rate actually was an improvement over the previous fiscal year (9.8% error rate).
“AUDIT: JUSTICE STATS ON ANTI-TERROR CASES FLAWED” (AP) February 2007
The Justice Department Inspector General found only two sets of data out of 26 concerning terrorism attacks were accurate. The Justice Department uses these statistics to argue for their budget. The Inspector General said the data “appear to be the result of decentralized and haphazard methods of collections … and do not appear to be intentional.”
“OOPS! TECH ERROR WIPES OUT Alaska Info” (AP) March 2007
A technician managed to delete the data and backup for the $38 billion Alaska oil revenue fund – money received by residents of the State. Correcting the errors cost the State an additional $220,700 (which of course was taken off the receipts to Alaska residents.)
"Good data management practice allows reliable verification of results and permits new and innovative research built on existing information. This is important if the full value of public investment in research is to be realized."
Managing and Sharing Data: Best Practices for ResearchersUK Data Archive
Benefits: Good Data Practices & Open Data
• Open data addresses social justice issues
• Open data enhances social welfare
• Open data benefits for effective governance and policy making
• Open data grows the economy
• Open data improves the integrity of the scholarly record
• Open data facilitates the education and training of new generations
• Open data enables validation or replication to support published results
• Open data accelerates the pace of discovery
• GDP increases the impact of your work by sharing your data, code & other products
• GDP improves the quality and consistency of research data you produce (save $$$)
• GDP improves the efficiency of your research (save time)
Personal Experience
“Please forgive my paranoia about protocols, standards, and data review. I'm in the latter stages of a long career with USGS (30 years, and counting), and have experienced much. Experience is the knowledge you get just after you needed it.
Several times, I've seen colleagues called to court in order to testify about conditions they have observed. Without a strong tradition of constant review and approval of basic data, they would've been in deep trouble under cross-examination. Instead, they were able to produce field notes, data approval records, and the like, to back up their testimony.
It's one thing to be questioned by a college student who is working on a project for school. It's another entirely to be grilled by an attorney under oath with the media present.”
- Nelson Williams, Scientist US Geological Survey
RDM AS A COMPONENT OF RCR
Concepts of Data Management
• Data ownership
• Data collection
• Data storage
• Data protection
• Data retention
• Data analysis
• Data sharing
• Data reporting
Steneck, 2004
The purpose of data management planning is to ensure that research data produced by a project are high quality, well organized, thoroughly documented, preserved, and accessible so that the validity of the data can be determined at any time.
ORI Guidelines for Responsible Data Management
The goal of data management is to produce self-describing data sets.
DataONE Primer on Data Management
Why is good data management so challenging?
ambiguity effect
availability heuristic
confirmation bias
experimenter’s or expectation bias
framing effect
hindsight bias
neglect of probability
optimism bias
planning fallacywell traveled road effect
ROLES & RESPONSIBILITIES
Funder progress towards openness
1985: National
Research Council
1999: Office of Mgmt & Budget, Circular A-110
revisions
2003: NIH Data
Sharing Policy
2008: NIH Public Access Policy
2011: NSF DMP Require-
ment
2012: NEH,
Office of Digital
Humanities DMP
Require-ment
2013: NSF Bio sketch change
2013: OSTP
Memo on Public
Access to Results of Federally-Funded
Research
2014: OSTP Memo on
Improving the Mgmt of & Access to Scientific
Collections
2014: OMB
Circular A-81
(Uniform Guidance)
takes effect
2015: Federal Funding agencies release plans
responding to 2013 OSTP memo
2016: Federal Funding Agency
Plans take effect
(DMP req)
Funder Policies: DMP & data sharing
• Association for Healthcare Research & Quality
• Centers for Disease Control & Prevention
• National Institutes of Health
• National Science Foundation
More agency policies at datasharing.sparcopen.org
Publisher Policies: Data availability
• DataDryad Publishers
• PLoS Journals
• Nature Publishing Group
• American Economic Review
• BioMedCentral
• JORD - Social Science Journals with a research data policy
• Data policies of Economic Journals
https://ulib.iupui.edu/digitalscholarship/datasupport/publisher_policies
Institutional Policies
• Vary greatly, with lots of gaps
• Distributed – address specific local or state requirements for specific types of data
• Often focus on institutional data rather than research data
• Do not provide practical guidance
• Do not distinguish between institutional and personal responsibilities
Roles/Responsibilities of project personnel
• On your own
– Fill in the team members responsible for key data activities
• In small groups (2-3 people), share and discuss
– What kind of training is provided for team members to complete these tasks accurately?
• Whole group discussion
– What barriers do you face in tracking roles and responsibilities?
– What barriers do you face in providing training?
Team Member Name Project Role Activity Description
Project design [+ documentation]
Determining the aims of the project, the methods used to achieve those aims, and identifying the products resulting from the project.
Translating the aims of the project into measurable research questions or hypotheses.
Instrument/measure/data collection tool design [+ documentation]
Creating tools that adapt the research questions or hypotheses into questions that can be addressed by discrete data points.
Validating tools through external review or pilot testing.Data collection [+ documentation]
Conducting surveys, interviews, experiments, and other project procedures according to the protocol in order to generate data.
Data processing [+ documentation]: entry, proofing/cleaning, preparation for analysis
Entering analog data into spreadsheet or database. Documenting procedures, date, and person responsible.
Checking data entry for accuracy and completeness. Documenting procedures, date, and person responsible.
Checking data for missing data, errors, and outliers. Documenting procedures, date, and person responsible.
Deciding what data to include/exclude. Documenting decision-making process and criteria used.Data analysis [+ documentation]:
Selecting analytical tools to be used. Documenting decision-making process and criteria used.
Conducting data characterization and screening tests, running analyses, generating results. Documenting process and files generated.
Deciding what data are relevant to the project aims and objectives. Documenting decision-making process and criteria used.
Data reporting: Creating summary tables, graphs, and other visuals to represent the data.
Writing up the project details and relevant results in the packages/format requested by the client, as specified by the deliverables agreed upon in the contract.
PRACTICAL RDM
Basic Principles: Good Research Data Practices
1. Have a plan & use it2. Follow the 3-2-1 rule of data storage3. Document4. Be consistent; when you aren’t, document the deviation5. Use common, standardized terminology6. Monitor the quality of the data as it is being created7. Report enough detail about your research so that others in your
field can reproduce it and others outside your field can evaluate it8. Be as open as possible9. Think about how your research might be useful to others
1: Functional data management plans support teams
• A tool for planning all the key activities related to data before you have a messy pile of bits on your hands
• A working document that reflects how a study is conducted
• Communication device for the team
• Documents the team members and their roles
• Customized to address the issues most relevant to your research
1: Planning…learning from Good Clinical Data Management Practices
• Begin with the end in mind OR Produce report-ready outputs
• Plan, test, revise, plan, test, revise…implement
• Include all stakeholders in the design of the protocol, data collection tools, data management plan, etc.
• Document, document, document– Specify documents required for reproducible research
– Facilitates clear communication and shared understanding throughout the project
– Specify roles and responsibilities from the beginning
2: Follow the 3-2-1 Rule
The accepted rule for backup best practices is the three-two-one rule. It can be summarized as: if you’re backing something up, you should have:
• At least three copies (in different places),
• in two different formats,
• with one of those copies off-site.
3: Document: How much?
More than you think you will need BUT less than everything
Information EntropyD
ATA
DE
TA
ILS
Time of data development
Specific details about problems with individual items or
specific dates are lost relatively rapidly
General details about datasets are lost through time
Accident or
technology
change may
make data
unusable
Retirement or career change makes access to
“mental storage” difficult or unlikely
Loss of data developer leads
to loss of remaining
information
TIME(From Michener et al 1997)
3: Document, document, document
Documentation should capture crucial details needed for post publication peer review or validation of results
• Study: research questions/aims, IRB protocol, informed consents/authorizations, etc.
• Data collection instruments or tools OR data sources
• Data collection process or workflow
• Can take many forms, but should be consistent with standards or norms of practice for your field (e.g., data dictionary, data model, codebook, readme.txt, lab notebook)
3A: Know what you have - Data Inventory
• On your own
– Fill in as much of the data inventory as you can
• In small groups (2-3 people), share and discuss
– Benefits of knowing exactly what data you have?
– How hard would it be to complete this fully and accurately?
• Whole group discussion
– How might this be helpful throughout various phases of the project?
– How might it be helpful to have an inventory for complete and active projects?
3A: Know what you have - Data Inventory Example
• Funding source
• Program or initiative
• Project title
• PI First Name
• PI Surname
• Other Researchers/Data Contacts
• Project Start Date dd-mm-yyyy
• Project End Date dd-mm-yyyy
• New datasets created?
• How many datasets created
• Data location(s)
• Dataset Type (qualitative, quantitative, mixed methods, model)
• Sharing data?– Deposit location?
– Licensing?
– Embargo?
http://www.data-archive.ac.uk/create-manage/strategies-for-centres/data-inventory
3B: Documentation Strategies
• Lab notebooks (print or electronic)
• Codebooks
• Data Dictionaries
• Procedures Manuals
• Protocols
• Readme.txt
3C: Structured documentation [metadata] is crucial for discovery, reuse, and interoperability
• Metadata describes the who, what, when, where, how, why of the data
• Metadata = documentation for machines (standardized, structured)
• Purpose is to enable evaluation, discovery, organization, management, re-use, authority/identification, and preservation
• Standards are commonly agreed upon terms and definitions in a structured format
• Good documentation builds trust in your data – provenance, data integrity, transparency, audit trail, etc.
4: Be consistent
• We’re human – recognize the challenge
• Prevention – design research instruments & processes to prevent mistakes
• Pilot everything to identify potential problem areas
• When you aren’t, document the deviation
• Train your project personnel to be consistent & monitor performance
• Do internal audits, quality checks, data screening periodically to detect inconsistencies
5: Use common, standardized terminology
• For things/concepts
– Diagnoses
– Species/cell lines
– Locations
– Variable names
– Samples & materials
• For formats, too
– Dates
– Codes
– Identifiers
6: Monitor the quality of the data
• Don’t wait until data collection is over
• Quality Assurance
• Quality Control
• Build it into the project timeline
• Make it someone’s job
• Document what you find and how it was corrected
7: Better reporting
• Report enough detail about your research so that others in your field can reproduce it and others outside your field can evaluate it
• This includes ALL aspects of the study: study design, data collection methods, sampling, population, data screening & processing, QA/QC procedures, analytical procedures, visualization procedures, etc.
I know you can’t fit this into a journal article but you can write up pieces of this as the study is conducted to support publications and reporting to the funder. Plus it makes writing those products much easier.
8: Be as open as possible (Open Science)
Open isn’t an all or nothing choice• Study registration• Open notebook science• Data sharing (raw data, processed data, data supporting published
results)• Open Data• Open Access publishing (deposit pre/post print in a repository,
choose OA journal, choose Gold OA option)
Want to learn more? Center for Open Science Why Open Research?
9: Think ahead
How might your research might be useful to others? To yourself in 5/15/50 years? Your students or trainees? Historians?
• Consider what you will forget in that time and document it
• Consider whether your data will be useful beyond the life of the project. If so, put it somewhere safe like an institutional or subject repository to share it and ensure long-term access.
Stakeholders in (Academic) RDM
• Research Administration
• Research Compliance
• University IT
• University Libraries
• University Archives
• Consortia (e.g., CIC)
• NIH CTSA Hubs/NCATS
• Research & Technology Corporation: http://iurtc.iu.edu/
Case studies: Discussion
1. http://retractionwatch.com/2015/11/05/got-the-blues-you-can-still-see-blue-after-all-paper-on-sadness-and-color-perception-retracted/
2. https://ori.hhs.gov/content/case-summary-anderson-david
Resources1. Uhlir, P. F. (2010). Information Gulags, Intellectual Straightjackets, and Memory Holes. Data Science Journal, 9, ES1-ES5.
2. DataONE Education Module: Data Management. DataONE. Retrieved December 2013. From
http://www.dataone.org/sites/all/documents/L01_DataManagement.pptx
3. Scientists are hoarding data and it’s ruining medical research: http://www.buzzfeed.com/bengoldacre/deworming-trials
4. Losing data from the National Centre for E-Social Science (NCESS) Portal:
http://datastories.jiscinvolve.org/wp/2015/08/10/losing-data-from-the-national-centre-for-e-social-science-ncess-
portal/
5. Biomedical data sharing and reuse: Attitudes and practices of clinical and scientific research staff:
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0129506
6. Over half of psychology studies fail reproducibility test: http://www.nature.com/news/over-half-of-psychology-
studies-fail-reproducibility-test-1.18248
7. Value of Open Data Sharing: https://www.fosteropenscience.eu/sites/default/files/pdf/2536.pdf
8. Michener, W. K., Brunt, J. W., Helly, J. J., Kirchner, T. B., & Stafford, S. G. (1997). Nongeospatial metadata for the
ecological sciences. Ecological Applications, 7(1), 330-342.
9. Society for Clinical Data Management. (2013). Good Clinical Data Management Practices. Washington, D.C.
10. UK Data Archive. (2015). Prepare and manage data. From http://ukdataservice.ac.uk/manage-data.