critical evaluation (web version)
DESCRIPTION
TRANSCRIPT
Critical Evaluation
of Research Information
James Bisset ([email protected] ) Academic Liaison Librarian (Research Support)
Session outline- Importance of evaluation- Forms of value - Group activity
- Evaluating Research Information
Session outline- Importance of evaluation- Forms of value - Group activity
- Evaluating Research Information
Self awareness –
what do you bring
to the table
Session outline- Importance of evaluation- Forms of value - Group activity- Evaluating Research Information
Self awareness –
what do you bring
to the table
Some practical
tips and
evaluation criteria
Part 1 Importance
of evaluation
• Much training is about directing you to the right information = searching and retrieval
• As postgraduate researchers you have to be critical and reflect on what you find.
• Be aware of your impact on your own research, and the research of others.
• What defines your evaluative criteria?
The need to evaluate information
• Resources are interconnected and they evolve
• Information resources are transformed into knowledge
• Knowledge becomes a resource
• Therefore prior knowledge shapes what we go on to create
Ecology of Resources
Role of the researcher
• In theory we can select almost any information to complete a task
• In practice we filter it by selecting resources we think most appropriate
• Motivation - affected by the learning we have already done
Other factors
• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP
Other factors
• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP
Other factors
• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP
Other factors
• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP
Other factors
• But, filtering is done for us BEFORE we get the chance to make a judgement• People• Technologies• Cost• Skills• Copyright, IP
• You need to be critical and reflect on all of the sources you find and use.
• You have a professional responsibility, as your research will impact on others.
• You are creating knowledge…… which evolves, and will shape what others create…… similarly, the information you discover will shape the knowledge you create.
• You need to be aware of the filters already impacting upon the information you use.
Part 1 Summary
Part 2 Forms of value
• Filtering process = value judgement– By researcher–Made on their behalf
• What forms of value are there and how do they work together to create information literate researcher?
Forms of Value
Objective form of value
• Scientific measures of validity or reliability
• Exists so that personal values don’t unduly influence work
• Omit this scheme of value and we risk information (and knowledge formed from it) becoming counterknowledge (Thompson, 2008).
Intersubjective form of value
• Based on the shared values of a community e.g. morals, ethics, laws, economics
• Allows for discussion of scientific method as it can’t explain everything
• Acceptance in a community• If we omit this then values are
relativist
Subjective form of value
• Decisions you make– Is this what I want, do I need this, is it
relevant?
• Privileges you as the researcher in the decision making process
• If we omit it we get groupthink (Janis, 1972) or battery cognition (Blaug, 2007)
• Importance therefore of asserting individual criticality
Cognitive biases• Subjectivity is vulnerable to bias &
hunches• Concept of cognitive bias was
developed in 1970s by Tversky and Kahneman
• Four main groups- Social - Probability/belief - Memory - Decision making
Social biases• Ascribe positive or negative traits to
self, individuals or groups• Loading values or anticipating action
based on prior experience or a bias against self, individuals or groups
• Academic impact: need to verify information and not rely on own views; important to remember when analysing human subjects
Memory biases• How you perceive past events• False memory, positive memory,
imbalanced memory• E.g. A Photo, a Suggestion, a False
Memory • Academic impact: importance of
accurate record keeping and note taking
Memory biases
Memory biases
Probability and belief
• To disregard or to pay too much attention to probability
• Academic impact: need to treat each research finding as distinct and to judge it in its own right
Decision-making biases
• Influences on your decisions by own biases or those of a group
• Academic impact: need to be objective and consider all possible routes of enquiry and treat all research findings as valid until proved otherwise e.g. Semmelweis reflex
Cognitive biases• On your table, group the
forty cards into four piles of ten• Social • Memory • Probability• Decision
Three forms of valueSubjectiv
e
Inter-subjectiveObjective
• Different concepts of ‘value’ and the need to evaluate in terms of each of these. - need to be objective and look at the measurable facts - need to be inter-subjective and apply to the wider context - need to be subjective, and assess the value based upon our own needs
• Explored some of the key bias which may subconsciously be impacting upon how your (subjective) measure of ‘value’ might effect how you filter information and your evaluative criteria.#
Part 2 Summary
Part 3 Evaluation of Research Information
Evaluating information In a literature review you need to
evaluate:• Relevance to the topic• Authority of the author, publisher etc• Objectivity• Presentation• Method of production and methodology• Currency
Evaluating information In a literature review you need to
evaluate:• Relevance to the topic• Authority of the author, publisher etc• Objectivity• Presentation• Method of production and methodology• Currency
• Read the abstract, introduction or summary.• Scan the bibliographic information which may
highlight key subject areas not specifically alluded to.
• Emphasis may not be clear until you read in full.
Relevance to the topic
• Read the abstract, introduction or summary.• Scan the bibliographic information which may
highlight key subject areas not specifically alluded to.
• Emphasis may not be clear until you read in full.
Relevance to the topic
Be aware of what is filtering your choices… - Vocabulary and broadness of interpretation. Are you under-estimating the value of a source because it doesn’t match your choice of keywords precisely?
• Are the authors acknowledged experts in the field? - frequently cited? - have you or colleagues heard of them? - do they have an h-index? - can you find any profile information where they work?
• Where is it published? - Impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige)
Authority
• Are the authors acknowledged experts in the field? - frequently cited? - have you or colleagues heard of them? - do they have an h-index? - can you find any profile information where they work?
• Where is it published? - Impact factors for a journal (not always an accurate measure of quality, but potentially one of prestige)
Authority
Be aware of what is filtering your choices y - Is the prestige of the author impacting on how you evaluate the content?
• Is the subject controversial?
• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?
Objectivity
• Is the subject controversial?
• If there are differing views on the subject area, does the author consistently fall into one ‘camp’?
Objectivity
Be aware of what is filtering your choices - Does the author demonstrate any hidden bias on the topic? - Evaluate yourself? Are you subconsciously over-valuing the resource because it confirms your own prejudices? Are you being objective?
• Is the article peer-reviewed?• Can you identify the editor/editorial
board for the publication?
Method of production
• Is the article peer-reviewed?• Can you identify the editor/editorial
board for the publication?
Method of production
Be aware of what is filtering your choices… - Be aware of editorial policy which may decide what is published.
2.23.39 – 2.25.56http://
news.bbc.co.uk/1/hi/help/3681938.stm
• Various criteria you can assess a resource by.
- a lot more ‘citation’ tools available for journal literature.
• How much time do you realistically have?
Part 3 Summary
Bibliography• Blaug, R. (2007) ‘Cognition in a hierarchy’, Contemporary Political Theory. 6: 24–44
• Goldacre, B. (2008) Bad science. London: Harper Press.
• Janis, I. (1972) Victims of groupthink: a psychological study of foreign-policy decisions and fiascoes. Boston: Houghton Mifflin.
• Kahneman, D. and Amos, A. (1972) ‘Subjective probability: a judgment of representativeness’, Cognitive Psychology. 3(3): 430-454.
• Luckin, R. (2010) Redesigning learning contexts: technology-rich, learner-centred ecologies. Abingdon: Routledge.
• Strange, D., Hayne, H. and Garry, M. (2008) ’A photo, a suggestion, a false memory’, Applied Cognitive Psychology. 22: 587–603.
• Thompson, D. (2008) Counterknowledge: How We Surrendered to Conspiracy Theories, Quack Medicine, Bogus Science and Fake History. London: Atlantic Books.
• Whitworth, A. (2009) Information Obesity. Oxford, UK: Chandos. In particular chapter 2.
• Whitworth, D (2010) “The three domains of value: Why IL practitioners must take a holistic approach” Available at: http://prezi.com/rxqnzpoooolb/the-three-domains-of-value-why-il-practitioners-must-take-a-holistic-approach/
• http://www.informationliteracy.ie/
Image Credits[Slide 5, 38] Via Flickr Creative Commons, by Martin LaBar. Original available at http://www.flickr.com/photos/32454422@N00/163107859/
[Slide 3] Via Flickr Creative Commons, by Kevin Dooley. Original available at http://www.flickr.com/photos/12836528@N00/2577006675
[Slide 10] Via Flickr Creative Commons, by shellorz. Original available at http://www.flickr.com/photos/59198719@N00/2192821345
[Slide 13] Via Flickr Creative Commons, by Richard Cocks. Original available at http://www.flickr.com/photos/richardland/3999234316/
[Slide 12] Via Flickr Creative Commons, by Photo Extremist. Original available at http://www.flickr.com/photos/thevlue/4839060646/
[Slide 11] Via Flickr Creative Commons, by What Dave Sees. Original available at http://www.flickr.com/photos/whatdavesees/2487875504/
Image Credits[Slide 14] Via Flickr Creative Commons, by vl8189. Original available at http://www.flickr.com/photos/27630470@N03/
[Slide 15] Via Flickr Creative Commons, by opensourceway. Original available at http://www.flickr.com/photos/47691521@N07/4371001458/
[Slide 17] Via Flickr Creative Commons, by otherthings. Original available at http://www.flickr.com/photos/18619970@N00/3057937540
[Slide 30-31] Photo provided by colleague
[Slide 54] ‘Vitae®, © 2010 Careers Research and Advisory Centre (CRAC) Limited‘ Available at www.vitae.ac.uk/rdf
Measuring Researche
r Developm
ent
Vitae Researcher Development Framework [see image credits]