universitat de barcelona facultat de biblioteconomia i documentació grau d’informació i...
Post on 17-Jan-2018
218 Views
Preview:
DESCRIPTION
TRANSCRIPT
UNIVERSITAT DE BARCELONAFacultat de Biblioteconomia i Documentació
Grau d’Informació i DocumentacióResearch Methods
Non-experimental designs
Professor: Ángel Borrego
Non-experimental designs
Longitudinal studies
Content analysis
Delphi studies
Bibliometrics
Qualitative research
Longitudinal studies
Research on a sample on more than one occasion over a period of time
Types of longitudinal studies:• Trend• Cohort• Panel
Longitudinal trend design
• Deals with changes in a population.
• The whole population (or a representative sample) is observed.
Time 1 Time 2 Time 3 Time n
Longitudinal trend design
Longitudinal cohort design
A cohort is a group of people who share a characteristic, usually age.
Time 1 Time 2 Time 3 Time n
Cohort is observed
Cohort is observed
Cohort is observed
Cohort is observed
Longitudinal panel design
The same set of subjects is always observed.
Time 1 Time 2 Time 3 Time n
Longitudinal panel design
http://www.kantarmedia1.es/sections/product/panel-audiencia
Non-experimental designs
Delphi studies
Content analysis
Bibliometrics
Log analysis
Qualitative research
Content analysis
Content analysis is the quantitative study of the features of recorded information.
It consists in classifying textual (or visual) material reducing it to more relevant and manageable categories.
That reduction leads to the calculation of descriptive statistics that summarize or characterize datasets.
Content analysis is essentially a coding operation. Coding is the process of transforming raw data into a standardized form.
Examples of content analysis in LIS
• Analysis of job advertisements to describe the library job market.
• Analysis of the use of Twitter by libraries to understand how to use social media successfully.
• Analysis of reference questions posed by library users in order to improve the reference service.
Steps in content analysis1) Formulate your research problem: define questions or research
aims.
2) Identify the contents to analyse (population or sample) and a chronological framework.
3) Define the variables precisely: use exhaustive and mutually exclusive categories. Create a book of codes and an analysis grid.
4) Codify the contents.
5) Apply a test of reliability: guaranties that conclusions are independent of the coder. At least 10% of the sample should be codified independently by two coders to reach an agreement of at least 80%.
Library job advertisements
• How many library positions are being offered in academic libraries?
• What qualifications are needed to entry the job market?
• What language and computer skills are required?
• What salaries are being offered?
• What new positions are emerging?
Library job advertisements
Use of Twitter by public libraries
• What do public libraries tweet about? What types of tweets are posted by academic libraries?
• What makes a tweet useful (measured by the number of retweets and favorites received)?
Use of Twitter by public libraries
Use of Twitter by public libraries
Reference questions
• What subjects do users ask about (computers, health, travel, etc.)?
• Do users ask for sources or for factual data?
• How long does it take to respond?
• How many questions are rejected? Why are they rejected? (unclear, not appropriate, erroneous email address, etc.)?
• How can we improve the service?
Test of reliability
Example of a reference question:
Hello, I am searching for information about the León Cathedral: architecture, history, opening hours...
How would you classify this question?
Art Travel
Intercoder reliability
Two independent analysts have classified ten reference questions. As you can see there are agreements and disagreements between them. Analyst A Analyst BQuestion 1 History HistoryQuestion 2 Travel TravelQuestion 3 Education LanguagesQuestion 4 History HistoryQuestion 5 Science ScienceQuestion 6 Languages LanguagesQuestion 7 Employment EmploymentQuestion 8 Law LawQuestion 9 Philosophy ReligionQuestion 10 Languages Languages
Intercoder reliability
We can calculate the degree of agreement (DA) between both analysists using the following formula:
where A is the number of agreements and n the number of questions.
Non-experimental designs
Longitudinal studies
Content analysis
Delphi studies
Bibliometrics
Qualitative research
Delphi studies• It aims to obtain consensus of opinion of a group of experts by a
series of questionnaires.
• Supports planning, allocation of resources and decision making.
Development Delphi study (I)
• Selection of experts.
• First questionnaire: open-ended questions asking for a list of opinions, predictions, recommended activities, etc.
Example: evaluation of institutional repositories
• The aim of the research is to develop a set of evaluation indicators for institutional repositories in academic libraries.
• Experts: 25 directors of academic libraries.
• First questionnaire: open-ended question asking for a list of indicators to evaluate these repositories.
• Researchers identify and group together items from the experts’ replies.
Evaluation of institutional repositories: replies to the first questionnaire
Number of documents uploaded Researchers with uploaded documents
Documents published in the last 3 years Documents downloaded
Link from library homepage to repository Number of users
Metadata completeness System failures per month
Integrated search of OPAC and repository Number of users’ training sessions
Document types (articles, dissertations…) User satisfaction
Cost per document uploaded Thematic diversity of the collection
Development Delphi study (II)
• Second questionnaire: a copy of the collective list is sent to each expert. They are asked to rate or evaluate each item according to its importance.
Evaluation of institutional repositories: second questionnaire
Less Moreimportant important
Number of documents uploaded 1 2 3 4 5
Documents published in the last 3 years 1 2 3 4 5
Link from library homepage to repository 1 2 3 4 5
Metadata completeness 1 2 3 4 5
Integrated search of OPAC and repository 1 2 3 4 5
Document types (articles, dissertations…) 1 2 3 4 5
Cost per document uploaded 1 2 3 4 5
Researchers with uploaded documents 1 2 3 4 5
Documents downloaded 1 2 3 4 5
Number of users 1 2 3 4 5
System failures per month 1 2 3 4 5
Number of users’ training sessions 1 2 3 4 5
User satisfaction 1 2 3 4 5
Thematic diversity of the collection 1 2 3 4 5
Evaluation of institutional repositories: results to second questionnaire
AverageNumber of documents uploaded 4,8Thematic diversity of the collection 4,6Cost per document uploaded 4,5Researchers with uploaded documents 4,5User satisfaction 4,0Metadata completeness 3,8Integrated search of OPAC and repository 3,6Document types (articles, dissertations…) 3,5Link from library homepage to repository 3,5Documents downloaded 3,3Number of users 3,2System failures per month 3,2Number of users’ training sessions 3,0Documents published in the last 3 years 3,0
Development Delphi study (III)
• Third questionnaire: includes the list and the average ratings. The experts are asked to either revise their opinions or discuss their reasons for not coming to consensus with the group. If they disagree with the general opinion they provide explanations for their ratings.
Evaluation of institutional repositories: third questionnaire
“I don’t agree with the high rating for the “thematic diversity of the collection”. Our University specializes in engineering, so most of the documents in our repository fall in this discipline. We won’t have any documents in Social Sciences or Humanities. It doesn’t make much sense to measure the thematic diversity of the documents in our institution.”
Expert #7
“Just measuring the number of researchers with uploaded documents is not informative. This indicator benefits big institutions with a large number of researchers. We would have to measure the percentage of researchers with uploaded documents”.
Expert #12
“User satisfaction is interesting… but how are we going to measure it? We already do a lot of surveys among our users and the response rate is quite low since users are tired of replying to questionnaires. I think conducting another survey is unwise.”
Expert #17
Evaluation of institutional repositories: results
Category Item Performance indicator
ContentCurrency No. documents published in last 3 years
Metadata Metadata completeness
ManagementBudgeting Cost per document
Staffing IR librarians (% of total)
System and networkIntegration (OPAC) Integrated search of OPAC and IR
IR homepage Link from library homepage to IR
Use and UsersUse rate No. of downloaded documents
Support for users No. of IR training sessions
Advantages of Delphi studies
• Anonymity of participants.
• Heterogeneity of participants.
• Controlled feedback: it allows interaction but reducing discord among panel members.
• More individuals can participate than those that can effectively interact in a face-to-face exchange.
Limitations of Delphi studies
• Judgments are those of a select group of people and may not be representative.
• Tendency to eliminate extreme positions and force a middle-of-the-road consensus.
• Time consuming, requires participant commitment (about 30 to 45 days to complete the entire process).
Non-experimental designs
Delphi studies
Content analysis
Bibliometrics
Log analysis
Qualitative research
Bibliometrics
Bibliometrics is the application of mathematical methods to the study of information products (books, journals, dissertations, etc.).
Typical bibliometric variables include: authorship; country of publication; language; journal titles; age; etc.
BibliometricsA common approach in bibliometrics is citation analysis, which involves examining the reference lists of documents.
In academic libraries, it is usual to analyse faculty publications or students’ dissertations to determine types and number of documents cited, languages, age, changes in citation patters over time, etc.
Applications of bibliometric studies
• Identification of core literatures: enables a library or a database producer to identify the most important journals that need to be included in a collection.
•Estimation of completeness of a library collection.
•Decisions regarding retention of items in a collection: it is possible to use an obsolescence study to predict when an item will no longer be likely to attract a meaningful level of use.
•Research assessment: evaluate the performance of authors, institutions, journals, etc.
Non-experimental designs
Delphi studies
Content analysis
Bibliometrics
Log analysis
Qualitative research
Qualitative research
Qualitative research emphasizes the importance of people’s views or behaviours, it seeks perceptions and opinions.
Qualitative research involves a small number of participants who are addressed open ended questions, without preconceived categories of responses.
Data analysis consists in repeatedly reviewing data to develop patterns and reconsider them. Data are organized into general themes.
Qualitative results often are not generalizable. This differs from quantitative research results, where statistical processing of data creates generalizable projections from the sample to the population it represents.
Bronisław Malinowski (1884–1942) was one of the most important 20th-century anthropologists and ethnographers.
Quantitative vs Qualitative research
top related