research, evaluation, and assessment · 2020. 2. 28. · validity, reliability, practicality, ......

Research, Evaluation, and Assessment

Chapter Eleven

©2017 Cengage Learning2

Learning Objectives (slide 1 of 2)• LO 1: Consider the importance of research, some of the major types of

quantitative and qualitative approaches to research, how to examine and discuss one’s research results, and how the human service professional can use research

• LO 2: Examine formative and summative types of and evaluation, define needs assessment and learn how it’s different from evaluation, and learn how the human service professional will use evaluation techniques

• LO 3: Define assessment, including the clinical interview, informal assessment, personality testing, and ability testing; learn how the different types of assessment are applied; learn how basic statistics are applied to test interpretation; understand test worthiness including validity, reliability, practicality, and cross-cultural fairness; understand how the human service professional uses assessment


Learning Objectives (slide 2 of 2)• LO 4: Understand the concept of informed consent in research and

assessment, the importance of conducting research ethically, and the significance of proper interpretation of test data when understanding assessment results

• LO 5: Understand how research and assessment is always evolving and can help lead direction of new ideas and procedures


Research (slide 1 of 9)

Conducting Research

Analysis and recording of observations

Answering questions about the world

Confirming hunches

Statistical analyses

Coming to conclusions

First the researcher needs a hypothesis or a research question


Research (slide 2 of 9) The Hypothesis, the Research Question, and the Literature Review

What is the difference between a hypothesis and a research question? Hypotheses: Looks at differences between groups or relationships

among groups Research question: Relationships among groups or describes events

or conditions Before you develop a research question or hypothesis, you must review

what has already been written in the area of exploration. Review professional literature and articles

Use electronic databases (ERIC, PsycInfo, EBSCO, etc.) Find peer-reviewed articles if possible

Identify variables (“characteristic, attribute, or trait that can be measured”)

Use variables and measurements to help formulate a hypothesis or a research question


Research (slide 3 of 9) Defining Quantitative and Qualitative Research Designs

Quantitative Research Looking for an objective reality Use of scientific methods Find variables and measure them

Qualitative Research Multiple ways of viewing knowledge Immersion by the researcher into the situation gives possible

explanations Broad analysis of a problem in context

Distinctions Between Quantitative and Qualitative Research See Table 11.1


Research (slide 4 of 9) Quantitative Research

True Experimental Research Independent and dependent variables Random assignment Can infer causal relationship (one thing causes another) See Reflection Exercise 11.1

Causal-Comparative (Ex Post Facto) Research Examination of intact groups (no random assignment) Cannot infer causal relationship (one thing causes another) For example, measure satisfaction of 100 human service

majors at four different kinds of jobs 1 year after graduation.


Research (slide 5 of 9) Quantitative Research

Correlational Research Simple correlational studies: Relationship between two variables Predictive correlational studies: Predicting scores on one variable

from scores on another variable (e.g., predicting grades on an exam based on how much students study)

Uses “r” goes from -1.0 to +1.0 (see bottom of p. 264)

Survey Research Questionnaire to gather information—easy to do with the Internet! Obtain information from respondents and examine the frequency

of responses and differences between different groups (e.g., males and females).

For example, survey of attendance in counseling found that 75% of human service practitioners, educators, and students had been in counseling with women seeking counseling at higher rates than men .


Research (slide 5 of 9) Qualitative Research

Grounded Theory Is a process in which a broad research question is examined in a multitude

of ways that eventually leads to the emergence of a theory. First, come up with a series of questions that can be asked of the

population which is the basis of forming your theory. For example, how are helping skills developed in human service work? Interview a number of human service professionals and eventually come up

with your theory. Process involves:1. Preparing (preparing self and reflecting on your bias as you get ready to

collect info from participants)2. Data Collection (e.g., focus groups, interviews)3. Note taking4. Coding (Finding common themes)5. Writing (As final theme emerges, write up findings)


Research (slide 7 of 9) Qualitative Research

Ethnographic Research (“Description of Human Culture”; popularized by Margaret Mead)1. Also called “cultural anthropology”2. Observation (Immersion in culture and observing culture)

Nonengaged observation Participant observation (Discuss Reflection Exercise 11.2)

3. Interviews4. Collection of Documents and Artifacts

Historical Research Systematic collection of information to understand past events from a

contextual viewpoint; primary and secondary sources1. Oral histories2. Documents3. Relics4. Discuss Reflection Exercise 11.3


Research (slide 8 of 9) Examining Results

Quantitative Research Statistics, analysis, and probability

Differences between groups: t-tests, ANOVA, correlations, probability levels

Survey research: Descriptive statistics, including measures of variability, central tendency, percentages, and frequency

Validity of assessment can be high if you have a highly controlled study.

Qualitative Research Inductive analysis—emergence of patterns and data categories Coding Review of data and drawing conclusions The difficulty of obtaining reliable and valid results

Important to “triangulate” date Important to “bracket” biases


Research (slide 9 of 9) Discussing the Results

Draw conclusions based on the results and data gathered and the presentation of theories.

Be wary of saying you have found “the truth.” Why is research useful in human service work?

Understand professional articles; find ideas that may be useful for working with clients

Validation of human service work that is being done and suggestion of new ways to help clients

Suggestions of new avenues to explore Allows program evaluation


Evaluation and Needs Assessment (slide 1 of 2) Evaluation

Tells you how well something has been done—has a program been effective?

Two main types (Formative and Summative)

Formative (Process) Evaluation Assessment of a program as it is going on Often verbal or anonymous written responses

Summative (Outcome) Evaluation Assessment of a program after its completion to decide if it is

worth repeating Formal assessment procedures used Results shown to funding bodies or agencies See Table 11.3


Evaluation and Needs Assessment (slide 2 of 2)

Needs Assessments Determining to assessing needs or “gaps” between current

conditions and desired conditions Used to improve existing programs/agencies Collect information through surveys, interviews, etc. Use results to form a plan of action Descriptive statistics often employed

Human Service Professional and Program Evaluation In these days of accountability, you will be asked to assess a

program or do a needs assessment to show that what you are doing is working.


Assessment (slide 1 of 7) Defining Assessment Evaluative procedures that yield information about a person Many kinds of assessment techniques (see Figure 11.2 and 11.3)

Assessment of Ability (Achievement and Aptitude—Cognitive Domain) Achievement tests (measure what the person knows)

Survey Battery (e.g., Stanford Achievement Test) Diagnostic (See Reflection Exercise 11.4) Readiness (e.g., to move into first grade)

Aptitude tests (measures what the person is capable of) Tests of intellectual and cognitive functioning

Individual intelligence (e.g., WAIS) Neuropsych tests (for brain damage)

Cognitive ability (e..g, SATs) Special aptitude (e.g., Mechanical ability) Multiple aptitude (e.g., ASVAB)


Assessment (slide 2 of 7) Assessment of Ability

Some laws related to ability assessment PL 94-142 (special ed. Law)

Students would be in “least restrictive environment”) Individual Education Plans should be developed

Individuals with Disabilities Education Act (IDEA) expansion of PL 94-142) Personality Assessment

Objective tests Multiple choice or True/False measurement of personality MMPI and Myers-Briggs Type Indicator (MBTI)

Projective techniques Interpret response to unstructured stimuli Rorschach inkblot test and sentence completions tasks

Interest inventories Measures likes and dislikes related to career counseling Measures personality style related to satisfaction at jobs Strong Interest Inventory


Assessment (slide 3 of 7) Informal Assessment Procedures

Rating scales (see Figure 11.5) Observation: Observe client in natural setting (e.g., home) Classification systems: Assess whether a person has, or does not

have, a certain attribute (see Table 11.4) Environmental assessment: Collecting information about a

client’s home, school, or workplace usually through observation or self-reports (e.g., checklists)

Records and personal documents: e.g., autobiography anecdotal info, biographical inventory, cumulative records, genograms, journals, diaries

Performance-Based assessment: Uses real world assessments (e.g., what a person might actually do on a job) to assess for ability instead of a traditional “cognitive based” test


Assessment (slide 4 of 7)

The Clinical Interview Goals

Gains an in-depth understanding of the client Sets tone for assessment process Helps desensitize client to discussion of intimate information Can assess nonverbals of clients Gives client and helper ability to see if they can work

together

Types Structured Unstructured Semi-structured


Assessment (slide 5 of 7) Different Types of Assessment

Norm-referenced assessment techniques The individual can compare his or her score to an average

score or peer group. Criterion-referenced assessment techniques

Assess specific goals or “criterions” to be reached of an individual

Standardized assessment Procedures administered in the exact same way and under

same conditions each time Nonstandardized assessment

Procedure may be administered in a different way under different conditions each time.

Tends to allow for more flexibility but less validity


Assessment (slide 6 of 7)

Basic Test Statistics Relativity of scores—How a person does is relative to:

Where they score relative to self. Where they score relative to others. How they feel about the score.

Standardized tests usually use measures of central tendency (mean, median, and mode) and measures of variability (e.g., standard deviation) to describe where a person scores.


Assessment (slide 7 of 7) Test Worthiness

Validity: Does test measure what it purports to measure? Four types: content, concurrent, predictive, and construct

Reliability: Is the test measuring what it’s supposed to measure consistently? If I were to take the same test over and over (without me changing) would I score about the same? Four types: internal consistency, split-half, parallel, test-retest

Practicality: Is the test practical to give in terms of amount of time, cost, focus of instrument, etc.?

Cross-cultural fairness: Is the test relatively free from bias? Does it predict well for all groups? Griggs v. Duke Power company: tests used for hiring and

advancement at work must show they the can predict job performance for all groups.


The Human Service Professional’s Use of Assessment Techniques APA adopted a three tier system for deciding who can give tests:

Level A (some Level B): associate- and bachelor-level individuals who have some knowledge of assessment and are thoroughly versed with the test manual (e.g., proctoring an achievement test)

Level B: Individuals with a master’s degree and have taken a tests and measurement course (some personality tests; career tests)

Level C: Minimum of master’s degree and advanced training (e.g., projective tests, intelligence tests)

It is important to have adequate knowledge of different assessment instruments.

The effective human service professional understands that research, evaluation, and assessment are not to be feared but are tools to help progress and development of new ideas and new programs.


Ethical, Professional, and Legal Issues Informed consent: Client has right to know purpose and nature of all aspects of

involvement with the helper, including when he or she will undergo testing or be involved in research.

Use of human subjects Stanley Milgram’s “shocking” experiments led to institutional review boards. Can’t do research today unless you can show there is little or no risk to

participants

Proper interpretation and use of test data Human service professionals should know the limitations of what kinds of

assessments they can give and research they can undertake. They should always see assessment and research in a broad perspective—it

tells you a small piece of information about a person or a group… Be careful about jumping to conclusions.


Effective Human Service Professionals Understand that the main purpose of research, program evaluation, and

assessment is to benefit our clients.

Understand that research, evaluation, and assessment is always evolving.

Want to participate in some fashion in the use and development of new research, evaluation, and assessment techniques


Summary

What is research and how to conduct it

The differences between quantitative research and qualitative research

Evaluation and needs assessment

Testing and assessment

Ethical, professional, and legal issues

Slide Number 1Learning Objectives (slide 1 of 2)Learning Objectives (slide 2 of 2)Research (slide 1 of 9)Research (slide 2 of 9) Research (slide 3 of 9)Research (slide 4 of 9)Research (slide 5 of 9)Research (slide 5 of 9)Research (slide 7 of 9) Research (slide 8 of 9)Research (slide 9 of 9)Evaluation and Needs Assessment (slide 1 of 2)Evaluation and Needs Assessment (slide 2 of 2)Assessment (slide 1 of 7)Assessment (slide 2 of 7)Assessment (slide 3 of 7)Assessment (slide 4 of 7)Assessment (slide 5 of 7)Assessment (slide 6 of 7)Assessment (slide 7 of 7)The Human Service Professional’s Use of Assessment TechniquesEthical, Professional, and Legal IssuesEffective Human Service ProfessionalsSummary