evaluating cctv and recording of child interviews and testimony
DESCRIPTION
Evaluating CCTV and Recording of Child Interviews and Testimony. M. Christine Kenty, PhD Sharon Elstein, MS ABA Center on Children and the Law. KENTY'S CLUES FOR EVALUATION. #1. Base any evaluation on your own goal pathway and logic model. - PowerPoint PPT PresentationTRANSCRIPT
Evaluating CCTV and Recording of Child Interviews
and Testimony
M. Christine Kenty, PhD
Sharon Elstein, MS
ABA Center on Children and the Law
KENTY'S CLUES FOR EVALUATION
#1. Base any evaluation on your own goal pathway and logic model.
#2. Organizations are social systems with their own cultures.
#3. Whatever people don't want you to study, that's the really important thing.
And those touchy things are likely to be:
Collaboration Quality of Forensic Interviewing Decision-making about arrest, prosecution
and child protection Relating to victims and families Children’s experiences before and in the
grand jury or courtroom
Kenty’s Clues
#4. It's trouble if only one person is doing all the thinking about evaluation.
#5. You can't keep partners and stakeholders too well informed about the evaluation process.
Kenty’s Clues
#6. Any evaluation (or program report) requires DATADATA, so decide early what you need to record and then keep it up.
#7. For any evaluation, there are many good designs, but no perfect ones.
#8. Don’t kill the messenger if you don't like the news!
Evaluation…
is a way to improve a program by systematically examining
and analyzing what the program is doing and what it has
accomplished.
The “I’ll know it when I see it” Rule
I’ll know that our program is working when I see……….
What evidence can convince us and others that our program is on target?
What would tell us that something has happened?
How can we count it or track it?
What evaluation can do:
Help improve the program from the beginning
Provide staff and stakeholders with a much-needed sense of accomplishment
Guide protocol, policy and law reform Assist in developing future funding
Don’t put the evaluation in the hands of just one individual
Whether that is an internal or external evaluator
A healthy organization needs to know the program mission, plan the work, develop enthusiasm, and bring things to fruition.
An organization also needs to look at what it’s doing, keep what's good and try to jettison what isn't working.
Evaluation isn't a separate topic….
it's just one more piece of the work
Forces will try to marginalize and minimize an evaluation
Don’t let that happen –
establish a strong committee!
Base the evaluation on your agency pathway.
“Logic model” and “pathway map” are popular phrases with funders.
Logic Model or Pathway MapEach part should logically follow from the last
Context(What we live in)
Strategy(Where we focus)
Activities(What we do)
Outcomes(Results)
We all work on underlying assumptions, which might be ---
Better technology will improve dispositions Technology will make things easier for kids –
fewer interviews, less testifying… Our forensic interviewing will stand up to
scrutiny The defense bar will not limit the potential of
recorded testimony We will know if this is working well
TALK ABOUT AND CLARIFY THESE!
Logic Model or Pathway Map
Context•Needs•Resources•Laws/man- dates•Stakeholder buy-in
Strategy Activities Outcomes
You may do a needs assessment to describe your context
A needs assessment is a systematic way to discover: what you “need” in order to accomplish a goal, and thenmake decisions based on that assessment
Typical needs assessments1. Estimate how many clients/professionals will
participate in a new program
2. Determine what resources are already in place and what has to be put into place
3. Decide what an agency or community needs to provide to get a particular result
4. Envision how technology and products will be used so that the equipment will be right
5. Decide what training people need
You may have already done one kind of needs assessment
but you may still want to do another piece as you begin to
implement your program
1. Estimate how many clients/professionals will be involved
Count # of allegations, investigations, interviews, arrests, prosecutions, hearings, dispositions in the last year
List all the professionals who will need to be trained or familiarized
2. Determine what resources are in place and needed
All professionals and what they do Adequacy of infrastructure: rooms,
technology, wiring, lighting Relevant state statutes re victims and
CCTV and/or recorded testimony Policies and procedures for
interagency work Other available funding
3. Decide what is needed for specific result (These are suggestions, not requirements)
What equipment, facilities, personnel, time, training, systems, policies, statutes do we need, for example:
To videotape all child interviews for children aged 3-13?
To decrease the number of child victims who testify in person?
To improve successful disposition rates? To make the system more child-friendly?
4. Envision how equipment/product will be used
Stationary or portable Professional technician or multiple users Video all children or selected cases or ages CCTV according to statutes Show videos in what rooms to what audiences Who needs a cut-off switch Maintenance, upkeep costs, back-ups
5. Determine what training people will need
Technical skill in equipment use and maintenance Scheduling and informing children/caregivers Interviewing skills Judiciary, Prosecutors and Bar
Permissibility/use of recording and CCTV according to all statutes Awareness of capacity Forensic use of recordings at multiple points
Quality assessment and record-keeping for recorders and prosecutors
Design method of data collection
Type of assessment question Possible Methods
1. # of clients and professionals
Document review: case records, annual reports, policies, inter-agency agreements
Interviews/surveys
2. Resources have/need Document review: case records, annual reports, policies, inter-agency agreements
Surveys/focus groupsObservation
Design methods of data collection
Type of assessment question
Possible Methods
3. What will achieve a specific result
Focus Group
InterviewsDocuments/Information Search: literature, state-of-the-art
4. Likely use of equipment Document Review – statutes, case records
Focus Group/Interview
Design methods of data collection
Type of assessment question
Possible Methods
5. Training needs Review of vendor materials
Surveys/interviews
Observation
Logic Model or Pathway Map
Context•Needs•Resources•Laws/man- dates•Stakeholder buy-in
Strategy•Technology•Staff•Training•Information
Activities Outcomes
Logic Model or Pathway Map
Context•Needs•Resources•Laws/man- dates•Stakeholder buy-in
Strategy•Technology•Staff•Training•Information
Activities•Acquire, install equipment•Train•Inform •Partner•Use
Outcomes
Logic Model or Pathway Map
Context•Needs•Resources•Laws/man- dates•Stakeholder buy-in
Strategy•Technology•Staff•Training•Information
Activities•Acquire, install equipment•Train•Inform •Partner•Use
Outcomes•Short term•Medium term•Long term
Logic Model or Pathway Map
Context Strategy Activities OutcomesShort term•Technology•Attitudes•Skills•Opinions
Ideas for short term outcomes
Plans are complete; we know who contributed and what went into planning
Stakeholders are aware and committed Equipment is acquired and installed
Logic Model or Pathway Map
Context Strategy Activities OutcomesMedium term•Action•Practice•Policies
Ideas for medium term results
Equipment being used regularly People skilled and knowledgeable Stakeholders have assessed use Data being collected on interviewing, use of
recordings and CCTV, and can be summed Data establishing a baseline for later
comparison
Logic Model or Pathway Map
Context Strategy Activities OutcomesLong term•Impact on the overall context
Ideas for long term results (impact)
Quality of forensic interviewing and recording is consistently high
Prosecutors use recording and CCTV regularly and effectively in known ways
Prosecution rates and/or case dispositions have been improved
Stakeholders, clients and families are satisfied with the process and use of recordings
Laws and policies have been changed
TYPES OF EVALUATIONS
NEEDS ASSESSMENTA systematic way to discover what you
need to accomplish a goal
Helps make informed planning decisions
TYPES OF EVALUATIONS
FORMATIVE EVALUATIONShort-term initial feedback on how the
program is working
Helps quickly readjust planned activities to be more effective
TYPES OF EVALUATIONS
PROCESS EVALUATION
Describes how something happened rather than outcomes
To understand the internal dynamics of organizations and relationships, and capture
what activities are actually happening.
TYPES OF EVALUATIONS
IMPACT or OUTCOME EVALdetermines whether a program
produced desired results.
Requires articulated outcomes and targets and they must be measurable.
RESEARCH METHODS
QUALITATIVE METHODS
QUANTITATIVE METHODS
RESEARCH METHODS
QUANTITATIVE METHODS:
numerical research by collecting data about pre-selected variables, and studying cause and effect
QUALITATIVE METHODS:
naturalistic research by studying participants' perceptions and experiences in context and the way they make sense of them
QUANTITATIVE METHODS
surveys with pre-determined categories and rating scales
document review – e.g. counting up numbers of arrests or prosecutions, and comparing them to other groups or time periods
evaluator attempts to keep at an objective distance from the people
QUALITATIVE METHODS
surveys with open-ended questions interviews – semi-structured observation document review – e.g. process, attitudes case studies focus groups evaluator gets close to the people to capture
what is actually happening
Collect both quantitative and qualitative data - they are
equally valid
Qualitative data help capture changes in processes and
relationships, and some things just aren’t countable.
Sampling: how many records or which people to ask
More important to have a representative sample than a
large sample
But, every professional might need to be heard so that no one
feels left out and there is no suspicion of bias
So sampling may not be acceptable
Shaping evaluation questions
Outcome Questions MethodShort term
Plans are complete
What are the plans and who did what?
Are the plans comprehensive?
Document review
Interviews
Shaping evaluation questions 2
Outcome Questions MethodShort term
Stakeholders aware and committed
Are key players knowledgeable and ready to run?
What are their reservations and perceived barriers?
Observation
Interviews
Focus Group
Shaping evaluation questions 3
Outcome Questions MethodsShort term
Equipment acquired/in-stalled
Is it up and running?
Can technician produce good quality video or transmission?
Should quality be improved?
Observation
Document review
Interviews
Shaping evaluation questions 4
Outcome Questions MethodsMedium Term
Equipment used regularly
What interviews or testimony are being recorded or transmitted via CCTV?
How are recordings stored and accessed?
Observation
Document review
Shaping evaluation questions 5
Outcome Questions Methods
Medium term
Key players skilled and/or knowledgeable
Who has been trained?
Do key players feel prepared, confidant?
What else is needed?
Observation
Interviews
Shaping evaluation questions 6
Outcome Questions Methods
Medium term
Stakeholders have assessed use
Does everyone like the program?
What doesn’t work for whom?
Surveys
Interviews
Focus group
Shaping evaluation questions 7
Outcome Questions Methods
Medium term
Data being collected on interviewing, use of recordings, etc.
Are logs and instruments completed consistently?
Do they have the right indicators?
Document review
Interviews
Shaping evaluation questions 8
Outcome Questions Methods
Medium term
Data establishing baseline
Have we assembled data on quality of interviewing, arrest, prosecution, disposition before video/CCTV?
Document review
Shaping evaluation questions 9
Outcomes Questions Methods
Long Term
Quality of forensic interviewing is high
How do they rate on adherence to protocol, state of the art, forensic value?
Do we peer review?
Observation
Review of selected recordings
Interviews
Shaping evaluation questions 10
Outcomes Questions Methods
Long term
Prosecutors, child protection use technology regularly, effectively
How do they use?
Are they satisfied?
What changes can increase use?
Observation
Document review
Interview
Survey
Shaping evaluation questions 11
Outcomes Questions Methods
Long term
Prosecution rates and/or case dispositions have improved
What are the rates and dispositions before and since video/CCTV
Document review
Shaping evaluation questions 12
Outcomes Questions Methods
Long term
Stakeholders, clients and families are satisfied
What works/does not work for each profession, group, agency?
Survey
Interviews
A monitoring system is part of the implementation of the
program, and is also a key element of any evaluation.
The monitoring system insures that....
Each part of the system can get the information it needs to proceed, i.e. arrest and prosecution, child protection, treatment, victim advocacy.
You can assess the effectiveness of your program.
If you know what is happening to your children and defendants from
start to finish...
Then you have an integrated data system
If you don’t............................
Then policies, technology or turf issues are preventing it, and
your evaluation will be affected.
Data collection for recorders
Intake log of all alleged victims: demo, perps Log of recorded interviews
Date, Number, Name, Age, Gender, Length Forensic Interviewer Observers – keep copy in central location, not just in evidence
Referred for prosecution – capture prosecutor # and date
Quick assessment of quality of interview by effectiveness, protocol and forensic value –highlight really good or not so good interviews for training and peer review library
Data collection instrument for assistant prosecutors Keep copies in a central location, not separately in each file or jacket!!
#____ Defendant _______Victim________AP________ Observed the forensic interview Reviewed the recording of the interview Forensic value of the recorded interview = Poor
Good Very GoodExplain:
Used the recording to encourage family cooperation w/ prosecution Explain:
Used the recording in diverting the caseExplain:
Used the recording to obtain a pre-indictment plea Explain:
Data collection instrument for assistant prosecutors 2
Used the recording or CCTV to obtain an indictment (GJ, deposition, other)
Explain: Used the recording to obtain a post-indictment plea
Explain: Used the recording or CCTV at pre-trial hearing or trial
Explain: The child testified before the grand jury or at
preliminary hearingExplain:
The child testified at trialExplain:
Verdict and sentence:
Other possible evaluation data
Answers to items on a survey Answers to questions in an interview
format Responses in a focus group Observations of group interactions in a
planning/protocol meeting Observations of forensic interviews or
children’s testimony
You can’t keep everybody too well informed about the evaluation.
Too much is never enough; despite your best efforts, somebody may still feel
blindsided.
Don’t kill the messenger if you don’t like the news!
Don’t allow the evaluation to be deep-sixed if there are negative
findings.
And don’t ever let the evaluator or committee twist in
the wind!
If the committee has done it right, everybody will understand
the evaluation process and what the findings are before a final
report comes out.