welcome! if you joined the conference via telephone, please enter your audio pin if you havent...
TRANSCRIPT
Welcome! If you joined the conference via
telephone, please enter your audio pin if you haven’t already. (This is an example, your audio pin will be different)
If you joined with a microphone and headset or speakers (VoIP), please be sure your device volume settings are properly adjusted.
If your mic & headset is not working, please try unplugging and re-plugging in your device.
If you have any questions, please enter them in the questions module.
We will begin promptly at the top of the hour.
DOC110: Evaluating Document Assembly Projects
September 26, 2008
GoToWebinar InterfaceMaximize/minimize your screen
with the chevron symbol Telephone participants need to
enter their audio pin Please share your questions
and knowledge!◦ Ask a question in the
questions log◦ Raise your hand
PanelistsKate Bladow, Pro Bono NetSteve Rapp, Idaho Legal Aid
ServicesBristow Hardin, Legal Services
Corp.
Today’s TrainingWhat is evaluation?Why evaluate?How do you evaluate a program?
◦ Planning for an evaluation◦ The process◦ Case study: Idaho Legal Aid Services
What is LSC looking for in evaluation plans and reports?
Evaluation DefinedSystematically collecting data to
answer questions of interest.
Reasons to Evaluate a Program
To find out if it was successfulTo build buy-inTo document lessons learned and
potential best practicesTo identify areas for improvementTo justify the cost and effort
Evaluation Process
1.Draft an evaluation plan.2.Design and test data
collection instruments.3.Collect baseline data.4.Collect additional data.5.Analyze data and report
findings.6. Integrate necessary changes.7.Repeat steps.
Draft an Evaluation Plan
Example Evaluation PlanProject Goal: Use A2J author and HotDocs to create guided interviews and court form templates that improve the ability of low income people to easily,
effectively and efficiently produce and file court forms without the assistance of an attorney.
Project Objectives Strategies/Activities Evaluation Questions Evaluation Data
Objective 1Use A2J Author and HotDocs to create guided interviews and court form templates that enable pro se litigants to easily produce accurate pro se pleadings.
Produce [insert number] guided interviews and [insert number] interactive court forms.
Identify forms/interviews to automate Hire forms developer to help develop
guided interviews and automated forms
Work with courts to ensure that interviews/forms are accurate, consistent with court requirements and will be acceptable to the courts.
Conduct usability testing of templates to ensure users (clients and/or advocates) can easily understand and use the interviews/forms. (Note: forms should be at the eight-grade reading level of lower.)
Modify templates to address any problems identified in usability testing
Publish forms/interviews on statewide website
What forms/interviews were selected for automation?
Was developer hired to develop guided interviews and automated forms?
What interviews were produced? What forms were automated? Have courts confirmed that the forms are
accurate, consistent with court requirements and acceptable for use?
What usability testing was conducted? What were the results of the usability testing? What changes were made to the automated
forms based on the initial testing? What automated forms and guided interviews
been published on the statewide website? To what extent have the automated forms and
guided interviews made it easier for self-represented litigants to prepare accurate court pleadings?
Confirmation that developer hired. List of interviews and automated
court forms produced. Court confirmation that forms are
forms are accurate, consistent with court requirements and acceptable for use
Description of usability testing Results of usability testing Description of notable changes
made to forms and interviews following initial testing
List of forms/interviews published on statewide website
User interviews/surveys with:• Users• Court staff
User feedback from the website Field observation data
Objective 2 Increase the availability and use by low-income persons of automated forms
Partner with courts and other stakeholders to develop and implement outreach initiatives to:
• foster courts’ acceptance of the forms
• maximize clients’ use of forms Partner with public libraries,
courthouses and other groups to increase access points where clients can:
• access and complete forms • receive assistance in completing
forms
What automated forms and guided interviews have been posted the statewide website?
Was an outreach plan developed? What outreach activities were conducted? To what extent are the different forms being
used by clients? How many judicial districts/courts have
accepted the automated forms? How many new access points have been
established where clients can access the forms?
At how many access points can clients receive assistance in completing the forms?
Copy of outreach plan Description of outreach activities Statistics from national server on
number of times different forms have been accessed and/or downloaded
List of judicial districts that have accepted the automated forms
List of new access points (with description of services available to clients)
Design and test instruments
Time TrialsFocus GroupsUsage StatisticsUsability TestsDocument
ComparisonsInterviewsSurveys
Collect baseline data
Allows you to compare before and after.
Can be collected while the document assembly content is being developed.
Keep this data collection consistent with your later data collection.
Collect additional data
Happens after implementation.
Stay consistent with your early data collection.
Analyze data and report findings Project Goals and Objectives Evaluation Data and Methodologies Summary of Major Accomplishments Recommendations and Future Steps In-Depth Analysis of Accomplishments Factors affecting project
accomplishments Strategies to address major challenges Major lessons and recommendations
Integrate necessary changesUse the feedback that you
gathered.
Tweak the program and then collect more evaluation data to find out if it really was a positive change.
Repeat
Plan
Design
CollectAnalyze
Report
Integrate
Case StudyIdaho Legal Aid Services
Idaho Project HighlightsPartnership between Idaho Legal Aid Services and the Idaho Supreme Court
Funding provided by the Legal Services Corporation and the State Justice Institute
Successful – good feedback and good filings
©2008 National Center for State Courts
A Conversation with LSC
Resources
DA and Legal Aid Specific
Document Assembly Supporthttp://www.probono.net/dasupport
LSC Evaluation Plan and Report Formshttp://tig.lsc.gov/eval_nonwebsite_grants.php
SelfHelpSupport.orghttp://bit.ly/y4c4s
Overview and TheoryW.K.K. Foundation Evaluation
Handbookhttp://bit.ly/1fg3n4
Outcome Measurement Resource Networkhttp://bit.ly/4rD2eY
Basic Guide to Program Evaluationhttp://bit.ly/eHHjW
Intelligent Measurement Bloghttp://intelligentmeasurement.wordpress.com/
Point Khttp://bit.ly/HQL2k
Surveys
LSNTAP Survey Toolhttp://bit.ly/2ARmCF
Creating Effective Online Surveyshttp://bit.ly/1bLpwg
Likert Scaleshttp://bit.ly/4nlZlh
Usability Testing
Usability Methodshttp://usability.gov/methods/
Assess Your Site’s Navigation & Structurehttp://bit.ly/1Bc1Pv
Useit.comhttp://www.useit.com/
Focus Groups
How to Use Focus Groupshttp://bit.ly/NHLBA
Data AnalysisExcel Skills for Advocates
http://www.lsntap.org/OFF105W.K.K Foundation Data Analysis
http://bit.ly/44nU2hHow to Analyze Data
http://bit.ly/3sRnvPOverview Introduction to Statistics
http://bit.ly/3TpaMdThe Complete Idiot's Guide to
Statisticshttp://bit.ly/2t0Hen or http://bit.ly/4hRQok
Attributions Image – Slide 6 – SMercury98
http://www.flickr.com/photos/smercury98/ Image – Slide 8 – markymarko
http://www.flickr.com/photos/cybercafe/ Image – Slide 10 – roboppy
http://www.flickr.com/photos/roboppy/ Image – Slide 10 – gergenzola
http://www.flickr.com/photos/gergenzola/ Image – Slide 10 – Eric Spiegelman
http://www.flickr.com/photos/ericspiegelman/ Image – Slide 10 – jimmedia
http://www.flickr.com/photos/jimmediaart/ Image – Slide 10 – cambodia4kids
http://www.flickr.com/photos/cambodia4kidsorg/ Image – Slide 10 – ifindkarma
http://www.flickr.com/photos/ifindkarma/ Images – Slide 13 – johnbullas
http://www.flickr.com/photos/johnbullas/ Image – Slide 27 – Faeryan
http://www.flickr.com/photos/faeryan/ Content – Slides 7, 8, 10-13, 16-18 – Adapted from Court Leadership and Self-
Represented Litigation Package: Module 7http://bit.ly/3qjLJ4