Field Testing in 5 Easy Steps

Field Testing Legal Documents

TIG Grant 15007

Maria Mindlin, TranscendPlain Language, Field Testing & Cultural Competency

1

Marias Background 25 years at Transcend, an agency that specializes in Plain English and certified translations. Lead author on the first (and so far only) quantitative test of Comparative Readability Study of Plain Language Court Forms in the U.S. (Scribes Journal of Legal Writing, 2005) Drafted and tested plain English documents for legal aid agencies and courts in 20+ states. Readability instructor, UC Davis Law School, under Professor Richard Wydick (author ofPlain English for Lawyers). Edited Bryan Garners Redbook text on visual accessibility.

TIG Grant 15007

2

Why field test?

TIG Grant 15007

What is this about?Who is it for?

3

Why field test?You want as many people as possible to read & understand the legal information you provide.

The feedback you get from FT helps you achieve this goal.

TIG Grant 15007

TIG Grant 15007

FT / FG is your opportunity to collect data and hopefully.Data-Driven LexiconYou are listening!What words do they find easy or hard? What do they prefer?

And will be shaped by what you hear5

Findings are based on what you hear and understand

Other options:Fairly quick, reasonably inexpensive, extremely instructive!Observations, interviews, chats, focus groups

TIG Grant 15007Focus Group Testing is one Kind of Qualitative Testing

No doubt you already know there are 2 overall types of testing : Quantitative & Qualitative

Under Quantitative, the most common ones are:

surveys, polls, other data collection methods

This type of testing produces reliable statistical data that can be generalized to the larger population. These methods are used for voting and large opinion polls.

Under Qualitative the most common ones are:

Observation, focus groups, interviews

This approach is usually the initial step in gathering information. Most quantitative tests start with qualitative pretests.

6

Record & Share Your Readers Preferences

TIG Grant 15007

Track writing styles, words, images they prefer. Let your writing be guided by your readers preferences and reading proficiency. I used to know how to write; then I went to law school.

Party v personPrevailing v won your case, decided in your favorDetermination b decision

7

Must-Have Resource

TIG Grant 15007

Focus Group Kit:1 Guidebook2 Planning3 Developing Questions4 Moderating5 Recruiting6 Analyzing & ReportingCost for all 6: $15-$50

Goals for Todays Webinar

You will learn:1How to Prepare Your Instrument (Script)2How to Recruit Participants3How to Train Your FG Team 4How to Conduct Your FG Testing5How to Report Your Data

TIG Grant 15007

9

Materials Needed:Instrument Template & Sample Test Document

TIG Grant 15007

Start With Your Best Draftart With Your Best DraftYou believe your draft is: clear, concise, focused correct good RGL 10

TIG Grant 15007

Template as a Starting Point

You dont have to reinvent the wheel!You see why we ask certain questionsWhen we all use similar formats, we can talk apples to apples.You wont commit the sin of running LONG!Youll see examples of open questions

11

Instrument Collects Data from basic CategoriesAn introduction Content questionsLanguage questions Likes and DislikesTitleOther suggestions

Writing the pre-test report also helps the team to record findings and make recommendations. We talked before about how sometimes you find small or large problems in the pretest.

The categories help us be efficient in a number of ways: You can re-use many of them, with a little adaptation

You can use them as a framework to structure data collection & write your report If gives you a good in a standard a predictable format that allows you to extend lessons learned to other documents.

For example, we keep a data base of the language questions (words that some people may find hard) and likes and dislikes to guide us on future projects.

When we talk to each other about our documents, it also gives us a common format to approach any challenges.

This is your chance to fix any small areas, and to decide if any major fixes are needed before the actual focus group.

12

How many questions?Researchers say no more than 12.

Reasons:Time is limited. You dont want to rush through the great questions you have!Ideal time is 1 hour, maybe 1.25 hours. People get tired, restless.

Having 12 questions also means you really have to focus on your main goals.

TIG Grant 15007

Researchers say no more than 12.

The reasons for this are:Time is limited. You dont want to rush through the great questions you have!Ideal time is 1 hour, maybe 1.25 hours. People get tired, restless.

Having 12 questions also means you really have to focus on your main goals.

13

Developing An Instrument

Your task:Look at your text. Highlight the areas you most want the readers to learn. Target those areas for questions. Lets see some examples.

TIG Grant 15007

Please have your content handy. You will need it for this next task.

Just follow along and well do this together.

Have you highlighted the areas that you most want your readers to learn? How many are there?

A tip: if there are more than 3 MOST important areas, you may have to break down your text into smaller pieces.

14

Target language translation will print on this side of page (Use as flyer, or half sheet w/ translation on back)Bottom area to be cut off and used as a card.

TIG Grant 15007

So if you were testing this draft, what would you want the reader to know?Take a minute or two and write that down.

Do you want the person to know: they have language rights?They can get a free interpreter?How to get the free interpreter?What to do if they dont get an interpreter?

15

How to Start

TIG Grant 15007

Introduction:My name is , and I work for Transcend. We are a company that helps make information easier for people to use. I am here today with my co-worker, . She will listen to your answers and make notes. This helps us to remember all of your suggestions!Today we need your help with a flyer. We want to know what you think about it and learn how we can make it better.We really want your honest opinions. Its OK to say anything, even if its negative. Telling us what you like and dislike will help us make the flyer better for all users. OK, does everybody feel comfortable about saying anything they want?Shall we begin? Make participants feel comfortable about saying anything they want. Explain why their contribution is so important. Facilitator does not correct answers or provide addl info. S/he responds with a warm OK, good, great. to all answers.

Content

TIG Grant 15007

1. What is this flyer about?

(Allow 30 seconds. Observer writes each answer given using each participants exact words.)

____________________________________________

____________________________________________

____________________________________________ If users cannot quickly identify the purpose, our text and design have failed.Noting the participants exact answers tells us what terms they typically use.

[Reading Interest]

Audience

TIG Grant 15007

2. Who is this flyer for?

(Observer writes each answer given using each participants exact words.)

____________________________________________

____________________________________________

____________________________________________ If users cannot identify the audience, they are less likely to persist.

[Reading Persistence]

Match / Mismatch

TIG Grant 15007

3. Do you think this flyer would be helpful for people who dont speak English well? With each yes, answer, ask: OK, so you think it could be helpful? In what way? ____________________________________________

____________________________________________ With each no, answer, ask: OK, so you said it may not be helpful? Can you tell me more about that?____________________________________________

____________________________________________ Their overall impression tells us about level of reading interest this flyer generates.

Contextualize

TIG Grant 15007

4.Lets say you did not speak English really well, and you want to apply for public services, such as Food Stamps or ask the police for help with a problem in your building. What does this flyer tell you to do? (Observer checks each item participants can name. Observer also notes any incorrect answers provided.) Go to the office where you want services Let them know you have the right to get an interpreter Let them know you have the right to get written translations of imp info Show them this card Call the Language Access Coalition to report it Text the Language Access Coalition to report it Go to the masslegalhelp.org website to get more info. Other: ______________________________________ Their answers reveal how clear each item is.

So is it clear that if the police dont help you, you would have to call us?20

Is this an open question?

TIG Grant 15007

5.Is it clear from the flier that calling or texting the number provided is only to report having been denied an interpreter, and that calling/texting will not get you an interpreter?

If not, how can that be made more clear? Their answers reveal how clear each item is.

So is it clear that if the police dont help you, you would have to call us?21

Make it open.

TIG Grant 15007

5.What is this telephone number for?

Point to tel # and see what they say.

(Accept all answers, do not correct or explain.)Their answers reveal how clear each item is.

So is it clear that if the police dont help you, you would have to call us?22

TIG Grant 15007

Always ask this!

are there any words that might be difficult for some people? So, even if its easy for you, mark any words that could be difficult for some people.

Dont ask, is this word difficult?

Keep track of these words. Over time you will have a wonderful glossary of words that readers find difficult and suggestions for easier words.

FYI: In this test, participants found these words difficultLanguage AccessCoalitionDeniedHousing AuthoritiesLanguage Rights

23

Always ask this!

are there any words that might be difficult for some people? So, even if its easy for you, mark any words that could be difficult for some people.

Dont ask, is this word difficult?

Keep track of these words. Over time you will have a wonderful glossary of words that readers find difficult and suggestions for easier words.

FYI: In this test, participants found these words difficultLanguage AccessCoalitionDeniedHousing AuthoritiesLanguage Rights

24

These sections should also be a part of every focus group instrument.

Each answer for likes and dislikes should be explored, Which color would be better?What did you think of when you first saw this image?What image would be better?

TITLEThis may be the most important part of the test. You can understand from the answers what the text means to them. You can get ideas for a title expressed in words that READERS will relate to.

Other improvementsAs you can imagine, this area is also potentially very rich.

25

Focus Group SizeLarge: 9-12 participantsSmall: 4-8 participants

TIG Grant 15007

TIG Grant 15007

Who to recruit:1. Recruits should match the profile of your typical consumers2. Recruits should also match profile of under-served populations, e.g., language / ethnic minorities3. With 8 spots shoot for a range!4. Different ages, education & literacy levels, primary languages, ethnic / racial backgrounds, gender, income levels

How to recruit

TIG Grant 15007

It takes time to build a database of participants.Get permission from people in charge first! Start by recruiting from places where typical consumers are waiting. Ex: health clinics, social services offices, jury pools, explore your local resources! If not available this date, ask if interested in the future. Collect demographic information now!

Your FG Team:

TIG Grant 15007

Facilitator / Moderator Smile, speak pleasantly, calmly. Repeat participants input, ex, OK, great, so you think that part is not clear. Be flexible! If someone arrives late, explain quickly what you are doing, and get right back on track. Address participants by name. (Have name tags ready.) Summarize key points, ask if others agree. Keep good pace. Not too fast. Not too slow. Keep people on track. Lets talk more about that later after the group so we can hear from everybody.

TIG Gant 15007

And most importantly

If you are attached to the document and cannot embrace negative comments Let someone else be the facilitator.

TIG Grant 15007

Note Taker & Observer Read the document and instrument carefully before the pretest. Stay in the background but sit close enough that you can hear what everyone says. Participate in FG report writing/review.Observers job is to take it all in without the distraction of note-taking, facilitating.

S/he can help recall information & contribute to report so it accurately reflects the FG.

TIG Grant 15007

Why Pretest?

1. Rehearsal for the FG team - get each person comfortable in his/her role3. Trouble shoot problems, e.g. typos, clumsy or ineffective questions, major flaws!2. Confirm logistics: where to sit, materials needed, timing 4. Additional data to compare w/ FG

It lets you all feel more comfortable and you can refine your instrument32

Demographic InfoName:

Tel:

Email:

Age:

First language:

Last grade completed:

TIG Grant 15007

A db with this information will help you round out your recruitment to make sure you get people from different slices of the population. As you move forward you will find that some participants are amazing dont hesitate to invite them frequently!

33

Pay

TIG Grant # 15007

Regular Group (1 to 1.5 hours)

$30 - $50 Cash

Or equal amount in Supermarket certificate

Enough to encourage people to juggle schedules and come!

Stipends also convey that you value their participation, opinions, and sets the right tone.

Makes it easier for you!

A db with this information will help you round out your recruitment to make sure you get people from different slices of the population. As you move forward you will find that some participants are amazing dont hesitate to invite them frequently!

34

TIG Grant 15007

Prep WorkReady Your Space Chairs, table Mock-up of text to test (print) Pens and yellow highlighters Address labels and markers for each participant to write his/her name Comfortable, cozy is OK. Toys & extra chairs if someone shows up with kids you will be ready! Make sure there is a clock. You and observer must keep track of time. Clipboards for Facilitator, Note taker, Observer

TIG Grant 15007

How to Collect Your DataNote Taker & Observer Make your instrument suitable to record comments Also use test document to make marksFacilitator Use document or instrument as needed for facilitating Example of suitable instrument

TIG Grant # 15007

What is this flyer for? (Allow 30 seconds.)

Getting translations

Getting interpreter

Getting someone to help you at a public office

Complaining about not getting help

Teaching you about your rights

Helping you to understand a new law

Other:

Other:

Other:

Try to anticipate likely responses so notetaker can check off quickly. You may want to update this based on your pretest results. For instance, if a lot of people in the pre-test said, agencies cant require you to bring your own interpreter any more, add it!

37

TIG Grant 15007

Quiet / Shy People Call them by name. Ana, do you agree with Bob? Which parts did you like most? Some questions are suited to quick polls: Who else thinks this word is difficult? Raise your hand. Some questions are suited to going around the table:Who do you think this flyer is for? Lets start with Florencia, and go around this way.

TIG Grant 15007

Difficult People

Very interesting. Lets talk about it more right after the group. Thank you for the comment. I dont want to interrupt you, but I need to finish these questions first. Lets talk more afterwards. Wow! You have such interesting experiences. Lets talk after we finish.

Target language translation will print on this side of page (Use as flyer, or half sheet w/ translation on back)Bottom area to be cut off and used as a card.

TIG Grant 15007

So if you were testing this draft, what would you want the reader to know?Take a minute or two and write that down.

Do you want the person to know: they have language rights?They can get a free interpreter?How to get the free interpreter?What to do if they dont get an interpreter?

40

Before I asked you to write your best draft. It doesnt have to be formatted yet, just enough basic layout features to give an idea of what would go where.

Our best draft had:A few icons, chunked text.Easy words172 words5.4 RGL41

TIG Grant # 15007

This is the flyer weve looked at before.

You may remember from our last video, we always start with a reading interest question, like what is this flyer about?

42

TIG Grant 15007

TIG Grant 15007

Thank you!Maria Mindlinme@transcend.net

TIG Grant 15007

In the next video, Video 3, you will learn about:1. Why you should pretest?2. Tester Roles3. How to Collect Language & Demographic Data 4. Possible revisions to instrument or document45

You have language rights!

The law says most public offices must give you services in your language. Public offices must not send you away because of your language, or ask you to bring your own interpreter.

You have the right to a free interpreter who will keep your information private. Ask for an interpreter at: Schools Courts Hospitals and doctors offices Police Government services Some housing authorities

Show the card below to ask for: a free interpreter, and written information in your language What if they do not give me an interpreter? Call or text the Language Access Coalition: 617 xxx-xxxx, orCall your local legal aid office.

Cut here Show this card to ask for an interpreter.I speak _________. I need an interpreter to talk to you.

Title VI of the Civil Rights Act says I have the right to an interpreter and information in my language.

Questions? Talk with your supervisor or look to your agencys language access policy. If you cannot give me an interpreter, contact the Language Access Coalition. Call or text: 617-xxx-xxxx (for English ) or 617 xxx-xxxx (target language)You have language rights!

The law says most public offices must give you services in your language. Public offices must not send you away because of your language, or ask you to bring your own interpreter.

You have the right to a free interpreter who will keep your information private. Ask for an interpreter at: Schools Courts Hospitals and doctors offices Police Government services Some housing authorities

Show the card below to ask for: a free interpreter, and written information in your language What if they do not give me an interpreter? Call or text the Language Access Coalition: 617 xxx-xxxx, orCall your local legal aid office.

Cut here Show this card to ask for an interpreter.I speak _________. I need an interpreter to talk to you.

Title VI of the Civil Rights Act says I have the right to an interpreter and information in my language.

Questions? Talk with your supervisor or look to your agencys language access policy. If you cannot give me an interpreter, contact the Language Access Coalition. Call or text: 617-xxx-xxxx (for English ) or 617 xxx-xxxx (target language)