©Yukiko Watanabe, John Davis, and John Norris Foreign Language Program Evaluation Project National Foreign Language Resource Center,

University of Hawaii at Manoa, 2012

A utilization-focused approach to developing, mapping, and assessing student learning outcomes in college

foreign language programs

1 of 22

Module  V  

Reflec&ng  on  assessment  

Implemen&ng  

U&lizing   Planning  

Situa&ng  

Reflec&ng  

Mapping  SLOs  

Developing  SLOs  

2 of 22

About Module V The goal of outcomes assessment is to enable academic programs to seek out, understand, and provide optimal conditions for student learning. A continual cycle of assessment inquiry and action calls for college faculty to take collective responsibility for educational effectiveness as part and parcel of what they do. To maximize the contributions of assessment, it is therefore essential to reflect on and improve associated practices as they are designed, implemented, and acted upon. Module V takes you through the steps of reflecting on and evaluating the assessment process. Note that the reflection phase is depicted at the center of the assessment cycle—although “reflection” is described here at the end of the modules, is not intended to be only a summative evaluation activity, but also a formative process that informs every step along the way towards useful assessment.

3 of 22

How  to  use  the  module  Because outcomes assessment should be a collective endeavor, we encourage you to go through this module together with your colleagues. The module includes tasks and resources that will help facilitate your outcomes assessment planning.

à Work on the questions with your colleagues.

à If you see this icon, access the resources for further information and examples by clicking on the link provided.

4 of 22

The goal of reflection is to improve our practices by examining the ways in which assessment is conducted. Recall the Standards of Practice we outlined in Module II (i.e., Utility, Accuracy, Feasibility, Propriety, and Accountability). These standards provide a benchmark for sound assessment practice, and should guide the decisions, execution, and evaluation of assessment. Going back to the basic premise of utilization-focused outcomes assessment, the assessment quality in our framework emphasizes actual uses of assessment, that is, whether the plan and process supported intended use and whether the assessment was actually used as intended.

Reflec&ng   Reflec&ng  on  the    assessment  path  

5 of 22

The next few slides list some key characteristics of utilization-focused assessment for reflection. The statements are formed based on utilization-focused evaluation premises (Patton, 2012), a standards-based meta-evaluation checklist created by Stufflebeam (1999), and our experiences facilitating assessment in various college foreign language programs.

Reflec&ng  on  the    assessment  path  

Reflec&ng  

6 of 22

Reflecting: Situating & Planning

Situa-ng  assessment   Not  at  all  

A  liBle  

Some-­‐what  

Verymuch    

N/A  

There  was  adequate  representa&on  of  primary  intended  users  and/or  key  stakeholders  in  the  assessment  working  group.    The  assessment  schedule  was  realis&c  (mee&ngs,  deadlines,  etc.).    Time  commitment  of  those  involved  was  respected.    

Assessment  working  group  members  were  clear  about  their  responsibili&es.    Assessment  working  group  members  followed  through  with  their  responsibili&es.    Different  posi&ons  and  percep&ons  of  different  interest  groups  were  respected.    There  was  a  consensus  on  the  assessment  approach  and  procedures.      Opportuni&es  to  learn  about  outcomes  assessment  were  provided  throughout  the  process  for  those  who  needed  them.    

 Judge  to  what  extent  the  following  were  accomplished.  

Reflec&ng  

7 of 22

Reflecting: Situating & Planning U&lizing  

Situa-ng  assessment     Not  at  all  

A  liBle  

Some-­‐what  

Very  much  

N/A  

Key  stakeholders  were  involved  in  iden&fying  assessment  needs.  Factors  that  may  hinder  assessment  use  were  iden&fied  from  the  beginning.    The  assessment  commiBee  strategized  to  overcome  assessment  barriers.  Planning  assessment   Not  

at  all  A  

liBle  Some-­‐what  

Very  much    

N/A  

Assessment  planning  was  transparent.    Faculty  were  kept  informed  about  assessment  updates  throughout  the  process.    Op&ons  for  assessment  ques&ons  were  solicited  from  various  users  (and  were  considered!).    Priority  was  given  to  the  most  important  assessment  ques&ons.    Assessment  ques&ons  were  linked  to  intended  purposes  and  uses.    Assessment  methods  elicited  informa&on  that  answered  the  priority  assessment  ques&ons.    

8 of 22

Reflecting: Planning U&lizing  

Planning  assessment  (con-nued)   Not  at  all  

A  liBle  

Some-­‐what  

Very  much  

N/A  

Selected  methods  were  chosen  on  the  basis  of  considering  strength  and  weaknesses  of  various  methods.    The  selec&on  of  methods  factored  in  the  &me  and  resource  constrains  of  the  program  context.    Instruments  went  through  pilot-­‐tes&ng  or  careful  review  to  eliminate  any  poten&al  biases,  errors,  and  irrelevances.    Analysis  and  interpreta&on  procedures  assured  credibility.    Poten&al  findings  were  an&cipated  to  improve  assessment  design.    

9 of 22

Reflecting: Implementing U&lizing  

Implemen-ng  assessment     Not  at  all  

A  liBle  

Some-­‐what  

Very  much    

N/A  

The  assessment  design  was  adapted  to  the  changing  condi&ons  of  the  context  and  needs.    Sufficient  informa&on  was  obtained  to  address  the  assessment  ques&on.    There  was  a  person  in  charge  of  data  management.    Data  was  managed  in  a  secure  manner  to  ensure  data  quality  and  ethics.    Data  collec&on  ensured  par&cipants’  privacy  rights.    The  primary  intended  users  were  involved  in  interpre&ng  findings.    Procedures  for  data  analysis  and  interpreta&on  were  monitored  and  recorded.    Factors  affec&ng  data  quality,  analysis,  and  interpreta&on  were  monitored  and  recorded.    Limita&ons  to  data  interpreta&ons  (due  to  design  or  unexpected  factors  affec&ng  data  gathering)  were  acknowledged.    Consistency  of  scoring,  categoriza&on,  and  coding  was  assured.      

10 of 22

Reflecting: Use U&lizing  

U-lizing  assessment     Not  at  all  

A  liBle  

Some-­‐what  

Very  much    

N/A  

The  primary  intended  users  were  ac&vely  involved  in  making  judgments  based  on  assessment  findings.    The  primary  intended  users  were  ac&vely  involved  in  genera&ng  recommenda&ons.      Nega&ve  findings  were  turned  into  construc&ve  feedback.    Recommenda&ons  considered  contextual  constraints.    Reports  were  delivered  in  a  &mely  manner  to  be  u&lized  for  decision-­‐making.        Repor&ng  was  clear  and  understandable  to  the  audience(s).    The  format  of  repor&ng  was  appropriate  and  maximized  findings  use.    Ac&on-­‐planning  was  feasible  to  those  who  are  involved  in  improving  and  developing  the  program.      Overall,  assessment  findings  were  actually  used  as  intended.    

11 of 22

Reflecting: Use U&lizing  

           Consider whether there were any unintended uses or even misuses of assessment. If so, how would you avoid these and their potential negative consequences next time?

Unintended uses? ��� Misuses?   Strategies for avoiding unintended uses and negative consequences?    

12 of 22

Conclusion

The rest of the slides provide additional useful resources on outcomes assessment. Module V suggested a list of questions to examine during and at the end of the assessment cycle to improve future assessment practices. Examples of good and reflective assessment practices are much needed in the field of foreign language education, and we hope you will consider documenting and disseminating what worked and what didn’t work in your context. ���

Reflecting: Useful assessment Reflec&ng  

13 of 22

We welcome any feedback from you on our Outcomes Assessment Modules. Please leave us some feedback about this module by accessing a

short online questionnaire here: ���[INSERT URL]

14 of 22

Look for examples: Duke University, Evansville University,

Georgetown University

Key Resources

15 of 22

•  Brindley, G. (2001). Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing, 18(4), 393-407.

•  Liskin-Gasparro, J. E. (1995). Practical approaches to outcomes assessment: The undergraduate major in foreign languages and literatures. ADFL Bulletin, 26(2), 21-27.

•  Brown, J. D., & Hudson, T. (1998). The alternatives in language assessment. TESOL Quarterly, 32(4), 653-675.

•  Byrnes, H. (2002). The role of task and task-based assessment in a content-oriented collegiate foreign language curriculum. Language Testing, 19(4), 419-437.

•  Dassier, J., & Powell, W. (2001). Formative FL program evaluation: Dare to find out how good you really are. In Dimension 2001: The odyssey continues. Selected proceedings of the 2001 conference on language teaching (pp. 15–30). Birmingham, AL: Southeast Council on Language Teaching.

Resources: SLO assessment in language education

16 of 22

•  Delett, J. S., Barnhardt, S., & Kevorkian, J. A. (2001). A framework for portfolio assessment in the foreign language classroom. Foreign Language Annals, 34(6), 559-568.

•  Mathews, T., J., & Hansen, C., M. (2004). Ongoing assessment of a university foreign language program. Foreign Language Annals, 37(4), 630-640.

•  Morris, M. (2006). Addressing the challenges of program evaluation: One department’s experience after two years. Modern Language Journal, 90(4), 599-602.

•  Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597.

•  Norris, J. M., & Pfeiffer, P. (2003). Exploring the use and usefulness of ACTFL Guidelines oral proficiency ratings in college foreign language departments. Foreign Language Annals, 36(4), 572-581.

•  Phillips, J. (2006). Assessment now and into the future. In A. Heining-Boynton (Ed.), 2005-2015: Realizing our vision of languages for all (pp. 75-103). Upper Saddle River, NJ: Pearson.

Resources: SLO assessment in language education

17 of 22

•  Allen, M. J. (2004). Assessing academic programs in higher education. Bolton, MA: Anker.

•  Bresciani, M. J., Zelna, C. L., & Anderson, J. A. (2004). Techniques for assessing student learning and development: A handbook for practitioners. Washington, DC: NASPA.

•  Heiland, D., & Laura, J. R. (Eds.). Literary study, measurement, and the sublime: Disciplinary assessment. New York: The Teagle Foundation.

•  Hernon, P., & Dugan, R. E. (eds.) (2004). Outcomes assessment in higher education: Views and perspectives. Westport, CT: Libraries Unlimited.

•  Knight, P. (Ed.). (1995). Assessment for learning in higher education. London: Kogan Page.

•  Maki, P. L. (2004). Assessing for learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.

•  Suskie, L. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco, CA: Jossey-Bass.

•  Walvoord, B. A. (2004). Assessment clear and simple: A practical guide for institutions, departments and general education. San Francisco, CA: Jossey-Bass.

•  Wiggins, G. (1998). Educative assessment: Designing assessments to inform and improve student performance. San Francisco: Jossey-Bass.

Resources: SLO Assessment in tertiary education

18 of 22

References •  American Association for Higher Education (1992). Nine principles of good practice

for assessing student learning. Sterling, VA: Stylus. •  Angelo, T. A. (1995). Reassessing (and Defining) Assessment. American Association

for Higher Education (AAHE) Bulletin, 48(2), 7-9. •  Banta, T. W. (2002). Characteristics of effective outcomes assessment. In T. W. Banta

(Ed.), Building a scholarship of assessment (pp. 261-283). San Francisco, CA: Jossey-Bass.

•  Grau Sempere, A., Mohn, M. C., & Pieroni, R. (2009). Improving educational effectiveness and promoting internal and external information-sharing through student learning outcomes assessment. In J. M. Norris, J. McE. Davis, C. Sinicrope, & Y. Watanabe (eds.), Toward useful program evaluation in college foreign language education (pp. 139-162). Honolulu: University of Hawaii, National Foreign Language Resource Center.

•  Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.). Newbury Park, CA: Sage.

19 of 22

References •  Mackay, R., Wellesley, S., Tasman, D., & Bazergan, E. (1998). Using institutional self-

evaluation to promote the quality of language and communication training programmes. In P. Rea-Dickens & K. P. Germaine (Eds.), Managing evaluation and innovation in language teaching: Building bridges (pp. 111-131). London: Longman. 

•  Norris, J. M. (2006). The why (and how) of student learning outcomes assessment in college FL education. Modern Language Journal, 90(4), 590-597.

•  Patton, M. Q. (2008). Utilization-focused evaluation (4th ed). Thousand Oaks, CA: Sage. •  Patton, M. Q. (2012). Essentials of utilization-focused evaluation. Thousand Oaks, CA:

Sage. •  Stufflebeam, D. L. (1999). Program evaluations metaevaluation checklist. Retrieved

from http://www.wmich.edu/evalctr/archive_checklists/program_metaeval_10point.pdf

•  Suskie, L. (2004). Assessing student learning. Bolton, MA: Anker.

20 of 22

References •  Walther, I. (2009). Developing and implementing an evaluation of the foreign

language requirement at Duke University. In J. M. Norris, J. McE. Davis, C. Sinicrope, & Y. Watanabe (eds.), Toward useful program evaluation in college foreign language education (pp. 117-138). Honolulu, HI: University of Hawaii, National Foreign Language Resource Center.

•  Watanabe, Y., Norris, J. M., & González-Lloret, M. (2009). Identifying and responding to evaluation needs in college foreign language programs. In J. M. Norris, J. McE. Davis, C. Sinicrope, & Y. Watanabe (Eds.), Toward useful program evaluation in college foreign language education (pp. 5–56). Honolulu: University of Hawai‘i, National Foreign Language Resource Center.

•  Wright, B. D. (2006). Learning languages and the language of learning. Modern Language Journal, 90(4), 593-596.

21 of 22

http://www.nflrc.hawaii.edu/evaluation

Visit our website

22 of 22