MESA Department, Boston College 2019 AERA

MESA at

AERA and NCME

TORONTO, CANADA

AERA: Friday, April 5 - Tuesday, April 9

NCME: Thursday, April 4 - Monday, April 8

MESA Department, Boston College 2019 AERA

Table of Contents

MESA at AERA & NCME

MESA Information p. 2 Program Overview

MESA Faculty Research Interests Research

Conference Information p. 6 Overview of AERA Theme

Overview of NCME Theme

Lynch School AERA Reception

MESA Participants p. 9 Overview

Presenters by Name and Schedule Pre-Conference Workshops

Exploring Toronto p. 15 Toronto Welcomes You

Toronto Transportation

Dining in Toronto

Thank you to undergraduate researcher Kathlyn Rooney for compiling this booklet!

MESA Department, Boston College 2019 AERA

Program Overview

MESA at Boston College

Drawing on the Lynch School aspiration to “enhance the human condition, to expand the human imagination, and to make the world more just":

The mission of the MESA program is to …

Contribute to national and international efforts to solve critical educational and human problems in a diverse global community;

Educate the next generation of scholars and practitioners who will engage in reflective so-cial inquiry;

Foster meaningful scholarly inquiry into both theoretical and applied aspects of social sys-tems;

Advance creative approaches to important measurement, evaluation, and research prob-lems in educational and other social science fields;

Provide students with the necessary theoretical and applied skills and experience to be-come research and evaluation leaders in their professional fields; and

Prepare students to make original and substantive contributions to the fields of measure-ment, evaluation, and statistics.

In service of this mission …

The MESA program is committed to high quality, rigorous, collaborative, and equity-minded research, teaching, and mentoring.

The MESA program trains students in state-of-the-art statistical, research, and evalua-tion methods.

Students in MESA develop expertise in research methodology, measurement, evalua-tion, and applied statistics.

MESA faculty members have expertise in classroom and large-scale assessment, measurement and item response theory, factor analysis and structural equation modeling, categorical data analysis, multilevel modeling, generalized linear models, program evalua-tion, psychometrics, etc.

Graduates of the MESA program find employment in colleges, universities, research centers, testing agencies, school systems, ministries of education and other government agencies, and industry.

Students completing the doctoral/master's degrees are capable of conducting inde-pendent research and evaluation, constructing measurement instruments, analyzing and interpreting quantitative and qualitative data, and contributing to the theory of research and evaluation methodology.

MESA Department, Boston College 2019 AERA

Faculty Research Interest

MESA at Boston College

Testing and education

policy; large-scale

assessment surveys;

achievement gaps; value-

added modeling;

standard setting; higher

education outcomes.

Measurement; psychometrics;

statistics; research design;

video analysis, interaction

analysis; cognitive ethnography;

embedded classroom

assessment for learning; large-

scale standardized assessment

of learning.

Henry Braun, Ph.D.

Nathaniel Brown, Ph.D.

Evaluation theory and practice;

mixed methods research and

evaluation; educational, health,

and social programs; STEM

teaching and learning; health

education; systems thinking &

complexity science; values &

valuing in evaluation.

Emily Gates, Ph.D.

Psychometrics and

educational research

methods, including

categorical data analysis,

latent variable modeling, and

educational measurement

issues.

Zhushan “Mandy” Li, Ph.D.

Larry Ludlow, Ph.D.

Scenario based scale

development; student

ratings of instruction;

longitudinal social network

change

Ina V. S. Mullis, Ph.D.

Large-scale assessment methods

in international, national, and

state contexts; innovative

approaches to measuring student

skills and understandings;

evaluating progress in

performance, and analyzing

achievement data; policy uses of

assessment information to

increase student learning.

Laura O’Dwyer, Ph.D.

Examining the impacts of

technology-based

interventions;

experimental design;

hierarchical linear

modeling and power

analysis; international

comparative studies.

Innovative uses of computer-

based technologies and

applications of Universal Design

to enhance educational testing

and assessment; large-scale

assessment and test design;

computer-based testing;

accessible portal item protocol

(APIP) Standards and assessment

interoperability standards. Michael Russell, Ph.D.

MESA Department, Boston College 2019 AERA

Research

MESA at Boston College

The TIMSS and PIRLS International Study Center

Located at Boston College’s Lynch School of Education, IEA’s TIMSS & PIRLS International Study

Center conducts regular international comparative assessments of student achievement in

mathematics and science (TIMSS) and in reading (PIRLS) in more than 60 countries. TIMSS (the

Trends in International Mathematics and Science Study) and PIRLS (the Progress in International

Reading Literacy Study) together comprise the core cycle of studies for IEA – the International

Association for the Evaluation of Educational Achievement. Headquartered in Amsterdam and

with a major data processing and research center in Hamburg, IEA has been conducting

international comparative studies of student achievement since 1959.

TIMSS and PIRLS enable participating countries to make evidence-based decisions for improving

educational policy. Some of the ways governments and ministries use TIMSS and PIRLS results

include:

Measuring the effectiveness of their educational systems in a global context

Identifying gaps in learning resources and opportunities

Pinpointing any areas of weakness and stimulating curriculum reform

Measuring the impact of new educational initiatives

Training researchers and teachers in assessment and evaluation

TIMSS and PIRLS also collect extensive data about the contextual factors that affect learning,

including school resources, student attitudes, instructional practices, and support at home. This

information can be examined in relation to achievement to explore factors that contribute to

academic success.

The Center for the Study of Testing, Evaluation, and Educational Policy (CSTEEP)

CSTEEP conducts its work on both small and large scales, working with individual schools,

districts, states, as well as countries to advance educational testing practices and policy, and to

improve the quality and fairness of education.

MESA Department, Boston College 2019 AERA

Conference Theme

AERA Conference Information

Leveraging Education Research in a “Post-

Truth” Era: Multimodal Narratives to

Democratize Evidence

From the meeting website:

The 2019 AERA Annual Meeting, based in the vibrant

multicultural city of Toronto and in the country of Canada,

known more for political cooperation than partisan squabbles, is

an opportunity to assess the state of education research. It is also

an opportunity to explore how our work can help overcome the

challenges of our time by becoming more relevant to

communities, practitioners, and policy makers who believe in

democratic principles and the public schools that should sustain

those principles.

Find out how to get the

mobile app with all

information on the AERA

Conference here!

MESA Department, Boston College 2019 AERA

Theme

NCME Conference Information

The theme of the 2019 NCME conference is “Communicating with the

Public about Educational Measurement.” In the context of this theme, the

“public” is meant to include many possible audiences—voters, parents,

educators, policymakers, journalists, and students of all ages. Ways of

communicating directly with students are of particular interest.

The “communications” theme encompasses two strands. One is technical in nature: We need to continue to work on developing creative and effective ways to communicate test results in a comprehensible way. This includes communicating clearly about measurement error—an ongoing challenge. Improvements in both verbal and visual means of communication, possibly including dynamic displays, are included here.

Another facet of communicating with the public involves engaging in a straightforward and honest way about the role of testing in society, recognizing that it has not always been a positive one. Acknowledging the pluses and minuses of testing can lead us to produce better, fairer tests, while increasing our credibility in the eyes of the public.

In recognition of this theme, the program committee will seek a broad range of proposals for inclusion on the 2019 conference program, in particular those concerned with:

innovative approaches for communicating test results verbally or visually, dynamic displays for communicating test results, communicating about the pros and cons of testing, communicating test results directly to students.

MESA Department, Boston College 2019 AERA

Saturday, April 6th 6:00-8:00pm

Lynch School AERA Reception 2019

The MESA network is invited to join Dean Stanton E. F. Wortham and Boston College Lynch School of Education and

Human Development for a reception at the 2019 Annual Meeting of the American Educational Research Association.

Register here!

MESA Department, Boston College 2019 AERA

Overview

MESA Participants

Current MESA Students Ella Anghel

Katrina Borowiec

Jing Jiang

Michael Kelly

Kelsey Klein

Sebastian Moncaleano

Katherine Reynolds

Amy Semerjian

Olivia Szendey

MESA Faculty Dr. Henry Braun

Dr. Zhushan Li

Dr. Larry Ludlow

Dr. Mike Russell

Alumni Dr. Lisa Abrams

Dr. Jessica Bailey

Dr. Mac Cannady

Dr. Courtney Castle

Dr. Wen-Chia Claire Chang

Dr. Ebru Erberber

Dr. Martin Hooper

Dr. Michael O’Leary

Dr. Todd Reeves

Dr. Joshua Littenberg-Tobias

Jessica Triant

Dr. Yang Caroline Wang

MESA Department, Boston College 2019 AERA

Presenters by Name and Schedule

MESA Participants

Conference Program Website

For more detailed information on sessions, view the Excel Sheet format here.

Friday, April 5 12:00-1:30pm

Katrina Borowiec: Initial Validity and Reliability for the Sense of Belonging in College Scale (Fairmont Royal York Hotel, Convention Floor, Concert Hall)

4:20-5:50pm

Dr. Mac Cannady: Fostering Environmental Activism through Community-Based Research Investigations (Metro Toronto Convention Centre, 300 Level, Hall C)

Saturday, April 6 8:00-10:00am

Dr. Henry Braun: Presenter—Robert Linn Distinguished Award Address. Data in the Social Sciences: It’s Time for Some Respect (Fairmont Royal York Hotel, Mezzanine Level, Confederation 5)

10:25-11:55am

Dr. Joshua Littenberg-Tobias: Designing Professional Learning MOOCs for Teacher Connected Learning (Metro Toronto Convention Centre, 300 Level, Hall C)

Katherine Reynolds: Student Perceptions of Faculty Availability: A Facet Theory Approach (Metro Toronto Convention Centre, 200 Level, Room 205A)

Kelsey Klein: Revising a Measure of High School Students’ Mathematics Anxiety (Fairmont Royal York Hotel, Convention Floor, Concert Hall)

Dr. Mac Cannady: Young Adult Learners: Meeting Expectations and Finding Value (Metro Toronto Convention Centre, 700 Level, Room 707)

12:20-1:50pm

Dr. Mike Russell: Digital Technologies Supporting and Advancing Assessment Practices in the Classroom (Fairmont Royal York Hotel, Mezzanine Level, Alberta)

MESA Department, Boston College 2019 AERA

Presenters by Name and Schedule

MESA Participants

2:15-3:45pm

Amy Semerjian: Inventing and Affect: Impact of an In-School Time Invention Project with a Targeted Measurement Suite (Metro Toronto Convention Centre, 700 Level, Room 707)

Michael Kelly: Integrated Student Support and English Language Learners: A Multisite Case Study(Metro Toronto Convention Centre, 200 Level, Room 205C)

Dr. Wen-Chia Claire Chang, Dr. Larry Ludlow: Measuring the Complexity of Teaching Practice for Equity: Results and Utility of the Scenario-Format Scale (Metro Toronto Convention Centre, 700 Level, Room 713B)

4:10-5:40pm

Dr. Mac Cannady: Clean Energy Literacy and Leadership: Engaging Youth in or After School (Metro Toronto Convention Centre, 300 Level, Hall C)

4:10-6:10pm

Dr. Jessica Bailey: Chair—Conducting Studies of Teacher Mobility and Retention in Partnership with State and Local Education (Fairmont Royal York Hotel, Mezzanine Level, Tutor 7)

Sunday, April 7 8:00-9:30am

Dr. Larry Ludlow, Dr. Henry Braun, Ella Anghel, Olivia Szendey: The BC-LAMP Portfolio Project (Metro Toronto Convention Centre, 700 Level, Room 717A)

9:55-11:25am

Amy Semerjian: A Visual Measure of Science Identity: Instrument Development Pilot; The Proximal Item: A New, DSL-Inspired Item Type, Contextualized within a “Technologically Enhanced Continuum” (Fairmont Royal York Hotel, Mezzanine Level, Confederation 3)

11:50-1:20pm

Jessica Triant: An Examination of Relationships Between Factors that Support STEM Pathways at Selective STEM High Schools (Metro Toronto Convention Centre, 800 Level, Hall F)

Dr. Yang Caroline Wang: Growth Mind-Set Versus Not a Fixed Mind-Set: Comparing Positively and Negatively Worded Survey Items (Metro Toronto Convention Centre, 200 Level, Room 206C)

MESA Department, Boston College 2019 AERA

Presenters by Name and Schedule

MESA Participants

3:20-4:50pm

Jing Jiang, Dr. Zhushan Mandy Li: Detecting Differential Item Functioning Using the Adaptive LASSO Penalty (Fairmont Royal York Hotel, Mezzanine Level, Manitoba)

3:40-5:10pm

Amy Semerjian: Opportunity to Learn Analysis Using an Engineering-Hydroponics Assessment in Urban, Suburban, and Suburban/Rural Schools (Sheraton Centre Toronto Hotel, Mezzanine Level, Willow West)

5:05-6:35pm

Dr. Yang Caroline Wang: Evaluating the Differentiation of Social-Emotional Learning Constructs Using Multilevel Factor Analysis (Fairmont Royal York Hotel, Mezzanine Level, Territories)

5:50-6:35pm

Dr. Mike Russell: Discussant—New Insights on Engagement, Learning, and Performance (Fairmont Royal York Hotel, Convention Level, Salon A)

Monday, April 8 8:00-10:00am

Dr. Henry Braun: What the Data Can Support: Avoiding Pitfalls in Interpreting International Assessment Results (Fairmont Royal York Hotel, Mezzanine Level, Alberta)

10:25-11:55am

Amy Semerjian: Investigating the Impact of Productive Failure Activities on Student Engineering Design (Sheraton Centre Toronto Hotel, Lower Concourse, Osgoode Ballroom)

12:20-1:50pm

Dr. Lisa Abrams, Dr. Michael O’Leary: The Context and Use of Standardized Testing Data for Educational Decision Making in Ireland (Fairmont Royal York Hotel, Mezzanine Level, Confederation 6)

Dr. Todd Reeves: More Than Just Test Scores: Teacher Use of Nonacademic Data (Fairmont Royal York Hotel, Mezzanine Level, Confederation 6)

MESA Department, Boston College 2019 AERA

Presenters by Name and Schedule

MESA Participants

2:15-3:45pm

Sebastian Moncaleano: Determining the Value of Techonology-Enhanced Items for Educational Assessments (Metro Toronto Convention Centre, 300 Level, Hall C)

Dr. Joshua Littenberg-Tobias: Designing Professional Learning MOOCs for Teacher Connected Learning (Sheraton Centre Toronto Hotel, Lower Concourse, Sheraton Hall E)

4:10-5:40pm

Dr. Joshua Littenberg-Tobias: Teachers’ Challenges in Learning and Implementing Design Thinking in a MOOC (Metro Toronto Convention Centre, 800 Level, Hall G)

4:10-6:10pm

Dr. Courtney Castle, Katrina Borowiec: Using Rater Cognition to Improve Generalizability of an Assessment of Scientific Argumentation (Fairmont Royal York Hotel, Mezzanine Level, Alberta)

Dr. Mike Russell, Sebastian Moncaleano: Examining Current Practice Regarding Technology-Enhanced Items in K-12 Testing Programs (Fairmont Royal York Hotel, Mezzanine Level, Territories)

Tuesday, April 9 10:25-11:55pm

Dr. Courtney Castle: Scoring and Dimensionality in Next Generation Science Standards Assessment (Fairmont Royal York Hotel, Mezzanine Level, Confederation 3)

Katrina Borowiec: Students’ Interactions with Faculty and Academic Achievement (Metro Toronto Convention Centre, 200 Level, Room 202A)

2:15-3:45pm

Dr. Lisa Abrams: Teacher Development in an Urban Residency Program: A Comparison of Two Cases (Metro Toronto Convention Centre, 800 Level, Hall F)

MESA Department, Boston College 2019 AERA

Pre-Conference Professional Development Courses

Additional Workshops

You must be registered for the Annual Meeting in order to purchase tickets to attend

courses. Potential participants can register for professional development and training courses by

logging into and updating their 2019 AERA Annual Meeting registration.

PDC01: Analyzing Data From International Large-Scale Assessments Using R

Instructors include: Dr. Ebru Erberber, American Institutes for Research (MESA PhD 2009)

Date: Thursday, April 4

Time: 9 a.m. – 5 p.m.

Fee: $135

This course will introduce participants to the procedures of analyzing data from international large-scale assessments

(ILSAs) using NCES’s EdSurvey R Package. Public-use TIMSS data files will be used as example data sets. Participants will

begin installing the EdSurvey R package and importing the data files into R. They will learn how to manipulate the data,

including merging, subsetting, and recoding data. The participants will learn how to use the EdSurvey package to perform

the statistical techniques used most often in ILSAs data analyses, including selecting an appropriate sample, estimating the

mean scale scores for groups of students, benchmark analysis, gap analysis, linear regression, logistic regression, and

correlations. This course will introduce unique design features of ILSAs data to researchers and provide guidance in data

analysis strategies that they require, including the selection and use of appropriate plausible values, sampling weights, and

variance estimation procedures. There will be designated time for participants using EdSurvey to practice the techniques

with the variables of their own interest.

PDC10: Multilevel Modeling With Large-Scale International Databases Using the

HLM Software Program

Instructors include: Dr. Martin Hooper, American Institutes of Research (MESA PhD 2017)

Date: Thursday, April 4

Time: 9 a.m. – 5 p.m.

Fee: $135

Data from international large-scale assessments (ILSAs) reflect the nested structure of education systems and is very well

suited for multilevel modeling (MLM). However, because these data come from complex cluster samples, there are

methodological aspects that a researcher needs to understand when doing MLM, such as the need for using sampling

weights and multiple achievement values for parameter estimation. This course will teach participants how to do MLM

with data from ILSAs, such as PIRLS, TIMSS, and PISA. The content of the course will include an overview of the ILSAs and a

presentation on the design of these studies and databases and implications for MLM analysis. Participants will learn how to

specify two-level models using the HLM software program and also learn about model comparison, centering decisions and

their consequences, and available resources for doing three-level models. Time will be allotted for participants to work on

practice exercises, with several instructors available to mentor and answer questions. Participants should have a solid

understanding of OLS regression and a basic understanding of MLM. Prior experience using a statistical software program,

such as Stata or SPSS, is helpful. Prior knowledge about ILSAs or prior experience using the respective databases or HLM

software is not required. To fully participate in the hands-on demonstrations and example analyses, participants should

bring their own laptops with HLM software (a free student version is available).

MESA Department, Boston College 2019 AERA

Toronto Welcomes You

Exploring Toronto

Average Temperatures in April:

High: 54°F

Low: 39°F

Toronto provides something for everyone! Go to the top of the CN Tower

to take in the view, check out the Kensington Market neighborhood for

shopping and food, or take advantage of the museums in the area, such

as the Art Gallery of Ontario or the Hockey Hall of Fame.

MESA Department, Boston College 2019 AERA

Toronto Transportation

Exploring Toronto

Toronto uses a variety of streetcars, buses, and subways as means for

transportation.

Street Car Information

Bus Information Subway Information

* There are many recommendations to take the train from the

airport towards Downtown, as this is cheaper, faster, and the

easiest method.

MESA Department, Boston College 2019 AERA

Quick Eats

Exploring Toronto

Ravi Soups— get a wrap

combo! Quick and easy.

Hidden Burger—

Cheeseburger combo for $10

Flock Rotisserie + Greens—

paleo, gluten free, vegan

and dairy free options/ SOMA Chocolatemaker—

everything is made from

scratch!

Dr. Alan Amtzis (The College of New Jersey), a friend of Dr.

Ludlow provided us with a list of his restaurant suggestions. He

notes that " these are all based on various sources (friends, on-

line websites, guidebooks, etc) but have not been experienced

first-hand by me...." Please click here to view more suggestions.