survey validity and reliability study · pdf filevalidity%and%reliability%of ......

Page | 1

Technical Report: Examination of the Psychometric Properties of the

Catholic School Program Effectiveness and Defining Characteristics Surveys

Prepared for: National Task Force, National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools

Center for Catholic School Effectiveness, Loyola University Chicago

Barbara and Patrick Roche Center for Catholic Education, Boston College

AdvancED

Author: Scott R. Weaver, Ph.D. AdvancED Consultant

July 31, 2012

Acknowledgments: Special thanks are extended to Dr. Eddie Krenson, Dr. Vicki Denmark, and Karmen Gary at AdvancED, for their assistance with the coordination of multiple elements of this project. The surveys examined in this psychometric study were developed by the following national task force with support from AdvancED: Lorraine A. Ozar, Ph.D, Loyola University Chicago (Chair); Susan Ferguson, Ed.D., University of Dayton; Adam Kruekeberg, MBA/MA Pastoral Ministry, Boston College; Kathleen Schwartz, Ed.D., Diocese of Venice, FL; Patricia Weitzel-‐O’Neill, Ph.D., Boston College.

Page | 2

Table of Contents

EXECUTIVE SUMMARY...............................................................................................................................................................4 SAMPLING AND METHODOLOGY ...............................................................................................................................................6

SAMPLING OVERVIEW ...........................................................................................................................................................................6 METHODOLOGY ...................................................................................................................................................................................6

PROGRAM EFFECTIVENESS SURVEYS........................................................................................................................................ 10

DESCRIPTION .....................................................................................................................................................................................10 ADULT SURVEY ..................................................................................................................................................................................10

Respondent Characteristics .......................................................................................................................................................10 Response Pattern Summaries....................................................................................................................................................11 Factorial Validity and Reliability................................................................................................................................................12 Summary and Recommendations .............................................................................................................................................16

STUDENT SURVEY (GRADES 5-‐8) ...........................................................................................................................................................19


STUDENT SURVEY (GRADES 9-‐12) .........................................................................................................................................................28


DEFINING CHARACTERISTICS SURVEYS..................................................................................................................................... 36

DESCRIPTION .....................................................................................................................................................................................36 ADULT SURVEY ..................................................................................................................................................................................36


STUDENT SURVEY (GRADES 5-‐8) ...........................................................................................................................................................43


STUDENT SURVEY (GRADES 9-‐12) .........................................................................................................................................................49

Page | 3


APPENDIX................................................................................................................................................................................ 54

APPENDIX A: TABLES FROM THE ITEM-‐LEVEL ANALYSES OF THE CATHOLIC SCHOOL – PROGRAM EFFECTIVENESS SURVEYS.......................................54 APPENDIX B: TABLES FROM THE ITEM-‐LEVEL ANALYSES OF THE CATHOLIC SCHOOL – DEFINING CHARACTERISTICS SURVEYS ....................................75

Page | 4

Executive Summary Arising from a widely recognized need by the community of Catholic educators for “a common framework of universal characteristics of Catholic identity and agreed upon criteria for Catholic school excellence,” the National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools was published in 2012.1 This publication delineates a set of Defining Characteristics of Catholic schools: (a) Centered in the Person of Jesus Christ, (b) Contributing to the Evangelizing Mission of the Church, (c) Distinguished by Excellence, (d) Committed to Educate the Whole Child, (e) Steeped in a Catholic Worldview, (f) Sustained by Gospel Witness, (g) Shaped by Communion and Community, (h) Accessible to All Students, and (j) Established by the Expressed Authority of the Bishop. These Defining Characteristics provide the foundation for a set of four Standards and Benchmarks for Catholic schools. The Standards articulate policies, programs, structures, and processes in four domains – Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality – that “should be present in mission-‐driven, program effective, well-‐managed, and responsibly governed Catholic schools….” Subsequently, the task force that authored the National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools document developed a set of survey instruments to measures schools’ alignment with the Defining Characteristics and the Standards for effective Catholic schools through the perceptions of school-‐affiliated adults and students (5th – 12th grades) and, in collaboration with AdvancED, designed a psychometric study to gauge the validity and reliability of the survey instruments. The focus of this technical report is on that psychometric study and analyses of the newly developed survey instruments. Participating were nearly 8,000 adults and students from 55 Catholic primary and secondary schools. Multilevel, ordinal factor analyses were conducted to examine the validity and reliability of the survey instruments. These analyses sought to elucidate the number and pattern of factors being measured by the surveys, including the congruence of the factor structure with the guiding theoretical framework; the validity and reliability of its items; and the reliability and distribution of scale and subscale composite scores. Based on the results of this study, each survey reliably captured a single factor or dimension reflecting the Defining Characteristics or Standards for effective Catholic schools. This was perhaps contrary to expectation for the Catholic School Program Effectiveness Surveys (Standards), for which four distinct factors relating to the four domains articulated in the Standards (viz., Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality) were expected. Most items were supported by the analyses as valid and reliable indicators of their respective factors, and reliability estimates of a composite (total) score and most subscale scores computed by averaging over respondents

1 Center for Catholic School Effectiveness, School of Education, Loyola University Chicago, in partnership with the Barbara and Patrick Roche Center for Catholic Education, School of Education, Boston College. (2012). National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools. http://www.catholicschoolstandards.org/

Page | 5

and items within the same school indicated exceptionally high reliability.2 The one exception was the Operational Vitality subscale on the Catholic School Program Effectiveness Survey of Students (5th – 8th grades); the reliability for the scale was marginal. Although nearly all items were found to be reliable indicators of their respective factors, all surveys had at least one item with missing data rates, due to either Don’t Know responses or being skipped, exceeding 10% and some approaching or exceeding 40%. These items can be problematic as high rates of missing data negatively affect the internal validity and generalizability of the survey measurements. The detailed technical report that follows elaborates on the sampling and statistical methodology and the psychometric analyses of these surveys. The results from the psychometric analyses are described in multiple sections, each with parallel, self-‐contained subsections, pertaining to each of the six surveys. These sections are concluded with a summary and recommendations for scale interpretation and possible revisions to the surveys.

2 Due to a limited sample of high schools, the 9th – 12th grade surveys were examined only at the respondent (student) level, which is not the intended level of measurement for these surveys. As such, inferences about the validity and reliability of the 9th – 12th grade surveys drawn from this study are limited.

Page | 6

Sampling and Methodology Sampling Overview

During January through March of 2012, a sample of 200 Catholic schools was selected to receive an invitation to participate in the pilot study by administering a battery of surveys to the students (5th – 12th grades) and/or adults (school-‐based and non-‐school-‐based) affiliated with their schools. Of the 200 schools invited, 55 schools agreed to participate and had at least one respondent for at least one of the surveys. Eligible respondents (all adults and students between 5th and 12th grades affiliated with the participating schools) were invited to complete the Catholic School Program Effectiveness Survey and/or Catholic School Defining Characteristics Survey. Separate versions of each survey were administered to adults, 5th – 8th grade students, and 9th – 12th grade students. Characteristics of the samples differed across the six surveys, and so individual respondent characteristics are provided in the survey-‐specific sections found later in this report. Survey administration was conducted through a web-‐interface using Survey Monkey, and participation was anonymous.

Methodology

Analyses began with a simple examination of item statistics at the respondent (adult/student) and school level.3 These analyses were focused on describing the distribution of responses for each item, with an eye toward the variability in responses across schools and across items. Inter-‐item variability is desirable for reliable measurement of the Standards for Program Effectiveness and Defining Characteristics of Catholic Schools across the range of scores on the respective factors. If most or all items are endorsed with high agreement responses, the items collectively might discriminate reliably amongst schools scoring high on these factors, but not reliably for schools intermediate or relatively weak on these factors. To evaluate the construct validity of these scales, factor analytic techniques were employed to evaluate their factorial validity and reliability. Factor analysis seeks to ascertain the underlying (i.e., latent, unobserved) structure of the measurement instrument (e.g., survey) and is an important prerequisite before other components of construct validity or reliability estimation are conducted. Examples of the questions that factor analysis can address include:

• How well does the underlying factor dimensionality and structure align with the theory that guided the development of the surveys?

Are there four dimensions corresponding to the four Standards for Program Effectiveness of Catholic Schools? Is there a single dimension capturing the Defining Characteristics for Catholic Schools? If not, how many dimensions are measured by the surveys?

Which items reliably measure which dimension(s)? • Are the identified factors reliably measured by the indicators (i.e., items)? Are items all valid indicators

of the underlying construct(s)?

3 Due to a limited sample of schools, school-‐level analysis was not possible for the 9th-‐12th grade surveys.

Page | 7

Which items, if any, need to be discarded or revised due to poor validity or reliability or due to measuring more than one dimension (item complexity)?

Do some items convey redundant information about the underlying construct and thus can be discarded without loss of information (reliability) to ease response burden?

Adult and students responses to the surveys were subjected to a multilevel, confirmatory factor analysis (CFA) to ascertain the degree of statistical fit between the data and models specified in accord with the theoretical intent of the survey's design.4 As it was considered a possibility that neither the four-‐factor model nor the one-‐factor model would accurately depict the factor structure of the surveys, an exploratory factor analysis (EFA) was also performed. EFA allows the scale developer to ascertain the degree of statistical fit between the observed data and a model with k factors, where the range of k examined depended on the eigenvalues from the reduced correlation matrix. Interpretability of the solution and statistical tests and indices of model fit were used to settle on a particular k factor solution. Exploratory factor analytical methods impose minimal a priori constraints (hypotheses/predictions on the model), beyond those required to statistical identify the estimated parameters of the model, and are aimed at building a model of the underlying factor structure in the absence of information about the structure. This analysis employed the Geomin rotation, which allows factors to correlate in models where k > 1. All factor analyses were conducted using the Mplus statistical software program (v. 6.12)5 and employed a mean-‐ and variance-‐adjusted weighted least squares estimator with numerical integration. Model fit tests and indices used consist of the chi-‐square test of exact fit, the Comparative Fit Index (CFI), the root mean square error of approximation (RMSEA), and standardized root mean residual (SRMR – school-‐level model). Although statisticians continue to debate the appropriate focus and thresholds on these tests and fit indices, most consider a statistically non-‐significant chi-‐square test of exact fit (reflected by probability values greater than .05), RMSEA values less than .06, CFI values greater than .96, and SRMR values below .06 to reflect good or adequate model fit to the data. In other words, a model with k factors that meets these criteria is said to be consistent with the data and therefore accepted for further consideration. Conversely, a model that fails all criteria is said to be rejected by the data. In practice, models often meet only some of the criteria, while being near but just outside the thresholds for the acceptable range on other criteria. In these cases, the theoretical interpretability of the results dominates. When two or more models exhibit similar fit, the model that is most interpretable and parsimonious is usually retained. For models that fit the data well, item-‐level statistics are examined to evaluate the validity and reliability of individual items. These analyses account for two important characteristics of the data: (1) response data are collected using a Likert scale and thus may not possess interval scale properties, and (2) responses to the surveys are nested within schools and thus are not independently distributed (i.e., since respondents affiliated to the same school are reporting on the same school, they are more correlated than with responses from respondents affiliated with different schools), an assumption of standard factor analytic methods. With respect to (1), common analytical methods (Pearson-‐product moment correlation analysis, linear factor analysis) often employed with survey data assume that the scale of measurement for the data are interval-‐

4 Due to a limited sample of schools, a single-‐level analysis at the respondent-‐level was conducted for the 9th-‐12th grade surveys. 5 Muthén & Muthén, 2011, http://www.statmodel.com

Page | 8

level (differences in adjacent response options reflect equal discriminations on the underlying agreement scale used by the respondent). As such, inappropriate use of these linear methods with binary or ordinal data can lead to statistical artifacts and biased results, particularly when the number of response options is fewer than seven and/or the response distribution is highly skewed (e.g., strong floor or ceiling effects). To address this limitation of conventional, linear factor analytic approaches, categorical confirmatory factor analysis methods6 were employed. Instead of modeling the variances and covariances (and means) of the response data as a linear function of the latent constructs given the model parameters: ,

one parameterization for ordinal factor analysis using a probit link function models the item-‐response probability distribution as a non-‐linear function of the latent constructs (given the model parameters) via a latent response distribution. In this model, latent response variates, y*, are conceptualized to reflect a latent, interval-‐scaled variable underlying each item, y, such that , where c

represents the categories (e.g., never, seldom, …) of y and are the estimated threshold parameters for the probit function linking y* with y. Based on the assumption of bivariate normality of y* (and other regularity assumptions) the estimated variance-‐covariance matrix of y* is modeled as a function of the

underlying latent factors given the estimated and constrained model parameters: .

With regards to (2), the ordinal factor analytic models described above were extended to account for the nested (multilevel or hierarchical) structure of the data (i.e., respondents are nested within schools) in line with the approaches described by Grilli and Rampichini7 and Asparouhov and Muthén.8 Once an acceptable factor model was established, an estimate of the reliability (internal consistency) for the scales and subscales was calculated using an approach described by Raykov.9 Raykov’s internal consistency reliability coefficient captures the precision of a scale score (e.g., computed by summing or averaging all items within the scale) and is similar to Cronbach’s alpha coefficient. Both coefficients typically range between 0 and 1 (with coefficients approaching 1 being indicative of more precise measurement and values greater than 0.8 being preferred). Values are interpreted as the proportion of scale variance that is “true score” variance deriving from the underlying common factor and which is not due to random measurement error or item specific variance. However, Cronbach’s alpha coefficient is unbiased only under a fairly stringent assumption about the relationship between the underlying factor and the observed item responses (viz. essential tau equivalence). When this assumption is not met, as is often the case, Cronbach’s alpha is a lower-‐bound estimate of reliability, though another issue can lead to upward bias.

6 Flora, D. B., & Curran, P. J. (2004). An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis with Ordinal Data. Psychological Methods, 9, 466-‐491. doi:10.1037/1082-‐989X.9.4.466

Wirth, R. J., & Edwards, M. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58-‐79. doi:10.1037/1082-‐989X.12.1.58 7 Grilli, L., & Rampichini, C. (2007). Multilevel factor models for ordinal variables. Structural Equation Modeling, 14(1), 1-‐25. doi:10.1207/s15328007sem1401_1 8 Asparouhov, T., & Muthén, B. (2007). Computationally efficient estimation of multilevel high-‐dimensional latent variable models. Proceedings of the 2007 Joint Statistical Meetings, Section on Statistics in Epidemiology (pp. 2531– 2535). Alexandria, VA: American Statistical Association. 9 Raykov, T. (1997). Scale reliability, Cronbach's Coefficient Alpha, and violations of essential tau-‐equivalence with fixed congeneric components. Multivariate Behavioral Research, 32(4), 329-‐353. doi:10.1207/s15327906mbr3204_2

Page | 9

Raykov’s reliability coefficient (rho) is valid under a weaker assumption of congeneric structure and can account, if properly modeled, for a common source of upward bias.

Page | 10

Program Effectiveness Surveys Description

The Program Effectiveness battery of surveys is designed to gauge the extent to which Catholic Schools adhere to standards developed by the Task Force for National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools.10 Separate self-‐report surveys were designed to capture the perceptions of adults, middle school (5th – 8th grade) students, and high school (9th – 12th grade) students of the “policies, programs, structures, and processes that should be present in mission-‐driven, program effective, well-‐managed, and responsibly governed Catholic schools,” according to the National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools. Each survey has three or more items pertaining to each of the four domains articulated in the Standards: Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality. The Catholic School Program Effectiveness Survey of Adults consists of 42 items, with Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality subscales consisting of 13, 7, 14, and 8 items, respectively. The Catholic School Program Effectiveness Survey of 5th – 8th grade students consists of 27 items, with Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality subscales consisting of 9, 3, 12, and 3 items, respectively. The Catholic School Program Effectiveness Survey of 9th – 12th grade students consists of 35 items, with Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality subscales consisting of 13, 4, 14, and 4 items, respectively. Survey respondents self-‐reported their perceptions on a 5-‐point Likert response scale ranging from Strongly Disagree (1) to Strongly Agree (5). Specific items and their subscale grouping can be found in Appendix A: Tables from the Item-‐level Analyses of the Catholic School – Program Effectiveness Surveys.

Adult Survey

Respondent Characteristics

A total of 3,526 respondents from 55 schools (average of 64 respondents per school) completed the Catholic School Program Effectiveness Survey of Adults for this psychometric study. The majority of respondents were non-‐school based (75%). Of non-‐school based respondents, 94.6% were parents, 2.9% were community members, and 2.5% self-‐identified as “other”. Among the parent respondents, 40.2% reported their oldest child’s grade as being 4th grade or below, 38% in 5th – 8th grades, and 12% in 9th – 12th grades. Of school-‐based respondents (25% of the total sample), 71.8% were teachers, 19.1% were staff, and 9.1% were administrators. Most respondents were female (83%), and the racial composition of the sample was: White (92%), African American or Black (4.5%), multi-‐racial (1.7%), Asian (1.2%), American Indian or


Page | 11

Alaska Native (.3%), or Native Hawaiian or Pacific Islander (.1%). The majority of respondents identified as non-‐Hispanic (95.8%). Most respondents expressed their religious affiliation as Catholic (85.7%). In general, respondents reported multiple years of affiliation with the school: 30.7% reported 1-‐4 years, 31.5% reported 5-‐10 years, and 27.8% reported more than 10 years. Only 10% reported less than 1 year of affiliation with the school.

Response Pattern Summaries

Response frequencies and intraclass correlation coefficients (ICCs) for each of the 42 items are reported in Table A.1 in Appendix A: Tables from the Item-‐level Analyses of the Catholic School – Program Effectiveness SurveysAppendix . On 26 of the items, at least 75% of adults endorsed the Strongly Agree or adjacent response options, and all items were endorsed (Strongly Agree or adjacent response) by at least 50% of respondents. Though response frequencies are clustered towards endorsement of options indicating greater agreement, there was some degree of variability of frequency patterns across items. Item percentages for skipped or Don't Know responses ranged between 9.7% and 43.1% (Mean = 21.4%). Thirteen items were skipped or had a Don't Know response from 25% or more of respondents. All Operational Vitality items exhibited rates exceeding 28%. The most extreme instance was item 35 (Our school’s financial plan is the result of a collaborative process including expert advisors). Of note is that the rates of using the Don’t Know option were much higher for these items than most preceding items, which may indicate that respondents did not feel they had sufficient knowledge of these items to provide their ratings. However explained, these items should be considered for rewording, replacement or deletion from the scale, along with other items exhibiting the highest skip or Don't Know rates, given the problems missing data presents for estimation of subscale and scale scores.11 ICCs quantify the proportion of variability in responses that is attributable to variability between scores. In other terms, an ICC reflects the degree of non-‐independence amongst responses from adults affiliated with the same school, with 0 = independence (i.e., no systematic variation across schools) and 1 = complete dependence (i.e., all variation in responses is due to differences across schools). As this survey is intended to measure school-‐level adherence to Standards, ICCs greater than zero are to be expected and desired. Additionally, ICCs greater than .01 support the need for statistical methods that account for the observed non-‐independence. The ICCs for most items were moderate in magnitude, reflecting similarity in responses from adults affiliated the same school and justifying the need for statistical methods that can account for non-‐independence amongst the observations. ICCs across all items ranged from .03 to .11 (Mean = .08), indicating that between 3% and 11% of variation in item responses was attributable to respondents being affiliated with different schools. The Catholic School Program Effectiveness Survey of Adults is intended to provide data on Catholic schools’ adherence to the Standards. Given this intent, the focus of these analyses is on school-‐level aggregations of the responses. Table A.2 in Appendix A: Tables from the Item-‐level Analyses of the provides statistics to describe the distributions of these school-‐level aggregates, where adult responses for each item are

11 Although most analyses described in the Factorial Validity and Reliability use statistical procedures that can incorporate respondents that are missing data on one or more items, respondents missing data on all items are necessarily omitted from these analyses. Depending on the particular analysis, between 286 and 694 respondents were omitted from subsequent analysis due to missing data on all items.

Page | 12

averaged with other adults from their school. The theoretical range for these aggregated item averages is 1 (Strongly Disagree) -‐ 5 (Strongly Agree), with higher scores indicating higher within-‐school average agreement for the particular item. On average, schools were rated highly on all of the items (mean average rating across items = 4.08). Along with average, minimum, and maximum ratings, standard deviations and quartiles are also provided for each item. The median (also known as the 50th percentile or second quartile) reflects the item score at which 50% of schools fall at or below. For all items, 50% or more of the schools scored at least a 3.59. Similarly to the respondent-‐level item statistics, scores were clustered toward the upper end of the distribution.

Factorial Validity and Reliability

Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Adults subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with seven to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Adults. CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model).12 The model fit tests and indices for these analyses are reported in Table 1Error! Not a valid bookmark self-‐reference.. These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration.

Table 1. Model Fit Tests and Indices for Full-Scale and Subscale CFA Models of the Catholic School Program Effectiveness Survey of Adults

Chi-‐Square df p-‐value RMSEA CFI SRMR

Mission and Catholic Identity

84.11 65 0.06 0.010 1.00 0.078

Governance and Leadership

30.77 14 0.006 0.020 1.00 0.039

Academic Excellence 93.31 77 0.10 0.008 1.00 0.056

Operational Vitality 28.09 20 0.10 0.012 1.00 0.040

1-‐Factor Model 889.37 819 0.04 0.005 1.00 0.080

4-‐Factor Model -‐-‐ -‐-‐ -‐-‐ -‐-‐ -‐-‐ -‐-‐

Note: df = degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CFI = Comparative Fit Index; SRMR = Standardized Root Mean Residual.

12 In these analyses, model parameters are being simultaneously estimated at two-‐levels, respondent-‐level and the school-‐level., with an unstructured model specified for the within-‐level (i.e., no over-‐identifying constraints included). However, the focus of these analyses is on discovering the factors that pertain to schools. Thus, the focus of this report is exclusively on results pertaining to the school-‐level models.

Page | 13

According to the conventional guidelines described earlier in the methodology section, all subscale models except for the Governance and Leadership model met exact fit standards. Although the Governance and Leadership model exhibited some degree of statistical misfit, this model fit the data approximately well according to the RMSEA, CFI, and SRMR fit indices. The four-‐factor CFA model, which combined the subscale models into a single analysis, did not converge to a solution when the specified number of iterations. This might be due to high collinearity amongst the factors as noted below. Therefore, a one-‐factor CFA model was examined. As reflected by the model fit indices, the one-‐factor model fit the data approximately well. Still, it is essential that the one-‐factor model solution be considered meaningful and interpretable where the meaningfulness or interpretability of a solution is determined by considering the strength and pattern of relationships between the items and underlying (latent) factor. The relationships between factors and indicators are typically depicted in a factor loading matrix. Standardized factor loadings and associated standard errors for the one-‐factor CFA model and for each one-‐factor subscale model are provided in Table in Appendix A: Tables from the Item-‐level Analyses of the . The standardized factor loadings were uniformly very high, with the vast majority greater than 0.8 and none below 0.58. These numbers indicate that each item is a salient and highly reliable measure of the factor. Moreover, the standardized factor loadings are estimated with a moderate degree of precision, as reflected by relatively small standard errors. Though the CFA analyses and higher inter-‐subscale correlations suggest a one-‐factor model, it is possible that another k-‐factor model not examined generated the data. Therefore, an exploratory factor analysis (EFA) was performed prior to settling on the one-‐factor model of the Catholic School Program Effectiveness Survey of Adults. An important and initial task in conducting exploratory factor analysis is to determine the number of factors or dimensions that are being measured by the survey instrument. This determination is guided by statistical tests and indices, evaluation of eigenvalues, and meaningfulness and interpretability of the solution. As with the confirmatory factor analyses, there are two-‐levels under consideration, respondent-‐level and the school-‐level, and the number of factors would typically be determined at each level. However, the focus of this report is on the school as the unit of analysis. Eigenvalues quantify the variance in the item responses that is explained by the factors. Factors that account for more variation are considered potentially more important or meaningful than factors that account for less variation. Eigenvalues from the school-‐level factor analysis of the survey data are reported in Table 2. An often cited rule of thumb is the Kaiser rule, which states that factors with eigenvalues greater than 1.0 should be extracted. According to the Kaiser rule, a model with four factors should be examined. This rule, however, has been criticized as leading to extraction of too many factors and thus typically should be considered an upper bound estimate of the number of factors.

Table 2. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Program Effectiveness Survey of Adults

Factor 1 2 3 4 5 6 7 8 9 10 ...

Eigenvalues 31.90 2.22 2.07 1.43 0.95 0.87 0.65 0.58 0.47 0.36

Another factor enumeration approach used is examination of the Scree plot, depicted in Figure 1. With this approach, one typically seeks the point where there is a pronounced bend (elbow) in the curve. Factors before the bend are given further consideration. According to the scree plot, a one-‐factor or perhaps a two-‐factor model should be considered for extraction. Although more sophisticated methods for factor

Page | 14

enumeration exists (e.g., parallel analysis), these were computationally infeasible or inaccessible for the present analysis, which involve a multilevel structure and ordinal item responses distributions that make it difficult to employ more advanced methods. Thus, the one-‐factor and two-‐factor models were given closer consideration.

Figure 1. Scree Plot of Eigenvalues (School-Level) for the Catholic School Program Effectiveness Survey of Adults

In conjunction with examination of the eigenvalues and scree plot, statistical tests and indices of model fit are often consulted. All models fit the data well (see Table 3). With most indices being within desired ranges for all models under consideration, the philosophical principle known as Occam's razor dictates that one would choose the most parsimonious model, which often is the model with the fewest factors. Consistent with the CFA, model fit indices, in conjunction with the eigenvalues, scree plot and factor pattern matrix, support a model that posits a single factor for explaining and summarizing the Catholic School Program Effectiveness Survey of Adults item responses when aggregated to the school-‐level. This conclusion was corroborated by the less interpretable solutions for the EFA models with two or more factors (e.g., factor loadings > 1, most items load on a general factor, with some Academic Excellence and other items cross-‐loading onto a second factor).

Table 3. Model Fit Tests and Indices for EFA Models of the Catholic School Program Effectiveness Survey of Adults

1-‐Factor Model

2-‐Factor Model

3-‐Factor Model

4-‐Factor Model

Chi-‐Square 889.37 812.09 715.60 652.03

df 819 778 738 699

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7 8 9 10

Eigenvalue

Factor

Page | 15

p-‐value 0.04 0.19 0.71 0.90

RMSEA 0.005 0.004 <0.001 <0.001

CFI 1.00 1.00 1.00 1.00

SRMR 0.080 0.065 0.047 0.039

Summary statistics for the distribution of composite scores formed by taking the average of school-‐level item averages for the entire Catholic School Program Effectiveness Survey of Adults and each subscale can be found in Table 4. The theoretical range for the composite and subscale scores is 1 -‐ 5, with 5 reflecting greater adherence to the Standards as reported by the respondents. On average, schools tended to score highly on the scale and each subscales, with 75% or more schools scoring greater than 3.69 on all scales.

Table 4. Descriptive Statistics for a Total Scale Composite Score and Subscale Scores (School-level) Derived from the Catholic School Program Effectiveness Survey of Adults

Mean Standard Deviation Minimum Maximum 25th

Percentile Median 75th Percentile Reliability SEM


4.15 0.26 3.39 4.65 4.01 4.15 4.32 0.98 0.04


4.22 0.33 2.94 4.74 4.09 4.23 4.42 0.99 0.03

Academic Excellence

4.08 0.33 2.87 4.63 3.99 4.10 4.29 0.98 0.05

Operational Vitality

3.86 0.39 2.56 4.53 3.69 3.90 4.14 0.98 0.06

Total Score 4.08 0.30 3.12 4.56 3.98 4.09 4.27 0.99 0.03

The composite scale and subscale scores exhibited excellent reliability, with all coefficients estimated > .98. This is largely due to the Catholic School Program Effectiveness Survey of Adults having a large number of highly correlated items, each being rated by several respondents per school. Based on an estimate of the standard deviation for the composite (scale) and subscale scores and their respective reliability coefficients,

the standard errors of measurement can be calculated as . The standard error of

measurement can be used to form confidence bands around scores for specific schools. It should be noted that while the exceptionally high reliability for this scale is a positive attribute, achievement of the high reliability, in large part, comes at the cost of a relatively lengthy instrument. In the psychometric literature, reliabilities of .90 are often considered sufficient for most or all practical uses of an instrument. With reliabilities generally approaching or exceeding .99, one or more items could be trimmed from most subscales while retaining sufficient reliability and breadth of coverage. This might facilitate more efficient survey administration and less response burden.

Observed subscale correlations are presented in Table 5. Correlation Matrix for Subscale Scores (School-‐level) Derived from the Catholic School Program Effectiveness Survey of Adults. These correlations are very high, with an average correlation of .85, and indicate substantial overlap in the information conveyed by

Page | 16

the subscale scores and support a one-‐factor conceptualization and use of the Catholic School Program Effectiveness Survey of Adults.

Table 5. Correlation Matrix for Subscale Scores (School-level) Derived from the Catholic School Program Effectiveness Survey of Adults

Mission and

Catholic Identity

Governance and

Leadership

Academic Excellence



-‐-‐


0.90 -‐-‐

Academic Excellence

0.86 0.89 -‐-‐


0.77 0.85 0.81 -‐-‐

Note: All correlations are statistically significant at p < .001.

Summary and Recommendations

Based on the results of these analyses, the Catholic School Program Effectiveness Survey of Adults appears to measure a single construct or factor reflecting the general shared perceptions of schools’ adherence to the Standards. All items were supported as valid and reliable indicators of a general Standards construct, and reliability estimates of a composite score computed by averaging over respondents and items within the same school indicated exceptionally high reliability (i.e., very minimal measurement error). In fact, the attained degree of reliability affords some opportunity to shorten the scale by trimming items that exhibit high rates of missing data (skipped questions and Don’t Know responses) to ease respondent burden while maintaining excellent reliability. Perhaps contrary to initial expectation, there was no evidence that this survey instrument measures four distinct factors relating to the four domains articulated in the Standards (viz., Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality). This finding has one or more explanations, each with potentially important implications for the interpretation and use of the survey. These four theoretical constructs/domains may be ecologically valid and distinct, but could be highly correlated in that schools which tend do well in one domain will also tend do well in the other domains. This may be particularly true for the sample of schools that self-‐selected to participate in this psychometric study. Although psychometric concepts such as validity and reliability are often ascribed to instruments, they are more accurately considered properties of the intended inferences or interpretations made from measurement scores, which includes not only the instrument as stimulus but also the characteristics of the respondent population, conditions during measurement, and inferences made on the basis of the scores. It may be that regardless of item construction, the respondents in this sample may not cognitively distinguish amongst the domains. That is, although the domains were developed based on a considerable body of theoretical and empirical literature, this theoretical and empirical knowledge may not be sufficiently developed within the majority of respondents for them to differentiate amongst the domains in their individual or collective responses to the survey. Without this knowledge, discernment amongst the domains by the respondents may be unrealistic, regardless of the survey items. Even in the absence of such

Page | 17

exposure, all of this is not to suggest that schools should not consider computing and interpreting subscale scores. Examination of subscale scores might be useful, for instance, if a school is evaluating a concerted effort to improve its efforts with respect to a particular standard/domain. It might that such a concerted effort targeting improvements pertaining to a single domain might generate movement unique to the subscale measuring the targeted domain. It is also possible that schools that may not score uniformly across the subscales may be underrepresented in this present sample. If such is the case, the subscales might be of more individual importance for these schools. However, in general, the subscale scores corresponding to each of the domains would be expected to be highly correlated with one another and thus convey little unique information. In such cases, focusing on the total composite score would provide a more reliable, albeit general, measure of adherence to the Standards. Certain items, particularly those from the Operational Vitality or Academic Excellence subscales, exhibited high rates of either being skipped by the respondent or receiving a Don’t Know response. Based on the content of the items and pattern of Don’t Know responses, it seems likely that the respondents, particularly non-‐school-‐based adults, to the survey may not have adequate knowledge of the content being tapped by these items. To the extent that these items require knowledge of their school that the respondents may not have, some of the respondents who did respond to the item may have done so with little actual knowledge and either made a “guess” or answered in a way to be consistent with their responses on other items or overall impression of their school. These items should be reviewed and considered for deletion from the survey.13 It may be possible to re-‐write some items so as to be clearer to respondents. It would be desirable to have missing data rates less than 10%. The worst offending items with the rates of missing data greater than 30% are listed below:

Our school’s financial plan is the result of a collaborative process including expert advisors. Our school has an institutional advancement plan, based on our mission, which uses current and

effective strategies for communications, marketing, enrollment management, and development. Our school leaders take responsibility for ensuring that the financial plans and budgets are

implemented using best practices. Our school maintains and shares a technology management plan. Our school treats all personnel with consistency, fairness, and justice. Our school maintains and shares plans for managing facilities and equipment. At our school, all administrators, faculty and staff engage in ongoing professional development. Teachers vary the types of assessments they use to monitor individual and class-‐wide student

learning.

The above items were associated with the highest rates of missing data. But other items had higher than desired rates of missing data as well. These items tended to be towards the latter half of the surveys and may suggest respondent fatigue or disinterest in the survey. Trimming the scale of items that exhibited the highest non-‐response rates may ease the response burden and address this issue. It might also be prudent not to group subscale items together. Currently, the Academic Excellence and Operational Vitality subscales, which are in the latter half of the survey, experience a disproportionate share of missing data. It

13 Note that deleting reliable items can reduce subscale reliability. This is not as much of a concern for scales with many items, but may be for scales with fewer than five to seven items. If reliability becomes a concern after deleting items, one can attempt to address this issue by generating replacement items that measure the construct of interest for those that are deleted.

Page | 18

may be that these items/subscales are less familiar to respondents or it may be they are fatigued or bored by the time they reach these items. Any missingness due to fatigue/boredom may be more evenly distributed across subscales if a random mixing of items was employed. A 5-‐point Likert scale was employed with anchors provided for only the extreme points (Strongly Disagree, Strongly Agree). Adding labels for the intermediate points may (though not necessarily) increase the range of the scale being used by the respondents, which may translate into more variability in ratings and subscale scores across schools. To the extent that the increased variability is reliable variation, adding labels for all points would tend to increase reliability. It might also help standardize the cognitive mapping used by respondents to link scale points to internal sense of agreement with the items. This may increase reliability as random individual differences in this mapping with the current labeling scheme may manifest as measurement error. Importantly, using standard labels (e.g., Strongly Disagree, Disagree, Neither Agree/Disagree, Agree, Strongly Agree) may help achieve item and scale scores that better approximate interval-‐level properties and do so more consistently so across respondents. Ordinal scales only convey information about the rank, whereas interval-‐level measures also convey the amount of difference. Finally, a few cautionary statements pertaining to these analyses are necessary. These analyses were conducted with less than optimal number of schools for assessing the school-‐level psychometric properties of the survey. Although the large number of respondents within each participating school and the generally high reliability (and factor loadings) of most items facilitated the analysis, it would have been desirable to have at least 150-‐200 schools. Related and also impinging on the results is that the high non-‐response rate of schools introduces the possibility of self-‐selection effects. The implication of this is that the obtained sample may not be representative of the broader population of schools. Therefore, the results may not fully generalize to that broader population.

Page | 19

Student Survey (Grades 5-8)


A total of 4,343 respondents from 46 schools (average of 94 respondents per school) completed the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) for this psychometric study. Participating students were nearly evenly distributed across grade levels (21.2% to 28.5% in the four grades). Just over half of respondents were female (55%), and the racial composition of the sample was: White (81.5%), African American or Black (5.7%), multi-‐racial (7.1%), Asian (4.4%), American Indian or Alaska Native (.9%), or Native Hawaiian or Pacific Islander (.4%). The majority of respondents identified as non-‐Hispanic (92.4%). Most respondents expressed their religious affiliation as Catholic (87.1%). Most respondents also reported being affiliated with the school for more than 4 years (71.1%).


Response frequencies and intraclass correlation coefficients (ICCs) for each of the 27 items are reported in Table A.4. Item Response Frequencies and Intraclass Correlation Coefficients (ICC) for the Catholic School Program Effectiveness Survey of Students (5-‐8 Grades) in Appendix AAppendix . On 9 of the items, at least 75% of middle school students endorsed the Strongly Agree or adjacent response options, and all items were endorsed (Strongly Agree or adjacent response) by at least 50% of respondents. Though response frequencies are clustered towards endorsement of options indicating greater agreement, there was some degree of variability of frequency patterns across items. Item percentages for skipped or Don't Know responses ranged between 4.7% and 21.9% (Mean = 9.5%). One item was skipped or had a Don't Know response from 20% or more of respondents (viz., At our school, teachers use student work and student test results to improve how they teach.). Of note is that the rate of using the Don’t Know option was much higher for this item than all other items, which may indicate that respondents did not feel they had sufficient knowledge to provide their rating for this item. Three other items had a Don’t Know response rate greater than 10%. However explained, these items should be considered for rewording, replacement or deletion from the scale, along with other items exhibiting the highest skip or Don't Know rates, given the problems missing data presents for estimation of subscale and scale scores.14 ICCs quantify the proportion of variability in responses that is attributable to variability between scores. In other terms, an ICC reflects the degree of non-‐independence amongst responses from students affiliated with the same school, with 0 = independence (i.e., no systematic variation across schools) and 1 = complete dependence (i.e., all variation in responses is due to differences across schools). As this survey is intended to measure school-‐level adherence to Standards, ICCs greater than zero are to be expected and desired. Additionally, ICCs greater than .01 support the need for statistical methods that account for the observed non-‐independence. The ICCs for most items were moderate in magnitude, reflecting similarity in responses


Page | 20

from students affiliated the same school and justifying the need for statistical methods that can account for non-‐independence amongst the observations. ICCs across all items ranged from .02 to .21 (Mean = .07), indicating that between 2% and 21% of variation in item responses was attributable to respondents being affiliated with different schools.

The Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) is intended to provide data on Catholic schools’ adherence to the Standards. Given this intent, the focus of these analyses is on school-‐level aggregations of the responses. Table A.5. School-‐level Statistics for the Catholic School Program Effectiveness Survey of Students (5-‐8 Grades) in Appendix A: Tables from the Item-‐level Analyses of the provides statistics to describe the distributions of these school-‐level aggregates, where student responses for each item are averaged with other students from their school. The theoretical range for these aggregated item averages is 1 (Strongly Disagree) -‐ 5 (Strongly Agree), with higher scores indicating higher within-‐school average agreement for the particular item. On average, schools were rated highly on all of the items (mean average rating across items = 4.01). Along with average, minimum, and maximum ratings, standard deviations and quartiles are also provided for each item. The median (also known as the 50th percentile or second quartile) reflects the item score at which 50% of schools fall at or below. For all items, 50% or more of the schools scored at least a 3.53. Similarly to the respondent-‐level item statistics, scores were clustered toward the upper end of the distribution.


Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with three to twelve indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades). CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model).15 The model fit tests and indices for these analyses are reported in Table 6. Model Fit Tests and Indices for Full-‐Scale and Subscale CFA Models of the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades)Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Adults subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with seven to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Adults. CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model). The model fit tests and indices for these analyses are reported in Table 1Error! Not a valid bookmark self-‐reference.. These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration.


Page | 21

Table . These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration.

Table 6. Model Fit Tests and Indices for Full-Scale and Subscale CFA Models of the Catholic School Program Effectiveness Survey of Students (5th - 8th grades)

Chi-‐Square df p-‐value RMSEA CFI SRMR


27.25 27 0.45 0.001 1.00 0.087

Governance and Leadershipa -‐-‐ -‐-‐ -‐-‐ -‐-‐ -‐-‐ -‐-‐

Academic Excellence 49.82 54 0.64 <0.001 1.00 0.101

Operational Vitalitya -‐-‐ -‐-‐ -‐-‐ -‐-‐ -‐-‐ -‐-‐

1-‐Factor Model 333.83 324 0.34 0.003 1.00 0.126

4-‐Factor Model 324.23 318 0.39 0.002 1.00 0.122

Note: df = degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CFI = Comparative Fit Index; SRMR = Standardized Root Mean Residual. aWith only 3 items for each of these subscales, these models are completely saturated (df=0) and thus no meaningful model fit tests or indices are available.

According to the conventional guidelines described earlier in the methodology section, all subscale models met exact fit standards. The four-‐factor CFA model, which combined the subscale models into a single analysis, similar exhibited good model fit. However, factor inter-‐correlations were notably very high (.80 through 1.093),16 indicating substantial overlap and little discriminant validity amongst the factors. Therefore, a one-‐factor CFA model was examined. As reflected by the model fit indices, the one-‐factor model fit the data according to exact-‐fit standards. Still, it is essential that the one-‐factor model solution be considered meaningful and interpretable where the meaningfulness or interpretability of a solution is determined by considering the strength and pattern of relationships between the items and underlying (latent) factor. The relationships between factors and indicators are typically depicted in a factor loading matrix. Standardized factor loadings and associated standard errors for the one-‐factor CFA model and for

16 Correlations are bounded by the interval [-‐1,1]. The model-‐estimated correlation between the Vitality and Academic Excellence factors exceeded 1 and is a non-‐admissible value. This likely stems from sampling and minor specification error in combination with a high population correlation between the factors.

Page | 22

each one-‐factor subscale model are provided in Table in Appendix A: Tables from the Item-‐level Analyses of the . The standardized factor loadings were mostly high, with the vast majority greater than 0.7 and all but two loadings above 0.48, indicating that these items are salient and highly reliable measures of the factor. Two items, however, had marginal factor loadings: (1) Our school offers guidance and resources (such as counselors, tutors and special teachers) to help students be successful and (2) Our school offers programs and activities (such as sports, drama, clubs, or band) for students to develop their gifts and talents. Only approximately 10% of the variation in the items is explained by variability in the underlying factor, and the items were noticeably poorer measures of the latent factor compared to other items.

Though the CFA analyses and higher inter-‐subscale correlations suggest a one-‐factor model, it is possible that another k-‐factor model not examined generated the data. Therefore, an exploratory factor analysis (EFA) was performed prior to settling on the one-‐factor model of the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades). An important and initial task in conducting exploratory factor analysis is to determine the number of factors or dimensions that are being measured by the survey instrument. This determination is guided by statistical tests and indices, evaluation of eigenvalues, and meaningfulness and interpretability of the solution. As with the confirmatory factor analyses, there are two-‐levels under consideration, respondent-‐level and the school-‐level, and the number of factors would typically be determined at each level. However, the focus of this report is on the school as the unit of analysis.

Eigenvalues quantify the variance in the item responses that is explained by the factors. Factors that account for more variation are considered potentially more important or meaningful than factors that account for less variation. Eigenvalues from the school-‐level factor analysis of the survey data are reported in Table 7. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades). An often cited rule of thumb is the Kaiser rule, which states that factors with eigenvalues greater than 1.0 should be extracted. According to the Kaiser rule, a model with five factors should be examined. This rule, however, has been criticized as leading to extraction of too many factors and thus typically should be considered an upper bound estimate of the number of factors.

Table 7. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Program Effectiveness Survey of Students (5th - 8th grades)

Factor 1 2 3 4 5 6 7 8 9 10 ...

Eigenvalues 16.26 2.51 2.19 1.55 1.31 0.91 0.64 0.54 0.47 0.46

Another factor enumeration approach used is examination of the Scree plot, depicted in Figure 2. Scree Plot of Eigenvalues (School-‐Level) for the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades). With this approach, one typically seeks the point where there is a pronounced bend (elbow) in the curve. Factors before the bend are given further consideration. According to the scree plot, a one-‐factor or perhaps a two-‐factor model should be considered for extraction. Although more sophisticated methods for factor enumeration exists (e.g., parallel analysis), these were computationally infeasible or inaccessible for the present analysis, which involve a multilevel structure and ordinal item responses distributions that make it difficult to employ more advanced methods. Thus, the one-‐factor and two-‐factor models were given closer consideration.

Page | 23

Figure 2. Scree Plot of Eigenvalues (School-Level) for the Catholic School Program Effectiveness Survey of Students (5th - 8th grades)

In conjunction with examination of the eigenvalues and scree plot, statistical tests and indices of model fit are often consulted. All models fit the data well (see Table 8. Model Fit Tests and Indices for EFA Models of the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades)). With most indices being within desired ranges for all models under consideration, the philosophical principle known as Occam's razor dictates that one would choose the most parsimonious model, which often is the model with the fewest factors. Consistent with the CFA, model fit indices, in conjunction with the eigenvalues, scree plot and factor pattern matrix, suggested a model that posits a single factor for explaining and summarizing the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) item responses when aggregated to the school-‐level. This conclusion was corroborated by the less interpretable solutions for the EFA models with two or more factors (e.g., factor loadings > 1, most items load on a general factor, with some Academic Excellence and other items cross-‐loading onto a second factor).

Table 8. Model Fit Tests and Indices for EFA Models of the Catholic School Program Effectiveness Survey of Students (5th - 8th grades)

1-‐Factor Model

2-‐Factor Model

3-‐Factor Model

4-‐Factor Model

5-‐Factor Model

Chi-‐Square 333.83 291.00 248.85 215.15 185.94

df 324 298 273 249 226

p-‐value 0.34 0.60 0.85 0.94 0.98

RMSEA 0.003 <0.001 <0.001 <0.001 <0.001

CFI 1.00 1.00 1.00 1.00 1.00

SRMR 0.126 0.102 0.080 0.064 0.049

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9 10

Eigenvalue

Factor

Page | 24

Summary statistics for the distribution of composite scores formed by taking the average of school-‐level item averages for the entire Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) and each subscale can be found in Table 9. Descriptive Statistics for a Total Scale Composite Score and Subscale Scores (School-‐level) Derived from the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades). The theoretical range for the composite and subscale scores is 1 -‐ 5, with 5 reflecting greater adherence to the Standards as reported by students. On average, schools tended to score highly on the scale and each subscales, with 75% or more schools scoring greater than 3.67 on all scales.

Table 9. Descriptive Statistics for a Total Scale Composite Score and Subscale Scores (School-level) Derived from the Catholic School Program Effectiveness Survey of Students (5th - 8th grades)




4.06 0.22 3.52 4.49 3.92 4.08 4.23 0.93 0.06


4.12 0.45 2.71 4.85 3.92 4.22 4.44 0.99 0.05

Academic Excellence

3.97 0.28 3.19 4.56 3.87 4.00 4.19 0.88 0.10


3.90 0.29 3.20 4.44 3.67 3.92 4.11 0.78 0.14

Total Score 4.01 0.26 3.36 4.51 3.92 4.01 4.23 0.96 0.05

The composite scale and subscale scores exhibited excellent reliability, except for the Operational Vitality scale. The reliability estimate for the Operational Vitality subscale was marginally acceptable (.78). Based on an estimate of the standard deviation for the composite (scale) and subscale scores and their respective

reliability coefficients, the standard errors of measurement can be calculated as . The

standard error of measurement can be used to form confidence bands around scores for specific schools. It should be noted that while the high reliability for this scale is a positive attribute, achievement of the high reliability, in large part, comes at the cost of a relatively lengthy instrument. In the psychometric literature, reliabilities of .90 are often considered sufficient for most or all practical uses of an instrument. With reliabilities generally exceeding .93 on three subscales, one or more items could be trimmed from these subscales while retaining sufficient reliability and breadth of coverage.17 This might facilitate more efficient survey administration and less response burden.

Observed subscale correlations are presented in Table 10. Correlation Matrix for Subscale Scores (School-‐level) Derived from the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades). The

17 Due to the Governance and Leadership subscale being comprised of just three items, it is not recommended that items be trimmed from this subscale.

Page | 25

correlations are smaller than the factor inter-‐correlations reported earlier as these may be attenuated by measurement error and are obtained using a different estimator. However, these correlations were also very high, with an average correlation of .76. These statistics indicate substantial overlap in the information conveyed by the subscale scores and support a one-‐factor conceptualization and use of the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades).

Table 10. Correlation Matrix for Subscale Scores (School-level) Derived from the Catholic School Program Effectiveness Survey of Students (5th - 8th grades)

Mission and

Catholic Identity

Governance and

Leadership

Academic Excellence



-‐-‐


0.73 -‐-‐

Academic Excellence

0.83 0.80 -‐-‐


0.69 0.64 0.84 -‐-‐



Based on the results of these analyses, the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) appears to measure a single construct or factor reflecting the general shared perceptions of schools’ adherence to the Standards. Most items were supported as valid and reliable indicators of a general Standards construct, and reliability estimates of a composite score computed by averaging over respondents and items within the same school indicated exceptionally high reliability (i.e., very minimal measurement error) for all constructs except Operational Vitality. The reliability estimate for Operational Vitality was marginal, which is partially attributable to this scale having few (3) items. Adding additional, reliable and valid items to measure Operational Vitality would likely increase this subscale’s reliability. For the other subscale scores and the total score, the attained degree of reliability affords some opportunity to shorten the scale by trimming two items that exhibited relatively lower factor loadings and other items that exhibit high rates of missing data (skipped questions and Don’t Know responses). Trimming select items could ease respondent burden while maintaining or, in the case of the two items with low factor loadings, increasing reliability. Perhaps contrary to initial expectation, there was no evidence that this survey instrument measures four distinct factors relating to the four domains articulated in the Standards (viz., Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality). This finding has one or more explanations, each with potentially important implications for the interpretation and use of the survey. These four theoretical constructs/domains may be ecologically valid and distinct, but could be highly correlated in that schools which tend do well in one domain will also tend do well in the other domains. This may be particularly true for the sample of schools that self-‐selected to participate in this psychometric study. Although psychometric concepts such as validity and reliability are often ascribed to instruments,

Page | 26

they are more accurately considered properties of the intended inferences or interpretations made from measurement scores, which includes not only the instrument as stimulus but also the characteristics of the respondent population, conditions during measurement, and inferences made on the basis of the scores. It may be that regardless of item construction, the respondents in this sample may not cognitively distinguish amongst the domains. That is, although the domains were developed based on a considerable body of theoretical and empirical literature, this theoretical and empirical knowledge may not be sufficiently developed within the majority of respondents for them to differentiate amongst the domains in their individual or collective responses to the survey. Without this knowledge, discernment amongst the domains by the respondents may be unrealistic, regardless of the survey items. Even in the absence of such exposure, all of this is not to suggest that schools should not consider computing and interpreting subscale scores. Examination of subscale scores might be useful, for instance, if a school is evaluating a concerted effort to improve its efforts with respect to a particular standard/domain. It might that such a concerted effort targeting improvements pertaining to a single domain might generate movement unique to the subscale measuring the targeted domain. It is also possible that schools that may not score uniformly across the subscales may be underrepresented in this present sample. If such is the case, the subscales might be of more individual importance for these schools. However, in general, the subscale scores corresponding to each of the domains would be expected to be highly correlated with one another and thus convey little unique information. In such cases, focusing on the total composite score would provide a more reliable, albeit general, measure of adherence to the Standards. Two items exhibited with relatively low factor loadings: (1) Our school offers guidance and resources (such as counselors, tutors and special teachers) to help students be successful and (2) Our school offers programs and activities (such as sports, drama, clubs, or band) for students to develop their gifts and talents. These items thus did not emerge as good measures of the Academic Excellence construct or the overarching latent construct. They should be reviewed and considered for re-‐wording or deletion from the scale. Alternatively, if they are considered informative as stand-‐alone items, they may be retained, but should not be averaged or summed with the other items when computing subscale or total scale composite scores. Certain items, particularly those from the Academic Excellence subscale, exhibited high rates of either being skipped by the respondent or receiving a Don’t Know response. Based on the content of the items and pattern of Don’t Know responses, it seems likely that the respondents to the survey may not have adequate knowledge of the content being tapped by these items. To the extent that these items require knowledge of their school that the respondents may not have, some of the respondents who did respond to the item may have done so with little actual knowledge and either made a “guess” or answered in a way to be consistent with their responses on other items or overall impression of their school. These items should be reviewed and considered for deletion from the survey.18 It may be possible to re-‐write some items so as to be clearer to respondents. It would be desirable to have missing data rates less than 10%. The worst offending items with the rates of missing data greater than 15% are listed below:

At our school, teachers use student work and student test results to improve how they teach. At our school, teachers work together to help each other become better teachers.


Page | 27

In our school, adults do service on behalf of the poor and those in need. Our school’s mission affects everything we do.

The above items were associated with the highest rates of missing data. But other items had higher than desired rates of missing data as well. A 5-‐point Likert scale was employed with anchors provided for only the extreme points (Strongly Disagree, Strongly Agree). Adding labels for the intermediate points may (though not necessarily) increase the range of the scale being used by the respondents, which may translate into more variability in ratings and subscale scores across schools. To the extent that the increased variability is reliable variation, adding labels for all points would tend to increase reliability. It might also help standardize the cognitive mapping used by respondents to link scale points to internal sense of agreement with the items. This may increase reliability as random individual differences in this mapping with the current labeling scheme may manifest as measurement error. Importantly, using standard labels (e.g., Strongly Disagree, Disagree, Neither Agree/Disagree, Agree, Strongly Agree) may help achieve item and scale scores that better approximate interval-‐level properties and do so more consistently so across respondents. Ordinal scales only convey information about the rank, whereas interval-‐level measures also convey the amount of difference. Finally, a few cautionary statements pertaining to these analyses are necessary. These analyses were conducted with less than optimal number of schools for assessing the school-‐level psychometric properties of the surveys. Although the large number of respondents within each participating school and the generally high reliability (and factor loadings) of most items facilitated the analysis, it would have been desirable to have at least 150-‐200 schools. Related and also impinging on the results is that the high non-‐response rate of schools introduces the possibility of self-‐selection effects. The implication of this is that the obtained sample may not be representative of the broader population of schools. Therefore, the results may not fully generalize to that broader population.

Page | 28



A total of 114 respondents from 4 schools (average of 29 respondents per school) completed the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades) for this psychometric study. Participating students were mostly in the 9th (43%) or 11th (30%) grades. Just over half of respondents were male (54%), and the racial composition of the sample was: White (73.7%), African American or Black (13.2%), multi-‐racial (8.8%), Asian (1.8%), American Indian or Alaska Native (.9%), or Native Hawaiian or Pacific Islander (1.8%). The majority of respondents identified as non-‐Hispanic (95.6%). Most respondents expressed their religious affiliation as Catholic (71.9%). Respondents were fairly uniformly distributed in their number of years attending the school (10.5% -‐ 23.7%).


Response frequencies for each of the 35 items are reported in Table A.7. Item Response Frequencies for the Catholic School Program Effectiveness Survey of Students (9-‐12 Grades) in Appendix AAppendix . On 17 of the items, at least 75% of high school students endorsed the Strongly Agree or adjacent response options, and all items were endorsed (Strongly Agree or adjacent response) by at least 62% of respondents. Though response frequencies are clustered towards endorsement of options indicating greater agreement, there was some degree of variability of frequency patterns across items. Item percentages for skipped or Don't Know responses ranged between 0% and 32.5% (Mean = 21.0%). Eleven items were skipped or had a Don't Know response from 25% or more of respondents. The most extreme instances were items 23 (At our school, teachers work together to help each other become better teachers), 24 (At our school, all administrators, faculty and staff engage in ongoing professional development), and 26 (At our school, teachers work together to use data on student performance to review and improve courses and instruction). These items on the Operational Vitality subscale and other items in this survey may ask respondents about aspects which they have little or no knowledge. These items tended to have the highest rates of Don’t Know responses. However explained, these items should be considered for rewording, replacement or deletion from the scale, along with other items exhibiting the highest skip or Don't Know rates, given the problems missing data presents for estimation of subscale and scale scores.19 A pattern of items in the second-‐half of the survey tending to exhibit higher skipped or Don’t Know response rates than the first-‐half suggests that respondents may have become have felt rushed or less motivated (e.g., bored, fatigued) as they worked towards completing the survey.

The Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades) is intended to provide data on Catholic schools’ adherence to the Standards. Given this intent, the desired focus of analysis is on school-‐level aggregations of the responses. However, too few high schools were successfully recruited for


Page | 29

this psychometric study to permit school-‐level analyses. Therefore, all subsequent analyses report on analyses at the respondent-‐level. In interpreting the results of these analyses, understand that these results may not extend to the school-‐level and any such generalization is discouraged.20


Ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades) subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with four to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model).21 The model fit tests and indices for these analyses are reported in Table 11. Model Fit Tests and Indices for Full-‐Scale and Subscale CFA Models of the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades)Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Adults subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with seven to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Adults. CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model). The model fit tests and indices for these analyses are reported in Table 1Error! Not a valid bookmark self-‐reference.. These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration. Table . These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration.

Table 11. Model Fit Tests and Indices for Full-Scale and Subscale CFA Models of the Catholic School Program Effectiveness Survey of Students (9th - 12th grades)

Chi-‐Square df p-‐value RMSEA CFI


163.16 65 <0.001 0.122 0.97


0.25 2 0.88 <0.001 1.00

Academic Excellence 172.34 77 <0.001 0.118 0.97

20 In addition, the respondent-‐level sample size was particularly modest for these analyses, particularly those of the entire scale. Caution is warranted when interpreting the results until more data is available. 21 In these analyses, model parameters are being simultaneously estimated at two-‐levels, respondent-‐level and the school-‐level., with an unstructured model specified for the within-‐level (i.e., no over-‐identifying constraints included). However, the focus of these analyses is on discovering the factors that pertain to schools. Thus, the focus of this report is exclusively on results pertaining to the school-‐level models.

Page | 30

Operational Vitality 0.23 2 0.89 <0.001 1.00

1-‐Factor Model 923.81 560 <0.001 0.080 0.96

4-‐Factor Model 763.71 554 <0.001 0.061 0.97

Note: df = degrees of freedom; RMSEA = Root Mean Square Error of Approximation; CFI = Comparative Fit Index.

According to the conventional guidelines described earlier in the methodology section,22 all subscale models except for the Governance and Leadership and Operational Vitality models did not meet exact fit standards. The Mission and Catholic Identity and Academic Excellence models also failed to meet approximate fit standards for the RMSEA, though they did meet the conventional threshold for the CFI. This discrepancy between the RMSEA and CFI is due to differences in how they “define” or quantify misfit and their proclivity to positive bias in small sample sizes. The CFI essentially compares the misfit of the model at hand with the misfit of a null-‐model that specifics no correlation amongst variables. Because the items are highly correlated, the models examined, which all allow some degree of correlation amongst the items, fits much better than the null model; hence the high CFI. The RMSEA estimates absolute misfit per degree of freedom in the model and is positively biased, particularly for small sample sizes. Taken all together, the validity of the subscale models for Mission and Catholic Identity and for Academic Excellence is suspect. In contrast, the subscale models for Operational Vitality and Governance and Leadership exhibited excellent model fit. However, these tests are based on only two degrees of freedom and thus may be underpowered or underspecified to provide a meaningful test of the model’s validity. The four-‐factor model provides for a stronger, more powerful test of the theoretical model. The four-‐factor CFA model, which combined the subscale models into a single analysis, exhibited statistical misfit but borderline approximate model fit. However, factor inter-‐correlations were notably very high (.82 through .93), indicating substantial overlap and little discriminant validity amongst the factors. Therefore, a one-‐factor CFA model was examined. As reflected by the model fit indices, the one-‐factor model fit the data marginally well. Standardized factor loadings and associated standard errors for the one-‐factor CFA model and for each one-‐factor subscale model are provided in Table A.8. Standardized Factor Loadings for the 1-‐factor CFA Model (Student-‐level) of the Catholic School Program Effectiveness Survey of Students (9-‐12 Grades) in Appendix A: Tables from the Item-‐level Analyses of the . The standardized factor loadings were uniformly very high, with the vast majority greater than 0.7 and none below 0.68. These numbers indicate that each item is a salient and highly reliable measure of the factor. Moreover, the standardized factor loadings are estimated with a moderate degree of precision, as reflected by relatively small standard errors.23 Though the CFA analyses and higher inter-‐subscale correlations suggest a one-‐factor model, it is possible that another k-‐factor model not examined generated the data. Therefore, an exploratory factor analysis (EFA) was performed prior to settling on the one-‐factor model of the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). An important and initial task in conducting exploratory factor analysis is to determine the number of factors or dimensions that are being measured by the survey

22 The SRMR fit index is not available for these analyses. 23 Given evidence of potential model misspecification, caution in interpreting the factor loadings is warranted until these results can be cross-‐validated on a larger data set and, importantly, conducted at the school-‐level.

Page | 31

instrument. This determination is guided by statistical tests and indices, evaluation of eigenvalues, and meaningfulness and interpretability of the solution. Eigenvalues quantify the variance in the item responses that is explained by the factors. Factors that account for more variation are considered potentially more important or meaningful than factors that account for less variation. Eigenvalues from the school-‐level factor analysis of the survey data are reported in Table 12. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). An often cited rule of thumb is the Kaiser rule, which states that factors with eigenvalues greater than 1.0 should be extracted. According to the Kaiser rule, a model with six factors should be examined. This rule, however, has been criticized as leading to extraction of too many factors and thus typically should be considered an upper bound estimate of the number of factors.

Table 12. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Program Effectiveness Survey of Students (9th - 12th grades)

Factor 1 2 3 4 5 6 7 8 9 10 ...

Eigenvalues 20.85 2.12 1.54 1.3 1.1 1.05 0.89 0.77 0.75 0.7

Another factor enumeration approach used is examination of the Scree plot, depicted in Figure 3. Scree Plot of Eigenvalues (School-‐Level) for the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). With this approach, one typically seeks the point where there is a pronounced bend (elbow) in the curve. Factors before the bend are given further consideration. According to the scree plot, a one-‐factor or perhaps a two-‐factor model should be considered for extraction. Although more sophisticated methods for factor enumeration exists (e.g., parallel analysis), these were computationally infeasible or inaccessible for the present analysis, which involved ordinal item responses distributions that make it difficult to employ more advanced methods. Thus, the one-‐factor and two-‐factor models were given closer consideration.

Page | 32

Figure 3. Scree Plot of Eigenvalues (School-Level) for the Catholic School Program Effectiveness Survey of Students (9th - 12th grades)

In conjunction with examination of the eigenvalues and scree plot, statistical tests and indices of model fit are often consulted. All models exhibited statistical misfit as indicated by the Chi-‐square test of exact fit (see Table 13. Model Fit Tests and Indices for EFA Models of the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades)). With most indices being within desired ranges for all models under consideration, the philosophical principle known as Occam's razor dictates that one would choose the most parsimonious model, which often is the model with the fewest factors. Consistent with the CFA, model fit indices, in conjunction with the eigenvalues, scree plot and factor pattern matrix, suggested a model that posits a single factor for explaining and summarizing the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades) item responses. This conclusion was corroborated by the less interpretable solutions for the EFA models with three or more factors. The two factor solution exhibited a factor characterized by the Catholic Identity and Mission items and item 32 from the Operational Vitality subscale and a second factor characterized by the remaining items and item 10 (Catholic Identity and Mission subscale), which loaded saliently on both factors. Although relatively interpretable, the factor correlation of .74 was considerably large and suggests that there may not be two conceptually distinct factors. Therefore, focus is on the one factor solution, with caution given the possible model misspecification.

Table 13. Model Fit Tests and Indices for EFA Models of the Catholic School Program Effectiveness Survey of Students (9th - 12th grades)

1-‐Factor Model

2-‐Factor Model

3-‐Factor Model

4-‐Factor Model

5-‐Factor Model

Chi-‐Square 923.81 763.61 677.77 611.06 549.36

df 560 526 493 461 430

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

Eigenvalue

Factor

Page | 33

p-‐value <0.001 <0.001 <0.001 <0.001 <0.001

RMSEA 0.080 0.067 0.061 0.056 0.052

CFI 0.96 0.97 0.98 0.98 0.99

SRMR 0.087 0.069 0.06 0.054 0.047

Summary statistics for the distribution of composite scores formed by taking the average of item scores across respondents for the entire Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades) and each subscale can be found in Table 14. Descriptive Statistics for a Total Scale Composite Score and Subscale Scores (Student-‐level) Derived from the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). The theoretical range for the composite and subscale scores is 1 -‐ 5, with 5 reflecting greater adherence to the Standards as reported by students. On average, students tended to rate their schools highly on the total scale and each subscale.

Table 14. Descriptive Statistics for a Total Scale Composite Score and Subscale Scores (Student-level) Derived from the Catholic School Program Effectiveness Survey of Students (9th - 12th grades)




4.17 0.64 2.69 5.00 3.69 4.15 4.69 0.96 0.13


4.18 0.79 1.00 5.00 4.00 4.25 4.75 0.93 0.21

Academic Excellence

3.95 0.80 1.21 5.00 3.55 4.07 4.50 0.97 0.14


3.79 0.94 1.25 5.00 3.44 4.00 4.50 0.91 0.28

Total Score 4.13 0.63 2.40 5.00 3.79 4.19 4.66 0.98 0.09

The composite scale and subscale scores exhibited excellent reliability, with all coefficients estimated > .91. This is largely due to the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). Based on an estimate of the standard deviation for the composite (scale) and subscale scores and their respective reliability coefficients, the standard errors of measurement can be calculated as

. The standard error of measurement can be used to form confidence bands around

scores for specific schools.

Observed subscale correlations are presented in Table 15. Correlation Matrix for Subscale Scores (Student-‐level) Derived from the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades). The correlations are smaller than the factor inter-‐correlations reported earlier as these may be attenuated by measurement error and are obtained using a different estimator. However, these correlations were also very high, with an average correlation of .85. These statistics indicate substantial overlap in the information conveyed by the subscale scores and support a one-‐factor conceptualization and use of the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades).

Page | 34

Table 15. Correlation Matrix for Subscale Scores (Student-level) Derived from the Catholic School Program Effectiveness Survey of Students (9th - 12th grades)

Mission and

Catholic Identity

Governance and

Leadership

Academic Excellence



-‐-‐


0.80 -‐-‐

Academic Excellence

0.75 0.83 -‐-‐


0.68 0.76 0.72 -‐-‐



Based on the results of these analyses, the Catholic School Program Effectiveness Survey of Students (9th -‐ 12th grades) appears to measure a single construct or factor reflecting individual perceptions of their school’s adherence to the Standards. All items were supported as valid and reliable indicators of a general Standards construct as measured at the respondent level, and reliability estimates of a composite score computed by averaging over items generally indicated high reliability (i.e., very minimal measurement error). In fact, the attained degree of reliability affords some opportunity to shorten the scale by trimming items that exhibit high rates of missing data (skipped questions and Don’t Know responses) to ease respondent burden while maintaining excellent reliability. Perhaps contrary to initial expectation, there was no evidence that this survey instrument measures four distinct factors relating to the four domains articulated in the Standards (viz., Mission and Catholic Identity, Governance and Leadership, Academic Excellence, and Operational Vitality). The possible explanations and implications for this are identical to those described for the other Program Effectiveness surveys. However, limitations of the 9th – 12th grade student data on this survey limits the conclusions that can be drawn from these results. Too few high schools were recruited to permit analyses at the school-‐level, the desired level of inference for these surveys. Analyses were thus conducted at the respondent (i.e., student) level, but it is very possible that the factor structure and reliability of the instrument can differ at the school-‐level from that at the student-‐level. Moreover, the sample size was modest and less than optimal for the type of analyses being conducted. As such, caution is warranted in generalizing the results beyond the schools and respondents sampled to the broader population of Catholic high schools. It is recommended that a multi-‐level factor analysis be conducted on data collected from a larger administration of this survey involving at least 100-‐200 schools. Certain items, particularly those from the Academic Excellence and Operational Vitality subscales, exhibited high rates of either being skipped by the respondent or receiving a Don’t Know response. Based on the content of some of these items and pattern of Don’t Know responses, it seems likely that the respondents to the survey may not have adequate knowledge of the content being tapped by these items to provide a rating. To the extent that these items require knowledge of their school that the respondents may not have, some of the respondents who did respond to the item may have done so with little actual knowledge

Page | 35

and either made a “guess” or answered in a way to be consistent with their responses on other items or overall impression of their school. These items should be reviewed and considered for deletion from the survey.24 Re-‐writing some items so as to be clearer to respondents is another option. It would be desirable to have missing data rates less than 10%. The worst offending items with the rates of missing data greater than 25% are listed below:

At our school, teachers work together to use data on student performance to review and improve courses and instruction.

At our school, teachers work together to help each other become better teachers. At our school, all administrators, faculty and staff engage in ongoing professional development. At our school, teachers follow school-‐wide policies and procedures to fairly evaluate and

communicate student performance. Our school communicates how well students are achieving in comparison to similar students locally

and/or nationally. Teachers vary the types of assessments they use to monitor individual and class-‐wide student

learning. Our school uses different ways to communicate all that is happening in our school to

parents/guardians, the school community and beyond. Our school does a good job of attracting new students to our school. Our school provides programs and services that help students successfully complete the school

program (for example, guidance and resource programs). Our school provides opportunities for parents/guardians to be involved in the education of their

students.

The above items were associated with the highest rates of missing data. But other items had higher than desired rates of missing data as well. A 5-‐point Likert scale was employed with anchors provided for only the extreme points (Strongly Disagree, Strongly Agree). Adding labels for the intermediate points may (though not necessarily) increase the range of the scale being used by the respondents, which may translate into more variability in ratings and subscale scores across schools. To the extent that the increased variability is reliable variation, adding labels for all points would tend to increase reliability. It might also help standardize the cognitive mapping used by respondents to link scale points to internal sense of agreement with the items. This may increase reliability as random individual differences in this mapping with the current labeling scheme may manifest as measurement error. Importantly, using standard labels (e.g., Strongly Disagree, Disagree, Neither Agree/Disagree, Agree, Strongly Agree) may help achieve item and scale scores that better approximate interval-‐level properties and do so more consistently so across respondents. Ordinal scales only convey information about the rank, whereas interval-‐level measures also convey the amount of difference.


Page | 36

Defining Characteristics Surveys Description

The Defining Characteristics battery of surveys is designed to gauge the extent to which Catholic Schools operate in accord with characteristics articulated by the Task Force for National Standards and Benchmarks for Effective Catholic Elementary and Secondary Schools.25 Separate self-‐report surveys were designed to capture the perceptions of adults, middle school (5th – 8th grade) students, and high school (9th – 12th grade) students of their schools’ “Catholic identity” with respect to being (a) Centered in the Person of Jesus Christ, (b) Contributing to the Evangelizing Mission of the Church, (c) Distinguished by Excellence, (d) Committed to Educate the Whole Child, (e) Steeped in a Catholic Worldview, (f) Sustained by Gospel Witness, (g) Shaped by Communion and Community, (h) Accessible to All Students, and (j) Established by the Expressed Authority of the Bishop. The Catholic School – Defining Characteristics surveys each consist of 17 similar or identical items. Survey respondents self-‐reported their perceptions on a 5-‐point Likert response scale ranging from Strongly Disagree (1) to Strongly Agree (5). Specific items and their subscale grouping can be found in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics Surveys.

Adult Survey


A total of 2,080 respondents from 52 schools (average of 40 respondents per school) completed the Catholic School Defining Characteristics Survey of Adults for this psychometric study. The majority of respondents were non-‐school based (74%). Of non-‐school based respondents, 94.7% were parents, 3.1% were community members, and 1.6% self-‐identified as “other”. Among the parent respondents, 38.9% reported their oldest child’s grade as being 4th grade or below, 50.2% in 5th – 8th grades, and 10.9% in 9th – 12th grades. Of school-‐based respondents (26% of the total sample), 73.8% were teachers, 16.8% were staff, and 9.4% were administrators. Most respondents were female (84%), and the racial composition of the sample was: White (93.8%), African American or Black (3.3%), multi-‐racial (1.2%), Asian (1.2%), American Indian or Alaska Native (.3%), or Native Hawaiian or Pacific Islander (.2%). The majority of respondents identified as non-‐Hispanic (96.5%). Most respondents expressed their religious affiliation as Catholic (86.9%). In general, respondents reported multiple years of affiliation with the school: 29.2% reported 1-‐4 years, 31.9% reported 5-‐10 years, and 29.3% reported more than 10 years. Only 9.6% reported less than 1 year of affiliation with the school.


Page | 37


Response frequencies and intraclass correlation coefficients (ICCs) for each of the 17 items are reported in Table B.3. Item Response Frequencies and Intraclass Correlation Coefficients (ICC) for the Catholic School Defining Characteristics Survey of Adults in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics SurveysAppendix . On all items, at least 75% of adults endorsed the Strongly Agree or adjacent response options, and all items were endorsed (Strongly Agree or adjacent response) by at least 77% of respondents. Though response frequencies are clustered towards endorsement of options indicating greater agreement, there was some degree of variability of frequency patterns across items. Item percentages for skipped or Don't Know responses ranged between 3.9% and 18.9% (Mean = 6.2%). Two items were skipped or had a Don't Know response from 10% or more of respondents. The most extreme instances were item 16 (Our school does everything it can to eliminate obstacles that hinder or exclude students from receiving a Catholic education) and item 17 (Our school operates with the expressed approval and support of our Bishop). Of note is that the rates of using the Don’t Know option were noticeably higher for these items, particularly item 17, than most preceding items, which may indicate that respondents did not feel they had sufficient knowledge of these items to provide their ratings on these items. However explained, these items should be considered for rewording, replacement or deletion from the scale, along with other items exhibiting the highest skip or Don't Know rates, given the problems missing data presents for estimation of subscale and scale scores.26 ICCs quantify the proportion of variability in responses that is attributable to variability between scores. In other terms, an ICC reflects the degree of non-‐independence amongst responses from adults affiliated with the same school, with 0 = independence (i.e., no systematic variation across schools) and 1 = complete dependence (i.e., all variation in responses is due to differences across schools). As this survey is intended to measure school-‐level alignment with the Defining Characteristics, ICCs greater than zero are to be expected and desired. Additionally, ICCs greater than .01 support the need for statistical methods that account for the observed non-‐independence. The ICCs for most items were moderate in magnitude, reflecting similarity in responses from adults affiliated the same school and justifying the need for statistical methods that can account for non-‐independence amongst the observations. ICCs across all items ranged from .05 to .13 (Mean = .08), indicating that between 5% and 13% of variation in item responses was attributable to respondents being affiliated with different schools. The Catholic School Defining Characteristics Survey of Adults is intended to provide data on Catholic schools’ alignment to the Defining Characteristics. Given this intent, the focus of these analyses is on school-‐level aggregations of the responses. Table B.4. School-‐level Statistics for the Catholic School Defining Characteristics Survey of Adults in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics Surveys provides statistics to describe the distributions of these school-‐level aggregates, where adult responses for each item are averaged with other adults from their school. The theoretical range for these aggregated item averages is 1 (Strongly Disagree) -‐ 5 (Strongly Agree), with higher scores indicating higher within-‐school average agreement for the particular item. On average,

26 Although most analyses described in the Factorial Validity and Reliability use statistical procedures that can incorporate respondents that are missing data on one or more items, 70 respondents missing data on all items were necessarily omitted from these analyses.

Page | 38

schools were rated highly on all of the items (mean average rating across items = 4.41) and minimum ratings averaged 3.30. These statistics indicate little variability in mean ratings across schools for the items, with some items (#’s: 2, 4, and 5) exhibiting very little variability. Along with average, minimum, and maximum ratings, standard deviations and quartiles are also provided for each item. The median (also known as the 50th percentile or second quartile) reflects the item score at which 50% of schools fall at or below. For all items, 50% or more of the schools scored at least a 4.45. Similarly to the respondent-‐level item statistics, scores were clustered toward the upper end of the distribution.


A multilevel, ordinal confirmatory factor analyses (CFA) was performed on the Catholic School Defining Characteristics Survey of Adults. A single-‐factor model of the survey was fit to the data.27 This model fit the data well by both exact fit and approximate fit standards [χ2(119) = 120.36, p = .45, RMSEA = 0.002, CFI = 1.00, SRMR = 0.06]Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Adults subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with seven to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Adults. CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model). The model fit tests and indices for these analyses are reported in Table 1Error! Not a valid bookmark self-‐reference.. These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration. Table . Still, it is essential that the one-‐factor model solution be considered meaningful and interpretable where the meaningfulness or interpretability of a solution is determined by considering the strength and pattern of relationships between the items and underlying (latent) factor. The relationships between factors and indicators are typically depicted in a factor loading matrix. Standardized factor loadings and associated standard errors for the one-‐factor CFA model are provided in Table B.3. Standardized Factor Loadings for the 1-‐factor CFA Model (School-‐level) of the Catholic School Defining Characteristics Survey of Adults in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics Surveys. The standardized factor loadings were uniformly very high, with most loading values greater than 0.8 and none below 0.66. These numbers indicate that each item is a salient and highly reliable measure of the factor. Though the CFA analysis support a one-‐factor model, it is possible that another k-‐factor generated the data. Therefore, an exploratory factor analysis (EFA) was performed prior to settling on the one-‐factor model of the Catholic School Defining Characteristics Survey of Adults. An important and initial task in conducting exploratory factor analysis is to determine the number of factors or dimensions that are being measured by the survey instrument. This determination is guided by statistical tests and indices, evaluation of eigenvalues, and meaningfulness and interpretability of the solution. As with the confirmatory factor analyses, there are two-‐levels under consideration, respondent-‐level and the school-‐level, and the number


Page | 39

of factors would typically be determined at each level. However, the focus of this report is on the school as the unit of analysis. Eigenvalues quantify the variance in the item responses that is explained by the factors. Factors that account for more variation are considered potentially more important or meaningful than factors that account for less variation. Eigenvalues from the school-‐level factor analysis of the survey data are reported in Table 15. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Defining Characteristics Survey of Adults. An often cited rule of thumb is the Kaiser rule, which states that factors with eigenvalues greater than 1.0 should be extracted. According to the Kaiser rule, a model with two factors should be examined. This rule, however, has been criticized as leading to extraction of too many factors and thus typically should be considered an upper bound estimate of the number of factors.

Table 15. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Defining Characteristics Survey of Adults

Factor 1 2 3 4 5 6 7 8 9 10 ...

Eigenvalues 13.55 1.04 0.7 0.52 0.46 0.36 0.23 0.14 0.13 0.1

Another factor enumeration approach used is examination of the Scree plot, depicted in Figure 4. Scree Plot of Eigenvalues (School-‐Level) for the Catholic School Defining Characteristics Survey of Adults. With this approach, one typically seeks the point where there is a pronounced bend (elbow) in the curve. Factors before the bend are given further consideration. According to the scree plot, a one-‐factor or perhaps a two-‐factor model should be considered for extraction. Although more sophisticated methods for factor enumeration exists (e.g., parallel analysis), these were computationally infeasible or inaccessible for the present analysis, which involve a multilevel structure and ordinal item responses distributions that make it difficult to employ more advanced methods. Thus, the one-‐factor and two-‐factor models were given closer consideration.

Page | 40

Figure 4. Scree Plot of Eigenvalues (School-Level) for the Catholic School Defining Characteristics Survey of Adults

In conjunction with examination of the eigenvalues and scree plot, statistical tests and indices of model fit are often consulted. All models fit the data well (see Table 16. Model Fit Tests and Indices for EFA Models of the Catholic School Defining Characteristics Survey of Adults). With most indices being within desired ranges for all models under consideration, the philosophical principle known as Occam's razor dictates that one would choose the most parsimonious model, which often is the model with the fewest factors. Consistent with the CFA, model fit indices, in conjunction with the eigenvalues, scree plot and factor pattern matrix, support a model that posits a single factor for explaining and summarizing the Catholic School Program Effectiveness Survey of Adults item responses when aggregated to the school-‐level. This conclusion was corroborated by the less interpretable solutions for the EFA models with two factors (e.g., factor loading > 1, item cross-‐loadings, factor correlation = .82).

Table 16. Model Fit Tests and Indices for EFA Models of the Catholic School Defining Characteristics Survey of Adults

1-‐Factor Model

2-‐Factor Model

Chi-‐Square 120.36 89.66

df 119 103

p-‐value 0.45 0.82

RMSEA 0.002 <0.001

CFI 1.00 1.00

SRMR 0.064 0.044

0

2

4

6

8

10

12

14

16

1 2 3 4 5 6 7 8 9 10

Eigenvalue

Factor

Page | 41

Summary statistics for the distribution of composite score formed by taking the average of school-‐level item averages for the entire Catholic School Defining Characteristics Survey of Adults can be found in Table 17. Descriptive Statistics for a Total Scale Composite Score (School-‐level) Derived from the Catholic School Defining Characteristics Survey of Adults. The theoretical range for the composite and subscale scores is 1 -‐ 5, with 5 reflecting greater alignment with the Defining Characteristics as reported by the respondents. On average, schools tended to score highly on the scale, with 75% or more schools scoring greater than 4.3.

Table 17. Descriptive Statistics for a Total Scale Composite Score (School-level) Derived from the Catholic School Defining Characteristics Survey of Adults



Total Score 4.41 0.25 3.48 4.78 4.30 4.42 4.55 0.98 0.04

The composite scale exhibited excellent reliability (.98), largely due to the survey having a large number of highly correlated items, each being rated by several respondents per school. Based on an estimate of the standard deviation for the composite (scale) and subscale scores and their respective reliability coefficients,


measurement can be used to form confidence bands around scores for specific schools.

Summary and Recommendations Based on the results of these analyses, the Catholic School Defining Characteristics Survey of Adults appears to measure a single construct or factor reflecting the shared perceptions of schools’ alignment with the Defining Characteristics. All items were supported as valid and reliable indicators of the Defining Characteristics construct, and the reliability estimate of a composite score computed by averaging over respondents and items within the same school indicated exceptionally high reliability (i.e., very minimal measurement error). In fact, the attained degree of reliability affords some opportunity to shorten the scale by trimming items that exhibit high rates of missing data (skipped questions and Don’t Know responses) to ease respondent burden while maintaining excellent reliability. A couple items exhibited higher than desired rates of either being skipped by the respondent or receiving a Don’t Know response. Based on the content of the items and pattern of Don’t Know responses, it seems likely that the respondents, particularly non-‐school-‐based adults, to the survey may not have adequate knowledge of the content being tapped by these items. To the extent that these items require knowledge of their school that the respondents may not have, some of the respondents who did respond to the item may have done so with little actual knowledge and either made a “guess” or answered in a way to be consistent with their responses on other items or overall impression of their school. These items should be reviewed and considered for deletion from the survey. It would be desirable to have missing data rates less than 10%. The items with rates above 10% are listed below:

Our school does everything it can to eliminate obstacles that hinder or exclude students from receiving a Catholic education.

Our school operates with the expressed approval and support of our Bishop.

Page | 42

A 5-‐point Likert scale was employed with anchors provided for only the extreme points (Strongly Disagree, Strongly Agree). Adding labels for the intermediate points may (though not necessarily) increase the range of the scale being used by the respondents, which may translate into more variability in ratings and subscale scores across schools. To the extent that the increased variability is reliable variation, adding labels for all points would tend to increase reliability. It might also help standardize the cognitive mapping used by respondents to link scale points to internal sense of agreement with the items. This may increase reliability as random individual differences in this mapping with the current labeling scheme may manifest as measurement error. Importantly, using standard labels (e.g., Strongly Disagree, Disagree, Neither Agree/Disagree, Agree, Strongly Agree) may help achieve item and scale scores that better approximate interval-‐level properties and do so more consistently so across respondents. Ordinal scales only convey information about the rank, whereas interval-‐level measures also convey the amount of difference. Finally, a few cautionary statements pertaining to these analyses are necessary. These analyses were conducted with less than optimal number of schools for assessing the school-‐level psychometric properties of the surveys. Although the large number of respondents within each participating school and the generally high reliability (and factor loadings) of most items facilitated the analysis, it would have been desirable to have at least 150-‐200 schools. Related and also impinging on the results is that the high non-‐response rate of schools introduces the possibility of self-‐selection effects. The implication of this is that the obtained sample may not be representative of the broader population of schools. Therefore, the results may not fully generalize to that broader population.

Page | 43



A total of 3,973 respondents from 43 schools (average of 92 respondents per school) completed the Catholic School Defining Characteristics Survey of Students (5th -‐ 8th grades) for this psychometric study. Participating students were nearly evenly distributed across grade levels (21.7% to 27.5% across the four grades). Just over half of respondents were female (55%), and the racial composition of the sample was: White (81.7%), African American or Black (5.5%), multi-‐racial (7%), Asian (4.5%), American Indian or Alaska Native (.9%), or Native Hawaiian or Pacific Islander (.4%). The majority of respondents identified as non-‐Hispanic (92.4%). Most respondents expressed their religious affiliation as Catholic (87.1%). Most respondents also reported being affiliated with the school for more than 4 years (71.6%).


Response frequencies and intraclass correlation coefficients (ICCs) for each of the 17 items are reported in Table B.4. Item Response Frequencies and Intraclass Correlation Coefficients (ICC) for the Catholic School Defining Characteristics Survey of Students (5-‐8 Grades) in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics SurveysAppendix . On 13 of the items, at least 75% of middle school students endorsed the Strongly Agree or adjacent response options, and all items except one were endorsed (Strongly Agree or adjacent response) by at least 50% of respondents. Though response frequencies are clustered towards endorsement of options indicating greater agreement, there was some degree of variability of frequency patterns across items. Item percentages for skipped or Don't Know responses ranged between 2% and 12.6% (Mean = 4.5%). One item (viz., Our school operates with the expressed approval and support of our Bishop) had both the highest Don’t Know (9.4%) and the highest skip (3.1%) rates. These rates were much higher for this item than other items, indicating that respondents did not feel they had sufficient knowledge to provide their rating for this item. This item should be considered for rewording, replacement or deletion from the scale.28 ICCs quantify the proportion of variability in responses that is attributable to variability between scores. In other terms, an ICC reflects the degree of non-‐independence amongst responses from students affiliated with the same school, with 0 = independence (i.e., no systematic variation across schools) and 1 = complete dependence (i.e., all variation in responses is due to differences across schools). As this survey is intended to measure school-‐level alignment with the Defining Characteristics, ICCs greater than zero are to be expected and desired. Additionally, ICCs greater than .01 support the need for statistical methods that account for the observed non-‐independence. The ICCs for most items were moderate in magnitude, reflecting similarity in responses from adults affiliated the same school and justifying the need for statistical methods that can account for non-‐independence amongst the observations. ICCs across all items ranged

28 Although most analyses described in the Factorial Validity and Reliability use statistical procedures that can incorporate respondents that are missing data on one or more items, 62 respondents missing data on all items were necessarily omitted from these analyses.

Page | 44

from .04 to .18 (Mean = .08), indicating that between 4% and 18% of variation in item responses was attributable to respondents being affiliated with different schools.

The Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) is intended to provide data on Catholic schools’ adherence to the Standards. Given this intent, the focus of these analyses is on school-‐level aggregations of the responses. Table B.5. School-‐level Statistics for the Catholic School Defining Characteristics Survey of Students (5-‐8 Grades) in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics Surveys provides statistics to describe the distributions of these school-‐level aggregates, where student responses for each item are averaged with other students from their school. The theoretical range for these aggregated item averages is 1 (Strongly Disagree) -‐ 5 (Strongly Agree), with higher scores indicating higher within-‐school average agreement for the particular item. On average, schools were rated highly on all of the items (mean average rating across items = 4.24) and minimum ratings averaged 2.91. These statistics indicate little variability in mean ratings across schools for the items, with some items (#’s: 1, 4, 5, and 9) exhibiting little variability. Along with average, minimum, and maximum ratings, standard deviations and quartiles are also provided for each item. The median (also known as the 50th percentile or second quartile) reflects the item score at which 50% of schools fall at or below. For all items, 50% or more of the schools scored at least a 3.31. Similarly to the respondent-‐level item statistics, scores were clustered toward the upper end of the distribution.


A multilevel, ordinal confirmatory factor analyses (CFA) was performed on the Catholic School Defining Characteristics Survey of Students (5th -‐ 8th grades). A single-‐factor model of the survey was fit to the data.29 This model fit the data well by both exact fit and most approximate fit standards [χ2(119) = 114.68, p = .60, RMSEA < 0.001, CFI = 1.00, SRMR = 0.078]Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Adults subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with seven to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Adults. CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model). The model fit tests and indices for these analyses are reported in Table 1Error! Not a valid bookmark self-‐reference.. These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration. Table . Still, it is essential that the one-‐factor model solution be considered meaningful and interpretable where the meaningfulness or interpretability of a solution is determined by considering the strength and pattern of relationships between the items and underlying (latent) factor. The relationships between factors and indicators are typically depicted in a factor loading matrix. Standardized factor loadings and associated standard errors for the one-‐factor CFA model are provided in Table in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics Surveys. The standardized factor


Page | 45

loadings were uniformly very high, with most loading values greater than 0.8 and none below 0.47. These numbers indicate that each item is a salient and highly reliable measure of the factor. Though the CFA analysis support a one-‐factor model, it is possible that another k-‐factor generated the data. Therefore, an exploratory factor analysis (EFA) was performed prior to settling on the one-‐factor model of the Catholic School Defining Characteristics Survey of Students (5th -‐ 8th grades). An important and initial task in conducting exploratory factor analysis is to determine the number of factors or dimensions that are being measured by the survey instrument. This determination is guided by statistical tests and indices, evaluation of eigenvalues, and meaningfulness and interpretability of the solution. As with the confirmatory factor analyses, there are two-‐levels under consideration, respondent-‐level and the school-‐level, and the number of factors would typically be determined at each level. However, the focus of this report is on the school as the unit of analysis. Eigenvalues quantify the variance in the item responses that is explained by the factors. Factors that account for more variation are considered potentially more important or meaningful than factors that account for less variation. Eigenvalues from the school-‐level factor analysis of the survey data are reported in Table 18. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Defining Characteristics Survey of Students (5th – 8th grades). An often cited rule of thumb is the Kaiser rule, which states that factors with eigenvalues greater than 1.0 should be extracted. According to the Kaiser rule, a model with three factors should be examined. This rule, however, has been criticized as leading to extraction of too many factors and thus typically should be considered an upper bound estimate of the number of factors.

Table 18. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Defining Characteristics Survey of Students (5th – 8th grades)

Factor 1 2 3 4 5 6 7 8 9 10 ...

Eigenvalues 11.86 1.23 1.07 0.86 0.53 0.51 0.41 0.22 0.19 0.18

Another factor enumeration approach used is examination of the Scree plot, depicted in Figure 5. Scree Plot of Eigenvalues (School-‐Level) for the Catholic School Defining Characteristics Survey of Students (5th – 8th grades). With this approach, one typically seeks the point where there is a pronounced bend (elbow) in the curve. Factors before the bend are given further consideration. According to the scree plot, a one-‐factor or perhaps a two-‐factor model should be considered for extraction. Although more sophisticated methods for factor enumeration exists (e.g., parallel analysis), these were computationally infeasible or inaccessible for the present analysis, which involve a multilevel structure and ordinal item responses distributions. Thus, the one-‐ and two-‐factor models were given closer consideration.

Page | 46

Figure 5. Scree Plot of Eigenvalues (School-Level) for the Catholic School Defining Characteristics Survey of Students (5th – 8th grades)

In conjunction with examination of the eigenvalues and scree plot, statistical tests and indices of model fit are often consulted. All models fit the data well (see Table 19. Model Fit Tests and Indices for EFA Models of the Catholic School Defining Characteristics Survey of Students (5th – 8th grades)). With most indices being within desired ranges for all models under consideration, the philosophical principle known as Occam's razor dictates that one would choose the most parsimonious model, which often is the model with the fewest factors. Consistent with the CFA, model fit indices, in conjunction with the eigenvalues, scree plot and factor pattern matrix, support a model that posits a single factor for explaining and summarizing the Catholic School Program Effectiveness Survey of Students (5th -‐ 8th grades) item responses when aggregated to the school-‐level. This conclusion was corroborated by the less interpretable solutions for the EFA model with two factors (e.g., factor loading > 1, item cross-‐loadings, high factor correlation = .65).

Table 19. Model Fit Tests and Indices for EFA Models of the Catholic School Defining Characteristics Survey of Students (5th – 8th grades)

1-‐Factor Model

2-‐Factor Model

Chi-‐Square 114.68 90.87

df 119 103

p-‐value 0.60 0.80

RMSEA < .0001 <.0001

CFI 1.00 1.00

SRMR 0.078 0.062

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10

Eigenvalue

Factor

Page | 47

Summary statistics for the distribution of composite score formed by taking the average of school-‐level item averages for the entire Catholic School Defining Characteristics Survey of Students (5th -‐ 8th grades) can be found in Table 20. Descriptive Statistics for a Total Scale Composite Score (School-‐level) Derived from the Catholic School Defining Characteristics Survey of Students (5th – 8th grades). The theoretical range for the composite scores is 1 -‐ 5, with 5 reflecting greater alignment with the Defining Characteristics as reported by students. On average, schools tended to score highly on the scale, with 75% or more schools scoring greater than 4.12.

Table 20. Descriptive Statistics for a Total Scale Composite Score (School-level) Derived from the Catholic School Defining Characteristics Survey of Students (5th – 8th grades)



Total Score 4.24 0.30 3.06 4.77 4.12 4.25 4.42 0.97 0.05





Based on the results of these analyses, the Catholic School Defining Characteristics Survey of Students (5th -‐ 8th grades) appears to measure a single construct or factor reflecting the shared perceptions of schools’ alignment with the Defining Characteristics. All items were supported as valid and reliable indicators of the Defining Characteristics construct, and the reliability estimate of a composite score computed by averaging over respondents and items within the same school indicated exceptionally high reliability (i.e., very minimal measurement error). In fact, the attained degree of reliability affords some opportunity to shorten the scale by trimming items that exhibit high rates of missing data (skipped questions and Don’t Know responses) to ease respondent burden while maintaining excellent reliability. One item (viz., Our school operates with the expressed approval and support of our Bishop) exhibited a higher than desired rate of either being skipped by the respondent or receiving a Don’t Know response. It seems likely that the respondents to the survey may not have adequate knowledge of the content being tapped by this item. To the extent that this items require knowledge of their school that the respondents may not have, some of the respondents who did respond to the item may have done so with little actual knowledge and either made a “guess” or answered in a way to be consistent with their responses on other items or overall impression of their school. This item should be reviewed and considered for deletion from the survey. It would be desirable to have missing data rates less than 10%. A 5-‐point Likert scale was employed with anchors provided for only the extreme points (Strongly Disagree, Strongly Agree). Adding labels for the intermediate points may (though not necessarily) increase the range of the scale being used by the respondents, which may translate into more variability in ratings and subscale scores across schools. To the extent that the increased variability is reliable variation, adding

Page | 48

labels for all points would tend to increase reliability. It might also help standardize the cognitive mapping used by respondents to link scale points to internal sense of agreement with the items. This may increase reliability as random individual differences in this mapping with the current labeling scheme may manifest as measurement error. Importantly, using standard labels (e.g., Strongly Disagree, Disagree, Neither Agree/Disagree, Agree, Strongly Agree) may help achieve item and scale scores that better approximate interval-‐level properties and do so more consistently so across respondents. Ordinal scales only convey information about the rank, whereas interval-‐level measures also convey the amount of difference. Finally, a few cautionary statements pertaining to these analyses are necessary. These analyses were conducted with less than optimal number of schools for assessing the school-‐level psychometric properties of the surveys. Although the large number of respondents within each participating school and the generally high reliability (and factor loadings) of most items facilitated the analysis, it would have been desirable to have at least 150-‐200 schools. Related and also impinging on the results is that the high non-‐response rate of schools introduces the possibility of self-‐selection effects. The implication of this is that the obtained sample may not be representative of the broader population of schools. Therefore, the results may not fully generalize to that broader population.

Page | 49



A total of 128 respondents from 5 schools (average of 26 respondents per school) completed the Catholic School Defining Characteristics Survey of Students (9th -‐ 12th grades) for this psychometric study. Participating students were mostly in the 9th (42%) or 11th (25%) grades. The majority of participants were female (66.4%), and the racial composition of the sample was: White (82%), African American or Black (9.4%), multi-‐racial (5.5%), or Asian (3.1%). The majority of respondents identified as non-‐Hispanic (94.5%). Most respondents expressed their religious affiliation as Catholic (77.3%). Respondents were fairly uniformly distributed in their number of years attending the school (10.9% -‐ 21.1%).


Response frequencies for each of the 17 items are reported in Table B.7. Item Response Frequencies for the Catholic School Defining Characteristics Survey of Students (9-‐12 Grades) in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics SurveysAppendix . On all items, at least 75% of high school students endorsed the Strongly Agree or adjacent response options. Though response frequencies are clustered towards endorsement of options indicating greater agreement, there was some degree of variability of frequency patterns across items. Item percentages for skipped or Don't Know responses ranged between 5.5% and 12.5% (Mean = 7.0%). The items with more than 10% missing data were: (a) The adults in our school show that they are partners with our parents/guardians in our Catholic education, (b) Our school does everything it can to eliminate obstacles that hinder or exclude students from receiving a Catholic education, and (c) Our school operates with the expressed approval and support of our Bishop. These items may ask respondents about aspects which they have little or no knowledge on which to base a rating. These items were among a few items with a non-‐zero rate of Don’t Know response. These items should be considered for rewording, replacement or deletion from the scale.30

The Catholic School Defining Characteristics Survey of Students (9th -‐ 12th grades) is intended to provide data on Catholic schools’ alignment to the guiding Defining Characteristics of Catholic schools. Given this intent, the desired focus of analysis is on school-‐level aggregations of the responses. However, too few high schools were successfully recruited for this psychometric study to permit school-‐level analyses. Therefore, all subsequent analyses report on analyses at the respondent-‐level. In interpreting the results of these analyses, understand that these results may not extend to the school-‐level and any such generalization is discouraged.31

30 Although most analyses described in the Factorial Validity and Reliability use statistical procedures that can incorporate respondents that are missing data on one or more items, seven respondents missing data on all items were necessarily omitted from these analyses. 31 In addition, the respondent-‐level sample size was particularly modest for these analyses. Caution is warranted when interpreting the results until more data is available.

Page | 50


An ordinal confirmatory factor analyses (CFA) was performed on the Catholic School Defining Characteristics Survey of Students (9th -‐ 12th grades). A single-‐factor model of the survey was fit to the data. The model exhibited statistically significant misfit and mixed results on the approximate fit indices [χ2(119) = 237.01, p < .001, RMSEA = 0.091, CFI = 0.97, SRMR = 0.057]Multilevel, ordinal confirmatory factor analyses (CFA) began with the separate examination of Catholic School Program Effectiveness Survey of Adults subscales corresponding to each of the four Standards for Catholic School Program Effectiveness. These CFA models, consisting of one-‐factor with seven to fourteen indicators each, were fit to the data. Then, these models were combined into a single, four-‐factor CFA model of the Catholic School Program Effectiveness Survey of Adults. CFA concluded with estimation of a model where all items were specified as measures of a single factor (one-‐factor CFA model). The model fit tests and indices for these analyses are reported in Table 1Error! Not a valid bookmark self-‐reference.. These tests evaluate how well each model fits the data. The aim is to identify a model that fits the data well (within sampling error) or approximately well and is parsimonious. If a model fits the data poorly, the model would be rejected from further consideration. Table . Modification indices and residuals were examined, but these were not informative as to source of misfit. Standardized factor loadings and associated standard errors for this model are provided in Table in Appendix B: Tables from the Item-‐level Analyses of the Catholic School – Defining Characteristics Surveys, but they should be interpreted with caution given the evidence of possible misfit. The standardized factor loadings were uniformly very high, with most loading values greater than 0.8 and none below 0.75. It is possible that another k-‐factor generated the data. Therefore, an exploratory factor analysis (EFA) was performed. An important and initial task in conducting exploratory factor analysis is to determine the number of factors or dimensions that are being measured by the survey instrument. This determination is guided by statistical tests and indices, evaluation of eigenvalues, and meaningfulness and interpretability of the solution. As with the confirmatory factor analyses, there are two-‐levels under consideration, respondent-‐level and the school-‐level, and the number of factors would typically be determined at each level. However, the focus of this report is on the school as the unit of analysis. Eigenvalues quantify the variance in the item responses that is explained by the factors. Factors that account for more variation are considered potentially more important or meaningful than factors that account for less variation. Eigenvalues from the school-‐level factor analysis of the survey data are reported in Table 21. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Defining Characteristics Survey of Students (9th – 12th grades). An often cited rule of thumb is the Kaiser rule, which states that factors with eigenvalues greater than 1.0 should be extracted. According to the Kaiser rule, a model with one factor should be examined further.

Table 21. Eigenvalues for the Exploratory Factor Analysis of the Catholic School Defining Characteristics Survey of Students (9th – 12th grades)

Factor 1 2 3 4 5 6 7 8 9 10 ...

Eigenvalues 12.36 0.94 0.65 0.49 0.46 0.43 0.37 0.28 0.26 0.2 …

Another factor enumeration approach used is examination of the Scree plot, depicted in Figure 6. Scree Plot of Eigenvalues (School-‐Level) for the Catholic School Defining Characteristics Survey of Students (9th –

Page | 51

12th grades). With this approach, one typically seeks the point where there is a pronounced bend (elbow) in the curve. Factors before the bend are given further consideration. According to the scree plot, a one-‐factor model should be considered. Although more sophisticated methods for factor enumeration exists (e.g., parallel analysis), these were computationally infeasible or inaccessible for the present analysis, which involve ordinal item responses distributions that make it difficult to employ more advanced methods. This model is identical to the one-‐factor CFA model.

Figure 6. Scree Plot of Eigenvalues (School-Level) for the Catholic School Defining Characteristics Survey of Students (9th – 12th grades)

For comparison, the fit indices for the one-‐ and two-‐factor EFA models are provided in Table 22. Model Fit Tests and Indices for EFA Models of the Catholic School Defining Characteristics Survey of Students (9th – 12th grades). Although the one-‐factor model exhibits evidence of misfit, the EFA results provide more support for this model over another k-‐class model.

Table 22. Model Fit Tests and Indices for EFA Models of the Catholic School Defining Characteristics Survey of Students (9th – 12th grades)

1-‐Factor Model

2-‐Factor Model

Chi-‐Square 237.01 162.01

df 119 103

p-‐value < .0001 0.0002

RMSEA 0.091 0.069

CFI 0.98 0.99

SRMR 0.057 0.040

0

2

4

6

8

10

12

14

1 2 3 4 5 6 7 8 9 10

Eigenvalue

Factor

Page | 52

Summary statistics for the distribution of composite score formed by taking the average of student-‐level item averages for the entire Catholic School Defining Characteristics Survey of Students (9th -‐ 12th grades) can be found in Table 23. Descriptive Statistics for a Total Scale Composite Score (Student-‐level) Derived from the Catholic School Defining Characteristics Survey of Students (9th – 12th grades). The theoretical range for the composite is 1 -‐ 5, with 5 reflecting greater alignment with the Defining Characteristics as reported by students. On average, students tended to rate their schools highly on the scale, with 75% or more students rating their school higher than 4.00.

Table 23. Descriptive Statistics for a Total Scale Composite Score (Student-level) Derived from the Catholic School Defining Characteristics Survey of Students (9th – 12th grades)



Total Score 4.41 0.65 1.00 5.00 4.00 4.59 4.99 0.98 0.09





Based on the results of these analyses, the Catholic School Defining Characteristics Survey of Students (9th -‐ 12th grades) appears to measure a single construct or factor reflecting the shared perceptions of schools’ alignment with the Defining Characteristics, through model fit tests and statistics were not conclusive. All items were supported as valid and reliable indicators of the Defining Characteristics construct as measured at the respondent level, and the reliability estimate of a composite score computed by averaging over items generally indicated high reliability (i.e., very minimal measurement error). In fact, the attained degree of reliability affords some opportunity to shorten the scale by trimming items that exhibit high rates of missing data (skipped questions and Don’t Know responses) to ease respondent burden while maintaining excellent reliability. However, limitations of the 9th – 12th grade student data on this survey limits the conclusions that can be drawn from these results. Too few high schools were recruited to permit analyses at the school-‐level, the desired level of inference for these surveys. Analyses were thus conducted at the respondent (i.e., student) level, but it is very possible that the factor structure and reliability of the instrument can differ at the school-‐level from that at the student-‐level. Moreover, the sample size was modest and less than optimal for the type of analyses being conducted. As such, caution is warranted in generalizing the results beyond the schools and respondents sampled to the broader population of Catholic high schools. It is recommended that a multi-‐level factor analysis be conducted on data collected from a larger administration of this survey involving at least 100-‐200 schools. A few items exhibited higher than desired rates of either being skipped by the respondent or receiving a Don’t Know response. Based on the content of the items and pattern of Don’t Know responses, it seems likely that the respondents, particularly non-‐school-‐based adults, to the survey may not have adequate

Page | 53

knowledge of the content being tapped by these items. To the extent that these items require knowledge of their school that the respondents may not have, some of the respondents who did respond to the item may have done so with little actual knowledge and either made a “guess” or answered in a way to be consistent with their responses on other items or overall impression of their school. These items should be reviewed and considered for deletion from the survey. It would be desirable to have missing data rates less than 10%. The items with rates above 10% are listed below:

The adults in our school show that they are partners with our parents/guardians in our Catholic education.

Our school does everything it can to eliminate obstacles that hinder or exclude students from receiving a Catholic education.

Our school operates with the expressed approval and support of our Bishop.

A 5-‐point Likert scale was employed with anchors provided for only the extreme points (Strongly Disagree, Strongly Agree). Adding labels for the intermediate points may (though not necessarily) increase the range of the scale being used by the respondents, which may translate into more variability in ratings and subscale scores across schools. To the extent that the increased variability is reliable variation, adding labels for all points would tend to increase reliability. It might also help standardize the cognitive mapping used by respondents to link scale points to internal sense of agreement with the items. This may increase reliability as random individual differences in this mapping with the current labeling scheme may manifest as measurement error. Importantly, using standard labels (e.g., Strongly Disagree, Disagree, Neither Agree/Disagree, Agree, Strongly Agree) may help achieve item and scale scores that better approximate interval-‐level properties and do so more consistently so across respondents. Ordinal scales only convey information about the rank, whereas interval-‐level measures also convey the amount of difference.

Page | 54

Appendix Appendix A: Tables from the Item-level Analyses of the Catholic School – Program Effectiveness Surveys

Table A.1. Item Response Percentages and Intraclass Correlation Coefficients (ICC) for the Catholic School Program Effectiveness Survey of Adults

Subscale Item

ICC Strongly Disagree

Strongly Agree

Don't Know

Skipped

Mission and Catholic Identity 1. Everyone in the school community –

administrators, faculty and staff, students, parents/guardians, alums, and supporters – knows and understands the school’s mission.

0.08 2.2% 3.7% 12.7% 35.5% 45.8% 2.8% 8.2%

2. Everything we do in our school is guided and directed by our mission.

0.09 2.2% 4.7% 14.1% 34.7% 44.2% 2.4% 8.4%

3. Our school mission clearly expresses a commitment to Catholic identity.

0.08 1.4% 2.1% 5.9% 25.2% 65.3% 1.4% 8.6%

4. Our school provides an academically rigorous Catholic religion program, taught by qualified teachers.

0.09 2.5% 3.8% 10.0% 29.3% 54.4% 1.4% 8.5%

5. In all subjects, teachers help students think critically and ethically about the world around them, using the lens of Gospel values and Catholic doctrine and beliefs.

0.06 2.4% 4.4% 14.3% 36.0% 42.8% 3.0% 8.4%

6. Our school provides opportunities outside the classroom for students to participate in service activities for social justice.

0.09 1.7% 3.5% 10.3% 28.7% 55.9% 2.2% 8.5%

7. Administrators, faculty, and staff serve as role models of faith and service to students.

0.09 3.2% 6.7% 16.6% 30.2% 43.3% 5.6% 8.4%

8. Our school provides opportunities outside the classroom for student faith formation, and participation in retreats, prayer, mass, sacraments, and other spiritual experiences.

0.06 2.6% 4.1% 11.8% 31.4% 50.1% 1.0% 8.7%

9. Our school provides opportunities for faith formation for faculty and staff.

0.05 1.2% 3.6% 11.9% 31.0% 52.3% 19.7% 8.8%

10. Our school provides opportunities for faith formation for parents/guardians and other adult members of the school community.

0.06 4.0% 8.6% 20.6% 29.9% 36.9% 7.1% 8.8%

11. Our school helps parents/guardians support the faith life of their child.

0.06 2.0% 4.7% 13.8% 30.6% 48.8% 1.2% 8.8%

Page | 55

12. Our school provides opportunities for adult members of the school community to participate in service activities for social justice.

0.03 4.5% 8.2% 20.9% 30.3% 36.1% 10.8% 8.8%

13. Every adult in the school supports the faith life of the school community.

0.05 4.4% 8.0% 19.2% 32.1% 36.3% 9.4% 8.9%

Governance and Leadership 14. There is a person or group (such as a

pastor or a board) who collaborates with the school administration to make or recommend decisions for the success of the school.

0.08 2.1% 2.8% 9.6% 27.7% 57.9% 10.2% 11.6%

15. A person or group (such as a pastor or a board), in collaboration with the school administration, takes responsibility for monitoring that the school is faithful to its mission, academically excellent and sound in its business decisions.

0.09 2.8% 3.3% 12.1% 29.3% 52.5% 12.0% 11.8%

16. Our school administration effectively carries out its responsibilities in the areas of faith formation and instructional leadership.

0.09 2.5% 3.9% 10.7% 31.0% 51.9% 2.9% 11.8%

17. Our school administration has authority to realize and implement the school’s mission and vision.

0.07 1.3% 2.8% 9.4% 28.6% 57.9% 5.6% 12.2%

18. Our school administration involves all members of the school community to ensure a school culture that embodies the mission and vision.

0.07 4.4% 5.9% 14.3% 28.1% 47.2% 5.0% 12.1%

19. Our school administration takes responsibility for development and oversight of school programs, personnel, and school operations.

0.09 3.2% 4.1% 10.9% 28.1% 53.8% 3.6% 12.1%

20. Our school has a strong culture of collaboration within the school at all levels to advance excellence.

0.10 4.4% 5.8% 13.1% 29.6% 47.0% 4.3% 12.1%

Academic Excellence 21. Our school has a clearly articulated

rigorous curriculum infused with Gospel values, preparing students for life and work.

0.07 2.4% 3.9% 13.3% 31.7% 48.7% 1.7% 16.6%

22. In classes in our school, students spend most of the time solving problems, discussing ideas, creating their own work, reading, writing, speaking, and researching.

0.04 2.7% 5.0% 14.3% 33.7% 44.3% 4.3% 16.7%

23. Curriculum and instruction in our school prepares students to be capable and critical users of media and technology.

0.11 3.6% 6.2% 13.7% 33.3% 43.1% 1.6% 16.7%

24. Teachers use effective instruction to deliver the curriculum.

0.08 2.5% 5.1% 13.3% 34.5% 44.6% 2.0% 16.8%

Page | 56

25. At our school, teachers use different teaching approaches to meet the diverse needs of all students.

0.05 5.5% 7.1% 14.7% 31.5% 41.1% 6.9% 16.8%

26. At our school, teachers collaborate systematically and regularly in order to increase student achievement and improve teaching effectiveness.

0.06 3.7% 5.8% 14.9% 31.4% 44.2% 10.6% 17.0%

27. At our school, all administrators, faculty and staff engage in ongoing professional development.

0.09 2.6% 3.2% 11.4% 30.0% 52.8% 13.3% 16.7%

28. Our school uses standardized and teacher-‐developed assessments to document student learning and report the outcomes to parents/guardians.

0.07 2.1% 3.4% 9.6% 29.7% 55.1% 5.2% 16.9%

29. Our school uses the results of standardized and teacher-‐developed assessments to improve the curriculum and increase learning.

0.09 3.2% 4.2% 13.1% 29.1% 50.4% 12.3% 16.8%

30. Teachers vary the types of assessments they use to monitor individual and class-‐wide student learning.

0.05 4.7% 6.0% 14.8% 33.1% 41.4% 13.0% 17.1%

31. Our school communicates how well students are achieving in comparison to similar students locally and/or nationally.

0.09 8.5% 8.8% 16.5% 27.6% 38.6% 5.4% 17.0%

32. Our school provides programs and services that help students successfully complete the school program (for example, guidance and resource programs).

0.09 5.2% 7.2% 16.2% 28.5% 42.9% 4.6% 16.9%

33. Our school provides enriching programs for students to develop their gifts and talents, and enhance their creative, artistic, social/emotional, physical, and spiritual potential.

0.08 5.5% 7.0% 15.7% 28.2% 43.5% 1.7% 17.1%

34. Our school provides opportunities for parents/guardians to be involved in the education of their children.

0.05 4.0% 5.8% 13.1% 27.6% 49.5% 1.3% 17.0%

Operational Vitality 35. Our school’s financial plan is the result

of a collaborative process including expert advisors.

0.11 5.2% 6.2% 14.7% 33.3% 40.5% 25.1% 18.0%

36. Our school consistently shares its financial plan with the school community.

0.11 10.7% 11.2% 19.7% 26.6% 31.8% 10.7% 18.0%

37. Our school leaders take responsibility for ensuring that the financial plans and budgets are implemented using best practices.

0.10 4.3% 5.9% 14.5% 31.5% 43.7% 19.0% 18.1%

38. Our school treats all personnel with consistency, fairness, and justice.

0.08 6.3% 6.6% 13.3% 28.5% 45.2% 14.1% 18.0%

Page | 57

39. Our school maintains and shares plans for managing facilities and equipment.

0.10 7.9% 8.7% 17.3% 29.4% 36.7% 12.7% 18.2%

40. Our school maintains and shares a technology management plan.

0.08 8.2% 9.0% 18.1% 27.9% 36.8% 14.9% 18.4%

41. Our school’s facilities, equipment, and technology management plans are designed to enhance teaching and learning.

0.11 3.9% 5.8% 16.3% 31.0% 43.0% 11.0% 18.2%

42. Our school has an institutional advancement plan, based on our mission, which uses current and effective strategies for communications, marketing, enrollment management, and development.

0.11 5.7% 6.0% 16.5% 31.0% 40.8% 20.8% 18.2%

Note. Percentages for Never through Always options are mutually exclusive and exhaustive and sum to 100%; participants who selected Don't Know or skipped the question were excluded from the denominator in calculation of these percentages. The percentages reported for Don't Know or skipped responses include all participants in the denominator.

Table A.2. School-level Statistics for the Catholic School Program Effectiveness Survey of Adults

Subscale Item

Average Rating

Standard Deviation

Minimum Rating

Maximum Rating

25th Percentile

Median 75th

Percentile



4.15 0.35 2.90 4.67 4.00 4.22 4.41


4.11 0.36 2.82 4.57 3.98 4.14 4.35


4.51 0.25 3.75 4.89 4.40 4.52 4.69


4.29 0.34 3.25 5.00 4.13 4.29 4.53


4.13 0.33 3.00 4.75 4.00 4.15 4.37


4.37 0.29 3.75 4.92 4.15 4.37 4.57


4.04 0.33 3.39 4.80 3.84 4.01 4.29

8. Our school provides opportunities outside the classroom for student faith formation, and participation in retreats, prayer, mass, sacraments, and other spiritual

4.25 0.31 3.33 4.76 4.10 4.25 4.45

Page | 58

experiences.


4.31 0.26 3.57 4.83 4.17 4.32 4.49


3.88 0.34 3.14 4.75 3.65 3.84 4.10


4.20 0.30 3.43 4.82 4.02 4.22 4.42


3.85 0.30 3.29 4.67 3.59 3.86 4.05


3.90 0.33 2.91 4.83 3.73 3.89 4.10


14. There is a person or group (such as a pastor or a board) who collaborates with the school administration to make or recommend decisions for the success of the school.

4.34 0.31 2.77 4.76 4.27 4.39 4.48


4.22 0.36 2.76 4.82 4.06 4.29 4.43


4.25 0.35 2.83 4.75 4.09 4.29 4.50


4.37 0.28 3.25 4.81 4.28 4.41 4.54


4.08 0.37 2.60 4.80 3.93 4.08 4.30


4.23 0.38 2.92 4.82 4.11 4.24 4.50


4.08 0.40 2.83 4.75 3.91 4.09 4.36

Academic Excellence

21. Our school has a clearly articulated rigorous curriculum infused with Gospel values, preparing students for life and work.

4.19 0.31 3.18 4.82 4.00 4.21 4.43

22. In classes in our school, students spend most of the time solving problems,

4.12 0.29 3.09 4.68 3.97 4.14 4.32

Page | 59

discussing ideas, creating their own work, reading, writing, speaking, and researching.


4.02 0.40 2.91 4.72 3.78 4.10 4.29


4.14 0.37 2.50 4.76 3.99 4.15 4.34


3.99 0.38 2.70 4.67 3.83 4.02 4.20


4.07 0.39 2.64 4.80 3.94 4.13 4.28


4.26 0.36 2.92 4.80 4.15 4.36 4.47


4.33 0.32 3.40 4.91 4.20 4.39 4.53


4.20 0.39 3.00 5.00 4.05 4.25 4.41


4.03 0.36 2.91 5.00 3.88 4.08 4.21


3.82 0.46 2.42 4.50 3.60 3.91 4.18


3.93 0.41 2.55 4.73 3.75 3.94 4.21


3.93 0.36 2.91 4.64 3.73 3.98 4.20


4.13 0.38 2.64 4.75 4.00 4.21 4.35


35. Our school’s financial plan is the result of a collaborative process including expert advisors.

3.92 0.43 2.48 4.67 3.67 3.95 4.26

36. Our school consistently shares its financial plan with the school community.

3.54 0.48 2.50 4.65 3.20 3.59 3.87

37. Our school leaders take responsibility for ensuring that the financial plans and budgets are implemented using best

4.02 0.41 2.52 4.79 3.80 4.10 4.27

Page | 60

practices.


3.96 0.44 2.55 4.80 3.82 4.05 4.26


3.76 0.44 2.31 4.76 3.61 3.81 4.00


3.75 0.44 2.57 4.50 3.52 3.79 4.13


4.00 0.40 3.08 4.67 3.74 4.11 4.31


3.89 0.45 2.45 4.67 3.67 3.95 4.21

Table A.3. Standardized Factor Loadings for the 1-factor CFA Model (School-level) of the Catholic School Program Effectiveness Survey of Adults

Full-‐Scale 1-‐Factor Model Subscale 1-‐Factor Models

Subscale Item)

Standardized Factor Loading

Standard Error


Standard Error



0.90 0.04 0.89 0.05


0.96 0.02 0.97 0.03


0.94 0.03 0.99 0.03


0.93 0.04 0.92 0.04


0.87 0.05 0.91 0.04


0.68 0.08 0.75 0.07


0.60 0.10 0.58 0.11

Page | 61


0.93 0.03 0.95 0.03


0.82 0.07 0.86 0.06


0.58 0.09 0.67 0.08


0.82 0.06 0.95 0.02


0.85 0.07 0.88 0.07


0.91 0.04 0.97 0.03


pastor or a board) who collaborates with the school administration to make or recommend decisions for the success of the school.

0.80 0.07 0.86 0.05


0.89 0.04 0.94 0.03


0.95 0.02 0.96 0.02


0.98 0.03 0.96 0.02


0.97 0.02 0.98 0.02


0.98 0.02 0.98 0.02


0.98 0.02 0.96 0.02

Academic Excellence

21. Our school has a clearly articulated rigorous curriculum infused with Gospel values, preparing students for life and

0.89 0.04 0.89 0.03

Page | 62

work. 22. In classes in our school, students spend

most of the time solving problems, discussing ideas, creating their own work, reading, writing, speaking, and researching.

0.85 0.06 0.93 0.04


0.86 0.05 0.86 0.05


0.89 0.04 0.94 0.03


0.88 0.05 0.96 0.03


0.91 0.04 0.94 0.03


0.90 0.04 0.85 0.05


0.90 0.04 0.94 0.03


0.87 0.05 0.95 0.03


0.86 0.05 0.96 0.03


0.77 0.06 0.87 0.05


0.71 0.07 0.71 0.08


0.75 0.07 0.73 0.07


0.88 0.04 0.86 0.05

Operational Vitality 35. Our school’s financial plan is the result of

a collaborative process including expert advisors.

0.93 0.04 0.94 0.03

36. Our school consistently shares its 0.78 0.06 0.88 0.04

Page | 63

financial plan with the school community.

37. Our school leaders take responsibility for ensuring that the financial plans and budgets are implemented using best practices.

0.96 0.02 0.96 0.02


0.83 0.05 0.78 0.06


0.89 0.03 0.97 0.01


0.84 0.06 0.95 0.03


0.83 0.06 0.93 0.04


0.89 0.04 0.95 0.02

aThe error variance for this item was original estimated in the subscale CFA analysis as a negative number and therefore was subsequently constrained to a small positive constant (i.e., .001).

Table A.4. Item Response Frequencies and Intraclass Correlation Coefficients (ICC) for the Catholic School Program Effectiveness Survey of Students (5-8 Grades)

Subscale Item

ICC Strongly Disagree

Strongly Agree

Don't Know

Skipped


the principal, teachers and staff, students, and parents/guardians – knows and understands the school’s mission.

0.04 2.8% 4.9% 16.7% 36.3% 39.4% 9.6% 3.2%

2. Our school’s mission affects everything we do.

0.03 5.4% 8.8% 20.9% 35.2% 29.7% 11.6% 3.6%

3. Our school’s mission statement clearly says that we are a Catholic school.

0.09 1.1% 2.1% 5.8% 17.8% 73.2% 4.9% 4.7%

4. Religion class in our school is very important, and is taught by knowledgeable teachers.

0.11 2.9% 3.7% 7.3% 20.3% 65.7% 0.8% 3.9%

5. In all subjects, teachers help students think about how Gospel values and Catholic beliefs can help to make the world a better place.

0.06 8.1% 13.3% 23.8% 29.0% 25.8% 3.7% 3.8%

6. Our school provides opportunities outside the classroom for students to develop their faith and participate in Mass, sacraments and prayer.

0.06 3.1% 5.3% 11.2% 27.4% 53.0% 2.9% 4.4%

Page | 64

7. Our school provides opportunities outside the classroom for students to do service on behalf of the poor and those in need.

0.04 5.2% 7.5% 15.3% 28.8% 43.3% 4.9% 3.9%

8. In our school, adults do service on behalf of the poor and those in need.

0.02 5.1% 8.4% 18.3% 33.3% 35.0% 10.5% 4.6%

9. The principal, teachers, and staff serve as role models of faith and service to students.

0.06 6.3% 7.1% 14.9% 27.4% 44.3% 1.7% 3.5%

Governance and Leadership 10. The principal makes sure that we have

a successful school. 0.16 3.8% 3.8% 7.6% 22.9% 61.8% 1.8% 3.6%

11. The principal involves everyone in the school community to make sure that everything we do furthers the school’s mission and vision.

0.08 5.2% 6.2% 15.4% 33.2% 40.1% 6.7% 3.8%

12. The principal finds ways to help teachers, staff, students, and parents work together in striving for excellence in all aspects of the school.

0.09 5.7% 6.3% 13.1% 28.7% 46.2% 5.4% 3.9%

Academic Excellence 13. Our school prepares students for the

best high schools. 0.08 3.7% 4.7% 11.5% 29.0% 51.1% 2.9% 4.2%

14. Our school prepares students to use and judge media and technology.

0.07 6.8% 8.3% 19.7% 33.7% 31.4% 7.0% 4.7%

15. In our classes, students spend most of the time solving problems, discussing ideas, creating their own work, reading, writing, speaking, and researching.

0.02 5.6% 7.5% 15.2% 29.1% 42.5% 1.3% 4.7%

16. Our teachers do a good job of helping all students learn.

0.09 5.0% 7.1% 14.8% 28.1% 45.0% 0.8% 4.8%

17. At our school, teachers use many different approaches in their classrooms to meet the different needs of all students.

0.04 6.1% 8.3% 15.3% 28.6% 41.7% 2.6% 4.8%

18. At our school, teachers work together to help each other become better teachers.

0.06 6.0% 7.3% 16.7% 29.3% 40.6% 13.8% 4.7%

19. At our school, student work is graded fairly.

0.07 8.5% 7.8% 14.3% 26.8% 42.6% 3.1% 4.9%

20. At our school, teachers use student work and student test results to improve how they teach.

0.04 8.4% 9.2% 15.8% 30.2% 36.4% 16.4% 5.5%

21. At our school, teachers find many different ways to see how much students have learned.

0.04 5.7% 7.7% 16.5% 30.1% 40.1% 5.9% 5.2%

22. Our school offers guidance and resources (such as counselors, tutors and special teachers) to help students be successful.

0.17 6.9% 6.8% 12.1% 22.9% 51.2% 4.0% 4.7%

Page | 65

23. Our school offers programs and activities (such as sports, drama, clubs, or band) for students to develop their gifts and talents.

0.21 3.6% 4.1% 7.7% 16.9% 67.7% 1.0% 5.0%

24. Our school invites parents/guardians to be involved in the school.

0.07 3.2% 3.8% 11.2% 25.0% 56.8% 2.4% 5.2%

Operational Vitality 25. Our school provides the space,

equipment and technology we need to learn.

0.07 5.6% 7.0% 14.5% 31.6% 41.3% 1.5% 4.8%

26. Our school uses different ways to keep parents/guardians informed about all that is happening in our school.

0.04 4.5% 5.0% 11.9% 30.7% 47.9% 2.6% 5.0%

27. Our school does a good job of attracting new students to our school.

0.08 9.0% 7.7% 17.6% 27.9% 37.8% 6.9% 5.0%

Table A.5. School-level Statistics for the Catholic School Program Effectiveness Survey of Students (5-8 Grades)

Subscale Item

Average Rating

Standard Deviation

Minimum Rating

Maximum Rating

25th Percentile

Median 75th

Percentile

Mission and Catholic Identity 1. Everyone in the school community – the

principal, teachers and staff, students, and parents/guardians – knows and understands the school’s mission.

4.03 0.27 3.33 4.51 3.87 4.10 4.21


3.83 0.32 3.22 4.75 3.60 3.83 4.00


4.61 0.22 4.07 5.00 4.51 4.63 4.78


4.43 0.31 3.50 5.00 4.23 4.43 4.66


3.60 0.43 2.50 4.75 3.31 3.53 3.82


4.23 0.28 3.50 4.70 4.01 4.28 4.43


3.98 0.28 3.32 4.54 3.80 4.00 4.15


3.90 0.25 3.41 4.75 3.71 3.88 4.07


3.95 0.38 3.00 4.64 3.77 3.98 4.23

Page | 66


10. The principal makes sure that we have a successful school.

4.34 0.48 2.71 5.00 4.12 4.47 4.69


3.97 0.44 2.71 4.80 3.74 3.99 4.32


4.04 0.47 2.71 4.80 3.81 4.15 4.32

Academic Excellence

13. Our school prepares students for the best high schools.

4.17 0.35 2.93 4.70 3.96 4.24 4.43


3.72 0.38 2.75 4.45 3.43 3.75 4.01


3.95 0.28 3.33 4.70 3.73 3.95 4.15


4.03 0.39 3.00 4.81 3.80 4.08 4.32


3.93 0.37 2.86 4.60 3.69 3.98 4.11


3.88 0.44 2.27 4.70 3.68 3.97 4.17


3.91 0.41 2.71 4.71 3.64 4.00 4.22


3.76 0.44 2.36 4.88 3.56 3.80 4.04


3.91 0.31 2.75 4.50 3.75 3.96 4.11


3.91 0.58 1.86 4.88 3.49 4.07 4.31


4.21 0.58 2.43 4.88 3.95 4.31 4.65


4.26 0.35 2.71 4.78 4.10 4.35 4.47


25. Our school provides the space, equipment and technology we need to learn.

3.89 0.37 2.75 4.62 3.70 3.92 4.09


4.13 0.31 2.86 4.60 3.98 4.18 4.33

Page | 67


3.70 0.45 2.40 4.50 3.54 3.79 3.96

Table A.6. Standardized Factor Loadings for the 1-factor CFA Model (School-level) of the Catholic School Program Effectiveness Survey of Students (5-8 Grades)


Subscale Item


Standard Error


Standard Error

Mission and Catholic Identity 1. Everyone in the school community – the

principal, teachers and staff, students, and parents/guardians – knows and understands the school’s mission.

0.96 0.04 0.90 0.06


0.77 0.09 0.73 0.11


0.75 0.11 0.73 0.11


0.61 0.13 0.69 0.11


0.71 0.09 0.78 0.10


0.75 0.09 0.89 0.06


0.55 0.10 0.58 0.10


0.70 0.12 0.81 0.08


0.97 0.03 0.94 0.05


10. The principal makes sure that we have a successful school.

0.84 0.07 1.00 0.04


0.88 0.05 0.95 0.04


0.82 0.07 0.99 0.03

Academic Excellence

13. Our school prepares students for the best 0.63 0.09 0.64 0.12

Page | 68

high schools.


0.40 0.15 0.45 0.15


0.88 0.08 0.99 0.07


0.88 0.05 0.93 0.04


0.88 0.06 0.97 0.03


0.89 0.06 0.96 0.04


0.85 0.07 0.78 0.09


0.84 0.06 0.88 0.06


0.94 0.04 0.97 0.03


0.34 0.16 0.28 0.19


0.34 0.16 0.26 0.19


0.80 0.07 0.70 0.09


25. Our school provides the space, equipment and technology we need to learn.

0.49 0.13 0.50 0.11


0.81 0.08 0.50 0.12

27. Our school does a good job of attracting new students to our school.a

0.75 0.07 1.00 0.00

aThe error variance for this item was original estimated in the subscale CFA analysis as a negative number and therefore was subsequently constrained to a small positive constant (i.e., .001).

Table A.7. Item Response Frequencies for the Catholic School Program Effectiveness Survey of Students (9-12 Grades)

Subscale Item

Strongly Disagree

Strongly Agree

Don't Know

Skipped


Page | 69

1. Everyone in the school community – administrators, faculty and staff, students, parents/guardians, alums, and supporters – knows and understands the school’s mission.

1.1% 3.2% 20.0% 38.9% 36.8% 6.1% 10.5%


2.0% 6.1% 17.3% 46.9% 27.6% 3.5% 10.5%


0.0% 2.0% 10.0% 37.0% 51.0% 1.8% 10.5%


1.0% 5.9% 10.9% 38.6% 43.6% 0.0% 11.4%


3.0% 10.9% 23.8% 33.7% 28.7% 0.0% 11.4%


1.0% 2.0% 9.2% 30.6% 57.1% 3.5% 10.5%


0.0% 2.1% 11.3% 30.9% 55.7% 0.0% 0.0%


4.0% 4.0% 13.1% 41.4% 37.4% 0.9% 12.3%


1.1% 1.1% 17.0% 43.2% 37.5% 10.5% 12.3%


2.2% 3.3% 24.2% 37.4% 33.0% 8.8% 11.4%


1.1% 7.8% 24.4% 34.4% 32.2% 9.6% 11.4%


6.5% 3.3% 18.5% 32.6% 39.1% 6.1% 13.2%


6.5% 4.3% 20.4% 35.5% 33.3% 7.0% 11.4%


board or a pastor) who collaborates with the school administration to make or recommend decisions for the success of the school.

2.2% 3.2% 9.7% 38.7% 46.2% 5.3% 13.2%


2.1% 4.3% 8.5% 43.6% 41.5% 4.4% 13.2%

Page | 70

16. Our school administration involves all members of the school community to make sure that everything we do embodies the school’s mission and vision.

4.1% 3.1% 16.5% 37.1% 39.2% 1.8% 13.2%

17. Our school administration finds ways to help faculty, staff, students, and parents collaborate in striving for excellence in all aspects of the school.

2.1% 1.1% 13.7% 44.2% 38.9% 3.5% 13.2%

Academic Excellence 18. Our school has a challenging

curriculum infused with Gospel values, preparing students for college, work, and life.

3.4% 7.9% 15.7% 41.6% 31.5% 0.0% 21.9%


4.5% 9.0% 20.2% 41.6% 24.7% 0.0% 21.6%


2.3% 11.4% 20.5% 33.0% 33.0% 0.0% 22.8%


5.7% 6.8% 22.7% 40.9% 23.9% 0.9% 21.9%


8.0% 4.6% 20.7% 37.9% 28.7% 0.9% 22.8%


5.2% 5.2% 13.0% 49.4% 27.3% 9.6% 22.8%


1.3% 7.7% 14.1% 46.2% 30.8% 8.8% 22.8%

25. At our school, teachers follow school-‐wide policies and procedures to fairly evaluate and communicate student performance.

3.7% 4.9% 14.8% 42.0% 34.6% 7.0% 21.9%

26. At our school, teachers work together to use data on student performance to review and improve courses and instruction.

3.9% 5.2% 15.6% 39.0% 36.4% 8.8% 23.7%


3.6% 2.4% 19.3% 45.8% 28.9% 5.3% 21.9%


7.4% 8.6% 18.5% 34.6% 30.9% 7.0% 21.9%

Page | 71


2.4% 5.9% 11.8% 40.0% 40.0% 3.5% 21.9%

30. Our school provides co-‐curricular and extra-‐curricular programs for students to develop gifts and talents and enhance their creative, artistic, social/emotional, physical, and spiritual potential.

1.2% 3.6% 16.7% 34.5% 44.0% 2.6% 23.7%

31. Our school provides opportunities for parents/guardians to be involved in the education of their students.

4.7% 4.7% 21.2% 40.0% 29.4% 3.5% 21.9%

Operational Vitality 32. Our school treats everyone who works

at the school with consistency, fairness, and justice.

6.9% 6.9% 17.2% 32.2% 36.8% 0.9% 22.8%

33. Our school manages facilities, equipment, and technology in ways that enhance teaching and learning.

8.0% 4.6% 25.3% 36.8% 25.3% 0.9% 22.8%

34. Our school uses different ways to communicate all that is happening in our school to parents/guardians, the school community and beyond.

2.4% 4.8% 21.7% 42.2% 28.9% 2.6% 24.6%


7.1% 4.8% 17.9% 39.3% 31.0% 3.5% 22.8%

Table A.8. Standardized Factor Loadings for the 1-factor CFA Model (Student-level) of the Catholic School Program Effectiveness Survey of Students (9-12 Grades)


Subscale Item


Standard Error


Standard Error



0.70 0.05 0.75 0.04


0.78 0.04 0.85 0.03


0.70 0.05 0.72 0.04


0.75 0.05 0.81 0.04

5. In all subjects, teachers help students think critically and ethically about the world around them, using the lens of Gospel values and Catholic doctrine and

0.69 0.05 0.73 0.05

Page | 72

beliefs. 6. Our school provides opportunities outside

the classroom for student faith formation, and participation in retreats, prayer, mass, sacraments, and other spiritual experiences.

0.68 0.05 0.74 0.04


0.72 0.05 0.72 0.05


0.77 0.04 0.83 0.04


0.83 0.04 0.88 0.03


0.85 0.03 0.82 0.03


0.68 0.05 0.75 0.05


0.79 0.05 0.85 0.04


0.76 0.05 0.78 0.04


board or a pastor) who collaborates with the school administration to make or recommend decisions for the success of the school.

0.77 0.05 0.79 0.05


0.88 0.03 0.91 0.03

16. Our school administration involves all members of the school community to make sure that everything we do embodies the school’s mission and vision.

0.84 0.03 0.87 0.03

17. Our school administration finds ways to help faculty, staff, students, and parents collaborate in striving for excellence in all aspects of the school.

0.85 0.03 0.89 0.03

Academic Excellence

18. Our school has a challenging curriculum infused with Gospel values, preparing students for college, work, and life.

0.77 0.04 0.83 0.04


0.74 0.04 0.76 0.05

Page | 73


0.68 0.05 0.73 0.05


0.78 0.04 0.80 0.04


0.77 0.04 0.80 0.04


0.84 0.03 0.88 0.03


0.88 0.03 0.88 0.03

25. At our school, teachers follow school-‐wide policies and procedures to fairly evaluate and communicate student performance.

0.85 0.04 0.87 0.03

26. At our school, teachers work together to use data on student performance to review and improve courses and instruction.

0.87 0.03 0.87 0.03


0.81 0.04 0.83 0.04


0.88 0.03 0.90 0.02


0.81 0.04 0.83 0.04

30. Our school provides co-‐curricular and extra-‐curricular programs for students to develop gifts and talents and enhance their creative, artistic, social/emotional, physical, and spiritual potential.

0.80 0.04 0.83 0.04

31. Our school provides opportunities for parents/guardians to be involved in the education of their students.

0.73 0.04 0.74 0.04

Operational Vitality 32. Our school treats everyone who works at

the school with consistency, fairness, and justice.

0.68 0.06 0.73 0.06

33. Our school manages facilities, equipment, and technology in ways that enhance teaching and learning.

0.74 0.04 0.83 0.05

34. Our school uses different ways to communicate all that is happening in our school to parents/guardians, the school community and beyond.

0.86 0.03 0.93 0.03

35. Our school does a good job of attracting 0.71 0.05 0.81 0.04

Page | 74

new students to our school.

Page | 75

Appendix B: Tables from the Item-level Analyses of the Catholic School – Defining Characteristics Surveys

Table B.3. Item Response Frequencies and Intraclass Correlation Coefficients (ICC) for the Catholic School Defining Characteristics Survey of Adults

Item ICC

Strongly Disagree

Strongly Agree

Don't Know

Skipped

1. Students in our school are encouraged, through all aspects of their school experience, to develop a closer relationship with Jesus Christ.

0.07 0.8% 1.9% 6.5% 22.8% 68.0% 0.7% 3.4%

2. Our school is a community that prays together.

0.09 1.0% 1.3% 4.4% 18.5% 74.8% 0.3% 3.6%

3. Our school is a community that lives the Gospel message through service to the poor and to those in need.

0.08 1.2% 3.0% 7.9% 25.7% 62.3% 0.8% 3.7%

4. Our school makes Jesus and the teachings of the Catholic Church known to all students.

0.06 0.7% 1.0% 3.0% 18.3% 77.1% 0.5% 3.5%

5. Symbols of the Catholic faith are displayed throughout our school.

0.07 0.6% 0.6% 1.7% 14.2% 82.9% 0.3% 3.8%

6. Our school upholds high standards of excellence in all it offers.

0.08 2.7% 4.2% 8.6% 25.2% 59.3% 0.6% 3.6%

7. In addition to academics and faith formation, our school offers experiences in the arts, athletics, and other extracurricular and service opportunities that contribute to the education of the whole child.

0.11 2.0% 3.3% 8.1% 23.5% 63.2% 0.2% 3.9%

8. Our school supports the social, emotional and spiritual growth of every student.

0.07 3.0% 3.2% 9.6% 25.5% 58.7% 0.8% 3.9%

9. The program of instruction in our school leads students to seek wisdom and truth, with a clear understanding of right and wrong.

0.07 1.3% 2.6% 8.3% 29.4% 58.4% 0.9% 4.0%

10. The learning environment in our school fosters self-‐discipline so that students can become more independent learners.

0.07 2.2% 3.6% 11.4% 31.2% 51.5% 1.0% 4.0%

11. Our school instills in students the responsibility to promote Gospel values and social justice in the world.

0.07 1.1% 3.3% 9.3% 30.4% 56.0% 1.3% 4.2%

12. Administrators in our school understand, accept and model the teachings of the Catholic Church.

0.10 2.0% 3.1% 7.1% 23.5% 64.3% 2.0% 4.2%

13. The teachers in our school understand, promote, demonstrate and teach Catholic values and beliefs.

0.05 1.9% 2.4% 8.3% 27.8% 59.6% 1.3% 4.4%

Page | 76

14. Our school helps parents/guardians fulfill their role as the primary teachers of the faith to their children.

0.05 1.6% 3.6% 14.8% 34.4% 45.5% 2.1% 5.1%

15. Everyone connected with our school works together, respecting each other’s gifts, for the sake of building a strong, faith-‐filled learning community.

0.07 4.1% 5.0% 13.8% 31.4% 45.6% 2.4% 5.1%

16. Our school does everything it can to eliminate obstacles that hinder or exclude students from receiving a Catholic education.

0.06 2.8% 3.8% 11.7% 26.6% 55.0% 5.6% 5.5%

17. Our school operates with the expressed approval and support of our Bishop.

0.13 1.4% 1.1% 5.0% 19.9% 72.6% 13.5% 5.4%

Table B.4. School-level Statistics for the Catholic School Defining Characteristics Survey of Adults

Item

Average Rating

Standard Deviation

Minimum Rating

Maximum Rating

25th Percentile

Median 75th

Percentile


4.56 0.24 3.88 5.00 4.42 4.56 4.71


4.66 0.22 4.04 5.00 4.57 4.69 4.80


4.45 0.26 3.55 5.00 4.35 4.50 4.62


4.69 0.18 4.24 5.00 4.56 4.71 4.80


4.77 0.16 4.29 5.00 4.69 4.78 4.87


4.30 0.36 3.00 5.00 4.15 4.33 4.50


4.38 0.35 3.29 4.89 4.27 4.48 4.62


4.30 0.33 3.00 4.82 4.14 4.31 4.54


4.39 0.33 2.86 5.00 4.28 4.43 4.57

10. The learning environment in our school fosters self-‐discipline so that students can

4.22 0.34 2.86 4.76 4.08 4.27 4.43

Page | 77

become more independent learners. 11. Our school instills in students the

responsibility to promote Gospel values and social justice in the world.

4.36 0.26 3.64 4.88 4.20 4.37 4.50


4.43 0.42 2.50 5.00 4.31 4.49 4.65


4.38 0.34 2.71 5.00 4.29 4.42 4.57


4.19 0.26 3.57 4.69 4.01 4.21 4.39


4.08 0.38 2.74 4.67 3.88 4.13 4.29


4.27 0.34 3.16 5.00 4.06 4.30 4.48


4.59 0.38 2.83 5.00 4.45 4.66 4.85

Table B.3. Standardized Factor Loadings for the 1-factor CFA Model (School-level) of the Catholic School Defining Characteristics Survey of Adults

1-‐Factor Model

Item Standardized Factor Loading

Standard Error


0.90 0.04


0.85 0.06


0.74 0.08


0.96 0.04


0.89 0.06


0.88 0.04

7. In addition to academics and faith formation, our school offers experiences in the arts, athletics, and other extracurricular and service opportunities

0.66 0.10

Page | 78

that contribute to the education of the whole child.


0.93 0.03


0.98 0.02


0.85 0.05


0.94 0.04


0.94 0.03


0.95 0.04


0.93 0.05


0.91 0.04


0.92 0.05


0.72 0.10

Table B.4. Item Response Frequencies and Intraclass Correlation Coefficients (ICC) for the Catholic School Defining Characteristics Survey of Students (5-8 Grades)

Item ICC

Strongly Disagree

Strongly Agree

Don't Know

Skipped

1. My school teaches me to love Jesus. 0.09 0.8% 0.9% 3.9% 16.3% 78.1% 0.5% 1.6% 2. In my school, we pray throughout the

day. 0.11 1.6% 2.6% 7.9% 19.4% 68.5% 0.3% 1.7%

3. We do things at my school to serve the poor and those in need.

0.07 2.0% 3.6% 13.7% 30.4% 50.4% 1.7% 2.1%

4. All students in my school learn about Jesus and what Catholics believe.

0.06 1.3% 1.5% 4.8% 19.5% 72.9% 1.3% 2.3%

5. There are crucifixes and other Catholic symbols throughout the school.

0.09 1.0% 1.2% 3.3% 13.0% 81.5% 1.5% 2.0%

6. My school strives to be excellent in all areas.

0.06 4.7% 5.0% 13.9% 29.8% 46.6% 3.0% 2.2%

Page | 79

7. My school offers extra-‐curricular activities and service projects.

0.09 3.2% 4.7% 10.3% 27.2% 54.6% 2.6% 2.4%

8. Adults in my school show that they care about how students are doing.

0.08 5.8% 6.8% 14.1% 29.7% 43.6% 3.3% 1.9%

9. It is important at my school to learn what is right and what is wrong, and act accordingly.

0.04 1.9% 2.0% 6.8% 24.9% 64.4% 1.2% 2.2%

10. Students at my school know how to behave and respect others so that everyone can learn.

0.05 13.8% 13.6% 24.6% 26.9% 21.1% 1.4% 2.1%

11. Even during subjects other than religion, we learn to act according to what Jesus taught, and to help make the world a better place for everyone.

0.08 6.4% 9.2% 19.2% 31.0% 34.2% 1.6% 2.7%

12. The principal leads us in prayer and is involved in school Masses, and makes sure everyone knows that my school is a Catholic school.

0.18 5.0% 5.2% 10.4% 22.5% 57.0% 1.8% 2.5%

13. The teachers in my school teach and act in a way that shows they understand Catholic values and beliefs.

0.06 4.0% 4.7% 12.0% 30.3% 49.0% 1.6% 2.6%

14. The teachers and the principal at my school show respect for parents/guardians, and keep them well informed about their students and the school.

0.08 4.5% 5.0% 11.4% 28.6% 50.4% 1.8% 2.6%

15. My school works together in ways that show respect for everyone’s unique talents.

0.06 7.3% 7.6% 16.6% 31.3% 37.2% 2.0% 2.6%

16. My school is a community that welcomes all students.

0.09 4.3% 4.3% 8.9% 22.0% 60.5% 2.1% 3.0%

17. The students in my school know that the school is part of the diocese and connected to the Bishop.

0.06 4.1% 4.5% 11.8% 27.8% 51.7% 9.4% 3.1%

Table B.5. School-level Statistics for the Catholic School Defining Characteristics Survey of Students (5-8 Grades)

Item

Average Rating

Standard Deviation

Minimum Rating

Maximum Rating

25th Percentile

Median 75th

Percentile

1. My school teaches me to love Jesus. 4.72 0.17 4.23 5.00 4.64 4.72 4.84

2. In my school, we pray throughout the day. 4.49 0.34 3.00 5.00 4.31 4.50 4.72


4.24 0.31 3.50 5.00 4.12 4.27 4.40


4.64 0.21 4.00 5.00 4.54 4.67 4.76


4.73 0.19 4.18 5.00 4.64 4.78 4.88

Page | 80


4.15 0.38 3.00 5.00 3.97 4.21 4.37


4.24 0.41 2.50 4.83 4.07 4.31 4.48


4.03 0.53 1.50 5.00 3.84 4.01 4.24


4.54 0.21 4.04 5.00 4.46 4.53 4.67


3.35 0.46 2.00 4.50 3.04 3.31 3.63


3.90 0.41 3.12 5.00 3.55 3.80 4.20


4.25 0.55 2.00 4.89 4.05 4.42 4.65


4.20 0.38 3.00 5.00 4.01 4.21 4.42


4.20 0.37 3.00 4.80 4.07 4.22 4.42


3.88 0.44 2.00 4.48 3.73 3.88 4.23


4.38 0.31 3.44 5.00 4.18 4.40 4.63

17. The students in my school know that the school is part of the diocese and connected to the Bishop.

4.20 0.57 1.00 4.83 4.10 4.28 4.46

Page | 81

Table B.6. Standardized Factor Loadings for the 1-factor CFA Model (School-level) of the Catholic School Defining Characteristics Survey of Students (5-8 Grades)

1-‐Factor Model


Standard Error

1. My school teaches me to love Jesus. 0.88 0.06 2. In my school, we pray throughout the

day. 0.47 0.15


0.47 0.14


0.85 0.06


0.70 0.10


0.91 0.05


0.62 0.11


0.96 0.04


0.96 0.04


0.74 0.10


0.84 0.07


0.78 0.08


0.95 0.04


0.86 0.05


0.97 0.03


0.92 0.04

17. The students in my school know that the 0.85 0.07

Page | 82

school is part of the diocese and connected to the Bishop.

Page | 83

Table B.7. Item Response Frequencies for the Catholic School Defining Characteristics Survey of Students (9-12 Grades)

Item

Strongly Disagree

Strongly Agree

Don't Know

Skipped


2.5% 3.3% 10.7% 24.0% 59.5% 0.0% 5.5%


2.5% 0.8% 2.5% 23.1% 71.1% 0.0% 5.5%


1.7% 1.7% 12.4% 31.4% 52.9% 0.0% 5.5%


0.8% 0.8% 5.8% 26.4% 66.1% 0.0% 5.5%


0.8% 0.8% 7.6% 31.4% 59.3% 0.8% 7.0%


1.7% 2.5% 9.2% 34.5% 52.1% 0.8% 6.3%


0.8% 0.8% 8.3% 24.8% 65.3% 0.0% 5.5%


2.5% 1.7% 9.2% 35.3% 51.3% 0.0% 7.0%


1.7% 0.8% 9.9% 38.0% 49.6% 0.0% 5.5%


1.7% 3.3% 14.0% 30.6% 50.4% 0.0% 5.5%


3.3% 1.7% 12.4% 38.0% 44.6% 0.0% 5.5%


0.8% 1.7% 9.9% 37.2% 50.4% 0.0% 5.5%


0.8% 0.8% 16.5% 31.4% 50.4% 0.0% 5.5%

14. The adults in our school show that they are partners with our parents/guardians in our Catholic education.

1.8% 1.8% 15.8% 30.7% 50.0% 3.9% 7.0%

Page | 84


4.3% 2.6% 12.1% 36.2% 44.8% 2.3% 7.0%


2.6% 1.8% 13.2% 28.9% 53.5% 3.1% 7.8%


2.7% 0.0% 9.8% 24.1% 63.4% 5.5% 7.0%

Table B.8. Standardized Factor Loadings for the 1-factor CFA Model (Student-level) of the Catholic School Defining Characteristics Survey of Students (9-12 Grades)

1-‐Factor Model


Standard Error


0.86 0.03


0.87 0.03


0.88 0.03


0.91 0.03


0.85 0.03


0.89 0.03


0.82 0.04


0.83 0.03


0.85 0.03


0.88 0.03

Page | 85


0.89 0.02


0.78 0.03


0.82 0.03

14. The adults in our school show that they are partners with our parents/guardians in our Catholic education.

0.83 0.03


0.86 0.03


0.91 0.03


0.75 0.05

survey validity and reliability study · pdf filevalidity%and%reliability%of ......

Documents