ORGANIZATION OF AMERICAN STATES

Inter-American Council for Integral Development(CIDI)

FOURTH MEETING OF MINISTERS OF EDUCATION OEA/Ser. K/V.7.1August 10-12, 2005 CIDI/RME/INF.4/05 Scarborough, Republic of Trinidad and Tobago 11 July 2005

Original: English

QUESTIONNAIRE ON CURRENT PRACTICES IN EDUCATIONAL REPORTINGPresentation of results

Technical Secretariat of the Inter-American Committee on Education

17th Street and Constitution Avenue, NW, Washington, D.C. 20006

QUESTIONNAIRE ON CURRENT PRACTICES IN EDUCATIONAL REPORTING

Presentation of resultsTechnical Secretariat of the Inter-American Committee on Education

The Questionnaire stems from a mandate of the Special Summit of the Americas, held in Monterrey, Mexico in January 2004. In the declaration of the Summit, the heads of state and government stated that they “commit[ed] to increase access to and dissemination of information concerning [their] educational systems with the objective of improving their performance.” In response to the mandate, during the II Meeting of the authorities and Executive Committee of the Inter-American Committee on Education, the member countries requested that the Technical Secretariat develop a questionnaire that would collect information on “current practices in member countries for informing the public on the characteristics and results of education systems.”

Both the Technical Secretariat of the Inter-American Committee on Education and the Department of Education of the United States participated in the design of the questionnaire. Contributions were also made by the Secretariat of Public Education of Mexico and the Canadian Council of Ministers of Education and Culture. From its inception, it was considered important that the questionnaire asked the countries, at a basic level, about the production and dissemination of educational information. The questionnaire also sought to recognize that dissemination requires those within an education system to decide what information is relevant, how it should be produced and what actors must be informed.

The questionnaire was structured under the assumption that the dissemination of educational information contributes to the improvement of education through strengthening teachers’, schools’ and parents ’ capabilities to identify the problems and solutions. The dissemination of information can also guide and strengthen compensatory actions of national and local governments as well as contribute to a legislative process that is more sensitive to the specific needs, timelines and processes of the educational system.

The Questionnaire on Current Practices in Educational Reporting was sent to the heads of international relations or international cooperation as well as to the CIE delegates of the 34 national education authorities of the Organization of American States (OAS) Member States. The Technical Secretariat sent an electronic file of the Questionnaire on July 21 and 23, 2004 to the 34 member countries and made an online version available August 3rd of the same year. Thirty-two national authorities sent their questionnaires to the offices of the Technical Secretariat, or they completed them online, from August 11th to October 13th , 2004. The Technical Secretariat then compiled the data supplied by the 32 countries that answered the questionnaire and produced a report based solely on that data.

The structure of the questionnaire and those who completed it

The questionnaire is divided into 10 sections, each beginning with a key question.

17th Street and Constitution Avenue, NW, Washington, D.C. 20006

The first five key questions focus on public dissemination practices generally and are divided into five categories according to the type of information. Each of these sections aims to outline the type of information reported, the actors involved in the interpretation of the information, the intended recipients and the dissemination methods used. In all of the sections the respondents were asked to provide information on the latest published education reports (including title, year, edition and possible website, if applicable).

The sixth and seventh questions focus on the dissemination of information specifically for families and schools. These sections seek to identify government efforts to disseminate relevant information for decision making among teachers, schools, parents and students. These sections focus on themes emanating from comments made by experts from the member countries during the launch meeting of the second phase of the Regional Education Indicators Project that was held in Cancun, Mexico, in June 2004. During that meeting, these experts highlighted the importance of using information about academic performance as a central input for reflection and action among educational actors, especially teachers and school personnel.

17th Street and Constitution Avenue, NW, Washington, D.C. 20006

The eighth and ninth questions refer to efforts to educate the media and legislative bodies to ensure that they are adequately informed about the workings of the education system and can actively participate in the interpretation of this information.

Lastly, question 10 allows the ministries to explain how the dissemination of educational information fits within the broader education policy framework. This section is divided into three parts. The first part considers information as a possible input for key actors in education (such as, principals, teachers and educational authorities). The second part offers possible criteria for the allocation of resources based on educational performance. The third part seeks information for building consensus and collaboration among various actors.

General considerations

The immediate conclusion garnered from the data collected by the survey is that practically all national educational authorities in the hemisphere disseminate, to varying degrees and in multiple ways, the information that they have gathered. Countries devote important efforts especially to the dissemination of educational achievement data as measured by national student assessments. However, there are cases in which this information has little relevance for main actors and certain ethnic groups, as is often the case with many teachers and principals. Also, there are few policies that are specifically designed to provide useful information to

parents or to legislators.

Table 1. Who answered the questionnaire?

Antigua and Barbuda

Coordinator - Technical & Vocational Education & Training, Ministry of Education

Argentina National Director of Information and Assessment of the Quality of Education (DINIECE), Ministry of Education

Barbados Deputy chief education officer (planning, research and development), Ministry of Education

Belize Education Officer- StatisticsBolivia Planning OfficerBrazil General Coordinator of the Consolidated System on

Educational Information (INEP).Canada Coordinator, Research and Statistics, Council of

Ministers of EducationChile Chief Officer, Department of Research and

Development, Ministry of Education Colombia Information Coordinator, Office of Planning,

Ministry of Education Costa Rica Sub-director, Division of Curricular Development,

Ministry of EducationDominica Senior Planning OfficerEcuador Director, Division of Statistics, Ministry of

EducationEl Salvador National Director of Monitoring and Assessment,

Ministry of EducationUnited States of America

Director, Annual Reports Program

Grenada Education Officer – Statistics and Assessment

Guatemala Director of Information Unit, Ministry of EducationGuyana Chief Planning OfficerHonduras Principal Consultant to the MinisterJamaica Acting Director, Planning & DevelopmentThe Bahamas The Education OfficerMexico General Director of Assessment, Secretariat of Public

EducationNicaragua Director of Public Information and Director of

Prospecting and Policies, Ministry of EducationPanama General Director of Education, Ministry of EducationParaguay Technical Officer, Ministry of EducationPeru Chief of the Unit of Education Statistics, Ministry of

EducationDominican Republic

Director of Statistics, Ministry of Education

St. Kitts and Nevis

Technical Coordinator

St. LuciaSuriname

Statistician lllHead of Department of Research and Planning

Trinidad and Tobago

Research Officer, I

Uruguay Officer, Research Unit

Venezuela Director of Budget and Planning, Ministry of Education

2

One issue unanticipated in the questionnaire was the participation of local and state/provincial authorities in the management of information. While the questionnaire focused on the actions of national governments, in some countries local authorities bear the responsibility for the dissemination, collection and regulation of educational information, giving national authorities a minimal role in the diffusion process. The questionnaire does not explicitly inquire about the details of cases where the dissemination of information is decentralized. Chile, Canada and the United States each indicated the use of this system of dissemination. Chile stated that the Ministry of Education’s commitment to the dissemination of educational information has led to the development of the Municipal Education Information System (SIEM), which offers educational information to municipalities via its website. The municipalities, in turn, take on the responsibility of disseminating that information among schools and communities. The Coordinator of Research and Statistics from the Canadian Council of Ministers of Education indicated that although his responsibilities for the

3

dissemination of information at the national level are limited, he could still offer some information on the general characteristics of the publications developed in the provinces. In the United States, individual states, as well as local school districts and schools, disseminate information regarding school characteristics, basic education statistics, and student performance data. This is generally done through documents known as “report cards,” which are often available online. At the national level, the National Center for Education Statistics is responsible for collecting and disseminating education statistics, and student achievement data from the National Assessment of Education Progress are disseminated online.

The dissemination of enrollment information, basic educational indicators and educational information garnered from the census

The large majority of countries in the hemisphere has developed information diffusion activities that are based on annual enrollment and graduation data in primary and secondary education. This information and the associated indicators (called “basic indicators” in the survey) are traditional inputs for the evaluation and planning carried out by educational authorities.

Only Antigua and Barbuda and the Bahamas completely lack the periodic publication of the enrollment figures and basic education indicators. Belize, Dominican Republic, St. Kitts and Nevis and Suriname, on the other hand, do not mention any printed publications, websites or other reporting methods. Though fifteeni countries annually print and publish statistical compendia on enrollment information, there are few editions distributed. Six countries manage to print one thousand or more annual units: Argentina, Brazil, Chile, Mexico, Peru and the United States.

The dissemination of basic indicators is similar to that of disseminating enrollment figures. Twelveii countries reported that they publish compendia of indicators, but print fewer than 1000 copies, with the exceptions of Brazil (5,000), Canada (2,500), Peru (2,000), Chile (1,000) and Mexico (1,000). Despite stating that they publish basic indicators, Barbados, Belize, Ecuador, Jamaica, Panama, Dominican Republic and Venezuela do not specify public access to a publication nor a website or other diffusion method.

As it has been mentioned within the context of the Regional Educational Indicators Project (PRIE), information on education systems would have a greater impact if it could be converted into a useful input for the daily work of teachers, principals, families and communities, and not only for

national or regional authorities. Therefore, educational information put at their disposal should reflect both the situation of their community and of their schools. The countries’ information dissemination practices for enrollment data seem to move in

i Argentina, Barbados, Brazil, Chile, Dominica, Granada, Guatemala, Guyana, Mexico, Paraguay, Peru, St. Kitts and Nevis, St. Lucia, Trinidad and Tobago, Uruguayii Brazil, Chile, Canada, Dominica, Granada, Guatemala, Guyana, Mexico, Paraguay, Peru, St. Lucia, Trinidad and Tobago

10

6

1 1 0 0

Educationauthorities

Academia,researchers

Teachers NGO's TeachersUnions

Lawmakers

Figure 1: Sector participation in writing of narrative reports based on census data (# of countries)

4

this direction. Of the 30 countries that publish enrollment information, 22 disaggregate this information at municipal, local or school levels. Bahamas, Belize, Colombia, Guyana, Mexico, Paraguay, Trinidad and Tobago and Uruguay are the eight countries that do not publish information at any of these levels. Seventeen countries publish information on enrollment for individual schools. However, there are few countries that publish basic educational indicators by municipality, community or school. Only Bolivia, Costa Rica, Dominica, Dominican Republic, El Salvador, Guatemala, Panama and St. Lucia publish drop out and retention data for individual primary and secondary schools.

Often, the greatest educational inequalities in the hemisphere are drawn along ethnic lines within nations. With this in mind, it is important to note that only 5 countries publish annual enrollment information disaggregated by ethnic group. These countries are Bolivia, Costa Rica, United States, Guatemala and Panama. Other countries with significant indigenous populations collect and publish information on indigenous populations served by bilingual-intercultural modalities, without specifying the ethnic group of the students. Costa Rica, Panama and the United States publish drop out and retention indicators disaggregated by ethnic group.

The majority of the countries (30) that answered the survey did collect educational information based on census data. Twenty-oneiii of these countries published reports using data collected from the census.

As demonstrated in Figure 1, the sectors that had the highest level of participation in the writing of reports using this data were educational authorities (10 countries) and academia (6 countries). Those that did not participate in the writing of the report were teachers’ unions and lawmakers (0 countries).

On the other hand, Figure 2 displays the institutions to which these narrative reports were sent. On the whole, the institutions in the hemisphere that most often received the reports based on census data from their governments were educational authorities (in 13 countries), and other ministries. While teachers’ unions and congresses (10 countries) also received the report from their governments, they did not participate in its development. Only three countries (Belize, Dominican Republic and Paraguay) indicated that they distributed the report to schools.

The Dissemination of the Results of Student Assessments and Achievement Tests

iii Bahamas, Belize, Bolivia, Canada, Chile, Colombia, Costa Rica, El Salvador, Jamaica, Mexico, Nicaragua, Panama, Paraguay, Dominican Republic, St. Kitts and Nevis, St. Lucia, Trinidad and Tobago, United States, Uruguay, Venezuela

5

Considerable effort is required to effectively report on the results of national standardized exams administered to primary and secondary level students. Of the 28 countries that administer these exams, 23iv have consistently produced descriptive reports. The number of these publications exceeds on average the number of publication of enrollment and basic indicators (in the 20 countries that publicized the results of secondary education performance exams, the reported average of copies published was 3,696, in comparison to an average of 841 publications for enrollment information.)

Of the 28 countries that reported on national standardized exam performance, Colombia, Panama and the Dominican Republic did not specify the title of any publication, related Web address or other method for the publication of the results. According to Figure 3, those that participate the most in the writing of the reports are educational authorities. Those that have a low level of participation are NGOs and teachers unions. The five countries that involved primary and secondary teachers in the writing of these reports were Antigua and Barbuda, Argentina, Grenada, Nicaragua and St. Kitts and Nevis. Regarding the dissemination of the reports, 19 of the 28 countries sent the reports to primary and secondary schools, 12 countries sent reports to the universities and 10 countries sent reports to teachers’ unions.

With regard to comparative international assessments, 14v countries participated in at least one of the assessments found listed in Table 3. Eight of those countries published the results through printed reports, three are in the process of generating the aforementioned report and three countries did not compile or publish any type of printed document containing the assessment results.

Dissemination of Information to Schools and Families

Although 19 countries reported that they publish education evaluation reports specifically designed for schools, only 16 provided detailed information on these reports. Using the information supplied by these 16 countries, three types of reports can be identified:

Statistical Reports: Belize, Bolivia, and Brazil provided statistical information concerning individual schools. Bolivia published 15,674 copies and Brazil published 216,000 copies.

Reports on Student Performance: These are reports that national and state (provincial, departmental, etc.) authorities provide individual schools with information about student performance as measured by national standardized tests. These reports may also contain information on exam results in other communities or regions, in order to provide a basis for comparison of schools. Examples include the Report of Results, System of Measuring of Quality of Education SIMCE, of Chile; The Report of National Evaluation of grades 2-4 of Dominica; the Report Cards with

iv Antigua and Barbuda, Argentina, Bahamas, Belize, Bolivia, Brazil, Canada, Chile, Dominica, El Salvador, Granada, Guyana, Honduras, Jamaica, Nicaragua, Paraguay, Peru, Dominican Republic, St. Kitts and Nevis, St. Lucia, Trinidad and Tobago, United States, Venezuelav Argentina, Belize, Bolivia, Brazil, Canada, Chile, Colombia, Honduras, Jamaica, Mexico, Paraguay, Peru, St. Lucia, United States

Table 3. International Standardized Exams

Cambridge Internal Exam,Caribbean Advanced Proficiency Exam Caribbean Secondary Examinations CertificateInvestigación Iberoamericana de Escuelas Eficaces Laboratorio Latinoamericano de Evaluación de la Calidad EducativaPISA PISA PLUSTIMSS 2003Primer Estudio Internacional Comparativo en lenguaje y matemática en 3o y 4o de educación primaria y factores asociadosProgress in International Reading Literacy Study

6

12

4 4

0 0 0

EducationAuthorities

Academia,Researchers

Teachers TeachersUnions

NGOs Lawmakers

Figure 4: Sector participation in the writing of the reports for schools (# of countries)

information on the performance of individual schools of the United States (shared with parents and the general public); the Results of School Advantage of México; School Report: Evaluation of School Academic Performance of Students 3rd and 6th grade of Nicaragua and other reports that were described but not specified from Canada, El Salvador, Honduras and the Dominican Republic.

Annual School Reports: These are reports that are delivered to schools at the end of a school year about particular challenges that each school faces. Here we have grouped together the reports of Jamaica, St. Kitts and Nevis and Saint Lucia. Jamaica presents reports to all schools about certain weaknesses but does not publish the report for wider consumption. St Lucia’s (End of Term/Year Assessment Report) provides schools with information about individual teachers and students and simultaneously gives recommendations to schools. St. Kitts and Nevis did not detail the information sent out in their annual report.

This categorization does not completely respond to the purpose of classifying each report, but helps us identify components that can guide future efforts to systematize experiences and identify relevant lessons: dissemination of information on the behavior and composition of school enrollment in each school, dissemination of information on standardized tests on students’ performance or school achievement in each school, dissemination of evaluations of relevant aspects of the education cycle in each school.

Figure 4 illustrates sector participation in the writing of the above-mentioned reports. Of 16 countries, 12 involved educational authorities in the drafting of the reports, while academics and teachers participate each participate in 4 countries.

Concerning reports designed specifically for families, only Bolivia, Canada, Chile, United

States, Mexico, Peru and Saint Kitts & Nevis publish these types of reports. We can identify three profiles on dissemination practices from the information provided by the countries:

Dissemination of reports through Internet, done by Bolivia, Mexico, Peru and the United States (only the latter two countries specified that the information published on Internet for families is designed to be user-friendly and written in clear and simple language).

The second profile consists of reports based on standardized tests. Examples include Chile’s SIMCE Report for Teachers and Guardians and the use of state “report cards” in the United States. Their purpose is to inform parents on educational performance, measured at the schools where their children are enrolled. These reports include information on the behavior of performance in time, as well as their relative position in comparison with regional and nationals groups of schools.

A third profile is presented by the isolated case of Saint Kitts & Nevis, where educational authorities, teachers and principals deliver reports to parents three times per year on school

7

performance. These reports are presented at meetings where students are also participating and where performances as well as students’ individual attitudes are discussed.

Education of the Media and Congress to ensure a correct reading and understanding of the data from the educational system

Fifteenvi countries described actions to educate media. Most of them mention press conferences where opportunities for dialogue are opened among authorities, specialists and the media on the meaning and implication of educational data. Some specific initiatives are highlighted, such as permanent counseling to the press provided by Brazil’s INEP; meetings in Canada to explain to the media the information that is going to be published, detailed expositions for media delivered in Chile; workshops for journalists and press editors and with press officers from the ministries of education in Colombia and workshops for the media at the design phase in the Dominican Republic.

Tenvii countries described actions to better educate the Congress. Most of them prefer to present and explain educational information through occasional counseling sessions with the legislative committees that oversee education. Two cases that could be mentioned are the effort of INEP in Brazil to maintain a team of specialists that responds to the needs of the Congress and the meetings of the Education Committee of the Legislative Assembly of Panama within the framework of the Dialogue for Integral Transformation of the National Educational System of Panama.

Policy Purposes

The last section of the questionnaire gave educational authorities the opportunity to specify the purposes that the dissemination of information has in the broad framework of national education policies (see Figure 5). With regard to the allocation of resources, the information disseminated is not usually integrated into policies of prices or incentives for schools or teachers. This information is more frequently seen as a component used to define the allocation of compensatory resources among schools. More common purposes for disseminating information were to report on the process of school management and to foment public debate about education.

vi Antigua and Barbuda, Argentina, Belize, Brazil, Canada, Chile, Colombia, Costa Rica, Jamaica, Mexico, Nicaragua, St. Kitts and Nevis, Suriname, United States, Venezuelavii Antigua and Barbuda, Belize, Brazil, Chile, Honduras, Panama, Paraguay, St. Kitts and Nevis, United States, Venezuela

8

Conclusions

Most countries in the hemisphere annually publish statistical data on the enrollment of students in primary and secondary school. This majority also publishes basic educational indicators and produces information through the censuses of their respective countries. The sectors that receive the reports most often are the same educational authorities, other ministries and academic entities. Very few countries design reports on educational assessment that are directed to the sector of parents or guardians and students.

Through an examination of the levels of sector participation in the writing of reports on the census, standardized exams and schools themselves, it can be concluded that “educational authorities” is the sector that has the highest level of participation in the majority of countries. Non-governmental organizations (NGO’s), teachers unions and legislators have a low level of participation in the writing of these reports.

Regarding the policy implications of the dissemination of educational information, the majority of countries indicated that these reports serve to inform the process of school management and to foment public debate about education. Many countries concurred that they did not use these reports to identify deserving teachers or schools for awards or incentives. Further research must be conducted to ascertain the effectiveness of these reports in improving school management and pedagogy.

Less than half of the countries in the hemisphere participate in comparative international standardized exams of student performance at the primary and secondary levels.

Approximately half of the countries of the hemisphere attempt to educate the media about the educational data published by the government. The effect of these efforts on the media and its public remain unexplored.

Only a third of the countries in the hemisphere try to implement informative activities and training for their Congresses. These countries most often present this information to the education committees within the congresses.

Very few countries have indicators that disaggregate data by ethnic group, especially with regard to the educational situation of indigenous peoples. The majority of countries with indigenous populations report their programs and modalities in intercultural and bilingual education without specifying the ethnic or linguistic group to which they belong.

document.doc

9