o3-i3 general academic competence and...
TRANSCRIPT
Draft August, 2015
Guiding tool 2 - General
academic competence and
potential
1
Table of Contents
1. Introduction 2 2. Definition 4
3. Debate standardized testing 5 4. Instruments 7
5. Implementation 11
Annexes (separate documents):
A. Questions on general academic competences in application process B. Examples of Master’s programmes currently using standardized tests
C. Suggestions further reading
2
1. Introduction
This Guiding Tool describes and analyses the categorisations of general academic competence and potential most commonly used in the
admissions process of graduate students to Master’s programmes. It provides an overview of the tests used in admission processes, looks into
the competences addressed there and the ways in which these are
measured.
This guiding tool on general academic competences provides you with relevant background information on general academic competences, food-
for-thought to support you in shaping your own ideas on how relevant and, more precisely, how decisive they are for a successful completion of
your Master’s programme – and therefore to what extent they are a critical or determining factor in your programme’s application process.
This guiding tool provides an overview of available assessment instruments to gain insight into an applicant’s strengths and weaknesses
in these competences. This guiding tool does not aim to convince you that you should use a specific test. If it helps you to clarify what you are
looking for in terms of general academic competencies and how you look for it, it will already serve its purpose.
Appendix A to this guiding tool offers a set of questions for Master coordinators in order to determine whether it could be worthwhile to
include a measurement of general academic competences in the application process. Appendix B to this guiding tool offers you examples of
Master’s programmes that are currently using standardized tests in their admission process to include general academic competence in their
criteria for admission. Appendix C to this guiding tool provides you with an overview of resources should you want to dive deeper into (aspects of)
the world of general academic competences.
2. Definition
The paradigm shift described in the report Changing paradigms in admission to master’s programmes in Europe describes the need to evolve
admission processes from “recognition of diplomas” to “assessment of competencies”. This implies the necessity to measure an applicant’s
competence on a level that is comparable to what can be expected after completing a Bachelor’s level. The shift from diploma recognition towards
the assessment of competency level requires a different framework to judge an applicant’s qualification, but not (necessarily) a different
competency level. And the same goes for the necessity to measure an
3
applicant’s potential with respect to the learning outcomes of the Master’s
programme. In the terminology of the Dublin Descriptors this means
assessing the applicant’s competence at the level of completion of the first cycle:
- have demonstrated knowledge and understanding in a field of study that builds upon and their general secondary education,
and is typically at a level that, whilst supported by advanced textbooks, includes some aspects that will be informed by
knowledge of the forefront of their field of study; - can apply their knowledge and understanding in a manner
that indicates a professional approach to their work or vocation, and have competences typically demonstrated
through devising and sustaining arguments and solving problems within their field of study;
- have the ability to gather and interpret relevant data (usually within their field of study) to inform judgements that include
reflection on relevant social, scientific or ethical issues;
- can communicate information, ideas, problems and solutions to both specialist and non-specialist audiences;
- have developed those learning skills that are necessary for them to continue to undertake further study with a high
degree of autonomy.
But the tools needed should do more, they should also predict the potential
to develop competences in line with completion of the second cycle:
- have demonstrated knowledge and understanding that is
founded upon and extends and/or enhances that typically associated with Bachelor’s level, and that provides a basis or
opportunity for originality in developing and/or applying ideas,
often within a research context; - can apply their knowledge and understanding, and problem
solving abilities in new or unfamiliar environments within broader (or multidisciplinary) contexts related to their field of
study; - have the ability to integrate knowledge and handle
complexity, and formulate judgements with incomplete or limited information, but that include reflecting on social and
ethical responsibilities linked to the application of their knowledge and judgements;
- can communicate their conclusions, and the knowledge and rationale underpinning these, to specialist and non-specialist
audiences clearly and unambiguously;
Tools should relate to Dublin Descriptors:
measure competence at level of
completion first cycle;
predict potential to complete second cycle.
4
- have the learning skills to allow them to continue to study in a
manner that may be largely self-directed or autonomous.
And finally the tools needed should be able to measure general academic
competence and potential in a way that ensures the exclusion of biases that may lead to an undesired limitation of the Master programme’s
attractiveness to an international population of students.
This guiding tool focuses on general academic competence and potential as one of the essential fields to determine whether a potential Master
student should be admitted to a specific Master’s programme. Together with subject-specific knowledge and skills, and with personal
competencies and traits, general academic competence and potential is considered to be a decisive aspect of a student’s chance on successful
completion of a Master’s programme. To phrase it differently: what an applicant knows, who he/she is, and how smart he/she is.
General academic competence Obviously, years of debate can pass discussing which competences and
skills can be labelled as being general academic, and how these can best be assessed. But fortunately years of academic debate have already been
spend on this question. In preparing the feasibility study for the assessment of higher education
learning outcomes, the OECD used the following description of generic skills: “generic skills are considered the general abilities, capacities or
capabilities deployed in all kinds of cognitive activity. These outcomes are seen as broad, enduring skills that, although not domain specific, are
developed by frequent practice and exercise in specific domains. These
skills are those that would be desired of any student regardless of subject matter or discipline, and are teachable and learnable. They are
distinguished from the knowledge or skills of particular performances or domains. Such general abilities are the flexible, fluid capacities developed
by and used in learning new things.”1
When using this description as the background against which to further specify to general
academic competence and skills, the following three main categories can be used to distinguish
general academic from personal competence: analytical skills, verbal reasoning and quantitative reasoning. And obviously, these categories can
be further split into underlying skills and competences.
1 http://www.oecd.org/edu/skills-beyond-school/AHELOFSReportVolume1.pdf, p.115
General Academic Competence:
Verbal reasoning
Quantitative reasoning
Critical thinking
Ability to analyse symbolical
information
5
These three categories are the framework for the most commonly used
standardized tests of potential Master students; Graduate Records
Examinations (GRE) and Graduate Management Admission Test (GMAT). Other tests (IE GAT) also add analysing and handling symbolical
information as a competence to be distinguished.
3. Debate
The debate on both the importance of general academic competence and potential for successful participation in and (timely) graduation from a
Master’s programme, as well as the possibility to measure these, has been
long and intensive. Although the debate still goes on today, several developments took place over the years.
Nowadays, the notion of
academic competence itself is no longer disputed. The
(labelling of) exact components of what makes a
person academic competent may vary, but generally
aspects like (the ability for) verbal and quantitative
reasoning and critical thinking are considered to be crucial
elements. Taking the IE test
(see below) into consideration, we may add analysing and
handling symbolical information as a third category. With reference to the Dublin Descriptors, students who start with a Master’s programme may be
expected to:
be able to apply their knowledge and understanding in a manner that indicates a professional approach to their work or vocation, and
have competences typically demonstrated through devising and sustaining arguments and solving problems within their field of
study; have the ability to gather and interpret relevant data (usually within
their field of study) to inform judgments that include reflection on relevant social, scientific or ethical issues;
be able to communicate information, ideas, problems and solutions
to both specialist and non-specialist audiences. (part of the qualifications to signify completion of the first cycle)
6
Students who successfully complete a Master’s programme may be
expected to:
be able to apply their knowledge and understanding, and problem solving abilities in new or unfamiliar environments within broader
(or multidisciplinary) contexts related to their field of study; have the ability to integrate knowledge and handle complexity, and
formulate judgements with incomplete or limited information, but that include reflecting on social and ethical responsibilities linked to
the application of their knowledge and judgements; be able to communicate their conclusions, and the knowledge and
rationale underpinning these, to specialist and non-specialist audiences clearly and unambiguously.
(part of the qualifications that signify completion of the second cycle)
It takes common sense to understand that without the previously mentioned skills of verbal and quantitative reasoning and critical thinking
the abovementioned capabilities will be impossible to develop and to
prove. Nowadays the discussion focuses much more on influences that stimulate
or hinder the development of academic competence, both before and during a student’s enrolment at university. And especially on the ways to
measure a student’s academic competence and the predictive value of standardized tests.
4. Instruments This chapter provides an overview of available tests for admission
processes taking general academic competence into account. Graduate Record Examinations (GRE), developed and offered by Educational Testing
Service (ETS), and the Graduate Management Admission Test (GMAT), developed and offered by the Graduate Management Admission Council
(GMAC) are the two most-known standardized tests. Both originate in the United States and have – due to criticism on earlier versions of the tests –
fairly recently seen adaptations in the tests.
GMAT always was the preferred test when applying for an MBA, but
nowadays also GRE is accepted by a large number of MBA programmes although GMAT is still accepted more widely. There seems to be a
difference in both tests in how hard prospective students find them (GMAT being considered the harder), and the accents the tests put in the three
components analytical skills, verbal reasoning and quantitative reasoning, also differ.
An alternative test has been developed by IE Business School in Madrid; this claims to be more practical and designed for a more diverse audience
7
than the first two. This test is less well-known than GRE and GMAT,
possibly because the latter two are successfully marketed globally and
offer test facilities world-wide, whilst the IE test can only be taken in Madrid. But IE offers an additional perspective on testing academic
competence (in short, diagrammatical or symbolic reasoning) that may be an important added value for certain Master’s programmes.
Also available is the Miller Analogies Test (MAT), developed by Pearson to offer a somewhat cheaper and shorter alternative to GRE and GMAT. This
test traditionally focuses more on Humanities than other academic fields.
PLEASE NOTE: the information provided below in paragraph 2a-2d is based only on sources from the organisations involved in offering these
tests!
2a. Graduate Record Examinations Graduate Record Examinations (GRE) was developed over 60 years ago,
and is and offered by Educational Testing Service (ETS). In 2011 a revised
version of the test was introduced, the GRE revised General Test. ETS claims thousands of Higher Education Institutions use GRE as part of their
admission process, most of them universities in the US. Some 350 universities in Europe use GRE, sometimes for specific Master’s
programmes. The GRE focuses on verbal reasoning, quantitative reasoning, and critical
thinking and analytical writing skills. Below these are described with an eye on the changes made in the revised version.
Verbal Reasoning
The Verbal Reasoning measure focuses on a test taker’s ability to understand what they read and how they apply their reasoning skills —
with an emphasis on how they analyze and evaluate written material. Plus, antonyms and analogies are no longer part of the test. Vocabulary is
still important, but it is used in the context of a reading passage. This measure demonstrates an applicant’s ability to:
analyze and draw conclusions from discourse; reason from
incomplete data; identify author’s assumptions and/or perspective; understand multiple levels of meaning such as literal, figurative and
author’s intent select important points; distinguish major from minor or relevant
points; summarize text; understand the structure of a text understand the meanings of words, sentences and entire texts;
understand relationships among words and among concepts
Quantitative Reasoning The Quantitative Reasoning measure focuses on a test taker’s ability to
apply basic mathematical skills and concepts (arithmetic, algebra, geometry and data analysis), with an emphasis on data interpretation and
8
real-life scenarios that test takers will encounter in graduate or business
school. This measure demonstrates an applicant’s ability to:
understand quantitative information
interpret and analyze quantitative information solve problems using mathematical models
apply basic mathematical skills and elementary mathematical concepts of arithmetic, algebra, geometry, probability and statistics
Critical Thinking and Analytical Writing
The Analytical Writing measure focuses on a test taker’s critical thinking and analytical writing skills — specifically their ability to articulate complex
ideas clearly and effectively. Test takers are asked to provide focused responses to questions so they can accurately demonstrate their skill in
directly responding to the task presented. This measure demonstrates an applicant’s ability to:
articulate complex ideas clearly and effectively
examine claims and accompanying evidence support ideas with relevant reasons and examples
sustain a well-focused, coherent discussion control the elements of Standard Written English
2b.Graduate Management Admission Test
The Graduate Management Admission Test (GMAT) is also developed over 60 years ago and offered by the Graduate Management Admission Council
(GMAC). A revised version of the test was introduced in 2012. GMAT focuses on analytical writing, integrated reasoning, quantitative and
verbal skills. The analytical writing assessment focuses on the analysis of an argument. Integrated reasoning consists of questions to assess multi-
source reasoning, the interpretation of graphics, two-part analysis and table analysis. Quantitative questions focus on assessing data sufficiency
and problem solving. And verbal questions focus on assessing reading
comprehension, critical reasoning, and sentence correction.
Analytical Writing The Analytical Writing section consists of one writing task—Analysis of an
Argument. The format is one 30-minute written essay on the analysis of an argument. What it measures is the test-taker's ability to formulate an
appropriate and constructive critique of a specific conclusion based on a specific line of thinking. A specific knowledge of the essay topic is not
necessary; only the capacity to write analytically is assessed.
Integrated Reasoning The Integrated Reasoning section gives another data point to differentiate
candidates. This part consists of 12 questions covering four question types: Multi-Source Reasoning, Graphics Interpretation, Two-Part
9
Analysis, and Table Analysis, to be completed in 30 minutes. What it
measures is the test takers ability to analyze and synthesize data in
different formats from multiple sources.
Verbal The Verbal section allows candidates to showcase their verbal abilities.
The format is 41 questions comprised of reading comprehension, critical reasoning, and sentence correction to be completed in 75 minutes. It
measures the test taker's ability to read and comprehend written materials, reason and evaluate arguments, and correct written material to
confirm to standard written English.
Quantitative The Quantitative section highlights data analysis and reasoning skills. This
section consists of 37 questions on Data Sufficiency and Problem Solving, to be completed in 75 minutes. It measures the test taker's ability to
reason quantitatively, solve quantitative problems, and interpret graphic
data. Understand problems involving arithmetic, elementary algebra, and common geometry concepts. Evaluate the amount of information needed
to solve quantitative problems.
2c. IE Admission Test The IE Admission Test is developed by IE Business School Madrid as an
alternative for tests like the GRE and GMAT. This test has been developed as a means of determining a candidate’s ability to make management‐
style decisions under pressure. This exam is different to other standardized tests in a number of ways:
It is aimed at an international audience, with a diversity of educational backgrounds (thus, it does not favor any particular
educational system or area of study); It is a practical exam (as opposed to an academic exam), presenting
participants with more real life situations; No preparation is required (this exam determines participants’
ability to process information and think strategically, rather than repeat previously studied patterns or equations).
The ieGAT consists of three types of tasks: verbal, numerical and
diagrammatical. Questions are scattered randomly throughout the exam, requiring participants to switch rapidly from one line of thinking to
another.
Comprehension and reasoning This section consists of a text box which must be read and understood by
the participant. Each text box is followed by two questions regarding the text. This portion of the test determines participants’ comprehension of
written English in a professional context.
10
Comprehension, reasoning and calculation
This section consists of graphs, and data analysis. Advanced mathematics
is not required. Rather, participants need to think quickly and process large amounts of information and quick calculations.
Logical‐abstract reasoning with symbols
In this section, participants take a predetermined shape through a series of modifications to determine the final product. This section requires
strong visualization skills and the ability to maintain track of multiple
“images” at a single time.
2d. Miller Analogies Test (MAT) Miller Analogies Test is developed by Pearson, focusing on testing an
applicant’s analytical skills. The MAT is a high-level mental ability test requiring the solution of
problems stated as analogies. Performance on the MAT is designed to reflect candidates’ analytical
thinking, an ability that is critical for success in both graduate school and professional life.
Through analogies with content from various academic subjects, MAT scores help graduate schools identify candidates whose
knowledge and abilities go beyond mere memorizing and repeating information.
The analytical skills measured by the MAT are key to success in
graduate school and are highly valued by graduate programs. For more than 50 years, the MAT has proven to be a reliable and
valid test for assessing candidates for graduate school. Predictive validity studies conducted by Pearson and by independent
researchers have shown positive correlations between MAT scores and subsequent success in graduate programs.
5. Implementation Below an overview is provided of steps to be taken after the decision has
been made to implement the assessment of general academic competence and potential into the admission process of your Master’s.
Step 1: See also step 1 knowledge requirements; bring together the core
lecturers of the programme.
Step 2: Identify the academic competences that are most important for successful completion of the master programme and/or entrance into the
job market and that you wish to measure.
11
Consider using the following questions in identifying these:
a. Does your admissions process require proof of an applicant's
writing/mathematical/analytical skills? If so, what kind of proof do you require?
b. Have you formulated criteria against which to determine whether the writing/mathematical/analytical skills are sufficient for
enrolment in your Master's programme? Could you briefly describe these criteria?
Are these criteria communicated to the applicants? c. Did you ever encounter any difficulties in assessing an applicant's
writing/mathematical/analytical skills? If so, could you briefly describe these difficulties?
d. Do you experience difficulties with the writing/mathematical/ analytical skills of students currently enrolled in your Master's
programme? If so, do you offer support to these students?
Step3: Adjust the descriptions of the academic competences into
transparent requirements for applicants.
Step 4: Decide in which way you wish to measure these academic competences (already existing standardized test, newly developed test
etc)
Step 5: Decide how to measure/rate these academic competences
Step 6: Decide what the meaning of the ratings is in your admission process.
One quick and dirty way to handle “General Academic Competencies” for your master’s
programme, could be:
- Sit down with the senior academics of your programme and discuss if your
current indicators for GAC (often the GPA and the reputation of the previous
university) are adequate – also to limit the risk of false negatives.
- Determine if you want to improve admission in terms of GAC.
- If you do, go for the GRE as the most broadly used and validated test with more
than 500 000 test takers per annum.
- If you are concerned about the costs for the students, you can consider to abduct
the GRE fee from the tuition for admitted students.
There may be other and better ways, but this is a good start.