EXPERT SURVEYS and

PARTY MANIFESTOSLimits and potentials

STRUCTURE

Expert survey by Benoit and Laver on party policy

Some reviews: Curini’s analysis Steenbergen and Marks Party Manifestos by CMP Curini’s analysis

DESIGNING THE EXPERT SURVEY

Benoit and Laver conducted a set of systematic surveys in order to collect the information required to locate all politically relevant parties on a wide range of policy dimensions

The surveys were conducted from 2002-03, covered 47 countries, resulted in 1,491 valid expert responses, locating 387 different political parties on scales relating to a total of 37 unique policy dimensions.

1.1 IDENTIFYING EXPERTS

The typical expert is an academic specializing in political parties and electoral politics of his or her country.

Method divided in steps:

Contact the national political science association of the country, if it exists, with a request for its membership lists.

1.1 IDENTIFYING EXPERTS

If it does not exist, “snowball” strategy is used: short list of well-known experts from the country, and ask each to name as many additional experts as possible

Then contact each of those experts, asking them in turn to name as many additional experts as possible

Additional lists from universities and NGO, and exclude journalists and political actors

1.2 IDENTIFYING PARTIES

“Politically relevant” parties where chosen, when they met any one of three criteria:

Every existing national party that won seats in the national legislature at the country’s most recent election

Every existing national party that had won at least 1 percent of the vote nationally at the country’s most recent election

1.2 IDENTIFYING PARTIES

Any other parties indicated by local experts as politically relevant despite not meeting the other two criteria

Even small parties are included since they’re crucial for the analysis of political competition (ex. in forming gov)

also to avoid to exclude parties that would play significant roles in party competition over a longer period.

1.3 POLICY DIMENSIONS

Core set of four substantive policy dimensions in order to allow for direct comparison between countries:

1.1.ECONOMIC POLICYECONOMIC POLICY2.2.SOCIAL POLICYSOCIAL POLICY3.3.ENVIRONMENTAL POLICY ENVIRONMENTAL POLICY 4.4.DECENTRALIZATION OF DECISION MAKINGDECENTRALIZATION OF DECISION MAKING

For each dimension a scale from 1 to 20, with the lower numbers indicating a ‘left-wing’ position and the higher a ‘right-wing’ position.

2.1 HOW TO EVALUATE BIAS

BL are aware that there could be a systematic relationship between the experts’ own ideologies and their judgments about party policy positions

SYMPATHY SCALE:Experts are asked to place all parties on a scale indicating their own closeness to each party’s policies.

LIMITS of EXPERT SURVEYS

To assess bias BL used the following regression-based technique:

For each country-party section, they regress experts’ placements on their sympathy scores for the party in question

They compute the mean and the variance of the predicted left–right score for an expert with an ‘indifferent’ sympathy score of 10.5 (the midpoint of the 1-to-20 scale), taking this to be the ‘corrected’ placement

they compare the ‘corrected’ expert placements with the ‘actual’ placements (i.e. the mean score of parties resulting from the expert survey) to see whether or not these differences are statistically distinguishable

From this analysis only seven parties appeared to involve sistematic bias that cuold be statistically distinguished from zero.

2.2 SAMPLING/IDEOLOGICAL BIAS

Ideological bias known as rationalisation or projection: respondents tend to place the parties they like closer to where they perceive themselves to be (assimilation effect) and to place those parties they dislike farther away (contrast effect).

Sampling bias: respondents are systematically unrepresentative of all possible experts (more left-wing or liberal)

2.3 SOLVING IDEOLOGICAL BIASFocus on the sympathy question alone: risk of under/overstimate IB

Curini:Use an unfolding model in order to estimate the location of the ideal point using respondent’s preferences“corrected score”: focus on the relationship between respondents’ ideal points and parties’ ideological stance

Scoreij = j + βj [s(Policyi – Scorei)] + εi

Policyi : respondents’ position (unfolding method) on the left-right scale

Scorei : mean party position on the same scale

j : indicator of ideological bias if it is statistically distinguishable from zero(as it was the case for 53 parties out of 158)

a direct way of “correcting” biased score, neutralising the ideological bias

2.4 EVALUATING EXPERT SURVEYS

Budge four type of problems:

what ‘party’ is being judged by the expert?what criteria do experts bring to bear when they judge party positions?do experts judge the intentions of parties or their behavior?what is the time frame for the judgments that we ask experts to make?

Steenbergen and Marks: these concerns can be alleviated in the expert survey design:

Avoid ambiguous termsGive them a more circumscribed meaning

limit interpretation

but there may still be an interpretative space for experts that could distort their judgments

HOW DO EXPERTS INTERPRET THE QUESTIONS IN EXPERT SURVEYS AND HOW DO THEY LINK SUBSTANTIVE KNOWLEDGE ABOUT PARTIES TO THOSE QUESTIONS?

The 1999 expert survey of national party positions on

European integration

Ray: “What was the overall orientation of the party leadership towards European integration in 1999”7 response optionsSpecification of the object that was to be evaluated (party leadership)Specification of the time frame (1999)

provide to the experts a common frame of mind

Assess variance in the experts’ judgments: Compute the standard deviation of their

placements of parties Perform a variance components analysis:

limited statistically significant variation across the experts

rather similar judgments about EU stances of the political parties

Attributes of a party or party system: measure party differentiation within a party system as the standard deviation of party positions in that country

variation is a function of party differentiation, salience and dissent

Steenbergen and Marks’ model

The “congeneric test model”: the correlation between the expert judgments should be greater, perhaps much greater, when they are judging the same trait than when they are judging different traits

similarity coefficients: compare the correlational patterns across experts, we should expect to see a high similarity coefficient between two experts if those experts indeed evaluate the same trait

THE ITALIAN EXCEPTIONReliability of expert judgments on Italian parties is worse than that for other countries.

This is primarily due to one expert (E4).

The correlations between the judgments of this expert (E4) and the remaining five experts are low, suggesting that he or she may have used different criteria for placing Italian parties.

good grounds for doubting the validity of this expert responses

although it is worth emphasizing that even after including this expert the final results for Italy look quite good

ALTERNATIVE MEASURES

Manifesto Research Group: extract party positions from party manifestos. It has coded both favorable and unfavorable mentions of European integrationVoter perceptions of party positions. The EES project asked respondents where parties stood on the issue of European integration. Subject to projection effects voters may project their own stance on European integration onto the partyUse members of parliament (MPs) and members of the European Parliament (MEPs) to gauge party support for European integration

ANALYSING PARTY MANIFESTOS

In 1979 the Manifesto Research Group (MRG) was formed to analyze documents in 90 democracies comparatively and within a common framework.

Manifesto: written as a whole and as overall balance of subjects carefully considered IT IS OVERALL PRIORITIES THAT MATTER RATHER

THAN PARTICULAR STANCES TAKEN IN ISOLATION

The idea behind the MRG is that parties argued with each other by emphasizing different policy priorities rather than confronting each other on the same issues

Since 1989 CMP (the Comparative Manifesto Project) has coded and standardized the statements of party election programs of 25 democracies since 1945 into 56 policy categories, such that each quasi-sentence of every election program is coded into one, and only one, specific category.

SENTENCE WHICH IS THE VERBAL EXPRESSION OF ONE POLITICAL IDEA OR ISSUE(GENERAL CODING UNIT)

Basis of the policy estimates: textual data, examination of parties’ and governments’ own statements of policy, in the shape of election programmes (manifestos) and declarations in the Parliamentary debate before a vote of confidence or investiture

estimates on what actors themselves have said rather then on people’s judgementsMajor source of evidence: texts, autoritative documents issued by parties and governmentsCollect and organize it in statistical form

13 categories as indicators of policy positions emphasised by the “left” and another 13 as indicators of policy positions emphasised by the “right”. The left-right position is constructed for each party in each election by subtracting the sum of its left statements from the sum of its right statements.

As a result the range of this scale is between –100 (i.e., the score of a party devoting its entire programme to Left-wing issues) and +100 (i.e., the score of a party devoting its entire programme to Right-wing issues).

ADVANTAGES OF PARTY MANIFESTOS

They allow three types of comparison:

Changes in policy position over time within specific partiesDifferences in policy positions across parties even in different countriesDifferences across countries

RELIABILITY and VALIDITY

Two measurement concerns:Reliability: ARE THE PROCEDURES DERIVED FROM THE

THEORY RELIABLE, GIVING THE SAME RESULTS EVERY TIME THEY ARE APPLIED? (computers will always apply some procedures to the same text in the same way: this is not always true of the human coders who have produced the Manifesto data)

Validity: ARE PROCEDURES VALID IN THE SENSE OF ACTUALLY TELLING US WHAT WE THINK THEY TELL US ABOUT CONTENT (OF MANIFESTOS) AND SENDERS (PARTIES)? (is the underling theory itself correct? Are the codings correlated with other measures of the same conceptual construct?)

THE ITALIAN CASE

CURINI using Manifesto scale, a trade-off can emerge: if on one hand this methodology guarantees a more theoretically sound comparison between countries and time, on the other hand it does not allow for specific characteristics of individual party systems (that could therefore be buried under a uniform left-right dimension).

The Italian party system was characterized for a long period by:

THE PRESENCE OF TWO ANTI-SYSTEM PARTIES, THE COMMUNIST PARTY AND THE ITALIAN SOCIAL MOVEMENT ON THE RIGHT

THE ABSENCE OF ANY CREDIBLE ALTERNATIVE TO A GOVERNMENT FORMULA BUILT AROUND THE CHRISTIAN DEMOCRATIC PARTY

A HIGH NUMBER OF RELEVANT PARTIES

POLARIZED PLURALISM

PLACEMENT OF THE 4 MAIN ITALIAN PARTIES DURING THE “FIRST REPUBLIC” (1948-92), USING THE MANIFESTO SCORE

Some results seem quite odd: in two occasion the DC is located at the right of the MSI, along the 70s the PSI is at the left of the PCI, and the PCI in the 1953 election is the furthest right party.

For scholars who want to apply models that rely on the policy preferences of the Italian political parties and that wish to use PMD, Curini proposes a method that consists of the following steps:

CURINI’S METHOD

The first change needed is the application of a reliability analysis to the original Manifesto scale, in order to assess the scale’s internal consistency: the Cronbach’s alpha coefficient is a famous indicator of consistency. Ideally, it should be above .7. The scalability of the original Manifesto scale for the country investigated should be assessed

If the Cronbach’s alpha is satisfactory, parties’ score should be smoothed by recognizing how the score of a party at time t does not translate directly in its position

(If however the reliability analysis shows some inconsistency, the possibility to drop some categories included in the original left-right scale should be considered before smoothing the parties’ scores)

CRONBACH’S ALPHA RESULTS POSITIVE (.54) ALTHOUGH LOW BY THE STANDARDS FOR LIKERT SCALE. TO TEST WHETHER ALL THE INDICATORS BELONG TO THE SAME SCALE, VALUES FOR THE CRONBACH ALPHA ARE COMPUTED ALSO WHEN SPECIFIC ITEMS ARE REMOVED FROM THE SCALE. AS A RESULT THE CATEGORY “DEMOCRACY” SEEMS NOT TO TAP THE SAME CONCEPTUAL DIMENSION AS THE OTHER INDICATORS. REMOVING THIS CATEGORY HELPS A BIT THE CRONBACH ALPHA RESULT (.58). HOWEVER, THE STILL UNSATISFACTORY VALUE OF THE SCALE SUGGESTS THAT THE EQUAL WEIGHTING ASSUMPTION OF ALL THE ITEMS WHICH CHARACTERISES THE ADDITIVE MANIFESTO SCALE BE DROPPED.

Second modification: the Manifesto score for a given party need not be identical to its actual positional value

THE GENERAL PUBLIC IS EXPECTED TO WEIGHT INFORMATION FROM THE PAST AS WELL AS PROMISES AND DECLARATIONS OF THE LATEST MANIFESTOS TO IDENTIFY THE ACTUAL POSITION OF A GIVEN PARTY

Then the position of a party at time t should reflect not only the score obtained in its current electoral program but also the party’s general image emerging from the cumulation of all its previous manifestos

To produce measures that capture this dynamics, the exponentially weighted moving average of current and past manifesto positions is computed:

POSITIONt = MANIFESTOt + (1.0 – )POSITIONt-1

Curini analyzes the 1953 general election and, by adding the scores obtained from the Manifesto scale without the category “Democracy”, and his final scores, he obtains more plausible results: the PCI moves from being the most rightist party to the most leftist, the PSI moves to the left of the DC, MSI becomes the most rightist one.

CURINI’S CONCLUSIONS

Considering the 2001 Italian general election, an advantage of his method stands out:

IN THAT ELECTION MOST PARTIES (WITH THE EXCEPTION OF COMMUNIST REFOUNDATION PARTY) DID NOT ISSUE THEIR OWN ELECTORAL MANIFESTOS, BUT FORMED PRE-ELECTORAL COALITIONS AND SUBSCRIBED TO A JOINT PLATFORM. THIS IMPLIED THAT THE CMP WAS ABLE TO CODE JUST ONE MANIFESTO FOR EACH COALITION. NONETHELESS, ESTIMATION OF EACH PARTY’S INDIPENDENT POLICY POSITIONS IS ALSO REQUIRED AS PARTIES ARE STILL THE PRE-EMINENT POLITICAL ACTORS WITHIN ITALY

CURINI’S CONCLUSIONS

By contrast, the smoothing procedure included in his method allows him to estimate different scores for different parties even if at time t parties agree on the same manifesto, precisely because different parties (usually) have different histories.

Policy estimates of Manifesto data are ideal also for operationalizing and testing coalition theories. As an additional check for the validity of his scores, Curini uses them as dependent variables to test a widely accepted theory: the central role of the median party in policy bargaining.

CONCLUSIONS

McDonald/Mendes: expert surveys are deficent and the manifesto measures are useful on two concerns:

The experts place the parties in such stable locations that there is little hope of using expert surveys to investigate party policy dynamics. The CMP is the only viable data source for observing such dynamics and analysing the party movements

Expert surveys may produce suspect results about where the parties stand on different dimensions of politics and policy. There is doubt that expert respondents actually can and do make clear distinctions between policy dimensions.

Steenbergen & Marks: there are good reasons to trust expert survey results on party positions:Consistency of experts’ responsesExpert placements of political parties converge with other measures

As we have seen both methods have confident support. BL admit and analyse some divergent estimates between expert survey and party manifestos.They claim that these differences are due to the understimation of some issues in the CMP analysis, and obviously that their method is the “correct” one…

OUR CONCLUSIONS

Expert surveys rely on a certain leeway accorded to respondents and therefore are subject to the risk of bias

But when using Manifestos, still the analysis is based on certain categories that can be ambiguously assigned both to the left and the right side, or on a certain statements that sometimes could be more linked to strategies (due to the elections) then to the real nature of that party. How can one really find the “hidden” real party position on certain issues?