QUALETRA“QUALITY IN LEGAL TRANSLATION”

WS 4Testing, Evaluation & Assessment

KU LeuvenAntwerpen

16-17 October 2014

QUALETRAJUST/2011/JPEN/AG/2975

With financial support from the Criminal Justice Programme of the European Commission Directorate General Justice

© Hendrik J. Kockaert & Winibert Segers

Translation Evaluation

Translation product (target text, the result of the translation process)

Translation process (think aloud protocol, eye tracking, key logging)

Translation service (contact with the client, offering the quotation, invoicing, compliance agreements, complaints, etc.)

© Hendrik J. Kockaert & Winibert Segers

Translation EvaluationTranslation product: methods

Holistic method Analytical methodCDI method (Calibration of Dichotomous Items)PIE method (Preselected Items Evaluation)

Translation EvaluationTranslation product: methods

Research Reports:How do evaluators carry out the following evaluation methods?

(1) Holistic(2) Analytical (3) PIE

Research ReportsTranslation Evaluation

Holistic & Analytical MethodsResearch Method

A sample translation from English to Spanish of a UK robbery judgment was allocated to ten evaluators who received a translation brief (structured translation specifications)

Shortly after submitting their assessment, the evaluators were asked to complete a questionnaire

These case studies were carried out by UAH (Holistic method) and DCU (Analytical method)

© Hendrik J. Kockaert & Winibert Segers

Research ReportTranslation Evaluation

PIE Method

Preselected Items Evaluation

Translation briefLT specific criteriaEssential documentsEAWECQA criteria

Case study

KU Leuven carried out a case study where 46 level A professional translators from federal ministries in Belgium did a translation from French to Dutch

The text contained 56 elements, including punctuation. Seven preselected items were used

An intern corrected the translations using the analytical evaluation method, referring to the error categories of the ATA evaluation grid, without implementing the actual ATA format and criteria

The standard error of difference between the scores obtained on the basis of the analytical method and the PIE method was 0.177, which is not statistically significant

Research ReportTranslation Evaluation

PIE Method

Sample TestJUGEMENT CORRECTIONNELDEMANDE DE MISE EN LIBERTÉDEBATS Avant l’audition de [PI 1] XXX, le président a constaté que celui-ci ne parlait pas suffisamment la

langue française ;Il a désigné YYY, interprète inscrit sur la liste du tribunal ; l’interprète a ensuite prêté son

ministère [PI 2] chaque fois qu’il a été utile.A l’appel de la cause [PI 3], le président a donné connaissance de [PI 4] l’acte qui a saisi le tribunal

[PI 5] et constaté la présence et l’identité de XXX, dont [PI 6] il a reçu les déclarations [PI 7].

Maître ZZZ, conseil [PI 8] du prévenu, a été entendu [PI 9] en sa plaidoirie.Le ministère public a été entendu en ses [PI 10] réquisitions.Le prévenu a eu la parole en dernier.Le greffier a tenu note du déroulement des débats.

© Hendrik J. Kockaert & Winibert Segers

PIE: Case StudyStatistics: Analytical Score vs. PIE Score

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 400.0

2.0

4.0

6.0

8.0

10.0

12.0

PIE Score (../10)Analytical Score

© Hendrik J. Kockaert & Winibert Segers

PIE: Case StudyStatistics: T-Test

P value and statistical significance: The two-tailed P value equals 0.1376By conventional criteria, this difference is considered to be

Not statistically significant

Confidence interval:The mean of Group One minus Group Two equals 0.268 95% confidence interval of this difference: From -0.090 to 0.626

Intermediate values used in calculations:t = 1.5151df = 40standard error of difference = 0.177

© Hendrik J. Kockaert & Winibert Segers

Conclusion:

No statistically significant difference between analytical scores and PIE scores

PIE proves a justifiable evaluation method

objectivitytransparencyequality

Research ProposalsTranslation Evaluation

PIE Method

ActivitiesTranslation Evaluation

ECQA Test Design

ECQA Test on three successive stages:

(1) MC questions on legal knowledge (specifically EDs and EAWs)

(2) MC and open questions on legal language (phraseologies, terminology, style, register), possibly including a recognition test (to recognise whether a proposed translation is correct or not); after passing each of the two previous stages

(3) A translation test corrected with PIE

Survey disseminated, analysed and survey report

Legal Translation Product Quality Assurance

(1) Three research reports drafted (holistic, analytical and PIE method)

(2) Strategy and methods have been decided for reporting on the evaluation methods

(3) Submission of “Objective Translation Evaluation through PIE” (KU Leuven), 1st International Young Researchers’ Conference on Translation and Interpreting (UAH, 7-8/11/2013)

Deliverables

No. Deliverable name/type (a) Format (b) Target group (d)

1 Survey on the testing, assessment and evaluation on the current legal translation practices in criminal proceedings in the EU (product, translator and service/process quality assurance)

Websites and manuals (updatable and online information on product, translator and service/process quality assurance)

Translator trainers Legal practitioners Professional associations

Deliverables

When you evaluate legal translations, how important do you find the following?

Very Im

portant

Importa

nt

Neither

Importa

nt nor U

nimporta

nt

Unimporta

nt

Not at a

ll Importa

nt0

10

20

30

40

50

60

70

80

90

100

Being able to differentiate between good and bad translationsAn objective evaluationThe efficiency of the overall evaluation pro-cedure

1 (highest ranking)

2 3 4 5 6 (lowest ranking)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Evaluation by comparing the translation to the source text and highlighting any errorsEvaluation of the translation of preselected segments in the source textEvaluation by means of a checklist or an evaluation gridEvaluation based on the general impression of the quality of the translation

Which evaluation method do you find most suitable for legal translations?

How important do you find the following?

Equiva

lence

betwee

n source

text

and ta

rget te

xt

Faith

fulness to

the s

ource te

xt

Lingu

istic c

orrectn

ess

Meaning t

ransfe

r

Idiomatic c

orrectn

ess

Style

and re

gister

Readab

ility o

f the t

arget

text

Term

inology0

20

40

60

80

100

120

Not at all ImportantUnimportantNeither Important nor UnimportantImportantVery Important

Which are the most important issues in evaluating legal translations?

1 (highest ranking)

2 3 4 5 6 7 (lowest ranking)

0

10

20

30

40

50

60

70

80

90

100

Having a suitable source text for legal translation evaluationDetermining if the candidate translator passes or failsDetermining the severity of errorsApplying the same criteria to each translationAvoiding subjective evaluation

How can the evaluator justify the evaluation?

0102030405060708090

100

AgreeNeutralDisagree

Test Formats

Three different testing formats:

Translation testRevision test Recognition test

Certification Procedure

Prerequisite

Only European Master’s in Translation Network (EMT) graduates may apply to take the ECQA certification examination.

Examination

The examination consists of 3 stages

A candidate must pass each element of each stage before passing to the next stage

A candidate who fails one or more elements of a stage only has to repeat the failed element(s)

ExaminationThe 3 stages of the examination are:

Multiple choice questions: LTiCP.U1 Legal KnowledgeU1.E1 Criminal law and procedure U1.E2 Monolingual legal terminology (in both languages) Internet access is not permitted at this stage

Open questions: LTiCP.U2 Professional aspects & LTiCP.U3 Instrumental competenceU2.E1 Professional practice U2.E2 Professional conductU3.E1 Information acquisition U3.E2 Legal terminology managementInternet access is not permitted at this stage

Translation: 2x250-word texts, one for each element of LTiCP.U4 TranslationU4.E1 Translation of essential documents (Directive 2010/64/EU) U4.E2 Translation of European Arrest WarrantsInternet access is permitted at this stage

ECQA Skills Card

Skill Card Type of ExaminationLTiCP.U1 Legal Knowledge

U1.E1 Criminal law and procedure MCQ

U1.E2 Monolingual legal terminology (in both languages) MCQ

LTiCP.U2 Professional aspects

U2.E1 Professional practice Open Questions

U2.E2 Professional conduct Open Questions

LTiCP.U3 Instrumental competence

U3.E1 Information acquisition Open Questions

U3.E2 Legal terminology management Open Questions

LTiCP.U4 Translation

U4.E1 Translation of essential documents (Directive 2010/64/EU) Translation

U4.E2 Translation of European Arrest Warrants Translation

Translation Test

maximum 2x250-words PIE method

Test FormatPIE Items

Report a lost or stolen passport [1] You must report the loss or theft of your passport as soon as possible, even if you don’t

want to replace it immediately. You’ll need to complete a Lost or Stolen Notification form [2]. Find out how to get the form and what to do if your passport is lost abroad.

If your passport is lost or stolen in the UK [3] You must report the loss or theft of your passport to the Identity and Passport Service [4]

(IPS) [5]. This will reduce the risk of anyone else using your passport or your identity. To report your passport lost or stolen, fill in and sign a Lost or Stolen (LS01) [6] Notification form [7] and return it to IPS [8]. The address is on the form.

You should [9] report all passport thefts to the police. You’ll need the crime reference

details for the LS01 form [10].

Revision Test

maximum 10 items maximum score = 10

Test FormatCandidates have to revise the English translation; have to recommend corrective measures for elements in the English translation that are not correct. Vérifiez la date du document.Check the data of the document. A candidate can make three types of erroneous revision:

1) The candidate revises an element in the translation that was correct, and proposes, for instance, to replace ‘Check’ by ‘Control’ or by ‘Verify’;

2) The candidate does not revise an element in the translation that was not correct: in our example the word ‘data’, which has another meaning then the French word ‘date’ (‘date’);

3) The candidate revises an element in the translation that was not correct, but her/his revision is not correct, and proposes, for instance, to replace ‘data’ by ‘information’.

Recognition Test

maximum 10 itemsmaximum score = 10

Example

Candidates have to indicate if the proposed English translation is correct or incorrect. Le témoignage de l’épouse de l’accusé n’est pas crédible.The testimony of the wife of the accused is not credible. correct incorrect

Test FormatPIE Translation, Revision, Recognition

After passing the three tests (translation, revision and recognition), the candidate will obtain a score with a maximum of thirty

The proposed testing approach is a better guarantee for objectivity than the traditional translation test

The score of the candidate does not depend on the personal appreciation and interpretation of the evaluator

Another advantage of this testing approach is the time-saving aspect: The time you spend to develop the three tests is largely compensated by the short evaluation time

Validity of Tests

Candidate 1 Candidate 2 Candidate 3 Candidate 4 Candidate 5 Candidate 6 Candidate 7 Candidate 8 Candidate 9

0

5

10

15

20

25

Score

RecognitionRevisionTranslation

ECQA Overview Question TypesSample

Learning Elements to be covered consistent with skill card

Type of Examination Developer

CLTiCP.U1Professional competence

U1.E1Professional practice Open Questions

U1.E2Professional conduct Open Questions

ECQA QuestionsSamples

LTICP.U1.E1.PC2: The candidate has a sound understanding of criminal procedure in the legal systems involved (e.g. levels of jurisdiction, legal structures, institutions, settings, parties).

1. A person who __________________ a crime may be sentenced to imprisonment. Fill in the blank with one of the four choices:

A doesB is convictedC rendersD performs

ECQA QuestionsSamples

1.1. LTICP.U1: Legal knowledge1.1.1. LTICP.U1.E1: Criminal law and procedure

CLTICP.U5.E1.PC5: The candidate masters the main domains and sub-domains of criminal law, especially the most frequent offences in essential documents and European Arrest Warrants, e.g. drugs, fraud and theft.

1. From the following list, choose one example of an offence against the person:

A theft B smugglingC assaultD possession

ECQA QuestionsSamples

LTICP.U1.E1.PC3 The candidate has a general awareness of current legal issues and their development in the relevant countries.

1. From the following list, choose one example of a current legal issue of special concern in the EU cross-border criminal context:

A Illegal sale and distribution of food products B Trafficking in human beingsC Electronic surveillance by state security servicesD Non-payment of fines

ECQA QuestionsSamples

LTICP.U1.E1.PC4: The candidate is familiar with the EU directives on legal translation.

1. A criminal suspect has the right to translation of (choose one of the following):

A Questions from the judgeB Prosecution statementsC Transcripts of evidence given in court by witnesses D Other