advanced validation techniques for xliff 2 › sites › default › files › ... · the third...

Localisation Focus Vol.14 Issue 1The International Journal of Localisation

43

1. Introduction

According to the W3C definition, Extensible Mark-up Language (XML) is designed to describe data byproviding a flexible text format derived from SGML(Standard Generalized Mark-up Language) (Bray et.al. Eds. 2008 ). As XML is a software- and hardware-independent format for carrying information, it hasbecome an important tool for information exchangeamong applications. XML is meant to be easilyunderstood by humans, yet – at the same time –machine-readable (w3schools 2014).

XML provides a minimum set of rules for creatinguser-defined tags to describe the stored andexchanged data. Therefore, XML – as ametalanguage – provides logical space for creatingXML vocabularies. In turn, instances of XMLvocabularies need to be validated against a specificvocabulary’s schema. An XML schema is adocument, which describes the structure of thedocument and indicates the order of elements withadmissible and mandatory attributes and their values.A number of schema languages are available, whichcan check the XML instances against a variety ofrules; from very basic ones – such as hierarchicalstructure or order of elements –, to complicatedconditions, restrictions, and even admissible states ina workflow progression.

XSD (XML Schema Definition) (Thompson et. al.

Eds. 2004) is the most popular schema language thatis being used widely by XML consumers. In spite ofits popularity, XSD is not expressive enough forXLIFF 2 and is not able to target many constraintsincluding complex and advanced rules. Although, thelatest version of XSD, 1.1 (Gao et. al. Eds. 2012), canhandle many issues omitted in XSD 1.0 (Delima et.al. 2009), it is not broadly implemented and used(Quin et. al. 2012).

Relax NG schema language (Clark et. al. Eds. 2001)can be considered the next step in XML validation. Itprovides more possibilities for describing complexconditions than XSD. In addition, Relax NG is easierto learn and its expressions and syntax are moreintuitive and user-friendly (Quin et. al. 2009; Vlist2011). This schema has the potential to providevalidation solutions to some of the XLIFF 2constraints that XSD 1.0 cannot express.

Finally, the most powerful and expressive schemalanguage for XML validation is Schematron (JelliffeEd. 2002). It can define the most complicated rulesand constraints of XML standards and vocabularies.Schematron also provides room for delivering user-defined diagnostics and customized error messages,which can be enriched by detailed information aboutthe objects that failed to comply with any of thevalidation rules.

All of the mentioned validation techniques aregathered and together make DSDL (Document

Advanced Validation Techniques for XLIFF 2

Soroush Saadatfar and David FilipLocalisation Research Centre, CNGL,

University of Limerick, [email protected], [email protected]

AbstractThis paper aims to present an overview of validation issues for the XML Localization Interchange File Format(XLIFF), while XLIFF 1.2 validation is mentioned, the paper concentrates on validation of XLIFF 2.0 andsuccessors (XLIFF 2). The goal is to propose an optimal set of DSDL (Document Schema Definition Languages)(Brown 2010) compliant methods to provide a set of XLIFF TC (and in due course OASIS and ISO) guaranteedstandardized machine readable artefacts that would support validation of all XLIFF 2 normative statements. Thediscussed methods include XSD files, proposed possible schemas in the Relax NG schema language, Schematronrules, and finally a master NVDL (Namespace-based Validation Dispatching Language) file that brings all thenecessary and guaranteed methods together. Development of a lightweight demonstrator RESTful web service thatwraps all the proposed DSDL methods in one validator platform is also discussed and foreshadowed.

Keywords: XLIFF, validation, Relax NG, XSD, Schematron, NVDL, DSDL, XML, RESTful, web-service, webservice


44

Schema Definition Languages). As a multipart ISOstandard, DSDL defines a set of schema languagesfor XML validation (Brown 2010).

2. XLIFF Validation

XLIFF 2.0, as an OASIS standard, is presented in thespecification created by the Technical Committee(Comerford, T., Filip, D., Raya, R.M., Savourel, Y.;Eds., 2014). XLIFF contains a large variety ofdifferent constraints and rules, which an XLIFFinstance must not violate in order to be valid, it alsoaddresses various application conformance targetswith its processing requirements. Because of variedrequirements for validation expressivity, differenttechniques for automated validation need to be usedto cover the specification in full.

There are a number of tools already developed andavailable for the purpose of validating differentversions of XLIFF. The first tool that addressesXLIFF validation beyond XSD was designed forXLIFF 1.2, it is the XLIFFChecker (Raya, 2012). Weare however primarily concerned with XLIFFvalidation, because XLIFF 2.0 does provide, in itsprose, specification statements that allow for moreadvanced DSDL validation.

The third party validators currently available for the2.0 edition are XLIFF 2.0 XMarker Checker(Schnabel) and Okapi-lynx (Savourel). Unlike theapproach used in this paper, these validators are notDSDL-based, except the basic XSD validation. Theirvalidation methods that go beyond a simple XSDcheck are implementation dependent in the above.On the plus side, both of the cited XLIFF 2 validators

do provide validation for XLIFF 2.0 fragmentidentifiers, which goes beyond the scope of thispaper. This DSDL based approach addresses thewellformedness of an XLIF fragment identifier onlyas far as it is required to check attributes of the URIor IRI type.

The main target of this paper is to identify andelaborate validation methods and artefacts on thebasis of XML standardised (implementationindependent) methods of validation, i.e. DSDLschema languages. DSDL artefacts that coverautomated validation for a maximal subset of aXLIFF 2 specification are suitable for becomingXLIFF TC deliverables (as part of the multipartstandard product according to OASIS definitions) inXLIFF 2.1 and successor editions.

XSD schema – the first part of DSDL – for XLIFF2.0 core is already provided by as a part of the XLIFF2.0 multipart OASIS standard. However, this schemais able to provide only very basic validation andmoreover, many patterns that pass XSD validationare in fact violating normative statements of the prosespecification. For instance, many attributes which areencoded as optional in the XSD schema, may be infact required or forbidden conditionally, dependenton specific values of other attributes, availability ofspecific content or elements etc. These advancedconstraints are explained in detail further in thissection, as we are tackling them with other DSDLmethods.

As the first step, the Relax NG schema for XLIFF 2.0core was developed (Saadatfar 2014). Relax NGschema for XLIFF validation allows one to conductmore detailed document checks than the originalXSD. For instance, Listing 1 illustrates how it ispossible to handle the constraint for the subTypeattribute (used in some of XLIFF inline elements).The XLIFF specification states:

If the attribute subType is used, the attribute typemust be specified as well.

The schema is defining two valid cases (<choice>element allows either of its children to be valid); firstif both type and subType attributes are present inthe element, and the second case where typeappears only. <ref> elements will describe thecontent of each declared node later in the schema.

Despite its advantages, Relax NG is not expressiveenough to describe all the normative content of anXLIFF 2 specification. For instance, it does notListing 1: subType dependency in Relax NG


support default values for attributes directly, DTDcompatibility annotations must be embedded for thispurpose. Definition of some rules may turn intoextremely long code which is very hard to read andmaintain. For instance, according to the specification,the trgLang attribute of <xliff> element isrequired if and only if the XLIFF document containsa <target> element. This statement could betargeted in Relax NG by defining two patterns for<xliff>. This approach would duplicate most ofthe schema and therefore does not present a practicalsolution.

For such complex constraints, Schematron offers asuitable solution, the rule-based validation. Listing 2illustrates the rule for the earlier mentionedstatement;

Schematron uses XPath language to access elementsinside documents. This approach helps to track errorsat all levels.

The rule in listing 3 first seeks <target> elementsinside the document (as context attribute defines)and then applies the constraint; i.e. if the file does notcontain any <target>, the rule will be ignored as itis not compulsory anymore. After the element wasfound, at the <assert> element the validation test

is conducted (assertion). The expression inside thetest attribute will return “true” if <xliff>element has trgLang attribute and otherwise, anerror will be raised. In the case of an error, themessage placed inside <assert> will be shown aswell as a link (inside see attribute) to the XLIFFspecification. The <name> element returns the nameof the picked node (target in this case). Thiselement becomes very helpful when dealing withseveral nodes at the same time and it will bediscussed later. Finally, beside all that, thanks to<let> elements, which are variables in Schematron,we can save additional information about the error. Inthis example, we are retrieving the id attribute of theenclosing <unit> element as well as the immediateparent of the <target> element which caused theerror (it can be either <segment> or

<ignorable>). Then by taking advantage ofdiagnostic mechanism of Schematron, an advanceddiagnostic message is formed and delivered to theuser which makes the error tracking and furtherfixing much faster and more efficient. It is notablethat messages are human-made and are consideredindividually rather than machine-generated generalerrors.

One of the other examples that offers a good

45

Listing 2: trgLang check in Schematron


46

comparison between the Relax NG and Schematronapproaches is the <skeleton> element’sconstraint. The <skeleton> element must containeither an href attribute and no text (empty) or textwithout the attribute. Listing 3 shows how this issueseems to have been resolved in the Relax NGschema;

Schema again is giving two valid scenarios; either theelement is empty and has the attribute, or it is notempty and no attributes are present.

However, both of the following invalid patterns willpass the validation against this schema;<skeleton/> and <skeleton></skeleton>. The point is that an empty elementor a white-space is still considered text in Relax NG,which allows the mentioned patterns to pass althoughin fact not valid according the normative prose

specification.

Listing 3 shows how this constraint can be addressedby a Schematron rule defining all the possible cases.

This rule perfectly targets the <skeleton>constraint by describing it within the expressioninside the test attribute.

Schematron can cover all remaining XLIFFnormative statements including the most importantand sophisticated ones, i.e. uniqueness of idattribute values among elements in differentuniqueness scopes of an XLIFF document.

ID-uniqueness in XLIFF 2.0 comes in three distinctlevels of complexity. First, when children elements(i.e. siblings to each other) must have unique idvalues within their parent element. This is the case of<file> elements within the root <xliff>element. The second level then requires elementsgrandchildren to have unique identifiers within theentire grandparent scope: the case of <data>elements within <unit> elements. These first twotypes are fairly easy to handle, so we will discuss indetail only the third and most complex one.

Unit level in XLIFF is a logical container for arelatively independent portion of content. Theextracted text along with codes, represented byXLIFF inline elements, are stored in <source>elements, and later the translations will be added to<target> elements, both grandchildren of<unit>; each pair of the <source> and<target> siblings is placed within a <segment>or an <ignorable> element in order to capturesegmentation of the content unit.

The constraints and other provisions for the unit levelare provided and explained in many different placesin the specification, and moreover many of them areconditional to specific governing attribute values.

Listing 3: skeleton “pseudo-solution”, Relax NG

Listing 4: <skeleton> solution, Schematron


47

The most general, though, is that any id attributemust be unique among all <segment>,<ignorable>, <mrk>, <sm>, <pc>, <sc>,<ec>, or <ph> elements within the enclosing<unit> element. But, the inline elements inside<target> must use the duplicate id values of theircorresponding inline elements in the <source>sibling, as long as such corresponding elements doexist, given the conditional logic of specificgoverning attributes.

The trickiest validity conditions to check occur here.The translated text of a content unit may be presentedin a different order, compared to the original text.This means that the <target> sibling content doesnot need to logically or linguistically correspond tothe content of its <source> sibling, provided that adifferent order has been explicitly specifies throughthe order attribute that is optional at the<target> element.

The order attribute defines the actual position of its<target> element in the sequence of the unitcontent. Therefore the corresponding inline elementsmust be searched for inside the relevant <source>that can be a “cousin” (same grandparent, differentparent) of the <target>. The value of the orderattribute is a positive integer between 1 and N, whereN is the number of <segment> and<ignorable> elements within the unit. Finally,inline elements that have their canDelete attributeset to ‘no’, must appear in the corresponding<target>, but other elements may be deleted,because the default value of the canDeleteattribute is ‘yes’.

As the first step for this task, we will check theoutermost elements of the unit; <segment> and<ignorable>. The id is optional for these

elements, so only those with id attributes will beselected and checked for duplication within the unit.Listing 5 demonstrates this step.

Here, the variables pick the elements which followthe node but still within the same <unit>, itspreceding elements and all its descendants. Theremust not be any node matching this pattern.

As the second step, all inline elements inside<source> elements will be checked for duplicationamong themselves, which is shown in the Listing 6.

For the demonstration purposes, the rules wereshortened for inclusion in this paper; the actual rulesprovided on the XLIFF TC SVN repository providedetailed error messages that are rigorously based onthe normative statements of the specification.

From now on (for the third ID-uniqueness validationstep), we are going to assume that the validatedXLIFF instance is in the final stage of validation(going to merge back into the original format as thenext workflow stage), which means that all rules thatapply to the <target> element are enforced, whilethis element is strictly speaking optional (from thespecifications point of view) at any other stage.

This third step consists itself of sub-steps. First, wecheck whether values (explicit and implicit) of theorder attributes are legal (between 1 and N) andthen for their uniqueness using the method explainedearlier. Then, all inline elements inside the<target> will be verified against all of other inlineelements except themselves; they are allowed to haveone element duplicating their id in thecorresponding <source>, we will look for them abit later.

Listing 5: id uniqueness, first step, Schematron


48

Listing 6: Inline id uniqueness check, Schematron

Listing 7: order constraints described in Schematron


The next sub-step of the third step is to check if theorder attributes are used correctly within units. If a<target> element’s natural order in the unitsequence has been overridden by explicitlyspecifying a different position via the orderattribute, the <target>, which had occurred at thenatural position corresponding to the former<target> element’s explicitly set order, the latterhas been “displaced”, and therefore has to use its ownorder attribute to specify another available positionwithin the unit. And so forth, until all target segmentshave explicitly set available positions, or otherwiseoccur on positions with their natural order not“displaced” by any other explicitly set orderattributes . Listing 7 demonstrates the rule forvalidating this.

This rule will first identify those order attributesthat use an illegal positive integer. The naturalpositions within the sequence are given by the ordinalnumbers designating the positions of the <source>elements within the given unit, so the rule searchesfor <target> elements, which are not at the naturalposition (the order attribute may actually match theactual position), and saves the values, then checks for<target> elements at those positions that in turnmust have an explicitly set order attribute. Thisloop continues until the rule verifies that all of the<target> elements are ordered properly or until aviolation is identified.

After the corresponding <source> elements havebeen identified, previously described methods areused to match legal inline ID duplications betweentarget and source content.3. Web Service

To implement Relax NG and XSD validation in Java,the Java library developed by the Thai Open SourceSoftware Center, Jing (Clark, 2008) can be used. Thislibrary, as well as other standard Java libraries forXML processing, provides sufficient tools to developa Java programme, which can validate XMLdocuments against the given schemas; XSD or RNGby user choice. This programme also returns a list ofidentified errors in case of a failed validation.

Based on the above, a RESTful web service wasdeveloped that consumes (receives) an *.xlf filewithin an HTTP POST request and produces(returns) the validation result (along with the errorreport, if any errors or warnings occurred) in JSONformat. This web service can be called to validateXLIFF instances against the XSD or RNG schemasor both. The functionality of the web service isillustrated with the screenshots below, with an XLIFFfile passing the validation and another failing.

Integrating the Schematron rules is planned as thenext step and will give more informative messages incase of a validation failure.

4. Next Steps towards the full advancedvalidation of XLIFF 2

By using the above demonstrated Schematronmethods and techniques, the XLIFF core can bedescribed in full, and hence fully validated withDSDL methods only. The expressivity of Schematron(together with XSD) is sufficient to guarantee the fullvalidation of the XLIFF core and modules includingthe fragment identifiers. All DSDL based methods ofXLIFF validation originating from this research arebeing proposed as the technical solution (normative

49

Picture 1: The instance passes the validation

Picture 2: The instance fails the validation


artefacts to become part of the multipart standard) forthe Advanced Validation feature of XLIFF 2.1 via theOASIS XLIFF TC.

To be able to check arbitrary XLIFF files includingmodules (and extensions), against all providedschemas (schemas for extensions can be alsointegrated if provided), constraints and processingrequirement, and in order to handle dynamicvalidation, NVDL (Namespace-based ValidationDispatching Language) needs to be used. This willprovide a suitable mapping tool for the advancedvalidation of different parts of XLIFF documentagainst different namespaces, using mixed DSDLmethods and schema languages. Eventually, alsoNVDL based techniques will be embedded in thevalidation web service that is being developed basedon this research. The web service, however, is notexpected to achieve any normative status through theXLIFF TC due to being implementation dependent.Nevertheless, the web service should be madepublicly available by the CNGL within the first halfof 2015 in order to provide easy access to theadvanced validation methods for the standard’s endusers.

References

Bray, T., Paoli, J., Sperberg-McQueen, C.M., MalerE., Yergeau, F. (Eds.) (2008) XML Version 1.0 (FifthEdition) [online], available:http://www.w3.org/TR/2008/PER-xml-20080205/[accessed 15 Mar 2015].

Brown, A. (2010) Document Schema DefinitionLanguages [online], available: http://www.dsdl.org/[accessed 12 Mar 2015].

Clark J. (2008) A RELAX NG validator in Java[Computer program], available:http://www.thaiopensource.com/relaxng/jing.html[accessed 1 Nov 2014].

Clark, J., Makoto, M. (Eds.) (2001) RELAX NG[online], ISO standard, available: https://www.oasis-open.org/committees/relax-ng/spec-20011203.html[accessed 10 Nov 2014].

Comerford, T., Filip, D., Raya, R.M., Savourel, Y.(Eds.) (2014) XLIFF Version 2.0 [online], OASISStandard. ed, Standard, OASIS, available:http://docs.oasis-open.org/xliff/xliff-core/v2.0/os/xliff-core-v2.0-os.html [accessed 22Aug 2014].

Delima, N., Gao, S., Glavassevich, M., Noaman, K.(2009) XML Schema 1.1, Part 2: An introduction toXML Schema 1.1 [online] available:http://www.ibm.com/developerworks/library/x-xml11pt2/ [accessed 20 Mar 2015].

Gao, S., Sperberg-McQueen, C.M., Thompson, H.S.(Eds.) (2012) XSD 1.1, w3c [online] available:http://www.w3.org/TR/xmlschema11-1/ [accessed12 Mar 2015].

Jelliffe, R. (Ed.) (2002), ISO Schematron,[online],ISO standard, available:http://www.schematron.com/spec.html [accessed 16Dec 2014].

Quin, L.R.E., Fawcett, J., Ayers, D. (2012)Beginning XML, 5th ed., Indianapolis: John Wiley& Sons.

Raya, M.R. (2012) XLIFFChecker (Version 1.0-8)[Computer program], available:http://www.maxprograms.com/products/xliffchecker.html [accessed 13 Mar 2015].

Saadatfar, S., Relax NG Schema for XLIFF 2.0 core[Computer program], available: http://tools.oasis-open.org/version-control/browse/wsvn/xliff/branches/Soroush[accessed 1 Feb 2015]

Savourel, Y., Okapi-lynx XLIFF 2.0 Validation[Computer program], available: http://okapi-lynx.appspot.com/validation [accessed 18 Mar2015].

Schnabel, B., XLIFF 2.0 XMarker Checker[Computer program], available:http://xmarker.com/node/13 [accessed 13 Mar2015].

Thompson, H.S., Beech, D., Malony, M.,Mendelsohn, N. (Eds.) (2004) XML Schema, w3c[online], available:http://www.w3.org/TR/xmlschema-1/ [accessed 12Mar 2015].

Vlist , E. (2003) RELAX NG, Sebastopol: O’Reilly& Associates.

w3schools XML Tutorial [online], available:http://www.w3schools.com/xml/ [accessed 19 Dec2014].

50

advanced validation techniques for xliff 2 › sites › default › files › ... · the third...

Documents