alfred raczkowsky connect to struval - circabc.europa.eu · slide 4 | 22 may 2017 webinar...

www.statistik.at We provide information

Connect to StruVal Server at ESTAT - PoC

Alfred Raczkowsky Alexander Bertsch

Statistics Austria

Luxembourg 09 June 2017

www.statistik.at slide 2 | 22 May 2017

Agenda

• Aim of StruVal PoC

• Webinar

• Preparation for SoapUI test

• Test in SoapUI

• Preparation for Java command line client

• Test with Java command line client

• SOAP Response for error-free data file

• SOAP response for data file with errors

• Non-numerical values

• Conclusions


Aim of StruVal PoC

• The StruVal PoC is based on SDMX Converter version 6.0.2.

• PoC: Use StruVal as a web service centrally hosted in Eurostat.

• The web service offers the operations „validate“ and „customValidation“.

• Structure Validation is validation against the DSD: check if the file format is SDMX compliant, check against the code lists, check if numbers are numeric.

• The validation service produces a SOAP validation response.


Webinar

• Participants of the webinar on 18 January 2017: AT, FR, IE, LU, PT

• StruVal WSDL much smaller than SDMX Converter WSDL

• Web Service Clients for the Test

SoapUI (Open Source)

Simple Java Command Line Client (Eurostat)

• Service only available with authentication (user, password)

• Only for non-confidential data


Preparation for SoapUI test

• Tests in SoapUI v5.2.1 as used for the webinar

Create a new SOAP project

• Add the service WSDL

• Test uses the same DSDs and data files as with Java command line client


Test in SoapUI (1)

Only operation „validate“ was used, because „customValidation“ was still under development

MTOM (Message Transmission Optimization Mechanism)

was activated in the Request Properties: ForceMTOM = true (EnableMTOM = true is default)


Test in SoapUI (2)


Test in SoapUI (3)

Edit the SOAP request

<v1:inputData>cid:data</v1:inputData> <v1:dsdStructure>cid:dsd</v1:dsdStructure> <v1:maxErrorNumber>10</v1:maxErrorNumber>


Test in SoapUI (4)

Attach the SDMX-ML data file and the DSD file

Change content type to „application/xml“

Change entry in column „Part“ to „data“ resp. „dsd“

Change value in column „ContentID“ to same value as for column „Part“ (recommended by Eurostat)

Entry in column „Type“ changes to „XOP“

The DSD must contain the same dataflow as the one used for the creation of the data file.


Preparation for Java command line client

• Download the Apache CXF framework from http://cxf.apache.org/ and include it into the project

• Installing the Maven project creates struval-client.jar

Command: mvn clean install

• Compile TestStruvalClient.java

• Edit and execute batch file runStruVal.bat

• Test uses the same DSDs and data files as with SoapUI

http://cxf.apache.org/

http://cxf.apache.org/


Test with Java command line client

• Run batch command file runStruVal.bat:

Set the environment variables: SET JAVA_HOME=C:\Progra~1\Java\jdk1.8 SET CXF_HOME=%~dp0lib\apache-cxf-3.1.9 SET CLASSPATH=%JAVA_HOME%\lib; %CXF_HOME%\lib\*; target\*

Set the parameters: set DataFile="D:\StruVal_Tests\NA_Test\NAMAIN_T0101_A_AT.xml" set DSDFile="D:\StruVal_Tests\NA_Test\NA_MAIN_2.0_V1.4.xml“

Call of TestStruvalClient in runStruVal.bat: eu.eurostat.struval.TestStruvalClient %user% %userpw% %DataFile% %DSDFile% (change DataFile and DSDFile for every execution)


SOAP Response for error-free data file

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope">

<soap:Header/>

<soap:Body xmlns:S="http://schemas.xmlsoap.org/soap/envelope/">

<stru:validateResponse xmlns:stru= "http://xmlns.ec.eu/BusinessActivityService/Core/StruVal/V1“>

<stru:return xmlns:struerr="http://xmlns.ec.eu/BusinessObjects/Core/StruVal/ErrorReport/V1">

<struerr:errorsFound>0</struerr:errorsFound>

<struerr:moreErrors>false</struerr:moreErrors>

<struerr:returnCode>0</struerr:returnCode>

</stru:return>

</stru:validateResponse>

</soap:Body>

</soap:Envelope>

http://xmlns.ec.eu/BusinessActivityService/Core/StruVal/V1


Manipulated data row with errors

• We modify a data row in the xml data file with errors (in red). • Number of passengers: 70, but only 7 seats available

<air:Series FREQUENCY="Y" TABLE="A1" COUNTRY="LO"

AIRPORT="LOWX" PART_AIRPORT="EBLG" DIRECTION="10"

SCHEDULED="2" SERVICE="1" AIRLINE_INFO="1EU"

AIRCRAFT_TYPE="C650">

<air:Obs TIME_PERIOD="2016_01" OBS_VALUE="70"

FREIGHT_MAIL="0" COMMERCIAL="1" SEATS_AVAIL="7"/>

</air:Series>


SOAP response for data file with errors

• <struerr:string>Dimension 'AIRPORT' is reporting value 'LOWX' which is not a valid representation in referenced Codelist 'urn:sdmx:org.sdmx.infomodel.codelist.Codelist=ESTAT:CL_AIR_AIRPORT(4.1)'</struerr:string>

• <struerr:string>Dimension 'DIRECTION' is reporting value '10' which is not a valid representation in referenced Codelist 'urn:sdmx:org.sdmx.infomodel.codelist.Codelist=ESTAT:CL_AIR_DIRECTION(4.1)'</struerr:string>

• <struerr:string>Dimension 'DIRECTION' is reporting value '10' which is ' is greater then the maximum allowed length of '1'</struerr:string>

• <struerr:description> <struerr:string>Invalid Date Format `2016_01`</struerr:string> </struerr:description>


SOAP response detail for one error

• Error code is always 150: <struerr:error>

<struerr:attachedTo>SERIES</struerr:attachedTo> <struerr:code>150</struerr:code> <struerr:column>0</struerr:column> <struerr:description> <struerr:string>Dimension 'DIRECTION' is reporting value '10' which is greater then the maximum allowed length of '1'</struerr:string> </struerr:description> <struerr:dimensions> … </struerr:dimensions> <struerr:errorClass>DATA</struerr:errorClass> <struerr:fatal>false</struerr:fatal> <struerr:line>0</struerr:line>

</struerr:error>


Non-numerical values

• Change: OBS_VALUE="7X"

• <struerr:description> <struerr:string>Primary Measure 'OBS_VALUE' is reporting value '7X' which is ' is not of expected type 'Integer‘ </struerr:string> </struerr:description>

• StruVal checks if values are numeric and if numbers are integer


Conclusions (1)

• Error reporting was only SOAP XML response

• For every single error all elements of the data row were repeated

• Some error messages are incomprehensible (like „Could not resolve reference from structure“)

• For automatic processing error codes are necessary (even in production system error code is always 150)

• The report should show which validation rule was violated

• Validation didn´t have good performance (several minutes)

either slow transmission (13 MB)

or slow processing (57600 data rows)


Conclusions (2)

• It would be of high value if Eurostat would provide a comfortable StruVal Client:

on a central server in Eurostat premises and/or

for installation in the MSs

equiped with GUI and command line interface (like SDMX converter) with a parameter file


Please address queries to: [email protected] [email protected]

Contact information: Guglgasse 13, 1110 Vienna phone: +43 (1) 71128-7145 phone: +43 (1) 71128-7375

fax: +43 (1) 714 75 50

2.3.2 Connect to StruVal Server at ESTAT - PoC

alfred raczkowsky connect to struval - circabc.europa.eu · slide 4 | 22 may 2017 webinar...

Documents