automated analytics administration guide - help.sap.com

CUSTOMER

SAP BusinessObjects Predictive Analytics 3.12017-10-26

Unicode CompatibilityAutomated Analytics Administration Guide

Content

1 Unicode Design Principles and Impacts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.1 Impacts on the data access. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Access to ODBC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5User-Space. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2 User Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61.3 Using the APIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.4 Code generation impacts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Conclusion for Unicode Impacts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

3 Annex: C/C++ code snippets for string conversion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2 C U S T O M E RUnicode Compatibility

Content

1 Unicode Design Principles and Impacts

The software releases prior to version 3.1.0 of SAP InfiniteInsight were ‘character set agnostic’, which means that whatever character set was used in the data was used in the application’s internal representations. This could lead to some problems when mixed with strings automatically generated within the application, which are coded in the ASCII character set (examples of such strings are names of parameters, or names of generated variables). These strings could potentially mix two sub-strings coded in different character sets.

Dealing with a normalized representation of strings in this application has impacts of the user on several sides:

● Data Access: Data can contain strings in their own character set.● User Interfaces: The application user interfaces have been adapted for the new architecture.● API usage: Strings are used as arguments in many function calls.● KMX: Generated codes are now compatible with Unicode data.

In order to be able to append internally generated strings, and strings coming from the external world, we needed an internal normalized representation that can handle strings representations in any character set. This is why we have chosen to represent all strings in UTF-8.

As we wanted to keep the compatibility with previous releases of the application, we had to make sure we could translate strings contained into files, for example into UTF-8. This implies to know the character set in which this data is encoded. We have adopted the convention to use the ‘server-native character set’ of the application server as the default source character set. This conversion has a slight impact on the performance for file data access.

Full Documentation

Complete documentation can be found on the SAP Help Portal at http://help.sap.com/pa.

Conversion implementation details

The default character set used to read and write data can be forced to be UTF-8 for the application through the configuration file variable: MultiLanguageIsDefault set to ‘true’.

When MultiLanguageIsDefault is set to ‘False’, which is the default, the native character set of the server is used. This ‘server native character set’ can be found, on UNIX, through the LANG environment variable, and, on Windows, through the CODEPAGE environment variable if exists, if not the logical constant CP_ACP. It must be noted that the retrieval of the native char set on Windows is not as clearly defined as it could be (there are some subtle differences if the application is in windows or console mode for example).

We will call below ‘default character set’ either the native char set or UTF-8 depending on the value of MultiLanguageIsDefault.

In the next sections, we will present the impacts of this new architecture on data access, user interfaces, and library usage by integrators.

Unicode CompatibilityUnicode Design Principles and Impacts C U S T O M E R 3

http://help.sap.com/pa

1.1 Impacts on the data access

Impacts on the data access layers can always be decomposed into two parts:

● Reading and writing string data properly● Reading and writing meta-data properly

Access to Files

Concerning the data, since version 3.1.0 of SAP InfiniteInsight, the application has been able to read files encoded in ‘server native character set’, UTF-8. Files stored using other character sets must be translated prior their usage within the application.

Concerning the metadata description (such as the file and directory names), in order to translate any file information to the application, we are using the following:

● On UNIX: Wide string calls from the operating system and iconv conversions (which must be properly setup for the conversions between the character set used in the OS and at least UTF-8).

● On Windows (> 2K): wide string calls from the operating system, and, because these version of Windows rely on Unicode storage, no extra conversion.

Reading DataSince version 3.1.0 of SAP InfiniteInsight, the application has been using the magic number to recognize files in UTF-8 (0xEFBBBF). By default, any existing file with no magic number is read as if encoded in the ‘default character set’. The variable MultiLanguageIsDefault allows coping with the situations where files encoded into UTF-8 do NOT contain the proper magic number.

Writing Data: Creating a new fileSince version 3.1.0 of SAP InfiniteInsight, any new file generated by the application is supposed to be encoded using the ‘default character set’. This is particularly true for the ‘KxAdmin’ files, and the files used to save models.

For the newly generated data files (containing the output scores for example), the user has the possibility (though the Locale parameter of the space) to force the character set within ‘native character set, UTF-8).

Writing Data in an existing fileAppending new data in an existing file will take the character set of the previously existing file found through the simple logic using the magic number and the configuration variable MultiLanguageIsDefault.

Restoring/Writing models from/to filesWhen restoring a model from version 2, the application reads the description in ‘KxAdmin’ that will be assumed to be in ‘default character set’. The file containing the model will be assumed to be in ‘default character set’ also.

When saving a model into an existing file (a new version of a model previously saved in version 2 for example), the application will append the model data to the existing file: It is the responsibility of the user to check that the encoding is compatible (do not force a UTF-8 character set for models in version 3 as models in version 2


Unicode Design Principles and Impacts

are already in the same files using native character set for example). When the user wants to save a model into a directory in which he has already saved some models and when this model contains strings that are not compatible with the ‘default character set’, he must save a model with a name compatible with ‘default character set’ and ask for a nonexistent file in which to save the model (which will be created according to the server default strategy).

1.1.1 Access to ODBC

Concerning the data, since version 3.1.0 of SAP InfiniteInsight, the application has been using standard ODBC calls, and has assumed that the drivers respect standard ODBC Unicode convention:

● When reading any value as CHAR, ODBC layer always translates the value in the character set of the server asking for the value whatever internal storage the DBMS uses.

● When reading any value as WCHAR, ODBC layer always translates the value in UTF-16 format, whatever storage the DBMS uses.

The application asks all ODBC data in CHAR storage if MultiLanguageIsDefault is false (the default), and data in WCHAR storage if MultiLanguageIsDefault is true and if the DBMS is known to actually support it.

Concerning the meta-data, since version 3.1.0, the application has been using the wide string calls to ODBC if the DBMS is known to actually support it.

Writing Data: Creating a new Table

Strings columns are created using the VARCHAR type or the DBMS native Unicode type following the value of MultiLanguageIsDefault. Please note that, when a key variable is declared as the DBMS Unicode type in the original table, we create it as a VARCHAR in the score table if MultiLanguageIsDefault is false. This small inconvenience is corrected in a later release: for this we plan to keep the physical storage type to be able to recreate a column containing this data with the same physical storage.

Restoring/Writing models from/to tables

As for files, the system uses MultiLanguageIsDefault in order to determine if models should be saved using Unicode or ‘server native character set’.

1.1.2 User-Space

The application comes with an internal C API that can be used to connect to custom storage. The signature of this API has not changed with respect to Unicode feature (see the API section).

Strings arguments are still provided and returned as char*, but the semantics of these char arrays has changed: When declaring the User Data Access in the configuration file, the implementer has to provide a new


configuration setting for this User Data Access which is MultiLanguage (the default value uses the global MultiLanguageIsDefault). This allows having different behaviors for each User Data Access.

For those who want to provide User Data Access with this flag at true, the application provides some code snippets in C/C++ that can be used as templates for string conversions.

Furthermore, this API has been augmented with new functions calls to be able to transfer more meta-information such as the length and precision of fields, as well as some error reporting facilities.

For example, supposing you have your own data access library, which supports natively UTF-8 encoding, you should configure it as follows:

UserStore MyData: MyDataLibUserStoreOPtion.MyData.MultiLanguage true

1.2 User Interfaces

There are two major user interfaces delivered with Automated Analytics. Impacts are described below. Interfaces have been upgraded following the recommendations that we give for any API users (see section on the APIs).

Java Wizard Interface

Java natively manipulates strings in Unicode format. Wrappers for both the C++ and CORBA APIs have been upgraded. Be aware that, even if the software itself handles Unicode strings, the proper display of these strings depends on the fonts that have been installed with the Java environment.

Because of the changes needed in the CORBA layer to transport Unicode strings and because of the changes in the CORBA server implementation (now fully POA), the Java runtime environment version must be 1.6 or above.

KxShell Interface

The application has updated KxShell source code as any regular user of the C++ API with string encoding and decoding. These operations must be used to read shell scripts and to display strings, because C routines (as the C functions “xprintf” for example) expect strings encoded in ‘server native character set’.

Furthermore, in order to force the reading of a UTF-8 script, the user can specify –UTF8 when firing KxShell.

Similarly, by default KxShell assuming that scripts are encoded in native charset, the output of the interpreter is converted back to native characters. An option -UTF8OUT has been added if UTF-8 output is desirable.

The encoding/decoding source code example is provided in annex of this document.



1.3 Using the APIs

Since version 3.1.0, the application has been delivered with Unicode internal management, as opposed to previous versions 3.0.x. Strings arguments are still provided and returned as char*, but the semantics of these char arrays has changed since the application expects strings to be encoded in UTF-8 (instead of an undefined character set as previously assumed).

C++ API

When integrating within C++, all strings going into or coming from the API are encoded in UTF-8. An example code is provided in annex.

CautionIntegrators that are using the C++ API have to upgrade their code in order to manage eventual conversion between ‘server native character sets’ and UTF-8, which is the same integration exercise that the application has performed for the migration of KxShell.

CORBA

The CORBA API has not changed, but, again, the semantics of the strings used in our calls have changed as they are now encoded in UTF-8. As CORBA manages the strings transport between inhomogeneous environments, it must be character set aware (in order to force eventual conversions). This is possible if the CORBA GIOP version is above 1.2.

Name servers making the junction between CORBA clients and servers must be compatible with the 'nativeCharCodeSet' option.

If the omniORB name service is used, all CORBA servers should be launched with the ‘supportBootstrapAgent’ option set to 1 for inter-operability.

COM-DCOM

In COM/DCOM, all API strings are encoded/decoded in UTF-16 within specific wrappers because UTF-16 is the default format for Windows > 2K: This way, no conversion is needed in this framework.

Integrators using Unicode-compatible graphical layers from Visual Basic or C# have no extra-developments to perform. In case of Microsoft environments, we strongly recommend to switch to ‘.Net’ environment which allows users to use a packaged Unicode graphic layer (all basic controls in the .Net environment are Unicode enabled).


Java Wrappers

We have upgraded all Java wrappers for both the Java/JNI and Java/CORBA wrappers. Integrators that have used previous versions of the Java wrappers can upgrade to the new versions.

1.4 Code generation impacts

If data can contain now Unicode strings, because these strings can appear in generated code, the code generator has been upgraded to manage this situation.

XML/PMML/JavaScript

As PMML uses UTF-8 as its natural character set, the generated PMML codes can directly embed UTF-8 characters, and the code is correctly applied against UTF-8 data. Of course, the PMML interpreter used to execute the generated PMML code must also support UTF-8 PMML models and processing of UTF-8 data.

C

The generated C code has been adapted. Strings are not anymore user-readable, but the behavior is correct.

Java

The generated Java code has been adapted: the UTF-8 data is generated as byte array, and then properly converted to Java Strings objects.

1.6.4 SQL/UDF

The generated SQL and UDF code has been adapted. Precise support of each database is specified in appendix.

SAS

We have not upgraded SAS code, so it can only process data in native charset.



2 Conclusion for Unicode Impacts

We report here the main impacts for integrators and users of the application since SAP InfiniteInsight version 3.1.0.

● Java runtime environment version must be 1.6 or above.● The integrators who are using the C++ API have to upgrade their code in order to manage possible

conversion between ‘server native character sets’ and UTF-8, as the® API expects strings to be encoded in UTF-8.

Unicode CompatibilityConclusion for Unicode Impacts C U S T O M E R 9

3 Annex: C/C++ code snippets for string conversion

The following code is used in our KxShell interface and is provided for integrators that have to integrate the API in C/C++ or want to write User-Access libraries compatible with versions of SAP InfiniteInsight 3.1.0 onwards.

For UNIX

Here is the example of the UNIX version:

#ifndef Kxen_NativeFile_UNIX_cpp #define Kxen_NativeFile_UNIX_cpp#if 0// ----------------------------------------------------------------------------// Copyright....: (c) KXEN 1999-2001// Project......: KXEN Analytic framework// Library......: KxShell// File.........: NativeFile_UNIX.cpp// Author.......: Serge Danzanvilliers// Created......: Mon Jul 21 13:08:42 2003// Description..: // // CVS infos....: // ... $Id: NativeConv_UNIX.cpp,v 1.14 2004/04/20 14:42:50 port Exp $// ... $Date: 2004/04/20 14:42:50 $// ... $Revision: 1.14 $// ----------------------------------------------------------------------------#endif#include "UCommon.h"#include "NativeConv.h"#include <assert.h>/*****************************************************************************/// A class managing the instanciation of native iconv convertersclass cIconvNativeTrans{public: static cIconvNativeTrans& getTheIconv () { static cIconvNativeTrans lTheIconv; return lTheIconv; } ~cIconvNativeTrans (); iconv_t getFromNativeConverter () const { return m_NativeToUTF8_; } iconv_t getToNativeConverter () const { return m_UTF8ToNative_; } char const* getCP() { return m_codepage.c_str(); }private: cIconvNativeTrans (); std::string m_codepage; iconv_t m_NativeToUTF8_; iconv_t m_UTF8ToNative_;


Annex: C/C++ code snippets for string conversion

};/*****************************************************************************//*****************************************************************************/cIconvNativeTrans::cIconvNativeTrans () : m_NativeToUTF8_ (0), m_UTF8ToNative_ (0){ // The LANG environment variable is used to get the // native character set. Be sure it is defined to // whatever value you need before starting. char const* lLANG = getenv ("LANG"); if (lLANG == NULL) { // if LANG is not defined we assume the native character set // is latin1 (western europe). lLANG = Kx_LATIN1_str; } assert (lLANG != NULL); // The LANG variable value should have the following format : // en_US.ISO8859-1 (sample) // We are only interested ing the last part (ISO8859-1), hence // the following lines of code. std::string lLANGstr (lLANG); m_codepage = lLANGstr; std::string::size_type idx = lLANGstr.find ('.'); std::string charset; if (idx == std::string::npos) { charset = Kx_LATIN1_str; } else { charset = lLANGstr.substr (idx + 1); } // Instanciation of iconv converters. m_NativeToUTF8_ = iconv_open (Kx_UTF8_str, charset.c_str()); if (m_NativeToUTF8_ == reinterpret_cast<iconv_t> (-1)) { fprintf (stderr, "Unable to instanciate iconv converter from %s to UTF-8\n", charset.c_str()); exit (-1); } m_UTF8ToNative_ = iconv_open (charset.c_str(), Kx_UTF8_str); if (m_UTF8ToNative_ == reinterpret_cast<iconv_t> (-1)) { fprintf (stderr, "Unable to instanciate iconv converter from UTF-8 to %s\n", charset.c_str()); exit (-1); }}/*****************************************************************************/cIconvNativeTrans::~cIconvNativeTrans (){ // Clean up. if (m_NativeToUTF8_ != reinterpret_cast<iconv_t> (-1)) { iconv_close (m_NativeToUTF8_); } if (m_UTF8ToNative_ != reinterpret_cast<iconv_t> (-1)) { iconv_close (m_UTF8ToNative_); }}/*****************************************************************************/char const*get_native ()

Unicode CompatibilityAnnex: C/C++ code snippets for string conversion C U S T O M E R 11

{ return cIconvNativeTrans::getTheIconv().getCP();}/*****************************************************************************/// Conversion from native to UTF8 charset.// We take a char vector holding an empty terminated string as input.// The string is then converted in place.voidnative_to_UTF8 (std::vector<char>& ioLine){ size_t slen = ioLine.size() + 1; size_t tlen = 6 * slen; std::vector<char> lRaw (tlen); char const* in = &ioLine[0]; char* out = &lRaw[0]; size_t o = iconv_wrap (cIconvNativeTrans::getTheIconv().getFromNativeConverter(), &in, &slen, &out, &tlen); if (o == (size_t)(-1)) { fprintf (stderr, "Iconv failed (native to UTF8).\n"); } lRaw.resize (strlen(&lRaw[0]) + 1); ioLine.swap (lRaw);}/*****************************************************************************/voidUTF8_to_native (std::vector<char>& ioLine){ size_t slen = ioLine.size(); size_t tlen = slen * 6; std::vector<char> lRaw (tlen); char const* in = &ioLine[0]; char* out = &lRaw[0]; size_t o = iconv_wrap (cIconvNativeTrans::getTheIconv().getToNativeConverter(), &in, &slen, &out, &tlen); if (o == (size_t)(-1)) { fprintf (stderr, "Iconv failed (UTF8 to native).\n"); } lRaw.resize (strlen(&lRaw[0]) + 1); ioLine.swap (lRaw);}/*****************************************************************************/#endif

For Windows

The following code is for Windows:

#ifndef Kxen_NativeFile_WIN32_cpp #define Kxen_NativeFile_WIN32_cpp#if 0// ----------------------------------------------------------------------------// Copyright....: (c) KXEN 1999-2001// Project......: KXEN Analytic framework// Library......: KxShell// File.........: NativeFile_Win32.cpp// Author.......: Serge Danzanvilliers// Created......: Thu Jul 17 17:27:13 2003



// Description..: Native encoded file access on WIN32 platforms.// // CVS infos....: // ... $Id: NativeConv_WIN32.cpp,v 1.5 2004/05/04 17:44:12 bertrand Exp $// ... $Date: 2004/05/04 17:44:12 $// ... $Revision: 1.5 $// ----------------------------------------------------------------------------#endif#include "NativeConv.h"#include <stdlib.h>#include <windows.h>/*****************************************************************************/static UINT gCODEPAGE = CP_ACP;/*****************************************************************************/// A class responsible for finding the default codepage.struct cKxShellNativeCodePageFinder{ cKxShellNativeCodePageFinder () { // The environment variable CODEPAGE is used // to get the native codepage. You can use it // if windows uses a strange one by default // (which is very often indeed). char const* lCode = getenv ("CODEPAGE"); if (lCode != NULL) { std::string lCODEPAGEstr (lCode); if (!lCODEPAGEstr.empty()) { UINT lCP = atoi(lCODEPAGEstr.c_str()); if (lCP != 0) { gCODEPAGE = lCP; } } } char codepage[64]; sprintf (codepage, "%lu", gCODEPAGE); m_codepage = codepage; } std::string m_codepage; } gKxTheNativeCodePage;/*****************************************************************************/char const*get_native (){ return gKxTheNativeCodePage.m_codepage.c_str();}/*****************************************************************************/// To perform the conversion we use the standard windows functions// MultiByteToWideChar and WideCharToMultiByte. That is we convert// first from native to UTF16, then from UTF16 to UTF8.voidnative_to_UTF8 (std::vector<char>& ioLine){ int lNbRequiredWChars = MultiByteToWideChar (gCODEPAGE, MB_PRECOMPOSED, &ioLine[0], ioLine.size(), NULL, 0); if (lNbRequiredWChars > 0) { std::vector<char> lRaw; lRaw.resize (lNbRequiredWChars * sizeof (WCHAR)); if (MultiByteToWideChar (gCODEPAGE,


MB_PRECOMPOSED, &ioLine[0], ioLine.size(), (LPWSTR)(&lRaw[0]), lNbRequiredWChars)) { int lNbReqUTF8 = WideCharToMultiByte (CP_UTF8, 0, (LPWSTR)(&lRaw[0]), lNbRequiredWChars, NULL, 0, NULL, NULL); if (lNbReqUTF8) { ioLine.resize (lNbReqUTF8); WideCharToMultiByte (CP_UTF8, 0, (LPWSTR)(&lRaw[0]), lNbRequiredWChars, &ioLine[0], lNbReqUTF8, NULL, NULL); } } }}/*****************************************************************************/voidUTF8_to_native (std::vector<char>& ioLine){ int lNbRequiredWChars = MultiByteToWideChar (CP_UTF8, 0, // Required when using UTF8 &ioLine[0], ioLine.size(), NULL, 0); if (lNbRequiredWChars > 0) { std::vector<WCHAR> ubuffer (lNbRequiredWChars); if (MultiByteToWideChar (CP_UTF8, 0, &ioLine[0], ioLine.size(), &ubuffer[0], lNbRequiredWChars)) { int lNbRequiredNative = WideCharToMultiByte (gCODEPAGE, 0, &ubuffer[0], lNbRequiredWChars, NULL, 0, NULL, NULL); if (lNbRequiredNative) { std::vector<char> nbuffer (lNbRequiredNative); WideCharToMultiByte (gCODEPAGE, 0, &ubuffer[0], lNbRequiredWChars, &nbuffer[0], lNbRequiredNative, NULL, NULL);



ioLine.swap (nbuffer); } } } }/*****************************************************************************//*****************************************************************************//*****************************************************************************/#endif // Kxen_NativeFile_WIN32_cpp


Important Disclaimers and Legal Information

Coding SamplesAny software coding and/or code lines / strings ("Code") included in this documentation are only examples and are not intended to be used in a productive system environment. The Code is only intended to better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, unless damages were caused by SAP intentionally or by SAP's gross negligence.

AccessibilityThe information contained in the SAP documentation represents SAP's current view of accessibility criteria as of the date of publication; it is in no way intended to be a binding guideline on how to ensure accessibility of software products. SAP in particular disclaims any liability in relation to this document. This disclaimer, however, does not apply in cases of willful misconduct or gross negligence of SAP. Furthermore, this document does not result in any direct or indirect contractual obligations of SAP.

Gender-Neutral LanguageAs far as possible, SAP documentation is gender neutral. Depending on the context, the reader is addressed directly with "you", or a gender-neutral noun (such as "sales person" or "working days") is used. If when referring to members of both sexes, however, the third-person singular cannot be avoided or a gender-neutral noun does not exist, SAP reserves the right to use the masculine form of the noun and pronoun. This is to ensure that the documentation remains comprehensible.

Internet HyperlinksThe SAP documentation may contain hyperlinks to the Internet. These hyperlinks are intended to serve as a hint about where to find related information. SAP does not warrant the availability and correctness of this related information or the ability of this information to serve a particular purpose. SAP shall not be liable for any damages caused by the use of related information unless damages have been caused by SAP's gross negligence or willful misconduct. All links are categorized for transparency (see: http://help.sap.com/disclaimer).


Important Disclaimers and Legal Information

http://help.sap.com/disclaimer/

Unicode CompatibilityImportant Disclaimers and Legal Information C U S T O M E R 17

go.sap.com/registration/contact.html

© 2017 SAP SE or an SAP affiliate company. All rights reserved.No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP SE or an SAP affiliate company. The information contained herein may be changed without prior notice.Some software products marketed by SAP SE and its distributors contain proprietary software components of other software vendors. National product specifications may vary.These materials are provided by SAP SE or an SAP affiliate company for informational purposes only, without representation or warranty of any kind, and SAP or its affiliated companies shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP or SAP affiliate company products and services are those that are set forth in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies.Please see http://www.sap.com/corporate-en/legal/copyright/index.epx for additional trademark information and notices.

https://go.sap.com/registration/contact.html

https://go.sap.com/registration/contact.html

http://www.sap.com/corporate-en/legal/copyright/index.epx

http://www.sap.com/corporate-en/legal/copyright/index.epx

automated analytics administration guide - help.sap.com

Documents