talend open studio advanced training material. p2 v080424 copyright © 2008 talend. all rights...

TALEND OPEN STUDIO

ADVANCED

Training Material

p2V080424 Copyright © 2008 Talend. All rights reserved

Copyright

Any total or partial reproduction without the consent of the author or beneficiary, devisee or legatee is not allowed (law of 11 March 1957, par. 1 of article 40). Representation or reproduction, by any means, would be considered an infringement of copyright under articles 425 et.seq. of the Penal Code. The law of 11 March 1957, par. 2 and 3 of article 41, allows the creation of copies and reproductions exclusively for the private use of the copier and not for collective use on the one hand while on the other it allows analysts to use short quotes for purposes of illustration.


Summary

Reminders and best practices Context management Mastering complex components Error handling Performance monitoring Monitoring and automating management of log files Deployment and scheduling Exploiting the power of programming languages Routines Debugging Creating components

tCiviliteGenerator (no properties view/retrieval of an exercise) tPrintln (properties view) tDecryptRow or tChangeDevise or tAddColumnTotal tPrintOutput to redirect to printer Virtual component (combining tFileInput and tSortRow)

p4

Let's carry on!

V080424 Copyright © 2008 Talend. All rights reserved

Reminders and best practices

Reminders Job designs Lookups

Best practices


Job Designer: reminders

A Job: components that are linked to one another

Sub-Job

Job


Job Designer: tMap and Lookup

Main and Lookup links set the order of processing

CUSTOMER WITH STATES


Job Designer: best practices


Job Designer: order of components

Order of execution of components

StartingStartingStartingEnd End End

StartingStarting

End End

p9

Let's carry on!


Management of Contexts

Reminders The Context node in the Repository

Dedicated components


A context contains several types (dev/prod) The prompt functionality refers to different types of

variables (pathDir, pathFile)

Context management


Selecting the execution context

When Talend Open Studio starts:

During deployment:


Advice

F5: Declare a variable

The component tRunJob


Practice area

Exercise 1 and 2

p14

Let's carry on!


Mastering complex components

Using regular expressions: tFileRegex tIntervalMatch / tDenormalize XML input, validation & XSL Using Webservices


Metadata file regexp

An example of reading an Apache log file


tFileInputRegexp

Define a regexp, example of reading from Apache log Aug 18 06:31:29 cplemon02.d075.cp logger: 66.102.9.104 - - [18/Aug/2006:07:49:27 +0200] "GET /server-status/ HTTP/1.1" 200 12891 "-" "CactiScript/1.0" "-" localhost Aug 18 06:31:30 cplemon02.d075.cp logger: 66.102.9.104 - - [18/Aug/2006:07:49:28 +0200] "GET /portail/accueil.pl HTTP/1.1" 200 2907 "-" "CactiScript/1.0" "-" localhost Aug 18 06:31:32 cplemon01.d075.cp logger: 66.102.9.104 - - [18/Aug/2006:08:06:48 +0200] "GET /server-status/ HTTP/1.1" 200 56765 "-" "CactiScript/1.0" "-" localhost


tIntervalMatch


tNormalize


tDenormalize

2

3


tXSDValidator


tXSLT: generate documents


tFileOutputXML: 1 special case


tAdvancedFileOutputXML


tWebServiceInput

p25

Let's carry on!


Error management

Trigger ifOk and ifError Components tLogCatcher / tWarn / tDie Create specific logs


Error handling

Each component has its own error handling routine (OnComponentError)


tLogCatcher schema

Default schema


Practice area

Exercise 3: Handling errors


Practice area

Exercise 4: Customizing error logs

p30

Let's carry on!


Performance monitoring

tStatCatcher tFlowMeter


tStatCatcher

Monitoring the performance of each component

p32

Let's carry on!


Monitoring and automation of log file management

Automating the management of logs Using log files


Log management

3 types of log

Manual log management and error management

(Java or Perl errors)

Management of component

execution start, end and duration

Management of metrics


Management of logs/preferences

In order to collate logs, configure preferences:


Management of logs/preferences

In the Properties and Job Designs view, preferences are entered automatically:

This is in Built-in mode, which is not great for maintaining preferences!

It is better to create metadata in the Repository and specify it in each job.


Practice area

Exercise 5: Creating the tables that are needed


Practice area

Exercise 6


Activity Monitoring Console /PE

Centralize the monitoring of Talend jobs Single user version Harvesting of all execution server reports

Functionalities Monitoring of:

Events triggered by the jobs Execution time Volumes of processed data

Harvest the local and remoteserver data simultaneously

User-definable interface Integrated in Talend Open Studio or independent

Benefits Single console for all integration jobs Custom indicators and thresholds Global view

p39

Let's carry on!


Exploiting the power of programming languages

tGroovy, tJava* & tPerl* Create specific classes/subs Use jars/ external Perl modules


GroovyFile component

GroovyFile: a simplified syntax

The script is simply entered in a text file

def name='World';

println "Hello $name";


Component execution in 3 stages

tPerlFlex/tJavaFlex component

Starting

loop

End

Start

Main

End

Component tPerlFlex/tJavaFlex: a component in 3 parts


Practice area

Exercise 7

Use tPerlFlex/tJavaFlex to generate the following flow:

key value

0 Miss

1 Mrs.

2 Mr.


Start

Main

End

tJava / tPerl = Start section of a tPerlFlex/tJavaFlex

tJava / tPerl components


Practice area

Exercise 8:

Create a table and use it in tMap


Start

Main

End

tJava / tPerl = Main section of a tJavaFlex / tPerlFlex

tJavaRow / tPerlRow components


Practice area

Exercise 9:

Use a tPerlRow/tJavaRow to modify a flow:

INPUT OUTPUT

0 Miss

1 Mrs.

2 Mr.


Practice area

Exercise 10: Use tPerlRow/tJavaRow to modify the flow:

INPUT OUTPUT

NSS month yeardepartment

2700392000000 03 1970 92

1760991000000 09 1976 91

p48

Let's carry on!



Master the complex components: tGroovy, tJava & tPerl Create specific classes/subs Use jars/ external Perl modules


Editor with embedded Java/Perl

Integrated Java / Perl editor Auto-complete Syntax-based colours Explanation of errors Javadoc / Perldoc

More information about the Java editor

http://jmdoudoux.developpez.com/java/eclipse/?page=Chap_006#L6.4


Managing shared code: routines

Structure your classes and subs Group together existing:

Business classes Connectors


Creating a class or a sub

Example: Java Class

The comments allow you to make the method available in

Expression Builder and tRowGenerator


Practice area

Exercise 11: Create that allows the following to be extracted from an NSS:

Month Year Department

INPUT OUTPUT

NSS month year dept

2700392000000 03 1970 92

1760991000000 09 1976 91

p53

Let's carry on!



Mastering complex components: tGroovy, tJava & tPerl Create specific classes/subs Use jars/ external Perl modules


Use a jar file or an external module

The external jar files are declared in a routine


Practice area

Exercise 12: Import the jakarta-oro-2.0.8.jar package Create a routine isValideEmail()

public static boolean isValideEmail(String email) { Perl5Matcher matcher = new Perl5Matcher(); Perl5Compiler compiler = new Perl5Compiler(); Pattern pattern; try { pattern = compiler.compile("^[\\w_.-]+@[\\w_.-]+\\.[\\w]+$"); if (!matcher.matches(email, pattern)) { return false; } } catch (MalformedPatternException e) { throw new RuntimeException(e); } return true; }


Practice area

Exercise 13: Create a job and name it UseJar Use the function IsValideEmail()

p57

Let's carry on!


Implementing jobs

Advanced debugging mode Define Breakpoints Learn step-by-step mode (step in, step into, step over) View variables in real time “Hot fix” variables in memory


Debug view

"Step-by-step" mode Hot fix variables

JasperETL

On-line resources

http://www.jmdoudoux.fr/java/dejae/chap008.htm


Practice area

Exercise 14: Using the debugger

p60

Let's carry on!


Implementing jobs

Deploy your jobs to a production system Export your jobs

Executable Web services

Launch your jobs from the command line Plan execution of your jobs via a scheduler


Deployment and optimisation

Deploy jobs Retrieve the generated code Conditions under which they function


Deploy a web service


Talend products and deployment

Talend Integration Suite

• Shares Repository• Job Conductor• Activity Monitoring Console / Dashboard• Distant Run• Grid Conductor• CPU Balancer

Talend on Demand

• Hosted Repository

Su

bs

criptio

n

GP

L

Op

en S

ou

rce

En

terprise

Ind

ividu

al

Activity Monitoring Console/Personal Edition

Talend Open Studio

• Business Modeler• Job Designer• Metadata Manager

Deploy your jobs remotelyDeploy your jobs remotely

p64

Let's carry on!


Implementing jobs

Create your own components: design and implementation Understand the concepts of code generation via a template Understand the 3 template files: start/main/end Understand the XML description of components Internationalize your components


Architecture of Talend Open Studio

Talend Open Studio

Business

Model

XML

Job

XML ...

Perl

Jet Template

Java

Jet Template

C

Jet Template

Perl Program

Java Program

C Program

Storage

CodeGenerator

GeneratedProgram

GUI

Pre

lim

inar

y sk

ills

: JA

VA

1.5

Pre

lim

inar

y sk

ills

: P

erl

5.8


Execution of components

Order in which components are executed

StartingStartingStartingEnd End End

StartingStarting

End End


Design a specific component

tJavaFlex / tPerlFlex : Start: action triggered by calling the component Main: action triggered with each line of data End: action triggered at the end of processing


Module containing components

The module org.talend.designer.components.localprovider

Do not modify this file by hand!


User components

Store your components in a specific file


Component’s Designer

Your components currently in development


Best practices

Differentiate between your development and production environments!

Install an instance of TOS to develop your components

Install an instance of TOS or of TIS Client to manage your Talend projects

Simply separating the storage folders for production and development components is not sufficient.


Component files

<component>_<language>.xml Description of a component

<component>_icon32.png The Palette icon

<component>_messages.properties The contents of the properties view

<component>_begin.<language>jet The Start part of the code

<component>_main.<language>jet The Main part of the code (the loop)

<component>_end.<language>jet The End part of the code


Working with files

The Component Designer view


<component>_<language>.xml

<component>_<language>.xml, description of component The Header tag presents the attributes

AUTHOR, VERSION, COMPATIBILITY STARTABLE SCHEMA_AUTO_PROPAGATE DATA_AUTO_PROPAGATE

The CONNECTORS tag contains: <CONNECTOR CTYPE="FLOW" MAX_INPUT="0"/> <CONNECTOR CTYPE="ITERATE"

MAX_OUTPUT="1" MAX_INPUT="1"/> <CONNECTOR CTYPE="THEN_RUN" MAX_INPUT="1"/> <CONNECTOR CTYPE="RUN_OK"/> <CONNECTOR CTYPE="RUN_ERROR"/> <CONNECTOR CTYPE="RUN_IF"/> <CONNECTOR NAME="UNIQUE" CTYPE="FLOW"

COLOR="086438" BASE_SCHEMA="FLOW" /> <CONNECTOR NAME="DUPLICATE" CTYPE="FLOW"

LINE_STYLE="2" COLOR="f36300" BASE_SCHEMA="FLOW" /> PARAMETERS...following page ;)

http://www.talendforge.org/wiki/doku.php?id=component_creation#xml_description


<component>_<language>.xml

<component>_<language>.xml, description of component: parameters

<PARAMETER NAME="FILENAME" FIELD="FILE" NUM_ROW="2" REQUIRED="true" >

<DEFAULT>'C:\talend_files\in.csv'</DEFAULT> </PARAMETER>

http://www.talendforge.org/wiki/doku.php?id=component_creation#xml_description

•CHECK•CLOSED_LIST•DIRECTORY•FILE•MEMO•MEMO_PERL

•MEMO_JAVA•MEMO_SQL•PROCESS_TYPE•PROPERTY_TYPE•SCHEMA_TYPE•TABLE•TEXT

The different forms of FIELD


Practice area

Exercise 15: First component! Objective is to understand the description xml file


Internationalize these components

CompoName_messages_zh.properties


JET Templates

Eclipse Modeling Framework (EMF) contains a very powerful tool for generating source code: JET (Java Emitter Templates)


JET Templates: general information

JET Template is made up of several parts: The code to be generated The functional code allowing you to determine the

parameters of the code to be generated

<%

/*Java code which will not be included in the generated code

otherwise called the functional code

*/

%>

/* Jave Code constituting the generated code*/


Practice area

Exercise 16: First JET Templates!


JET Templates / Start :: imports

JET Templates header

Then provides information according to the functionality of components

<%@ jet imports="

org.talend.core.model.process.INode org.talend.core.model.process.ElementParameterParser org.talend.core.model.metadata.IMetadataTable org.talend.core.model.metadata.IMetadataColumn org.talend.designer.codegen.config.CodeGeneratorArgument java.util.List

"

%>


JET Templates / tPrint

The tPrint component:

<%@ jet imports="

org.talend.core.model.process.INode org.talend.core.model.process.ElementParameterParserorg.talend.designer.codegen.config.CodeGeneratorArgument

"%><% CodeGeneratorArgument codeGenArgument = (CodeGeneratorArgument) argument; INode node = (INode)codeGenArgument.getArgument(); String msg = ElementParameterParser.getValue(node, "__COMMENTAIRE__");%> String msg = <%=msg %>; if(msg.equals("")) { System.out.println("Message is empty"); } else { System.out.println(msg); }


JET Templates/tPrint/generated code

Generated code

String msg = "nbrLignes: " + ((Integer)globalMap.get("tLogRow_1_NB_LINE")) ;

if(msg.equals("")) { System.out.println("Message is empty"); } else { System.out.println(msg); }


Practice area

Exercise 17: Create the component tPrint A single parameter: Comments

Use it in a job!


JET Templates/error management

Native handling: when a Java exception occurs The OnError link is triggered instead of the OnOk link The variable compoName_ERROR_MESSAGE is instantiated The error message is sent to tLogCatcher

Manual handling: compoName_ERROR_MESSAGE must be instantiated throw new Exception("myErrorMessage"); System.err.println("myErrorMessage");


JET Templates / tPrint

Handling an error in the tPrint component:

<%@ jet imports="

org.talend.core.model.process.INode org.talend.core.model.process.ElementParameterParserorg.talend.designer.codegen.config.CodeGeneratorArgument

"%><% CodeGeneratorArgument codeGenArgument = (CodeGeneratorArgument) argument; INode node = (INode)codeGenArgument.getArgument(); String msg = ElementParameterParser.getValue(node, "__MSG2PRINT__");

String cid = node.getUniqueName(); %>String msg = <%=msg %>; if(msg.equals("")) { String errMsg = "<%=cid %>: message is empty"; globalMap.put("<%=cid %>_ERROR_MESSAGE", errMsg); System.err.println(errMsg); throw new Exception(errMsg); } else { System.out.println(msg); }


Practice area

Exercise 18: Manage errors from within a component Create the component tPrint Display an alert in the case of an empty message Trigger the OnError link


JET Templates

Eclipse Modeling Framework (EMF) contains a very powerful tool for generating source code: JET (Java Emitter Templates)

In Talend, the JET Templates are split into 3 parts:

Start

Main

End


tInputCivilite


Define a schema by default

<PARAMETERS> <PARAMETER NAME="SCHEMA" FIELD="SCHEMA_TYPE" REQUIRED="true" NUM_ROW="1" READONLY="true"> <TABLE> <COLUMN NAME="id" TYPE="id_Integer" LENGTH="1"/> <COLUMN NAME="value" TYPE="id_String" LENGTH="30"/> </TABLE> </PARAMETER> </PARAMETERS>

<PARAMETERS> <PARAMETER NAME="SCHEMA" FIELD="SCHEMA_TYPE" REQUIRED="true" NUM_ROW="1" READONLY="true"> <TABLE> <COLUMN NAME="id" TYPE="id_Integer" LENGTH="1"/> <COLUMN NAME="value" TYPE="id_String" LENGTH="30"/> </TABLE> </PARAMETER> </PARAMETERS>


Practice area

Exercise 19: Create the tInputCivilite (or tInputtitle) component Configure the xml file and properties Create the 3 JET Templates


Practice area

Exercise 20: Manipulate multiple flows Modify the xml file in order to allow multiple outputs Modify the main Jet Template


Mode Row: Propagation of the schema

Let's pass to the Row components


Propagation of the schema/ the code


Practice area

Exercise 21: Create a tDemoRow component


tInt2StringCiviliteRow


Practice area

Exercise 22: Produce the tInt2StringCivilite component Begin: initialize the String [] valueArray table Main: loop on the columns and determine the content of the

column __CIVILITEOUT__


tReplaceCiviliteRow

<PARAMETERS> <PARAMETER NAME="CIVILITEOUT" FIELD="TEXT" REQUIRED="true" NUM_ROW="1" NB_LINES="1"> <DEFAULT>"newColumn"</DEFAULT> </PARAMETER> </PARAMETERS>

<PARAMETERS> <PARAMETER NAME="CIVILITEOUT" FIELD="TEXT" REQUIRED="true" NUM_ROW="1" NB_LINES="1"> <DEFAULT>"newColumn"</DEFAULT> </PARAMETER> </PARAMETERS>


Practice area

Exercise 23: Produce the tRowCivility component


Reject

Case where the key is superior to 2


Practice area

Exercise 24: Manage a reject flow


Load a jar into a component


Practice area

Exercise 25: Import a Jar into a component


Improve the created components

The Java editor allows you to identify unused imports


Virtual component

Certain components can use other components in a transparent manner

<CODEGENERATION> <TEMPLATES INPUT="AGGOUT" OUTPUT="AI"> <TEMPLATE NAME="AGGOUT" COMPONENT="tAggregateOut"> <LINK_TO NAME="AI" CTYPE="THEN_RUN"/> </TEMPLATE> <TEMPLATE NAME="AI" COMPONENT="tArrayIn"/> <TEMPLATE_PARAM SOURCE="self.OPERATIONS"

<TEMPLATE_PARAM SOURCE="self.OPERATIONS" <TEMPLATE_PARAM SOURCE="self.GROUPBYS"

TARGET="AGGOUT.GROUPBYS"/> <TEMPLATE_PARAM SOURCE="self.SCHEMA"

TARGET="AGGOUT.SCHEMA"/> <TEMPLATE_PARAM SOURCE="self.SCHEMA "

TARGET="AI.SCHEMA"/> <TEMPLATE_PARAM SOURCE="self.UNIQUE_NAME"

TARGET="AGGOUT.DESTINATION"/> <TEMPLATE_PARAM SOURCE="self.UNIQUE_NAME "

TARGET="AI.ORIGIN" /> </TEMPLATES> </CODEGENERATION>


Component and graphic interface

The use of the graphic user interface by a component requires the creation of a dedicated plugin


Rely on the Talend community

Your tools: Wiki BugTracker

Exchange with the community: Forum Ecosystem

p108

Congratulations!


Contact : [email protected]

Your turn to play!

talend open studio advanced training material. p2 v080424 copyright © 2008 talend. all rights...

Documents