TOS / TIS Code Architecture

•Copyright © 2008 Talend. All rights reserved

Introduction Page 2

• Difference between TOS and TIS– Different builds include different plugins– Branding plugins.– Some plugins are used only for TIS. (DB

repository, SVN respository…)– Some plugins are even use only for different

builds of TIS. (AMC …)– EVERY plugin of TOS are used in all TOS or TIS

build.

TOS / TIS?

Introduction Page 3

SVN Organization

• Different SVN Repository– TOS (Open Source Code)

• Anyone can access to the code. But need a SVN Account from Talend to commit.

– TIS Shared (First TIS part Shared to China)• Code not Open Source. This SVN Repository was created to

allow people from China to commit some plugins for TIS.• This include Joblets, DataPreview, Documentation…• All SVN Accounts created in China got this access.

– TIS Private• This SVN was created first to have all TIS plugins. This

access is only for people in France (not other countries).• This include DB, SVN Repository / WebApp.• People from Talend Beijing should have an access later.

Introduction Page 4

What about the teams?

• Plugins for Component Team and UI Team– Components Team:

• org.talend.designer.components.localprovider• org.talend.designer.codegen• org.talend.core (folder about mapping)• org.talend.designer.components.tisprovider (TIS components)

– UI Team:• All others… and include also the DB Mapping also depends the

tasks.• So about 60 plugins maybe in TOS + TIS Shared, and 20-30 in TIS

Private.

This doesn’t mean one team can’t commit on the plugins of other team. Only show which plugin you will use most of the time.

Introduction Page 5

Plugins organization

• Different plugins to remember for– Code Generation (Components team)– Repository– Job Editor– Routines– Business Modeler– Tools (SQL Builder, Ecosystem…)– External Components (tRowGenerator, tMap…)– Documentation– (…)

Introduction Page 6

Build System

• Build Made with both TOS jobs and Ant.– Several types of build exists, these builds will simply list

the plugins needed (depends the build)– Build depends of features dependencies.

• Build Version of TOS or TIS:When Create a new plugin, don’t forget that this plugin should be included in a feature !If the new plugin is not included in a feature, maybe what you did will work in Eclipse.But this will never be included in the Build version !

When you add dependencies between plugins, never set the version of the plugin.Take care, by default with Eclipse 3.4 it will add the version…

Introduction Page 7

Main Classes

• Main classes are linked to:

– EMF Model

– Job Creation / Edition– Parameters– Code Generation– Link between Job Creation and Code

Generation.– Repository– Update Manager– Migration System

Introduction Page 8

TOS / TIS PLUGINS

Organization of the different plugins

Introduction Page 9

TOS/TIS Plugins

• Code Generation (1 TOS)– org.talend.designer.codegen (TOS)

• Repository (2 TOS, 3 TIS Private)– org.talend.repository (TOS)– org.talend.repository.localprovider (TOS)– org.talend.repository.remoteprovider (TIS Private)– org.talend.repository.databaseprovider (TIS Private)– org.talend.repository.svnprovider (TIS Private)

• Job Editor (1 TOS, 2 TIS Shared)– org.talend.designer.core (TOS)– org.talend.designer.joblet (TIS Shared)– org.talend.designer.jobtemplate(TIS Shared)

Introduction Page 10

TOS/TIS Plugins

• Routines (1 TOS, 1 TIS Shared)– org.talend.librairiesmanager (TOS)– org.talend.designer.routines.tisprovider (TIS Shared)

• Business Modeler (1 TOS)– org.talend.business.diagram (TOS)

• Documentation (1 TIS Shared)– org.talend.repository.documentation (TIS Shared)– Standard documentation done in org.talend.core.

• Tools: SQL Builder, EcoSystem, Scheduler... (4 TOS)– org.talend.sqlbuilder (TOS)– org.talend.designer.components.ecosystem (TOS)– org.talend.scheduler (TOS)– org.talend.expressionbuilder (TOS)

Introduction Page 11

TOS/TIS Plugins

• External Components (9 TOS)– Simple components:

• org.talend.designer.trowgenerator (TOS)• org.talend.designer.scd (TOS)• org.talend.designer.fileoutputxml (TOS)• org.talend.designer.alfrescooutput (TOS)

– tMap• org.talend.designer.mapper (TOS)• org.talend.designer.abstractmap (TOS)

– ELT Map• org.talend.designer.dbmap (TOS)• org.talend.designer.dbmap.mysql (TOS)• org.talend.designer.dbmap.oracle (TOS)

Introduction Page 12

TOS/TIS Plugins

• Metadata wizard plugins– org.talend.repository (TOS)

• This one contains all TOS wizards (DB, Delimited, Positional…)

– org.talend.repository.ebcdic– org.talend.repository.sapwizard– org.talend.wsdl.advancement

• Contains some other options to existing wizard in TOS.

Introduction Page 13

TOS/TIS Plugins

• Main TOS plugins used for development:– org.talend.commons– org.talend.commons.ui– org.talend.core– org.talend.designer.codegen– org.talend.designer.components.ecosystem– org.talend.designer.components.localprovider– org.talend.designer.core– org.talend.designer.rowgenerator– org.talend.designer.runprocess– org.talend.expressionbuilder– org.talend.librairiesmanager– org.talend.model– org.talend.repository– org.talend.repository.localprovider– org.talend.sqlbuilder

Introduction Page 14

TOS/TIS Plugins

• Main TIS Shared plugins used for development:– org.talend.designer.cdc– org.talend.designer.components.preview– org.talend.designer.joblet– org.talend.designer.jobtemplate– org.talend.designer.routines.tisprovider– org.talend.repository.documentation– org.talend.repository.ebcdic– org.talend.repository.importation– org.talend.repository.sapwizard

Introduction Page 15

MAIN PLUGINS / CLASSES

Details about the main plugins and classes in TOS / TIS

Introduction Page 16

EMF Model: org.talend.model

• Different Model Generated by EMF:– Business

• Used by the Business Modeler.

– Component• Used to load the components XML.• Modifications must be done first on XSD file, then XSD

transformed to EMF model.

– Expression• Used in the Expression Builder

– joblet• Used for the joblets, contains only the informations specific

to the joblets.

Introduction Page 17

EMF Model: org.talend.model

– metadata• Base for the Metadata wizards. Each informations from the

Wizard will be saved from this model.

– properties• Used by any object that can be saved in the repository. Job,

Db Connection… (Metadata Wizards model link to this one)

– talendfile• Job information are saved in this model. Same as the job

used in memory, most of the informations are saved in the paramters.

• Depends the modification of the model, it might need first a validation either from nrousseau or mhelleboid before commit (depends the modification)

Introduction Page 18

Job Creation: org.talend.designer.core

• Process (Job in fact!)– Each job is saved into EMF model, but is loaded and

modified during edition in memory in other classes– loadXml / SaveXml.

• Load or Save the complete Job from or to EMF.

• Node– Graphical object of the components

• Connection– Object that store informations about the links

All these classes extends the class Element.It’s in fact the GEF Model classes. GEF parts related are: ProcessPart,

NodePart, ConnectionPart.Some other classes are related to GEF also, for the labels, or some

icons , container displayed.

Introduction Page 19

Parameters (Components setup)

• ElementParameter (UI Design of parameters)– This class will store parameters of:

• Process (Job)• Node (Components)• Connection (links)

– These parameters will be used for:• Display of information in the UI• Code Generation (Mainly…)

• ElementParameterParser (Use of parameters in the code generation)– This class will be used to read easily the parameters from

any object contain parameter (Process, Node, Connection…)

Introduction Page 20

Global view of the Process

Process ElementParameter

Connection

Node ElementParameter

ElementParameter

1

11

n

1

1

n

n

n

n

• This is just a global graph to show that one process will contains several nodes, and several connections. And that each contains parameters. (Reality is a little more complex)

• You can consider that this model exists in:– EMF (Job information saved to repository)– Model part of GEF. So directly UI part.– Model used for code generation.

Introduction Page 21

Code Generation

• ProcessorUtilities– Allow to generate easily any code generation for one job.– Code generation MUST call this class, never use Processor

classes directly (unless specific use, like shadow process).• JavaProcessor / PerlProcessor

– These classes help for the code generation for the different languages. This will add all specific code for any language generated.

• CodeGeneratorEmittersPoolFactory (Comp. team)– Will generate code for each component. It will

generate .java files for each jet file.• CodeGeneratorService / CodeGenerator

– Will generate the code once the components are setup, from a DataProcess (Not from the UI Process)

Introduction Page 22

Link between Designer and CodeGeneration

• Take Care !– Graphical Job is different than the job we generate !– Generated process can be linked differently, and can

have additional components.• Difference between :

– Process.getGraphicalNodes• Get all “Graphical” nodes…. This means get all components

directly visible on the • This method return Node objects only.

– Process.getGeneratedNodes• Get all components really used for the code generation.• This method can return DataNode and ExternalNode

• DataProcess, DataNode, DataConnection– This will be a copy of the graphical process, but with the

modifications added to generate the process.

Introduction Page 23

Links between process and code generation

Process

EMFProcess

ProcessorUtilities

Can Use graphical Process Or EMF Process

Load

DataProcessContains

Processor

Contains

Use to generate the code

Introduction Page 24

Repository

• ProxyRepositoryFactory– Main class used to get any item from the repository. Any

other repository shouldn’t be called directly, unless specific use.

– To get this class from anywhere, for example:• CorePlugin.getRepositoryService().getProxyRepositoryFactory()

• LocalRepositoryFactory– This is one of the classes for repository implementation.

Other classes, no matter the repository type, got the same kind of implementation.

• RepositoryView– Implementation of the view for the repository

• RepositoryContentProvider– Model part of the repository view, this will get information

from the repository used.

Introduction Page 25

Classes linked to the language Java/Perl

• JavaProcessor / PerlProcessor– Code generation.

• Code viewer (tJava / tPerl):– TalendJavaSourceViewer– TalendPerlSourceViewer

• LanguageManager– Will get informations like current language used

• TypesManager, PerlTypesManager, JavaTypesManager– Functions related to schema types and types generated

in the code.

Introduction Page 26

Repository Wizards

• All TOS wizards are in the plugin org.talend.respository• org.talend.repository.ui.wizards.metadata.connection

• All wizards will store informations directly to the EMF Model.• Some wizards have links with a ShadowProcess, this kind of

process will create a job same as in the designer and will use the components directly to display datas in the table.

• To get informations from the wizard to any property displayed in the components:– Class: RepositoryToComponentProperty

• This will allow to get any information from the repository wizard (Emf Model), to use it directly in the properties.

Introduction Page 27

Component properties or Component Settings View

• ComponentSettingsView– This one can have different sections (called also category

or just “tab”)– Each section can contain different parameters– The section depends on the setup of the components XML

• Classes MultipleThreadDynamicComposite and DynamicComposite– Will check all the parameters from the current section,

and will display the related controls if needed.– Refresh all the composite if need to redraw only.– Refresh all the controls to refresh data if needed.

• Parameter UPDATE_COMPONENTS (need set to True to force refresh).

Introduction Page 28

Component properties or Component Settings View

• xxxController, xxxGenerator– Needed for each control of the properties.– For example there is controller for TEXT, CLOSED_LIST

(combo), SCHEMA_TYPE, …• Extension point: org.talend.designer.core.generators

– One extension point defined for each control.– Extension point will call the Generator class for the

control, this one will give the related Controller

• EParameterName– This enum will define the global parameters with the

display name for each. (specific components parameter will never be defined here)

Introduction Page 29

Migration System

• Plugin org.talend.migrationTool and org.talend.core contains basic classes for migrations.

• Plugin org.talend.repository contains most of the migrations needed for TOS.

• Extension point to use to create migration task:– org.talend.core.migrationTask– For any example, check any migration task in:

org.talend.repository.model.migration

Introduction Page 30

Migration System

• Take Care !

Migration task will set compatibility or not between each version. If one migration is created at one moment, this means that new job can’t be imported directly in a previous version of TOS.

Worse, if one migration is created in one version, then deleted later, a Job created in a previous version can’t be imported in a future version, this should never happen !

Rename a migration task id in the extension point is considered the same as delete a migration.

Introduction Page 31

Update Manager

• ProcessUpdateManager– Nothing related with the update from Eclipse.– This will help any job or joblet to be “up to date” with the

repository.Means that it will update any property field if anything has been modified in the repository.

Introduction Page 32

Bugs ?

• For some parts of the code, need to take care, because it’s quite easy to add a bug…

• When create/modify a metadata wizard:– Remember that this wizard is or will be linked to one

component. So if you do only the wizard and that’s all, it’s not really usefull for the user. The link to the component to set the properties must be added.

– Each wizard should also have some drag & drop.– Take care to the property type field, and the link with the

repository. Same with the UpdateManager.

• When modify the “Script”(code) Export:– Don’t forget that there is Java AND Perl mode to test.

Introduction Page 33

Bugs ?

• When modify the components:– There should be a migration task at least if:

• The component has been renamed.• One parameter has been renamed or have too much changes.• … any time you think the components from a previous version won’t

work in development version.

Don’t forget that migration system is here to make possible to use a job even from TOS 1.0 to TOS x.y.

• When modify the link system of the job design, there is some rules for each link… so even a small modification on this, will need maybe lots of tests.

• …