data stage ppt materials
TRANSCRIPT
-
8/9/2019 Data Stage PPT Materials
1/216
Copyright Sennovate 2010. All rights
1-1Ver.1.0
IBM Information Server
InfoSphere Information Server provides a
single unified platform that enables companies
to understand, cleanse, transform, and delivertrustworthy and context-rich information.
-
8/9/2019 Data Stage PPT Materials
2/216
Copyright Sennovate 2010. All rights
1-2Ver.1.0
Products in IBM Inor!ation Server
IBM InfoSphere DataStage IBM InfoSphere QualityStage
IBM InfoSphere Information Services Director
IBM InfoSphere InformationAnalyzer
IBM Information Server ast!rac" IBM InfoSphere Business #lossary
$ther %ompanion &ro'ucts
IBM InfoSphere e'eration Server
(ational Data Architect InfoSphere (eplicationServer
)vent &u*lisher
-
8/9/2019 Data Stage PPT Materials
3/216
Copyright Sennovate 2010. All rights
1-"Ver.1.0
IBM Inor!ation Server architecture
IBM Information Server architecture is a
client+server architecture ma'e up of client+*ase''esign, a'ministration an' operation tools thataccess a set of server+*ase' 'ata integration
capa*ilities through a common services layer.
-
8/9/2019 Data Stage PPT Materials
4/216
Copyright Sennovate 2010. All rights
1-#Ver.1.0
IBM Inor!ation Server architecture
-
8/9/2019 Data Stage PPT Materials
5/216
Copyright Sennovate 2010. All rights
1-$Ver.1.0
Client tier
!he %lient tier *asically inclu'es the follo-ing
IBM InfoSphere DataStage an' QualityStage clients
A'ministrator
Director
Designer
-
8/9/2019 Data Stage PPT Materials
6/216
Copyright Sennovate 2010. All rights
1-%Ver.1.0
Server tier
!he Server tier inclu'es
Services
)ngine
(epository /or"ing area
/or"ing areas
Information Services Director resource provi'ers
-
8/9/2019 Data Stage PPT Materials
7/216 Copyright Sennovate 2010. All rights
1-&Ver.1.0
Services tier
!hree general categories of Services
Design
)ecution
Meta'ata
-
8/9/2019 Data Stage PPT Materials
8/216 Copyright Sennovate 2010. All rights
1-'Ver.1.0
(epository tier
!he Share' (epository is use' to share all the IBMInformation Server pro'uct mo'ule o*ects.
!he common repository contains the follo-ing types of
meta'ata that are re2uire' to support InfoSphereDataStage
Pro)ect !etadata
*perational !etadata
+esign !etadata
-
8/9/2019 Data Stage PPT Materials
9/216 Copyright Sennovate 2010. All rights
1-,Ver.1.0
ngine tier
!his is a parallel engine that eecutes IBM InformationServer tas"s.
-
8/9/2019 Data Stage PPT Materials
10/216 Copyright Sennovate 2010. All rights
1-10Ver.1.0
or/ing areas
!hese are the temporary storage areas use' *y thecomponents.
-
8/9/2019 Data Stage PPT Materials
11/216 Copyright Sennovate 2010. All rights
1-11Ver.1.0
Inor!ation Service providers
Information Service provi'ers are sources ofoperations for the services li"eDataStage,QualityStage.
-
8/9/2019 Data Stage PPT Materials
12/216 Copyright Sennovate 2010. All rights
1-12Ver.1.0
opologies
IBM InfoSphere Information Server multiple topologiesto support variety of 'ata integration , har'-are an'*usiness re2uirements.
%onsi'er the performance nee's to select the
topology !opologies supporte' are as follo-s
!-o+tier
!hree+tier
%luster#ri'
-
8/9/2019 Data Stage PPT Materials
13/216 Copyright Sennovate 2010. All rights
1-1"Ver.1.0
opologies
o-tier
!he engine, application server an' the meta'atarepository are all on the same computersystems -hile client are in 'i3erent machines.
hree-tier
!he engine is on one machine , the applicationserver an' meta'ata repository is co+locate' on
other machine. %lients are in the thir' machine.
-
8/9/2019 Data Stage PPT Materials
14/216 Copyright Sennovate 2010. All rights
1-1#Ver.1.0
opologies
Cluster!his is a slight variation of a three tier topology.
!he engine is 'uplicate' over multiple computers.
Ina cluster environment, a single parallel o* eecution
can span multiple computer each -ith its o-n engine.!he processing of a o* on multiple machines is 'riven
*y a con4guration 4le associate' -ith the o*.
-
8/9/2019 Data Stage PPT Materials
15/216
-
8/9/2019 Data Stage PPT Materials
16/216 Copyright Sennovate 2010. All rights
1-1%Ver.1.0
o-tier
-
8/9/2019 Data Stage PPT Materials
17/216 Copyright Sennovate 2010. All rights
1-1&Ver.1.0
hree tier
-
8/9/2019 Data Stage PPT Materials
18/216 Copyright Sennovate 2010. All rights
1-1'Ver.1.0
Clusters and rids
-
8/9/2019 Data Stage PPT Materials
19/216 Copyright Sennovate 2010. All rights
1-1,Ver.1.0
IBM Inosphere +ataStage
&opular )!5 tool
)na*les $rganization to 'esign 'ata 6o-s that etractinformation from multiple source systems, transform itto ma"e it more valua*le, an' then 'eliver it to one or
more target 'ata*ases. IBM InfoSphere DataStage is a part of IBM Information
Server architecture
It has
Share' components (untime Architecture
-
8/9/2019 Data Stage PPT Materials
20/216
Copyright Sennovate 2010. All rights1-20
Ver.1.0
+ataStage architecture
-
8/9/2019 Data Stage PPT Materials
21/216
Copyright Sennovate 2010. All rights1-21
Ver.1.0
Shared Co!ponents o +ataStage
Common User Interface
Designer
Director
dministrator
Common services
Common repository
Common parallel processing engine
Common connectors
-
8/9/2019 Data Stage PPT Materials
22/216
Copyright Sennovate 2010. All rights1-22
Ver.1.0
(unti!e architecture
*S3 Script
7sing the 'esigner, o*s are create'.
!he o*s are compile' into parallel o* 6o-s an'reusa*le components that eecute on the parallel
information server engine.Designer generates the $S89$rchestrate Shell
script:.
$S8 script
7ses the familiar script of 7ni shell.
-
8/9/2019 Data Stage PPT Materials
23/216
Copyright Sennovate 2010. All rights1-2"
Ver.1.0
4our core capa5ilities o +ataStage
%onnectivity to -i'e range of mainframe, legacy an'enterprise application 'ata*ases, 4le formats an'eternal information sources.
&re*uilt li*rary of more than ;00 functions inclu'ing
'ata vali'ation rules an' comple transformations. Maimum throughput using parallel, high+
performance processing architecture.
&rovi'es 'evelopment, 'eployment an' maintenancefeatures. It leverages meta'ata for analysis an'maintenance.
-
8/9/2019 Data Stage PPT Materials
24/216
Copyright Sennovate 2010. All rights1-2#
Ver.1.0
InoSphere +ataStage ele!ents
he Central +ataStage ele!ents are
&roects
%reate' -ith A'ministrator.
)ach proect contains other components such as o*s,stages, lin"s, containers an' ta*le 'e4nitions.
-
8/9/2019 Data Stage PPT Materials
25/216
Copyright Sennovate 2010. All rights1-2$
Ver.1.0
A 6o5
-
8/9/2019 Data Stage PPT Materials
26/216
Copyright Sennovate 2010. All rights1-2%
Ver.1.0
7a!ple o a )o5
-
8/9/2019 Data Stage PPT Materials
27/216
-
8/9/2019 Data Stage PPT Materials
28/216
Copyright Sennovate 2010. All rights1-2'
Ver.1.0
Parallel 6o5
)ecute' *y the DataStage parallel engine.
Built+in functionality for pipeline an' partitionparallelism .
%ompile' into $S8 9$rchestrate Scripting 5anguage:.
$S8 eecutes $perators )ecuta*le %== classinstances.
(untime monitoring in DataStage Director
-
8/9/2019 Data Stage PPT Materials
29/216
Copyright Sennovate 2010. All rights1-2,
Ver.1.0
Server 6o5s
)ecute' *y the DataStage server engine
%ompile' into Basic
(untime monitoring in DataStage Director
-
8/9/2019 Data Stage PPT Materials
30/216
Copyright Sennovate 2010. All rights1-"0
Ver.1.0
6o5 Se8uences
Master Server o*s that "ic"+o3 server or parallel o*san' other activities.
(untime monitoring in DataStage Director
)ecute' *y the Server engine
-
8/9/2019 Data Stage PPT Materials
31/216
Copyright Sennovate 2010. All rights1-"1
Ver.1.0
Stages
Active stage
Active stages mo'el the 6o- of 'ata an' provi'emechanisms for com*ining 'ata streams,aggregating 'ata, an' converting 'ata from one
'ata type to another Alters the num*er of ro-s from source to target.
&assive Stage
A passive stage han'les access to 'ata*ases for
the etraction or -riting of 'ata. Does not alter the num*er of ro-s from sourceto target.
-
8/9/2019 Data Stage PPT Materials
32/216
Copyright Sennovate 2010. All rights1-"2
Ver.1.0
Parallel processing
&arallel processing is the use of multiple processors toeecute the 'i3erent parts of the same programsimultaneously.
-
8/9/2019 Data Stage PPT Materials
33/216
Copyright Sennovate 2010. All rights1-""
Ver.1.0
(epresentation o )o5 ithoutparallelis!
-
8/9/2019 Data Stage PPT Materials
34/216
Copyright Sennovate 2010. All rights1-"#
Ver.1.0
o type o parallel processing
&ipeline
&artitioning
%om*ining an' &artitioning
-
8/9/2019 Data Stage PPT Materials
35/216
Copyright Sennovate 2010. All rights1-"$
Ver.1.0
Pipeline Parallelis!
!ransform, clean, loa' processes eecutesimultaneously
5i"e a conveyor *elt moving ro-s from process toprocess
Start 'o-nstream process -hile upstream processis running
A'vantages
(e'uces 'is" usage for staging areas
>eeps processors *usy
Still has limits on scala*ility
-
8/9/2019 Data Stage PPT Materials
36/216
Copyright Sennovate 2010. All rights1-"%
Ver.1.0
Pipeline Parallelis!
-
8/9/2019 Data Stage PPT Materials
37/216
Copyright Sennovate 2010. All rights1-"&
Ver.1.0
Partition Parallelis!
Divi'e the incoming stream of 'ata into su*sets to *eseparately processe' *y an operation .
Su*sets are calle' partitions 9no'es:
!his is "ey to Scala*ility
)ach partition of 'ata is processe' *y the sameoperation
).g., if operation is ilter, each partition -ill *e4ltere' in eactly the same -ay
acilitates near+linear scala*ility ? times faster on ? processors
@ times faster on @ processors
!his assumes the 'ata is evenly 'istri*ute'
-
8/9/2019 Data Stage PPT Materials
38/216
Copyright Sennovate 2010. All rights1-"'
Ver.1.0
Partitioned Parallelis!
-
8/9/2019 Data Stage PPT Materials
39/216
Copyright Sennovate 2010. All rights1-",
Ver.1.0
hree-9ode Partitioning
-
8/9/2019 Data Stage PPT Materials
40/216
Copyright Sennovate 2010. All rights1-#0
Ver.1.0
Parallel 6o5s Co!5ine Partitioningand Pipelining
-
8/9/2019 Data Stage PPT Materials
41/216
-
8/9/2019 Data Stage PPT Materials
42/216
-
8/9/2019 Data Stage PPT Materials
43/216
Copyright Sennovate 2010. All rights1-#"
Ver.1.0
Inor!ation in con:guration :le
9odes
It i'enti4es the num*er of no'es in the parallelprocessing.
(esource dis/
Data 4les are store' here (esource scratch dis/
8ere path is speci4e'. !he path is use' *yparallel o*s for *u3ering
-
8/9/2019 Data Stage PPT Materials
44/216
Copyright Sennovate 2010. All rights1-##
Ver.1.0
Sa!ple Con:guration :le
{node "dev1"
{
fast name "etltools-dev"
pool ""
resource disk "/data/etltools-tutorial/d1" { }
resource disk "/data/etltools-tutorial/d2" { }
resource scratchdisk "/data/etltools-tutorial/temp" { }
}
node "dev2"
{
fastname "etltools-dev"
pool ""resource disk "/data/etltools-tutorial/d1" { }
resource scratchdisk "/data/etltools-tutorial/temp" { }
}
}
-
8/9/2019 Data Stage PPT Materials
45/216
Copyright Sennovate 2010. All rights1-#$
Ver.1.0
Partitioning and Collecting
&artitioning *rea"s incoming ro-s into multiple streams ofro-s 9one for each no'e:
)ach partition of ro-s is processe' separately *y thestageoperator
%ollecting returns partitione' 'ata *ac" to a single stream
&artitioning %ollecting is speci4e' on stage input lin"s
-
8/9/2019 Data Stage PPT Materials
46/216
Copyright Sennovate 2010. All rights1-#%
Ver.1.0
Partitioning !ethods
(oun' (o*in &artitioner
(an'om &artitioner
Same partitioner
)ntire &artitioning
8ash partitioner
Mo'ulus partitioner
(ange partitioner
DB@ &artitioner
Auto &artitioner
-
8/9/2019 Data Stage PPT Materials
47/216
Copyright Sennovate 2010. All rights1-#&
Ver.1.0
(ound (o5in Partitioner
!he 4rst recor' goes to the 4rst partitioning no'e,secon' to the secon' an' so on.
/hen the 'ataStage reaches the last it starts overagain.
-
8/9/2019 Data Stage PPT Materials
48/216
Copyright Sennovate 2010. All rights1-#'
Ver.1.0
(ando! Partitioner
(ecor's are ran'omly 'istri*ute' over all partitioningno'es.
5i"e roun' ro*in, ran'om partitioning can re*alancethe partitions of an input 'ata set to guarantee that
each processing no'e receives an approimatelye2ual+size' partition.
!he ran'om partitioning has a slightly higheroverhea' than roun' ro*in *ecause of the etraprocessing re2uire' to calculate a ran'om value for
each recor'.
-
8/9/2019 Data Stage PPT Materials
49/216
Copyright Sennovate 2010. All rights1-#,
Ver.1.0
Sa!e partitioner
!he stage using the 'ata set as input performs norepartitioning an' ta"es as input the partitions output*y the prece'ing stage.
/ith this partitioning metho', recor's stay on the
same processing no'eE that is, they are notre'istri*ute'.
Same is the fastest partitioning metho'.
!his is normally the metho' DataStage uses -henpassing 'ata *et-een stages in your o*.
-
8/9/2019 Data Stage PPT Materials
50/216
Copyright Sennovate 2010. All rights1-$0
Ver.1.0
ntire Partitioning
)very instance of a stage on every processing no'ereceives the complete 'ata set as input.
It is useful -hen you -ant the *ene4ts of paralleleecution, *ut every instance of the operator nee's
access to the entire input 'ata set.
-
8/9/2019 Data Stage PPT Materials
51/216
Copyright Sennovate 2010. All rights1-$1
Ver.1.0
3ash partitioner
Set *ase' on a zip co'e 4el', -here a large percentage of yourrecor's &artitioning is *ase' on a function of one or more columns9the hash partitioning "eys: in each recor'. !he hash partitionereamines one or more 4el's of each input recor' 9the hash "ey4el's:.
(ecor's -ith the same values for all hash "ey 4el's are assigne' tothe same processing no'e.
!his metho' is useful for ensuring that relate' recor's are in thesame partition, -hich might *e a prere2uisite for a processingoperation.
8ash partitioning 'oes not necessarily result in an even 'istri*ution
of 'ata *et-een partitions. or eample, if you hash partition a 'ata are from one or t-o zip
co'es, you can en' up -ith a fe- partitions containing most of yourrecor's. !his *ehavior can lea' to *ottlenec"s *ecause some no'esare re2uire' to process more recor's than other no'es.
-
8/9/2019 Data Stage PPT Materials
52/216
Copyright Sennovate 2010. All rights1-$2
Ver.1.0
Modulus partitioner
&artitioning is *ase' on a "ey column mo'ulo thenum*er of partitions. !his metho' is similar to hash *y4el', *ut involves simpler computation.
-
8/9/2019 Data Stage PPT Materials
53/216
Copyright Sennovate 2010. All rights1-$"
Ver.1.0
(ange partitioner
Divi'es a 'ata set into approimately e2ual+size'partitions, each of -hich contains recor's -ith "eycolumns -ithin a speci4e' range. !his metho' is alsouseful for ensuring that relate' recor's are in the
same partition. A range partitioner 'ivi'es a 'ata set intoapproimately e2ual size partitions *ase' on one ormore partitioning "eys. (ange partitioning is often apreprocessing step to performing a total sort on a 'ata
set. In or'er to use a range partitioner, you have to ma"e a
range map. Fou can 'o this using the /rite (ange Mapstage.
-
8/9/2019 Data Stage PPT Materials
54/216
Copyright Sennovate 2010. All rights1-$#
Ver.1.0
+B2 Partitioner
&artitions an input 'ata set in the same -ay that DB@G-oul' partition it.
or eample, if you use this metho' to partition aninput 'ata set containing up'ate information for an
eisting DB@ ta*le, recor's are assigne' to theprocessing no'e containing the correspon'ing DB@recor'. !hen, 'uring the eecution of the paralleloperator, *oth the input recor' an' the DB@ ta*lerecor' are local to the processing no'e. Any rea's an'
-rites of the DB@ ta*le -oul' entail no net-or"activity.
-
8/9/2019 Data Stage PPT Materials
55/216
Copyright Sennovate 2010. All rights1-$$
Ver.1.0
Auto Partitioner
5eaving it to DataStage to 'etermine the *estpartitioning metho' to use 'epen'ing on the type ofstage, an' -hat the previous stage in the o* has'one.
!ypically DataStage -oul' use roun' ro*in -heninitially partitioning 'ata, an' same for theinterme'iate stages of a o*.
-
8/9/2019 Data Stage PPT Materials
56/216
Copyright Sennovate 2010. All rights1-$%
Ver.1.0
Collecting
%ollecting the process of oining the multiple partitionsin to single 'ataset.
%ollecting metho's
(oun' ro*in
$r'ere' collector Sort merge collector
Auto collector
-
8/9/2019 Data Stage PPT Materials
57/216
Copyright Sennovate 2010. All rights1-$&
Ver.1.0
(ound ro5in
(ea's a recor' from the 4rst input partition, then fromthe secon' partition, an' so on. After reaching the lastpartition, starts over.
After reaching the 4nal recor' in any partition, s"ips
that partition in the remaining roun's
-
8/9/2019 Data Stage PPT Materials
58/216
Copyright Sennovate 2010. All rights1-$'
Ver.1.0
*rdered collector
(ea's all recor's from the 4rst partition, then allrecor's from the secon' partition, an' so on.
!his collection metho' preserves the or'er of totallysorte' input 'ata sets. In a totally sorte' 'ata set,
*oth the recor's in each partition an' the partitionsthemselves are or'ere'.
!his might *e useful as a preprocessing action *eforeeporting a sorte' 'ata set to a single 'ata 4le.
-
8/9/2019 Data Stage PPT Materials
59/216
Copyright Sennovate 2010. All rights1-$,
Ver.1.0
Sort !erge collector
pro'uces a glo*ally sorte' se2uential stream from-ithin partition sorte' ro-s.
Sort Merge pro'uces a non+'eterministic on un+"eye'columns sorte' se2uential stream using the follo-ing
algorithmal-ays pic" the partition that pro'uces the ro- -ith
the smallest "ey value.
-
8/9/2019 Data Stage PPT Materials
60/216
Copyright Sennovate 2010. All rights1-%0
Ver.1.0
Auto collector
!he 'efault algorithm rea's ro-s from a partition assoon as they are rea'y.
!his may lea' to pro'ucing 'i3erent ro- or'ers in'i3erent runs -ith i'entical 'ata. !he eecution is
non+'eterministic.
-
8/9/2019 Data Stage PPT Materials
61/216
Copyright Sennovate 2010. All rights1-%1
Ver.1.0
Ad!inistrator
A'ministrator is a client program use' to carry outcon4guration tas"s in DataStage.
It has ; pages
#eneral
!he general page is use' to set server+-i'eproperties.
&roect
!his lists the proects availa*le an' options toa'', e'it an' 'elete proects.
C5S
Cational 5anguage support features.
-
8/9/2019 Data Stage PPT Materials
62/216
Copyright Sennovate 2010. All rights1-%2
Ver.1.0
Attaching to +ataStage
-
8/9/2019 Data Stage PPT Materials
63/216
Copyright Sennovate 2010. All rights1-%"
Ver.1.0
Ad!inistrator
-
8/9/2019 Data Stage PPT Materials
64/216
Copyright Sennovate 2010. All rights1-%#
Ver.1.0
Pro)ect Page
A'' !o a'' Ce- DataStage &roect
Delete !o 'elete a proect. !his *utton is ena*le onlyif you have a'ministrator status.
&roperties !o set the properties of the selecte'
proect. %leanup %leans up 4les in selecte' proect
C5S !o change proect map an' locales.
%omman' !o eecute DataStage )ngine comman's
'irectly from the selecte' proect
-
8/9/2019 Data Stage PPT Materials
65/216
Copyright Sennovate 2010. All rights1-%$
Ver.1.0
Pro)ect page
-
8/9/2019 Data Stage PPT Materials
66/216
Copyright Sennovate 2010. All rights1-%%Ver.1.0
Add Pro)ect
-
8/9/2019 Data Stage PPT Materials
67/216
Copyright Sennovate 2010. All rights1-%&Ver.1.0
Creating a pro)ect
-
8/9/2019 Data Stage PPT Materials
68/216
-
8/9/2019 Data Stage PPT Materials
69/216
Copyright Sennovate 2010. All rights1-%,Ver.1.0
Pro)ect Properties ; eneral ta5
-
8/9/2019 Data Stage PPT Materials
70/216
Copyright Sennovate 2010. All rights1-&0Ver.1.0
eneral ta5 in Pro)ect properties
)na*le o* a'ministration from the DataStage Director 5ets DataStage operators release the resources of a o* that
has a*orte' or hung, an' so return the o* to a state in -hichit can *e rerun -hen the cause of the pro*lem has *een 4e'.
)na*le runtime column propagation for parallel o*s.
Stages in parallel o* can han'le un'e4ne' columns.
De4ne a proect+-i'e setting for auto+purge of the o* log.!o prevent o* log 4le to *ecome too large this feature is
use'.
Set up environment varia*les.%an set values to )nvironment varia*les
%reate ne- )nvironment Varia*les -hich can *e use' li"eparameters.
-
8/9/2019 Data Stage PPT Materials
71/216
Copyright Sennovate 2010. All rights 1-&1Ver.1.0
Per!issions ta5
-
8/9/2019 Data Stage PPT Materials
72/216
Copyright Sennovate 2010. All rights 1-&2Ver.1.0
Per!issions ta5
ssign user categories to operating system user groups, or enableoperators to view all the details of an event in a !ob log file.
"he Permissions tab is enabled only if you have logged on to
DataStage using a name that gives you administrator status.
-
8/9/2019 Data Stage PPT Materials
73/216
Copyright Sennovate 2010. All rights 1-&"Ver.1.0
racing ta5
-
8/9/2019 Data Stage PPT Materials
74/216
Copyright Sennovate 2010. All rights 1-Ver.1.0
racing ta5
!his is to ena*le or 'isa*le tracing on the server.
-
8/9/2019 Data Stage PPT Materials
75/216
Copyright Sennovate 2010. All rights 1-&$Ver.1.0
Schedule
-
8/9/2019 Data Stage PPT Materials
76/216
Copyright Sennovate 2010. All rights 1-&%Ver.1.0
Schedule ta5
Set up a user name an' pass-or' to use for runningsche'ule' DataStage o*s.
!he Sche'ule ta* is ena*le' only if you have logge'on to a /in'o-s C! server.
-
8/9/2019 Data Stage PPT Materials
77/216
Copyright Sennovate 2010. All rights 1-&&Ver.1.0
Mainra!e )o5 properties
-
8/9/2019 Data Stage PPT Materials
78/216
Copyright Sennovate 2010. All rights 1-&'Ver.1.0
una5le ; Con:gure cache settings
-
8/9/2019 Data Stage PPT Materials
79/216
Copyright Sennovate 2010. All rights 1-&,Ver.1.0
Parallel ta5
-
8/9/2019 Data Stage PPT Materials
80/216
Copyright Sennovate 2010. All rights 1-'0Ver.1.0
Se8uence ta5
-
8/9/2019 Data Stage PPT Materials
81/216
Copyright Sennovate 2010. All rights 1-'1Ver.1.0
I!porting and 7porting *5)ects
rom Director choose !oolsJJ(un Manger %hoose )port JJ%omponents.
-
8/9/2019 Data Stage PPT Materials
82/216
Copyright Sennovate 2010. All rights
1-'2Ver.1.0
+esigner
A graphical user interface for creating DataStageapplications "no-n as
-
8/9/2019 Data Stage PPT Materials
83/216
Copyright Sennovate 2010. All rights
1-'"Ver.1.0
Stages
!he in'ivi'ual steps that ma"e up the o* are calle'stages.
Some of the DataStage &re*uilt stages are sort,merge, oin, 4lter, transform, loo"up an' aggregate.
Stages provi'e the ?0 to K0 percent of the applicationlogic re2uire' for enterprise 'ata integrationapplications.
)ach stage has properties that tell ho- to perform orprocess 'ata.
-
8/9/2019 Data Stage PPT Materials
84/216
Copyright Sennovate 2010. All rights
1-'#Ver.1.0
So!e o the Stages in +ataStage
i i
-
8/9/2019 Data Stage PPT Materials
85/216
Copyright Sennovate 2010. All rights
1-'$Ver.1.0
Co!!on
-
8/9/2019 Data Stage PPT Materials
86/216
Copyright Sennovate 2010. All rights
1-'%Ver.1.0
Stages list contd=
Sort Stage Surrogate >ey generator
!ransformer
(emove Duplicate stage
St i ti ) 5
-
8/9/2019 Data Stage PPT Materials
87/216
Copyright Sennovate 2010. All rights
1-'&Ver.1.0
Steps in creating a )o5
$pen 'esigner an' connect to the proect %hoose the type of o* to *e create'.
Import ta*le 'e4nition
Drag an' 'rop the stages
5in" the stages Set the properties of the stage
Save an' compile the o*
)ecute the o* *y choosing !oolsJJ(un Director
7a!pleSimple o* to group 'epartment+-ise an' sum
salary from 6at 4le
C t t th P ) t
-
8/9/2019 Data Stage PPT Materials
88/216
Copyright Sennovate 2010. All rights
1-''Ver.1.0
Connect to the Pro)ect
Ch th t 6 5
-
8/9/2019 Data Stage PPT Materials
89/216
Copyright Sennovate 2010. All rights
1-',Ver.1.0
Choose the type o 6o5
P ll l 6 5 Citory
-
8/9/2019 Data Stage PPT Materials
90/216
Copyright Sennovate 2010. All rights
1-,0Ver.1.0
Parallel 6o5 Canvas
Stage
s
&alat
te
(epo
sito
o*1ec
ts
%ACVAS
I t S ti l :l d : iti
-
8/9/2019 Data Stage PPT Materials
91/216
Copyright Sennovate 2010. All rights
1-,1Ver.1.0
I!port Se8uential :le de:nition
Se2uential 4le'e4nition
Ch +i t d :l t i t
-
8/9/2019 Data Stage PPT Materials
92/216
Copyright Sennovate 2010. All rights
1-,2Ver.1.0
Choose +irectory and :le to i!portImportoption
+ : C l d t
-
8/9/2019 Data Stage PPT Materials
93/216
Copyright Sennovate 2010. All rights
1-,"Ver.1.0
+e:ne Colu!ns and or!at
Stages and
-
8/9/2019 Data Stage PPT Materials
94/216
Copyright Sennovate 2010. All rights
1-,#Ver.1.0
Stages and
-
8/9/2019 Data Stage PPT Materials
95/216
Copyright Sennovate 2010. All rights
1-,$Ver.1.0
Se8uential :le - Source
4or!at or Se8uential :le
-
8/9/2019 Data Stage PPT Materials
96/216
Copyright Sennovate 2010. All rights
1-,%Ver.1.0
4or!at or Se8uential :le
$ther&roperti
es
Colu!ns ta5
-
8/9/2019 Data Stage PPT Materials
97/216
Copyright Sennovate 2010. All rights
1-,&Ver.1.0
Colu!ns ta5 -
-
8/9/2019 Data Stage PPT Materials
98/216
Copyright Sennovate 2010. All rights
1-,'Ver.1.0
Select the colu!ns needed
Colu!ns loaded
-
8/9/2019 Data Stage PPT Materials
99/216
Copyright Sennovate 2010. All rights
1-,,Ver.1.0
Colu!ns loaded
Se8uential :le arget Properties
-
8/9/2019 Data Stage PPT Materials
100/216
Copyright Sennovate 2010. All rights
1-100Ver.1.0
Se8uential :le ; arget Properties
Aggregate Stage Properties
-
8/9/2019 Data Stage PPT Materials
101/216
Copyright Sennovate 2010. All rights
1-
101Ver.1.0
Aggregate Stage Properties
Select#roup *ycolumn
5ist ofcolumn
s
Choose *utput colu!ns
-
8/9/2019 Data Stage PPT Materials
102/216
Copyright Sennovate 2010. All rights
1-
102Ver.1.0
Choose *utput colu!ns
Input page Aggregator Stage
-
8/9/2019 Data Stage PPT Materials
103/216
Copyright Sennovate 2010. All rights
1-
10"Ver.1.0
Input page ; Aggregator Stage
*utput page
-
8/9/2019 Data Stage PPT Materials
104/216
Copyright Sennovate 2010. All rights
1-
10#Ver.1.0
*utput page
6o5
-
8/9/2019 Data Stage PPT Materials
105/216
Copyright Sennovate 2010. All rights
1-
10$Ver.1.0
6o5
Save )o5
-
8/9/2019 Data Stage PPT Materials
106/216
Copyright Sennovate 2010. All rights
1-
10%Ver.1.0
Save )o5
-
8/9/2019 Data Stage PPT Materials
107/216
(un +irector
-
8/9/2019 Data Stage PPT Materials
108/216
Copyright Sennovate 2010. All rights
1-
10'Ver.1.0
(un +irector
Status >ie
-
8/9/2019 Data Stage PPT Materials
109/216
Copyright Sennovate 2010. All rights
1-
10,Ver.1.0
Status >ie
(un the )o5
-
8/9/2019 Data Stage PPT Materials
110/216
Copyright Sennovate 2010. All rights
1-
110Ver.1.0
(un the )o5
Annotation Stage
-
8/9/2019 Data Stage PPT Materials
111/216
Copyright Sennovate 2010. All rights
1-
111Ver.1.0
Annotation Stage
!his stage is use' to insert notes to the 'iagram-in'o-.
!-o types of Annotation
Annotation
Description Annotation
-
8/9/2019 Data Stage PPT Materials
112/216
Copyright Sennovate 2010. All rights
1-
112Ver.1.0
Stages in co!5ining data
Co!5ining +ata 5ased on ?eyl
-
8/9/2019 Data Stage PPT Materials
113/216
Copyright Sennovate 2010. All rights
1-
11"Ver.1.0
colu!n
5oo"up Stage Merge Stage
-
8/9/2019 Data Stage PPT Materials
114/216
Copyright Sennovate 2010. All rights
1-
11#Ver.1.0
-
8/9/2019 Data Stage PPT Materials
115/216
Copyright Sennovate 2010. All rights
1-
11$Ver.1.0
-
8/9/2019 Data Stage PPT Materials
116/216
Copyright Sennovate 2010. All rights
1-
11%Ver.1.0
-
8/9/2019 Data Stage PPT Materials
117/216
Copyright Sennovate 2010. All rights
1-
11&Ver.1.0
-
8/9/2019 Data Stage PPT Materials
118/216
Copyright Sennovate 2010. All rights
1-
11'Ver.1.0
oo up S age
6oin Stage
-
8/9/2019 Data Stage PPT Materials
119/216
Copyright Sennovate 2010. All rights
1-
11,Ver.1.0
6 g
I t performs oin operation on t-o or more inputs tothe stage
!his is similar to s2l oin.
It provi'es
Inner
ull $uter
5eft $uter
(igh $uter
6oin Stage
-
8/9/2019 Data Stage PPT Materials
120/216
Copyright Sennovate 2010. All rights
1-
120Ver.1.0
6 g
6oin Stage%hoosethe "eyfor oin
-
8/9/2019 Data Stage PPT Materials
121/216
Copyright Sennovate 2010. All rights
1-
121Ver.1.0
6 gfor oin
6oin Stage%hoosethe oin
type
-
8/9/2019 Data Stage PPT Materials
122/216
Copyright Sennovate 2010. All rights
1-
122Ver.1.0
6 gtype
6oin Stage
-
8/9/2019 Data Stage PPT Materials
123/216
Copyright Sennovate 2010. All rights
1-
12"Ver.1.0
6 g
Merge Stage
-
8/9/2019 Data Stage PPT Materials
124/216
Copyright Sennovate 2010. All rights
1-
12#Ver.1.0
g g
Merge stage is processing stage It can have
More than one input lin"
Single $utput lin"
Same num*er of reect lin" as that of up'ate lin"s.
Merge Stage
-
8/9/2019 Data Stage PPT Materials
125/216
Copyright Sennovate 2010. All rights
1-
12$Ver.1.0
g g
Merge Stage
%hoosethe
merge
-
8/9/2019 Data Stage PPT Materials
126/216
Copyright Sennovate 2010. All rights
1-
12%Ver.1.0
g g"ey
Merge
>eep or'rop
-
8/9/2019 Data Stage PPT Materials
127/216
Copyright Sennovate 2010. All rights
1-
12&Ver.1.0
Merge
Merge Stage
-
8/9/2019 Data Stage PPT Materials
128/216
Copyright Sennovate 2010. All rights
1-
12'Ver.1.0
Co!parisonMerge Join Lookup
-
8/9/2019 Data Stage PPT Materials
129/216
Copyright Sennovate 2010. All rights
1-
12,Ver.1.01
Merge Join Lookup
Stream Input 2 to N 2 To N 1
Reference Input NA NA 1-N
Output
Merged data
Master Update Type SQL-type o!ned data
If no dup"!cates !n t#e "oo$up data e%pected
t#en one for e&ery !nput stream record
'"se If one reference stream pro&!des
"eg!t!mate dup"!cates( t#en mu"t!p"e ro)s for
t#ose records
Sort!ng re*u!rements A"" !nput A"" Input Stream Input On"y
+up"!cates
Not a""o)ed e%cept !n
"ast update "!n$ A""o)ed
A""o)ed !n Stream Input
Upto 1 reference "!n$ can #and"edup"!cates, In ot#ers( s!ng"ef!rst. &a"ue
returned,
/art!t!on Merge 0ey o!n 0ey Usua""y set 'nt!re3 for "oo$up data
Unmatc#ed Ro)s
Master - drop4$eep(
)arn!ng4no )arn!ng
Update - drop4reect
+epends on o!n type
NULL &a"ues on outer
o!n Unmatc#ed stream - reect4$eep
Memory
5ery fe) ro)s !n memory
as data !s sorted 6 no
dup"!cates are e%pected
7e) ro)s as data !s
sorted,8!g#er se*uent!a"(
opt!m!9ed. I4O for #!g#-
speed sort on !nput 6
reference data sets
Loo$up data !n memory- may page for
"arge &o"umes, Not su!ta:"e for "argereference data,
;#en "oo$!ng up aga!nst a data:ase( t#e
+< stage can :e set to pro&!de sparse "oo$-
up support
Use ;#en Larger sorted data Large data Sma"" reference data "oo$-up,
4unnel Stage
-
8/9/2019 Data Stage PPT Materials
130/216
Copyright Sennovate 2010. All rights
1-
1"0Ver.1.0
It com*ines multiple input to single output !he stage can have any num*er of input lin"s *ut a
single $utput 5in".
!he meta'ata of all the inputs has to *e i'entical
unnel Stage $perates in ; mo'es%ontinuous funnel
Sort funnel
Se2uence funnel
4unnel Stage
-
8/9/2019 Data Stage PPT Materials
131/216
Copyright Sennovate 2010. All rights
1-
1"1Ver.1.0
4unnel Stage
-
8/9/2019 Data Stage PPT Materials
132/216
Copyright Sennovate 2010. All rights
1-
1"2Ver.1.0
Choose thefunnel type
4unnel stage
-
8/9/2019 Data Stage PPT Materials
133/216
Copyright Sennovate 2010. All rights
1-
1""Ver.1.0
ypes o unnel
-
8/9/2019 Data Stage PPT Materials
134/216
Copyright Sennovate 2010. All rights
1-
1"#Ver.1.0
%ontinuous unnel
%ontinuous funnel com*ines recor's of the input 'ata in nogurantee' or'er
It ta"es one recor' from each input lin" in turn.
If 'ata is not availa*le on an input lin", the stage s"ips tothe net lin" rather than -aiting.
Sort unnelSort unnel com*ines the input recor's in the or'er 'e4ne'
*y the value9s: of one or more "ey columns, an' the or'erof the output recor's is 'etermine' *y these sorting "eys.
Se2uence unnel
Se2uence copies all recor's from the 4rst input 'ata set tothe output 'ata set, then all the recor's from the secon'input 'ata set, an' so on.
+evelop!ent and +e5ug Stages
-
8/9/2019 Data Stage PPT Materials
135/216
Copyright Sennovate 2010. All rights
1-
1"$Ver.1.0
#ead Stage "ail Stage
$ee% Stage
Column &enerator Stage
'ow &enerator Stage (rite 'ange )ap Stage
3ead Stage
-
8/9/2019 Data Stage PPT Materials
136/216
Copyright Sennovate 2010. All rights
1-
1"%Ver.1.0
It can have a single input lin" an' a single output lin"
Select 4rst C ro-s from each partition of an input 'ataset an' copies selecte' ro-s to output 'ata set.
!his is use' to De*ug large Data Sets
&roperty settings inclu'es the follo-ing
Cum*er of recor's to copy
&artition from -hich recor's are copie'
5ocation
Stage Page ; 3ead Stage
-
8/9/2019 Data Stage PPT Materials
137/216
Copyright Sennovate 2010. All rights
1-
1"&Ver.1.0
#eneral #eneral &roperties can *e provi'e' here
&roperties
&roperties -hich inclu'es num*er of ro-s per
partition, all ro-s or s"ip ro-s A'vance'
Di3erent eecution mo'es an' com*ina*ilitymo'e
Input Page ; 3ead Stage
-
8/9/2019 Data Stage PPT Materials
138/216
Copyright Sennovate 2010. All rights
1-
1"'Ver.1.0
#eneral &artitioning
%olumn
A'vance'
*utput Page ; 3ead Stage
-
8/9/2019 Data Stage PPT Materials
139/216
Copyright Sennovate 2010. All rights
1-
1",Ver.1.0
#eneral Mapping
%olumn
A'vance'
ail Stage
-
8/9/2019 Data Stage PPT Materials
140/216
Copyright Sennovate 2010. All rights
1-
1#0Ver.1.0
It can have single input lin" an' single output lin" It selects last C recor's from each partition an' copies
it to output 'ata set
Pee/ Stage
-
8/9/2019 Data Stage PPT Materials
141/216
Copyright Sennovate 2010. All rights
1-
1#1Ver.1.0
It has single input lin" an' any num*er of output lin"s
It let to print the recor' column values either in the)o5 log or separate output lin" as it copies recor'sfrom input to output.
It is helpful in monitoring the progress of theapplication or 'iagnose the *ug in the application.
Sa!ple Stage
-
8/9/2019 Data Stage PPT Materials
142/216
Copyright Sennovate 2010. All rights
1-
1#2Ver.1.0
It has single input lin" an' any num*er of output lin"s
Samples an input 'ataset
&ercent mo'e+It etracts ro-s *y selecting them *ymeans of a ran'om num*er generator an' -rites apercentage to output 'ata set.
Colu!n enerator Stage
-
8/9/2019 Data Stage PPT Materials
143/216
Copyright Sennovate 2010. All rights
1-
1#"Ver.1.0
It can have a single input lin" an' a single output lin".
%olumn #enerator a''s columns to the incoming Dataan' generates moc" 'ata for these columns for eachro- processe'.
!he ne- 'ata set is the output.
(o enerator Stage
-
8/9/2019 Data Stage PPT Materials
144/216
Copyright Sennovate 2010. All rights
1-
1##Ver.1.0
(o- generator has no input lin" an' a single outputlin"
(o- generator pro'uces moc" 'ata 4tting the givenmeta'ata.
It is use' to test -hen there is no 'ata availa*le
It has Stage page an' $utput &age
rite (ange Map Stage
-
8/9/2019 Data Stage PPT Materials
145/216
Copyright Sennovate 2010. All rights
1-
1#$Ver.1.0
!he /rite (ange Map stage ta"es an input 'ata setpro'uce' *y sampling an' sorting a 'ata set an'-rites it to a 4le in a form usa*le *y the rangepartitioning metho'.
A typical use for the /rite (ange Map stage -oul' *ein a o* -hich use' the Sample stage to sample a 'ataset, the Sort stage to sort it an' the /rite (ange Mapstage to -rite the resulting 'ata set to a 4le.
*+BC Stages
-
8/9/2019 Data Stage PPT Materials
146/216
Copyright Sennovate 2010. All rights
1-
1#%Ver.1.0
$DB% stage is use' to etract, -rite or aggregate'ata.
)ach $DB% stage can have any num*er of input lin"sor output lin"s.
Specify the input lin" using the follo-ing metho's
An SQ5 statement
A user 'e4ne' SQ5 2uery
A store' proce'ure
I!port ; *+BC ta5le de:nition
-
8/9/2019 Data Stage PPT Materials
147/216
Copyright Sennovate 2010. All rights
1-
1#&Ver.1.0
*+BC stage
-
8/9/2019 Data Stage PPT Materials
148/216
Copyright Sennovate 2010. All rights
1-
1#'Ver.1.0
Choose the ta5les to i!port
-
8/9/2019 Data Stage PPT Materials
149/216
Copyright Sennovate 2010. All rights
1-
1#,Ver.1.0
*+BC SA
-
8/9/2019 Data Stage PPT Materials
150/216
Copyright Sennovate 2010. All rights
1-
1$0Ver.1.0
*+BC stage
-
8/9/2019 Data Stage PPT Materials
151/216
Copyright Sennovate 2010. All rights
1-
1$1Ver.1.0
*utput Mapping
-
8/9/2019 Data Stage PPT Materials
152/216
Copyright Sennovate 2010. All rights
1-
1$2Ver.1.0
*uput !apping
-
8/9/2019 Data Stage PPT Materials
153/216
Copyright Sennovate 2010. All rights
1-
1$"Ver.1.0
*CI stage ; I!port Plugin !etadade:nition
-
8/9/2019 Data Stage PPT Materials
154/216
Copyright Sennovate 2010. All rights
1-
1$#Ver.1.0
*CI Stage
-
8/9/2019 Data Stage PPT Materials
155/216
Copyright Sennovate 2010. All rights
1-
1$$Ver.1.0
+ata source na!e and user detailsData*as
-
8/9/2019 Data Stage PPT Materials
156/216
Copyright Sennovate 2010. All rights
1-
1$%Ver.1.0
Data*ase name
-
8/9/2019 Data Stage PPT Materials
157/216
Choose the ta5les or i!porting
-
8/9/2019 Data Stage PPT Materials
158/216
Copyright Sennovate 2010. All rights
1-
1$'Ver.1.0
*CI Stage and transor!er stage
-
8/9/2019 Data Stage PPT Materials
159/216
Copyright Sennovate 2010. All rights
1-
1$,Ver.1.0
-
8/9/2019 Data Stage PPT Materials
160/216
Choose the ta5le
-
8/9/2019 Data Stage PPT Materials
161/216
Copyright Sennovate 2010. All rights
1-
1%1Ver.1.0
Choose the colu!ns
-
8/9/2019 Data Stage PPT Materials
162/216
Copyright Sennovate 2010. All rights
1-
1%2Ver.1.0
Surrogate ?ey generator stage
-
8/9/2019 Data Stage PPT Materials
163/216
Copyright Sennovate 2010. All rights
1-
1%"Ver.1.0
Surrogate ?ey enerator
-
8/9/2019 Data Stage PPT Materials
164/216
Copyright Sennovate 2010. All rights
1-
1%#Ver.1.0
Sort and 4ilter
-
8/9/2019 Data Stage PPT Materials
165/216
Copyright Sennovate 2010. All rights
1-
1%$Ver.1.0
Sort stage Properties%hoose
-
8/9/2019 Data Stage PPT Materials
166/216
Copyright Sennovate 2010. All rights
1-
1%%Ver.1.0
the "ey
*utput Mapping
-
8/9/2019 Data Stage PPT Materials
167/216
Copyright Sennovate 2010. All rights
1-
1%&Ver.1.0
4ilter Stageilter
con'ition
-
8/9/2019 Data Stage PPT Materials
168/216
Copyright Sennovate 2010. All rights
1-
1%'Ver.1.0
con'ition
ransor!er stage
-
8/9/2019 Data Stage PPT Materials
169/216
Copyright Sennovate 2010. All rights
1-
1%,Ver.1.0
-
8/9/2019 Data Stage PPT Materials
170/216
ransor!er StageStage
varia*les
%onstraints
-
8/9/2019 Data Stage PPT Materials
171/216
Copyright Sennovate 2010. All rights
1-
1&1Ver.1.0
ransor!er stage
-
8/9/2019 Data Stage PPT Materials
172/216
Copyright Sennovate 2010. All rights
1-1&2Ver.1.0
ransor!er
-
8/9/2019 Data Stage PPT Materials
173/216
Copyright Sennovate 2010. All rights
1-1&"Ver.1.0
ransor!er
-
8/9/2019 Data Stage PPT Materials
174/216
Copyright Sennovate 2010. All rights
1-1Ver.1.0
ransor!er conditions
-
8/9/2019 Data Stage PPT Materials
175/216
Copyright Sennovate 2010. All rights
1-1&$Ver.1.0
Scenario&ro'uct 4le has pco'e an' pro'uct colour
&ro'ucts -ith yello- colour are move' to one 4le,*lue are move' to one 4le an' rest are move' toother
!his tas" is 'one using !ransformer stageconstraints
ransor!er stage ith " outputlin/s
-
8/9/2019 Data Stage PPT Materials
176/216
Copyright Sennovate 2010. All rights
1-1&%Ver.1.0
ransor!er stage constraints
-
8/9/2019 Data Stage PPT Materials
177/216
Copyright Sennovate 2010. All rights
1-1&&Ver.1.0
Change +ata capture stage
-
8/9/2019 Data Stage PPT Materials
178/216
Copyright Sennovate 2010. All rights
1-1&'Ver.1.0
%ompares t-o 'ata sets an' recor's the 'i3erences*et-een them.
-
8/9/2019 Data Stage PPT Materials
179/216
-
8/9/2019 Data Stage PPT Materials
180/216
+e:ning Para!eter in 6o5 Propertiesindo
-
8/9/2019 Data Stage PPT Materials
181/216
Copyright Sennovate 2010. All rights
1-1'1Ver.1.0
&aramete
rs option
@sing 6o5 Para!eters
-
8/9/2019 Data Stage PPT Materials
182/216
Copyright Sennovate 2010. All rights
1-1'2Ver.1.0
!o 7se the parameters in the o* useLparameternameL
-
8/9/2019 Data Stage PPT Materials
183/216
Containers
-
8/9/2019 Data Stage PPT Materials
184/216
Copyright Sennovate 2010. All rights
1-1'#Ver.1.0
Container is a group of stages and lin%s. It is used to modulari*e Server !ob designs using Container Stage
DataStage provides + types of containers
ocal Container
Shared Container
ypes o Containers
-
8/9/2019 Data Stage PPT Materials
185/216
Copyright Sennovate 2010. All rights
1-1'$Ver.1.0
ocal Container"hese are created within a !ob and are accessible only within a
!ob.
Shared Containers
"hese are created and stored separately in repository as !obs.
"here are + types of Shared Containers
Server Shared Containers
Server shared containers can be included in the parallel
!obs
$arallel Shared Containers
Creating
-
8/9/2019 Data Stage PPT Materials
186/216
Copyright Sennovate 2010. All rights
1-1'%Ver.1.0
If a o* is comple group stages an' lin" through acontainer to save an eisting stages an' lin"s in localcontainer
Select the Stages
)'it+%onstruct %ontainer+5ocal
!o insert an empty containerclic" %ontainer
Dou*le clic" the stage an' a'' stages an' lin"s
s
+econstructing local Container
-
8/9/2019 Data Stage PPT Materials
187/216
Copyright Sennovate 2010. All rights
1-1'&Ver.1.0
!o %onvert container in to group of 'iscrete stages an'lin"s in the o*.
Select the %ontainer Stage an' choose Deconstructfrom shortcut menu
diting local containers
-
8/9/2019 Data Stage PPT Materials
188/216
Copyright Sennovate 2010. All rights
1-1''Ver.1.0
%hoose the container an' clic" )'it++
&roperties
Shared Container
-
8/9/2019 Data Stage PPT Materials
189/216
Copyright Sennovate 2010. All rights
1-1',Ver.1.0
!o store the eisting stages an' lin"s in the share'container
%hoose the stages an' lin"s
%hoose )'it%ontainer+Share'
&arameters to the components are copie' to share'
container as %ontainer &arameters Saving it is same as saving a o*.
6o5 Se8uences
-
8/9/2019 Data Stage PPT Materials
190/216
Copyright Sennovate 2010. All rights
1-1,0Ver.1.0
Speci4es a se2uence of o*s to run. Se2uence can contain control information
ie, It is possi*le specify 'i3erent course of action to*e ta"en 'epen'ing on -hether a o* succee's orfails.
-
8/9/2019 Data Stage PPT Materials
191/216
Copyright Sennovate 2010. All rights
1-1,1Ver.1.0
-
8/9/2019 Data Stage PPT Materials
192/216
(estarta5le se8uence
-
8/9/2019 Data Stage PPT Materials
193/216
Copyright Sennovate 2010. All rights
1-1,"Ver.1.0
-
8/9/2019 Data Stage PPT Materials
194/216
Copyright Sennovate 2010. All rights
1-1,#Ver.1.0
ile ++Ce-+++
-
8/9/2019 Data Stage PPT Materials
195/216
Copyright Sennovate 2010. All rights
1-1,$Ver.1.0
&alette
(epository
Activity Stages
-
8/9/2019 Data Stage PPT Materials
196/216
Copyright Sennovate 2010. All rights
1-1,%Ver.1.0
-
8/9/2019 Data Stage PPT Materials
197/216
Copyright Sennovate 2010. All rights
1-1,&Ver.1.0
Ceste' con'itions
Allo-s you to further *ranch the eecution of ase2uence 'epen'ing on a con'ition.
Se2uencer
Allo-s you to synchronize the control 6o- of
multiple activities in a o* se2uence. Start an' en' loop
!ogether these t-o stages allo- you to implement aor...Cet or or...)ach loop -ithin your se2uence
!erminatorAllo-s you to specify that, if certain situations occur,
the o*s a se2uence is running shut 'o-n cleanly
Activity stages
-
8/9/2019 Data Stage PPT Materials
198/216
Copyright Sennovate 2010. All rights
1-1,'Ver.1.0
7ser Varia*le
Allo-s you to 'e4ne varia*les -ithin a se2uence.!hese varia*les can then *e use' later on in these2uence, for eample to set o* parameters.
)ceptional han'ler
It is eecute' if a o* in the se2uence fails to run9other eceptions are han'le' *y triggers: or if a
o* a*orts an' the Automatically han'le activitiesthat fail option is set for the se2uence.
$nly one )ception han'ler for a se2uence.
riggers
-
8/9/2019 Data Stage PPT Materials
199/216
Copyright Sennovate 2010. All rights
1-1,,Ver.1.0
!riggers provi'e control information to the StageActivities
Speci4es 'i3erent courses of action to *e ta"en *ase'on o*s status.
!rigger names must *e uni2ue
!ypes of !riggers%on'itional
7ncon'itional
$ther-ise
6o5 Se8uence Properties
Speci4esparameters
Displays co'e
-
8/9/2019 Data Stage PPT Materials
200/216
Copyright Sennovate 2010. All rights
1-200Ver.1.0
Scenario or )o5 Se8uence
N i 4l il *l i f l' i h h
-
8/9/2019 Data Stage PPT Materials
201/216
Copyright Sennovate 2010. All rights
1-201Ver.1.0
N input 4les are availa*le in a fol'er -ith the same
layout
Single Server
-
8/9/2019 Data Stage PPT Materials
202/216
Copyright Sennovate 2010. All rights
1-202Ver.1.0
/aits fortrigger 4le toappear
/hen any failureoccurs, control istransferre' here
)ecutes Sort
-
8/9/2019 Data Stage PPT Materials
203/216
Copyright Sennovate 2010. All rights
1-20"Ver.1.0
ppear option / does not deletethe file
after finding it
Current value of counter stored in
stage_label.2Counter
Progra!!ing in +ataStage
& i t
-
8/9/2019 Data Stage PPT Materials
204/216
Copyright Sennovate 2010. All rights
1-20#Ver.1.0
&rogramming components
(outines
!ransforms
unctions
)pressions
Su*routines
Macros
&rece'ence rules
(outines
( ti t ' i th ( ti f l' * ' f lt
-
8/9/2019 Data Stage PPT Materials
205/216
Copyright Sennovate 2010. All rights
1-20$Ver.1.0
(outines are store' in the (outines fol'er *y 'efault.
!he follo-ing components are classi4e' as routines
!ransform functions
BeforeAfter Su*routines
/hile 'esigning a o* it is possi*le to specify
%ustom 7niverse functions
ActiveP functions
7ecuting )o5s ro! co!!and line
d j b [ d [ NO!# $ %&%' $ (#)*'%
-
8/9/2019 Data Stage PPT Materials
206/216
Copyright Sennovate 2010. All rights
1-20%Ver.1.0
dsjob -run [ -mode [ NO!# $ %&%' $ (#)*'%+ + [ -param name,value + [ -arn n + [ -rosn + [ -ait + [ -stop + [ -jobstatus+ [-userstatus+ [-local+ [-opmetadata ['.% $#&%++ [-disableprjhandler+ [-
disablejobhandler+ [useid+ project job$job0id
Co!!ands
'sa'min comman'
-
8/9/2019 Data Stage PPT Materials
207/216
Copyright Sennovate 2010. All rights
1-20&Ver.1.0
'sa'min comman'
DSPImport Service comman'
Sync&roect comman'
Peror!ance tuning in +S
)nsure proper in'ees are create'
-
8/9/2019 Data Stage PPT Materials
208/216
Copyright Sennovate 2010. All rights
1-20'Ver.1.0
)nsure proper in'ees are create'.
&artition the ta*le -hereever re2uire'.
7se multiple no'es.
7se A&!D7M&S%$()
!ry to use or'er *y than sort.
Scenarios
Scenario 1
-
8/9/2019 Data Stage PPT Materials
209/216
Copyright Sennovate 2010. All rights
1-20,Ver.1.0
Scenario 1
If -e have ; o*s in se2uencer -hile running if o* 1is faile' then ho- to run other @ o*s
&roperties++trigger++++uncon'itional
Scenario @
!ry 5eft outer oin using 5oo"up stage
Server 6o5 Stages in '.1.2
%omple lat 4le Stage
-
8/9/2019 Data Stage PPT Materials
210/216
Copyright Sennovate 2010. All rights
1-210Ver.1.0
%omple lat 4le Stage
ol'er Stage
8ashe' 4le Stage
Se2uential 4le stage
Aggregator Stage
%omman' Stage
Interprocess Stage
!& plugin stage
5in" %ollecte' stage
5in" &artitioner stage
Server )o5 stages
Merge Stage
-
8/9/2019 Data Stage PPT Materials
211/216
Copyright Sennovate 2010. All rights
1-211Ver.1.0
Merge Stage
&ivot Stage
(o- merger Stage
(o- Splitter Stage
Sort Stage
!ransformer Stage
-
8/9/2019 Data Stage PPT Materials
212/216
Parallel )o5 Stages
&rocessing stages
-
8/9/2019 Data Stage PPT Materials
213/216
Copyright Sennovate 2010. All rights
1-21"Ver.1.0
&rocessing stages
!ransformer Stage
Basic !ransformer Stage
Aggregator Stage
-
8/9/2019 Data Stage PPT Materials
214/216
Copyright Sennovate 2010. All rights
1-21#Ver.1.0
%ompress Stage
)pan' stage
%opy Stage
Mo'ify Stage
ilter Stage
)ternal 4lter Stage
%hange capture stage
%hange apply Stage
Di3erence Stage
%ompare Stage
Parallel )o5 Stages
)nco'e stage
-
8/9/2019 Data Stage PPT Materials
215/216
Copyright Sennovate 2010. All rights
1-21$Ver.1.0
)nco'e stage
Deco'e Stage
S-itch Stage
!& )nterprise stage
#eneric stage
Surrogate "ey generator stage
Slo-ly %hanging 'imension Stage
&ivot )nterprise Stage
%hec"sum stage
(estructure stage
%olumn Import Stage
-
8/9/2019 Data Stage PPT Materials
216/216
%olumn Import Stage
%olumn )port Stage
Ma"e Su*recor' stage
Split Su*recor' stage
%om*ine recor' stage
&romote su*recor' stage
Ma"e Vector Stage
Split Vector Stage