jbatch with apache...
TRANSCRIPT
![Page 1: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/1.jpg)
Mark Struberg / INSO, TU Vienna
JBatch with Apache BatchEE
![Page 2: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/2.jpg)
About the Speakers
• http://github.com/struberg
• http://people.apache.org/~struberg
• Apache Software Foundation member
• Apache OpenWebBeans, MyFaces, BVal, OpenJPA, Maven, DeltaSpike, BatchEE, ...
• Java EG member
![Page 3: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/3.jpg)
Agenda
• JBatch and Apache BatchEE• JBatch Terms and Ideas• Writing a simple Batch• Threading and Transactions• JBatch Execution Environments• BatchEE add-ons
![Page 4: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/4.jpg)
About Apache BatchEE
• Apache Incubator Project since 2013-10• Homepage:
http://batchee.incubator.apache.org
• Source:git://git.apache.org/incubator-batchee.git
• [email protected]• irc.freenode.net channel #apache-batchee
• https://issues.apache.org/jira/browse/BATCHEE
![Page 5: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/5.jpg)
Diff JBatch and BatchEE
• JBatch– The specification
– The RI
• BatchEE– includes forked and bug-fixed RI code for Java EE
– adds tons of features on top of it:
– different SPI implementations
– various ItemReader and Writer,
– JAX-RS and simple Job UI,
– Standalone Java EE Batch Environment
![Page 6: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/6.jpg)
JBatch Basics
![Page 7: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/7.jpg)
What is JBatch
• A Java EE specification, JSR-352– https://jcp.org/en/jsr/detail?id=352
• Inspired by WebSphere-Batch, Spring-Batch, JES, JCL, ...
• To write and execute Java Batches• Why do I need Batches ?
– Migration
– Bulk Import/Export
– Mass processing
![Page 8: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/8.jpg)
Batch Classification
• Online vs Offline Tasks• Bulk vs On-demand Batches• Synchronous vs Asynchronous
processing
![Page 9: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/9.jpg)
Benefits of JBatch Spec
• improved maintainability• improved portability• reuse of existing app components• out-of-the-box monitoring and control
![Page 10: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/10.jpg)
JBatch Architecture
JobOperator
Job StepItemReader
ItemProcessor
ItemWriter
Batchlet
JobRepository
![Page 11: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/11.jpg)
JobRepository
JobRepository
JobInstance
JobExecution
JobInstance
StepExecution
![Page 12: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/12.jpg)
Batch JSL
• JSL: Job Specification Language• META-INF/batch-jobs/*.xml<job id="myjob" restartable="false">
<step id="firststep">
<chunk item-count="1">
<reader ref="myReader"/>
<processor ref="myProcessor"/>
<writer ref="myWriter"/>
</chunk>
</step>
</job>
![Page 13: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/13.jpg)
Batch Artifact Loading
• Available as @Named• Registered via META-INF/batch.xml
![Page 14: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/14.jpg)
Batch Properties (1/2)
• Defined In JSL:<properties>
<property name="myProp" value="myVal"/>
</properties>
• for:– <job>
– <reader>, <processor>, <writer>
– <batchlet>
• Dynamically via JobOperator
![Page 15: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/15.jpg)
Batch Properties (2/2)
• Usage in Code via • JobContext#getProperties();
• StepContext#getProperties();
• @Inject @BatchProperty("myProp") String val;
• Can leverage Expression Language:• <property name="csvFile"
• value="#{jobProperties['infile.name']}"/>
• <property name="jobDay"
• value="#{partitionPlan['schedule.day']}"/>
![Page 16: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/16.jpg)
Basic Step Structure
• Batchlet or• ItemReader, plus• ItemProcessor (optional), plus• ItemWriter• Steps can form Sequence Flows• Steps can use Partitioning and Splits
- each Partition gets an own Thread
![Page 17: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/17.jpg)
Batchlet
• performs all the processing in one go• Batchlet#process()
• Batchlet#stop()
• Return the exit status of the batch
![Page 18: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/18.jpg)
ItemReader
• Read single Item for a chunk step• Can set and read 'Checkpoints'• readItem() invoked until null returned
![Page 19: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/19.jpg)
BatchEE TypedItemReader
• <R, C> version of ItemReader– R: Type of item to Read
– C: Type of Checkpoint (? extends Serializable)
• <R> NoStateTypedItemReader• <R> BufferedItemReader
![Page 20: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/20.jpg)
ItemProcessor
• Optional!• Processes the read item• Returning null will skip the item
![Page 21: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/21.jpg)
BatchEE TypedItemProcessor
• <I, O> version of ItemProcessor– I: Type of item input
– O: Type of Output
![Page 22: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/22.jpg)
ItemWriter
• Writes n items (according to chunk-size)• Each writeItems is an own Transaction• Can set and read 'Checkpoints'
![Page 23: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/23.jpg)
BatchEE TypedItemWriter
• <W, C> version of ItemWriter– W: Type of item to Write
– C: Type of Checkpoint (? extends Serializable)
• <R> NoStateTypedItemWriter
![Page 24: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/24.jpg)
Batch Transaction Handling
• JBatch uses JTA by default• JTA TX is always per Thread in Java• Each 'Split' Partition has an own
Transaction• Commit packages can be set via
chunk-size
![Page 25: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/25.jpg)
Batch Chunks
• chunk-size=100 means– beginTransaction();
– 100x ItemReader#readItem();
– (optional) 100x ItemProcessor#processItem();
– 1x ItemWriter#writeItems(all100);
– commit();
![Page 26: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/26.jpg)
Chunk Exception Handling
• default: abort w FAILED on Exception• possible to define Exceptions for:
– retry <retryable-exception-classes>plus optionally: Retry*Listener
– skip <skippable-exception-classes>plus optionally: Skip*Listener
– Retry is always done with item-count=1
– Retry precedes Skip
![Page 27: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/27.jpg)
The JobOperator
• THE central interface to communicate with the Batch Runtime
• can start, stop and restart Jobs• can list Jobs
![Page 28: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/28.jpg)
Batch Listener
• JobListener• beforeJob()
• afterJob()
• StepListener• beforeStep()
• afterStep()
• e.g. for setting up Authentication• e.g. for starting CDI Contexts
![Page 29: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/29.jpg)
BatchEE SPI
![Page 30: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/30.jpg)
BatchEE SPI activation
• batchee.properties• BatchThreadPoolService=....
• SecurityService• PersistenceManagerService
• BatchThreadPoolService• TransactionManagementService• http://batchee.incubator.apache.org/configuration.html
![Page 31: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/31.jpg)
Batch Environments
![Page 32: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/32.jpg)
Batch WAR
• BatchEE Servlet• UI for
– showing status
– creating Jobs
– starting Jobs
![Page 33: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/33.jpg)
JAX-RS Integration
• Exposes JobOperator via JAX-RS• Allows to start Jobs from UC4, JES, etc
![Page 34: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/34.jpg)
CLI Batch
• We introduced our own BAR formatBATCH-INF
+-- batch-jobs/*.xml
+-- classes/..
+-- lib/..
• $> java -jar batchee-cli.jar start \ -archive demo-1.0.bar -name mybatch
![Page 35: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/35.jpg)
CDI Batch Scopes
• @StepScoped• @JobScoped• technically done via StepListener and
JobListener
![Page 36: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/36.jpg)
Clustering
• TODO... it's just not yet done...• ... helping hands are welcome...
![Page 37: JBatch with Apache BatchEEpeople.apache.org/~struberg/jax2014/apache_batchee_jbatch_jax2014.pdfAgenda •JBatch and Apache BatchEE •JBatch Terms and Ideas •Writing a simple Batch](https://reader033.vdocuments.us/reader033/viewer/2022042314/5f020c197e708231d4025056/html5/thumbnails/37.jpg)
Legal stuff
• Apache, BatchEE, OpenWebBeans, OpenEJB and OpenJPA are trademarks of the Apache Software Foundation