Spring Batch Groovy & Gradle integrationSpring Batch Groovy & Gradle integration

Because we like it…Because we like it… because we have fun with itbecause we have fun with it

About me:About me:

Born with C and Java

Github: http://github.com/amr390 (mostly drops, and tests)

Working with Java and Javascript, take any opportunity to use groovy and python

(I love for both of them)

Grown up with Javasince 2000

Twitter: 390amr (not much into it)

Spring Batch. Groovy and GradleSpring Batch. Groovy and Gradle

What is Spring BatchWhat is Spring Batch

Why integrating spring-batch with Groovy and GradleWhy integrating spring-batch with Groovy and Gradle

Requirements Requirements ➢ DependenciesDependencies

➢ Gradle tasksGradle tasks

What is Spring BatchWhat is Spring Batch

➢Configuring Jobs, Tasks, Steps and stufConfiguring Jobs, Tasks, Steps and stuf

➢Running and Handling JobsRunning and Handling Jobs

Improving Spring Batch performanceImproving Spring Batch performance

➢Multi-threadingMulti-threading

➢Master-Slave approachMaster-Slave approach

➢Parallelizing Step executionsParallelizing Step executions

What is Spring Batch I

Batch Framework

Technical services

Functionality to process large volumes of records

ItemReader, ItemWriter, ItemProcessor

Stateless or Stateful step executions

High-volumeHigh-performanceHigh-scalability

Trough optimization Partitioning techniques

What is Spring Batch IIWhat is Spring Batch II

What is Spring Batch IIIWhat is Spring Batch III

What is Spring Batch IVWhat is Spring Batch IV

STEP:➔ Custom tasklet➔ Restartable➔ Skippable➔ Retry➔ Rollback➔ Listeners Before/After

✗ Reader✗ Processor✗ Writer✗ Chunk✗ Step

➔ Step flow (on status)

What is Spring Batch: exampleWhat is Spring Batch: example

Philae Lander dataPhilae Lander data

Philae Lander is a spacecraft that sends analysis data from comet 67P/Churyumov–Gerasimenko (https://en.wikipedia.org/wiki/Philae_(spacecraft))

Sends tones data in files encrypted in Klingon or Murciano.

Files are zipped, and organized with a naming rule of file1, file2 etc ...

It is always fun using groovy

Can be nicely coupled to other platforms Grifon, Grails...

Power of Gradle

Homogeneous Architecture

Why integrating Spring-batch with Why integrating Spring-batch with groovy and Gradlegroovy and Gradle

Plug insPlug insRepositoriesRepositories

DependenciesDependencies TasksTasks

Requeriments IIRequeriments II

Repositories

Plugins

Dependencies IIIDependencies III

Dependencies

Requirements IVRequirements IV

Gradle tasks

MainClassName: Sets the project main class (because it is a global definition)

run: sets the required arguments for run task, in this case the job configuration class and the Job to be launched.

@Configuration@EnableBatchProcessing

Injects

a JobRepository a JobLauncher a JobRegistry a PlatformTransactionManagera JobBuilderFactorya StepBuilderFactory

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Job definition configurations, steps, tasklets and stuf

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Running and Handling Jobs

Spring batch admin

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Running and Handling Jobs

Job Launchers

org.springframework.batch.core.launch

Interface JobLauncher

All Known Implementing Classes:

SimpleJobLauncher

JobExecution run(Job job, JobParameters jobParameters) throws JobExecutionAlreadyRunningException, JobRestartException, JobInstanceAlreadyCompleteException, JobParametersInvalidException

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Running and Handling Jobs

Job Launchers

org.springframework.batch.core.launch.support

Class CommandLineJobRunner

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Performance improvement

There different approached in order to improve the job execution performance

Multi-threading: Using a ThreadPoolTaskExecutor

Remote chunking:Master executes read phase of step slaves are distributed in nodes and execute processing and writing from a chunkProvider.

Partitioning: Clone Job execution step and run steps in parallel

Remote Partitioning:

Instead of running the steps clone locally steps are

distributed

Parallel steps:step execution is splitted to

execute several steps in parallel flows.

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Multi

threading

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Multi

threading

Using task namespace <task:executor id="taskExecutor" pool-size="10" />

By java configuration method:ThreadPoolTaskExecutor taskExecutor ...

Steps must be thread safe

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Master-Slave

approach

PartitionHandler: Interface defining the responsabilities of controlling the execution of partitioned step

Partitioner: Interface to determine the way of creating partitions (key-range, file-chunks...)

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Master-Slave

approach

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Parallelizing Step executions

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

Parallelizing Step executions

@Beanpublic SimpleFlow mainFlow() { SimpleFlow splitFlow = new FlowBuilder<SimpleFlow>("Split Flow") .split(new SimpleAsyncTaskExecutor()) .add(flow2(), flow3()) .build(); return new FlowBuilder<SimpleFlow>("Main Flow") .start(flow1()) .next(splitFlow) .end();}

<batch:step id="step1" next="">...</batch:step><batch:split id="splitSteps" next="step3">

<batch:flow> <batch:step … /></batch:flow><batch:flow> <batch:step … /></batch:flow>

</batch:split><batch:step id="step3">

<batch:tasklet ref="tasklet" /></batch:step>

Demo Github: https://github.com/amr390/groovy-springbatch-demo

Spring batch reference page: http://docs.spring.io/spring-batch/trunk/reference/Spring batch in action: http://www.manning.com/templier/

Spring batch horizontal-vertical scaling:http://www.ontheserverside.com//blog/2014/07/23/horizontal-and-vertical-scaling-strategies-for-batch-applications

Spring batch admin: http://docs.spring.io/spring-batch-admin/

Spring Integration reference page: http://docs.spring.io/spring-integration/reference/

Groovy: http://www.groovy-lang.org/documentation.html

Gradle: https://docs.gradle.org/current/release-notes

Spring Batch Groovy & Gradle Spring Batch Groovy & Gradle integrationintegration

References