oracle goldengate and apache kafka a deep dive into real-time data streaming

[email protected] www.rittmanmead.com @rittmanmead 1

[email protected] www.rittmanmead.com @rittmanmead

Michael Rainey | KScope16

Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming

2


Introduction

3

• Michael Rainey - Data Integration Lead - America- Oracle Data Integration expertise - Blog: http://ritt.md/mRainey - Oracle ACE Director

@mRainey

http://ritt.md/mRainey


About Rittman Mead

4

•World’s leading specialist partner for technical excellence, solutions delivery and innovation in Oracle Data Integration, Business Intelligence, Analytics and Big Data

•Providing our customers targeted expertise; we are a company that doesn’t try to do everything… only what we excel at

•70+ consultants worldwide including 1 Oracle ACE Director and 3 Oracle ACEs, offering training courses, global services, and consulting

•Founded on the values of collaboration, learning, integrity and getting things done

Unlock the potential of your organization’s data

•Comprehensive service portfolio designed to support the full lifecycle of any analytics solution


Visual Redesign Business User Training

Ongoing SupportEngagement Toolkit

Average user adoption for BI platforms is below 25%

Rittman Mead’s User Engagement Service can help

More info: http://ritt.md/ue

http://ritt.md/ue


Today’s New Data Challenge

6


Data Integration Today

7


Typical Example - Marketing

8

• Financial data stored in RDBMS• Social media data, web logs, Google analytics, etc all in

various formats• Bring it all together for analysis ‣ Marketing campaign effect on sales


Relational Data Replication - Oracle GoldenGate

9


Oracle GoldenGate for Big Data (Then)

10


Oracle GoldenGate for Big Data (Now)

11


Streaming Data - Apache Kafka

12

“Publish-subscribe messaging rethought as a distributed commit log”

Image source: kafka.apache.org/

http://kafka.apache.org/


Streaming Data - Apache Kafka

13

Image source: kafka.apache.org/

http://kafka.apache.org/


Kafka - How is it used?

14

• Pure Event Streams• System Metrics• Derived Streams• Hadoop Data Loads / Data Publishing• Application Logs• Database Changes- Log Compaction - Data cleansing

Image source: confluent.io

http://confluent.io


Let’s Jump Right In

15

• An example…near and dear to my heartOne single view of the Oracle Data Integrator logs!

- Oracle Data Integrator session logs stored in the repository - ODI Agent logs are text based log files - To see the full picture of your ODI environment, they must be

combined


oracle.com/technetwork/database/bigdata-appliance/downloads

http://www.oracle.com/technetwork/database/bigdata-appliance/downloads/index.html


Oracle GoldenGate for Big Data - we talked about this…

17


ODI Agent Logs to Kafka via Logstash

18


Extract from the ODI Repository with GoldenGate 12c

19

• Prepare the database • Setup GoldenGate for Oracle Database- Install and configure • Setup Manager, Extract and Pump parameter files• Add Extract and Pump process groups• Start!


Prepare the Database for GoldenGate Extract - OGG User

20


Prepare the Database for GoldenGate Extract - Logging

21


Add Table Supplemental Logging

22


GoldenGate Manager Parameter File - Source

23


GoldenGate Extract Parameter File

24


GoldenGate Pump Parameter File

25


Add Extract and Pump Process Groups

26


Stream ODI Agent Logs to Kafka via Logstash

27

• Application log processing is a standard use for Kafka- Many approaches to extract logs • Logstash- Part of the Elastic (formerly ELK) stack - Robin Moffatt blogged —> http://ritt.md/kafka-elk - Producer configuration for Kafka

http://ritt.md/kafka-elk


Logstash to Kafka - Setup and Startup

28

• Startup Zookeeper- Already installed on Big Data Lite • Set Kafka server.properties- Broker ID - Number of partitions - Log retention period - Zookeeper connection • Start Kafka


Setup Logstash Configuration File

29

• Configuration File

• Start Logstash


ODI Agent Logs to Kafka!

30

• Start the Kafka Console Consumer - delivered with Kafka

• Start the ODI Agent and…messages!


ODI Agent Logs to Kafka!

31


GoldenGate Transactions to Kafka

32


Oracle GoldenGate for Big Data

33

• Kafka one of many handlers- HDFS, HBase, Flume • Pluggable Formatters- Convert trail file transactions to alternate format - Avro, delimited text, JSON, XML • Metadata Provider- Handles mapping of source to target columns that differ in structure/name - Similar to SOURCEDEF file in GoldenGate - Avro or Hive


Oracle GoldenGate for Big Data - Kafka Handler

34

• Standard GoldenGate Extract / Pump processes- We just set this up • Replicat parameter file & process group• Kakfa Handler configuration• Kafka Producer properties- Note: Kafka 0.9.0+ now certified with GoldenGate for Big Data

12.2.1.1


GoldenGate and Kafka…Prerequisites

35

• Zookeeper & Kafka up and running• Add topic to broker up front vs dynamically- Option to create a topic per table (12.2.0.1.1) • Kafka Handler must have access to broker server• Kafka libraries must match Kafka version


GoldenGate and Kafka…Replicat Parameters

36


GoldenGate and Kafka…Kafka Handler Properties

37

• Properties allow communication between the GoldenGate adapter and Kafka



38

• gg.handlerlist = kafkahandler• gg.handler.kafkahandler.type = kafka• gg.handler.kafkahandler.KafkaProducerConfigFile = kafka_producer.properties• gg.handler.kafkahandler.TopicName = odirepo- Kafka topic name • gg.handler.kafkahandler.format = json- Pluggable Formatter - Avro recommended for Kafka… • gg.handler.kafkahandler.BlockingSend = true|false • gg.handler.kafkahandler.includeTokens = true|false • gg.handler.kafkahandler.mode = tx- Transaction vs Operation mode


GoldenGate and Kafka…One Topic Per Table

39

• gg.handler.kafkahandler.topicPartitioning=table - Option to split schema into one topic per table- Can be created dynamically• gg.handler.kafkahandler.mode=op - Operation mode to track individual table operations



40

• goldengate.userexit.timestamp = utc• goldengate.userexit.writers = javawriter• javawriter.stats.display = TRUE• javawriter.stats.full = TRUE• gg.log = log4j• gg.log.level = INFO• gg.report.time = 30sec• gg.classpath = dirprm/:/u01/kafka/kafka_2.10-0.8.2.1/libs/*:- Location of the Kafka libraries• javawriter.bootoptions = -Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar


GoldenGate and Kafka…Kafka Producer Configuration

41

• Access to the Kafka producer configuration parameters

More on this later


GoldenGate and Kafka…Startup

42

• Create a topic in Kakfa

• Add Replicat process group to GoldenGate on target

• Start Kafka console consumer

• Start GoldenGate extract/pump on source, replicat on target


GoldenGate and Kafka Integration Complete!

43


Schemas…

44

• Schema automatically created - Stored in <ogg_home>/dirdef directory - Based on gg.handler.kafkahandler.format setting


GoldenGate Big Data Adapter - What to Think About

45

• GoldenGate could be a single point of failure - Kafka is a fault-tolerant, distributed system • Source transactions may end up larger than expected - max.request.size • Need for speed?- batch.size - linger.ms - BlockingSend = false


Why GoldenGate with Kafka?

46

• GoldenGate- …is non-invasive - …has checkpoints for recovery - …moves data quickly - …is easy to setup


In conclusion

47

• The new data challenge, not quite as challenging with Kafka- Kafka throughout


Questions?

48


Questions?

49

• Websites- kafka.apache.org - rittmanmead.com/blog • Contact- [email protected] - [email protected] • Twitter- @rittmanmead - @apachekafka - @mRainey

http://kafka.apache.org

http://www.rittmanmead.com/category/biapps

mailto:[email protected]

mailto:[email protected]?subject=

https://twitter.com/rittmanmead

https://twitter.com/apachekafka

http://www.apple.com


Questions?

49

• Websites- kafka.apache.org - rittmanmead.com/blog • Contact- [email protected] - [email protected] • Twitter- @rittmanmead - @apachekafka - @mRainey


Rittman Mead at KScope16

50

Oracle GoldenGate and Apache Kafka: A Deep Dive into Real-Time Data Streaming

Michael Rainey | Monday Jun 27, 4:30pm | Level 2 - Missouri

Free-Form Data Visualizations: First Session

Charles Elliott | Tuesday Jun 28, 8:30am | Level 2 - Superior ALunch & Learn: BI and Data Warehousing

Michael Rainey | Tuesday Jun 28, 12:45pm | Ballroom Level - Sheraton IILunch & Learn: Big Data and Advanced Analytics

Mark Rittman | Tuesday Jun 28, 12:45pm | Ballroom Level - Sheraton IIIOBIEE 12c and Essbase: What’s New for Integration and Reporting Against EPM Sources

Mark Rittman | Wednesday Jun 29, 10:15am | Ballroom Level - Sheraton III

A Walk Through the Kimball ETL Subsystems with Oracle Data Integration

Michael Rainey | Wednesday Jun 29, 11:30am | Level 2 - MayfairHow to Brand and Own Your OBIEE Interface: Past, Present, and Future

Andy Rocha & Pete Tamisin | Wednesday Jun 29, 2:00 pm | Ballroom Level - Sheraton III

Free-Form Data Visualizations: Second Session

Charles Elliott | Wednesday Jun 29, 2:00pm | Level 2 - Superior AOracle Big Data Discovery: Extending into Machine Learning and Advanced Visualizations

Mark Rittman | Wednesday Jun 29, 3:15pm | Level 2 - Missouri


Join Your Community! Tonight from 8:00 – 10:00 in Chicago IX

BIWarriorTriviaTimetogetoutthefunnyhats,ba6leaxes,andyourpassionfortriviabecauseBICommunityTriviaisback!ThisquirkyeventisquicklybecomingaKscopemust-see.ThisyearwearegeDngdownmedieval-stylewithBIWarriorTrivia,sogatheryourclanandpreparetoclash(mentally,ofcourse)aswesquareoffinthistrulysillyba6leofwits.Joinusaswedrinksomebeer,giveawayprizes,andputourknowledgetothetestatBITriviaNight!

oracle goldengate and apache kafka a deep dive into real-time data streaming

Technology