where to start etl developer career
TRANSCRIPT
Where to Start? ETL Developer Career
Rodrigo Radtke
DevEpm.com
@RodrigoRadtke@DEVEPMAbout the Speaker
Radtke, Rodrigo• Oracle Ace• Graduated in Computer Engineering• Software Developer Sr. Advisor at
Dell• ODI, Oracle and Java Certified• Blogger @ devepm.com• ODTUG Leadership Program 2016
Class• Working with ETL for 9 years
DevEpm.com
@RodrigoRadtke@DEVEPMWhat we'll learn
• What ETL means?• Is ETL dead?• What the job looks like - The good, The bad• ETL Tools - Oracle Data Integration Portfolio• The path to ETL Mastery• Education/Certification• Social Media• Community/User Groups
DevEpm.com
@RodrigoRadtke@DEVEPMWhat ETL means?
(E)xtract, (T)ransform and (L)oad is the process responsible for pulling data out of source systems, transform it and placing it into a target system.
Extract• the data from source systems (Oracle, SAP, ERP, other operational systems, text files, Excel, etc)
Transform• cleaning (mapping NULL to 0, "Male" to "M“, "Female" to "F" etc.)• filtering (select only certain values to load)• joining together data from multiple sources (merge)• data validation (if the first 3 columns in a row are empty then reject the row from processing)
Load• the data into a data warehouse, data repository for reporting applications, a different/new application,
etc.
DevEpm.com
@RodrigoRadtke@DEVEPMWhat ETL means?
Kimball Group• www.kimballgroup.com
34 Subsystems of ETL• Three subsystems focus on extracting data from source systems.• Five subsystems deal with value-added cleaning and conforming, including
dimensional structures to monitor quality errors.• Thirteen subsystems deliver data as dimensional structures to the final BI
layer, such as a subsystem to implement slowly changing dimension techniques.
• Thirteen subsystems help manage the production ETL environment.Useful Links
• http://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/etl-architecture-34-subsystems/
• http://www.slideshare.net/MichaelRainey3/a-walk-through-the-kimball-etl-subsystems-with-oracle-data-integration
DevEpm.com
@RodrigoRadtke@DEVEPMWhat ETL means?
Where is ETL used? Basically everywhere!
• Reporting• Data warehouse/Business
Intelligence• Data consolidation• Data replication• Data migration• Big Data/IOT• Cloud
DevEpm.com
@RodrigoRadtke@DEVEPMIs ETL dead?
If there is data, there is ETL. And it is always changing/evolving…
Disconnected Mainframes Separate Interconnected Systems Big Data/IOT explosion
Manual ETL? Data Warehouse Data Lakes
DevEpm.com
@RodrigoRadtke@DEVEPM
What the job looks like - The good, The bad
What is expected from an ETL Developer
• Understand data models (Relational, Dimensional)
• Analyze business requirements• Develop and unit test mappings• Debug ETL executions• Development estimations• Performance tuning• Integration design
Soft Skills
• Analytical mindset• Good communication (specially for
requirement gathering)• Easiness to work with multiple
technologies• “Out of the box” thinking• Team player
DevEpm.com
@RodrigoRadtke@DEVEPM
What the job looks like - The good, The bad
Is it good to be an ETL Developer? Yes it is!
• Diverse technologies• Contact with business/power users• You learn the systems you load• Easiness to shift to other job positions• Work with one of the most valuable assets of the companies: Data
DevEpm.com
@RodrigoRadtke@DEVEPM
What the job looks like - The good, The bad
The bad: “Plumber” syndrome
• ETL developers are similar to plumbers but you need to learn that this is a good thing!
• Your work may not show up as much as you would liked
• Sometimes you will feel that you are only requested when the pipe is “broken”
• Do not let it discourage you: there are no houses without pipes as there are no business analyses without ETL
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Oracle Data Integration Portfolio Other ETL Tools:• OWB• SAP Data Services• IBM Infosphere• Informatica• SSIS• CloverETL• Pentaho
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Oracle Enterprise Metadata Management (OEMM)
• Metadata governance• Impact Analysis• Source Lineage
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Oracle Enterprise Data Quality (EDQ)• Profiling, Audit and Dashboards• Parsing and Standardization• Match and Merge• Address Verification
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Oracle GoldenGate• Real-Time Data Integration• Query offloading• Data distribution• Heterogeneous support• Zero-down time operations• Disaster recovery and data protection
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Oracle Data Integrator• E-LT Architecture• Knowledge Modules and Mappings
• Focus on “what” rather than “how”• Complete development and execution
platform• Imbedded Data Quality process• Heterogeneous Support• Data Warehousing, BI, Big Data, SOA,
Migration, Real Time Integration, etc…
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Oracle SQL Developer• Probably the most used tool in
ETL development (more than ODI itself)
• Free integrated development environment for databases
• Run queries, create DB objects, run install scripts, create PL/SQL, data export, DB compare, manage DB sessions and so much more
DevEpm.com
@RodrigoRadtke@DEVEPMETL Tools - Oracle Data Integration
Get start with ODI at Oracle site “Getting Started” Section• http://
www.oracle.com/technetwork/middleware/data-integrator/overview/index.html• https://apexapps.oracle.com/pls/apex/f?p=44785:24:15503233161396::::
P24_CONTENT_ID,P24_PREV_PAGE:7952,24
DevEpm.com
@RodrigoRadtke@DEVEPMThe path to ETL Mastery
• You need to know what your ETL tool is capable of• SQL is the key to performance and to understand what is going on• A computer language will give you the ability to modify/create new
things• You will only be able to influence your business if you understand it• Theory behind any development is key to success
ETL tool SQL Java/Jython/Groovy
Business processes
Study the theory of what you are working on
DevEpm.com
@RodrigoRadtke@DEVEPMEducation/Certification
Oracle training• http://education.oracle.com
Why get certified?• Study a new tool/technology• Get recognized by the industry
Useful certifications for ETL development• Oracle Database SQL Expert – Exam: 1Z0-047• Oracle Data Integrator 12c Implementation Essentials - Exam: 1Z0-448• Java SE 8 Programmer – Exam: 1Z0-808
DevEpm.com
@RodrigoRadtke@DEVEPMSocial Media
Social Media: your endless source of informationForums
• https://community.oracle.com/community/business_intelligence/system_management_and_integration/data_integrator
Twitter• @OracleDI, @orclbiblogs, @oracleotn, @odtug
LinkedIn Groups• Oracle Data Integrator (ODI)• Oracle Data Integration• Data Migration/Integration & ETL Group
Blogs• https://blogs.oracle.com/dataintegration/• http://www.ateam-oracle.com/?cat=63
DevEpm.com
@RodrigoRadtke@DEVEPMCommunity/User Groups
Benefits of joining an User Group• Networking• Getting Knowledge/Experience• Sharing Knowledge/Giving Back• Friendship• Business Opportunities• Passionate people
Oracle User Groups• http://www.oracle.com/us/corporate/customers/user-groups/index.html• https://community.oracle.com/community/usergroups
ODTUG/Kscope changed my life• Kscope 13, 14, 15 and 16
presenter• Kscope 16 Reviewer• ODTUG webinars• OTN publications/2MTT/Podcasts• Oracle ACE Associate, ACE• ODTUG Leadership Program
DevEpm.com
@RodrigoRadtke@DEVEPMCommunity/User Groups
Conferences: Kscope• The brightest minds working with Oracle Technology
in one single place• More than 250 sessions• 40 hands-on lab sessions• More than 55 Oracle ACE Directors and ACEs in
attendance• 1600 attendees• Attendees have access to recorded presentations
post-conferenceODTUG Kscope16 New Oracle Professional Scholarship
• http://kscope16.com/registration/scholarshipKscope16 New Oracle Professional Tracks
• http://kscope16.com/content/new
DevEpm.com
@RodrigoRadtke@DEVEPMCommunity/User Groups
The Career Track is one of many communities that an ODTUG member can follow. It contains information, resources and presentations specifically geared towards people just starting their careers. In addition to the materials and access that will make you successful, the Career Track community will have exclusive networking events with the current (and future) leaders within the Oracle communities.
Follow us on #orclCareer
DevEpm.com
@RodrigoRadtke@DEVEPM
Rodrigo Radtke de Souza – Dell@Rodrigo Radtke
[email protected]://br.linkedin.com/in/rodrigoradtkedesouza/en
Thank you!
