[wso2con eu 2017] ballerina: exploring data integration
TRANSCRIPT
![Page 1: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/1.jpg)
Architect, WSO2
Ballerina: Exploring Data Integration
Anjana Fernando
![Page 2: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/2.jpg)
• Source: Heterogeneous Data Sources• Processing:
– Data Quality/Cleansing– ETL
• Store/Mining:– Data Warehouse
• Expose:– Data Services
Data Integration Explained
![Page 3: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/3.jpg)
• Built-in support for popular data formats, XML/JSON/Tables– Natural manipulation of data
• Extensive Data Connector Support (SQL, NoSQL)• Transactions (Local, Distributed)• Graphical data modelling with Composer• Scheduled data processing jobs with tasks• Rich HTTP REST service development support for
rapid data services exposure
How Ballerina Fits In
![Page 4: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/4.jpg)
• Value Types– int/float/boolean/string/blob
• Arrays– type[]
• Maps– map
• XML– “xml” type with fully functional XML sequence API
Ballerina Data Types
![Page 5: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/5.jpg)
• JSON– “json” type with constrained JSON support for
schema enforcement• Datatable
– “datatable” type in representing a resultset from data connectors
• User defined types– “struct” type for data binding – Automatic data validation– Record representation in datatable type
Ballerina Data Types
![Page 6: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/6.jpg)
Data Connectors• SQL Connector (built-in)
– MySQL, MSSQL, Oracle, DB2... – Supports DML / DDL operations– Stored procedures / batch
operations– UDT support– XA transaction support
• Multiple databases / Message queues / topics
![Page 7: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/7.jpg)
Data Connectors...• SQL Connector
![Page 8: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/8.jpg)
Data Connectors...• Cassandra Connector (https://github.com/ballerinalang/connector-cassandra)
– CQL based data querying
– Streaming results with datatable integration
![Page 9: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/9.jpg)
Data Connectors...• Cassandra Connector
![Page 10: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/10.jpg)
Data Connectors...• MongoDB Connector (https://github.com/ballerinalang/connector-mongodb)
– MongoDB CRUD API support
• insert/find/update/delete
• Native JSON data handling
– Ballerina ‘json’ data type
• JSON streaming support
![Page 11: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/11.jpg)
Data Connectors...• MongoDB Connector
![Page 12: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/12.jpg)
Transactions• Local / XA Transactions
– Declarative transactions by defining a transaction boundary in the program
– The mode is inferred from the participants in the transaction scope
– Currently support SQL databases and message brokers for distributed transactions
![Page 13: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/13.jpg)
Transactions...
![Page 14: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/14.jpg)
Graphical Data Modelling
Data Mapping with Ballerina Transformer
![Page 15: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/15.jpg)
• ETL jobs can be scheduled with the Ballerina tasks features:– Timers: delay + interval scheduling– Appointments: cron based scheduling
Scheduled Data Jobs
https://www.informatec.com/sites/default/files/inline-images/etl_2.jpg, https://www.cosculture.com/static/images/[email protected]
![Page 16: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/16.jpg)
Scheduled Data Jobs Timers:-
![Page 17: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/17.jpg)
Scheduled Data Jobs Appointments:-
![Page 18: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/18.jpg)
• HTTP/WS server connectors• Ability to generate REST APIs rapidly• Native data streaming capability
– XML / JSON• Batch data request support for high performance data
loading• Power of a general purpose programming language
with the ease of a visual programming approach– Data preprocessing, filtering, access control etc...
Data Services
![Page 19: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/19.jpg)
Data Services...
![Page 20: [WSO2Con EU 2017] Ballerina: Exploring Data Integration](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a64906b7f8b9a27568b687b/html5/thumbnails/20.jpg)
• Ballerina provides an efficient set of connectors, tools and technologies for easy data access/manipulation and serving.
More information, visit https://ballerinalang.org/.
Summary