use brms for data quality in your bigdata pileline

13
Session title * Red Hat & Dell Confidential Use RED HAT BRMS For DataQuality In Your BigData PileLine Anurag Saran Sr Solution Architect, Red Hat Financial Services Email: [email protected] Twitter: @anuragsaran

Upload: anurag-saran

Post on 18-Feb-2017

41 views

Category:

Data & Analytics


2 download

TRANSCRIPT

Session title

*Red Hat & Dell Confidential

Use RED HAT BRMS For DataQuality In Your BigData PileLine

Anurag SaranSr Solution Architect, Red Hat Financial ServicesEmail: [email protected]: @anuragsaran

AGENDA

● Data Quality Using BRMS With Demo○ Using Sparse Decision Table○ Using Consolidated Decision Table○ Using Decision Table To Invoke Relevant Rules○ Using Decision Table To Invoke Relevant Rules + Data

Lookup/Translation + Data Correction○ Using Domain Specific Language

HOW TO AUTHOR RULES AS AN ANALYST?WEB DECISION TABLEDesign and/or Edit decision tables in the web editor

EXCEL SPREADSHEETAnnotate an excel spreadsheet with rule logic, then execute in the embedded app, or service.

WEB GUIDED RULE A web based interface will allow you to create rules, while reflecting on the data model you uploaded or designed through the data modeler.

WEB DOMAIN SPECIFIC LANGUAGEUsing a mapping between a domain specific natural language and technical syntax - you can compose rules with conditions and consequences, entering data into placeholders.

HOW TO AUTHOR RULES AS A DEVELOPER?DRL EDITORThe drools syntax is Java like. The when clause may be formed with Java or MVEL syntax. All rules formats get translated based to DRL.

EXCEL SPREADSHEETAnnotate an excel spreadsheet with rule logic, then execute in the embedded app, or service.

DOMAIN SPECIFIC LANGUAGE MAPPINGUsing a mapping between a domain specific natural language and technical syntax. The eclipse tooling includes both a DSL translation and DSL rule editor.

DOMAIN SPECIFIC LANGUAGE RULEUsing a mapping between a domain specific natural language and technical syntax. The eclipse tooling includes both a DSL translation and DSL rule editor.

Using Sparse Decision Table

Using Sparse Decision Table

Using Consolidated Decision Table

Using Consolidated Decision Table

Using Decision Table To Invoke Relevant Validation Rules

Using Decision Table To Invoke Relevant Validation Rules

RecordObjrecord_id = Rec001offer_id = 001offer_type_cd = 09Brate = 55payment_method_type_cd = CASH

FieldRestrictions:fieldname: record_idvalue: Rec001Pattern: R*LengthMin: 10Lenghtmax:20

FieldRestrictions:fieldname: ratevalue: 55RateMin: 10RateMax:22

FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn: CASH, DEBIT

FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern: [0-9][A-Z][A-Z]

LenghtRule

PatternRule

ListRule

RangeRule

CrossDataRefRule

ListRule

Result

FieldRestrictions:fieldname: record_idvalue: Rec001Pattern:LengthMin: Lenghtmax:

FieldRestrictions:fieldname: ratevalue: 55RateMin: RateMax:

FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn:

FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern:

createRestrictions.drl populateRestrictions.xls validateFields.drl

Using Decision Table To Invoke Relevant Validation Rules +Data Lookup/Translation +Data Correction

RecordObjrecord_id = Rec001offer_id = 001offer_type_cd = 09Brate = 55payment_method_type_cd = CASH

FieldRestrictions:fieldname: record_idvalue: Rec001Pattern: R*LengthMin: 10Lenghtmax:20

FieldRestrictions:fieldname: ratevalue: 55RateMin: 10RateMax:22

FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn: CASH, DEBIT

FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern: [0-9][A-Z][A-Z]

LenghtRule

PatternRule

ListRule

RangeRule

CrossDataRefRule

ListRule

Result

FieldRestrictions:fieldname: record_idvalue: Rec001Pattern:LengthMin: Lenghtmax:

FieldRestrictions:fieldname: ratevalue: 55RateMin: RateMax:

FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn:

FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern:

createRestrictions.drl populateRestrictions.xls validateFields.drl

Data Translat

ionRule

DataGrid

Data Correcti

onRule

Data TranslationRule

On TooMany ErrorsRule

Result

Domain Specific language

Thankyou