use brms for data quality in your bigdata pileline
TRANSCRIPT
Session title
*Red Hat & Dell Confidential
Use RED HAT BRMS For DataQuality In Your BigData PileLine
Anurag SaranSr Solution Architect, Red Hat Financial ServicesEmail: [email protected]: @anuragsaran
AGENDA
● Data Quality Using BRMS With Demo○ Using Sparse Decision Table○ Using Consolidated Decision Table○ Using Decision Table To Invoke Relevant Rules○ Using Decision Table To Invoke Relevant Rules + Data
Lookup/Translation + Data Correction○ Using Domain Specific Language
HOW TO AUTHOR RULES AS AN ANALYST?WEB DECISION TABLEDesign and/or Edit decision tables in the web editor
EXCEL SPREADSHEETAnnotate an excel spreadsheet with rule logic, then execute in the embedded app, or service.
WEB GUIDED RULE A web based interface will allow you to create rules, while reflecting on the data model you uploaded or designed through the data modeler.
WEB DOMAIN SPECIFIC LANGUAGEUsing a mapping between a domain specific natural language and technical syntax - you can compose rules with conditions and consequences, entering data into placeholders.
HOW TO AUTHOR RULES AS A DEVELOPER?DRL EDITORThe drools syntax is Java like. The when clause may be formed with Java or MVEL syntax. All rules formats get translated based to DRL.
EXCEL SPREADSHEETAnnotate an excel spreadsheet with rule logic, then execute in the embedded app, or service.
DOMAIN SPECIFIC LANGUAGE MAPPINGUsing a mapping between a domain specific natural language and technical syntax. The eclipse tooling includes both a DSL translation and DSL rule editor.
DOMAIN SPECIFIC LANGUAGE RULEUsing a mapping between a domain specific natural language and technical syntax. The eclipse tooling includes both a DSL translation and DSL rule editor.
Using Decision Table To Invoke Relevant Validation Rules
RecordObjrecord_id = Rec001offer_id = 001offer_type_cd = 09Brate = 55payment_method_type_cd = CASH
FieldRestrictions:fieldname: record_idvalue: Rec001Pattern: R*LengthMin: 10Lenghtmax:20
FieldRestrictions:fieldname: ratevalue: 55RateMin: 10RateMax:22
FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn: CASH, DEBIT
FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern: [0-9][A-Z][A-Z]
LenghtRule
PatternRule
ListRule
RangeRule
CrossDataRefRule
ListRule
Result
FieldRestrictions:fieldname: record_idvalue: Rec001Pattern:LengthMin: Lenghtmax:
FieldRestrictions:fieldname: ratevalue: 55RateMin: RateMax:
FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn:
FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern:
createRestrictions.drl populateRestrictions.xls validateFields.drl
Using Decision Table To Invoke Relevant Validation Rules +Data Lookup/Translation +Data Correction
RecordObjrecord_id = Rec001offer_id = 001offer_type_cd = 09Brate = 55payment_method_type_cd = CASH
FieldRestrictions:fieldname: record_idvalue: Rec001Pattern: R*LengthMin: 10Lenghtmax:20
FieldRestrictions:fieldname: ratevalue: 55RateMin: 10RateMax:22
FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn: CASH, DEBIT
FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern: [0-9][A-Z][A-Z]
LenghtRule
PatternRule
ListRule
RangeRule
CrossDataRefRule
ListRule
Result
FieldRestrictions:fieldname: record_idvalue: Rec001Pattern:LengthMin: Lenghtmax:
FieldRestrictions:fieldname: ratevalue: 55RateMin: RateMax:
FieldRestrictions:fieldname: Payment_type_codevalue: CASHListIn:
FieldRestrictions:fieldname: Offer_type_Codevalue = 09BPattern:
createRestrictions.drl populateRestrictions.xls validateFields.drl
Data Translat
ionRule
DataGrid
Data Correcti
onRule
Data TranslationRule
On TooMany ErrorsRule
Result