Refactoring Toward Deeper Insight

DDD Findings In Batch Processing A Case Study

Andreas Brink, factor10

Disclaimer!!! .NET not Java

Work In Progress…

Danica Pension

• Core financial business system.

• ASP.NET / RDBMS / Integrations

• Generations of technologies / architectures.

• Lots of financial business rules.

• Lots of Batch Processing.

• Mission: Taking Control of the Code Base – DRY, Understandability, Testability, Automation

• DDD/DM + ORM fits well!

My View of the DDD Toolbox

Philosophy & Principles • Ubiquitous Language • Model Driven • Declarativity • Distilling The Model • Breaktrhough • …

Model Design & Impl. • Entity • Aggregate • Value Object • Respository • Service • Specifiction • Side-Effect Free Functions • …

Strategic Design • Bounded Contexts • Responsibilty Layers • …

Domain Model Pattern (DM)

Supple Design • Assertions • Side-effect free

Functions • Standalone Classes • Closure of Operations

Design Sweet-Spot • Understanding & Communication • Testability, Executable Specifications But, not like the Office Object Model… • Must scale • Not one single file on disk

Basic Building Blocks • Entity, • Aggregate • Value Object • Specification • Factory

Implementability

Service

Repository

• Object Navigation does not scale Repositories

• DM does not scale well with composition & coupling Services

• Problem Solved !?!?

Some Layer…

ORM

Implementation Mess

Service

Repository

• Less focus on Domain Model • Services – The Ultimate Loophole

– Touches big parts of the system – horizontally, vertically – Side Effects Understandability & Testing Problems

• Decentralized Flow Control & Data Access… Global Optimization & Generic Processing hard or impossible Performance Problems

ORM Some Scenario…

Why DDD is Hard

Service

Repository

• Model Design is hard to begin with – OO Expertise is still quite rare

• Have to be a Design Patterns / ORM / Architecture Expert

• Fail to get an Easily Consumable & Testable Model

• Same old Procedural Service Soup (+ some entities…)

ORM Some Scenario…

My DDD Challenge

• Reclaiming the Domain Model – Easy Reasoning, Consumption & Testing

• REAL Separation of Concerns – Not just a complex web of objects and method

calls behind superficially simple interfaces

• And with Batch Processing in the mix…

IS THIS POSSIBLE??

Batch in the mix…

• ”ORM is not for Batch, use the right tool…”

• DDD/ORM vs Stored Procedures

• Service Chattiness Performance Problem

• Batch becomes a domain service in itself

– Business Workflow as a mini program

– Hard to decompose/compose without Service

– I want the business rules in the Domain Model…

Billing Batch – Pseudo Code

foreach (PlanAgreement planAgreement in GetPlanAgreements())

{

Agreement agreement = GetAgreement(planAgreement.Id);

foreach (PlanCategory planCategory in GetPlanCategories(planAgreement.Id))

{

PremiumTable premiumTable = GetPremiumTable(planCategory.Id);

foreach (PlanInsurance planInsurance in GetPlanInsurances(planCategory.Id))

{

Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);

InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);

AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);

double premium = CalculatePremium(planAgreement, agreement, planCategory,

premiumTable, planInsurance, insurance);

List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);

...

...

...

}

}

}

Billing Batch – Levels

Agreement

Category

Insurance

• Agreement

• PlanAgreement

• PlanCategory

• PremiumTable

• PlanInsuranceHistory

• LatestPlanInsurance

• InsuranceHistory

• LatestInsurance

• InsuranceAccount

• AdviceHistory

• …

Misc…

• …

• …

Batch Observations

• High input/output entity ratio – 11 Entity Types as Input – Often Complex

– 2 as Output (1 Create, 1 Update) – Simple

– Simple State Semantics

– Opportunities for Caching

– (Responsibility Layers Analysis…)

• Data is centered around a few central business keys.

Potential for generalizing / streamlining the batch processing pipeline??

Billing Batch – Loops Flattened

PlanAgreement planAgreement = null;

Agreement agreement = null;

PlanCategory planCategory = null;

PremiumTable premiumTable = null;

foreach (PlanInsurance planInsurance in GetPlanInsurances()) {

if (planInsurance.PlanAgreement != planAgreement) {

planAgreement = planInsurance.PlanAgreement;

agreement = GetAgreement(planAgreement.Id);

}

if (planInsurance.PlanCategory != planCategory) {

planCategory = planInsurance.PlanCategory;

premiumTable = GetPremiumTable(planCategory.Id);

}

Insurance insurance = GetInsurance(planInsurance.InsuranceNumber);

InsuranceAccount account = GetAccount(planInsurance.InsuranceNumber);

AdviceHistory adviceHistory = GetAdviceHistory(planInsurance.InsuranceNumber);

double premium = CalculatePremium(planAgreement, agreement, planCategory,

premiumTable, planInsurance, insurance);

List<Advice> advices = CalculateBillingAdvice(adviceHistory, premium, account);

...

}

Billing Batch – Levels Flattened

• Agreement

• PlanAgreement

• PlanCategory

• PremiumTable • PlanInsuranceHistory

• LatestPlanInsurance

• InsuranceHistory

• LatestInsurance

• InsuranceAccount

• AdviceHistory

Agreement Category Insurance Misc…

• …

• …

Agreement Category Insurance Misc…

Agreement Category Insurance Misc…

Billing Batch – Entity Level Keys

• Agreement

• PlanAgreement

• PlanCategory

• PremiumTable • PlanInsuranceHistory

• LatestPlanInsurance

• InsuranceHistory

• LatestInsurance

• InsuranceAccount

• AdviceHistory

Agreement Category Insurance Misc…

• …

• …

Agreement Category Insurance Misc…

Agreement Category Insurance Misc…

Agreement Level Key

Category Level Key

Insurance Level Key

Entities

Billing Batch – Generic Cursor Style

Plan Agreement

Plan Category

Insurance History

Master Keys

Agreement Level

Category Level

Insurance Level

Agreement

Plan Insurance History

Advice History Premium Table

… …

• Cursor Semantics • A Set of Master Keys Drives the Cursor • Entities Associated with Keys in Master • Each Row Contains Entities for a Unit-Of-Work

Entity Level Keys

Level Keys

Agreement Level = 4501

Category Level = 78

Insurance Level = ”56076”

Plan Insurance

Agreement ID = 4501

Category ID = 78

ID = ”56076”

• Map of EntityLevel & Values

– Dictionary<EntityLevel, object>

• Or derived from Entity Properties

The Entity Level Abstraction

public class PlanAgreement

{

[Level(typeof(AgreementLevel), IdentityType.Full)]

public int Id;

}

class AgreementLevel : EntityLevel {}

class CategoryLevel : EntityLevel {}

class InsuranceLevel : EntityLevel {}

Entity Cursor: Master + Entities

void Initialize()

{

var cursor = EntityCursor.For(SessionFactory, MetaData);

// MASTER: IEnumerable<object[]> OR IEnumerable<TEntity>

cursor.Master = GetMyMaster();

cursor.MasterLevels(new AgreementLevel(), new InsuranceLevel());

cursor.Add(Query.For<PlanAgreement>());

// ADD MORE ENTITIES TO THE CURSOR...

while (cursor.MoveNext()) {

var currentPlanAgreement = cursor.Get<PlanAgreement>();

// PROCESS EACH ROW IN THE CURSOR...

}

}

IoC Style + Syntactic Sugar class MyBatch : BaseBatch

{

PlanAgreement planAgreement;

EntityLevel[] Levels() { return ... }

object[] Master() { return ... }

void Initialize() {

// Query Defintions that are not simple

// Query.For<MyEntity>()

Add<PlanAgreement>()

.Where(pa => pa.Foo != null);

}

void ProcessRow() {

var foo = this.planAgreement.Foo ...

// PROCESS THE ROW...

}

}

Row Processing

Master Keys Agreement Insurance

Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2

Key 2: Insurance No

(1, “InNo-1")

(1, “InNo-2")

(1, “InNo-3")

(2, “InNo-4")

(2, “InNo-5")

(3, “InNo-6")

...

(n, “InNo-n")

Master Keys Agreement Insurance

Key 1: Agreement Id ChunkSize: 2 ChunkSize: 2

Key 2: Insurance No

(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")

(1, “InNo-2") -||- Insurance(1, “InNo-2")

(1, “InNo-3") -||-

(2, “InNo-4") Agreement(2)

(2, “InNo-5") -||-

(3, “InNo-6")

...

(n, “InNo-n")

Row Processing Chunked Data Fetch

Query<Agreement>()

.Where(a => a.Id)

.IsIn(1, 2)

Query<Insurance>()

.Where ...

• Entities are fetched in Chunks

• Multiple chunk queries executed in one DB round-trip.

• NHibernate MultiCriteria (or Futures).

Master Keys Agreement Insurance

Key 1: Agreement ChunkSize: 2 ChunkSize: 2

Key 2: Insurance

(1, “InNo-1") Agreement(1) Insurance(1, “InNo-1")

(1, “InNo-2") Agreement(2) Insurance(1, “InNo-2")

(1, “InNo-3") Insurance(1, “InNo-3")

(2, “InNo-4") Insurance(2, “InNo-4")

(2, “InNo-5") Insurance(2, “InNo-5")

(3, “InNo-6") Agreement(3) Insurance(3, “InNo-6")

... ...

(n, “InNo-n")

Row Processing Indexing

• Each entity is indexed with the identifying level key(s). • Entities in chunks synced with key for current row as the

cursor proceeds forward.

Entity Grouping

class InsuranceHistory : GroupingEntity<Insurance>

{

static readonly Grouping Grouping = Grouping.For<Insurance>()

.By(i => i.AgreementId)

.By(i => i.InsuranceNumber);

public InsuranceHistory(IList<Insurance> values) { ... }

}

• Groups as a 1st class modeling concept

• Enriching the Domain Model

• “Virtual Aggregate Root” – Model Integrity

• Declarative expression (By, Where, Load)

Cursor.Add<PlanInsuranceHistory>();

Cursor.Add<PlanInsuranceHistory, PlanInsurance>()

.Where(...); // OK to override filter??

Complex Grouping – PremiumTable

• Rich Model Abstraction

• Complex data structure with lookup semantics

• No natural aggregate root

• Not cacheable in NHibernate session

• Fits well as a GroupingEntity

Value

10-22 23-30 31-45

0-20 100 120 135

20-40 110 130 150

40-65 130 160 190

Row Interval

ColumnInterval

Querying

• Filter per Entity – Cursor “Joins” using Shared Level Keys • ORM-semantics: Where, Load • Grouping Entity has query like qualities • Level Queries are statically defined query using Entity Levels

Keys to construct underlying ORM query (yes, coupling)

Conceptual API:

Cursor.Add(Query entityProvider)

Query.For<PlanInsurance>()

.Where(insurance => insurance.IsActive)

.Load(insurance => insurance.Category)

Query.For<AdviceHistory>()

Query.For(PremiumTable.ByAgreement)

.IndexBy(table => table.TableId)

Versioning

public class PlanInsurance

{

[Level(typeof(AgreementLevel), IdentityType.Partial)]

public int AgreementId;

[Level(typeof(InsuranceLevel), IdentityType.Partial)]

public string InsuranceNumber;

[VersionLevel(typeof(PlanInsurance), IdentityType.Partial)]

public int Version;

}

• Core to many business domains

• Has its own set of semantics

• Common in Groups – Latest<Insurance> vs InsuranceHistory

• Implemented in different ways in the DB

• Expressed declaratively

• Uniform Query Semantics

What About The Services? void ProcessRow()

{

...

var premiumService = new PremiumService

{

PlanAgreement = Cursor.Get<PlanAgreement>(),

PlanInsurance = Cursor.Get<PlanInsurance>(),

Insurance = Cursor.Get<Insurance>(),

Insured = Cursor.Get<Person>(),

PriceBaseAmountTable = Cursor.Get<PriceBaseAmountTable>(),

PremiumTable = Cursor.Get<PremiumTable>(),

RiskTable = Cursor.Get<RiskTable>()

};

var premium = premiumService.CalculatePremium(advicePeriod);

...

} • Service has pure calculation responsibility

• Dependencies are injected by client

• Coupling…? Boilerplate Smell…?

Conclusions

• Data Access Abstraction with Power & Ease of Use • Declarative & Composable Entity Pipeline • Minimizes DB Round-trips; Favors Eager Loading • Repositories Become Redundant • No More Unconstrained Services – “Calculators” / …??? • Richer Domain Model – Less Supporting Objects, More Domain

Related Objects • DDD/ORM + Legacy DB == True • Composite DB Key Thinking Essential to the Solution • Patching the DB Model with Entity Level Abstraction… • What’s Next? – Lots of Low Hanging Fruit… TOWARDS AN EXECUTABLE ARCHITECTURE…???

What’s Next? – Entity Injection

Cursor.Add<PremiumCalculator>();

void ProcessRow()

{

...

var calculator = Get<PremiumCalculator>();

var premium = calculator.Calculate(advicePeriod);

...

}

• Cursor can inject entity dependencies automatically

• Calculators dependencies can be inferred and added to cursor automatically

• ”Calculator” define Cursor Entities Implicitly

What’s Next? – Stateful Calculators?

class PremiumCalculator

{

...

double CalculatePremium(...) {}

...

}

• What if we treated a calculation as a stateful object? • Calculations become data flows through the system • Stateful Objects as the Uniform Expression – Simplifies

declarative programming • Captures Multiple / Intermediate Calculation Results • Can be bound to a UI • Additional state in the cursor – UI could add presentation

model/wrapper to the cursor

class PremiumCalculation

{

...

double Premium;

...

}

What’s Next? – Entity Pipeline

class BillingCalculation : EntityPipeline

{

void Initialize() {

Add<PlanAgreement>();

...

}

}

var monthlyBatch = new BillingCalculation();

monthlyBatch.Master = GetMasterForMonthlyBatch();

monthlyBatch.Persist<AdviceCalculation>(ac => ac.Advice).BatchSize(20);

monthlyBatch.Execute();

var singleInstance = new BillingCalculation();

singleInstance.Master = new object[]{ 24, "InNo-1"};

singleInstance.Persist<AdviceCalculation>(ac => ac.Advice);

singleInstance.Execute();

var nextUIPage = new BillingCalculation();

nextUIPage.Add<MyUIModel>();

nextUIPage.Master = GetMasterForNextPage();

myGrid.DataSource = nextUIPage.Select(cursor => cursor.Get<MyUIModel>())

What’s Next? – New Data Providers

• File Processing for Data Imports – Prototyped batch framework

• Document Based Persistence – Premium Table for example

• Hybrid Persistence – Serialized object graphs in SQLServer

• SOA Integrations – Loosely Coupled Bounded Contexts

• Parallel data fetch – Multiple DBs / Data Services

What’s Next? – Business Events

• Entity Processing Pipeline seems to be a good environment for triggering and/or handling business events based on persistence events.

• Poor man’s Business Events!?!?

What’s Next? – Greenfield

• Search the Core Domain/Application Semantics – Built-in Versioning from the start e.g. – Semantic Storage…

• Streamline – Uniform Expression – Semantics – Patterns

• Be Opinionted – Constraints are Liberating

• Executable Architecture

Thanks For Listening!!!

Questions?