20131028 btug.be - biztalk tracking

BTUG – BizTalk TrackingToon Vanhoutte – Integration Architect

2

Configuration

Under the hood

Best Practices

Demo Visualization

Tools

Agenda

3

SetupDemo

Specify unique reference for messages #search

Specify simulated response times of backend systems #sla

Specify customer and transporter names #inspiration

4

Configuration

5

| Port Tracking Configuration

Configuration

Only track what’s really required #performance

6

| Pipeline Tracking Configuration

Configuration

Bug BTS2010 CU2 CU5 #solved #CU6 #kb2728652

7

| Orchestration Tracking Configuration

Configuration

Avoid Shape start and end #performance #orchestrationdebug

8

| Property Schema Tracking Configuration

Configuration

Make context properties searchable #troubleshooting

Global + specific property schema #generic #consistent

9

| Disable Global BizTalk Tracking

Configuration

10

Under the hood

11

| BizTalk Database Structure

Under the hood

11

SQL Agent

Host Instance processes message

12

| BizTalk Database Structure

Under the hood

12

SQL Agent

Host Instance processes message

Message stored in MessageBox

13

| BizTalk Database Structure

Under the hood

13

SQL Agent

Host Instance processes message

Message stored in MessageBox

TDDS copies events to DTA + BAM

14

| BizTalk Database Structure

Under the hood

14

SQL Agent

Host Instance processes message

Message stored in MessageBox

TDDS copies events to DTA + BAM

SQL Agent copies bodies to DTA

15

| Store messages in MessageBox database| Physical runtime message bodies are stored in:

| Spool| MessageParts| Parts| Fragments

Under the hood

15

Fragmentation controlled by large message size

16

| Process messages in MessageBox database| Per Host, these tables are created:

| HostNameQ: Main queue: each host instance polls this queue table to collect messages that are pending processing

| HostNameQ_Suspended: Suspended queue: queue table that holds supended messages

| HostNameQ_Scheduled: Scheduled queue: not used| InstanceStateMessageReferences_Host: State queue:

holds list of messages that have been processed but are required later

Under the hood

16

Stores only reference to physical message in Spool table

17

| Store messages in MessageBox database

Under the hood

17

#probts2006

18

| Store tracking data in MessageBox database| Host Instance serializes data and stores it in the

MessageBox| Message events:

| TrackingData_0_x [x: 0 - 3]| TrackingData_1_x [x: 0 - 3]

| Message bodies:| Spool| MessageParts| Parts| Fragments

Under the hood

18

#probts2006

19

| Copy Events from MessageBox to DTA database

| Tracked Message Events copied by TDDS Service:| Tracking Data Decode Service| Runs in each host instance with tracking enabled

Under the hood

19

Create a dedicated tracking host

20

| Copy Events from MessageBox to DTA database

Under the hood

20

BizTalk MessageBoxTrackingData_0_0TrackingData_0_1TrackingData_0_2TrackingData_0_3

TrackingData_1_0TrackingData_1_1TrackingData_1_2TrackingData_1_3

BAM Primary ImportActivity_ActiveActivity_ActiveRelationshipsActivity_CompletedActivity_CompletedRelantionshipsActivity_Continuations

BizTalk DTA DatabaseDta_MessageInOutEventsDta_ServiceInstances...

TDDS

21

| Copy Events from MessageBox to DTA database

Under the hood

21

BizTalk MessageBoxTrackingData_x_x

BAM Primary ImportTDDS_StreamStatus TDDS_FailedTrackingData

BizTalk DTA DatabaseTDDS_StreamStatus TDDS_FailedTrackingData

TDDS

TDDS_StreamStatus contains last copied sequence n°

TDDS moves events with > sequence n° from TrackingData_x_x to DTA

TDDS_FailedTrackingData contains tracking failures

22

| Copy Bodies from MessageBox to DTA database

| Tracked Message Bodies copied by SQL Agent Job:| TrackedMessages_Copy_BizTalkMsgBoxDb| Scheduled by default each minute (development: 10

seconds)

Under the hood

22

Monitor the SQL Agent Jobs #SCOM #blog

23

| Copy Bodies from MessageBox to DTA database

Under the hood

23

BizTalk MessageBoxSpoolMessagePartsPartsFragments

BizTalk DTA DatabaseTracking_Spoolx Tracking_Partsx Tracking_Fragmentsx

SQLAgent

[x: 1 - 2]:Always one online / one offline

24

| Remove data from MessageBox| This is performed by SQL Agent Jobs:

| MessageBox_Message_ManageRefCountLog_BizTalkMsgBoxDb:| Runs continuously (due to former SQL limitation of 1 min job

schedule)| This jobs checks weather a message is still referenced by instances

| MessageBox_Message_Cleanup_BizTalkMsgBoxDb| Is disabled (keep it like this)| It’s called from with the infinite loop of the ManageRefCount Job| This job deletes physical messages from MessageBox that have no

outstanding references anymore

Under the hood

24

25

| Remove data from DTA| Enable + Configure the DTA Purge and Archive Job

#avoidlargeDTA

| Remove data from BAM| SQL Agent Job BAM_DM_Activity moves data to BAMArchive

Under the hood

25

26

| BizTalkDTADb Database model| No foreign keys #performance

| Designed for fast insert| Results in slower query

| No single chain ID| Difficult to reproduce the “route” within BizTalk

Under the hood

26

27

| BizTalkDTADb Database model| No foreign keys #performance

| Designed for fast insert| Results in slower query

| No single chain ID| Difficult to reproduce the route within BizTalk

Under the hood

27

| Create a chain of messages

Under the hood

29

| Create a chain of messages

Under the hood

29

SI = 1MI = A

SI = 1MI = B

SI = 2MI = B

SI = 2MI = C

SI = 3MI = C

SI = 3MI = D

dta_MessageInOutEvents Service Instance ID Message Instance ID

30

Tools & SDK

30

31

| BizTalk Administration Console #technical #notsexy

Tools & SDK

31

32

| BizTalk MessageBox Viewer #troubleshooting

Tools & SDK

32

33

| BizTalk Terminator #troubleshooting

Tools & SDK

33

Import MBV scripts

34

| Performance Counters #troubleshooting| Throttling state find reason on MSDN| TDDS

Tools & SDK

34

35

| Retrieve tracked message body #3ways| Microsoft.BizTalk.Operations.dll

Tools & SDK

35

36

Visualization

36

37

Visualization

37

| Bad practice| Develop a dashboard directly on BizTalkDTADb

#performance| No control on user queries| Impacts directly the BizTalk runtime| Bad dashboard performance

38

Visualization

38

| Bad practice| Develop custom tracking mechanism

39

Visualization

39

| Good practice| Use default BizTalk Tracking #CID

40

Visualization

40

| Good practice| Use BizTalk Tracking #CID

41

Visualization

41

| Business Activity Monitoring? #notideal| Development impact

| Tracking Profile Editor is limited| BAM API is rather complex

| SQL Server Analysis Services| 2008 R2 required, when using SQL 2012

| Deprecated BAM portal| Impact on operations| Not ideal OLAP cube structure| Termination of activities not straightforward

42

Visualization

42

| Use case #1: Message Search| BizTalk Administration Console

| Codit Integration Dashboard

43

Visualization

43

| Use case #1: Message Search| Search on context properties:

| Application| Flow| Sender| Receiver| Reference

Demo

44

Visualization

44

| Use case #1: Message Search

Demo

45

Visualization

45

| Use case #2: Operational reports | Message Load| Message Size| Database Size

#managedservices

46

Visualization

46

| Use case #3: Pro-Active Monitoring| Custom SCOM alert: monitoring non-events #vbs

| E.g.: Just-In-Time ordering| Daily check on expected amount of received orders| Query on BizTAlkDTADb| Automated email alert / support ticket

#managedservices

47

Visualization

47

| Use case #4: SLA Reporting| Processing times of web services

| Total processing time| Processing time of every backend system| Processing time in BizTalk

48

Visualization

48

| Use case #4: SLA Reporting

Demo

rp_MasterDataService.WCF

sp_CRM_GetCustomer.WCF

sp_SAP_GetItem.WCF

sp_AX_GetEmployee.WCF

ORCH

GetCustomerGetCustomerItemList

GetItemGetEmployee

49

Visualization

49

| Use case #4: SLA Reporting

Demo

rp_MasterDataService.WCF sp_SAP_GetItem.WCF

Tota

l Pro

cess

ing

Tim

e SAP Processing Time

BizTalk Processing Time = Total Processing Time – SAP Processing Time

1 2 3 4

8 7 6 5

Same MessageID

Same MessageID

50

Visualization

50

| Use case #4: SLA Reporting| SQL Agent Job

| BizTalkDTADB Custom database #chaining

Demo

51

Visualization

51

| Use case #4: SLA Reporting

Demo

52

Visualization

52

| Use case #4: SLA Reporting

Demo

53

Visualization

53

| Use case #5: Process Overview| Visualization of related integration flows

54

Visualization

54

| Use case #5: Process Overview| Example: EDI Order Processing

Demo

55

Visualization

55

| Use case #5: Process Overview

Demo

56

Visualization

56

| Use case #6: Dashboards| BizTalk 360: Operational focus| AIMS for BizTalk: Behavior / trend analysis| Codit Integration Dashboard: Advanced message

handling| Minotaur| Frends Helium| Moesion| …

57

Visualization

57

| Use case #6: Dashboards| Codit Integration Dashboard

| Friendly Naming of technical artifacts| Message Search| Message Resume / Resubmit

| Editing| Multi selection

Demo

58

Visualization

58

| Use case #6: Dashboards| Codit Integration Dashboard: resubmit

Demo

59

Questions?

59

You

BTUG

Microsoft