What is SOA

8

(Module 1: Getting a feeling for SOA)

Logosworld SOA Camp OverviewLorum Ipsum Dolor DelendatSed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo.

Nemo enim ipsam voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui dolorem ipsum, quia dolor sit, amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt, ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel illum, qui dolorem eum fugiat, quo voluptas nulla pariatur?

[33] At vero eos et accusamus et iusto odio dignissimos ducimus, qui blanditiis praesentium voluptatum deleniti atque corrupti, quos dolores et quas molestias excepturi sint, obcaecati cupiditate non provident, similique sunt in culpa, qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio, cumque nihil impedit, quo minus id, quod maxime placeat, facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet, ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.

know the key terminology

know the core technologies used in SOA

have a clear view how SOA governance takes place

be able to judge costs, risks and benefits of proceeding with SOA.

GoalThe goal of the SOA Camp is to enable the IT savvy to understand in depth the new trend topic of the Service Oriented Architecture and the correlated prominent use case Web 2.0 and Enterprise 2.0.

In the end of the course the active participants will

understand the concept of SOA,

know the key terminology

know the core technologies used in SOA

have a clear view how SOA governance takes place

be able to judge costs, risks and benefits of proceeding with SOA.

Visual SOA: Practical Training ConceptTell me and I shall forget it; Show it me and I may remember it.The concept of this training is based on seeing it working. The key driver is Example-First. Before diving into the theory of SOA and Web 2.0 we shall build a small real life demo application on top of a virtual enterprise to demonstrate the difference of the SOA paradigm versus the traditional enterprise computing approach. The examples will make use of a mash-up of SAP, Microsoft and Open Source technologies spiced up with some other, prominent SOA products. For SAP addicts this will bring an additional benefit in understanding how SAP process Infrastructure (PI) works and how competing technologies seamlessly cog into each other.

Our Vision for this CourseDruing this course we want to build a small, working prototyping application that allows vsiualizing the core matters of SOA. We want to start with a small communication scenario between two applications, then add more applications via different channels and finally end up in a loosly coupled web of several computers, something which is known as computing cloud and stands firm as the basis for planning Ultra Large Systems.Module 1: Getting a feeling for SOADuring this first day we want you to walk through the SOA City and understand a little bit more how a world of SOA looks like, what you can achieve with SOA and how some basic principles work.

What is SOASOA is a new paradigm in computing architecture that will build an underlying common infrastructure that allows breaking applications into reusable and recombinable components. In the end we shall have a large society of small service components that are loosly coupled and temporarily interact with each other. The ultimate goal of SOA is to obfuscate the distinction between individual computers and let the whole internet appear as one single computer as the base ground of building Ultra Large Systems.

Looking back on history of sciene we soon realize that SOA follows the same solution path of modularization that other industry sectors have undergone over time: weaving in the 18th century, automotive in the 1970ties, electronics in the 1980ties or workstation computing in the 1990ties. SOA will make enterprise computing far easier to compared to the steep hybrid architectures of nowadays all-in-one solutions in terms of building it, controlling its behaviour, maintaining operation and continous evolution and enhancement.

SOA is a common term for many sub-topics like application independent frontend design, message oriented communication, Middleware and Enterprise Service Bus, Virtualization, Webservices, Software as a Service (SaaS) and more.

Hurray! It said Mama! An Excursion in SOA Terminology

First the words confuse and then the things.

Since we are all good scientists and knowledge engineers we shall start our learning sessions with an exercise in naming the things and disambiguate the meaning of extensively used expression.

There is two fairly interesting observation to be made about SOA terms: most of the terms are as old as the mid 1960ties and there are often several synonyms for the same entity.

SOA Service Oriented ArchitectureSynononyms:

COA Component Oriented Architure;

CDA Component Driven Architecture

EDA Event Driven Architecture

MDA Message Driven Architecture

The synonyms indicate the different viewpoints on SOA. COA and CDA are inter-changable and put the focus on breaking software into components to shape the architectural landscape in the consequence. EDA and MDA focus more on the communication level.

Definition

SOA is an architecture concept that tries making it as attractive as possible for a software vendor to expose its atomic functionalities as components and services in a way that other software can easily consume the service at discretion.

The sector of enterprise computing is now the last area where we still have a dominance of ALL-in-One application suites. All-in-One is attractive for a software vendor but a honey trap for the customer. For the software marketeer it is attractive, since he can easily sell his whole barrow while the customer asked for a small part of it only. Since it is difficult to mix and match the software packages of different makes it is normally not economical for a customer to run several makers products. And this is the honey trap: it drags the buyer of a suite slowly into a dependency, where he cannot escape easily.

The SOA Paradigm PrinicplesSOA is much more than architecture

Despite the name, SOA is definitely more than just architecture. SOA is a new paradigm, a new way of thinking. Traditional programming favoured a dictatorial top-down approach where a single process tried controlling other processes. In SOA you refrain from coammnding other processes and rather ask them to render a service. The core difference is essential: in classical development the caller of a service needed to tell who the service is executed; in SOA the service provider decides how to produce the result.

SOA favors Darwinistic governance

SOA transforms IT landscapes from autocratic rulings into democratic governance, where market-oriented and the Darwinistic principle and strategies of the Survival of the Fittest make the tone. Once you have thousands or illions of services actively running in an Ultra large Web there is now way to control their genesis nor their evolution or disappearence.

SOA favors collaboration

SOA is a collaboration infrastructure that does not only allow for collaboration but encourages the latter by making it easier to reuse common components over the web instead of building isolated silo solutions.

SOA implies agility

Agility to change business processes in real-time is one of the core visions of a Service Oriented Architecture. The infrastructure must provide tools to guarantee stability of the existing world while components are in permanent transformation.

SOA is not bottom-up nor top-down

SOA is neither bottom-up nor top-down but a healthy blend of both. It will be the duty of the management to drive the vision and to orchestrate and mediate the daily changing situation

SOA fails by too much governance

Plans are the natural enemy of any SOA. While you can use some standard template to implement the most wanted and needed services you cannot control the growth of a SOA without risking throttling its further evolution. Like democracy cannot be ordered by a dictating ruler, SOA cannot be ordered from the top. SOA lives and dies with the participation and non-participation of the community members.

SOA has no purpose on its own

SOA has no purpose on itself but to assist the software implementation teams to build and transform business process quickly.

SOA requires developer satisfaction

SOA lives from the agility to build and change processes. Hence, developer satisfaction is crucial and requires easy to use development tools and services that satisfies their needs and their mind.

SOA creates new and better business models

A collaboration market place as provided by SOA opens up for completely new opportunities to do business. Information and commodities can be exchanged much quicker than today. Associative marketing like the one those-who-have-bought-A-have also-bought-B bring a completely different way of waltzing a customer. SOA also brings the agility to build and change processes. Many an effort wasted in decision making processes will void in future.

SOA ChallengesSOA Challenge 1: Making Computers talk with each other

This topic used to be here long before SOA became popular. There were several names for it like

Enterprise Application Integration

Application-to-Application (A2A)

Business-to-Business (B2B)

There are basically two methods that allow communication between computers.

Calling a program remotely

Exchange data documents

Calling programs remotelyProtocol API

SAP: librfc32.dll

Proxy-Communication

A proxy is a program that plays the broker between two communication partners.

Exchange documents between computersFile exchange Managed Data Transfer

Exchnaging data via file systems is one of the most wanted services. Unfortunately the data transfer possibilities are fairly limited in nowadays world and mostly dominated by the limitations of the underlying network and file system.

What users really need are managed file transfer services that work as a post master who receives a file stream and delivers it to a destination. Problems like an interrupted network connection, file write errors, missing authorization should be handled conveniently by a managed service.

Message Queues

Message Queues are the heart of any distributed computing environment. When a non-deterministic number of loosely coupled tasks and services start cooperating, there is an immediate need to store and serialize data temporarily.

Database Exchange

Database exchange is a very delicate issue. Reading database formats and converting them between each others is fairly easy but not satisfying at. Since the IT world made the fatal decision to base all the critical developments on to of relational database systems there are more mechanisms necessary to compile the logical compounds out of the individual tables. Generally this requires the assistance of the application that controls the database tables.

Database exchange is a good example where SOA can help a better structuring of data stores in future. Since a SOA requires breaking down applications to their atomic component level, every application needs first to define the semantic data capsules before it can use them. Once the capsules exist for the core applications they can also be accessed by peripheral uses.

Multichannel architecture

SOA is based on a strict abidance of the OSI 7-Layer structuring. Although it is still heart to bash the discipline into the heads of modern time programmers, the seperation of layers will quickly lead into a dramatic simplification of data flows and transactional processing.

Instead of working with many different data sources like local and network files, databases, memory blocks, manual data entry etc. an application will only work with universal channels that stream data in and out.

Adapters and Protocol Converters

Adapters will translate any physical data format into a universal data stream. Protocols will be seen like envelopes to data and every application will rely on protocol converters to wrap and un-wrap the data for transport between services.

Proxy-Communication Managed Execution

Proxies play an essential role in SOA. In fact not every single process step in a SOA environment will be executed via a dedicated proxy. A proxy (or broker) in its general form is a service that receives a requests and initiates an appropriate processing; eventually it pipes the resulting data to the outbound channel as response.

SOA Challenge 2: Common Look and Feel and MobilesThere is more than a Windows computer. There are Macs, there are Linux PCs; there are several browsers now out there, besides the Internet Explorer there are Opera, Firefox, Safari and maybe more. And there are Mobile devices like the Windows Mobile and the Apple iPhone. But also other computers are trying to access another computer via the same protocol channel.Coping with multiple browsers

Protocols for different needs

HTTP

HTTP is a synchronous request-reply protocol. This means that every message sent will wait for a reply that describes the status of the delivery.

One request One Response

Open Connection

Send Request

Wait for Reply

Close Connection

### We need a small demo here, maybe using HTTP analyzer

Request via HTTP:GET /ftp/public/ HTTP/1.1Accept: */*Accept-Language: deAccept-Encoding: gzip, deflateUser-Agent: Mozilla/4.0 Host: logosworld.comConnection: Keep-AliveResponse from HTTPHTTP/1.1 200 OKDate: Thu, 14 Aug 2008 13:22:05 GMTServer: Apache/1.3.34 Ben-SSL/1.55Keep-Alive: timeout=2, max=200Connection: Keep-AliveTransfer-Encoding: chunkedContent-Type: text/html

FTPSMTPSecure Variations: HTTPS, SFTPSOA Challenge: Traffic Congestion and bad routes

There are situations where your server suddenly receives more requests than you are able to process. You might now shut the gates. But this is no solution. The requests should actually be not lost. Since you cannot expect all the senders being prepared for bouncing messages, there should be an alternative solution.

Message Queues

Messages queues are highly optimized databases. There main purpose is to buffer messages and allow them to redistribute in the same sequence as it had been received: First-in-First-Out.

Quality of Service

There are situations when information is transferred across slow and unreliable connections. Imagine a builder company in the middle of the Amazonas forest. When they need access to their drawings and have to make updates it might be nearly impossible to transfer with reliability a several Megabyte large file over a wireless satellite phone connection. There are specialized services that cn guarantee that even large files are transferred with reliability.

SOA Challenge: Message delivery servicesPostman as messengerPostman as message storeSubscription Services

Imagine that you have a site where you permanently create new documents. There may be many people out there in the world who whould like to know if something new appears that matches their interest. Instead of alerting your friends or subscribers personally, we look for a service that takes care of organizing the distribution. In real life you may hire a marketing agency, that has not only a great distribution organisation but may also have the right addresses for you and can review and filter the replies on your behalf.

Example: Drupal Newsletter Alert

A good example for such a service is delivered the now ubiquitous newsletter distrtibution. Imagine how complicated it is to take care of registrations

SOA Challenge: Simple Automation

SOA Challenge: On Demand Computing

Imagine a highly seasonal business, e.g. selling wooden traditional nutcrackers. They are sold mainly throughout the months before Christmas with a peek shortly before Holy eve. A bookstore faces a similar phenomenon in that 50% of the turnover is accumulated within the single month of December.

Today there are so many computers all over the world. While some applications are suffering a momentary shortage of capacity, there are many others that have a surplus in computing power at the same time.

A service oriented design allows a flexible assignment of computing power. An application will be loaded on as many computers as needed for the moment.

### Drawing that demonstrates how an array of computer is assigned to applications at different time.

Morning: SAP1-SAP5 Logistic Execution SAP6 Sales and FI

Afternoon: SAP1-SAP2 Logistic Execution SAP4-SAP6 Sales and FI, SAP3 Development

SOA Challenge: Computing Clouds

Computing clouds are a true vision that will come true when the transistion to SOA has proceeded far enough. A computing clouds is a network of computers that share the work load with each other.

In a cloud the requesting service will not know which computer is foing to execute the delivering service.

Examples:Timestamps

E.g. If your application needs to retrieve a timestamp, it will not call the unreliable system time but it may call the central atomic time over the internet.

Calculate Road Tolls

A more realistic example would be the occasional calculation of road tolls for street transports. Instead of maintaining a database with complex algorithms that calculate the tariff, it would be much cheaper to let some web service calculate the value. Since such a calculator will be visited by many users, the costs per request will be much lower than an own in-house solution.

Consult Expert Systems

In a more and more specilized industry there are areas where the inhouse know-how is far not enough to fulfil a required task or they are so expensive to setup, that it is more rentable to consult an expert system that has been designed for a special work. Production planning systems beong to such systems; picture analysis software in life sciences are other areas. Virus and identity checkers are also expert systems. The demand is to allow easily transfer the necessary information as inoput for expert systems to retrieve a quick and affordable result.

This is another example that we know from our city lives. If we need a new suit, we consult a tailor. If we need a new bed, we call for a carpenter. If we want a good piece of bread, we go to the baker. And we do not want to see how a cow is slaughtered, so we better leave this to the butcher. We consume the indulgence of convenience services by leaving them to the experts.

SOA LiveSOA Examples that we find in the WWW

We shall start with having a look how SOA already helps us enhancing the work of IT.

Internet as the SOA for Webservices

In this beginning we shall show the use of webservices in enhancing real applications and how the internet architecture assists the use of such services.

Playing with AmazonAmazon is the web service pioneer

Amazon is one of the real pioneers in practical web service enablement. No wonder that we shall use Amazon to demonstrate how easy it can be to make program calls to retrieve and post information across the internet.

Simple REST queryWeb Service descriptionsAMAZON WSDLhttp://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl?

AMAZON Webservice Queryhttp://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=[your personal access key ID]&Operation=ItemSearch&SearchIndex=Books&Author=Axel%20Angeli&Title=&Version=2008-08-19

Mind that you need first specify a personal AccessKeyID that you can get from aws.amazon.com upon registration.

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=0RQS425M2GV9QMCE6NG2&Operation=ItemSearch&SearchIndex=Books&Author=Axel%20Angeli&Title=&Version=2008-08-19

Generate Code from WSDLService that are not webservices

SOA does not necessarily mean web services. A service is any kind of program that allows the retrieval or posting of information across the limits of a single program.

DCOM and IntellisenceSMTP&POP: Sender&receiver exampleModule 2: Writing your own web servicesCreating web services yourself is fairly simple, at least if you have the right toolboxes and development environments at hand. In this first example we shall use Visual Studio as the elementary component and see how we can integrate components of arbitrary legacy like SAP or AMAZON.

Writing a simple backend applicationMaking call-offs from an existing contract

Your management has decided that from now on your company will interact with the external field service engineers through a web-based self-service kiosk. For that purpose a framework contract for each contracting party is created.

F:\BLUEBOOK\Dokumente\Visual Studio 2005\Projects\SimpleTimeSheetService01\SimpleTimeSheetService01\2008-08-06_0726.swf

Writing a backend test proxyWrite request to DB and acknowledgeWriting a simple frontendAgentID, ServiceID, Date, Start, EndModule 3: Test First Paradigm: Pig Runs and crash test dummiesMonitoring and Control DashboardsA dashboard is a utility that lets us easily define and manipulate a test scenario and view the result of the process execution.Monitoring SubjectsProcess RuntimeExceptions

Dashboard PlugsException restartExercise: Simple HTTP Client

Create a simple HTTP client in your preferred programming environment. The plugin should allow you entering an URL and a payload dynamically, then calling the given URL and displaying the result.

Crash Test DummyCrash test dummies are known from the automobile test. They are used to see what effect a certain predictable accident has in reality. For our testing scenarios a crash test dummy is a message that is designed deliberately false.Pig RunPig runs are patterns that simulate a flow without carrying information from end to end. In fact compares to a transport without payload. Such transports are done in real life every time when a route is new or it is unknown if the route is free of barriers. Before a life transport with a genuine payload is sent over the dedicated route an empty dummy vehicle (the pig) is fired across the parcours. Exercises: Sample PigsMessage Drivethrough (Null-proxy, Message-Forwarder)

This is the most simple of all patterns. The system receives a message and sends a message to itself. No payload manipulation takes place here. The message is taken from a data source and stored in a second data source on the same system.

Exercise 1:

Using PI read a message from a file with fixed file name and store it in another file.

Exercise 2: Multichannel test

Data sources are data channels only, so they must not be part of the process definition. In this example we want to change the datasource from case to case to see if different channels are treated equally well.

Message Loopback

A special but extremely important variation of a message passthrough is sending the message to the address of provenance. It allows both easily testing the message flow without knowing the target system and verifying that a message has not been tampered with during a transport.

Parallel executionExercise Pig run with notification:

o a pig run and every time the message is send on its way an email is sent out to a fixed address

Exercise: Send message to multiple destinations and waitr for reply

Send out messages to two destinations and then wait until all have replied, then send out a message with the results.

Exercise: Monitor behaviour of abov

Some scenario but this time monitor the replies for time-outs.

Service health checkTerminologyFighting Babylonian Cacophony

EAI projects are communication projects. This means communication between computers, but also between people and between organizations. But there is still no commonly known and understood language and terminology. This is what makes EAI projects so delicate. Many expressions go around without a clear understanding, and many terms are coined every day, just to disappear a few weeks later.

This Babylonia of terminology is a fate shared by every complex project (and which project isnt complex?). The path to overcome it is just as old as science: start to be clear on terminology before starting a design. So we will abide by this virtue, and start to sort the most common expressions used.

Message Oriented Middleware

For computers to communicate with each other, the developer of the programs has to know about the protocol and the peculiarities of the communication partner. This leads to a simple estimate; if you have four software components that communicate, you have to maintain twelve (four times three; formula: n * (n-1)) communication channels. This can be avoided if every program communicates via an interpreter program, commonly called middleware. This middleware exposes an interface for every participating program, and reroutes the data to its final receiver. Hence, middleware allows every program to communicate in its native language only, and the hassle of protocol conversion is delegated entirely to the single software component, the middleware.

Middleware can do more than this. It can provide public services that are needed by all participating software, such as sending out information mail, watching over secure delivery, encrypting data sent over a channel, and more generalized triggering and monitoring of other workflow activity.

Communication between disparate systems is done through messages. Message-based communication is the opposite of imperative communication. The latter would have to know exactly the execution plan to trigger a certain action on the remote computer, while a message-driven communication puts the decision on the execution plan, where it would naturally belong: the computer then executes the service. In addition to that, the executing computer can also decide whether to execute a command or not, and it can monitor the execution of every command, because monitoring is not done within the called service, but by the instantiater.***Axel/Lynton: I cannot find the word instantiater in SAP or in the dictionary. Is there another word you can use here to convey your meaning?cc ###Cheryl: I think the word instantiater comes out of pure programming dialect / jargon Perhaps we could put instantiater in quotes (instantiater) or I suppose we could use requestor or invoker instead? The meaning is: A service creates an instance from a class template. So the analogy is that the object that creates an instance is an instantiator;.initiator would work as well, but maybe invoker would be right one in this context.AXX More sophisticated, but very important tasks, are the mediation between the systems. This might include conflicts in locking data components, or concurrent updates.

But all the theory about communication about messaging, message flow, and harmonizing the protocols is void unless there is something on the terminal points of a communication channel that performs a meaningful and useful action. The problem is, of course, not a lack of useful functionality of the participating software, but that this functionality is not easily (or not at all) accessible from an external requester. The software needs to expose its functionality as a public service. We need a service oriented architecture.

SOA - Service Oriented Architecture

A SOA requires every participating software component within a network to expose its business functionality in a way that allows any action of a business process to be automated, and initiated from an external requester. SOA is also known as ESA - Enterprise Services Architecture.

The idea behind SOA is to break down the business functionality modeled in an ERP system, to atomic entities, that can be executed from anywhere.

What is the motivation? The purpose of an ERP system is to model certain business activities, like creating or changing a sales order, a delivery, some master data, etc. Traditionally those activities are proprietary tasks of the individual software. Think of the ways to create a sales order in SAP R/3. Before the introduction of BAPIs you could help yourself by simulating the manual data entry, via a screen through a batch input mechanism. Although this is a smart solution, it has its limitations.

Thinking of the SAP sales order transaction VA01, you may think of atomic activities like creating a header, sales items, delivery schedules, price items, variant configuration settings, etc. The idea is now to create small, mini transactions that allow changing those atomic entities, of course, with the possibilities to allow a proper rollback of those actions.

Such atomic entities are called services, and an ERP architecture that consequently implements a complete set of services for a business model is called an SOA.

The web service architecture is quite simple actuallyinstead of the traditional 3-tier architecture we now add in a 4th tier the Service Tier. This is depicted in the following diagram.

Figure 5: Four-tier Architecture of a web service Architecture

Lets take a quick look at a simple scenario of a client asking a Loan Application Service to determine whether a specific customer is loan-worthy or not.

Behind the scenes the Loan Application Service (written in .NET) accesses a Credit Rating Service (written in ABAP), to determine the clients credit rating. Lets take a look:

FUNCTION z_credit_rating.

*"---------------------------------------------

*"*"Local interface:

*" IMPORTING

*" REFERENCE(CUSTNO) TYPE STRING

*" EXPORTING

*" REFERENCE(CREDIT_OK) TYPE CHAR01

*"---------------------------------------------

DATA: profile TYPE REF TO zcl_credit_profile.

CREATE OBJECT profile

EXPORTING customer_number = custno.

CALL METHOD profile->check_credit_rating

IMPORTING

credit_ok = credit_ok.

ENDFUNCTION.

namespace LoanApplication {

public class LoanApplicationService :

System.Web.Services {

public LoanApplicationService {

Initialize( ) ;

}

[WebMethod]

public boolean ApplyForLoan (string custno) {

Loan l = new Loan ( );

return l.ApplyForLoan (new Client (custno));

}

}

}

Figure 6: A Simple Web Service Scenario

So we can immediately see that one of the big bonuses of using web services is that you dont need to worry about the implementation of the underlying service code. For all we know, any one orchestrated web service scenario could involve languages like Java, C#, ABAP (and possibly many more).This allows you to utilize your current resources to their full potential, without having to necessarily re-skill, in any particular area.

Although an SOA already defines an orderly way to produce open and collaborative systems, the degree of freedom to implement such architecture in reality, is infinite. Just think of the possibilities to define a communication protocol to exchange the data between systems, imagine one being SAP R/3 and the other one Oracle or Microsoft EXCEL. To overcome these ambiguities, there are activities that define a common set of protocol standards for the communication between services. One such activity that builds ontology for Business Process Modeling, is the open source movements around WSDL (web service Description language) and BPML (Business Process Modeling Language BPMI.org).

To summarize, SOA is an architecture that exposes business functionality to external requesters. The Enterprise Service Architecture (ESA), SOA, and ESB, are all pretty much aiming for the same goal. That goal is to deliver messages between systems, which in turn will provide for business services they can execute in a certain sequence, and manner.

ESB-Enterprise Service Bus

An ESB forms the backbone of an event-driven, highly-distributed enterprise. It provides the necessary fabric that connects a large number of applications under one EAI umbrella. The loosely coupled nature of the ESB promotes agility in the enterprise, and allows IT staff to spend more time on the real business issues, rather than hard-wiring numerous applications together.

There are two different technologies to implement service architecture:

Central orchestration

Bus arbitration

There are star-like and bus-like network architectures. The central message queue architecture requires that all messages be sent to a central message hub (the message queue), that then decides how the message is routed and processed. Such central architecture allows monitoring and filtering every activity within the SOA. Typical representatives of such a star-like architecture are SAP XI or IBM Websphere/MQ.

A bus-like architecture follows a more democratic philosophy. Messages are all put on a commuting belt, the bus, that is connected to every computer. A service running on one of those computers can now decide whether it is interested in processing the message. That does not discriminate the advantages of a central message hub: one of the services attached to the bus will be the traditional message queue arbiter that monitors and mediates all messages, and delivers messages to computers that do not have their own service monitor to pick messages from the bus. In practical terms, the main difference between central and bus architecture, is that the bus architecture implements a publicly available message repository, that can be accessed by any participating software (of the whole service orchestra). An integration broker tends to be more or less specialized, while an ESB provides a more generalized, standard- based infrastructure. A product like SAP XI is more suited to an individual project, where the high cost can be justified by its support for vertical industry processes. An ESB, on the other hand, provides a moderately cheap, open infrastructure, that ensures compliance to any number of different environments and technologies. In a more narrow meaning, the ESB is associated by the technology proposed by Sonic Software, who coined the expression initially.

All web service work follows the same two-step process, first you publish, and then you orchestrate. Publishing a web service means taking an existing system, and exposing it as a service. Orchestrating is when you compose multiple web services into an end-to-end process flow. The industry standard BPEL (Business Process Execution Language), provides the glue that coordinates the execution of a web service process flow.

WSDL Web Service Description Language

This is an XML based formal language that describes the syntax of an interface exposed by a specific web service. It tells the caller what the technical name of the service is, what parameters are available, and how they are defined. A WSDL delivers a template for the XML document that is acceptable for calling the service, and defines the setup of the subsequent action that handles the received request.

UDDI Universal Discovery, Description, and Integration

When a software producer finally produces an excellent business service and creates a WSDL for it, it is no worth at all, if this information does not reach a potential consumer. It is like if you produced the best bicycle wheels in the world but nobody would find your shop. In that case you might advertise in the yellow pages, so the rest of the world would know that you are selling bicycle wheels. And this is just what UDDI is about. It is a universal directory, just like the yellow pages for businesses, that lists all publicly accessible web services, along with their proper WSDL.

BPML - Business Process Modeling Language

The BPML is an approach to describe business processes by means of a harmonized XML syntax. The BPML is used to catalog each business process, and its associated web service. Every business process in BPML implements web services through request activities, and corresponding response activities.

( - )

Figure 7: Example of a BPML Document

BPEL Business Process Execution Language

BPEL stands for Business Process Execution Language. This language was created to standardize the integration logic and process automation between web services. BPEL uses other web service standards (e.g. WSDL and SOAP) for interface description and communication. BPEL describes both the inbound and outbound process interfaces in WSDL, so that they can easily integrate into other processes or applications. This has the added advantage that the consumer of a process is able to inspect and invoke the BPEL process, just like any other web service. It is also important to note that BPEL allows processes to interact synchronously and asynchronously.

So why do we want to orchestrate web services? Well, the answer to that is quite simply that web services are rapidly being adopted as a common integration fabric, and this means that services, both internal and external to a company, can easily be incorporated into one, seamless process. This opens up great potential for very innovative and flexible business processes and opportunities. Furthermore, the loose coupling and run-time discovery of web services has allowed us to move one step closer to true real-time business process modeling and execution. This is a great improvement over the somewhat brittle solutions used by some proprietary EAI solutions in the past.

EDI - Electronic Data Exchange

EDI refers to exchanging electronic documents across company boundaries. The idea behind it is simple. The sender creates a document in a format that can be received by the computer in a foreign company, and processed automatically there. It is not the dream of the real-time enterprise that drove the EDI movement, but the prospect of the paper-free office. There are numerous standards around this concept ; the best known ones are EDIFACT, in Europe, and ANSI X.12, in North America.

EDI is often mistakenly understood as a synonym for the common EDI standard EDIFACT, which only defines the formats and requirements of a document for electronic data exchange. Those utterly academic and unnecessarily complicated formats defined by EDIFACT or ANSI.X12 contribute strongly to the negative reputation that builds the aura of EDI as being complicated, elusive, and consequently expensive.

EDI in SAP solutions is supported by Intermediate Documents (IDocs), which are standardized descriptions of the structure of a business document (such as an invoice or sales order). In the SAP world, a conversion is needed between these Idocs, and corresponding EDI messages. For this reason, many EDI software vendors have become certified by SAP to provide mapping, status tracking, and conversion of Idocs, into EDI standards.

Despite the numerous advantages of using EDI, it has its reverse side. Currently the use of EDI systems is restricted to large organizations, due to its high implementation costs.

The Internet, with all its open standards (particularly XML), has given the smaller to medium sized companies the ability to use EDI as well. Internet-EDI uses Internet services to transport EDI messages. These would include, for example, protocols like HTTP, FTP, and SMTP. The use of open standards provides platform independence, which allows for a significant reduction in investment costs.

Middleware and Communication PatternsThe term middleware has long time been the placeholder for SOA in general. Middleware is software that is designed to work as the central exchange market place for message and control all traffic between connected applications. In plain words it is the General Post Office for all message exchange within the network. Middleware delivers essential services at a central place, like data conversion and mapping, guaranteed delivery, rule based receiver determination, temporary data storage and collision and contention handling.Message Pattern in Middleware

Message Forwarding

Forwarding a messsge is an elementary and a heavily used in messaging scenarios. It corresponds to the general von-Neumann priniciple of

Input Process Output

The message works with two end points:

Input channel

Output channel

In between there is nothingthan a vector that builds a path between inpout and output channel.

Nature of datasource is not part of the pattern

An important rule in SOA is that the design of a message flow is independent from the physical nature of the channels. Channels are mapped to a physical source during run time (or configuration time) only.

Example:

The following scenarios are all the same, only the data channels are differently mapped later.

File Copy: Read file Process Write File

In this case we assign file datasources to both the input and output channel

Receive to File: Receive message via HTTP process Write File

In this case we assign a file datasources to the output channel and an HTTP-Channel to the input

Get file content: Read file Process Return as HTTP

In this case we assign a file datasources to the input channel and an HTTP-Channel to the output

Exercise for XI:Step 1 Define the Message Scenriao in Integration Builder:

Define a simple scenario in SAP PI integration builder that receives a message and hands it over to the output without manipulation. Use the following message structure:

Step 2 Configure sample uses in Integration Directory:

In the Integration Directory please define three different configurations for the cases in the examples given before.

File Copy

HTTP to File

File to HTTP

Inorder to keep the exercise simple, use the MEssageID as filenames if appropriate and assume the the directory of the file location is static for now.

Figure 1: Sample Middleware Landscape

The middleware is the central message brokerto coordinate the many-to-many communication in a network. The standard features of a middleware are:

Protocol conversion

Temporary data storages and queuing

Data serialization

In addition to the basic services all modern middleware products offer support for workflow, commonly known as Business Process Monitoring (BPM). BPM will be discussed in a separate chapter.

Message drivethrough

This is the most elementary messaging pattern. A message is received from an endpoint and handed to another endpoint.

In this process we see three layers of components.

Document Formats

In this example we are deling with three document formats.

Payload

We shall call the information carried by the native format the Payload.

Native Format

This is the format which is delivered by the external applications. E.g. it is the content of the file as a plain string or the HTML document when calling via HTTP.

File Format:The quick brown fox jumps over the lazy dog.HTML Format

The quick brown fox jumps over the lazy dog.

Adapter Standard Format (Internal Transport Format)

Since the middleware should be able to see the same information independent of the data source, the adapter is responsible to repackage the payload into a harmonized internal format.

1234567890The quick brown fox jumps over the lazy dog.

You may well compare the internal format with some standard container boxes that all have the same size, so that you can stack and pile them easily. No matter whether you transport bottles, meat, letters or toys: the crates will have the same size and for the transporter the content does not matter when he does his logistics plannings.

Logical Format

The internal format is somehow arbitrary and may not meet really the needs of your application. This is not so much a problem if you work within a single middleware or workflow system and are sure that this never changes. If you say, PI will stay forever, then it is fine to work on the internal format directly. If you, however, plan to juggle data between several middleware instances of different brands, then it is more handy to have a common meta data structure across the organisation. We call this the logical format.

For our exercises we define a simple meta format like this:

< id>1234567890The quick brown fox jumps over the lazy dog.

In real world applications you might rather consider to use JMS or AMQP formats instead.

Conversion layersThe Adapter layer

The adapter layer is responsible for building the interface between the application programming interface and the device specific protocol. The adapter converts the data rectrieved from the device into a harmonize format. After conversion of the data by the adapter, the data is in a standard format. No matter which adapter you pliug in, the result of the adapter will be the same. Other names for adapters are drivers or protocol converters.

The Mapping Layer

The mapping layer serves the sole purpose to produce a harmonized document container out of the adapter delivered format.

The Process Layer

The process layer does manipulations of the document that is handed over. It can be made up of a single step or a complete sub-workflow, which is seen as a single compound procedure.

VB.NET Example:Class AdapterFunction read as StringEnd FunctionFunction write (dataStream as string)End Class

Class FileAdapter Inherits AdapterDim lFileName as String = 'dummyIn.txt'Function Overwrites read as String return readfile(lFileName)End Function

Function Overwrites write(dataStream as string) as String return readwrite(lFileName, dataStream)End Function

Function ReadFile(filename as string) as StringDim buffer as variantbuffer = (read the file)return buffer.toBase16End Function

Function WriteFile(filename as string, fileString as string) as booleanDim buffer as variantbuffer = (read the file)return buffer.toBase16End FunctionEnd Class

Exercise: Define a simple File Copy Scenario in SAP PI

Exercise: Add a mapping of the file format to a document format to the File Copy Scenario in SAP PI

Module 4: What is SOA good for?

In-Sourcing External Services

Amazon

eBAY

Google

Unified messaging

(more)

Out-Sourcing ServicesBasis ServicesArchivingData backupPrintingMailing(more)Business ServicesInvoicingCentral orderingEDICompliance checksOnline Payments(more)SecurityVirus-ChecksAccess-verificationCertificatesCredit Card Verification(more)Decentralizing Enterprise ServicesFrom R/3 to NetweaverSAP ERPSAP APOSAP BISAP Enterprise portalFrom Netweaver to Best of BreedSAP ERPMicrosoft Dynamics SatellitesShared demand PlanningBusiness ObjectsMicrosoft SharePoint Portal and ServicesSOA GoalsInteroperabilityCentral Process ControlAutomationReduction of contradicting redundancyReuse of functionality and knowledgeElectronic CollaborationClouds: Smear borders between individual computersModule 5: The Components of the SOA MarketGovernance

Front-endMultiple BrowserAJAXThemes and stylesPortalsCMSCollaboration ToolsScreen-SharingMulti-MediaStreamingCustomizingAccess ControlRoaming ProfilesCookiesKerberos

Services

Atomic servicesMessage to and from FileMessage from and to storageCompressionArchivingStagingContent based routingTimed servicesEvent handlingAsynchronous decouplingAtomic services

Abstraction

Middleware

Protocol ConversionData mappingBusiness Process Execution (Workflow)AutomationDashboardsCockpitsBusiness Process MonitoringQuality of serviceGuaranteed deliveryRegistered deliveryContent based routingBroadcastTracking

Persistence

File SystemDatabasesMessage QueuesArchivesSynchronizationStagingUse case: Weak linesVirtualization

CPU-SharingVM-Ware, VPCSimulation and TestReset to defined stateCPU-ClusteringOn-Demand CloudsSafety

Firewall and Virus-ProtectionContent isolationData loss protectionSide-effectsAvailabilityRedundancyAmorph redundancySecurity SecurityEncryptionAddress verificationComplianceHardware

Module 6: Technical Components of SOATechnical Components of SOAMessage QueuesEnterprise Service BusUniversal Protocol ServicesHTTPFTPSMTPSOAPXMLRPCAsynchonous vs Synchronous CommunicationsConvenience ServicesService DiscoveriesUDDI, SAP ESRFrontend Design ConceptsModel-View-ControllerCSS and XSLTAJAXMulti-Browser and Multi-Client frontendsMinimum SOA EquipmentBasic ServicesSMTP, FTP, HTTPSOAP, JSON, XMLRPC, SAP, ODBC, ORACLE, EXCELUnified Data Storage: WebDAVManaged RuntimeMessage QueueAgility ToolsUnified Data BrowserService DirectoriesBuilding on Top of an MQ

Use cases of SOAModule 7: Business Process ModellingThe main activity in enterprise driven IT is building automated workflows between existing processes. In this chapter we explain the concepts of BPM and what basic architecture is needed to make BPM useful for everybody and allow mashups of process flows across arbitrary software applications.Module 8: Guide to Introduction of a SOA FrameworkModule 9: Principles of SOA GovernanceSOA requires a completely different methodology to control the whole interaction of processes and flows. The challenge lies in getting a nice control over myriads of components that appear and vanish often quicker than you can register them.Module 10: Managing SOA Life Cycle

Atomic PatternsPig Runs

Pig runs are patterns that simulate a flow without carrying information from end to end. In fact compares to a transport without payload. Such transports are done in real life every time when a route is new or it is unknown if the route is free of barriers. Before a life transport with a genuine payload is sent over the path the vehicle (the pig) is fired across the parcours.

Round Trip (Loopback test)

This is the most simple of all patterns. The system sends a message to itself. No payload manipulation takes place here. The message is taken from a data source and stored in a second data source on the same system.

Exercise 1:

Using PI read a message from a file with fixed file name and store it in another file.

Exercise 2: Multichannel test

Data sources are data channels only, so they must not be part of the process definition. In this example we want to change the datasource from case to case to see if different channels are treated equally well.

Exercise 2a: Dynamic file naming

Exercise 2b: Database to database

Watchdogs

Watchdogs are used to control the execution of services. In a traditional development environment this is something that is very difficult to achieve. In a proper SOA, a watchdog is a basic service. Any service action can be run under the control of a watchdog. The watchdog controls whether the started service meets certain conditions and behaves well.

Running under the control of a watchdog can explicitelky be requested by the service itself or set as a policy by the execution environment. Am example of a policy enforced watchdog is in In SAP the execution of a dialog process that cannot run longer then 300 seconds without having a user interaction. This is handled by a built-in watchdog of the ABAP runtime, therefore set by the system policy.

Implementing a watchdog

There are several strategies to implement a watchdog.

If the service runs on the same machine it may be possible to kick off an asynchronous process and kill the process through a system command.

If the service is run remotely this may be much more difficult. A proven strategy is working with a message queue.

Start the watchdog as a proxy

Watchdog pokes the message in a queue and calls the external service

If the request does not return in time, the watchdog returns a default response and cuts the external connection

Heartbeats

Heartbeats are special form of watchdog control. In this case the watchdog requires that the controlled service reports regularly its activity to show a sign of live. Naturally they require the active cooperation of the controlled service.

Timer

Timers are one of the most commonly used components in a service based environment. There are several kinds of timers.

Timer Triggers

Such timers are uded to start execution of a service or a flow periodically or at a preset moment of time.

In a traditional programming environment you would define a timer and then call a program when the time has reached the preset lap.

In SOA we mainly think in terms of loosely coupled services. Therefore the timer will not call a program, buit rather raise an event. The ebent will then go to an event dispatcher that decides which services need to be alerted and triggered.

Timer raises event Dispatcher caches the event one or more services are triggered

Figure 2: ### Drawing for above Timer Events

Watchdog timers

Watchdog timers are a special form of a watchdog. They control if a service is run within a given time window and runtime does not exceed a preset amount of time.

Countdown (Wait) Timers

Countdown timers are mainly used to delay the start of execution of a service.

Exercise: Execute the pig run every 10 seconds

Important: Protocol the runs somehow

Fire & Forget

Exercise: Send an email with a payload

Parallel execution

Exercise Pig run with notification:

o a pig run and every time the message is send on its way an email is sent out to a fixed address for monitoring purposes.

Exercise: Send message to multiple destinations and wait for reply

Send out messages to two destinations and then wait until all have replied, then send out a message with the results.

Exercise: Monitor behaviour of above

Some scenario but this time monitor the replies for time-outs.

Exercise: Making an HTTP call to an external service

ZIP a list of files

We have written a small web service in .NET that allows adding a list of files to a ZIP archive. Define a client call to send a list of filenames and receive the filename of the resulting ZIP-archive.

In order to test the service call, define a process that does the following.

Copy some existing files from a known location into a work directory

Call the service with the name of the files that have been copied

Copy the received archive to a known output directory.

Splitting a file

We have written a small web service in .NET that allows splitting a file into several small chunks. Define a client call to send the name of an existing file and receive a list of filenames that are the created file chunks.

In order to test the service call, define a process that does the following.

Call the service with the name of the file to chunk

Copy the received files from its current locatiuon to a predefined directory.

Frontend

Frontends are delivered nowadays mainly in form of web browser applications. However, the browser is only a special variant of an HTTP client.

Browser is only an HTTP client with presentation

Browsers differ in capabilities

Some browsers allow callback, called AJAX

Exercise 1:

Create a simple web service in your favourite application development environment, that returns a lost of the current time in different time zones:

ZoneTimeUTC23.05.2008 12:00:01UTC-123.05.2008 11:00:01UTC-323.05.2008 09:00:01UTC+123.05.2008 13:00:01

Test first; prepare the pig run

Before we even start implementing any middleware pattern we need to be prepared to test the defined processes and components. We must not consider any component or process as being finished and ready to use until we have a clearly defined testing scenario that demonstrates the use of the implemented work and

Quality of Service

Coping with massive traffic

Throttling

Staging

Pseudo-Synchronous Execution

Sizing

Managing Threads

Clustering and Hologramming

Exercise: File Copy

Define a process in SAP PI that copies the content of a file from one place to another.

Solution:

Copying a file is actually a special case of a content forwarding scenario. So the process should be defined as a simple content passthrough.

Sender (process) receiver

The theory behind:

There are many ways to solve the file copy problem. The usual quick and dirty approach would simple open the file from the adapter and juggle the received information to the destination port where another adapter would take the load and save the file into another destination.

This is a working solution but not something that not universal enough. Please keep in mind that we build now a scenario that should be independent of the datasource. In fact the process that we design is very simple, it does indeed nothing at all. Kit receives the data and lets the data flow onwards to the next step, which is the adapter endpoint.

Inbound.Adapter (Process that does nothing) Outbound adapter

What does the adapter do?

This is a device specific driver, that reads the data from the data source here: our file and converts the data into a harmonized data format, the adapter meta data.

Why do we need the meta format?

There are many distinct data channels. Files, databases, TCP/IP channels. Since our process is only interested in the payload delivered from the external channel, the adapter needs to unwrap the envelope from information. After transforming the information into a meta-format we can easily swap the adapter without making any changes in the process itself. We use the same process for any adapter format. The adapter that we are going to use in reality will be assigned by the configuration of the process after deploying it.

Defining a document format

Since the data of the file may be encoded and we might also need to pass additional information we define an envelope data structure that will both carry the data payload and the extra meta information pragmas. Example:

hier goes the payload

Since the adapter engine (file adapter) may deliver a format that is not 100% compatible with our document format, it might be necessary to add some meta-adapter on process level, that maps the information delivered by the adapter into the transport structure.

Sender File adapter Meta-FileAdapter -> (process) Meta-FileAdapter File adapter receiver

Meta File Adapter: Maps between the format delivered by the file adapter into the payload segment of the XML

FileAdapter outputs FileContent.

Mapping:

D:\\inputfile.txtFileContent

A real life scenarioBluefant Corporation

Bluefant Corp is a large niche manufacturing business. It sells, procures and produces cooking ingredients and convenience dishes for the nutrition industry and gastronomy. With this business description it belongs to the non-discrete manufacturing industries.

Bluefant is a global business with factories in several continents. The main plants are located in Paris, Capetown, Singapore and San Francisco. There are following distribution channels for the business.

Direct shipment just-in-time to large food-processors like Nestle, Nabisco, Dr.Oetker, Procter&Gamble and more. The production line of the latter are designed in a way that they can buffer no more than two hours of supply chain disruption before it causes significant impact on the smooth flow of production.

Bluefant franchises the semi-processed dishes to smaller restaurant under the name Bluefant Deli. The Bluefant Deli offers snacks from across the world like various noodle specialities, Pakora, Samosa, Baji and Dim Sun variations (see the menu here: ### menu ###). The franchise line Bluefant Deli, which is an independent company owned 51% by Bluefant and the rest belonging to some investors. Bluefant Deli furnishes the restaurants, including the equipment for the kitchen like stoves and cooling machinery. All restaurants report daily their consumption and Bluefant Deli calculates then the required supply to be delivered from a local distribution centre.

Abstracting the Business Process

Self-Service Service for Field Service

Your management has decided that from now on your company will interact with the external field service engineers through a web-based self-service kiosk. For that purpose a framework contract for each contracting party is created. The contract lists all the items that are subject to the self-service scenario. Whenever a service is provided or a product is delivered against one of the existing items in the contract, the respective booking is done against the item line.

A simple case

The contract shall be created in SAP ERP. The external supplier should be presented a kiosk style web page, where he can choose the contract, view the possible contract items and make postings against the item.

Solution PathModule test

A mini application in SAP will first be created to mimic the data entry from the web and to allow an isolated test of the module

In order to make sure that frontend and backend functionality are truly independent, create the screen in another SAP client than the processing routine

Agility-Test

Generate a simple data entry web form on the framework of your choice where you can enter data and save the data through calling a backend service.

Rapid Frontend DesignWebDynpro

BSP-WebForm

Microsoft ASP.NET

Making the call to backendCGI

REST

SOAP

XMLRPC

JSon

Making Webservice Calls

Some of the information for the postings like the price has now been stored in a different place at the vendors site. We want to retrieve this information by making an HTTP call to the other server.

http://ecs.amazonaws.com/onca/xml?Service=AWSECommerceService&AWSAccessKeyId=0C95TZGS3T612ZW2T1R2&Operation=ItemSearch&SearchIndex=Books&Author=Mark%20Twain&Version=2008-04-07

Proxy-Scenario

The solution architects have made a decision that for security reasons no direct connection between front end and backend can be permitted. More than that: the web server has been chosen to run on a framework that uses a different technology than the backend.

Outsourcing Special ServicesCentral billing

Recently Bluefant Corporation had some trouble in finding skilled accountants to cope with the enormous amount of invoices they send out and receive. This triggered the responsible to have the efficiency of the accounting analyzed. The result was a bit surprising for everybody. Although every accountant worked at a considerable high personal performance, the overall efficiency of the department was fairly poor if compared to other bigger organisations. They also found out that both Bluefant Corporation and Bluefant Deli were using expensive equipment like scanning and document archives only at 40% of their capacity. Waiving the initial idea to consolidate the accounting departments of Deli and Corporation the external business process experts suggested outsourcing the billing and accounting process completely to a specialized firm. Since Bluefant had no experience in Central Billing before the decision was made to outsource to three independent firms, one in Poland, one in India and one in Mexico. The idea was that the Central Billers receive the printed documents or the electronic invoices and deliver the data to a central electronic inbox. Depending on capacity any one of the three firms would then pick the next available document and process it accordingly, i.e. either verify an incoming invoice or produce an outgoing one.

IT Challenge for Central billing

We find here a situation of many-to-many communication. There are several companies on the side of Bluefant and a limited number of service providers. Data needs to be sent between any of the providers to any of the corporate members. Since the external providers shall have access to a common pool of documents we need to cater for mechanism that allows a collaborative access and synchronization of works.

Another challenge peeks out of the situation that the involved service providers and the corporate plants are spread around the Globe and hence need to cope with different time zones, multiple languages and legal variations. The possibility that processing paths may change rapidly requires a higher degree of agility just as well.

The chosen solution strategy is based on a procedure that is well known from manual processing. Both inbound and outbound invoices are stored in a central storage that can be accessed by any authorized Bluefant user and by the concerned external clerks simultaneously.

Business Abstraction

Solution Path:Day 1: Install a common Data Storage

In the initial scenario we defined a server that was centrally accessible through normal file system from any participating member. This is created on a Microsoft SharePoint with access to it via normal file system or WebDAV.

Every time a document is scanned it is automatically routed to the central storage.

Every time a document is processed it is flagged as in process so that other users cannot modify the document meanwhile.

Every time a document is changed an event is raised. The ESB will be responsible to consume the event.

Day 2: Unified data access

There have been identified a number of business partners who wanted to deliver their invoices directly to the servers. However, they all have different preferred protocols.

We had 100 larger Franchise partners. 30 of them were able to deliver data via WebDAV or FTP. Some of the larger business partners had a strict company policy that external data exchange needs to be executed via X400 services. The others were only able to deliver data reliable via email or by depositing them on a local server where the data can be picked up.

Instead of forcing the BP to write some dedicated protocol conversion, we decided to deliver a protocol conversion service that accepts data in multiple formats and saves them easily to a file.

Multichannel-Conversions:

All protocols have a common morphology. The blocks you find there are:

Envelope

Header

Sender information

Optional: Logical recipient

Optional: Routing information

Optional: Processing Pragmas

Payload

The data that is part of the message

Example:

FTP

SMTP

HTTP Put and Get

FILE

Header: Directory information

Payload: Content

SOA GovernanceSOA Communications and PRHow to tell your bossHow to tell businessHow to tell the ITHow to tell the business partnersSOA Team OrganisationSOA Implementation StrategyGeneral StrategiesIndustry specific strategiesIndustryGovernment and Public ServicesRegulated IndustriesDifferences SMB and big industriesSOA vendor politicsSOA Skill FormingSOA EnforcementSOA Cost Balance Sheet

100 Franchise Partner

25 could deliver WebDAV and/or FTP

5 needed to install sFTP instead of FTP for security reasons

Costs for each partner: 7 Mandays in average; Total=35 MD

Since we offered a central data transformation service, costs had been reduced as follows:

SOA in uncertain environmentsDealing with external SOA partnersSOA Risk ContainmentPlanning and Running SOA Projects

Alternate Outline

Module 1 Fundamentals of SOA v2.0.ppt

Module 2 Planning and Running SOA Projects v2.0.ppt

Module 3 Building the SOA Infrastructure v2.0.ppt

Module 4 Creating the SOA Metamodel v2.0.ppt

Module 5 Building a Governance Framework v2.0.ppt

Module 6 Implementing SOBAs v2.0.ppt

Module 7 Managing the SOA Lifecycle v2.0.ppt AttachmentsThe Components of the SOA MarketGovernance

Front-end

Services Abstraction Middleware Persistence Virtualization Security & Safety Hardware

Contents1Overview1.1Lorum Ipsum Dolor DelendatSed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo.1.2GoalThe goal of the SOA Camp is to enable the IT savvy to understand in depth the new trend topic of the Service Oriented Architecture and the correlated prominent use case Web 2.0 and Enterprise 2.0.1.3What is SOASOA is a new paradigm in computing architecture that will build an underlying common infrastructure that allows breaking applications into reusable and recombinable components. SOA follows there the same solution path of modularization that automotive have undergone in the 1970ties, electronics in the 1980ties and workstation computing in the 1990ties. SOA will make enterprise computing far easier to build, control, to maintain and to enhance compared to the steep hybrid architectures of nowadays all-in-one solutions. SOA is a common term for many sub-topics like application independent frontend design, message oriented communication, Middleware and Enterprise Service Bus, Virtualization, Webservices, Software as a Service (SaaS) and more. The ultimate goal of SOA is to obfuscate the distinction between indibidual computers and let the whole internet appear as one single computer as the base ground of building Ultra Large Systems.1.4Visual SOA: Practical Training ConceptThe concept of this training is based on seeing it working. The key driver is Example-First. Before diving into the theory of SOA and Web 2.0 we shall build a small real life demo application on top of a virtual enterprise to demonstrate the difference of the SOA paradigm versus the traditional enterprise computing approach. The examples will make use of a mash-up of SAP, Microsoft and Open Source technologies spiced up with some other, prominent SOA products. For SAP addicts this will bring an additional benefit in understanding how SAP process Infrastructure (PI) works and how competing technologies seamlessly cog into each other.1.5Our Vision for this CourseDruing this course we want to build a small, working prototyping application that allows vsiualizing the core matters of SOA. We want to start with a small communication scenario between two applications, then add more applications via different channels and finally end up in a loosly coupled web of several computers, something which is known as computing cloud and stands firm as the basis for planning Ultra Large Systems.2Module 1: Getting a feeling for SOADuring this first day we want you to walk through the SOA City and understand a little bit more how a world of SOA looks like, what you can achieve with SOA and how some basic principles work.2.1Hurray! It said Mama! An Excursion in SOA TerminologySince we are all good scientists and knowledge engineers we shall start our learning sessions with an exercise in naming the things and disambiguate the meaning of extensively used expression.2.1.1.1SOA Service Oriented Architecture2.1.1.1.1Synononyms:2.1.1.1.2Definition2.2The SOA Paradigm PrinicplesSOA is much more than architectureSOA favors Darwinistic governanceSOA favors collaborationSOA implies agilitySOA is not bottom-up nor top-downSOA fails by too much governanceSOA has no purpose on its ownSOA requires developer satisfactionSOA creates new and better business models2.3SOA Challenges2.3.1SOA Challenge 1: Making Computers talk with each other2.3.1.1Calling programs remotely2.3.1.1.1Protocol API2.3.1.1.2Proxy-Communication2.3.1.2Exchange documents between computers2.3.1.2.1File exchange Managed Data Transfer2.3.1.2.2Message Queues2.3.1.2.3Database Exchange2.3.1.3Multichannel architecture2.3.1.3.1Adapters and Protocol Converters2.3.1.3.2Proxy-Communication Managed Execution2.3.2SOA Challenge 2: Common Look and Feel and MobilesThere is more than a Windows computer. There are Macs, there are Linux PCs; there are several browsers now out there, besides the Internet Explorer there are Opera, Firefox, Safari and maybe more. And there are Mobile devices like the Windows Mobile and the Apple iPhone. But also other computers are trying to access another computer via the same protocol channel.2.3.2.1Coping with multiple browsers2.3.2.2Protocols for different needs2.3.2.2.1HTTPOne request One ResponseRequest via HTTP:Response from HTTP2.3.2.2.2FTP2.3.2.2.3SMTP2.3.2.2.4Secure Variations: HTTPS, SFTP2.3.3SOA Challenge: Traffic Congestion and bad routes2.3.3.1Message Queues2.3.3.2Quality of Service2.3.4SOA Challenge: Message delivery services2.3.4.1Postman as messenger2.3.4.2Postman as message store2.3.4.3Subscription Services2.3.4.3.1Example: Drupal Newsletter Alert2.3.5SOA Challenge: Simple Automation2.3.6SOA Challenge: On Demand Computing2.3.7SOA Challenge: Computing Clouds2.3.7.1.1Examples:TimestampsCalculate Road TollsConsult Expert Systems2.4SOA LiveSOA Examples that we find in the WWWInternet as the SOA for Webservices2.4.1.1Playing with AmazonAmazon is the web service pioneer2.4.1.1.1Simple REST query2.4.1.2Web Service descriptions2.4.1.2.1AMAZON WSDL2.4.1.2.2Generate Code from WSDL2.4.1.3Non-Webservices2.4.1.4DCOM and Intellisence2.4.1.5SMTP&POP: Sender&receiver example2.5Module 2: Writing your own web servicesCreating web services yourself is fairly simple, at least if you have the right toolboxes and development environments at hand. In this first example we shall use Visual Studio as the elementary component and see how we can integrate components of arbitrary legacy like SAP or AMAZON.2.5.1Writing a simple backend application2.5.1.1Making call-offs from an existing contract2.5.2Writing a backend test proxy2.5.2.1Write request to DB and acknowledge2.5.3Writing a simple frontend2.5.3.1AgentID, ServiceID, Date, Start, End2.6Module 3: Test First Paradigm: Pig Runs and crash test dummies2.6.1Monitoring and Control DashboardsA dashboard is a utility that lets us easily define and manipulate a test scenario and view the result of the process execution.2.6.1.1Monitoring Subjects2.6.1.1.1Process Runtime2.6.1.1.2Exceptions2.6.1.1.32.6.1.2Dashboard Plugs2.6.1.2.1Exception restart2.6.1.3Exercise: Simple HTTP Client2.6.2Crash Test DummyCrash test dummies are known from the automobile test. They are used to see what effect a certain predictable accident has in reality. For our testing scenarios a crash test dummy is a message that is designed deliberately false.2.6.3Pig RunPig runs are patterns that simulate a flow without carrying information from end to end. In fact compares to a transport without payload. Such transports are done in real life every time when a route is new or it is unknown if the route is free of barriers. Before a life transport with a genuine payload is sent over the dedicated route an empty dummy vehicle (the pig) is fired across the parcours.2.6.4Exercises: Sample Pigs2.6.4.1Message Drivethrough (Null-proxy, Message-Forwarder)2.6.4.1.1Exercise 1:2.6.4.1.2Exercise 2: Multichannel test2.6.4.2Message Loopback2.6.5Parallel execution2.6.5.1Exercise Pig run with notification:2.6.5.2Exercise: Send message to multiple destinations and waitr for reply2.6.5.3Exercise: Monitor behaviour of abov2.6.6Service health check3Middleware and Communication PatternsThe term middleware has long time been the placeholder for SOA in general. Middleware is software that is designed to work as the central exchange market place for message and control all traffic between connected applications. In plain words it is the General Post Office for all message exchange within the network. Middleware delivers essential services at a central place, like data conversion and mapping, guaranteed delivery, rule based receiver determination, temporary data storage and collision and contention handling.3.1Message Pattern in Middleware3.1.1Message ForwardingNature of datasource is not part of the pattern3.1.1.1.1Example:3.1.1.2Exercise for XI:3.1.1.2.1Step 1 Define the Message Scenriao in Integration Builder:3.1.1.2.2Step 2 Configure sample uses in Integration Directory:3.1.2Message drivethrough3.1.2.1Document Formats3.1.2.1.1Payload3.1.2.1.2Native FormatFile Format:HTML Format3.1.2.1.3Adapter Standard Format (Internal Transport Format)3.1.2.1.4Logical Format3.1.2.2Conversion layers3.1.2.2.1The Adapter layer3.1.2.2.2The Mapping Layer3.1.2.2.3The Process Layer3.1.2.3VB.NET Example:3.1.2.4Exercise: Define a simple File Copy Scenario in SAP PI3.1.2.5Exercise: Add a mapping of the file format to a document format to the File Copy Scenario in SAP PI4Module 4: What is SOA good for?4.1.1In-Sourcing External Services4.1.1.1Amazon4.1.1.2eBAY4.1.1.3Google4.1.1.4Unified messaging4.1.1.5(more)4.1.2Out-Sourcing Services4.1.2.1Basis Services4.1.2.1.1Archiving4.1.2.1.2Data backup4.1.2.1.3Printing4.1.2.1.4Mailing4.1.2.1.5(more)4.1.2.2Business Services4.1.2.2.1Invoicing4.1.2.2.2Central ordering4.1.2.2.3EDI4.1.2.2.4Compliance checks4.1.2.2.5Online Payments4.1.2.2.6(more)4.1.2.3Security4.1.2.3.1Virus-Checks4.1.2.3.2Access-verificationCertificatesCredit Card Verification4.1.2.3.3(more)4.1.3Decentralizing Enterprise Services4.1.3.1From R/3 to Netweaver4.1.3.1.1SAP ERP4.1.3.1.2SAP APO4.1.3.1.3SAP BI4.1.3.1.4SAP Enterprise portal4.1.3.2From Netweaver to Best of Breed4.1.3.2.1SAP ERP4.1.3.2.2Microsoft Dynamics Satellites4.1.3.2.3Shared demand Planning4.1.3.2.4Business Objects4.1.3.2.5Microsoft SharePoint Portal and Services4.1.4SOA Goals4.1.4.1Interoperability4.1.4.2Central Process Control4.1.4.3Automation4.1.4.4Reduction of contradicting redundancy4.1.4.5Reuse of functionality and knowledge4.1.4.6Electronic Collaboration4.1.4.7Clouds: Smear borders between individual computers4.2Module 5: The Components of the SOA Market4.2.1.1Governance4.2.1.2Front-end4.2.1.2.1Multiple Browser4.2.1.2.2AJAX4.2.1.2.3Themes and styles4.2.1.2.4Portals4.2.1.2.5CMS4.2.1.2.6Collaboration ToolsScreen-SharingMulti-MediaStreaming4.2.1.2.7Customizing4.2.1.2.8Access ControlRoaming ProfilesCookiesKerberos4.2.1.3Services4.2.1.3.1Atomic servicesMessage to and from FileMessage from and to storageCompressionArchivingStagingContent based routingTimed servicesEvent handlingAsynchronous decoupling4.2.1.4Atomic services4.2.1.5Abstraction4.2.1.6Middleware4.2.1.6.1Protocol Conversion4.2.1.6.2Data mapping4.2.1.6.3Business Process Execution (Workflow)AutomationDashboardsCockpitsBusiness Process Monitoring4.2.1.6.4Quality of serviceGuaranteed deliveryRegistered deliveryContent based routingBroadcastTracking4.2.1.6.54.2.1.7Persistence4.2.1.7.1File System4.2.1.7.2Databases4.2.1.7.3Message Queues4.2.1.7.4Archives4.2.1.7.5Synchronization4.2.1.7.6StagingUse case: Weak lines4.2.1.8Virtualization4.2.1.8.1CPU-SharingVM-Ware, VPC4.2.1.8.2Simulation and TestReset to defined state4.2.1.8.3CPU-ClusteringOn-Demand Clouds4.2.1.9Safety4.2.1.9.1Firewall and Virus-Protection4.2.1.9.2Content isolation4.2.1.9.3Data loss protection4.2.1.9.4Side-effects4.2.1.9.5Availability4.2.1.9.6Redundancy4.2.1.9.7Amorph redundancy4.2.1.10Security4.2.1.10.1SecurityEncryptionAddress verificationCompliance4.2.1.11Hardware4.3Module 6: Technical Components of SOA4.3.1Technical Components of SOA4.3.1.1Message Queues4.3.1.2Enterprise Service Bus4.3.1.3Universal Protocol Services4.3.1.3.1HTTP4.3.1.3.2FTP4.3.1.3.3SMTP4.3.1.3.4SOAP4.3.1.3.5XMLRPC4.3.1.4Asynchonous vs Synchronous Communications4.3.1.5Convenience Services4.3.1.5.1Service DiscoveriesUDDI, SAP ESR4.3.1.6Frontend Design Concepts4.3.1.6.1Model-View-Controller4.3.1.6.2CSS and XSLT4.3.1.6.3AJAX4.3.1.6.4Multi-Browser and Multi-Client frontends4.3.2Minimum SOA Equipment4.3.2.1Basic Services4.3.2.1.1SMTP, FTP, HTTP4.3.2.1.2SOAP, JSON, XMLRPC, SAP, ODBC, ORACLE, EXCEL4.3.2.1.3Unified Data Storage: WebDAV4.3.2.2Managed Runtime4.3.2.3Message Queue4.3.2.4Agility Tools4.3.2.4.1Unified Data Browser4.3.2.4.2Service Directories4.3.3Building on Top of an MQ4.3.3.14.3.3.24.3.4Use cases of SOA4.4Module 7: Business Process ModellingThe main activity in enterprise driven IT is building automated workflows between existing processes. In this chapter we explain the concepts of BPM and what basic architecture is needed to make BPM useful for everybody and allow mashups of process flows across arbitrary software applications.4.5Module 8: Guide to Introduction of a SOA Framework4.6Module 9: Principles of SOA GovernanceSOA requires a completely different methodology to control the whole interaction of processes and flows. The challenge lies in getting a nice control over myriads of components that appear and vanish often quicker than you can register them.4.7Module 10: Managing SOA Life Cycle4.85Atomic Patterns5.1Pig Runs5.1.1Round Trip (Loopback test)5.1.1.1Exercise 1:5.1.1.2Exercise 2: Multichannel test5.1.1.3Exercise 2a: Dynamic file naming5.1.1.4Exercise 2b: Database to database5.1.2Watchdogs5.1.2.1Implementing a watchdog5.1.2.1.1Heartbeats5.1.3Timer5.1.3.1.1Timer Triggers5.1.3.1.2Watchdog timers5.1.3.1.3Countdown (Wait) Timers5.1.3.2Exercise: Execute the pig run every 10 seconds5.1.4Fire & Forget5.1.4.1Exercise: Send an email with a payload5.1.5Parallel execution5.1.5.1Exercise Pig run with notification:5.1.5.2Exercise: Send message to multiple destinations and wait for reply5.1.5.3Exercise: Monitor behaviour of above5.1.5.4Exercise: Making an HTTP call to an external service5.1.5.4.1ZIP a list of files5.1.5.4.2Splitting a file5.2Frontend5.3Test first; prepare the pig run5.4Quality of Service5.5Coping with massive traffic5.5.1.1.1Throttling5.5.1.1.2Staging5.5.1.1.3Sizing5.5.1.1.4Managing Threads5.5.1.1.5Clustering and Hologramming5.5.2Exercise: File Copy5.5.2.1Solution:The theory behind:5.5.2.1.1What does the adapter do?5.5.2.1.2Why do we need the meta format?5.5.2.1.3Defining a document format6A real life scenario6.1Bluefant Corporation6.1.1Abstracting the Business Process6.2Self-Service Service for Field Service6.2.1A simple case6.2.1.1Solution Path6.2.1.1.1Module test6.2.1.1.2Agility-TestRapid Frontend DesignWebDynproBSP-WebFormMicrosoft ASP.NETMaking the call to backendCGIRESTSOAPXMLRPCJSon6.2.1.2Making Webservice Calls6.2.1.3Proxy-Scenario6.3Outsourcing Special Services6.3.1Central billing6.3.1.1IT Challenge for Central billing6.3.1.2Business Abstraction6.3.2Solution Path:6.3.2.1Day 1: Install a common Data Storage6.3.2.2Day 2: Unified data access6.3.2.2.1Multichannel-Conversions:Example:7SOA Governance7.1SOA Communications and PR7.1.1How to tell your boss7.1.2How to tell business7.1.3How to tell the IT7.1.4How to tell the business partners7.2SOA Team Organisation7.3SOA Implementation Strategy7.3.1General Strategies7.3.2Industry specific strategies7.3.2.1Industry7.3.2.2Government and Public Services7.3.2.3Regulated Industries7.3.3Differences SMB and big industries7.4SOA vendor politics7.5SOA Skill Forming7.6SOA Enforcement7.7SOA Cost Balance Sheet7.8SOA in uncertain environments7.9Dealing with external SOA partners7.10SOA Risk Containment7.11Planning and Running SOA Projects8Alternate Outline8.1Module 1 Fundamentals of SOA v2.0.ppt8.2Module 2 Planning and Running SOA Projects v2.0.ppt8.3Module 3 Building the SOA Infrastructure v2.0.ppt8.4Module 4 Creating the SOA Metamodel v2.0.ppt8.5Module 5 Building a Governance Framework v2.0.ppt8.6Module 6 Implementing SOBAs v2.0.ppt8.7Module 7 Managing the SOA Lifecycle v2.0.ppt9Attachments9.1The Components of the SOA Market9.1.1.1Governance9.1.1.2Front-end9.1.1.3Services9.1.1.4Abstraction9.1.1.5Middleware9.1.1.6Persistence9.1.1.7Virtualization9.1.1.8Security & Safety9.1.1.9Hardware10Contents

Inbound Mapping

Adapter-Format

Document-Format

Document process

Document

Document

Outbound Mapping

Document-Format

Adapter-Format

Inbound Mapping

Adapter-Format

Document-Format

Document process

Document

Document

Outbound Mapping

Document-Format

Adapter-Format

Adapter

external Format

Adapter-Format

Adapter

external Format

Adapter-Format

Front-end

Services

Abst-raction

Persis-tence

Hard-ware

Virtua-lization

Security & Safety

SOA Market

Governance

Middle-ware

Front-end

Services

Abst-raction

Persis-tence

Hard-ware

Virtua-lization

Security & Safety

SOA Market

Middle-ware

Governance

8