Introduction

Any large organization will typically have many disparate and independent systems that it would have built or acquired from third party or as a result of mergers and acquisitions of companies over the years. The process of seamlessly connecting these dissimilar systems to work as a whole is known as Enterprise Application Integration (EAI) . BizTalk is Microsoft’s answer to solving the EAI problem. It is also used in Business-to-Business ( B2B ) and Business-to-Customer ( B2C ) scenarios in a Service Oriented Architecture (SOA). BizTalk Server 2004 is the latest and totally revamped version of BizTalk server. A major part of it has been re-written using .NET and has undergone major changes compared to its previous releases. The tools for developing BizTalk applications and artifacts such as schemas , pipelines , maps and orchestrations are now seamlessly integrated within Visual Studio .NET 2003, so that developers can avoid having to learn another development tool and at the same time leverage their existing skills of Visual Studio .NET 2003. Note that BizTalk Server 2004 will not work with the recently released Visual Studio .NET 2005

BizTalk Architecture

Figure 1 illustrates the architecture of BizTalk Server 2004.

Figure 1. BizTalk architecture and its components

The architecture of BizTalk Server 2004 can be broadly divided into:

• The Messaging System (Consisting of adapters , pipelines and the MessageBox )


• The Business Process (Consisting of orchestrations )

The job of the messaging system is to interface with external systems which involve receiving, storing and forwarding of messages between BizTalk and various external systems. The messaging system makes use of transport level adapters such as SOAP, HTTP, SMTP, FTP etc. for connecting to external systems. The pipelines are used to convert the incoming and outgoing messages into the required format. The MessageBox stores every message received and then forwards it to the end points or destinations that have subscribed to the particular message such as send ports and orchestrations.

An orchestration is a graphical representation of the business process logic, it is similar to a flow chart that shows the flow of the business process visually using various shapes for various activities. The business process may require connecting to various external systems such as web services to retrieve information that may be used for making decisions. The orchestration constructs (shapes) that make up the business logic such as loops, decisions, ports etc. are dragged on to the orchestration design surface within Visual Studio and connected in a meaningful way to build the business logic. The orchestration designer surface is similar to a canvas where the shapes are dragged and droped from the Toolbox.

Now let us see how messages are processed inside BizTalk Server end-to-end. That is, right from when a message is received into BizTalk until the message leaves BizTalk.

Receive Ports, Receive Locations and Receive Adapters

A message is received into BizTalk using a suitable receive adapter at one of the many possible receive locations of a defined receive port .

A receive port is an end point where the messages arrive. Each receive port can have multiple receive locations for receiving messages from different sources. For example a BizTalk application that processes purchase orders may receive the purchase order messages from various client applications, such as a website posting the orders using the HTTP protocol, a web service client connecting using SOAP or from a legacy application which might simply output the purchase order as a text file for BizTalk to pick up and process. In such cases you define and configure one receive location for each client connecting using a particular protocol. This is possible due to the fact that each receive location can have a separate transport adapter and a separate pipeline for preprocessing the message. Each receive location will preprocess the incoming message and transform it into a normalized XML format so that all messages look the same irrespective of where they came from. This makes it easy for the BizTalk application to process the message without having to know anything about the client application such as its original format or the protocol used.

A receive adapter implements the communication mechanism used to receive the incoming message from the external system, such as reading a file by monitoring a folder on the Windows file system or receiving a request over the HTTP protocol and so forth. BizTalk provides the following built-in receive adapters:

• EDI Adapter – For receiving messages in the Electronic Data Interchange format. This Base EDI Adapter supports both X-12 and EDIFACT standards


• File Adapter – For receiving messages as files from a Windows file system. You can filter the files to be picked up by giving the file names or the wild characters. For example if you are interested only in text files, you can give a filter such as *.txt


• FTP Adapter – For receiving messages as files from a FTP server using the File Transfer Protocol


• HTTP Adapter – For receiving messages over the HTTP protocol by exposing an URL


• SOAP Adapter – For receiving messages using SOAP over a HTTP connection, such as exposing a web service


• SQL Adapter – For receiving data from an SQL Server in the form of a XML message. This adapter polls a SQL Server at a fixed interval by sending it a SQL query or a Stored Procedure name. The result is sent back as an XML message


• BizTalk Message Queuing adapter – For receiving messages using MSMQT protocol. MSMQT is an implementation for using MSMQ transport with BizTalk Server

BizTalk also provides you with an Adapter Framework so that you can build your own custom adapter for situations where there are no ready to use adapters such as a raw socket listener for instance. Apart from the transport level adapters I mentioned above, there are a lot of other transport level as well as specialized application level adapters available that are used to connect to systems such as JD Edwards, Oracle Financials, SAP, Peoplesoft, Siebel, CORBA and so forth. These are sold separately by Microsoft and other third party vendors such as iWay, Attunity etc.

Building the Message Schema

Every message that BizTalk deals with (with few exceptions), has an XML schema (XSD) defined for it. This helps BizTalk to validate the message and also work with it internally. BizTalk provides a tool called the BizTalk Schema Editor , which is integrated into Visual Studio .NET 2003 for creating a schema. Using this tool you can easily create the structure of the XML message as a tree without having to know much about the XSD language. The XSD is generated dynamically. Figure 2 shows you a sample schema being created using the BizTalk Editor.

Figure 2. Sample schema being created using the BizTalk Editor

Receive Pipelines

Once the message is received by the adapter, it is passed through the configured receive pipeline where the message is preprocessed in stages. The output of the pipeline can be one or more messages that are normalized and converted to an XML format to be persisted into the MessageBox. This is important because BizTalk works only with XML internally and requires that all messages passing into BizTalk be converted to an XML format before reaching the MessageBox.

The various stages of the receive pipeline are shown in Figure 3

Figure 3. Various stages of the receive pipeline and their components

• Decode Stage – used to decode messages and attachments in either MIME or Secure MIME (S/MIME) format, and to verify their digital signatures


• Disassemble Stage – used to convert the incoming message formats into the normalized XML format defined by the schema. This stage is also used to copy certain content of the message into the message context known as property promotion so that they are accessible to the messaging system as well as the orchestration for routing and decision making


• Validate Stage – used to validate the disassembled message against a specified schema


• Resolve Party Stage – determines an identity for the sender of the message

BizTalk provides two built-in receive pipelines known as PassThruReceive and XMLReceive . It also provides individual components for each of the stages so that you have the freedom and flexibility of assembling your own receive pipeline by configuring the individual components using the Pipeline Designer tool inside Visual Studio .NET 2003 as shown in Figure 4. It is also possible to develop your own custom component for any of the stages using .NET

Figure 4. Pipeline Designer Tool showing various stages of the receive pipeline

The MessageBox

Once the message is normalized and converted into the XML format by the receive pipeline, it is persisted into the MessageBox which is a SQL Server database. The MessageBox is central to the Messaging Engine of BizTalk from where messages are routed to orchestrations and send ports depending on the subscription. There may be multiple orchestrations and send ports, which may be waiting for a particular message, so when a message arrives into the MessageBox, it is forwarded to whatever has subscribed to that message.

Orchestrations

An orchestration is where the business process / business logic is implemented. You will see an example of this soon when I discuss the publisher example. The orchestration manages the interactions between various external systems like connecting to web services or other applications, be it a legacy application or an Enterprise Resource Planning application such as SAP, or a Customer Relationship Management such as Siebel, and based on the response messages from the external system, business decisions can be taken. The orchestration can do things like transform messages using Maps as explained in the next section and can even call upon the Microsoft Rules Engine to apply the necessary business rules which is discussed further down.

Orchestrations are built visually using the Orchestration Designer as shown in Figure 5. With this tool and since orchestrations are visual in nature it is easy for a business analyst to visually verify the business logic without having to dig deep into the developer’s code. In fact business analysts can themselves create orchestrations in Visio by using a snap-in called Orchestration Designer for Business Analysts (ODBA) – this can be downloaded separately from the Microsoft Download Center using the URL provided in the related links for this article. This enables business analysts to create orchestrations and then pass it on to the developers for implementation.

Figure 5. Orchestration Designer tool within Visual Studio .NET 2003 showing a sample orchestration

In Figure 5 you see a sample business process built using the Orchestration Designer . Various shapes (activities) are dragged from the Toolbox and connected to form a meaningful business process. The business process shown in Figure 5 is that of a fictitious book publishing company processing the quotation request for bulk ordering of books as follows:

• Customers who want to obtain a quotation for bulk ordering of a particular book, will log on to the publishers website and enter the required details such as the title of the book, ISBN and number of copies to order. They also provide other details such as his name, email address, and the shipping address.


• The request is forwarded to a BizTalk server using the web service exposed by BizTalk.


• Once BizTalk receives the request, it first checks with the inventory system to ascertain if sufficient stock of the particular book is available. It will also retrieve the unit price and unit weight of the book from the inventory system for pricing and shipping purposes. In this case the inventory system has SQL Server as its database and does not have any other interfacing capabilities such as an API. Therefore BizTalk will have to directly access the SQL Server database using the built-in SQL Adapter to check for stock availability.


• If sufficient stock is not available, BizTalk will inform the customer of the same through the website using the Request-response port.


• If on the other hand, stock is available, BizTalk continues processing and will connect to an external shipping company through a web service using the SOAP Adapter to obtain shipping charges and the number of days required to ship the consignment.


• It will then call the business rules engine to check if any discounts are applicable and also compute the final figures including the shipping charges.


• Finally it will send out the quotation to the customer via the website as well as through email.


• It will also dump a file containing the same information for internal purposes like logging the request.

Maps and Functoids

Maps are used inside BizTalk to transform XML messages from one structure to another. BizTalk provides a tool called BizTalk Mapper that is integrated into Visual Studio .NET 2003 for visually mapping the fields from source schema to the destination schema. The fields are mapped by simply dragging a line between the fields and connecting them. Maps use XSLT internally to do the transformation. Maps contain built-in functionality in the form of Functoids for performing common functions such as database lookups, string manipulations, mathematical calculations and logical operations among other things. Figure 6 shows the BizTalk Mapper tool illustrating a sample mapping that is also using a multiplication functoid to compute the TotalBookPrice by multiplying the Qty and UnitPrice .

Figure 6. BizTalk Mapper tool with a sample mapping that uses a multiplication functoid

Microsoft Rules Engine

Business rules are dynamic in nature and change frequently based on the nature of the business. As such, it is always a good idea to identify the rules that change frequently and separate them out of the business process. Implementing the business rules inside the orchestration would mean that every time there is a change in the business rule, you need to make changes to the orchestration, which means you need to un-deploy the current orchestration, recompile and redeploy it. This would also mean that you would need the help of a BizTalk developer every time your business rules change. You can use the Microsoft Rules Engine to avoid the above scenario. Microsoft ships the Microsoft Rules Engine with BizTalk and provides a mechanism to call the rules engine from the orchestration. You can compose rules using a standalone tool called the Business Rule Composer , you need to first create a policy which is the basic unit of versioning and deployment for the rules engine. You can have one or more rules inside a policy.

The rules engine can be used to dynamically retrieve or compute values depending on the state of the message at runtime. It can retrieve data from datastores such as a database or an XML file or a .NET component.

Send Pipelines and Send Adapters

The output from the orchestration is usually a transformed message that is persisted back into the MessageBox to be routed to the appropriate destination using a suitable send adapter. Before sending the message out to the target system using the send adapter, the output message undergoes processing in stages inside the send pipeline where the message is converted from an XML format to the format required by the target system. The send pipeline also has stages similar to the receive pipeline for converting the XML to the format required by the target system.

The various stages of the send pipeline are shown in Figure 7

Figure 7. Stages of the send pipeline and their components

• Pre-assemble Stage – This stage can be used to perform any final processing before the message is converted, such as filling in any empty fields by looking up a database


• Assemble Stage – used to convert the XML message to the format required by the target application.


• Encode Stage – used to wrap outgoing messages into MIME or S/MIME format (encrypt the messages and/or use a digital signature).

Similar to the receive pipeline, BizTalk provides two built-in send pipelines known as PassThruTransmit and XMLTransmit . BizTalk also provides individual components for each of the stages so that you have the freedom and flexibility of assembling your own send pipeline by configuring the individual components using the Pipeline Designer tool inside Visual Studio .NET 2003 as shown in Figure 8. It is also possible to develop your own custom component for any of the stages using .NET

Figure 8. Pipeline Designer Tool showing various stages of the send pipeline

Compiling, Deploying and Running a BizTalk Application

Once you have built or configured your BizTalk artifacts such as schemas, pipelines, orchestrations, Maps and Rules, you need to compile and deploy them into the BizTalk Configuration Database .

Building / Compiling your BizTalk solution is similar to building a .NET solution. The compiler will generate an assembly file. Make sure you have strong named the assembly file. If you are calling any external .NET component from the BizTalk application make sure the component is also strong named and registered in the Global Assembly Cache (GAC), so that BizTalk will be able to locate the component at runtime.

Once the BizTalk solution is compiled and ready for deployment, you can deploy it into the configuration database in multiple ways depending on whether you are deploying it in the development environment or in the production environment. For deploying and running it in the development environment, you can use the BizTalk Explorer tool integrated into Visual Studio .NET 2003. First you need to deploy the solution by right clicking on the BizTalk Project in the Solution Explorer window and selecting Deploy . You can verify that the application got deployed in the configuration database using the BizTalk Explorer under the Assemblies node as shown in Figure 9

Figure 9. BizTalk Explorer

You can also use the BizTalk Explorer at design time to perform the following activities:

• Create the physical send port and receive ports


• Bind the physical ports to the orchestration’s ports


• Enlist / unenlist and start / stop the orchestration


• Enable / disable the receive locations


• Enlist / unenlist and start / stop the send ports

To deploy the BizTalk solution in a production environment you need to use a different set of tools provided by BizTalk such as the BizTalk Deployment Wizard and the BizTalk Administration Console as shown in Figure 10 and Figure 11.

Figure 10. BizTalk Deployment Wizard

Figure 11. BizTalk Administration Console

Other BizTalk Tools

Until now you have seen some of the tools used to build and deploy a BizTalk application such as the BizTalk Schema Editor, Pipeline Designer, Orchestration Designer, Business Rule Composer, BizTalk Explorer, the BizTalk Deployment Wizard and the BizTalk Administration Console. The following sections will briefly introduce you to some of the other tools used for managing and monitoring the BizTalk application such as Health and Activity Tracking ( HAT ), Human Workflow Services ( HWS ), Business Activity Monitoring ( BAM ), Business Activity Services ( BAS ).

Health and Activity Tracking (HAT)

The Health and Activity Tracking (HAT) tool (shown in Figure 12), as the name suggests, is used to debug, track and monitor the activities inside BizTalk. It can do things like:

• Finding Messages


• Check whether the messages are getting processed as expected or getting terminated due to some problems


• Inspect the details of instance messages and orchestrations at any given point of time


• Debug and set Breakpoints inside orchestrations


• Configure tracking for ports, pipelines, orchestrations, rules, and messages (Note that since tracking has overhead it should be used carefully, especially in a production environment)








Figure 12. Health and Activity Tracking tool
This figure has been reduced in size to fit in the text. To view the full image Click here

Human Workflow Services (HWS)

Human Workflow Services (HWS) is used in workflow activities that involve human intervention such as sign offs and approvals, delegating tasks to people etc. People who participate in the workflow are known as Actors . Each workflow, also known as Activity Flow is composed of one or more Actions which are implemented as orchestrations. The actions in turn contain one or more tasks that will be executed by the Actors. The user interface for the workflow can be any client that can connect to HWS using web services and even Microsoft Office tools such Word, Outlook, InfoPath etc.

For example, let us consider an online leave processing system. An employee in a company submits an online leave request. The leave request is forwarded to the Human Resource Department of the company where the workflow service looks up the leave history and the available balance and submits it to the HR manager along with the new leave request. The HR manager may approve or reject the leave based on the leave policy of the company. If the leave application is rejected, the employee is informed about the same and the workflow is terminated. If on the other hand the HR manager approves the leave, the leave application is again forwarded to the employee’s immediate manager who will have the final say in the matter and his decision (approval or rejection) is conveyed to the employee through the workflow. The employee also has the facility to check the status of the leave application anytime during the workflow.

Business Activity Monitoring (BAM)

Business Activity Monitoring (BAM) can provide high level information about the daily business activities to the information workers such as Business Analysts. The Business Analysts can analyze this information to identify bottlenecks or new opportunities in the business among other things.

Business Activity Services (BAS)

Business Activity Services (BAS) is a set of services used for managing and collaborating with Trading Partners through SharePoint Services and MS Office tools such as InfoPath. You need to create and deploy things like agreements for each trading partner (Eg: if PurchaseOrder > $1000, accept the order etc.).

BizTalk Roadmap

Microsoft is planning on two new releases of BizTalk server. The first one will be called BizTalk Server 2006 (code named Pathfinder). BizTalk Server 2006 requires Visual Studio .NET 2005 for development and can use either SQL Server 2000 or SQL Server 2005 as its datastore. The second release known as "vNext" or "Beyond BizTalk Server 2006", is due sometime in 2007 or maybe later and is expected to take advantage of the new capabilities of Windows Vista (code named Longhorn) such as the Windows Workflow Foundation and Windows Communication Foundation (codenamed Indigo).

Conclusion

This article introduced you to BizTalk Server 2004, its architecture and the various tools used for developing and deploying BizTalk artifacts such as schemas, pipelines, orchestration, business rules and ports. I hope by now you have a fair idea of the capabilities of BizTalk and how it can be used to integrate various applications and to execute the business process by orchestrating various external services to form a Service Oriented Architecture. It also briefly touched upon the tools used by BizTalk Administrators and Information Workers such as Business Activity Services (BAS), Business Activity Monitoring (BAM), Human Workflow Services, Health and Activity Tracking (HAT)