Thursday, March 31, 2005

XML Explicit and SQL receive adapter

Following on from my theme of 'hacking' around the limitations of the various wizards / auto-generated artefacts in BizTalk, here's an excellent posting on how to consume xml produced from SQL Server using XML EXPLICIT rather than XML AUTO.10:22 AM 3/31/2005

Tuesday, March 29, 2005

Wednesday, March 16, 2005

Ultra-hydroxy-lipo-activase III (TM)

I've long been fascinated by cod science, and no industry on earth understands the power of a completely made up word as well as Cosmetics. Here a few gems to get you going. Feel free to add any you come across.

  • Boswelox

  • Retin-ox Correxion

  • BioSync Activating™ Complex

  • Radiance Hydra Cell Vectors

Monday, March 14, 2005

BizTalk Primer

People keep on asking how they can get started with BizTalk, and so a big thanks to Luke Nyswonger for putting together a great collection of resources, along with a study plan.

Sunday, March 13, 2005

BizTalk Configuration Documenter

As promised, a few thoughts on the above utility.

First impressions are great. It produces a fantastic-looking output, using the "Description" metadata properties of orchestrations and ports to provide the detail. It even includes the diagrams of the orchestrations - how does it do that (same way as the debugger and tracking and profile editor I guess - is it some graphics API)?

I have a couple of gripes - one being that it documents external web reference schema, which can swamp the documentation with content that is not strictly relevant.

On a broader note, it provides the sort of documentation that is actually useful. If you extend the raw output by adding in custom HTML introduction pages you can generate documentation that would make any SDK developer proud. My friend and ex-colleague James, as a certified SCRUM Master, would I'm sure agree that less documentation is more, and specifically that documentation generated from source code (or source diagrams, in the BizTalk realm) is the most powerful technical documentation of all. Functional specifications are of course extremely valuable in defining how an application should behave, but should be written without regard to the implemented technology, IMHO.

I am fed up with a.) reading, and b.) being asked to write, documentation that includes database table desciptions in Word documents. They are worthless, and invariably obselete the minute they are committed to paper.

People who need to know about a database schema should be able to read a schema diagram, ditto object models. Most source applications (e.g. Oracle, SQL Server, VS.NET, Java) can now be documented using tools that use introspection to determine datatypes, interrelationships etc. The missing information, namely a broader description of the 'problem domain' that the code addresses, can be added in a separate, simple, overview document. And if the documentation tool produces compiled HTML help files (as this utility does), then this overview documentation can be integrated with the technical documentation, if that's desirable.

Let's just have a more intelligent approach to documentation, and specifically an acknowledgement that detailed technical documentation serves a very different purpose from its functional / project proposal equivalents.

One more thing

When using the Sql adapter to receive batches of data, a couple of handy hints:

1. Use a stored procedure. It allows you more control and security, and abstracts the underlying table structure.

2. Always add a parameter that allows you to restrict the size of the output (using SET ROWCOUNT). This allows you to control the flow of input messages - if there's a chance that your sproc is going to pick up 10,000 records, it's probably going to be easier to control if you poll for 100 messages every x seconds, rather than attempting to take the whole lot in one go.

Hacking files in BizTalk

The previous post on Sql debatching mentioned 'hacking' auto-generated files, something that may alarm some people (though not many, if my experience of developers is anything to go by.)

Another example of this is the extraction of primitive values from auto-generated schemas, specifically (in my case) consumed web service responses. As an example, say that you consume a web service that returns a single value (acknowledgement id, boolean flag indicating that an update went through, you know the sort of thing.) Because these values are contained within the auto-generated reference.xsd files they are neither promoted (in fact they couldn't be promoted, as only passthru pipelines are supported for SOAP calls) nor distinguished, and therefore not immediately accessible within expression or assignment shapes.

I think the documentation, if there were any, would suggest using the xpath function to extract the values, but being lazy I find it easier to simply find the relevant reference.xsd schema definition and make the propert distinguished just as you would with your own schema. The danger with this is that the distinguished property status is lost if you refresh the web reference.

However, and this to me is the distinction between good and bad practice, this would cause a compile-time error, and would therefore have to be fixed before being redeployed. It'll never cause a runtime exception, and so this does not strike me as bad practice. It's just a shortcut, which in my book is a Good Thing.

Debatching Sql receive adapter resultsets

I spent Friday with a client advising them on some basic design principles, specifically around the use of the Sql receive adapter. It was a fairly simple scenario - extracting data from a SQL Server database, then processing the output messages with some very simple orchestrations. It seemed fairly obvious to me that the easiest way to process the data was to split the output from the receive location, then process each record individually, which is where it started to go awry.

Problem I
The easiest way to split out messages from a batch is to use an envelope schema within a receive port. However, if you create a schema using the SQL adapter wizard (as most people do when using the SQL adapter), the resulting auto-generated schema is not an envelope.
Solution: the solution to this is to map this schema to one that is an envelope.

Problem II
Maps are executed after receive pipelines; if you put the map in the receive port, by the time the map is executed, it is too late to debatch, and you would end up with the envelope in the messagebox, rather than the split documents.
Solution: use a loopback send port to get the envelope back into a receive port for debatching.

Problem III
As per the discussion here, it is not possible to map to an envelope schema in a send or receive port (fixed post-SP1?).
Solution: pass the original batch message into an orchestration, map to the envelope, then use a loopback to get the split messages into the messagebox.

Phew.

By now, the proposed solution was so complicated that, although it worked, it was hard to persuade anyone that it represented a 'best practice'.

A day on the train and a bit of free thinking time got me to the better solution - hacking the auto-generated schema.

Although the generated sql schema isn't initially marked as an envelope, there is nothing to prevent you from doing just that. This means that the direct output of the sql receive location will be split in the pipeline, and that all of the individual messages will be arrive at the messagebox. As a note, the documents that are split out conform to a partial schema, and so if you want to use these, or better still, map them to some common canonical form, then you will need to create a schema with the same namespace that matches this portion of the original sql schema. (If you do decide to do this, remember to mark each element 'Form' as "Qualified".)

I have a sample project that is available on request (I have nowhere to host files, so I'll have to email it out I'm afraid) that selects first and last names from a source table, debatches the output and then maps each document message to a canonical Fullname message. Two separate orchestrations then subscribe to the Fullname message, one that extracts the firstname and inserts into a second table, and another that does the same with the lastname. The sample includes the BizTalk project, sql scripts and a binding file.

STOP PRESS: after three years, I've finally gotten around to uploading the solution for all those who are interested. It's on Codeplex - http://www.codeplex.com/biztalkdebatch

Tuesday, March 01, 2005

Rules engine and repeated element facts

Thanks to Stephen Kaufman for posting on the subject of repeated element fact assertion within the rules engine - admirably filling yet another hole in the available documentation.