Wednesday, November 24, 2004

XSD.exe and namespaces

I've spent many hours this afternoon tracking down a schema issue - the classic "The part or fragment may not exist in the database." exception that means that a message cannot be resolved to a known schema. This is usually caused by having deployed two identical schema, or by not having deployed the schema at all.

A common fix is to undeploy and redeploy all schemas. This wouldn't work in my case, and I even went as far as reconfiguring BizTalk (and deleting all the databases).

Unfortunately this still didn't fix the issue, which was really confusing, as I only had two assemblies deployed, so it shouldn't have been too hard to work through all the available schemas!

I eventually tracked it down to a local web service that I was calling from within an orchestration. I'd used the BizTalk xsd schema and the xsd.exe tool to create the .cs file containing all of the classes behind my test web service. The web service worked fine through WebServiceStudio, so I knew that it was working.

The problem is that xsd.exe applies the same schema target namespaces to the System.Xml.Serialization.Xml*Attribute attibutes, which meant that I had a namespace clash through the imported Reference.xsd in the orchestration.

So - when using xsd.exe to create classes, always check the namespace attributes.

Tuesday, November 23, 2004

Convoy processing

Those of you who subscribe to the various microsoft.public.biztalk.* newsgroups may have seen a number of posts from myself re. convoy processing, and the perils of correlation set subscriptions. The end result of all of this has been quite dramatic, in terms of our understanding of convoys, and has resulted in us removing them from our design. This is obviously quite a serious step, and so I thought it might be worth passing on some of our recent experience.

First, a quick recap on the messaging architecture. The pub-sub model is described pretty well here, amongst others, so I won't repeat it; suffice to say that messages are delivered to the messagebox through receive locations, and then matched to subscriptions using a combination of message type (as defined by the schema), and any applied filters. Subscribers include send ports (for content-based routing), orchestrations, and orchestration instances (in the case of correlation, and therefore convoys.)

If a subscription exists, then any matching message that arrives in the messagebox will be consumed.

So - how are subscriptions created, and when? The easiest way to check this is to use the BTSSubscriptionViewer utility, found in the SDK\Utilities directory. Using this you can see that in the case of send ports and orchestrations, the subscriptions exist all the time that the artefact is enlisted. For correlation sets things are a little more complex, and this is where it started to unravel for us.

Correlation subscriptions are created when the correlation set is initialized - through a receive or send shape within an orchestration. In a common sequential convoy scenario, the set is intialized, and then a Listen shape is used to pick up the correlated messages (see Alan Smith's sample here.) Without thinking about it, it's easy to assume that before the listen shape is reached, messages will be discarded, regardless of correlation matches, however this is not so! Once the subscription is created, messages will be consumed by the orchestration regardless of the listen shape.

There are two scenarios that we have come across where this causes problems:

1. When theres is a delay of some sort after the initialisation, during which new messages might reasonably be expected to be discarded, and not consumed.
2. When receiving correlated messages at a faster rate than the orchestration is capable of processing them.

We hit the first scenario when using an orchestration to manage a publication schedule. We were receiving an initial message, sleeping until a given date, then sending the first message on, and entering a Listen-Loop, which was being used to hoover up updates to the initial message. We found that updates received during in the initial delay were being published rather than discarded. There is a work around for this, and that is to use a fake Send shape just before the Listen-Loop to initialise the correlation at the last minute. It's ugly, but it works.

The second scenario is much more serious, and has proved a show-stopper for us. Consider the situation where an orchestration is not only using a convoy to batch up messages, but processing the messages as well. As in Alan's sample, the batch limits ("completeness conditions") are set by one of two parameters - a batch size, and a timeout value. This means that either the number of messages processed reaches a set limit, at which point the batch is delivered, and the orchestration dies, or there is a sufficiently long delay in between individual incoming messages for the orchestration to deliver the batch as it currently stands and then die. (e.g. If the convoy picks up 10 messages, output them, else if the convoy picks up 5 messages, then sits for 10 minutes waiting for the next message, output the batch of 5 only.)

In our test orchestration we had the following setup:

- A receive location delivering messages at a rate of 2/second.
- An orchestration picking up the messages in a convoy, and processing each message.
- The processing of each message takes 2 seconds (simulated using a delay.)
- A batch limit of 10 messages, output to a flat file, after which the orchestration dies.

We then sent 100 messages in to the receive location, expecting to see 10 flat files appear.

What we actually saw was 3 files appearing, with no sign of the missing messages. The explanation appears to be as follows:

The correlation set is initialised when the first message is received, at which point a subscription is created for all further messages (all 100 messages matched the correlation.)
The orchestration takes 20 seconds to process the ten messages it requires. However, as the messages are being received at a rate of 2/sec, 40 messages have been delivered to the messagebox in this time, and all of them match the subscription of the correlation set. They are therefore consumed, and not discarded (yet). Neither is a new orchestration instance created. This means that a net 30 messages are consumed, but NOT processed. At the end of the 20 seconds, the orchestration dies, and the outstanding 30 messages discarded.

If you then look at the services report in HAT, you should see that the orchestration is marked as "Completed with discarded messages". The missing messages should be visible in the messages report, again with the status "Suspended", "Completed with discarded messages". You could, of course, save these messages and manually resubmit them, but obviously in a production environment this is not an option.

The lesson from all of this seems to be that you should always think of convoys in light of the subscriptions that they use to consume messages, and understand when these subscriptions are created, and what might happen to the messages that fall in between the gaps.

Caveat convoy, as they say.

UPDATE: see this for a very informative posting on the background to this problem.

MSMQT configuration and testing

I've been struggling along with MSMQT this morning, trying to get to grips with its installation and configuration, and how to get messages delivered through an MSMQT adapter. A couple of pointers for others who are trying this for the first time, and having problems:

1. Once MSMQT is installed (add a new adapter through the Admin console - see the documentation for details), the easiest way to test it, and the only way when developing on a single machine, is to set up an MSMQT send adapter to point to an MSMQT receive location, as below:

a.) Set up a receive location that uses the FILE adapter.
b.) Set up a send port that subscribes to the above port (using BTS.ReceivePortName filter) to consume the file, and uses the MSMQT send adapter to send it to a queue.
c.) Set up a second receieve location (in a different receive port - if it's in the same one as above you'll get a recursive message loop), that uses MSMQT to receive the message sent by b.) above.
d.) Set up a second send port that subscribes to the MSMQT receive location, and outputs the message to a file.

Stick a file into a.) and it should pop out of d.)

This'll demonstrate FILE -> MSMQT -> FILE transfer of a message, and demonstrates the MSMQT is correctly installed.

The more interesting issue is how to send a message to the receive queue using the System.Messaging API, which is what you'll want to be doing most of the time.

The easiest way to do this is to use the SDK\Samples\Adapters\SendMSMQMessage sample. Just open the solution and run it as is.

The gotcha is that you can't use this sample from the same machine - it must be running on a separate machine, that has MSMQ (not MSMQT) installed.


Well, the experiment has come to an end, and it's time to report back. I never got round the duplication issue, and ended
up with several contacts / appointments duplicated numerous times, which proved a real pain.

The key seems to be not to have both ActiveSync and Plaxo sync'ing with more than one machine. The following steps should help you keep everything in order, without duplication.

1. Update Plaxo.
2. Clear all PIM from computers, delete ActiveSync partnerships, uninstall Plaxo.
3. Reinstall Plaxo on a single 'master' computer, and sync data.
4. Attach Smartphone, create a new partnership, and Sync data to phone.
5. Create further partnership with second computer, but do NOT install Plaxo, and sync from phone.

I don't know what you do if you want more than two computers in sync, as ActiveSync is limited to two partnerships (I believe.)

An alternative might be to synchronize the computers using Plaxo, and then only create a single ActiveSync partnership.

Good luck.

Friday, November 12, 2004

Firefox 1.0

It's finally arrived - get it here.

When you've installed it, get the excellent SAGE rss extension here.

Monday, November 08, 2004

BPM whitepaper

Excellent whitepaper from David Chappell here, that puts BizTalk into context, alongside other "Business Process Management" servers.

Smartphone update

I've installed the Jeyo Mobile Extender software so that I can now send / receive SMS from my desktop when my Smartphone is connected. I've been using the Mobile Companion, which is a great bit software, so hopefully it'll prove it's worth (it's only $14.95, which at today's exchange rate makes it practically free for UK users :-)) - I've also downloaded the Personalizer software, which I'll report back on when I get a chance to play.

Whilst on the Smartphone subject, time for a few gripes. First and foremost - I can't find a way to send contact details via SMS (only to "beam" them via IrDa / BT), which is related to the second gripe - it appears as if the SMS stack doesn't understand vCards. This lack of standard phone functionality is a huge hole IMHO, and so I'm still giving M$ the benefit of the doubt, and assuming that it's just my stupidity in not understanding how to do it.

Parallel convoy sample

Stephen Thomas has posted a new sample, this time of a concurrent (or parallel) convoy.

Sunday, November 07, 2004

Windows Train Edition

Sitting on the train on the way home on Friday evening, when it slowed down for an unplanned stop. The driver then explained that he had to "reboot the train". The lights went out, then gradually everything started up again and we were on our way.

Sell Panasonic

It's easy to see why Apple is so revered amongst the design community, when digital home products like this are still being produced.

Technically it may well be unsurpassed, but it looks like a video from the 1980's, and is called the "DMRE55EBS".

Jonathan Ives can't be the only designer in the world interested in working with technology. It's so depressing.

Monday, November 01, 2004

Next Blog

One of the great features of Blogger is the "Next Blog" button, which goes to someone else's public blog. This can be any blog on Blogger, and so not necessarily a technical blog.

My first go brought up The Shelter 1/2.


Stacy Martin, from Plaxo, has actually replied to my earlier post about my sync'ing problems... I'm astounded. Not only do Plaxo produce the best-looking software around, they're also surfing the darkest recesses of the web to bring aid to the undeserving.

I'm glad to say that I spent some time over the weekend singing Plaxo's praises to some of my unbelieving friends.

Something from nothing

Richard Veryard has posted an interesting reply to my acknowledgement of non-arrival of messages quandary, pointing out the value of 'nothing' in certain circumstances...