Wednesday, October 20, 2010

Guardian Open Platform – IT, or Innovation?

Earlier this week I attended the Enterprise Search Meetup that was held at the Guardian’s office, and entitled “Search at the Guardian”.

It was a great event, lots of interesting details and a good insight into the Guardian, who, through their Open Platform initiative, seem to running away from the rest of Fleet Street in terms of innovation and open data. Who knows, perhaps they even “get” online?

Plenty of others have written reviews of the event, and most of the speakers themselves blog regularly about the Open Platform, so I won’t go into that here – suffice to say that, in the words of Martin Belam, “Open Data is basically one big advanced search query”.

What I found most interesting about the evening was the philosophy / approach behind the initiative, something that I believe is neatly illustrated by their choice of search software.

The Guardian held a search “bake-off” when they had to decide whether to upgrade their Endeca implementation to a new version, and they invited Endeca, FAST and Apache Solr. The eventual winner, and apparently this was a decision that ignored commercial factors, was the ‘free’ one – Solr. And why – well, because when it came down to it, Solr was more developer-friendly, and they knew that when they started to push the boundaries of the product, they could either reach out to the community, or, heaven-forbid, do it themselves.

This is not a rant about Open Source Software, however, as I’m not sure that’s the point. I think the Guardian is probably spending more on people tweaking their free software by hand than they would have spent on the licence fee for the expensive commercial software equivalents.

To me, the point is this: on the plus side, the commercial product comes with a supportable SLA, a sense of security around the product vendor, and a contract, which allows the user to hold the vendor to account in the event of failure. It also comes with a set of documented tools and user manuals – a product is designed to slot into an operational model.

On the downside, it is very hard to innovate around a commercial product – if it doesn’t do what you want it to, you simply have to sit and wait for the next release.

To an IT-style "CIO”, this is perfect – a fixed contract allows them to manage costs and budget for the year ahead – as well as safe-guarding their position (it would be hard to fire someone for choosing to buying Endeca over Solr in a large organisation.) It reinforces the idea of technology as an operational cost, to be managed down. This is is the essence of “IT”.

The alternative approach is to invest that same money in people – clever people, empowered to innovate. For the same amount of money they are now treating technology as a competitive advantage, something that can be used to put clear water between themselves and their peers. It’s a huge gamble, and the person responsible almost certainly will lose their job if it doesn’t pay off.

In answer to my question on the night - “What is the commercial model, and why would anyone pay to support this initiative”, it was clear that whilst there is no direct commercial model as such (it’s free, after all) they were already seeing benefits. As Stephen Dunn (whose decision it may have been) pointed out, by opening up their platform they are capable of supporting far more partners than they would have been by trying to integrate them into their core systems. (I’m not sure this was the original design goal, but I may be wrong.)

A brave initiative from the Guardian, and one worth supporting.

Tuesday, October 19, 2010

DNS Propagation

We all tell clients that DNS changes take 24-48 hours to propagate, but we very rarely get to see any evidence of this. Well, now I have a nice chart showing just this.

A website that I am, ahem, involved with, went offline for a period of time earlier this week on account of a ‘miscommunication’ with the company hosting the DNS name servers. As a result, they removed the DNS record from their servers, and the domain simply disappeared from the internet. The solution was pretty simple – update the name servers to the current hosting provider (the ever reliable Rackspace, who I have no hesitation in plugging – their service desk is fantastic). Then all I had to do was sit back and wait.

Fortunately, at the same time the site was being monitored by the equally reliable Pingdom, who monitor the site from 25 locations in 10 countries across Europe and North America (not representative of the world, I know, but enough to make this interesting).

This meant that I could watch the propagation of the DNS change as it was picked up by their servers. Again, this is not a scientific survey, but interesting nonetheless.