Thursday, March 31, 2011

Google lives in Perpetual Beta, whilst Microsoft shoots for RTM.

Something came up in conversation today that struck a chord with the (my) favourite topic of the moment – why is MSFT not responding to everything going on around it?

I don’t really know anything at all about MSFT’s business – I’m a .net guy, I’ve been to Redmond, I’ve seen the inside of the machine, but I don’t know all the ins-and-outs. But I do know what they do – and more importantly, how they do it, and that gives me enough to form an opinion.

They produce software. And the software they produce, even to this day, is distributed on physical disks. That’s right – they burn their software onto shiny plastic disks, which are then distributed around the world (“filling the channel”). And this is their problem.

Culturally, MSFT is still working in a physical disk world. If you’re burning your software onto 100m disks, you can’t afford to get it wrong, and you can’t easily get it back again once it’s left the building. So they create enormous product development cycles, building towards huge, monolithic, releases. They still call their final releases RTM (Release to Manufacturing). Their development processes are built around this concept, their product development process are built around this concept. Worse still, they have partnerships with disk-pressing plants and disk-selling retail channels. And all of this baggage they haul around with themselves to this day.

Back in the 90s, the talk was of how MSFT missed the boat, misunderstood the internet, and then woke up. They suddenly Got It. The built a better browser, killed Netscape, and now they owned it. Except that they didn’t. They thought that the internet was the browser – and since they owned the desktop, they could fold in a browser for free and their job was done. Wrong.

They still don’t get the internet; only 18 months ago I was being told, in Redmond, that whilst the internet was nice, desktop applications were nicer, so could we please put our internet-only Silverlight idea on hold, and just get with the Windows 7 program.

Google has none of this this heritage – it was born from the internet. As was Facebook. They can innovate and release at internet speed. A Google search results page is a continuously evolving multi-variate test. When they say that they live life in beta – they’re not lying – they really are trying new ideas every single day, live. Google is never “RTM”.

And Microsoft cannot compete – because of its legacy. They have a vault of cash, and some fantastic talent, but until they start to shed some of their physical dead-weight, until they acknowledge what the internet has done to their business (a business they invented), they are heading into retirement. The world they built is innovating away from them, and they’re just too slow to react.

Wednesday, March 30, 2011

webOS Developer Connect (London) – notes from a (very) small island

I’ve just come back from the HP webOS “Connect” meeting in London, where a bar full of eager mobile web developers were being courted (with free* beer no less) by the worldwide VP of Developer Relations, Richard Kerris.

I didn’t stay the full distance, leaving after about an hour and a half, and that in itself is not a good sign.

Kerris himself gave a great, and sincere, introduction to the evening. He seems like a nice guy – competent, articulate, honest. He promised the best mobile developer program in the world, access to 100m devices in the coming years (HP have committed to getting webOS onto notebooks – or possibly netbooks, one of them). The devices look good, the OS seems to be right in the 2011 sweetspot, and HP has a mountain of money – so anticipation levels were high.

And then the presentations began.

I stayed for about half-a-dozen ten minute presentations, and frankly they were lamentable. Someone from the Guardian (where isn’t there a Guardian presenter these days) gave a great little talk on how important iOS mobile devices are to their future strategy. Unfortunately they don’t have any webOS applications.

Then came a German games developer – seriously, flown in from Germany. As if there’s no one comparable in the UK. Except apparently there isn’t. He too gave a nice talk – predominantly on how the first SDK didn’t really work, and how he ported his first webOS app to iOS in a couple of days. And he’s a paid (in expenses at least) advocate.

Then we had a succession of very nice people, all of whom had various degrees of hello world apps to show, but nothing commercially viable. At which point I had to leave.

I wasn’t the biggest fan of the Pirate at QCon, but at least he had a company, employing real people, making money from iOS.

I like Mr. Kerris, and I like webOS. Hell, I even quite liked an HP laptop I saw advertised the other day, but this was a pretty poor show from HP. They couldn’t even pull together a set of speakers willing to endorse the platform professionally. We wanted Rovio to turn up, or possibly Zynga. Or someone, anyone, who was doing something that might make money out of this. If HP are putting on a roadshow for developers (as is their stated intent) they need a lot more firepower, and if that means taking some US developers on the road with them, then that’s what they need to do.


* That’s free as in beer.

Sunday, March 27, 2011

Color may have something to it after all…

I was at an event on Thursday evening at @TechHub and the host for the evening invited everyone there to install Color and at least give it a go before wading in to the “$41m for what?” debate. His teaser was quite an interesting one – by firing up Color on his phone he was able to see photos coming in from the party upstairs, and by commenting on the photos (i.e. socialising) he got himself invited to the party.

Edge-case or mainstream – not sure, but it did at least counter the “wtf would I want to share photos with people I’m already in room with” argument.

(NB I still think $41m is absurd, and we are definitely in a bubble – Color may not be, but we’re only one high-profile flop from dotcom 2.0)

Saturday, March 26, 2011

The Government is “joining up”– be afraid!

Today I applied online for a new driving licence – and as part of the process I was warned that the photo they had on record for me was only valid for a further six months. However, when I put in my passport number it found the photo I had scanned in when I got my new passport a few weeks previously. I was asked if I would like to use that instead – as it was valid for the next ten years.

Aside from the great user experience (from a Government website no less), this points to two things:

  1. There is now inter-department “joined-up” Government
  2. I now have a de facto identity card – whether it’s physical or not

The ramifications of this are enormous, and - in conjunction with recent treasury annoucement re. the combining of tax and national insurance – we are closing in on a time when the Government might have a “single customer record” – which is not necessarily a good thing.

Wednesday, March 23, 2011

Using a foreign language to challenge UX

I have spent the morning trying to test a new website that is currently available to me in Ukranian and Swedish – neither of which I speak. I have a basic understanding of the site, having seen it in its original english some time ago, and I know the functional requirements. Not being able to speak Swedish meant that when filling in a form I had to make educated guesses based on the pure UX itself – was the warning text in red, was it in a relevant position etc.

This turned out to be a fantastic way of testing the site. I repeatedly failed to submit a form because I hadn’t filled in a mandatory field, and the shape / style of the visual cues weren’t obvious enough to me.

Another example – something that was obvious to me, but not to the native speakers, was an inconsistency in button styles – which caused me to cancel my form submission because I thought the cancel button was the confirmation button.

I think I may have invented a new technique.

(In a similar fashion I do know someone who used to proof-read professionally, and she would read a document backwards – so that she could spell-check each word individually and out of context!)

Friday, March 18, 2011

How we use HipChat

Inspired by a blog post (many years ago) by the 37Signals team on their use of Campfire, their own chat application, I have been trying, unsuccessfully, to wean the development team off Windows Live Messenger, Skype, et al. for team chat.

I had been looking for a long, long time for something that would allow us to maintain long-lived (and searchable) chat archives, but that would also allow us to plug in some of our automated systems. Campfire itself didn’t work for us as it’s web-based, and no one was willing to give up the taskbar icon / notification process.

Cue HipChat – which we’ve been using for a few weeks now, and which I am now unilaterally declaring a hit. I thought it might be interesting to show how we use the chat, and how our build / vcs systems interact with the chat application, so here you are – Profero developer chat, now in colour (click to view full-size):


Monday, March 14, 2011

QCon London 2011 Review

I’ve done a quick round-up day-by-day, but here are my overall thoughts on QCon 2011.

The conference as a whole seemed slightly subdued after last year – it was sold out, so I guess it had the same number of attendees, but many of the sessions were half-full, and there was often an uncomfortable silence at the end of presentations when they were opened up for questions.

There was no stand-out presentation of the calibre of last year’s Facebook keynote, nor anything as technically impressive as the LMAX talk, despite presentations from Facebook, Google, Twitter, Netflix, VISA, Best Buy and a host of other big names. This year seemed to be more about getting things done – the nuts-and-bolts rather than the vision.

On the plus side, there was an entire track devoted to NoSQL this year, and many of the other presentations included large slices of NoSQL goodness – be in no doubt, NoSQL is no longer a niche – this is mainstream technology, in production across the web today. A big thank you to Alex Popescu not only for hosting the track, but also for his tireless efforts to bring NoSQL to the attention of the community with his blog

I’ve touched on it my daily digest, but I got a real feeling that some of the more extreme team dynamics (see Dan North and Fred George’s talks in particular) are gaining traction – there seems to be a split forming between the radical side which is eschewing almost all project management practices, and those who are artlessly turning Agile into everything Waterfall had been (process over people).

Maybe continuous deployment is the secret sauce that moves development from being project-based to business-as-usual; no need for a project plan, it’s just the job – come in, develop a feature, deploy it, go home. It looks like developers are now working with Ops (DevOps is everywhere) so closely that they are now testing and depoying their own code, and …. drum roll … it’s working! I have a train of thought going at the moment that says that if development costs moved from Capex to Opex (and innovation becomes a cost of doing business on the internet) companies would understand the requirements much better. No end date, no fixed scope – development just rolls on, day after day.

HTML5 was also hot this year – Google and Facebook were both touting their experiences / thoughts, and I wouldn’t be surprised if both didn’t settle on it as a core mobile development platform – which makes the demise of WebOS all the more poignant – right idea, wrong time?

The functional programming track was down from last year – server-side JS and Node.js in particular were this year’s favourites, with asynchronous programming the new concurrency paradigm. Since no one could claim production experience with Node, it’s hard to see where that’s going right now.

Overall, another great conference, good speakers, and some interesting experiences shared. There wasn’t a truly memorable talk, but that aside, as someone said (and I’ve said before) – we are in a golden age for software development – we may never have it so good again, so get out there and make the most of it.

Sunday, March 13, 2011

QCon 2011 (London): Day Three

Day three went a bit more webby – and technical, and frankly, a bit flat. It seemed slightly lacklustre.

Keynote: Things I Wish I’d Known (Rod Johnson) 7/10:

Co-founder of SpringSource, Rod is a successful entrepeneur, and entertaining speaker. He gave us a very personal take on the lessons learned, although I’m not sure I personally took anything away from it. I didn’t take any notes, so can’t really comment.

Single Page Apps and Future of History (Michael Mahemoff) 7/10:

Good talk from Michael (Googler) on the history of the web, and what single page apps will mean. Not much new here (for me) – this was more of a review session, but Michael put it all into context.  A good discussion of the hashbang debate followed, along with things like REST, Hijax, and the history API of HTML5 (push/pop/replace) – I’m still not totally convinced by SPAs, but they do seem to be gaining ground. For me, the summary is that many apps these days are very task focussed, and so a SPA may fit, but for a larger site, pages are still the way to go.

I had actually been hoping for a session on WebSockets, but apparently that was cancelled. Que sera, sera.

Node.js: Asynchronous I/O For Fun and Profit (Stefan Tilkov) 7/10:

Node.js is a very hot topic right now, and this session was packed to the rafters. Stefan gave us a great run-through of the model, and some possible uses. This was a code-level demo – although it was noted that no one in the room (inc. Stefan) had experience of running Node in production. Definitely one to watch, and I’m glad I went, but it was more out of interest than a practical requirement.

HTML5 @ Facebook (David Recordon) 7/10:

Key take-away here was the use of HTML/CSS/JS for mobile application development – clearly a focus at Facebook. David called out PhoneGap specifically, which kind of torpedos any other “appcelerator” framework given that it has the Facebook seal, and is free.

There was a bit of a review as to how Facebook use HTML – specifically the BigPipe process – which is very clever, but not exactly news.

Basically – they’re really clever at Facebook, and have more money than anyone else. Copy them with care – it may not work out quite as well as you had hoped.

ThingWorx and Neo4j (Rick Bullotta & Emil Eifrem) 7/10:

Bit of a sales pitch from Rick around his company, ThingWorx, which I didn’t really get (database of everything, and how it’s related to everything else), with not enough really information on Neo4j, the graph database.

HBase @ Facebook (Kannan Muthukkaruppan) 6/10:

OK, so it was the last presentation, but it was also very, very technical. This was pretty serious stuff, and I lost track early on. As before, they’re cleverer than you are, at Facebook, so copy carefully.

QCon 2011 (London): Day Two

Day two was (for me) primarily about the big guys – Google, Twitter, Netflix & Best Buy all made appearances, so this was less about the Special Forces of day one, and more about how that theory scales.

Keynote: Innovation at Google (Patrick Copeland) 8/10:

Great talk from Patrick Copeland on Google culture. Hard to summarise in short-form, but essentially the emphasis is on innovators (people) and not on ideas. Ideas are worthless without execution, everyone has an idea (Google themselves have 100-200k employee ideas in their idea-database), but very few people can make something out of an idea. Look for the spikey, difficult people – constantly challenging, never taking the easy route. Be prepared to fail, and fail fast – use data mercilessly to measure an idea, and if it’s not going anywhere kill it – Wave is the poster child for this philosophy. Google ‘dog-food’ anything and everything – not just new ideas per se, but even within the core product. Every search results page served is a complex multi-variate test (I’ve noticed this myself), measured and analysed. Like the previous day’s talks – no one should get too attached to anything – as everything is expendable. Fun section on the concept of the “pretendo-type” or “pretotype” – an example being the Palm guy who made a fake Palm Pilot out of wood and carried it around with him to see if it felt right. (See

One possible newsworthy item here was the impending launch of “Androgen” – a “v. fast prototyping tool for Android”. Watch this space.

Building Best Buy’s RESTful Commerce Engine (Brian Sletten) 8/10:

Starting to get into deep tech here, which was great. I worked with BBY a few years back, so have some appreciation of the complexity of creating software for such a behemoth, and they seem to have done a great job on their commerce API. The secret – well, there is no silver bullet (of course), but they seem to have taken a pragmatic approach – start simple, listen to your users and build out. I loved the purity of their implementation – particularly the use of hypermedia to provide fully-formed URLs to client software. You call the initial URL, and it will return all of the available service URLs. If they release a new service, they add it to the manifest, and you use it when you need to. Additional features included the use / extension of the link/rel attribute.

Our old friend “small teams, co-located” cropped up again – is there anyone who still believes that anonymous, distributed, teams of developer-drones can succeed?

One thing that they apparently found very useful is BDD – they have a very complete set (1,000+) of Cucumber tests that they can run – which apparently help enormously with smoke / regression testing new releases.

Data Architecture at Twitter Scale (Nick Kallen) 7/10:

Now we’re getting very deep – this was the first of a number of very technical presentations, which I can’t really do justice to here – though I would recommend the presentation when it finally appears on InfoQ. It was a really interesting data history lesson on Twitter – how they migrated from the original Rails / MySQL implementation to the (incredibly exotic) system they run today. Suffice to say, I now understand why the API restricts you to the last 3,200 tweets. They are fortunate in being to optimise for some incredibly precise scenarios, and so have built an architecture that supports those specific use cases. As one example, all of a user’s read-only timeline is cached in memory – so that when someone tweets all of their followers receive a message (pub-sub) that is then prepended to their own in-memory timeline. They have internal SLAs around every scenario – which includes propagating updates within a second – just imagine what that means when Ashton Kutcher tweets… (they can support 4.8m messages/second).

Couple of interesting bits: all engineering solutions are considered transient (following the theme), good enough is considered good enough, and until recently social graph events (following/unfollowing) exceeded tweet events. Counter-intuitive, but true.

Behind the Scenes at VISA (John Davies) 8/10:

This was a first – the only unrecorded session (I believe), this was a walk into the murky world of the world’s finance backbone. I hesitate to repeat anything here – I fear for my safety! John’s own presentation was neutered by the policy wonks at VISA, so this was largely a Q&A session. Interesting factoids: VISA runs on a mainframe (is not distributed), it runs on 99.99999% availability, it provides the US government with their GDP figures, it can afford to lose two data centres without interrupting service. No one knows where the DCs are, and their core systems have probably only changed 10% in twenty years (contrast with Dan North’s software half-life measured in months). This is the opposite of Agile, and probably good for it – this is, next to ICBM programming, about as serious as it gets. But that also presents VISA with a problem – when Hyves (Dutch Facebook) introduced a payment system, VISA card transactions dropped 25% in nine months – if Facecbook (/Google / ?) were to do the same, VISA’s entire business could collapse, and with their bunker-mentality they would have no way to react. So, they’re getting out into the community, hence John’s presence at QCon – VISA needs the development community more than we need them – but this could present some life-changing opportunites to some.

Be warned though – if you work at VISA you may have to sign quite a long, restrictive, covenant.

Netflix’s Cloud Data Architecture (Siddarth Anand) 8/10:

Some great facts re. Netflix – they were spending $600m/year on USPS postage before embracing the web, streaming costs 1% of posting a DVD, at peak they account for 20% of downstream bandwidth in the US, they have $2bn in revenue, and they do all this with 400 employees, 15 of which are in IT operations. That’s 15. Extraordinary stuff.

Anyway, they decided to move everything to AWS, which they did, in commendable JFDI style. One interesting point – they never attempted a wholesale data import from DC to cloud – but instead moved a user’s records on demand – when the user first accessed the new site. This meant that they were able to move everyone across over a two-three week period.

Yet again, as if it needed re-emphasising – small teams of capable people, empowered to do the job.

Using Hypermedia Services for System Integration (Tim Ewald) 6/10:

This was the end session of the REST track, and I felt a bit like I’d walked into a private conference. This talk was quite opinionated, and there was a lot of speaker-audience chat about how important it was to use link/rel for hypermedia, or some other document annotation. I don’t really care – the theory of hypermedia is interesting in general, and specific implementations are interesting (qv BBY session), but frankly the academic details are not. This was a classic case of over-thinking. A bit more JFDI, a little less ivory-tower please.

QCon 2011 (London): Day One

Great opening day to QCon, with quite a consistent theme – we appear to have moved beyond Agile to a new post-Agile “lean development” model. This was echoed from the keynote through three of the five talks. This is the anti-offshore, anti-project-management, “let us get on with it” model – which may not work for many companies, but does appear to pay huge dividends to those brave enough to embrace it.

Keynote: Scaling Lean & Agile (Craig Larman) 8/10:
Keynote speech, interesting, insightful take on how to manage effective development across very large (1000+) teams. Essentially the QCon basics - hire great people, get out of their way. Avoid job titles - foster a culture of the master programmer - banish the word architect, and create cross-functional teams. Spend 5-10% of a sprint digging deeper into requirements for the next sprint. Great software engineers are gardners, not architects. Oh, and avoid project management tools where possible - it adds little except overhead.

From months to minutes - upping the stakes (Dan North) 8/10:
Taking it up a notch, Dan North (a QCon regular) spoke about his conversion from Agile to the new post-Agile world - working in financial trading firm where software is released every three seconds on average. Or something like that. Dan's a good speaker, giving an entertaining talk, and a lot of what he said resonated with the keynote - small teams, tightly integrated (inc. the 'client'), with a lot of autonomy and control. You would have to build up a heck of a lot of trust to get away with this! Most interesting concept he spoke about is that of "Software Half-life" - this time taken for 50% of the codebase to be re-written. He estimated theirs in months - the codebase is constantly refactored out of existence; in a world like this, no one gets hung up on their own code perfection - get it done, and out of the door ("spike-and-stabilise"). He also spoke of the "path-to-production" being the most effective test-bed. True, but again, you'd need an awful lot of trust on account to be able to live like this. Fun though.

Lean Programmer Anarchy (Fred George) 8/10:
Taking it up another notch (or ten), Fred George introduced us to "Developer Driven Development" - a world without QA, BA or PM - just developers and the client (internal). This is a world where developers are measured on their contribution to the bottom line - so frankly no one gives a damn about the backlog or what you did yesterday. Hard to believe it really works like this - but apparently it does, and the business now thinks that more developers = more profit, so on they come. Developers hire their own, self-organise and manage their own time. Extraordinary, but a great fun presentation. One interesting point that reflected the keynote was the "cult of the programmer" - a/c Fred, in the bad old days (of waterfall), being a developer was a crappy job (true), and so any decent developer wanted to move on to becoming an "architect". This is no longer the case - as Fred put it - how many surgeons wake up one morning and decide to become a hospital administrator? Developers can now grow old at the keyboard, like geeky rock stars, which is a great thing (IMO).

Bringing Developers and Testers Together with Visual Studio (Giles Davies) 4/10:
Talk about coming down to earth with a bump! I walked into this talk by mistake - a last minute room-change meant that the "Do's and Don'ts on Android" talk I was expecting was in the adjacent room. In contrast to the previous talks (and the keynote) - the word from MSFT was "do you want developers having access to the test environment - probably not!" - cue tumbleweed. This was so at odds with the tone of the day (developer anarchy = profit) that it was hard to imagine it was the same conference. I felt sorry for the presenter - it wasn't his fault, and he did a grand job of demonstrating some convoluted Visual Studio features - I'll give him 4/5 for presenting and the content 0/5. Topped off with some Q&A around licensing. Wrong conference MSFT.

Agile Operations (Dan North and Chris Read) 8/10:
Dan teamed up with (another ex-ThoughtWorker) Chris Read here to demonstrate how it all comes together. Great fun, and a repeat of a (ThoughtWorks) presentation from QCon SF that I watched a while back - basically reinforcing the concept of "Infrastructure as Code" - a lot of which depends on tools like Chef / Puppet / CFEngine etc. Putting everything into version control, rolling out changes to infrastructure like you would with code. The advantage of versioning includes having a comment attached to each change - no more mystery DNS updates! NB - these guys still deploy application code using hand-rolled shell scripts - the Chefification is for the network infrastructure only. Echoed one of the other themes of the day - "optimise for change" - change will happen, so embrace it and build it into your work. Like Etsy - these guys often find it's quicker to redeploy than rollback - sign of a good process IMO.

Making Apps That Don't Suck (Mike Lee) 5/10:
Mike Lee is a legend, apparently. He's built some great apps on iOS, made some money (I guess), and is now travelling around the world dressed as a pirate. He's also a bit of a tit - and whilst I was interested in his views on iOS development, something he clearly knows about, I wasn't remotely interested in his views on life - which were of fortune-cookie sophistication. I have subsequently read that he was fired ( from his own company. No surprise to me.

NASA Keynote (Mark Powell) 6/10:
Second keynote of the day - this from a NASA guy, although not, much to the audience's dismay, someone writing software for spaceships. Still, it was interesting stuff - they clearly have some clever people working at NASA, but generally a downbeat end to the day.