Wednesday, August 24, 2011

Our economy is based on The Wealth of Nations, and it’s out of date…

This is a post I’ve been meaning to write for some time, and have never got around to as it’s just too big a thing to get straight in my head. I’m writing something now (and it’s little more than the title) just because I think I should lay down a marker. Apologies in advance to anyone who was looking for something more insightful.

The summary is this: The Wealth of Nations by Adam Smith is the de facto bible of western economics, and its core tenet is the Division of Labour, which he studied through the prism of industrialisation and specifically the manufacture of pins.

Problem is, we’re moving beyond a classic industrial society – possibly to something identified as the Information Age. And in this age, classical economics no longer apply. This is most clearly seen in my own industry – software development, which works in precisely the opposite fashion: productivity increases as work is consolidated into a smaller group of more talented (and expensive) developers. The Division of Labour is not only unproductive, but positively destructive, as anyone who has scrambled through an offshore project can attest.

This paradox undermines many well-meaning corporate initiatives, and until we have a clear, and open, debate on this we are, frankly, a bit f*ked.

More to follow.

[1] Notwithstanding the “Invisible Hand”, which is the other core message.

Thursday, July 07, 2011

Yes, Facebook has lots of users. So what?

Another day, another article about Facebook's global takeover. Yes, I know that FB now has 750m members, which is clearly a lot of people. However, the obsession with registered users feels very like the rush for pixels in the digital camera business a few back, or the stampede for ever larger app stores amongst the mobile vendors.

I don't care if FB has 1bn members (anecdotal evidence suggests that registrations peaks at apprx. 50% internet users within a country), a camera has 50m pixels or my mobile app store has 1m applications - if they're not my friends, my favourite apps, or the pictures still come out blurry it's a problem.

Google+ is the victim in the latest article - with the Facebook-Skype deal apparently sounding the death-knell for the new Hangouts video-chat service. I don't think it will kill it, and what is more, I think the niche audience that G+ is rapidly acquiring (albeit fanned by the limited access to invites) is more valuable than the mass of humanity represented by FB*.

In a similar vein, if I was a camera vendor I would pay a lot more to advertise on Flickr than Facebook (yes, I know Flickr doesn't have ads, it's a theoretical point), because people on Flickr care about photos, and people on Facebook don't. (I'm also sure there are hundreds of amateur photo groups on Facebook - but if they really cared they'd be on Flickr, as it's a destination defined by the quality of images available.)

I am sure that it makes sense for all commercial entities to have a presence on Facebook, and to invest in a "social media strategy", but they are supporting acts; the web is the superset of all properties, and will outlive Facebook.

* NB This obviously doesn't apply to the drinks brand WKD - whose natural habitat is clearly Facebook on a Friday night.

Sunday, June 05, 2011

What agile means to me

[Update: in response to some of the comments here and on HN, I thought it might be worth updating with a note on my personal experience with Agile, which appears at the bottom of the post.]

Agile (with a big 'A') has become so mainstream now that it has started to become the problem. An alarming number of people who espouse the virtues of Agile, and who quote the Agile Manifesto believe that Agile is a project management methodology, and that Agile really means SCRUM, XP, Kanban, and that it is embodied in the daily stand-up, whiteboards or writing requirements on post-it notes. I was once told by an Agile Trainer (LOL) that the correct way to phrase the requirement "we need advertising placeholders on the site, and some way to manage which ads appear where" was "As a User, I wish to be marketed to." Needless to say his company lost a $m project on the back of such BS.

As a result of all this, Agile (big 'A' again) has become a bit of an albatross - it doesn't really work, it doesn't deliver the benefits it promised, and it inevitably involves a lot of arguing amongst the team about who should be doing what, and when.

All of which distracts from the fact that agility (small 'a' this time) is a wonderful thing, that can be achieved, that does provide enormous benefits, and that should be encouraged in everything we (as an industry) do. Agile (small 'a') refers to team dynamics, delivery processes, the software itself (how easy is it to change), and operations. Everyone should aim to be agile.

So, a few lessons I've learned along the (hard) way:

  • The best way to achieve your goal is to start with the right people
  • If you've found the right people, let them get on with the job you're paying them for
  • The people with the best ability to plan the delivery are the people doing the work
  • A small team with a manageable backlog needs very little Project Management
  • Everyone on the team “needs to know”.

A few things that help this happen on a practical level:

  • Try and avoid building towards a pre-determined marketing / launch date
  • If you do have a fixed time, then scope is variable
  • Never try messing around with team sizes to massage delivery
  • Automate everything in sight - builds, deployments, testing, progress reporting, tea-making
  • Measure as much as you can - no feedback == no direction

And finally some other things that will prevent you from getting there:

  • If you have to have everything on your requirements list, you can't be agile
  • If you need to plan beyond the next sprint with any degree of accuracy, you're not agile
  • If you think you can "fix" an iteration by adding more people, you're not agile
  • If you are prepared to change the end date of an iteration to "fit something in", you're not agile
  • If you have to give a fixed date for delivery - it's very, very difficult to be agile.

Postscript: my personal experience with Agile.

I worked for several years at a company that was at the forefront of the Agile movement in the UK, and I believe that at one point they had more registered SCRUM Master practitioners than anyone else in the country. (I think Sensai Schwaber visited in person at one point, although I’d left by then.) They were generally acknowledged to be the leaders in the field.

The incident in the post occurred when I returned to the company as a client, parachuted into a large project that they were running with the full SCRUM toolkit (“Agile Coach” included), that was running into trouble, principally the fact that it was not agile. The project was rescued by hiring a team of people with the right combination of aptitude & attitude.

As I say in the post – agility is a great goal, and everyone should aim for it – but don’t make the mistake of thinking that Agile is the means to achieve it.

Driving to the airport, agile-style.

If you were travelling from A-B, and planning the journey yourself, you might look it up on Google Maps. GM would give you a route, and an estimated time for the journey - let's say it's one hour for our mythical trip. Now let's assume that the destination for our journey is the airport, as we have a plane to catch - i.e. we absolutely have to be there on time.

In this scenario, would you leave the house one hour before the flight departs (let's ignore the reality of parking / check-in / security etc. - we're on the company jet, just driving up to the plane)? Of course not - if you had to be there on time, and you were taking someone else's directions, even someone with the 'authority' of Google, you'd probably add at least 50%, and probably 100% to the estimate. Even then, you're still subject to the vagaries of the traffic - something that affects everyone, regardless of whether they've driven the route before.

If you were to approach the journey from an agile point of view, you wouldn't book the airline ticket in advance - you'd catch the plane when you got there. And you wouldn't plan the route either - you'd hire a driver, who knew the route, and simply tell him the destination. He'd make up the route as he went along, avoiding traffic problems if possible by using his previous experience. It’s pretty clear that if you just set out from A, travelling to B as fast as you could, it would take less time than if mentally started at B, then worked backwards to determine when you ought to leave A.

Agile behaviour in a software development team requires a similar faith in the ‘driver’ – in this case the people in the team. The quickest way to reach your destination is to have faith in the team to deliver, and to make sure that they understand the destination. If you then insist on them following a specific route, then you have just removed their capacity to avoid delays and dead-ends – removing the reason you hired them in the first place.

If you want to be agile, you have to trust in your team.

(And yes, this does mean that outsourcing a project cannot be agile. It also means that agile is NOT a project management methodology. It’s a way of working.)

Monday, May 16, 2011

Application Lifecycle Management

[Update: a couple of people have called me out on this post, and suggested that it would only work from the bottom up if you hire good developers. This is true. This entire blog is based on the philosophy that the cheapest and most efficient way to achieve any given goal in software is to hire a smaller number of better developers. If you have a different philosophy, then you may find this site more helpful]

ALM is described on Wikipedia as:

“…a continuous process of managing the life of an application through governance, development and maintenance. ALM is the marriage of business management to software engineering made possible by tools that facilitate and integrate requirements management, architecture, coding, testing, tracking, and release management.”

The phrase itself is enough to make the blood run cold, but I’ve been struggling recently to justify why I think it’s a Bad Thing, and I think I’ve worked it out: it’s all about the direction in which processes are created and applied.

ALM comes from the Old World (aka I.T.), where working practices are created by Architects and applied downwards to development teams. It’s a process designed to commoditise development, and to manage human beings out of the equation. If the process is sufficiently detailed and watertight, then developers can become replaceable – which in turn allows you to find cheapers versions. 

This is lowest common denominator thinking – by creating a process that allows poor developers to work effectively, that same process will prevent good developers from doing what they’re good at. It’s mediocrity by design. Which in I.T. terms is not necessarily a bad thing – ERP, Accounting, Warehouse Management Systems – these are things that are: a.) operated in a controlled environment, b.) used by specially-trained staff, c.) defined by stability, not innovation.

None of this applies to web application development – which is defined in contrast by: a.) public access, b.) self-service, c.) continuous innovation, and pace of change. In this New World (otherwise known as The Web), companies are defined by the quality of their development staff – they are the asset, and they need be allowed to flourish, and not capped by  process.

The kicker to this is that process is still a good thing in this New World – but the new processes are designed not to prevent developers from getting things wrong, but to make it easier for them to get things right. ALM should exist in practice, but it should (in fact must) come from the bottom up – the processes are organic and designed by the developers, for the developers.

There’s no requirement for management to even be aware of these processes – they should manage by outcome, not intervention.

NB If this rings any bells with you I would recommend Orson Scott Card’s “software developers as bees” article -

Monday, May 09, 2011

Web app or native - looks like HTML is winning the war.

When the iPhone launched, one of the major gripes was the lack of support for native apps. Steve Jobs launched the web-apps-look-like-native-apps program, was roundly criticised for the lack of a native SDK, and sure enough it turned out it was already in the pipeline. The App-Age was born.

(Not long after, Palm released webOS, a web-app platform from inception, and it was roundly praised for its innovative approach.)

Circle back round a few years, and the plethora of native platforms (iOS, Android, RIM, webOS, WP7) and form factors (phone, tablet) combined with the emergence of HTML5 as a "standard" (albeit one implemented (in)differently by each platform developer) means that web-apps are now back in favour.

PhoneGap in particular seems to have captured the mindshare of the mobile developer community; it will be interesting to see how this battle plays out over the coming year. Gaming aside, once local storage becomes a common feature, web apps look unstoppable.

Thursday, April 14, 2011

Ode to Dropbox

I’ve been using Dropbox for about 18 months, and before that I was a Mesh fan, and in the dim and distant past I spent many, many hours fighting with Plaxo and ActiveSync (see here for a post from 2004 on my frustrations at the time) - so I like to think I know a thing or two about file syncing issues.

I have a separate work Dropbox account, which for the past couple of months has been so far over its (free) limit that I've had to stop using it (there's only so much you can legitimately clear out!) In the meantime, two things have been happening - I've been adding to the local directory, in anticipation of a future upgrade, and other people have been adding content to the remote folders. I had enough faith in Dropbox already to be able to cope with this – I’d get another 3GB of new content coming down, and Dropbox would get a few MB of changes from my local machine. Easy.

Unfortunately, having upgraded, I discovered that in some overzealous spring-cleaning I had removed critical user profile files, and so the Windows user account associated with the work Dropbox account was corrupt, and I couldn't login and start Dropbox. This meant that all of my changes were now stuck in a local folder that couldn’t be synced, hidden amongst 2GB of data that had already been synced to the cloud.

My solution was to create a brand new Windows account and to copy over the entire 2GB archive to this new profile before hitting the sync button - but this meant that instead of adding just my recent changes, I would be syncing the entire 2GB local directory to the new user profile, 99% of which was already on Dropbox, uploaded from a different Dropbox sync. I was convinced this would result in conflict armageddon (and about 48 hours of file transfers – 2GB up, 3GB down, merge, sync, index.)

I left it running over lunch, came back and discovered... nothing. It had just worked.

Dropbox is an amazing piece of software, and if they don't sell up to Microsoft (Windows 8 - Live Mesh replacement), Apple (MobileMe replacement) or Google (Chrome OS file system) for several billion dollars something ain't right.

Thursday, March 31, 2011

Google lives in Perpetual Beta, whilst Microsoft shoots for RTM.

Something came up in conversation today that struck a chord with the (my) favourite topic of the moment – why is MSFT not responding to everything going on around it?

I don’t really know anything at all about MSFT’s business – I’m a .net guy, I’ve been to Redmond, I’ve seen the inside of the machine, but I don’t know all the ins-and-outs. But I do know what they do – and more importantly, how they do it, and that gives me enough to form an opinion.

They produce software. And the software they produce, even to this day, is distributed on physical disks. That’s right – they burn their software onto shiny plastic disks, which are then distributed around the world (“filling the channel”). And this is their problem.

Culturally, MSFT is still working in a physical disk world. If you’re burning your software onto 100m disks, you can’t afford to get it wrong, and you can’t easily get it back again once it’s left the building. So they create enormous product development cycles, building towards huge, monolithic, releases. They still call their final releases RTM (Release to Manufacturing). Their development processes are built around this concept, their product development process are built around this concept. Worse still, they have partnerships with disk-pressing plants and disk-selling retail channels. And all of this baggage they haul around with themselves to this day.

Back in the 90s, the talk was of how MSFT missed the boat, misunderstood the internet, and then woke up. They suddenly Got It. The built a better browser, killed Netscape, and now they owned it. Except that they didn’t. They thought that the internet was the browser – and since they owned the desktop, they could fold in a browser for free and their job was done. Wrong.

They still don’t get the internet; only 18 months ago I was being told, in Redmond, that whilst the internet was nice, desktop applications were nicer, so could we please put our internet-only Silverlight idea on hold, and just get with the Windows 7 program.

Google has none of this this heritage – it was born from the internet. As was Facebook. They can innovate and release at internet speed. A Google search results page is a continuously evolving multi-variate test. When they say that they live life in beta – they’re not lying – they really are trying new ideas every single day, live. Google is never “RTM”.

And Microsoft cannot compete – because of its legacy. They have a vault of cash, and some fantastic talent, but until they start to shed some of their physical dead-weight, until they acknowledge what the internet has done to their business (a business they invented), they are heading into retirement. The world they built is innovating away from them, and they’re just too slow to react.

Wednesday, March 30, 2011

webOS Developer Connect (London) – notes from a (very) small island

I’ve just come back from the HP webOS “Connect” meeting in London, where a bar full of eager mobile web developers were being courted (with free* beer no less) by the worldwide VP of Developer Relations, Richard Kerris.

I didn’t stay the full distance, leaving after about an hour and a half, and that in itself is not a good sign.

Kerris himself gave a great, and sincere, introduction to the evening. He seems like a nice guy – competent, articulate, honest. He promised the best mobile developer program in the world, access to 100m devices in the coming years (HP have committed to getting webOS onto notebooks – or possibly netbooks, one of them). The devices look good, the OS seems to be right in the 2011 sweetspot, and HP has a mountain of money – so anticipation levels were high.

And then the presentations began.

I stayed for about half-a-dozen ten minute presentations, and frankly they were lamentable. Someone from the Guardian (where isn’t there a Guardian presenter these days) gave a great little talk on how important iOS mobile devices are to their future strategy. Unfortunately they don’t have any webOS applications.

Then came a German games developer – seriously, flown in from Germany. As if there’s no one comparable in the UK. Except apparently there isn’t. He too gave a nice talk – predominantly on how the first SDK didn’t really work, and how he ported his first webOS app to iOS in a couple of days. And he’s a paid (in expenses at least) advocate.

Then we had a succession of very nice people, all of whom had various degrees of hello world apps to show, but nothing commercially viable. At which point I had to leave.

I wasn’t the biggest fan of the Pirate at QCon, but at least he had a company, employing real people, making money from iOS.

I like Mr. Kerris, and I like webOS. Hell, I even quite liked an HP laptop I saw advertised the other day, but this was a pretty poor show from HP. They couldn’t even pull together a set of speakers willing to endorse the platform professionally. We wanted Rovio to turn up, or possibly Zynga. Or someone, anyone, who was doing something that might make money out of this. If HP are putting on a roadshow for developers (as is their stated intent) they need a lot more firepower, and if that means taking some US developers on the road with them, then that’s what they need to do.


* That’s free as in beer.

Sunday, March 27, 2011

Color may have something to it after all…

I was at an event on Thursday evening at @TechHub and the host for the evening invited everyone there to install Color and at least give it a go before wading in to the “$41m for what?” debate. His teaser was quite an interesting one – by firing up Color on his phone he was able to see photos coming in from the party upstairs, and by commenting on the photos (i.e. socialising) he got himself invited to the party.

Edge-case or mainstream – not sure, but it did at least counter the “wtf would I want to share photos with people I’m already in room with” argument.

(NB I still think $41m is absurd, and we are definitely in a bubble – Color may not be, but we’re only one high-profile flop from dotcom 2.0)

Saturday, March 26, 2011

The Government is “joining up”– be afraid!

Today I applied online for a new driving licence – and as part of the process I was warned that the photo they had on record for me was only valid for a further six months. However, when I put in my passport number it found the photo I had scanned in when I got my new passport a few weeks previously. I was asked if I would like to use that instead – as it was valid for the next ten years.

Aside from the great user experience (from a Government website no less), this points to two things:

  1. There is now inter-department “joined-up” Government
  2. I now have a de facto identity card – whether it’s physical or not

The ramifications of this are enormous, and - in conjunction with recent treasury annoucement re. the combining of tax and national insurance – we are closing in on a time when the Government might have a “single customer record” – which is not necessarily a good thing.

Wednesday, March 23, 2011

Using a foreign language to challenge UX

I have spent the morning trying to test a new website that is currently available to me in Ukranian and Swedish – neither of which I speak. I have a basic understanding of the site, having seen it in its original english some time ago, and I know the functional requirements. Not being able to speak Swedish meant that when filling in a form I had to make educated guesses based on the pure UX itself – was the warning text in red, was it in a relevant position etc.

This turned out to be a fantastic way of testing the site. I repeatedly failed to submit a form because I hadn’t filled in a mandatory field, and the shape / style of the visual cues weren’t obvious enough to me.

Another example – something that was obvious to me, but not to the native speakers, was an inconsistency in button styles – which caused me to cancel my form submission because I thought the cancel button was the confirmation button.

I think I may have invented a new technique.

(In a similar fashion I do know someone who used to proof-read professionally, and she would read a document backwards – so that she could spell-check each word individually and out of context!)

Friday, March 18, 2011

How we use HipChat

Inspired by a blog post (many years ago) by the 37Signals team on their use of Campfire, their own chat application, I have been trying, unsuccessfully, to wean the development team off Windows Live Messenger, Skype, et al. for team chat.

I had been looking for a long, long time for something that would allow us to maintain long-lived (and searchable) chat archives, but that would also allow us to plug in some of our automated systems. Campfire itself didn’t work for us as it’s web-based, and no one was willing to give up the taskbar icon / notification process.

Cue HipChat – which we’ve been using for a few weeks now, and which I am now unilaterally declaring a hit. I thought it might be interesting to show how we use the chat, and how our build / vcs systems interact with the chat application, so here you are – Profero developer chat, now in colour (click to view full-size):


Monday, March 14, 2011

QCon London 2011 Review

I’ve done a quick round-up day-by-day, but here are my overall thoughts on QCon 2011.

The conference as a whole seemed slightly subdued after last year – it was sold out, so I guess it had the same number of attendees, but many of the sessions were half-full, and there was often an uncomfortable silence at the end of presentations when they were opened up for questions.

There was no stand-out presentation of the calibre of last year’s Facebook keynote, nor anything as technically impressive as the LMAX talk, despite presentations from Facebook, Google, Twitter, Netflix, VISA, Best Buy and a host of other big names. This year seemed to be more about getting things done – the nuts-and-bolts rather than the vision.

On the plus side, there was an entire track devoted to NoSQL this year, and many of the other presentations included large slices of NoSQL goodness – be in no doubt, NoSQL is no longer a niche – this is mainstream technology, in production across the web today. A big thank you to Alex Popescu not only for hosting the track, but also for his tireless efforts to bring NoSQL to the attention of the community with his blog

I’ve touched on it my daily digest, but I got a real feeling that some of the more extreme team dynamics (see Dan North and Fred George’s talks in particular) are gaining traction – there seems to be a split forming between the radical side which is eschewing almost all project management practices, and those who are artlessly turning Agile into everything Waterfall had been (process over people).

Maybe continuous deployment is the secret sauce that moves development from being project-based to business-as-usual; no need for a project plan, it’s just the job – come in, develop a feature, deploy it, go home. It looks like developers are now working with Ops (DevOps is everywhere) so closely that they are now testing and depoying their own code, and …. drum roll … it’s working! I have a train of thought going at the moment that says that if development costs moved from Capex to Opex (and innovation becomes a cost of doing business on the internet) companies would understand the requirements much better. No end date, no fixed scope – development just rolls on, day after day.

HTML5 was also hot this year – Google and Facebook were both touting their experiences / thoughts, and I wouldn’t be surprised if both didn’t settle on it as a core mobile development platform – which makes the demise of WebOS all the more poignant – right idea, wrong time?

The functional programming track was down from last year – server-side JS and Node.js in particular were this year’s favourites, with asynchronous programming the new concurrency paradigm. Since no one could claim production experience with Node, it’s hard to see where that’s going right now.

Overall, another great conference, good speakers, and some interesting experiences shared. There wasn’t a truly memorable talk, but that aside, as someone said (and I’ve said before) – we are in a golden age for software development – we may never have it so good again, so get out there and make the most of it.

Sunday, March 13, 2011

QCon 2011 (London): Day Three

Day three went a bit more webby – and technical, and frankly, a bit flat. It seemed slightly lacklustre.

Keynote: Things I Wish I’d Known (Rod Johnson) 7/10:

Co-founder of SpringSource, Rod is a successful entrepeneur, and entertaining speaker. He gave us a very personal take on the lessons learned, although I’m not sure I personally took anything away from it. I didn’t take any notes, so can’t really comment.

Single Page Apps and Future of History (Michael Mahemoff) 7/10:

Good talk from Michael (Googler) on the history of the web, and what single page apps will mean. Not much new here (for me) – this was more of a review session, but Michael put it all into context.  A good discussion of the hashbang debate followed, along with things like REST, Hijax, and the history API of HTML5 (push/pop/replace) – I’m still not totally convinced by SPAs, but they do seem to be gaining ground. For me, the summary is that many apps these days are very task focussed, and so a SPA may fit, but for a larger site, pages are still the way to go.

I had actually been hoping for a session on WebSockets, but apparently that was cancelled. Que sera, sera.

Node.js: Asynchronous I/O For Fun and Profit (Stefan Tilkov) 7/10:

Node.js is a very hot topic right now, and this session was packed to the rafters. Stefan gave us a great run-through of the model, and some possible uses. This was a code-level demo – although it was noted that no one in the room (inc. Stefan) had experience of running Node in production. Definitely one to watch, and I’m glad I went, but it was more out of interest than a practical requirement.

HTML5 @ Facebook (David Recordon) 7/10:

Key take-away here was the use of HTML/CSS/JS for mobile application development – clearly a focus at Facebook. David called out PhoneGap specifically, which kind of torpedos any other “appcelerator” framework given that it has the Facebook seal, and is free.

There was a bit of a review as to how Facebook use HTML – specifically the BigPipe process – which is very clever, but not exactly news.

Basically – they’re really clever at Facebook, and have more money than anyone else. Copy them with care – it may not work out quite as well as you had hoped.

ThingWorx and Neo4j (Rick Bullotta & Emil Eifrem) 7/10:

Bit of a sales pitch from Rick around his company, ThingWorx, which I didn’t really get (database of everything, and how it’s related to everything else), with not enough really information on Neo4j, the graph database.

HBase @ Facebook (Kannan Muthukkaruppan) 6/10:

OK, so it was the last presentation, but it was also very, very technical. This was pretty serious stuff, and I lost track early on. As before, they’re cleverer than you are, at Facebook, so copy carefully.

QCon 2011 (London): Day Two

Day two was (for me) primarily about the big guys – Google, Twitter, Netflix & Best Buy all made appearances, so this was less about the Special Forces of day one, and more about how that theory scales.

Keynote: Innovation at Google (Patrick Copeland) 8/10:

Great talk from Patrick Copeland on Google culture. Hard to summarise in short-form, but essentially the emphasis is on innovators (people) and not on ideas. Ideas are worthless without execution, everyone has an idea (Google themselves have 100-200k employee ideas in their idea-database), but very few people can make something out of an idea. Look for the spikey, difficult people – constantly challenging, never taking the easy route. Be prepared to fail, and fail fast – use data mercilessly to measure an idea, and if it’s not going anywhere kill it – Wave is the poster child for this philosophy. Google ‘dog-food’ anything and everything – not just new ideas per se, but even within the core product. Every search results page served is a complex multi-variate test (I’ve noticed this myself), measured and analysed. Like the previous day’s talks – no one should get too attached to anything – as everything is expendable. Fun section on the concept of the “pretendo-type” or “pretotype” – an example being the Palm guy who made a fake Palm Pilot out of wood and carried it around with him to see if it felt right. (See

One possible newsworthy item here was the impending launch of “Androgen” – a “v. fast prototyping tool for Android”. Watch this space.

Building Best Buy’s RESTful Commerce Engine (Brian Sletten) 8/10:

Starting to get into deep tech here, which was great. I worked with BBY a few years back, so have some appreciation of the complexity of creating software for such a behemoth, and they seem to have done a great job on their commerce API. The secret – well, there is no silver bullet (of course), but they seem to have taken a pragmatic approach – start simple, listen to your users and build out. I loved the purity of their implementation – particularly the use of hypermedia to provide fully-formed URLs to client software. You call the initial URL, and it will return all of the available service URLs. If they release a new service, they add it to the manifest, and you use it when you need to. Additional features included the use / extension of the link/rel attribute.

Our old friend “small teams, co-located” cropped up again – is there anyone who still believes that anonymous, distributed, teams of developer-drones can succeed?

One thing that they apparently found very useful is BDD – they have a very complete set (1,000+) of Cucumber tests that they can run – which apparently help enormously with smoke / regression testing new releases.

Data Architecture at Twitter Scale (Nick Kallen) 7/10:

Now we’re getting very deep – this was the first of a number of very technical presentations, which I can’t really do justice to here – though I would recommend the presentation when it finally appears on InfoQ. It was a really interesting data history lesson on Twitter – how they migrated from the original Rails / MySQL implementation to the (incredibly exotic) system they run today. Suffice to say, I now understand why the API restricts you to the last 3,200 tweets. They are fortunate in being to optimise for some incredibly precise scenarios, and so have built an architecture that supports those specific use cases. As one example, all of a user’s read-only timeline is cached in memory – so that when someone tweets all of their followers receive a message (pub-sub) that is then prepended to their own in-memory timeline. They have internal SLAs around every scenario – which includes propagating updates within a second – just imagine what that means when Ashton Kutcher tweets… (they can support 4.8m messages/second).

Couple of interesting bits: all engineering solutions are considered transient (following the theme), good enough is considered good enough, and until recently social graph events (following/unfollowing) exceeded tweet events. Counter-intuitive, but true.

Behind the Scenes at VISA (John Davies) 8/10:

This was a first – the only unrecorded session (I believe), this was a walk into the murky world of the world’s finance backbone. I hesitate to repeat anything here – I fear for my safety! John’s own presentation was neutered by the policy wonks at VISA, so this was largely a Q&A session. Interesting factoids: VISA runs on a mainframe (is not distributed), it runs on 99.99999% availability, it provides the US government with their GDP figures, it can afford to lose two data centres without interrupting service. No one knows where the DCs are, and their core systems have probably only changed 10% in twenty years (contrast with Dan North’s software half-life measured in months). This is the opposite of Agile, and probably good for it – this is, next to ICBM programming, about as serious as it gets. But that also presents VISA with a problem – when Hyves (Dutch Facebook) introduced a payment system, VISA card transactions dropped 25% in nine months – if Facecbook (/Google / ?) were to do the same, VISA’s entire business could collapse, and with their bunker-mentality they would have no way to react. So, they’re getting out into the community, hence John’s presence at QCon – VISA needs the development community more than we need them – but this could present some life-changing opportunites to some.

Be warned though – if you work at VISA you may have to sign quite a long, restrictive, covenant.

Netflix’s Cloud Data Architecture (Siddarth Anand) 8/10:

Some great facts re. Netflix – they were spending $600m/year on USPS postage before embracing the web, streaming costs 1% of posting a DVD, at peak they account for 20% of downstream bandwidth in the US, they have $2bn in revenue, and they do all this with 400 employees, 15 of which are in IT operations. That’s 15. Extraordinary stuff.

Anyway, they decided to move everything to AWS, which they did, in commendable JFDI style. One interesting point – they never attempted a wholesale data import from DC to cloud – but instead moved a user’s records on demand – when the user first accessed the new site. This meant that they were able to move everyone across over a two-three week period.

Yet again, as if it needed re-emphasising – small teams of capable people, empowered to do the job.

Using Hypermedia Services for System Integration (Tim Ewald) 6/10:

This was the end session of the REST track, and I felt a bit like I’d walked into a private conference. This talk was quite opinionated, and there was a lot of speaker-audience chat about how important it was to use link/rel for hypermedia, or some other document annotation. I don’t really care – the theory of hypermedia is interesting in general, and specific implementations are interesting (qv BBY session), but frankly the academic details are not. This was a classic case of over-thinking. A bit more JFDI, a little less ivory-tower please.

QCon 2011 (London): Day One

Great opening day to QCon, with quite a consistent theme – we appear to have moved beyond Agile to a new post-Agile “lean development” model. This was echoed from the keynote through three of the five talks. This is the anti-offshore, anti-project-management, “let us get on with it” model – which may not work for many companies, but does appear to pay huge dividends to those brave enough to embrace it.

Keynote: Scaling Lean & Agile (Craig Larman) 8/10:
Keynote speech, interesting, insightful take on how to manage effective development across very large (1000+) teams. Essentially the QCon basics - hire great people, get out of their way. Avoid job titles - foster a culture of the master programmer - banish the word architect, and create cross-functional teams. Spend 5-10% of a sprint digging deeper into requirements for the next sprint. Great software engineers are gardners, not architects. Oh, and avoid project management tools where possible - it adds little except overhead.

From months to minutes - upping the stakes (Dan North) 8/10:
Taking it up a notch, Dan North (a QCon regular) spoke about his conversion from Agile to the new post-Agile world - working in financial trading firm where software is released every three seconds on average. Or something like that. Dan's a good speaker, giving an entertaining talk, and a lot of what he said resonated with the keynote - small teams, tightly integrated (inc. the 'client'), with a lot of autonomy and control. You would have to build up a heck of a lot of trust to get away with this! Most interesting concept he spoke about is that of "Software Half-life" - this time taken for 50% of the codebase to be re-written. He estimated theirs in months - the codebase is constantly refactored out of existence; in a world like this, no one gets hung up on their own code perfection - get it done, and out of the door ("spike-and-stabilise"). He also spoke of the "path-to-production" being the most effective test-bed. True, but again, you'd need an awful lot of trust on account to be able to live like this. Fun though.

Lean Programmer Anarchy (Fred George) 8/10:
Taking it up another notch (or ten), Fred George introduced us to "Developer Driven Development" - a world without QA, BA or PM - just developers and the client (internal). This is a world where developers are measured on their contribution to the bottom line - so frankly no one gives a damn about the backlog or what you did yesterday. Hard to believe it really works like this - but apparently it does, and the business now thinks that more developers = more profit, so on they come. Developers hire their own, self-organise and manage their own time. Extraordinary, but a great fun presentation. One interesting point that reflected the keynote was the "cult of the programmer" - a/c Fred, in the bad old days (of waterfall), being a developer was a crappy job (true), and so any decent developer wanted to move on to becoming an "architect". This is no longer the case - as Fred put it - how many surgeons wake up one morning and decide to become a hospital administrator? Developers can now grow old at the keyboard, like geeky rock stars, which is a great thing (IMO).

Bringing Developers and Testers Together with Visual Studio (Giles Davies) 4/10:
Talk about coming down to earth with a bump! I walked into this talk by mistake - a last minute room-change meant that the "Do's and Don'ts on Android" talk I was expecting was in the adjacent room. In contrast to the previous talks (and the keynote) - the word from MSFT was "do you want developers having access to the test environment - probably not!" - cue tumbleweed. This was so at odds with the tone of the day (developer anarchy = profit) that it was hard to imagine it was the same conference. I felt sorry for the presenter - it wasn't his fault, and he did a grand job of demonstrating some convoluted Visual Studio features - I'll give him 4/5 for presenting and the content 0/5. Topped off with some Q&A around licensing. Wrong conference MSFT.

Agile Operations (Dan North and Chris Read) 8/10:
Dan teamed up with (another ex-ThoughtWorker) Chris Read here to demonstrate how it all comes together. Great fun, and a repeat of a (ThoughtWorks) presentation from QCon SF that I watched a while back - basically reinforcing the concept of "Infrastructure as Code" - a lot of which depends on tools like Chef / Puppet / CFEngine etc. Putting everything into version control, rolling out changes to infrastructure like you would with code. The advantage of versioning includes having a comment attached to each change - no more mystery DNS updates! NB - these guys still deploy application code using hand-rolled shell scripts - the Chefification is for the network infrastructure only. Echoed one of the other themes of the day - "optimise for change" - change will happen, so embrace it and build it into your work. Like Etsy - these guys often find it's quicker to redeploy than rollback - sign of a good process IMO.

Making Apps That Don't Suck (Mike Lee) 5/10:
Mike Lee is a legend, apparently. He's built some great apps on iOS, made some money (I guess), and is now travelling around the world dressed as a pirate. He's also a bit of a tit - and whilst I was interested in his views on iOS development, something he clearly knows about, I wasn't remotely interested in his views on life - which were of fortune-cookie sophistication. I have subsequently read that he was fired ( from his own company. No surprise to me.

NASA Keynote (Mark Powell) 6/10:
Second keynote of the day - this from a NASA guy, although not, much to the audience's dismay, someone writing software for spaceships. Still, it was interesting stuff - they clearly have some clever people working at NASA, but generally a downbeat end to the day.