Refer to Martin Fowler's recent post on how top-down imposition of agile is pretty un-agile by definition; he's far more calm and articulate then I am. But as someone who has seen agile succeed and fail, I thought I'd throw my opinion into the ring for a bit of perspective for the recently converted. The greatest success with which I have been involved personally was simply awesome. A true agile poster child project: the brochure for it would tell you that we estimated it almost perfectly, delivered it on time and on budget, have had so few bugs in two years of production use that you'd have several fingers left counting them digitally, and the system transformed the business in a truly fundamental way. And we're not talking about a small project here: 20 people all up for 18 months and several million dollars. This is all absolutely true. What the brochure would fail to mention most likely is that we clashed with several internal stakeholders to the point where I was excluded from the building for a week in the middle of the project, that we needed the best team of developers I've ever seen to pull it off, the best project manager I've ever seen, the best iteration manager, customer representative and analysts and we had to fight off any number of die-hard process nazis who did not share our value system of actually delivering something of value (think Vogons). The brochure would also fail to mention that the organisation that witnessed this startling occurrence was quite committed to hiding this anomalous project like an embarrassing relative; you see, we outsiders had been brought in as the third attempt at this project in three years, and were set up to fail. The brochure probably would not mention that either.

Where I've seen agile fail is quite simply when you don't have all of these things working for you. Like having a distracted customer who can't spend the time with the team. Like having developers who really don't care about the craftsmanship of their work. Like having analysts who just type requirements into documents without any actual analysis. Like testers who just do the happy scenarios because the other ones are too hard. Like having a project manager who focuses on avoiding failure rather than achieving success. The problem with agile is that any one of these can severely damage or even kill your project despite the health of the others.

So the good news is that agile can and does work. The bad news is that agile can and does fail. So what are you to do? Looking back on that successful project, what made it work was, unfortunately, probably the same old things that have made any project work in any human endeavour since human endeavours began; highly skilled and committed people who bust a gut, put their necks on the line and get really lucky as well. And guess what? If you took these circumstances and did waterfall, you'd be just fine. What agile is great at is failing fast and eliminating waste. The feedback cycles are there to detect and correct the course of the project almost as soon as it starts. I'm not sure how common this expression is, but when someone asks you a question to which the true answer will be very upsetting for them, the initial response is often, "Do you prefer the truth or a comforting lie?" That successful project had been estimated prior to my team's arrival. When we took a look, our estimate was 3.5 times higher and the duration was twice as long. We had extreme pressure to just hide our estimate and do the typical budget blowout thing later on. Not doing so and being honest nearly killed the project before it started. Agile is for those who can handle the truth; waterfall is for those who prefer the comforting lie. Either one may be right for you and your organisation.

* your Mac
* your ipod
* Project Gutenberg, my favourite source of great literature in goold old single-file ASCII text
* Cepstral William, the sexiest robot text-to-speech voice around, and
* Audio Hijack Pro to capture whatever William says and turn it into bookmarkable mp3 files to put on your ipod...?

A: A supply of the world's best audiobooks that will last longer than you will, that you can take anywhere with you, enjoy with your feet up and your eyes closed (unless you're driving, of course). But not only that, unlike a normal audiobook that you might buy from Audible, you can crank the speed up to whatever your brain can handle! Speed reading with your eyes shut. Cool.

A few features of the site:
* the constrained box design was chosen by the client so don't blame me!
* funky interactive map on the search page - go Stuart!
* little ajaxy forms to submit feedback without doing page refreshes
* main image on the property listing page changes as you hover over the thumbnails
* client-side scrolling of bio information on the meet the team page

A few 'behind the scenes' things:
* all the images are in a Postgres database but are cached by Rails as they are used
* images are uploaded in hi-res by the client and crunched on the server using ImageMagick to generate the various form factors required for the site - a real time saver for the client
* lots of admin screens for updating listings, team members, etc

A few lessons learned as a J2EE guy working on his first Rails app:
* Rails makes the backend stuff so easy that the pain in the butt becomes all about the HTML and CSS - by far the most painful part on such a pixel-perfect design like this. Normally on a J2EE app you don't notice because everything is that painful!
* Postgres rocks
* Rails caching is very cool. The site warms up nicely
* database migrations are good but you can't sustain writing them using model objects if your model is changing (which it will). So you're basically writing SQL for migrations, but that's ok
* some of the Rails association stuff is a bit weird, but the worst thing is some of the 'silent failures' you get when saving related records. Never a good idea.
* some of the Rails plugins are a bit naive. For example, acts_as_list, which lets a model include positional ordering, doesn't let you put a UNIQUE constraint in the database for the position column as the update code is not transactional. Hope to find some time to fix that one.
* Rails routes rock

So what's the conclusion about Rails? Is it perfect? Hell, no! Is it productive? Hell, yes! Does it beat the J2EE stack for database-backed web apps? By a country mile. I will not choose J2EE over Rails to build such an application, that's for sure. To be honest, it feels a little like the Mac/Windows discussions I find myself having reasonably regularly. To say something is way better than something else does not mean the thing is perfect. It just means it's better. Rails is like that - it's not perfect, it's just way better. I sincerely hope that Rails does not go mainstream or we'll all have to go find something else to be our competitive advantage. May the industry behemoths ignore Rails for many years to come!

But now I have stumbled across an enhancement to the methodology itself. If you're not familiar with it, it's basically a personal workflow system wit major steps: collect, process, organise, review, do. At first it's hard to get your head around how much work you have to do before you do any work! There are also several defined lists that you maintain, such as agendas for each person you interact with regularly, a list of things you're waiting for others to do, a 'someday maybe' list, next actions lists organised by context (such as home, work, online, travelling, out and about) etc, etc. The idea is that you capture everythhing (and I do mean everything), review it regularly and enable yourself to notch forward on many fronts in small increments in a context-optimised way just like a multitasking operating system.

So I offer to the GTD community my personal recommendation to maintain a 'things that shit me' list. Keep it up to date, capture new things into it often, review it every week, and use it as inspiration to make improvements. If you think I'm being flippant, think again. When you're angry, frustrated, stressed,or otherwise unhappy and the solution isn't immediately apparent, it's an effective short circuit just to name it, look at it next to all the other things that drive you nuts, and choose when you have the energy and time to pick some of them to turn into projects in your GTD system for improving your life. There is a very good feeling when you delete something from this list, let me tell you, and that's the real key. It's a smell if all you ever do is add to it, so make sure you pat yourself on the back each week by knocking something off your 'things that shit me' list. You'll be a nicer person to be around and that is probably the biggest benefit of all.

I assume you're interested in an example, so here's an excerpt from my current 'things that shit me' list:

The leaves in the skylight in the kitchen
I don't have enough storage space in the office at home
my house keys don't open the side door
the way my phone only starts telling you the battery is low when it has a minute of life left
I can't use my little dictaphone in bright sunlight 'cos I can't read the screen
the tree that overhangs the carport
I haven't figured out how to do my fortnightly trips with just a backpack
you can't get clip-on sunnies for my glasses
I'm not set up to scan things conveniently at home or at work
the handrail in the bathroom keeps falling off
the laundry door leaked in the last big storm we had
I never have time to play the drums
theres piles of mysterious crap in my book case
i am notsure if priorities in my gtd system should be related to projects or actions
i get so much comment spam on my blog
moving around the screen on the mac when the zoom is on is too slow
i cant print on my home printer when i'm connected to the vpn
the plumbago outt the front is out of control
wikiwords seems to have been turned off in the wiki
ruby barking drives us mad
the hard drive on the mac is just about full
i do not have a functional tickler file at least electronically
i have no way to track how much support work we're doing versus project work
we don't use a single issue tracking system
i can't print from the mac on my home network

"Advance Whose Skin Is Fair"

Australians all let us rejoice
For we are rich and white
With golden soil and lust for oil
We reckon she'll be right
Our land abounds in nature's gifts
We've never learned to share
In history's page let every stage
Advance whose skin is fair

Beneath our radiant southern cross
We'll toil with hearts and hands
To make sure noone different
Sets foot upon our lands
For those who come across the sea
We've prison cells to spare
With courage let us all combine to
advance whose skin is fair
In joyful strains then let us sing
advance whose skin is fair

Copyright © 2006 James Ross. All Rights Reserved. Of course. This machine kills fascists.

We now return you to your regularly scheduled program of agile software development....

Without getting all hysterical about it, there are many things in Rails that should give pause to the J2EE crowd. When I switched from C++ to Java, the thing I liked most about coding in Java was the much improved signal-to-noise ratio compared with C++. I found I spent much less time thinking about computer stuff and much more time thinking about the problem at hand and the concepts of the problem space, largely ignoring the problems of the implementation space. This was partly the language, partly the libraries, and partly the community around Java at the time, but when you add it all up, the result was better software faster, which is really all I want.

Now, having used the classic J2EE stack for a while (Struts/Tiles/Hibernate, et al), and having watched J2EE vendors bloat their products to the point of collapse, using Rails has given me the same feeling. No struts-config.xml, no tiles-defs.xml, no *.hbm.xml files, and much less time thinking about the machine and its implementation layers. The signal-to-noise ratio is simply better than in the form of J2EE I've learned. Ruby/Rails also appear to have the libraries and the community to make the whole stack work. The whole setup feels more supple and malleable than Java, and, since that feeling was what attracted me to Java, that has to be a concern for those who think Java is some kind of universal maximum, rather than just a local one, as all technologies are.

There are some things in Rails about which I retain mixed feelings. ActiveRecord seems to me to be less of an O/R mapping layer, and more of an O/R unification layer. This caused me severe brain ache for a while, but I'm suspending disbelief for the moment. The unit testing model involves database access (and lots of it), which is a definition of unit testing I've always disliked strongly. The functional testing model (so far as I have looked) doesn't use a model based on parsing and filling out and submitting the HTML forms served by the running application. It appears to be more a controller-testing setup than what I would call a functional testing setup. But I'm only a newbie and might have missed some stuff.

So in an effort to find some of the artists that were lacking and to try out the competition to see if they could do any better, I headed on over to Bigpond Music, which has been around for a bit but I never bothered trying. A nice web 2.0 interface, by the way. Now the deal here is that it's a Windows-only system (boo!) that uses the DRM capabilities of Windows Media Player to manage the licensing process. So I left the Powerbook for a minute and clicked away a morre comfortable fifteen bucks to see what Alex Lloyd's new album is like. Astute readers will notice it's only Australian musiic involved here - I can get your yankee junk on bit torrent and still sleep. That's a joke, by the way (also of an Australian variety)

Twenty minutes later and I had an absolutely pristine, fart-free version of some fine Aussie rock with which I am well pleased. I can even burn a CD and rip it into iTunes where it belongs. Oops. Did I say that out loud?

So as much as it pains me to say it, the Apple experience was not one I am likely to repeat in a hurry. Bummer.

Updated 2005-12-4: Apple got back to me after three and a half weeks to say they were looking into it and were planning to refund the purchase of the album. So I still don't have Paul Kelly's new album like I'm supposed to. Makes a visit to the bricks and mortar store look quite attractive at this point.

You say to your friend, "Hey, don't you want to go into second gear?"

But your friend says, "Don't interrupt me, man, I'm going as fast as I can and I really need to concentrate here"

You try again. "But we'll get there faster if you change gears!"

But your friend says, "No we won't! I tried that before and putting in the clutch __definitely__ slows you down, silly! And the last thing I can do now is slow down when we're running so late!"

The problem with development teams is that you can't actually hear the engine screaming and see the car shaking and feel the lack of speed. But it seems to me a lot of development teams are flat out in first gear. Agile development has a lot of practices that are all about changing gears.

But the biggest issue is (as usual) not a technical one but a human one, in this case the implicit belief that requirements can be 'captured' if you just think hard enough (and long enough) as opposed to the agile view that requirements need to be trawled for constantly, starting at a high level and increasing the resolution as implementation approaches, leaving the resolution lower for far-off things.

I unwittingly stumbled on a requirements parable the other day when, of all things, I attempted to buy a new pen. I succeeded in purchasing a pen, but I failed to get one that met all my requirements. I'm serious; I couldn't even buy a pen for myself that satisfied all my requirements. Read on if you are still on your chair.

So, why did I need this pen? I carry around a pocket notebook and a pen wherever I go, having learned from experience that freeing my brain from minutiae leaves it free to dream and create. I've tried other tools, PDA's, dictaphones, etc, but the good old pen and notebook just works. The definition of 'works' here is that the time and effort between being confronted with something that I need to remember but that I don't want to deal with emmediately must be minimised. I even have a kind of code for the notes I take to speed it up, and recently spent half an hour in a stationery shop buying just the right notebook (with high end features like one of those built in straps to mark your page). So I'm pretty serious about this pen. It has to be able to be connected to the notebook firmly so they can't be separated accidentally, and it has to have a broad enough point that my dim eyes can see what I've written.

So how did I go wrong? Well, the pen I've used for a long time started to become unreliable, so I had to replace it. It literally did not occur to me that the fact it was one of those ones that you press the top to extend the ballpoint was important. I was thinking maybe I'd try a felt tip pen rather than a ballpoint as the nibs come in wider formats. So I looked at the mountains of pens in the store and picked a broad-nib felt tip number with a clip that attached snugly to my notebook. So how long did it take to realise my error? The first thought I had that I wanted to write down, I pulled the unit out of my pocket, separated the pen and instinctively went to push the button to extend the ballpoint with one hand while flipping to the current page with the other. Only there was no button. There was a cap! That requires two hands (or one hand far more dextrous than mine) to pull off and put somewhere else, probably on the butt of the body of the pen. I was stupefied. I was now thinking about the damned pen and not the brilliant idea I'd just had.. If there is something you just want to stay out of your way, it's the thing between you and your great ideas.

So what's the moral of the story? I guess it's something like this: Assuming you are capable of defining requirements up front without exposing users to the solution is bogus. Even if the system is well established. Even if it's as simple as a pen. Even if you're so clever that you can read the customer's mind. :P

Anyway, I'm showing my age by saying it's 19 years since I last had a Mac, so I knew things would have changed quite a bit in the meantime. I've been reading 'Mac OS X The Missing Manual' while waiting for my machine to arrive so I already had some of the keyboard shortcuts down before I had the box open. Which brings me to the first of my observations about the oft-repeated cries of "It just works" by the MacCult members...don't get me wrong, this little baby is a beautiful thing, but in the interests of full disclosure, I thought I'd jot down my notes on the switching experience...

Getting the machine delivered was something Apple found quite challenging. I ordered the PowerBook, MS Office, iWork, .mac and Logic Express, and they got off to a bad start by shipping the software individually so that a couple of times when the courier's arrival at my house found nobody at home, they went away until we called them and could commit to waiting at home for a full day before they'd deliver. Not so bad for a single delivery, but when the order is part-shipped in four separate trips, this gets a bit old. Of course the software was no use without the machine, which didn't turn up at all. Problem was, the online order status page showed it as having been shipped many days ago. Several calls to the courier and Apple finally led to the discovery that the courier had returned it to Apple because the exterior box was slightly damaged in transit. Still no drama, except that Apple seemed to not have a process to keep it moving. I had to call them multiple times to get multiple people to find the box and ship it to me after testing that the machine was OK. Very annoying.

The machine arrived last Friday and with sweaty palms I fired it up. The warm, welcoming and soothing setup process came to a shuddering halt at the wireless network setup screen. No matter what I did, I could not get it to connect to my network. Now I know my 128-bit WEP key off by heart, so this should not be hard. Eventually I gave up in disgust and told it I didn't have a network at all. When I got into the OS proper, I found I had extra options, including to change its default WEP key size of 40 bits to the 128 I needed. Up and running immediately, but obviously unimpressed with the unnecessary dumbing down of the setup program.

A little later on, I cracked open my iWork 05 disc and stuck it in. Lo and behold, the machine refused to install the software because a later version was already installed on the machine. So I go hunting and find a trial version of iWork, which I start and am annoyed to find there is no 'enter the serial number from your boxed copy you just brought home from the store and I'll start working for you' option. So I drag the application to the trash, since that's what book said was all that was needed to uninstall an application. No dice. A call to a Mac buddy and a little Spotlight magic later and I find a couple of files in the /Library/Receipts folder, which has something to do with the auto-update feature for the software. Killing off the files and now I can install my software. This is software I bought with the machine, you realise. And the machine won't let me install it. This is not what I call 'It just works'

The rest of my installs go pretty much according to plan (after Office downloads the 50Mb of updates needed to actually get it to run properly). The only other disappointment was the Logic Express has not had the latest UI goodness like the rest of the apps that sport the Apple logo. They've done the box up nice, but the app must be one of those 'carbonised' ones they talk about.

Of course, before I could really start using the apps, I had to sort out the accessibility options. Being a visually impaired person, one of the things that tipped me to considering the Mac was the built in accessibility features, particularly the new VoiceOver software that speaks the UI to you as you use the machine. Annoying as all get out to anyone standing nearby, but pretty darn useful for me. In fact, as I type this, it's reading each word as I go. Overall, the software is good, and I can save the money I'd normally have to spend getting set up with specialist software, so I'm happy about that. It's got a few minor issues like jumping around a little crazily when the screen is magnified and the focus shifts, and not reading full paragraphs of text in Safari if they contain a (gasp!) hyperlink, but not too bad at all.

Things really started to crank when I got my head around expose, spotlight and some of the apps that come with it. iMovie is awesome in its elegance and simplicity and speed (I talk as someone who purchased Pinnacle video editing software for Windows) and iPhoto blew me away when I imported my six years worth of photos from its home in a nicely structured directory, and iPhoto turned the directory names into keywords that I can search, so 'Antonia's Wedding' instantly finds all the right pics. Awesome.

I find I'm not as productive when typing and editing text because I don't know all the keyboard shortcuts for selecting a word or a paragraph, but that will come with practice. It is true what they say that it just feels solid, and having a large number of apps open doesn't seem to have the same drag effect as it does in Windows. Now I just have to figure out how to replace all the apps I rely on like The Brain and Microsoft Money. I'm not in a great hurry as I invested in Virtual PC to ease the transition. I haven't actually used it yet, as I'm trying to just feel the vibe for now. And I must say it feels pretty gooood. I look forward to being snobbish and dismissive of my Windows colleagues very soon.

Having finally succumbed to the weight of my geek buddies buying Macs, I've just ordered a new Powerbook, so I reckon I'll be posting a bit about the experience. I've had to buy a copy of Virtual PC to ease the switch, but I'd like to retire all my Windows apps if I can.