My hovercraft is full of eels

The first time I met James I was amazed to see him sitting mere inches from the monitor. As I moved closer to see what it was all about I saw the text and graphics on the screen zooming in and out. It was enough to make my stomach turn.

James is legally blind. It's actually not as bad as it sounds (it does come with some perks like free public transport hahaha) and he uses a great bit of software called ZoomText to magnify the display. But the best bit is the Text-To-Speech (TTS) feature. Highlight a paragraph of text (for which there is a short-cut key combination) and Davros reads the text back to you. You can also adjust the speed at which text is read back. I can just understand it at about medium pace but James has it turned all the way up to Chipmunks.

Recently, James wondered what source code would sound like and, to his delight, discovered that not only did ZoomText read the source code back as intelligibly as (perhaps more so than) a human, it even understood CamelCase text, separating the words appropriately.

Like most developers, I guess, I've always tried to adhere to one of the longest serving mantras of the development community - to write readable code. As you've probably guessed, James and I pair program quite a bit and now that he uses ZoomText to read back our code, never has this meant as much to me as it does now.

As I'm coding away now, there is this little voice in the back of my head (yes I am nuts but that's a different voice) constantly reading back the statements, constantly reminding me of what it will sound like when James uses his TTS. And I think it's made a difference. I think because of it I've changed the way I name methods, classes, etc., and even the way I structure statements, hopefully, for the better.

Posted by Simon Harris at 11:04 PM | Permalink | Comments (4)

In all my years of software development, I have honestly never encountered a developer who really just wanted to be a drone. Someone who wanted to code directly off the spec without a care for what they were doing or why. Until today.

But then it's not so surprising when they're given advice like "Learn from me. Always remember my 3 rules: I didn't do it; It's not my area; and I don't know anything about it."

Surely it's not too much to ask that a developer actually understands the rationale behind the code they are producing. Surely it is reasonable to assume that a developer has questioned the design and implementation to the point they are at least happy that it conforms to their understanding of the problem domain and that it is traceable to some functional spec.

How can so many people have such a low care-factor for the work they do and the software they produce?

Posted by Simon Harris at 09:27 PM | Permalink | Comments (16)

I received a very interesting post from the JESS mailing list last night and thought it was worth a mention.

There's a weird threshold that can occur with any Java data structure that uses large arrays of Object references (Object[]). If the size of the Object array exceeds the maximum size of objects that can be placed in the "new generation", garbage collection performance can be severely impacted...

I'm not sure if I'm likely to see this problem really but I do ue HashMaps a fair bit so I thought it was interesting. In general I don't find the need to use arrays much if at all these days. In fact, for some inexplicable reason, I tend to use LinkedList over ArrayList and, except for Simian which uses lovingly hand-crufted data structures, I can't recall ever holding maps of data large enough to exhibit this behaviour. But then again I'm not implementing a Rules Engine.

Posted by Simon Harris at 10:36 AM | Permalink | Comments (2) | TrackBacks (1)

As Business Rules come into vogue (again?) and the tools proliferate, there will be the usual fumbling about as many come to terms with what it all really means. How do we use these things? What should I look out for, the pitfalls, the traps? Are there any "patterns"? But above all, the greatest difficulty it seems, is coming to terms with the idea that Rule Engines ARE NOT procedural scripting languages.

The Rete Algorithm (pronounced REE-tee and Latin for net) was developed by Charles L. Forgy at Carnegie-Mellon University in the 1970's and is used in most modern high-performance rule engines. Rete is able to efficiently handle very large numbers of rules.

One of the most important features of the Rete algorithm lies in its ability to identify and subsume rules with similar predicates. Because of this, predicates need only be evaluated once. This differs from procedural (java coded) rules where every predicate in every rule must be independently evaluated, regardless of whether the same predicate might already have been evaluated in another rule. It can also locate conflicting rules. Something that's almost impossible in traditional, procedural, languages.

When it comes to codifying business rules, well factored Java code can be rather difficult to understand. After a couple of weeks away, it can often take the original developer some time to get back up to speed with their own code, let alone someone elses. On the other hand, Rules are declarative statements of fact. That means no trudging through tens or even hundreds of lines of procedural code to understand what will happen under various conditions. Weeks, months or even years later you can go back to the rule definitions and immediately understand their meaning and intent.

Rule engines share much in common with Relational Databases. They are based on tuples and predicate calculus. You don't navigate Relations (Tables), you join them. Similarly you don't navigate facts, you join them. Both suffer (or at least have suffered) similar problems in terms of performance and optimisation.

Business Rules should be simple and atomic. They should make inferences. They should not be calling out to databases nor making countless remote calls. That's what application code is for. Much like the difference between queries and stored procedures.

Analogies aside, the fact remains (no pun intended) that rules are not procedural, they are declarative statements of fact! Writing business rules requires very clear, concise and logical thought, as much if not more so than procedural code.

Rule-flow, priority, salience, etc. are mechanisms that allow some degree of procedural control and should therefore be considered a last resort, not the basis for a rule engine framework. While sometimes useful, all are frowned upon by rule advocates in much the same way as OO design frowns upon public variables.

If you can't or won't make the necessary shift from a procedural to a declarative mind set then I suggest you try BeanShell, Rhino, Groovy or any of the myriad scripting languages available. There is nothing to be ashamed of with this approach but it is most certainly NOT the same thing.

Posted by Simon Harris at 10:06 PM | Permalink | Comments (1) | TrackBacks (3)

Desktop software products, especially of the Windows variety, invariably come with an "About" Dialog listing, among other things, the version of the software. The product version number helps support staff and developers solve problems when they occur out in the wild. Without a version number, tracking down a problem can often be rather difficult.

Especially on web-based applications, making the product build number, build date, and other configuration information (was it a production or a training build, etc.) accessible to the end user is an invaluable aid to developers, testers and support staff. In fact of you look to the side-bar on this blog, you'll find the version of MovableType used.

On our last project we made this meta-info available as, funnily enough, META tags (though we could just as easily have used comments) in our JSPs and HTML files. To source the info we simply passed the CruiseControl build number and date through into our Ant scripts to use as replacement parameters when copying the Struts (*sigh*) application.properties file into the web archive. The person responsible for deployment can always be sure that the correct version of the application has actually been deployed. Then, when testers and users need to report a problem, they simply view the source for the page they are on and, hey-presto, there it is.

On a project I worked on with Dave some time back, we stored the current build number in the database as well. The build number was inserted into a table at the end of the database schema update scripts. A DBA could then visually inspect the data in the table to ensure the correct updates had been applied. Our update scripts also checked this table to ensure a script could not be applied again. At runtime, the application would also double check that it was running against the expected database schema ensuring we never had a, possibly catastrophic, mismatch.

You can add build watermarks to templates used to generate documents such as PDFs, to XML messages sent between systems, in emails as header tags, you name it. In fact anytime traceability back to a particular version of an application might be useful, consider adding some kind of meta-information about your application. Your support staff will love you for it!

Posted by Simon Harris at 07:50 PM | Permalink | Comments (2)

It's amazing how after just 3 days on a motorbike, out on the open road, breathing in clean ocean air, all the worries of the world seem to dissapear; how the old "care-factor" rapdily falls away to hover tantelisingly above zero; only to be brutally re-awakened by my brain hitting the ground with a thud as I begin day one of a new, 3-month long, project. And what a project it is!

The last project I was on kept my brain working overtime. I had to push myself to stay afloat, to keep up with the brains on the team. It was fantastic. We all pushed each other, striving to get as close to the "ideal" solution, continually honing and refining the design to remove all, seemingly, unneccessary fluff. The Simplest Thing That Could Possibly Work While Still Being Good Design ™ was King.

But now, as if by way of some bizzare super-natural ying-yang thing struggling to balance the forces of nature, I find myself on a project where the Powers That Be have skillfully managed to find complexity where there rightfully should be none.

And yet I'm baffled that, at some level, it's a very simplistic design. A design that even with it's strange technology choices and inexplicable coding "standards", requires very little brain power to comprehend. I suppose that by adhering to all those wonderful™ J2EE Core Patterns, it can't help but be simplistic. It really is the epitomy of a Simple Arrangement of Complex Things. Need a service? JNDI is your friend. And for your troubles you'll get back an EJB. Need to extract a piece of data from some XML? Here's a static helper class we prepared earlier.

The other developers on the team (I've come in 3/4 of the way through) have no trouble wading through the hundred+ line methods that make up the half dozen or so EJBs mixed in amongst the various classes that together comprise the 20 or so WSAD "modules". In fact I found myself in a meeting honestly feeling stupid that I just didn't "get" the seemingly overly complex application architecture. I figured I must be missing something, something significant. And maybe I have.

With the exception of my good buddy Phil, it made perfect sense to all the rest to store XML as a CLOB in a relational database only to unpack it, extract a subset of the fields and use Lucene to index the records. No one else seemed to think that spawning threads inside the app server was a bad thing. I mean "we don't want to re-invent MQ". Of course not. Silly me. Are you sure 15 days isn't too fine-grained an estimate for you? We still have 6 weeks left, surely your gant chart would look a lot simpler if we just made everything finish then. :-P

I'm not sure if I simply suffer from a particularly severe form of geek snobbery (in the same way that Linux users look down on Windows users) or perhaps I expect too much from my work. Whatever it is, it's interesting to see another perspective on J2EE development. I will try to suspend for a while my preconceptions and at least give it a go. But I'm not giving up IntelliJ...yet!

My only consolation is that it will surely be an intellectual walk in the park and give me more than enough food for blog, not to mention oodles of free cerebral cycles to work on my various out-of-hours pet projects - the ones I had been neglecting for the past 6 or so months.

Posted by Simon Harris at 10:28 PM | Permalink | Comments (5) | TrackBacks (1)

James mentioned that one of his colleagues had been compiling stats on various projects and how they rated in terms of Cyclomatic Complexity (A.K.A. McCabe's Complexity). Quite rightly, everyone on the team was ecstatic to find that the project ranked top of the list (ie. lowest average complexity) in a field of various open and closed source projects.

Tools such as Checkstyle and PMD (to name but a few) make collecting these kinds of statistics very easy. What's more, incorporating these tools into your developer and continuous integeration builds makes enforcing limits on whatever metrics you like almost trivial. And that is exactly what we had done.

Comprehension and testability of code vary with complexity. Overly complex code is difficult to maintain, difficult to test and more likely to contain bugs (hidden flaws). Experience on previous projects (rather than just theory) had taught us that a low value for cyclomatic and it's cousin NPath complexity was indeed a good predicter of problematic code. On these projects, we had always cranked the threshold right the way down and had found this to have a positive influence on the code base.

In true Pavlovian style, and with nothing but ultruistic intentions, we decided to make cyclmatic complexity (along with dozens of others) a constraint on this project as well. It obviously follows then that our project would place at or near the top of the list. We had made it a constraint, enforced it in our build and therefore the only possible outcome was a low average.

But as always, the devil's in the detail. It soon became apparent that like so many other checks, it is possible to subvert the process. The worst part is that few developers do so out of malice. More often than not a check will fail and the developer will (often quite ingeniously I might add) code their way around it. Without knowing any better, it is quite easy to inadvertently increae actual complexity whilst adhering to simple metrics.

I have a love-hate relationship with development tools. In the right hands they are invaluable but you can't just give inexperienced developers tools and expect them to do as well. Tools + Monkeys != success. In fact as we discovered, left unchecked the opposite is much more likely to occur. Luckily for us we had a much better ratio of experienced/inexperienced developers so we caught most of the problems early and entered into intense education (ala Clockwork Orange - just kidding). I think it's safe to say that in the end, the benefits certainly outweighed the potential dangers.

While certainly interesting, relying on a single metric can be very dangerous. Blind adherence to any number of metrics can disatrous. In general it's often far safer to let tools do the grunt work to find the smells and even make recommendations, then employ the expertise of more experienced developers to deoderise. It's also a good idea to use several metrics in combination that address different, sometimes competing, concerns.

Posted by Simon Harris at 05:41 PM | Permalink | Comments (11)

All good things must come to an end they say. Tomorrow (or today I guess) is the last day on my current project. It is no exaggeration to say that it's been one of the two best projects I've ever worked on.

After six months of development I've managed to see it through to code freeze and although I'd have liked to see it go live, that's a luxury I will unfortunately miss out on this time. As strange as that sounds, there is nothing like nursing your own creation into production. I've always found production support of my own baby far easier than the actual development effort and certainly easier than supporting someone elses. Watching real, live users pounding away at your product is always good for the soul in what can often be a soul destroying lead-up hehehe.

I've laid my fair share of turds in the code base, I just hope I've obfuscated them well enough. Everyone has worked really hard to keep the code as clean as possible, employing as many checks and tests as we could. There are always things you wished you could have done differently but all in all I think it will survive the ultimate test - maintenence.

Technically, we used constructor based IoC really effectively. We have pretty good unit test coverage. Integration tests are up there too. Functional test coverage is arguably too good but hey I'll live with that. We have out of container tests, in container tests, automated builds and deployments, code checks, cruise control, you name. Thankfully no one managed to find a way to squeeze any AOP in though they gave it their best shot :-P. In the last couple of days I've even been seen partaking in the writing of perl scripts. Oh and of course nothing gave me more pleasure than to see all 10 developers happily writing declarative business rules!

The people I've worked with have been just sensational. I've certainly learned a lot more than I've given especially from people like Mad Dog Murdoch, Uncle Dave with his little nuggets of wisdom (or as he would put it "corn in the stool") and of course JRO who never ceases to make me realise why I'm just not cut out to be a "Technical Architect". Marcus, Trashy, Triggles, Phil, Jimmy, Happy, Big Daz, Bazza, Victor and Vorgahan not to mention my backup singers Andwea and Neats. I don't know how they did it but they all managed to endure my endless singing (if you can call it that) interrupted by bouts of ranting and the occasional tap dance.

It was gratifying to see customers, testers, BAs, developers and management all working together to get this thing out the door. It hasn't always been easy but then "nuffin is". In the end though I think pragmatism (along with threats of violence - just kidding) won through and we all just worked to ensure we got the job done.

I've quite a bit to digest and hopefully blog about in the coming days and weeks but before that I'm going to get my last 8 (ahem 10?) hours out the way then jump on my motorcycle for a couple of days and enjoy the sunshine that seems to be blessing this lovely city. Come Monday it's back to work on a new project. For as a wise developer once said "It ain't all sunshine and party pies."

Posted by Simon Harris at 02:02 AM | Permalink | Comments (2)

I was interested to see Martin Fowlers recent entry on Notifications. If you've ever used Struts (gasp) or similar "framework" you'll already be familiar with the concept so it's certainly nothing new but Martin has a fantastic ability to document and explain things in clear, unambiguous terms.

The most interesting thing to me was this statement "You should use Notification whenever validation is done by a layer of code that cannot have a direct dependency to the module that initiates the validation." and how this relates to the use of a rule engine within an application.

One of the biggest mistakes we've seen in using rule engines is to allow the business rules to become dependent on other than the business domain. For example, allowing rules to know or depend on what screen is currently displayed. Business rules should be statements of fact about business information not application workflow or navigation state. As much as possible, we want business rules to survive changes to the form, flow, layout and even number of application/s that depend on them.

Business rules make inferences about the business information (facts) presented to them. Some of these inferences will be new facts for other rules to consume and some will be facts for the caller to consume. It is this second class of facts which we classify as Notifications and that which the application collects and proccesses. At any point in time, some of these notifications will be relevant to the application and some will not. Some will cause the application to alter it's workflow, screen layouts, etc. and some may safely be ignored. The critical thing to understand is that it is the applications responsibility to filter the notifications.

For example, imagine we have an application that collects data on a customer. The data is collected over N (where typically N > 1) screens according to the business workflow requirements. After each screen of data is collected, the user hits Next to proceed at which time the state of the domain is asserted into the rule engine, the rules are executed, and the notifications processed. Now lets imagine that one of the rules states that a customers date of birth is required. You'll note there is no mention of a screen here meaning that until the date of birth is filled out, the application will receive a notification indicating some problem with that field. Rather than emed knowledge of the application into the rules, the application instead filters out any notifications that are not relevant. In this case by checking to see if the field specified in the notification exists on the curent page or not. If after filtering, there are no notifications, the application can proceed to the next screen; otherwise a message is displayed and the user cannot proceed. Finally on the last page, the application can check to ensure that there are no unfiltered messages before allowing the user to save. You can even get fancy and have the application take the user directly to the appropriate screen, something that would be difficult to achieve if the business rules were dependent on navigation state.

Another area we have used this approach is with authorisation. We have rules that assert Permissions (a type of Notification) based on the business data that the application can use to determine what a user is or isn't allowed to do. Again, the rules make no reference to screens or assume anything about the calling application for that matter. They simply state the facts as presented and inferred. The application then checks the results for the existence of the desired permission and if present the user is allowed to proceed; if not the user is denied access to that particular function. This also makes rendering links, enabling/disabling buttons etc. very easy while maintaining the ability to define the rules in purely business domain (ie application-agnostic) terms.

The concerns of a client application are typically to do with workflow and appropriate use of screen real estate. Business rules on the other hand are concerned with statements of fact about the underlying business data. Notifications allow the business rules and application workflow to vary independently according to these different concerns.

Posted by Simon Harris at 04:29 AM | Permalink | Comments (2) | TrackBacks (2)

Now for something completely geeky so as to fly directly in the face of my previous post, I quite literally just received notification that a Java feature request I had voted for (along with 861 others lol) has finally been closed off.

It may not seem like much but I've quite often felt the desire to further restrict the return type of an overidding method and soon, according to the release notes at least, I'll be able to do just that in the up and coming 1.5 ahem 5.0 release of J2SE.

A clear candidate for this is clone() meaning I will finally be able to do:

public ThisClass clone() {
    return (ThisClass) super.clone();
}

No longer requiring callers to perform the cast themselves.

Of course removing redundant casts is but the simplest and most obvious use. Anyone who's ever been forced to code on top of a "generic framework" implementing/overriding Object doStuff(Object) throws Exception methods will know what I'm talking about :-).

I guess it comes down to your preference for static versus dynamic typing in many cases and while I'm not averse to a degree of dynamic typing, I do like it when my APIs can be more explicit without the need for all that useful™ JavaDoc.

Posted by Simon Harris at 02:26 AM | Permalink | Comments (0)

I recently attended a half-day seminar thingy put on by Enterprise Java Victoria. During the day we had presentations from Mark Hapner and Gavin King along with a few vendors and some large-scale J2EE shops. Robert Watkins, James and I (apparently I talk too much and need to learn to shut up - like you didnt know that already) were among a dozen or so people fortunate enough to be invited to sit on various panels. All in all it was well attended and I think most people got something out of it.

Most of the talk centred around EJB Three-Dot-Oh and Es-Oh-Ay but there was also discussion on topics such as developer training, heavy versus light-weight containers, Java Web Start, etc. During all this, the one thing that struck me was the distinct lack of audience participation. We (the panels) were pretty much there to give our views (such that they are) on the future of J2EE but no matter how much I or Gavin or anyone else ranted, I kept wanting to ask the audience "so what problems are you having with J2EE?" I'm astonished the question never came up. Then it occurred to me that these people were here to find out what the next big thing in Java was going to be. And therein lies the problem.

If you've realised anything from reading this blog (apart from the fact that I'm a loud mouthed opinionated git) it's that I really do feel we as an industry (being IT) do more to justify our own existence and "innovate" than to actually think about what our customers need. I love playing with new tools and frameworks and whatever else seems to be in vogue at the moment as much as the next developer but I also feel we (yes that includes me) persist in creating the technology equivalent of whiter toothpaste and softer toilet paper. FUD drives our industry as it does fashion. Vendor X or "Thought Leader" Y come up with some new fandangled tools/methodologies/practices. Quite often its not even new. Then these ideas are disseminated to the masses (being we the developers) in IT shops who then, upon seeing this bright shiny new toy, go about selling it to the business to justify the R&D required to work out how we might actually use it. What's more we often shoot ourselves in the foot by over promising and under delivering.

Pause to blow off steam...

Why does the customer care about the latest technology unless it specifcally addresses some business need? "Why do they even care if it runs on Java or not?" was one of my questions for the throng and didn't that go down like the Mars Beagle Lander on a bad day! Surely the customers have problems not to do with technology but to do with the way they conduct business. They want to do business more effectively. They want to be more flexible. Shouldn't we be trying to find out what they do and help them do it better? Feel where their pain is, then go away and work out how we might make it go away. In doing so we would encounter a world of hurt and pain which should lead us to ask the tool vendors to help us out. Instead we end up with BEA telling us that what we really need is to turn everything into a web service and let business users clickedy-click to put together an application.

"So," I asked the audience, "who here has built their own house or their own car?" amazingly one person replied "yes" to both questions. What about build your own furnitiure or made your own clothes or even serviced your own car? The materials are all readily accessible. Do commercial pilots construct their own planes or astronauts their own rockets? Surely they could, they're some of the most highly trained people around but that's not their job, that's not where they add value.

Just take a look at SourceForge. We are very good at building IDEs and frameworks, etc. Well maybe not the last but certainly IDEs and tools in general. Whether you like it or not, Visual Studio really set the benchmark that all subsequent IDEs had to match and, thankfully, have now surpassed. It's no surprise that IntelliJ and Eclipse , etc. are so good. We as developers understand developers. We understand how we work. IntelliJ doesn't get in my way. It does just what I need and no more. It takes away the tedious tasks without trying to do my job for me. What we need is to build end-user software with as much thought as this. Not try to solve cool networking and infrastructure issues over and over and over again.

Sure, SOA and interoperability are laudible goals but they're hardly new. Whether you like it or not, EDI has been around for decades, CORBA for longer than RMI, telnet longer than HTTP. I'm happy to listen to our Thought Leaders tell me their vision for computing but don't try and convince me it'll solve a bunch of problems that quite frankly I just don't have right now.

I've taught martial arts for over a decade now and it's only just occurred to me that some people turn up because they want to be told what to do. Maybe they lack the capacity to learn for themselves or maybe they just don't want the responsibility but if I spend all my time teaching and none of it training, my reflexes get slow and I forget all the little things that one learns when actually forced to use technique in a meaningful way.

Neccessity they say is the mother of invention. If our Leaders long ago stopped actually dealing with customers and developing real code then they also stopped feeling the "pain" associated with developing business software. No amount of thinking makes up for this.

Posted by Simon Harris at 01:32 AM | Permalink | Comments (5)