My hovercraft is full of eels

This morning I went to push out a new version of an application to our staging environment on an Engine Yard slice. I knew I had done exactly that last night so I navigated through my bash history and hit enter. Two minutes later the new version had been deployed and I was about to walk out the door to do some chores before coming back to start playing with the app. Thankfully, Steve came online and informed me that production was broken.

A quick look through my bash history and it seemed I'd used the deploy to production rather than deploy to staging but, being in a hurry, hadn't looked carefully enough. Of course some might argue that I should have looked more carefully. That I shouldn't have deployed before heading out. All valid points but I very rarely have full control over what's going on around me. So, while it's all very well and good to hope that I will be more careful next time, that's a bit like hoping global warming isn't a reality: we all hope it's not but maybe we should do something about it just in case?

And so I added the following at the start of the :production task in my deploy.rb file:

  unless Capistrano::CLI.ui.agree("Are you sure you want to deploy to production? (yes/no): ")
    puts "Phew! That was a close call."
    exit
  end

For all other environments, the deployment goes through without question. Attempt to deploy to production however, and I'm now forced to be explicit about my intentions.

Posted by Simon Harris at 11:41 AM | Permalink | Comments (1)

When writing code that is largely algorithmic, I find I end up writing specs that sit in a loop repeating the same operations over a set of data.

This works well enough but it has the downside that the tests abort as soon as a single failing case is detected which can lead to vicious cycles of fixing one case only to find you've broken another.

The solution is of course to break out each expectation--inputs and expected outputs--so they are all run and reported individually. Doing so by hand however, is tedious to say the least so why not generate them on the fly instead:

describe Fibonacci do
  [[0, 0], [1, 1], [2, 1], [3, 2], [4, 3], [5, 5], [6, 8]].each do |input, output|
    it "should generate #{output} for #{input}" do
      Fibonacci.calculate(input).should == output
    end
  end
end

This is such a elegant solution I'm not sure why it only just occurred to me.

Posted by Simon Harris at 03:51 PM | Permalink | Comments (6)

If you're thinking about updating to Rails 2.1 to get the timezone support, you'll need to update all database records to UTC. Here's a quick migration script to do just that:

class ConvertTimestampsToUtc < ActiveRecord::Migration
  # Assume all times were in UTC+10:00
  OFFSET = "interval '10 hours'"

  # Adjust any date/time column
  COLUMN_TYPES = [:datetime, :timestamp]

  def self.up
    adjust("-")
  end

  def self.down
    adjust("+")
  end

  private
  
    def self.adjust(direction)
      connection = ActiveRecord::Base.connection
      connection.tables.each do |table|
        columns = connection.columns(table).select { |column| COLUMN_TYPES.include?(column.type) }
        updates = columns.map { |column| "#{column.name} = #{column.name} #{direction} #{OFFSET}"}.join(", ")
        execute("UPDATE #{table} SET #{updates}") unless updates.blank?
      end
    end
end

As you can see, I've assumed that the dates were previously stored as AEST (UTC+10:00) so you'll likely need to adjustthat and I'm also assuming PostgreSQL for date manipulation though it should be pretty simple to convert to run under MySQL. It may even work asis.

Posted by Simon Harris at 06:22 PM | Permalink | Comments (0)

This morning I had occasion to deploy a branch of a git repository to a staging server but hadn't the foggiest idea how. A quick search through the capistrano source code revealed that I could use set :branch "branch_name" in my deploy script. I tried it and it worked. I then figured I would need to make a similar change across all my branches. Of course, I'm a lazy sod and wondered if there wasn't a better way.

If you're not familiar with git, the output of the git branch command is a list of branches with an asterisk marking the one currently checked out on your local machine. For example:

$ git branch
* drupal_authentication
  fragment_caching
  master

So, I figured, what if I just parsed the output and searched for the branch marked as current:

set :branch, $1 if `git branch` =~ /\* (\S+)\s/m

Now I'm able to deploy whatever branch is current on my local machine from a single, shared, deploy script.

Posted by Simon Harris at 09:07 AM | Permalink | Comments (1)

Update: Ruby 1.8.7 also has this.

Ruby 1.9 has it but if you're not that bleeding edge, you can have it now:

class Array
  def index_with_block(*args)
    return index_without_block(*args) unless block_given?
    each_with_index do |entry, index|
      return index if yield(entry)
    end
    nil
  end
  alias_method :index_without_block, :index
  alias_method :index, :index_with_block

  def rindex_with_block(*args)
    return rindex_without_block(*args) unless block_given?
    
    index = size
    reverse_each do |entry|
      index -= 1
      return index if yield(entry)
    end
    nil
  end
  alias_method :rindex_without_block, :rindex
  alias_method :rindex, :rindex_with_block
end

If you're using Rails you can substitute the two calls each to alias_method with a single call to alias_method_chain.

Posted by Simon Harris at 03:01 PM | Permalink | Comments (0)

Yes, it's true, Scriptaculous already provides a Sortable that makes it almost trivial to enable drag'n'drop sorting of your HTML lists. Whenever an item is moved an onUpdate() event is called (if provided) allowing you to inspect the new order and presumably perform an AJAX request to record the change. In principle, this sounds great but I've never really liked it for a couple of reasons.

For a start, if you have any appreciable number of items updating each in the database just to re-order one seems somewhat unnecessary. Not withstanding the fact that we need to send all those ids to the server in the first place.

Secondly, if you're doing any kind of filtering, it's difficult at best to take the newly constructed ordering and apply that at the back-end; what happens to all the items that may be lurking in between that aren't presently displayed?

Enter Prioritizable (itself built on top of Sortable).

You use it in much the same way as Sortable with the major difference being that the onUpdate() event is called with three arguments: the item that was moved, the sibling relative to which it was moved, and the relative position ("higher" or "lower"). And, if like me, you're feeling a bit RESTful, it's pretty easy to turn these arguments into a nice semantic URL and parameters as shown:

Prioritizable.create($("chores"), {
  onUpdate: function(item, position, sibling) {
    id = item.substring(6);                           // "chore_17" => "17"
    sibling_id = to.substring(6);                     // "chore_2" => "2"
    url = "/chores/" + sibling_id + "/" + position;   // "/chores/2/higher"
    
    new Ajax.Request(url, {
      method: "post",
      parameters: { id: id }
    });
  }
});

When the onUpdate() event is called we POST the id of the item to be moved to a path constructed from the id of the sibling and the relative position. Assuming the the user moves chore_17 just above chore_2 we would POST "id=17" to /chores/2/higher.

In practice, I combine this client-side behaviour with some server-side code that provides move_higher_than() and move_lower_than() methods that efficiently handle all the necessary database updates.

All the pieces mentioned will eventually be available alongside Cogent's other Rails plugins but until then, here's enough of the Javascript side of things to get you going.

var Prioritizable = {
  create: function(element) {
    options = Object.extend(arguments[1] || {}, {
      onChange: Prioritizable.onChange,
      onUpdate: Prioritizable.onUpdate  
    });

    Sortable.create(element, options);
  },

  destroy: function(element) {
    Sortable.destroy(element);
  },

  onChange: function(item) {
    Sortable.options(item)._item = item;
  },

  onUpdate: function(element) {
    options = Sortable.options(element);
    item = options._item;
    options._item = null;

    sibling = item.previous();
    if (other) {
      position = "higher";
    } else {
      sibling = item.next();
      position = "lower"
    }
    
    options.onUpdate(item, position, sibling);
  }
};

Enjoy!

Posted by Simon Harris at 10:36 PM | Permalink | Comments (0)

In an attempt to push yet more behaviour from my Rails controllers into model classes, I was extracting some code into a yet-to-be-published plugin that allows date and time columns to be set using more human-readable values. For examples:

>> task.completed_at = "now"
>> task.completed_at
=> Wed Apr 09 14:39:12 +1000 2008

Although my actual requirement was to support "now" and "today" I figured it would be rather cool if I could support anything that Chronic does. (If you haven't used Chronic before, it's a natural language date/time parser written in pure Ruby that understands a vast array of expressions including ranges.)

Naturally (no pun intended), Chronic also supports explicit dates such as "1/2/08". As part of my testing however, I discovered that the date parsing is decidedly US-centric presuming, of course, that dates are specified as "month-day-year". So for anyone living in say, Australia, it can be pretty frustrating to have Chronic.parse("1/2/08") return "Wed Jan 02 12:00:00 +1100 2008" rather than the expected "Fri Feb 01 12:00:00 +1100 2008".

The good news is, there is a solution. The bad news is, the solution is far from elegant. But first some context.

Chronic is actually written very nicely and the code is fairly easy to follow. In essence it works as follows: the input string is tokenized; each token is inspected to see if it's a keyword such as a month name, day name, or a number, etc.; and finally tries to matches the sequence of tokens against a pattern such as "a day number followed by a month name and then a year." The problem arises because the pattern for matching "month-day-year" comes before the one for "day-month-year" meaning that unless the first number is greater than 12, Chronic will always consider it to be a month.

The less than elegant solution is almost trivial and involves switching the order in which the patterns are matched. Doing so returns the desired result:

>> Chronic.parse("1/2/08")
=> "Fri Feb 01 12:00:00 +1100 2008"

Which is all very well and good but now we have the reverse problem. What would be better is if we had a more general solution, one that allows us to specify the desired precedence when parsing:

>> Chronic.parse("1/2/08")                                    # Default to U.S. date formats
=> Wed Jan 02 12:00:00 +1100 2008

>> Chronic.parse("1/2/08", :explicit_date_format => :non_us)  # Prefer Non-U.S. formats
=> Fri Feb 01 12:00:00 +1100 2008

>> Chronic.parse("1/2/08", :explicit_date_format => :us)      # Prefer U.S. formats
=> Wed Jan 02 12:00:00 +1100 2008

Among other things, this allows us to have users in Australia enter dates with one format and users in the U.S. another.

For anyone interested, I've pasted a diff for each approach below.

A less than elegant solution:

--- a/chronic-0.2.3/lib/chronic/handlers.rb
+++ b/chronic-0.2.3/lib/chronic/handlers.rb
@@ -13,8 +13,8 @@ module Chronic
                  Handler.new([:repeater_month_name, :ordinal_day, :separator_at?, 'time?'], :handle_rmn_od),
                  Handler.new([:repeater_month_name, :scalar_year], :handle_rmn_sy),
                  Handler.new([:scalar_day, :repeater_month_name, :scalar_year, :separator_at?, 'time?'], :handle_sd_rmn_sy),
-                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
                  Handler.new([:scalar_day, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sd_sm_sy),
+                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
                  Handler.new([:scalar_year, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_day, :separator_at?, 'time?'], :handle_sy_sm_sd),
                  Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_year], :handle_sm_sy)],

A more general solution:

--- a/chronic-0.2.3/lib/chronic/chronic.rb
+++ b/chronic-0.2.3/lib/chronic/chronic.rb
@@ -43,7 +43,8 @@ module Chronic
       default_options = {:context => :future,
                          :now => Time.now,
                          :guess => true,
-                         :ambiguous_time_range => 6}
+                         :ambiguous_time_range => 6,
+                         :explicit_date_format => :us}
       options = default_options.merge specified_options
             
       # ensure the specified options are valid
@@ -51,6 +52,7 @@ module Chronic
         default_options.keys.include?(key) || raise(InvalidArgumentException, "#{key} is not a valid option key.")
       end
       [:past, :future, :none].include?(options[:context]) || raise(InvalidArgumentException, "Invalid value ':#{options[:context]}' for :context specified. Valid values are :past and :future.")
+      [:us, :non_us].include?(options[:explicit_date_format]) || raise(InvalidArgumentException, "Invalid value ':#{options[:explicit_date_format]}' for :explicit_date_format specified. Valid values are :us and :non_us.")
       
       # store now for later =)
       @now = options[:now]
--- a/chronic-0.2.3/lib/chronic/handlers.rb
+++ b/chronic-0.2.3/lib/chronic/handlers.rb
@@ -3,41 +3,50 @@ module Chronic
 	class << self
 	  
 	  def definitions #:nodoc:
-	    @definitions ||= 
-      {:time => [Handler.new([:repeater_time, :repeater_day_portion?], nil)],
+	    if @definitions.nil?
+        us_date = [Handler.new([:repeater_day_name, :repeater_month_name, :scalar_day, :repeater_time, :time_zone, :scalar_year], :handle_rdn_rmn_sd_t_tz_sy),
+                   Handler.new([:repeater_month_name, :scalar_day, :scalar_year], :handle_rmn_sd_sy),
+                   Handler.new([:repeater_month_name, :scalar_day, :scalar_year, :separator_at?, 'time?'], :handle_rmn_sd_sy),
+                   Handler.new([:repeater_month_name, :scalar_day, :separator_at?, 'time?'], :handle_rmn_sd),
+                   Handler.new([:repeater_month_name, :ordinal_day, :separator_at?, 'time?'], :handle_rmn_od),
+                   Handler.new([:repeater_month_name, :scalar_year], :handle_rmn_sy),
+                   Handler.new([:scalar_day, :repeater_month_name, :scalar_year, :separator_at?, 'time?'], :handle_sd_rmn_sy),
+                   Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
+                   Handler.new([:scalar_day, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sd_sm_sy),
+                   Handler.new([:scalar_year, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_day, :separator_at?, 'time?'], :handle_sy_sm_sd),
+                   Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_year], :handle_sm_sy)]
+
+        non_us_date = us_date.dup
+        non_us_date[7] = us_date[8]
+        non_us_date[8] = us_date[7]
+
+	      @definitions = 
+        {:time => [Handler.new([:repeater_time, :repeater_day_portion?], nil)],
         
-       :date => [Handler.new([:repeater_day_name, :repeater_month_name, :scalar_day, :repeater_time, :time_zone, :scalar_year], :handle_rdn_rmn_sd_t_tz_sy),
-                 Handler.new([:repeater_month_name, :scalar_day, :scalar_year], :handle_rmn_sd_sy),
-                 Handler.new([:repeater_month_name, :scalar_day, :scalar_year, :separator_at?, 'time?'], :handle_rmn_sd_sy),
-                 Handler.new([:repeater_month_name, :scalar_day, :separator_at?, 'time?'], :handle_rmn_sd),
-                 Handler.new([:repeater_month_name, :ordinal_day, :separator_at?, 'time?'], :handle_rmn_od),
-                 Handler.new([:repeater_month_name, :scalar_year], :handle_rmn_sy),
-                 Handler.new([:scalar_day, :repeater_month_name, :scalar_year, :separator_at?, 'time?'], :handle_sd_rmn_sy),
-                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_day, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sm_sd_sy),
-                 Handler.new([:scalar_day, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_year, :separator_at?, 'time?'], :handle_sd_sm_sy),
-                 Handler.new([:scalar_year, :separator_slash_or_dash, :scalar_month, :separator_slash_or_dash, :scalar_day, :separator_at?, 'time?'], :handle_sy_sm_sd),
-                 Handler.new([:scalar_month, :separator_slash_or_dash, :scalar_year], :handle_sm_sy)],
+         :date => {:us => us_date, :non_us => non_us_date},
                  
-       # tonight at 7pm
-       :anchor => [Handler.new([:grabber?, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
-                   Handler.new([:grabber?, :repeater, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
-                   Handler.new([:repeater, :grabber, :repeater], :handle_r_g_r)],
+         # tonight at 7pm
+         :anchor => [Handler.new([:grabber?, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
+                     Handler.new([:grabber?, :repeater, :repeater, :separator_at?, :repeater?, :repeater?], :handle_r),
+                     Handler.new([:repeater, :grabber, :repeater], :handle_r_g_r)],
                    
-       # 3 weeks from now, in 2 months
-       :arrow => [Handler.new([:scalar, :repeater, :pointer], :handle_s_r_p),
-                  Handler.new([:pointer, :scalar, :repeater], :handle_p_s_r),
-                  Handler.new([:scalar, :repeater, :pointer, 'anchor'], :handle_s_r_p_a)],
+         # 3 weeks from now, in 2 months
+         :arrow => [Handler.new([:scalar, :repeater, :pointer], :handle_s_r_p),
+                    Handler.new([:pointer, :scalar, :repeater], :handle_p_s_r),
+                    Handler.new([:scalar, :repeater, :pointer, 'anchor'], :handle_s_r_p_a)],
                   
-       # 3rd week in march
-       :narrow => [Handler.new([:ordinal, :repeater, :separator_in, :repeater], :handle_o_r_s_r),
-                   Handler.new([:ordinal, :repeater, :grabber, :repeater], :handle_o_r_g_r)]
-      }
+         # 3rd week in march
+         :narrow => [Handler.new([:ordinal, :repeater, :separator_in, :repeater], :handle_o_r_s_r),
+                     Handler.new([:ordinal, :repeater, :grabber, :repeater], :handle_o_r_g_r)]
+        }
+      end
+      @definitions
     end
     
     def tokens_to_span(tokens, options) #:nodoc:                   
       # maybe it's a specific date
       
-      self.definitions[:date].each do |handler|
+      self.definitions[:date][options[:explicit_date_format]].each do |handler|
         if handler.match(tokens, self.definitions)
           puts "-date" if Chronic.debug
           good_tokens = tokens.select { |o| !o.get_tag Separator }

Posted by Simon Harris at 06:47 PM | Permalink | Comments (0)

JavaScript is a fantastic little language and with the likes of Prototype, Scriptaculous, and my newest favourite, lowpro, you can build some quite frankly, remarkable web applications.

One area where most web browsers fall down however is in their error-reporting, or lack thereof. A fact that has caused me to waste seemingly countless hours trying to find the source of some problem or other only to realise that a typo that had been staring me in the face the entire time was to blame!

Now, like just about any programming library I use these days, most JavaScript libraries use American english. initialize, capitalize, you know what I'm talking about.

For the most part the use of 'z' instead of 's' isn't too much of a problem but just recently I consistently tried to use lowpro's addBehaviour method, only there isn't one. It's called addBehavior (sans 'u').

So today after about 20 minutes cursing and swearing at the spelling Steve asked "is there anyway you could create an alias?" Being JavaScript the answer is of course "abso-bloody-lutely!":

Event.addBehaviour = Event.addBehavior

You can alias just about anything this way.

No more will my code silently fail due to differences in spelling :)

Posted by Simon Harris at 06:43 PM | Permalink | Comments (2)

I just added very simple view support¹ to redhillonrails_core trunk. The changes give you the ability to create and drop views using create_view :name, "definition" and drop_view :name respectively as well as preserving the views in schema.rb.

¹WARNING: This is currently only supported for PostgreSQL. Creating views in MySQL will cause extra tables to be created in schema.rb! Probably not what you wanted.

Posted by Simon Harris at 10:03 AM | Permalink | Comments (1)

If you haven't checked out Ambition for generating ActiveRecord queries I highly recommend you do so. In a nutshell, it allows you to generate queries using the standard Ruby Enumeration idioms. Take for example the following snippet:

class Message < ActiveRecord::Base
  def self.unread
    select { |m| m.read_at == nil }.entries
  end
end

Message.unread
=> SELECT * FROM messages WHERE messages.read_at IS NULL

Notice anything missing? SQL perhaps?

Now clearly there's a whole lotta magic going on here. That said, ambition does use the standard ActiveRecord finders to query the database. One of the side-effects of this is that when navigating associations, you automatically get all the appropriate scoping.

So, for example, assuming a has_many relationship between User and Message we can do something like this:

user = User.detect { |c| c.name == 'Simon Harris' }
=> "SELECT * FROM users WHERE (users.name = 'Simon Harris') LIMIT 1"
user.messages.unread
=> "SELECT * FROM messages WHERE messages.read_at IS NULL AND messages.user_id = 3"

Here we navigated from user to messages and selected only those that have no read_at date.

As you can see, the restriction by user_id was automatically inserted for us by ActiveRecord as we navigated the association. We didn't have to do a thing. It just worked the way you would expect it to. Very cool!

However (there's always a but) this only worked because we included an execution trigger in Messages.unread. These "kickers" as they're known, include the standard Enueration methods first, entries, detect and size.

Without a kicker, the result of select is actually a query object that you can assign to a variable, call more selects on, or even store for later use! This last feature is kinda neat and leads to some nice stuff with partial caching but it also leads to some hidden complications.

If we remove the kicker from Message.unread so that it instead returns a query and call the kicker explicitly, this happens:

class Message < ActiveRecord::Base
  def self.unread
    select { |m| m.read_at == nil }
  end
end

user = User.detect { |c| c.name == 'Simon Harris' }
=> "SELECT * FROM users WHERE (users.name = 'Simon Harris') LIMIT 1"
user.messages.unread.entries
=> "SELECT * FROM messages WHERE messages.read_at IS NULL"

Where did all the scoping go?!

In the earlier example, the query was executed inside a method on an model class which, because it was called whilst navigating an association, means it was also executed using an appropriate with_scope.

In the second example however, because the query wasn't actually executed until sometime later--ie outside any model classes and associations--all the scoping information has been forgotten, as if it were never there in the first place. Bbbbbut, you want to have all that Ambition goodness and you'd like your scoping as well right? So what's a poor boy (or girl) to do?

I'm glad you asked. Here's a patch¹ to ambitious-activerecord that remembers the scoping that was in play at the time the query was constructed. It seems to work for all my current uses.

¹ DISCLAIMER: very quick-and-dirty and thoroughly untested

Posted by Simon Harris at 10:14 AM | Permalink | Comments (0)

I'm pretty sure I've blogged before about my dislike of prefix/suffix pairs such as from/to and start/end. They smell of a missing abstraction. There are however languages that provide a nice solution.

We have an application that stores a date against a record in the database. At various times, we want to see if that date falls between a specific date range. The initial implementation of the code looked something like this:

def within?(from_date, to_date)
  @date >= from && @date <= to
end

Which you would then be used along the lines of:

puts "Not available" if record.within?(start_date, end_date)

This immediately activated my olfactory senses: Not only do we have some common suffixes, we also have comparison code that looks like pretty much like every other kind of range comparison code you're ever likely to write.

As it happens, Ruby has a Range class built-in that I don't see being used nearly as much as I think it deserves. Using a range we could re-write the example to look something like:

def within?(date_range)
  date_range.includes?(@date)
end

And then in the worst case, call it like this:

puts "Not available" if record.within?(start_date..end_date)

I say worst case because here the client code still uses two separate dates which are then converted to a Range for the purposes of the call. (Interestingly the use of use of .. to construct a Range out of two individual values uses the same number of keystrokes as passing the parameters individually!) However, in the best case, we'd have been using a Range to store the dates in the first place.

Posted by Simon Harris at 11:36 AM | Permalink | Comments (2)

FWIW, I've just created two very new, very simple and VERY BETA Rails plugins.

The first, Simian, is pretty obvious: it adds a couple of rake tasks to run duplicate code checks against your rails project using, you guessed it, simian.

The second is Restful Transactions. This plugin ensures that your controller's create, update and destroy actions are wrapped in a database transaction meaning you don't have to think about it.

As always, you'll find these plugins and more over at the RedHill on Rails plugin page

Update: The plugin now wraps any action executed as a POST, PUT or DELETE rather than just the create, update and destroy actions.

Posted by Simon Harris at 11:51 PM | Permalink | Comments (0)

Yet another ridiculously simple Rails plugin, Stripper removes leading and trailing blanks from attribute values.

Leading and trailing spaces on attribute values can be a problem. They're almost never wanted nor intended. In fact, oracle actually treats empty strings as NULL. That's not to say we necessarily think that the database should be messing with our data mind you but it does show that at least someone agrees with us.

Posted by Simon Harris at 04:50 PM | Permalink | Comments (0)

As part of some other work I'm currently undertaking, I've just extracted a VERY new Rails plugin called, you guessed it, Conventional Validations.

As the README says, Conventional Validations is a plugin that attempts to apply validation based on the naming of column. Specifically, the plugin searches for validation methods such that a column named foo or ending in _foo would be validated using a method named validates_foo.

For example, by defining a class method as:

def self.validates_phone(*attr_names)
  validates_format_of attr_names, :with => /[0-9]+(\.[0-9]+)*/, :allow_nil => true 
end

Any columns named phone or ending in _phone would be automatically validated.

I've only just added it (it's only available in trunk) and the matching rules are VERY simplistic but if you have company-wide or even project-wide validation and you're strict about your naming conventions (as I am), consider extracting them into a module and mixing them into ActiveRecord::Base and letting the plugin apply them for you.

There are already some existing plugins such as (off the top of my head) "validates_email" that would most likely just work out of the box.

Enjoy.

Posted by Simon Harris at 05:32 PM | Permalink | Comments (2)

If you already have a continuous integration (ie build) server (such as Pulse, Bamboo, Hudson, or even the venerable old CruiseControl) and you've been trying to include some of your Rails applications then your life just got a whole lot easier.

Enter CI::Reporter. To quote the documentation:

...an add-on to Test::Unit and RSpec that allows you to generate XML reports of your test and/or spec runs. The resulting files can be read by a continuous integration system that understands Ant‘s JUnit report XML format, thus allowing your CI system to track test/spec successes and failures.

So, I simply installed the gem on the build-server and the plugin into my plugins project, configured the build to slurp up the generated XML files and voila! I now get build notifications letting me know how many tests passed and which ones failed.

Thanks Nick!

Posted by Simon Harris at 12:09 AM | Permalink | Comments (3)

After much installation pain and anguish experienced by some users of our plugins (due mostly to outages with RubyForge SVN servers and the inability of some users to access SVN from behind firewalls), I've finally managed to spend some time getting a plugin installer-friendly HTTP mirror up and running.

The root of the mirror can be found at http://www.redhillonrails.org/svn from which you can browse the entire repository.

If you live on the edge, you'll find them at http://www.redhillonrails.org/svn/trunk/vendor/plugins.

If you're after are the Rails 1.2 stable compatible plugins, you'll find them at http://www.redhillonrails.org/svn/branches/stable-1.2/vendor/plugins.

If you're after the older Rails 1.1.6 versions, they're available at http://www.redhillonrails.org/svn/tags/release-1.1.6/vendor/plugins.

The mirror is presently refreshed once a day which should be enough for most people.

As always, let me know if I've right royally screwed something up. I've manually tested all of the above but "it works on my machine" definitely applies in the case I'm sure.

Posted by Simon Harris at 11:52 PM | Permalink | Comments (0)

Possibly the only thing I like about MySQL is when performing a search, the values 'SIMON' and 'sImOn' are considered equal—case-insensitive searching. PostgreSQL on the other hand considers them to be different—case-sensitive searching. Now I don't know about you but for %99.999~ of the applications I've ever written, I'd rather 'Australia' and 'AuStRaLiA' weren't considered different countries.

The "standard" approach to solving this problem is to change a query from this:

SELECT * FROM countries WHERE name = ?;

To something like this:

SELECT * FROM countries WHERE LOWER(name) = LOWER(?);

Thereby forcing the database to perform a pseudo case-insensitive search. The only problem is that all those nice indexes you've created to ensure fast, efficient searching are totally ignored. (Who can spell full-table scan?) Performance issues aside (I mean after all we know that premature optimisation is the root of all evil right?) what's just as annoying is that I can't actually guarantee uniqueness, which is pretty much the whole point! No, even with a unique index on countries.name, the database will still quite happily allow me to:

INSERT INTO countries (name) VALUES ('Australia');
INSERT INTO countries (name) VALUES ('AUSTRALIA');
INSERT INTO countries (name) VALUES ('aUsTrAlIa');

So when I perform a case-insensitive search as previously discussed, I'll end up with three (count 'em 3) records. Thankfully, there is a solution (of sorts): expression indexes.

PostgreSQL allows you to create indexes based on expressions, say for example LOWER(name), allowing us to create a unique, case-insensitive index as simply as:

CREATE UNIQUE INDEX index_countries_on_name ON countries (LOWER(name));

Ok, so perhaps you knew this already and you're wondering what all this has to do with Rails? Well I'm glad you asked.

Rails (as of 1.2) has a new option for validates_uniqueness_of named, oddly enough, case_sensitive. This is assumed to be true by default (meaning all searches are case-sensitive). Set it to false however and you'll magically get validation queries that look like:

SELECT * FROM countries WHERE (LOWER(countries.name) = 'australia') LIMIT 1;

To compliment this feature, I've recently enhanced the RedHill on Rails Plugins in two interesting (and hopefully useful) ways.

The first is in the core and supports the creation of case-insensitive indexes during schema migration:

add_index :countries, [:name], :unique => true, :case_sensitive => false

The second is in schema validations and causes case_sensitive => false to be passed as an option to validates_uniqueness_of whenever a case-insensitive index is detected.

(I also looked at the possibility of automagically surrounding query parameters, etc. with LOWER() inside find methods but given the myriad forms queries can take, it seems altogether too difficult for my feeble mind at this point.)

The upshot of all this is that at the very least, it should now be possible to add case-insensitivity to your queries and be assured that (bugs not withstanding) the performance of your application won't suddenly plummet as a consequence.

Posted by Simon Harris at 10:11 AM | Permalink | Comments (4)

James has a home-grown GTD system which he'll hopefully write about if not release to the world one day. It's totally command-line driven using a combination of bash, perl and ruby to manipulate plain text files and, from what I can tell, it's not only blindingly fast, it seems to work seriously well.

One of the daily tasks that James performs is to prioritise the things he wants to (attempt) to get done and what better time to do that than on the 45 minute train trip to work. As you can well imagine, prioritising 40 items takes a fair amount of time not only due to the sheer number but also, being vision-impaired, reading (and re-reading) takes a bit of concentration. So, we came up with a nifty solution in two parts.

The first part was quite simple: use binary insertion. That way, instead of O(n^2) comparisons, we'd get O(n log n). So far so good. But the that was pretty obvious. The next bit was the real doozy!

Rather than have each pair printed out so that James could read them and make a decision, we instead chose to use some Text-To-Speech (TTS) to literally speak each pair; something along the lines of:

"Is calling your boss to discuss staff pay reviews more important than send wife a bunch of flowers for her birthday?"

In addition, we actually run the TTS in another thread so that we can easily interrupt it, primarily because James: has already chosen a suitable answer; needs to repeat the question; or wants to cancel the whole thing.

About the only problem we encountered was that, on his Powerbook, the say command (built-in to Mac OS X) ran a little too slowly for our liking so we simply aliased it to use the much faster swift command that comes as part of the commercial Cepstral William voice he uses for all his Text-To-Speech.

Now James can listen to the minimal set of questions, making the appropriate decision for each by way of a simple keystroke, and have it all done within 15 minutes all the while free to sit and enjoy the view (such that it is on a suburban train during peak hour) rather than staring at the screen.

Posted by Simon Harris at 10:13 PM | Permalink | Comments (2)

The Rails convention for naming foreign key fields is to use the singular name of the table with a suffix of _id.

In the past few days a number of people using the Foreign Key Migrations plugin together with ActiveRecord session store have discovered that the exception to the rule is session_id in the sessions table. In this case, session_id is not a recursive relationship but an unfortunately named field.

The solution is to update the migration script and add :references => nil to the line that adds the session_id. The plugin will then ignore the column and not attempt to generate a database foreign key constraint.

Posted by Simon Harris at 11:43 AM | Permalink | Comments (0)

With the pending release of Rails 1.2, I've taken the opportunity to begin updating the plugins to take advantage of various 1.2 features such as alias_method_chain and to remove where possible, work-arounds for bugs that have been fixed. So, as of today, the trunk will only run against Rails 1.2.

For those not lucky or perhaps not foolish enough to start using the latest and greatest that Rails has to offer, you can find all the Rails 1.1.6 compatible versions of the plugins at:

svn://rubyforge.org/var/svn/redhillonrails/tags/release-1.1.6/vendor/plugins

Enjoy!

Posted by Simon Harris at 11:02 AM | Permalink | Comments (0)

Since moving from lighttpd+FasCGI to Apache 2.2+mongrel our production rails application has been rock solid—one unexplained ruby core-dump notwithstanding.

To keep everything humming along, we run a few cron jobs which I thought I'd share.

The first is to ensure the application starts on boot. There is an rc-script to do this but I never bothered to get it running on FreeBSD. Instead, we use the @reboot keyword built into vixie-cron:

cd ~/www/production/current; mongrel_rails cluster::stop; mongrel_rails cluster::start

Next, session expiration. Even though plenty have argued against them in favour of memcached, we've found file-system sessions to be just fine for our, relatively low traffic, application. To keep timeout sessions after one hour—with a margin of error of an extra hour—we run an hourly cron job to delete session files that haven't been updated since it last ran:

cd ~/www/production/current; find tmp/sessions -name 'ruby_sess.*' -amin +60 -exec rm -rf {} \;

Next, to keep log file sizes manageable, we run a cron job once a day to rotate the log files using logrotate, followed by a re-cycle of the mongrel cluster:

cd ~/www/production/current; logrotate -s log/logrotate.status config/logrotate.conf; mongrel_rails cluster::restart

And here's config/logrotate.conf:

"log/*.log" {
  compress
  daily
  delaycompress
  missingok
  notifempty
  rotate 7
}

And finally, just because, we run another daily cron job to vacuum the PostgreSQL database:

cd ~/www/production/current; psql cjp_production -c 'vacuum full'

Posted by Simon Harris at 12:30 PM | Permalink | Comments (5)

Ruby on Rails provides a very simple mechanism for specifying that a model class is an abstract base class and therefore has no corresponding database table:

class MyAbstractClass < ActiveRecord::Base
  self.abstract_class = true
  ...
end

Code can then interrogate a model class to see if it is abstract:

puts "it's abstract" if MyAbstractClass.abstract_class?

Not so hard, however I pretty much always prefix the name of my abstract classes with, you guessed it, 'Abstract'. So, I added some code to the RedHill on Rails Core Plugin the other day to extend the definition of an abstract class to include the name:

def abstract_class?
  @@abstract_class || !(name =~ /^Abstract/).nil?
end

With that simple change, I no longer need to explicitly set self.abstract_class = true; it just works by magicconvention.

I suppose I could/should have created a plugin for it but I was feeling lazy :)

Posted by Simon Harris at 11:04 AM | Permalink | Comments (9)

On one Rails project, we have two deployment environments: production; and UAT. Using the default Capistrano configuration makes deploying to these two environments rather difficult so, I thought I'd share our deploy.rb with a bit of explanation along the way. Ok, here goes:

For a start, we deploy to a directory that includes the environment as part of the path:

set :deploy_to, lambda { "/home/#{user}/www/#{rails_env}" }

For subversion, we checkout the code as the user who is running the deployment making sure not to cache authentication details on the server:

set :svn_user, ENV['USER']
set :svn_password, lambda { Capistrano::CLI.password_prompt('SVN Password: ') }
set :repository, lambda { "--username #{svn_user} --password #{svn_password} --no-auth-cache svnurl/trunk/#{application}" }

In both cases, we run a mongrel cluster. Because the mongrel configuration files share a lot in common and because they largely duplicate information contained within the deployment script, we generate an appropriate configuration on deployment. More of that in a bit but for now, the common bits look like:

set :mongrel_address, "127.0.0.1"
set :mongrel_environment, lambda { rails_env }
set :mongrel_conf, lambda { "#{current_path}/config/mongrel_cluster.yml" }

Now, for the environment specific portions. For each environment we have a task that simply sets variables appropriately—I toyed with using an environment variable such as RAILS_ENV rather than the pseudo-tasks but it was more typing and I'm allergic to typing :).

For production, we want 3 mongrel instances in the cluster, listening on ports 8000-8002:

desc "Production specific setup"
task :production do
  set :rails_env, :production
  set :mongrel_servers, 3
  set :mongrel_port, 8000
end

For UAT, we want 2 mongrel instances in the cluster, listening on ports 8010-8011:

desc "UAT specific setup"
task :uat do
  set :rails_env, :uat
  set :mongrel_servers, 2
  set :mongrel_port, 8010
end

And finally, a custom deployment script based almost entirely on the built-in deploy_with_migrations with the major difference being the configuration of the mongrel cluster just prior to restart:

desc "Generic deployment"
task :deploy do
  update_code

  begin
    old_migrate_target = migrate_target
    set :migrate_target, :latest
    migrate
  ensure
    set :migrate_target, old_migrate_target
  end

  symlink
  
  configure_mongrel_cluster

  restart
end

That's it really. Now whenever we need to deploy to a particular environment, say for example UAT, we do something like:

cap uat deploy

Update: By request, here is our database.yml file :

common: &common
  adapter: postgresql
  username: <%= ENV['USER'] %>

development:
  database: foo_development
  <<: *common

test:
  database: foo_test
  <<: *common

uat:
  database: foo_uat
  <<: *common

production:
  database: foo_production
  <<: *common

As you can probably tell, we're lucky enough that the database user is always the same as the user under which the application will be run and is that the database itself is named according to the environment. That makes it very easy to wrap up most of the common parts—Thanks goes to Jon Tirsen for that YAML tip.

This could also easily be generated. I guess it just hasn't needed any attention since it was created so YAGNI overrode DRY ;-)

Posted by Simon Harris at 07:44 AM | Permalink | Comments (6)

I happened to be reading a blog entry last night that mentioned some "short comings" in Rails' ActiveRecord and its handling of record loading. Specifically, AR will load the same record twice, into two different instances, within the same transaction. Ie. the following test fails:

Customer.transaction do
  c = Customer.find_by_name('RedHill Consulting, Pty. Ltd.')
  assert_same c, Customer.find(c.id)
end

To be honest, I've not yet been burned by this but it may just catch-out some so I quickly whipped up a very basic plugin to see how difficult it would be solve:

module RedHillConsulting
  module IdentityMap
    class Cache
      def initialize
        @objects = {}
      end

      def put(object)
        objects = @objects[object.class] ||= {}
        objects[object.id] ||= object
      end
    end

    module Base
      def self.included(base)
        base.extend(ClassMethods)

        base.class_eval do
          alias_method_chain :create, :identity_map
        end
      end

      module ClassMethods
        def self.extended(base)
          class << base
            [:instantiate, :increment_open_transactions, :decrement_open_transactions].each do |method|
              alias_method_chain method, :identity_map
            end
          end
        end

        def instantiate_with_identity_map(record)
          enlist_in_transaction(instantiate_without_identity_map(record))
        end

        def enlist_in_transaction(object)
          identity_map = Thread.current['identity_map']
          return object unless identity_map
          identity_map.put(object)
        end

        private
          def increment_open_transactions_with_identity_map
            increment_open_transactions_without_identity_map
            Thread.current['identity_map'] ||= Cache.new
          end

          def decrement_open_transactions_with_identity_map
            Thread.current['identity_map'] = nil if decrement_open_transactions_without_identity_map < 1
          end
      end

      def create_with_identity_map()
        create_without_identity_map
        self.class.enlist_in_transaction(self)
        id
      end
    end
  end
end

The code essentially interferes with create and instantiate (called from find) and ensures that, within a transactions, the same record will always be returned for the same id (IdentityMap).

As I mentioned, unlike all my other plugins, I've never used nor needed to use this one—and I'm not sure I will unless it proves to be a problem for me—but it's yet another example of how easy it is to extend Rails to do pretty much whatever you might imagine.

Posted by Simon Harris at 12:51 PM | Permalink | Comments (8)

The first cut at Schema Validations only applied validates_uniqueness_of for single-column unique indexes. This removed 80% of the cases in my code base but there were still cases where a scope was specified that lingered. Not any more.

The plugin now automatically generates validates_uniqueness_of with scope for multi-column unique indexes as well.

As always, there are some assumed conventions—which I believe will handle close to 99% of cases—around how to decide which column to validate versus which columns to consider part of the scope. The column to validate is chosen to be either:

1. The last column in the index definition not ending in ‘_id’; or simply
2. The last column in the index definition.

With all remaining columns considered part of the scope, following, what I believe to be, a typical typical composite unique index column ordering.

So, for example, given either of the following two statements in your schema migration:

add_index :states, [:country_id, :name], :unique => true
add_index :states, [:name, :country_id], :unique => true

The plugin will generate:

validates_uniqueness_of :name, :scope => [:country_id]

My next stop is to have a look at simple column constraints such as IN('male', 'female') and turn them into validates_inclusion_of :gender, :in => ['male', 'female'].

Perhaps tomorrow :)

Posted by Simon Harris at 11:10 AM | Permalink | Comments (0)

The more I use Rails (and the more plugins I create) the more quirks I find.

Imagine I have a one:many relationship between Country and State:

State.belongs_to :country
Country.has_many :states

We then issue the following sequence of statements (I've interleaved the output of tailing the development log):

c = Country.find_by_name('Australia')
  Country Load (0.006506)   SELECT * FROM countries WHERE (countries."name" = 'Australia' ) LIMIT 1
s = c.states.build(:name => 'Victoria', :abbreviation => 'VIC')
s.country
  Country Load (0.009738)   SELECT * FROM countries WHERE (countries.id = 1)

Notice the SELECT to find the country? Now why would that be necessary? I just used .states.build on the country. I would have thought that would set the association but that doesn't appear to be the case.

Looking at the code, my suspicions were confirmed: only the parent's id is set. That seems decidedly odd given that we know for a fact the parent exists—we just used it to create the child.

So anyway, I'm pretty sure this is considered a "feature" but to be honest, I can't see why it is desired behaviour over and above the fact that doing otherwise would be more work and why would you need this if you already have the parent yada, yada, yada.

Well, for a start, I'd like this behaviour because I'd like to use validates_presence_of on foreign-keys and have it work for newly constructed graphs. Usually this barfs no matter what but I concocted a work-around last night and committed it to my Foreign Key Associations plugin which, if done manually, would look something like this:

class State < ActiveRecord::Base
  validates_presence_of :country_id, :if => lambda { |record| record.country.nil? }
  ...
end

Essentially this says to validate the presence of country_id but only if there isn't an associated country. This means that for cases where the parent record is also new, the validation checks for the presence of the associated object rather than the foreign-key column. If you had simply used validates_presence_of :country_id then save would fail because country_id was still nil.

OK that's all very well and good but it still doesn't help because, as shown above, the association isn't set anyway. So, I'm now back to manually setting the association; at least the validation works hehe

I'm sure someone far smarter than I will point out why the behaviour as it stands is obviously the most appropriate and that no one in their right mind would want to do anything else, of course ;-)

Posted by Simon Harris at 10:19 AM | Permalink | Comments (2)

As I was bookmarking something on del.icio.us today, I noticed the dates on which I had bookmarked the last couple of times and wondered if there was any correlation between frequency and day of the week. So, I downloaded a summary using https://api.del.icio.us/v1/posts/all? and whipped up a little ruby script to compile some statistics:

Wednesday = 41
Tuesday = 39
Thursday = 37
Friday = 32
Monday = 26
Saturday = 24
Sunday = 12

Looks like Wednesday is the biggest day for bookmarking—also known as procrastinating—and what do you know? Today is...Wednesday!

So then I thought I'd see if there was anything interesting in the time of day:

12 = 26
13 = 20
4 = 17
22 = 15
0 = 14
23 = 12
5 = 12
2 = 12
20 = 10
11 = 10
1 = 10
7 = 10
3 = 9
6 = 7
21 = 7
9 = 6
15 = 4
14 = 3
8 = 3
10 = 2
19 = 2

Phew! Most of my bookmarking is done around lunchtime although an awful lot were done at 4am!

Posted by Simon Harris at 06:54 PM | Permalink | Comments (2)

I mentioned in my previous entry that I'd done quite a bit of refactoring of the plugins. Among the various changes that will affect developers using them are:

• Schema Defining (schema_defining) has been deleted;
• Foreign Key Support (foreign_key_support) has been deleted; and
• RedHill on Rails Core (redhillonrails_core) has been added to replace the previous two as well as subsuming some of the more generic functionality from other plugins.

So, why all these changes?

The main reason is manageability. We're actually eating our own dog food and using these plugins in production applications and we're adding functionality at quite a surprising rate. Each time we add something, we first put it into the plugin that needs it directly. That works great for a while but then, someday, we decide we need that functionality in two or more plugins. What to do?

Our original idea had been to create new plugins and this worked for us up to a point. Unfortunately, of late, the number of extra plugins—with very specific functionality mind you—was just getting out of hand and needed to be simplified.

In the end, we decided on a two-tiered approach to plugins: those which add functionality but no (or at least minimal) behaviour; and those that add behavioural magic.

As an example, the new core plugin adds functionality to manage foreign keys, lookup indexes, add unique column meta-data, etc. but doesn't do anything particularly magic that will affect the running of your application.

On the other hand, the foreign key migrations, foreign key associations, schema validations, etc. plugins—which all rely on core—add funky rails magic to automatically generate foreign keys, associations, model validation, etc.

Another change we made was in the way documentation is generated. We used to manually generate a nice HTML file containing all the plugins. This was becoming rather tedious and meant that the documentation was often quite out of date. We've now remedied this with a nice ruby script using Erb and RDoc to generate the online documentation directly from the README files.

I also mentioned previously that we've added "lots" of tests. I say lots because we're still playing catchup so relatively, there are lots but we still need lots more. As a group of developers that are ardent TDD evangelists, the conspicuous lack of tests was somewhat embarrassing to say the least. Unfortunately, testing plugins (especially those related to schema and database) is pretty difficult so we opted to bypass the whole problem and just create a standard rails app with standard rails tests and all is well again.

And lastly, besides all the extrat features we've added (see the CHANGELOGs for the specific plugins), you'll notice that the subversion URL has changed slightly—it used to contain an extra slash (/) which was not only unnecessary but caused SVN to regularly crap out.

My aplogies to all those that have been trying to keep up but we hope that's the last of it. From now on, we'll continue to beef up core as we need and then add plugins only when we need new behaviour.

Of course we'll always reserve the right to change our minds ;-)

Posted by Simon Harris at 05:01 PM | Permalink | Comments (0)