My hovercraft is full of eels

Since moving from lighttpd+FasCGI to Apache 2.2+mongrel our production rails application has been rock solid—one unexplained ruby core-dump notwithstanding.

To keep everything humming along, we run a few cron jobs which I thought I'd share.

The first is to ensure the application starts on boot. There is an rc-script to do this but I never bothered to get it running on FreeBSD. Instead, we use the @reboot keyword built into vixie-cron:

cd ~/www/production/current; mongrel_rails cluster::stop; mongrel_rails cluster::start

Next, session expiration. Even though plenty have argued against them in favour of memcached, we've found file-system sessions to be just fine for our, relatively low traffic, application. To keep timeout sessions after one hour—with a margin of error of an extra hour—we run an hourly cron job to delete session files that haven't been updated since it last ran:

cd ~/www/production/current; find tmp/sessions -name 'ruby_sess.*' -amin +60 -exec rm -rf {} \;

Next, to keep log file sizes manageable, we run a cron job once a day to rotate the log files using logrotate, followed by a re-cycle of the mongrel cluster:

cd ~/www/production/current; logrotate -s log/logrotate.status config/logrotate.conf; mongrel_rails cluster::restart

And here's config/logrotate.conf:

"log/*.log" {
  compress
  daily
  delaycompress
  missingok
  notifempty
  rotate 7
}

And finally, just because, we run another daily cron job to vacuum the PostgreSQL database:

cd ~/www/production/current; psql cjp_production -c 'vacuum full'

Posted by Simon Harris at 12:30 PM | Permalink | Comments (5)

If you're not sure what I'm talking about, have a quick read of my earlier post.

The trick—as far as I can tell—to plugging the hole is to: disable guest logins to the server; and ensure each build configuration requires each agent to have a secret environment variable set to a, secret, value—something like a generated WEP key for example.

This seems to be a reasonable solution until JetBrains figures out a better mechanism. That said, in many respects, it's not that different to the way WEP authentication works anyway.

Oh, an why the need to disable guest login? TeamCity shows all logged in users—yes, even guests—which agents aren't compatible and, in particular, why.

Of course there may well be a way to interrogate programatically what the requirements are in which case, you're hosed anyway :(

Posted by Simon Harris at 01:30 AM | Permalink | Comments (0)

After nearly falling over at the ease with which I could use TeamCity to crush helpless machines, it suddenly occurred to me that I may have found yet another security hole.

As you probably already know, the TeamCity server doesn't do the builds itself; rather, it farms the work off to build-agents. You can connect as many build-agents as you like to a server: simply download (or otherwise obtain) a copy of the build-agent code; configure it to point at the server; and start it up. No server-side credentials are necessary.

When a build is required, the server checks out the source code and sends a delta to the agent. This has a number of benefits, one being that you can securely configure the source-code-repository credentials in one place—the server—and the agents will be sent the source code as needed. This also poses a potential security risk.

Let's imagine that disgruntled developer X from our previous exploit wants to obtain a copy of source code from a repository for which he has no access but for which he knows there is a TeamCity build. He simply configures his build-agent to connect to the server and waits. When the server decides it's his machine's turn to do a build, the server dutifully sends him a copy of the source-code...!

Posted by Simon Harris at 03:19 PM | Permalink | Comments (0)

As much as I've bitched about IntelliJ performance, and as much as I wish I didn't have to do Java development on a regular basis, the fact of the matter is I do and IntelliJ just rocks my world in terms of features—I guess I'll just have to wait for the new Core 2 Duo MacBook Pro to address the performance issue— so today I purchased a copy of the latest version.

IntelliJ 6.0 comes with TeamCity, a continuous-integration/build-server tool. It would appear that TeamCity isn't just restricted to building applications, but could really do anything assuming you do in ant script.

TeamCity has the concept of build agents so all the real work is farmed off to other machines, so I setup my machine as a build-agent which required opening up a port—9090 is the default—through the firewall on my machine. This immediately rang warning bells in my head: I had just run the agent using sudo and any application run using sudo that needs ports opened up scares me. In the end I realised I could easily run the agent as a non root user and all was happy. Excellent!

That did get me thinking however, as to what would happen on say, M$ Windoze machines where developers typically run with administrator priveleges or even poor saps like me who had stupidly run using sudo or even just as my own login.

Imagine a team of 20 developers, all of whom have obligingly run a build-agent so that their spare CPU cycles don't go to waste. One day developer X becomes disgruntled with his employer and decides to run wild. Before he goes home at night, he checks in a change to build.xml. Not a large change, nothing special, just a one line change that recursively deletes everything starting at the root directory or even just the user's home directory.

This change quickly makes it to the TeamCity server and then out to a build machine which dutifully executes the script, destroying everything taking down the machine with it!

Thankfully, the TeamCity server detects that the agent has gone down and, doing it's best to utilise the resources at its disposal, re-schedules the build on the next available agent...

To be fair, TeamCity isn't really to blame, if a developer runs build.xml without first ensuring that it does nothing macilicious, it's really no different to running an un-verified shell script. TeamCity just makes it really easy to setup agents and ensure that the script is executed.

Me thinks it's time make a seperate, locked-down account for running the build-agent!

Posted by Simon Harris at 03:48 PM | Permalink | Comments (6)

Whilst reading the latest news headlines on the ABC (Australian Broadcasting Corporation) web site just now, I happened upon what seemed like a rather interesting article entitled Brain fluid draining eases dementia: research. Fascinated, I read on.

...The study investigated 20 patients...71 per cent of our patients improved in memory and mental function and 94 per cent improved in balance and walking...

Hang on a second...71 percent of 20 patients would be...14.2 patients; 94 percent of 20 patients would be...18.8 patients. Huh?

Posted by Simon Harris at 02:18 PM | Permalink | Comments (1)

Ruby on Rails provides a very simple mechanism for specifying that a model class is an abstract base class and therefore has no corresponding database table:

class MyAbstractClass < ActiveRecord::Base
  self.abstract_class = true
  ...
end

Code can then interrogate a model class to see if it is abstract:

puts "it's abstract" if MyAbstractClass.abstract_class?

Not so hard, however I pretty much always prefix the name of my abstract classes with, you guessed it, 'Abstract'. So, I added some code to the RedHill on Rails Core Plugin the other day to extend the definition of an abstract class to include the name:

def abstract_class?
  @@abstract_class || !(name =~ /^Abstract/).nil?
end

With that simple change, I no longer need to explicitly set self.abstract_class = true; it just works by magicconvention.

I suppose I could/should have created a plugin for it but I was feeling lazy :)

Posted by Simon Harris at 11:04 AM | Permalink | Comments (9)

For anyone who is is unfortunate enough to work with Perforce—and so I don't have to remember—here's a quick-and-dirty kick-start guide for setting up a client workstation. (Note I'm on a Mac so your mileage may vary.)

First things first, install the perforce software available from http://www.perforce.com/perforce/downloads.

Next, in order to avoid various command-line arguments, I have the following environment variables set:

export P4EDITOR="$EDITOR"
export P4USER="username"
export P4PORT="1666"
export P4HOST="clientname.local"
export P4CLIENT="clientname"

If you're using SSH, you'll need to create a tunnel to the server. Something like this should work:

ssh -L1666:server:1666 -p 22 -N -t -x username@server

This sets things up so all requests to localhost:1666 are routed over ssh to the remote server. You can then setup the client:

p4 client

This will launch your default editor—in my case that's TextMate but nano/pico/emacs/vi/etc will do—and allow you to modify the following fields:

Client:	clientname
Owner:	username
Host:	clientname.local
Root:	/path/to/projects/
View:

See the documentation for an explanation on how to set-up the View. In my case, I'm running a rails application, so I have some rules to exclude various generated and client specific directories:

-//depot/projectname/config/database.yml //clientname/projectname/config/database.yml
-//depot/projectname/db/schema.rb //clientname/projectname/log/schema.rb
-//depot/projectname/log/... //clientname/projectname/log/...
-//depot/projectname/tmp/... //clientname/projectname/tmp/...

Finally, to get a copy of the latest source code in /path/to/projects/projectname, run:

p4 sync

And because I just can't help myself, by way of comparison, here's the equivalent instructions for subversion:

svn co svn+ssh://username@repositoryurl/trunk/projectname

Gosh, wasn't that difficult.

Posted by Simon Harris at 10:44 AM | Permalink | Comments (3)

Yes, it's been a while since I posted an entry related to Java! Believe it or not, we still do Java development, lots of it in fact, but it's mostly large-scale re-factoring and cleanup work on what can best be described as "legacy" applications so there's rarely much if anything to write home about. That said, I have a couple of posts just itching to be written when I find some time. Until then, a relatively short entry will have to do :)

A client distributes one particular Java-based web application to hundreds of customers using a zip file. The distribution contains, among other things, the war file and some scripts for database migration, etc. It's these scripts that cause us some headaches as they need to have execute permission. The problem arises because Ant's built-in zip task specifically doesn't handle file permissions. So, naturally, we concocted our own using macrodef:

<macrodef name="zipdir">
    <attribute name="destfile"/>
    <attribute name="sourcedir"/>
    <echo>Building zip: @{destfile}</echo>
    <exec executable="zip" dir="@{sourcedir}">
        <arg value="-qR"/>
        <arg value="@{destfile}"/>
        <arg value="*"/>
        <arg value="-x *.svn* "/>
    </exec>
</macrodef>

This simply calls the operating system's—read *nix—zip command to compress the specified directory thus preserving all the file permissions that SVN lovingly maintains.

Posted by Simon Harris at 11:31 AM | Permalink | Comments (2)