« RedHill on Rails Plugin Refactoring | Main | Procrastinating in Ruby is Delicious »

Not My SQL

Listen to this articleListen to this article

Everyone else's favourite database just gave me the shits, again!

As part of my Schema Validations plugin for rails, I needed to see if a column has a default value. If it does, then there's no point in adding a validates_presence_of as the database will add one in. Ok, sounds sensible. Works just fine under PostgreSQL but my tests were failing when run against MySQL. Specifically, there was no validation being added for integer columns marked as NOT NULL. Huh?!

After a little investigation, I noticed that the meta-data that rails was collecting for mandiatory integer columns included a default of 0. So I looked in the test database and sure enough the columns all had a default of 0. But how? Why? I didn't put a default in my migrations...

A little more investigation and I noticed that the schema dump that is generated out of the development database and then run against the test database did indeed include the very same defaults. I then looked in the development database and to my surprise found no such defaults there. Aha! Mystery solved I presumed. Rails must have a bug for MySQL.

So I go and look at the code but alas, the code is the same for both PostgreSQL and MySQL. Something else must be happening. Time to get down and dirty on the command-line.

mysql> create table foo (col1 int, col2 int not null, col3 int default null) engine=InnoDB;

mysql> show columns from foo;
+-------+---------+------+-----+---------+-------+
| Field | Type    | Null | Key | Default | Extra |
+-------+---------+------+-----+---------+-------+
| col1  | int(11) | YES  |     | NULL    |       | 
| col2  | int(11) | NO   |     |         |       | 
| col3  | int(11) | YES  |     | NULL    |       | 
+-------+---------+------+-----+---------+-------+

If I have a nullable column then the default default (if that makes sense) is NULL. If I mark a column as mandiatory, the default default is...an empty string!? I wonder what would happen if I tried inserting a row and letting MySQL default all the values:

mysql> insert into foo () values ();

mysql> select * from foo;
+------+------+------+
| col1 | col2 | col3 |
+------+------+------+
| NULL |    0 | NULL | 
+------+------+------+

You have to be shitting me! I attempted to insert a row into a table without specifying a value for a column that is marked as NOT NULL and it inserts 0!? Hold on a second...what if I force the default to be NULL so that it behaves just like every other sensible database on the planet:

mysql> create table bar (col1 int not null default null) engine=InnoDB;
ERROR 1067 (42000): Invalid default value for 'col1'

Egads! OK let me try that in PostgreSQL:

psql=# create table bar (col1 int not null default null);
CREATE TABLE

Thank-you!

Sure, I could make the assumption that 0 was never going to be a valid identifier for a record in another table but why should I have to? As far as I can tell, MySQL is just making shit up! No wonder my brother says it reminds him of using Microsoft Access.

So, now I'm left with the task of working out how to patch rails to get around this. I think I'll just have to presume that empty strings are equivalant to NULL for manditory columns. Sheesh.

Comments

Hi,

Your talkr links are not working. Just thought you might want to know that.

Regards,
Behi

I found this a while ago when noticing db:schema:dump doesn't generate correct schema.rb, tracked it down to mysql.rb in rails. Got test to review the bug but don't know how to fix or work around it. I've emailed the author of mysql.rb, not sure if I'll hear back from him so I'd be interested to find out your fix.

Ok, so I've now implemeted a monkey-patch in the core plugin which is mixed into the MysqlColumn class:

module MysqlColumn
def initialize(name, default, sql_type = nil, null = true)
default = nil if !null && default.blank?
super
end
end

Ideally I'd like the code in MysqlAdapter.columns that constructs the MysqlColumn instances to do this but that'll have to wait I guess. Maybe I'll submit a patch.

There is a setting in later versions of MySQL to disable this (annoying) "feature". Check the docs.

I had a cursory glance but couldn't see anything obvious. Any links?

My test passed with your patch :)
Btw did you get a link to how to turn off the feature?

No links I'm afraid. I did submit a patch which was rejected on the basis that I can enable "strict" mode. Although I didn't try it, I did find some documentation on strict mode which seemed to indicate that rather than affecting the meta-data, it merely says MySQL will no longer add in values for NOT NULL columns that don't have a default. So, whilst completely unsubstantiated by me, I suspect the strict mode doesn't solve the actualy problem I was having; namely, crappy meta-data. I don't use MySQL in anger so frankly I don't care enough to bother trying but I do want my plugins to be compatible with it (for much the same reason as most people write sotware to run on Microsoft Windows) so this work-around suits me just fine and doesn't require strict mode, even if it did work as needed.

I ran into this issue this evening as I was trying to test some validates_presence_of stuff. I had gone through nearly the exact sequence of testing that you did. I had remembered reading something about this wonderful mysql feature on the mysql gotcha's page, and was trying to figure out how to enable strict mode to see if that would help.

Thank goodness I found this blog entry, since requiring mysql strict mode isn't exactly a portable fix.

Hopefully some version of the patch gets accepted into the core, because this really needs to be fixed. This behavior is just way too surprising.


Glad we could help.

I can confirm that turning on STRICT_ALL_TABLES mode does not solve the problem (MySQL 5.0.22 on Linux). Perhaps this should be filed as a bug against MySQL, as well as making a work-around for Rails. I suspect you are right that it is in the meta-data, but I don't know how to write a test for this.

FWIW, The redhill on rails core plugin addresses this specifically. I've also raised it as a bug against Rails proper.

I am still a rails newbie and need to get around the above problem. I can see that the monkey-patch fixes the problem, but I'm not sure where or how to apply it to my app. Any chance of a pointer?

Thanks,
Dennis

Post a comment