#682 ✓resolved
Alex Arnell

strings containing null terminators get truncated

Reported by Alex Arnell | December 1st, 2008 @ 04:50 PM

If you have a model that contains a string property. If you try and insert a string containing a null terminator the string is truncated to that terminator.

Basically the following example will truncate the hash string to an empty string.

class User
  include DataMapper::Resource

  property :id, Serial
  property :password_hash, String
# string containing null terminator, and gets truncated
hash = "\000\340\336\303\352\2019\355]\200\300\306\344\374\177P\214\023\311\231"

puts "Hex Encoded Original Value: #{Digest.hexencode( hash )}"
User.create({ :password_hash => hash })
user = User.first

# => ''
puts "password_hash (hex encoded): #{Digest.hexencode( user.password_hash )}"

Attached is the full example program that showcases the issue in both mysql and sqlite3 adapters.

Comments and changes to this ticket

  • Dan Kubb (dkubb)

    Dan Kubb (dkubb) December 1st, 2008 @ 05:48 PM

    • Tag changed from adapter, bug, dm-core, do_mysql, do_sqlite3 to bug, do_mysql, do_postgres, do_sqlite3
    • State changed from “new” to “open”
    • Assigned user changed from “Dan Kubb (dkubb)” to “Dirkjan Bussink”

    I can confirm this is an issue with all 3 DO drivers: mysql, postgres and sqlite3. I've attached a modified script to the one Alex provided that demonstrates the problem on all the adapters, aside from the InMemory adapter.

    I spoke with Dirkjan about this briefly and he said it could be caused by not specifying the length of the strings in the C code. I've assigned this to Dirkjan to review.

  • Alex Arnell

    Alex Arnell December 2nd, 2008 @ 01:01 AM

    We had a dig through some of the do_mysql C code here at the office. We think it might be related to using the strlen function to measure the size of the string.

  • Dirkjan Bussink

    Dirkjan Bussink December 7th, 2008 @ 09:55 AM

    • State changed from “open” to “resolved”

    I've been digging into this, and it seems like it's not properly fixable by definition. I've added code to properly escape stuff and not use strlen anymore, but this doesn't fix the issue.

    In MySQL for example, inserting the following data won't work properly:

    INSERT INTO users (password_hash) VALUES ('\0app');

    This won't load up the data properly into the table, because MySQL itself will truncate it. You probably need a binary type to handle this properly. Other RDBMS' like Postgres even specify that truncating a value at a nil byte is defined behavior.

    It seems that the only way to properly fix this, is to make sure that you encode the data with for example Base64 encoding. This way it won't end up being clobbered by truncating nil bytes.

  • Alex Arnell

    Alex Arnell December 11th, 2008 @ 07:49 PM

    This kind of works for us now. We actually have a custom CharArray type that we have built using an Object as it's primitive. We map this to the mysql BINARY field type.

    Inserts now work perfectly for us, however reading this data back out still fails, the data is truncated again.

    I think you need to use http://dev.mysql.com/doc/refman/... to find the length of the fields instead of relying on strlen.

    I tried to get this working locally, but failed miserably.

  • Dirkjan Bussink

    Dirkjan Bussink December 12th, 2008 @ 01:45 AM

    • State changed from “resolved” to “confirmed”

    Could you verify whether the following patch works for you: http://gist.github.com/35047

    If that fixes the issue, I'll commit these changes to do_mysql also fix this for the other DO drivers.

  • Alex Arnell

    Alex Arnell December 12th, 2008 @ 03:04 AM

    That patch solves our problem.

    I had to make one minor change to my custom type to use a String primitive instead of an Object (which makes more sense anyways).

    So essentially our custom type looks like the following:

    class CharArray < DataMapper::Type
      primitive String
    repository.adapter.class.type_map.map( CharArray ).to( 'BINARY' )
    class User
      include DataMapper::Resource
      property :id, Serial
      property :password_hash, CharArray, :size => 20

    When primitive Object is used there is a marshal exception thrown

    /Library/Ruby/Gems/1.8/gems/dm-core-0.9.8/lib/dm-core/adapters/data_objects_adapter.rb:68:in `next!': marshal data too short (ArgumentError)

    Also I have not tested this with PostgreSQL.

  • Alex Arnell

    Alex Arnell December 12th, 2008 @ 03:06 AM

    BTW. Thanks for the speedy responses on this bug.

  • Dirkjan Bussink

    Dirkjan Bussink December 12th, 2008 @ 03:40 AM

    • State changed from “confirmed” to “resolved”

    Ok, fix has been pushed to do_mysql and do_postgres. I need to fix this for sqlite3, but when i looked at that code, I saw I need to refactor more in there than just fix this.

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »