Datamapper DataMapper

Comments and changes to this ticket

• 

  Dan Kubb (dkubb) October 7th, 2009 @ 12:59 PM

  • State changed from “new” to “hold”
  • Assigned user set to “Dan Kubb (dkubb)”

  @Brock: I'm not sure what you mean. DataMapper.auto_migrate! sets the table to the same encoding as whatever the connection's character set is, which you control when you configure MySQL. If for some reason, there isn't one then it defaults to "utf8".

  Can you provide an example of what your specific problem is, and how I could reproduce it on my end?

  Marking this as "on hold" until Brock replies.

• You flagged this item as spam.
  

  Brock Whitten October 7th, 2009 @ 01:39 PM

  I have tested this further. I have found that encoding in DM setup must be set to "UTF-8" rather than "utf8" to get table set to "utf8" in mysql. DM gives a warning if encoding is set to "utf8" so I would say that everything is fine there.

  As for part two (saving content), Im having troubles replicating this problem. I have been running into a lot of encoding issues and this may be due to contaminated content that I have in my database. I will try to narrow this down to irreducible complexity.

• 

  Dan Kubb (dkubb) October 8th, 2009 @ 03:28 PM

  • State changed from “hold” to “unconfirmed”
  • Assigned user cleared.

  When you specify the encoding in DataMapper.setup, you need to use "UTF-8" (specifying nothing defaults to UTF-8 BTW). The reason is that's what Ruby itself uses to designate a String as "UTF-8" encoded, and not "utf8 or "utf-8" and we map those onto whatever each DB uses, which can sometimes be quite different. We designated Ruby's POV as being authoritative, and have Hashes that map the Ruby encodings to whatever the DB supports.

  If you have alot of contaminated data, it might be worthwhile using something like Iconv to convert each "cell" into UTF-8 encoding, compare to the original and prompt you when they differ, allowing you to review them before accepting or rejecting the change. It can sometimes time consuming, but it totally depends on the size and state of your dataset.

• 

  Dan Kubb (dkubb) May 22nd, 2010 @ 01:26 AM

  @Brock: Any luck reproducing this problem with the latest DM/DO? There were a number of encoding fixes in DO 0.10.2 (released this week) that may have solved some of the problems.