#660 ✓resolved
Radosław Bułat

require "sqlite3" fails with do_sqlite3 0.9.7

Reported by Radosław Bułat | November 18th, 2008 @ 03:16 PM

$ gem list | grep sqlite do_sqlite3 (0.9.7) sqlite3-ruby (1.2.4)

$ ruby -r rubygems -e 'require "sqlite3"; puts "OK"' /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in gem_original_require': no such file to load -- sqlite3 (LoadError)

from /usr/local/lib/site_ruby/1.8/rubygems/custom_require.rb:36:in `require'
from -e:1

$ sudo gem uninstall do_sqlite3 -I Successfully uninstalled do_sqlite3-0.9.7

$ ruby -r rubygems -e 'require "sqlite3"; puts "OK"' OK

I investigated it and it's due to fact that do_sqlite install file sqlite3.dll under it's lib/ directory. I guess that this file is misinterpret by rubygems. After deleting this file do_sqlite3 doesn't make conflict with sqltie3.

Comments and changes to this ticket

  • Luis Lavena

    Luis Lavena November 20th, 2008 @ 04:18 PM

    I guess the idea was having that file inside the gem would avoid the need to install the shared library (dll) under Windows.

    I commented was a bad idea at the moment but 2 releases did go with that schema, introducing problems not only to different platforms but also gems like sqlite3-ruby coexisting under Windows.

    Having binaries of shared libraries in the repository is something we should avoid.

  • rick frankel
  • Dan Kubb (dkubb)

    Dan Kubb (dkubb) November 22nd, 2008 @ 03:26 AM

    • State changed from “new” to “open”

    Luis: Do you know if we can safely remove the file? What sort of impact will it have? What should we do to improve the Windows gems in the future?

    I'm totally unfamiliar with packaging up gems for Windows machines, so if you could impart some of knowledge I would great appreciate it.

    I would like to get the gem build process to the point where anyone can run "rake release" for DO and have everything bundled properly for Windows, Linux, OSX, etc machines, but I'm unsure what we'd need to do to get to that point.

  • rick frankel

    rick frankel November 22nd, 2008 @ 10:14 AM

    If you want to distribute a windows binary file for the underlying system library with the gem (a bad idea, windows binaries should not be installed on non windows boxen regardless), couldn't you rename it something that does not conflict with sqlite3-ruby and dynamically load the correct library based on architecture?

    Or, like many other gems, have a ruby version and an x86-win version?

  • Luis Lavena

    Luis Lavena November 22nd, 2008 @ 11:13 AM

    Hey Guys,

    I believe, for the sake of stability of end-users systems, the sqlite3.dll must be removed from the repository and excluded from being bundled in any pure-ruby or native gems for any platform.

    Instead, users should be notified by a post install message or by instructions in the website that sqlite3, mysql and postgres binaries are required to exist and be available in the system prior to usage of the gem.

    Is the same about installing the headers or the libraries for other platforms, so why Window should be different?

    At this time, when users want sqlite3-ruby gem to work on Windows they simply download the binary from www.sqlite.org and put on the PATH.

    That is a common practice. The same for mysql-gem or any other adapter.

  • Luis Lavena

    Luis Lavena November 22nd, 2008 @ 11:17 AM

    Regarding the generation and release of the gems I have my reservations.

    There is no officila "head's up" in preparation for a release

    There is no official announcement in the list

    There is no changelog of things introduced

    There are no details of the things published in RubyForge

    I believe that is more important get fixed than doing a rake release and pushing the gems to RubyForge.

  • Michael Guterl

    Michael Guterl November 22nd, 2008 @ 11:19 AM

    I agree with Luis on all matters mentioned above.

    I think a new gem needs to be pushed quickly, random breakages of unrelated gems is not cool.

  • Dan Kubb (dkubb)

    Dan Kubb (dkubb) November 22nd, 2008 @ 05:56 PM

    I released the last set of gems so I'll take full responsibility for the mistakes that were made, and do my best to make sure they aren't repeated in the future.

    In my defense though, I was asked to do an emergency release of DM, DO and dm-more to fix a problem where code was using the Addressable 1.x API, and Addressable 2.0 was released with a non-backwards compatible API. This was the first I've ever released these gems, and there were a number of barriers: (I'm actally surprised more didn't go wrong)

    • No documentation on how to do it properly so I basically had to figure out as I went. I did start on some basic documentation tho: http://gist.github.com/26009
    • No one else was available who had done it before to give me more instruction than basically "use rake release". Sure people came onto IRC, and I asked them if there was anything special I needed to know, but I don't blame them for forgetting -- the release process is stupidly complex.
    • dm-more had dozens of errors that needed to be fixed before a release would be possible.
    • There were approximately 35 gems altogether that need to be released together, each one requires validation by hand.

    All in all I had to take a day off work, spent over 9 hours fixing the bugs in dm-more and preparing the release. I finished sometime between 2 and 3 AM on Tuesday morning. I think we could do it much faster again, but there's going to need to be a number of changes in the commit and release process.

    There is absolutely no way the release can take more than 10-15 minutes in the future. No one wants to do it if it's going to take hours, which is the reason why it's been so long since the last release, and why this release fell to me after being delayed for a couple of days. That's despite the fatal error that the Addressable API change caused that was affecting more than just DM, for example Merb was unusable too.

    The bottom line is that we need to change our process so that the code in the mainline is in a releasable state at all times, and that the release process is as automated as possible to prevent future problems. We also need to work out the process so that gems can be distributed that work on Ruby 1.8.6, 1.8.7 and in the future 1.9 and jRuby. And that's for Windows, BSD, Linux and other flavors of Unix too. This is a big job and one person can't possibly know everything about every area, so I'm asking for help from the experts.

    -- end rant --

    Luis, I will update the release process I'm documenting so those critical points don't get overlooked again. However I either need specific instructions on how to make sure the do_* gems are working on Windows and/or someone who knows what they are doing needs to fix what's broken in the repository. At this point I do not consider myself enough of an expert to manage the DataObjects project since I neither know C or programming for Windows. Someone else is going to need to be responsible for the repository and possibly the gem releases unless it can be automated enough that I could perform the gem release role.

    Michael, I can feel your pain but we're not going to do any more releases until we know for sure that the problem is fixed and that we haven't caused any new problems to appear.

  • Michael Guterl

    Michael Guterl November 25th, 2008 @ 05:59 AM

    Dan, no need to apologize, things happen. As developers I feel like we've all been in the same position before. I commend your efforts in coordinating a release, your work is very much appreciated!

  • Luis Lavena

    Luis Lavena November 25th, 2008 @ 06:32 AM

    I agree with Michael and I commented to this on the #datamapper channel.

    There was no need to apologize since I was the one moving already bundled sqlite3.dll from ext/ into lib/ to be able to come up with a hot release for Merb.

    On the subject about getting DO adapters working on Windows, I stepped back from RubyInstaller to work on rake-compiler project and I'm almost close to finish the cross compilation functionality.

    I believe the need for a CI to verify all those small projects.

    On a side note, I believe there is too much coupling between the parts (15 gems cannot depend each other) to be considered individual packages.

    There should be a minimum set of compatibility across these packages that don't require sucha huge release burden.

  • Dan Kubb (dkubb)

    Dan Kubb (dkubb) November 25th, 2008 @ 03:40 PM

    Luis: ah, I never really checked the history to see who caused it -- I don't believe in placing blame on any one person anyway. Aside from malicious sabotage (which this clearly was not), problems like this are related to a failure in the process, not an individual person.

    In this case the problem occurred because of a larger issue: we don't have a good process in place to test on all our target systems. CI would go a long way towards making that better. We also need to be thinking of the master branch for our projects as needing to be in a releasable state at all times. In the last gem release I saw alot of breakage that could've been easily caught has the specs been run before checking code in. (almost half of dm-more didn't pass it's specs)

    It's also important that if any of the dependencies change, the code needs to be tested with the updated package -- so for example a change in DO doesn't break dm-core. People need to be running some regression tests before check-in, and need to be aware of the dependencies. I see CI as more of a safety net after local verification is done.

    Also, I'm going to bring it up at the contributor's summit that we need to adopt an organization similar to what merb's done, with combining dm-core and dm-more together. Before checkin you'd be expected to run rake specs from the root, which would test dm-core, and all of dm-more. At the same time, we've got to slim down dm-more to just include the essentials, and move the optional plugins to a dm-plugins repo. I don't necessarily know if we should combine DO and dm-core though, because the skill sets to manage the two are so different.

  • Luis Lavena

    Luis Lavena November 25th, 2008 @ 04:04 PM

    bq. Luis: ah, I never really checked the history to see who caused it -- I don't believe in placing blame on any one person anyway. Aside from malicious sabotage (which this clearly was not), problems like this are related to a failure in the process, not an individual person.

    I'm not blaming anyone, I just checked my comments in github and found that I already left some comments back then on that commit.

    My email was just ot expose that we are aware of this issue.

    Those things slept through the cracks due the same you mention: the lack of release process (and not running specs before commit).

    Following the commit bit model of Rubinius requires the harness ensuring everything is OK.

    Also the lack of diversity of platforms from developers.

    Those things like hardcoded extensions has no reason to be committed, yet still most of the developers ignores these facts. Mea culpa: I did it several times too (oops!)

    To be honest, there is no guide to hacking this stuff, just experience.

    I believe DO should keep separated from DM and be handled in a different release cycle.

    There aren't new changes introduced in DO to justify a bump of version every new DM release.

    We just stick to one version and on next DM release, we review the compatibility and bump or stick to it.

    Also there are other developers that can take advantage of DO standalone.

  • Dan Kubb (dkubb)

    Dan Kubb (dkubb) November 27th, 2008 @ 07:29 PM

    • State changed from “open” to “resolved”
    • Assigned user cleared.

    This should now be fixed as of DO 0.9.9: http://is.gd/9k4J

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »