Datamapper DataMapper

Comments and changes to this ticket

• 

  Yehuda Katz November 9th, 2007 @ 05:32 PM

  • Assigned user set to “Rein Henrichs”
  • State changed from “new” to “open”

• You flagged this item as spam.
  

  Sam Smoot December 28th, 2007 @ 10:48 PM

  • Milestone cleared.

  Definitely doesn't look like this is happening in 0.3.0. Let's shoot for 0.4.0. 4 is my fave number anyways. :-)

• 

  Paul Sadauskas (Rando) January 6th, 2008 @ 01:25 PM

  I would like to discuss the usefulness of this feature. Please see my thread about this here: http://groups.google.com/group/d...

• You flagged this item as spam.
  

  Michael Boutros February 10th, 2008 @ 11:33 PM

  I think that there are a lot of people who are willing to work on this, but no one is sure exactly what to do, because it has not yet been decided 100% how it should work.

  I'm leaning towards saying that if you're altering table columns, you're probably in development so any data destroyed is no problem. However, sometimes you do need to change/add things later on in production, so that should be taken into account.

  I know what I would do is just migrate the database manually and then change the property calls in my model class. However, that's quite hackish, so there does need to be a solution.

  What say you (to the 10 or so people watching this ticket)?

• You flagged this item as spam.
  

  Marcin 'swistak' Raczkowski February 11th, 2008 @ 05:49 AM

  Well.

  My plan and proposition was to create milestones.

  Unless you're in development mode migrations occour automatically - or when you call rake migrate BUT by alter table - so removing property will remove column - removing relationship and/or model will destroy table etc.

  when you deploy your application you create milestone, models are copied to diffirent directory, and are put into namespace - for example R1

  now if you plan to deploy new version you make another milestone - and create migration between 2 milestones like this:

  directory structure:

  current

  milestones\

  \1

  \2

  \3

  migrations\

  1_to_2.rb

  1_to_3.rb

  now lets say we have model where we want to change one to one relationship to one table

  R1::Parent.find(:all).each do |record|

  new_record = R3::Merged.new

  1. mege records here
  2. using for example record.child

  new_record.save

  end

  R1::Parent.drop # for example

  R1::Child.drop

  if there's name conflict - for example we want new table to have the same name - we can use temp name like this:

  R2::Same.table_name = "temp_name"

  R2::Same.create

  1. do migration here from R1:Same

  R2::Same.change_table_name "sames"

  this is of course example, but that was my plan - we can automate creating migrations and milestones with rake of course

• 

  Ted Han (knowtheory) February 11th, 2008 @ 07:25 AM

  Guys, there really isn't a need to produce a historical record of what your database structure is. In fact this is something i find really annoying about ActiveRecord, you can have like 12 migrations in a row, which could in theory just add a column, drop it, add it again, drop it, add... etc. There's no reason for it, and shit like that just takes up space (and is a pain in the ass to trawl). Take the parsimonious way out and just match current DB structure to the DB structure that you derive from current Model status.

  If you need to derive previous DB structure, just check out a previous version and automigrate. So long as you can get to the currently desired DB from any other sort of DB structure, then the steps you took to get there are effectively irrelevant, because all you care about is the end output (i.e. does your current db structure match what you want your ORM to map to :P ).

  Automigration should also give you a bunch of other options like "back up target database(s) before i tear up all your shit".

• You flagged this item as spam.
  

  jmoses February 11th, 2008 @ 07:31 AM

  I agree with knowtheory. I just want to be able to say "my DB is out of date with my models, fix it in the least destructive way possible."

  Backing up would be a "nice to have" but I don't really see the migration code being the place to have that.

• You flagged this item as spam.
  

  Sam Smoot February 11th, 2008 @ 09:46 AM

  Either way sounds good to me. Or pull it from Og. I don't think anyone will be too dissapointed with any of those solutions, so I leave this one up to you guys.

  What it really needs I think is someone just to sit down and do it. A piece at a time. Get column-adds in there. Tack on index reflection and creation (dropping the index with the matching name). Recluster the table if you end up modifying the clustered-index (postgres only I think). Create missing tables.

  You can't drop or rename columns without user intervention because one of the major goals of DM is to work with integration databases, which means it's expected that the table may have more columns than the model.

  You also can't blindly alter columns. There's probably some safe defaults like INT(4) to DECIMAL, or VARCHAR to TEXT. For the most part though, we probably just want to "fail-fast".

  I doubt anyone here is really interested in writing a robust merge-tool, so we'll leave that one alone. ;-) When it's not obvious, we fail and leave it up to the user.

• 

  Ted Han (knowtheory) February 11th, 2008 @ 10:00 AM

  Hey I'm more than willing to jump on it :)

  I definitely think that a smart non-destructive filter should be easy to use. I.e. easy (reversible) things should be done automatically, and users should have to approve things that irrevocably change your db.

• You flagged this item as spam.
  

  Mayo Jordanov February 11th, 2008 @ 10:54 AM

  I would side with knowtheory too.

  One thing that should always happen, imho, is not to run migration automatically. I think a file should be generated and it be up to the user to run it. OR even better, when a migration is about to run a prompt that will let her review all the queries/steps that would happen if the migration were to run and once the user signs off with "y" to "Are you sure you want to run this" the migration would be ran. Anyway, how this works is irrelevant right now, but if there is no historical record of migrations kept, it would be nice to be able to review what migrations will do to my DB.

• You flagged this item as spam.
  

  jmoses February 11th, 2008 @ 10:58 AM

  @knowtheory: If you seriously get into implementing this, and want some Ruby-smart/Datamapper-dumb help, drop me a line. I'm going to start looking at this a little (if I can get all the specs to run), but I haven't done any DM work yet.

• 

  Paul Sadauskas (Rando) February 11th, 2008 @ 11:34 AM

  I still think this automagic figure out how to get from schema A to schema B is a pretty worthless feature. In the case of development & testing modes, just drop and recreate the tables. In the case of production tables (with production data), don't automagically do anything. Let the user write a testable migration, and run that. If we do the crazy migrations, we'll end up with a mess of heuristics that will be non-trivial to test, and probably still not do the right thing in most cases, and the user will have to write a migration anyway.

• You flagged this item as spam.
  

  Marcin 'swistak' Raczkowski February 11th, 2008 @ 12:20 PM

  well knowtheory - and how you imagine migrating from one DB structure to another? without loosing data in production enviroment?

  You need to update data in non-trivial way

  okey for me plan is simple:

  • automatic migrations in development mode (without loosing data if possible)
  • rake tasks to generate migrations for production deployment - for serious milestones - so you'll want to have two versions of models - and testable migration right?

  mayby i described it wrong - but for me what's most annoying in rails is that you write all this migrations - I don't want to do that

  • in development mode - database should be updated once i save model and run rake task or start webserver.

  when i deploy something i want to save models(and old database structure) and be allowed to work in development mode again, then when comes time for version 2.0 - where i decided that by the book normalized structure is fucking slow - and i want cache for counters and tags - i can create migration (with rake and from template) where i can access two versions of models - currently deployed - and current - with cache - and when i deploy code that uses new database - i run rake db:migrate, and voila - i have models matching database - simple?

  I see that someone wants to do crazy-migrations - I'm all in favor,

  For my part i'm willing to do migrations between milestones - having both options is better then one i guess?

• 

  Rein Henrichs February 11th, 2008 @ 12:52 PM

  Folks,

  I've spent a few weeks wrangling with this issue (in my mind if not in code) and I tend to side with Paul here. I don't see "crazy" automigrations as a production-ready tool, no matter how they're done. It would be far easier and imo more useful to write an improved "normal" migrations handler that doesn't suffer from the issues that AR has, migration number collisions being most obvious. We're almost done with that as it is.

  There have been a few ideas put forward here that I'd like to discuss:

  "Smart" non-destructive filter: I don't think that automatically making changes, even presumably reversible changes, is an acceptable solution. If a "name" field is removed from a model and a "title" field is added, I would definitely not want my automigrations tool to automatically rename "name" to "title". If anything, I would want to be given the option. We could spend a lot of cycles trying to write heuristics to cover these situations for little real gain. I wouldn't use a tool that makes that sort of assumptions and I certainly wouldn't let it go anywhere near my production environment. Humans are better at making these sort of decisions.

  Versioned model directories: I'm sorry, but I don't see how this is really a desirable solution. I certainly don't want multiple model directories cluttering up my application. Testing would be a huge PITA as well.

  One idea that I did find interesting was Mayo's of creating a migration. My original idea was to use some sort of delta chain with a user prompting system to determine and store changes between database versions. Ideally, the delta format would be human readable. It seems obvious to use standard migrations as the delta format.

  If we do move forward on this, I would definitely suggest that automigrations generate standard migrations. I would also suggest that in the case of any non-trivial change the automigration either generate the migration without running it (which would be easy) or prompt the user to resolve the "merge conflict". Either way, the automigrations should generate a new standard migration, which is then run.

  This is the only idea I've heard so far that seems to solve the problem at hand without introducing more.

• You flagged this item as spam.
  

  Marcin 'swistak' Raczkowski February 12th, 2008 @ 06:47 AM

  Rein - I full agree with you.

  But about versioning models - i thought about storing delta - but what if you had triggers in old models? custom accessors etc.

  you can of course do incremental changes - then when you have to make migration you'd checkout old version from subversion - and generate kind of diff. Anyway i'll try to implement this soon

• You flagged this item as spam.
  

  Nathan Herald February 13th, 2008 @ 11:08 AM

  What I am interested in is something akin to what the Err dudes wrote: http://errtheblog.com/posts/65-a...

  It uses the activerecord schema.rb to automigrate the db.

  Also, has the issue of indexes been brought up? This is something I am really interested in being automagic as well.

  My ideas of moving to production would be using the automigrate feature to generate a single migration file (essentially a diff from a point I choose) as a starting point for my production migration. I could then alter/test the migration myself and run it with my own capistrano recipe or just manually.

• 

  Ted Han (knowtheory) February 14th, 2008 @ 11:14 PM

  So i think the tool being described here does the following:

  1) Reads current DB/Store structure and current model structure

  2) Check current model structure does not map properly to current DB/Store structure (i.e. models have been structurally changed)

  3) If there is a mismatch, spit out a Delta/Migration file which will realign the DB/Store with the Models, for all trivial changes (trivial changes being ones where there is a 1 to 1 correspondence between store and model structure. E.g. if a new model has suddenly appeared, create a new table for this model).

  4) For non-trivial changes (changes that are ambiguous, or may have multiple possible solutions), the user should be prompted for possible options (among which should be "just spit a stub for this change into the Delta/Migration file").

  And optionally the tool will run your migration/delta file for you.

  Thoughts?

  Suggestions?

• You flagged this item as spam.
  

  jmoses February 15th, 2008 @ 06:49 AM

  I think that sounds like a decent compromise between all the different suggestions.

  Personally, that would fit my needs pretty well.

• You flagged this item as spam.
  

  Sam Smoot February 15th, 2008 @ 10:03 AM

  knowtheory,

  Sounds good to me. I think in general the interface should be as simple as:

  rake dm:db:automigrate
  

  Then that script throws a warning that you should review it. Other than index updates, there doesn't seem to be a case where you could safely not review the migration except for a missing table.

  Even adding columns is troublesome because you don't know if that means renaming an unmapped column.

  I know I keep saying this, but it's important to keep in mind the Integration Database, and the expectation that it'll be very common to NOT map every column in a table. That single concern makes an almost completely automated process impossible. If we were dealing with an Application Database, sure, we could get pretty close...

  On the other hand, if we, down the line want to throw in an opt-in option into DataMapper::setup to specify :integrated => false, then we could extend the crazy-migrations to cover a lot more scenarios.

  But I believe that should definitely be a post 1.0 feature (if ever).

  The point is, outside of creating a new table, there really are no trivial use-cases for this functionality.

  I think there's also definitely value in the existing dumb DROP/CREATE auto-migrations for local development in conjunction with fixtures and autotest. So this crazy-migrations task is probably not something we want to replace the existing auto-migrations.

  In fact, thinking about it that way, the rake task should actually probably be more like:

  rake dm:gen:migrations
  

  Anyways, back to indexes... they're the one thing we can auto-migrate I think. Is there an index that matches these constraints? No? Go ahead and generate one. Especially since it's easy to imagine your different environments being out of sync with regards to indexes, and a linear apply-once migration probably isn't the best way to handle indexes. IMO they're not a database-evolution issue. They're a binary "are they present and correct?" issue.

  Any comments? Agree/Disagree with any of that?

• You flagged this item as spam.
  

  Nathan Herald February 15th, 2008 @ 10:38 AM

  As far as renaming columns, I am ok with removing the column that is now gone and adding in the new column. I don't think it's wise to try to tell if something has been renamed or not.

• You flagged this item as spam.
  

  jmoses February 15th, 2008 @ 10:50 AM

  Personally I'd prefer a prompt about what to do in that case, rather than just assuming. Or at least a notification so I can manually edit the generated migration before I run it.

• You flagged this item as spam.
  

  Mayo Jordanov February 15th, 2008 @ 10:59 AM

  Sounds good, and I agree there should be no magic with renaming columns. Drop old one, create new one is as far as the migration should go. The dev should change that to rename if in fact it is one.

  I like Sam's idea to have this as different task. I originally thought I'd hate the drop/create migrations and didn't really like using them (especially because I can't reload the old data or repopulate with fixtures) but I'm warming up to it in some cases to a point were I'd hate to see it go :)

  I'm having second thoughts about prompting the user to autorun the migration from the point of automation. I don't know where, but if somebody was to automate generation for multiple things at the same time, you have to deal with the whole Y/N answer issue and somehow work around it. Maybe DROP/CREATE migrations run automatically, crazy migrations just generate?

• You flagged this item as spam.
  

  Sam Smoot February 15th, 2008 @ 12:59 PM

  Nathan,

  Right, I guess what I'm trying to say is that we shouldn't handle renames. We should gen a migration for you, and in the case of a rename, it'd default to add_column, but it's up to you to edit/adjust that.

  So basically CrazyMigrations aren't going to auto-anything. It's just a generator to best guess a migration and give you a starting point to review/edit as needed.

  AutoMigrations will continue to live on in their current form, no changes, just a full drop/create in development mode.

• You flagged this item as spam.
  

  Nathan Herald February 15th, 2008 @ 12:56 PM

  I like the idea of generating a migration file for the developer to review.

  I myself would write my own rake task to do it automatically, and clean up the migration files, and I am fine with that.

  I think that generating migration files might also make it easier to write a tool to combine a set of migration files into one migration, for use in a rake task to be run on the production server. However, migration files are one of the current downfalls of activerecord, because of the naming. It is too easy to get migrations numbered the same in a team environment.

  I, myself, will end up using a workflow like this:

  • automagically migrate with one rake task during development (not actually producing migration files or I would delete them or archive them somewhere and not version control them)
  • checkout the old version of the app before deploying and run a diff rake command to generate a starter migration file for the production server (or maybe have it look at the current files residing on the server and diff from there)
  • use a capistrano task that will run a local migration file on a remote server (probly override db:migrate) or rake it by hand if it's pretty crazy and I don't trust myself

  I understand that doing things automagically is dangerous and I don't think the current automigrations should be replaced by any this. I do think that automagicness without migration files should be a choice tho.

  I suppose we could have rake rake dm:gen:migrations and rake db:gen:migrations:apply

  And if the apply task wanted to leave the migration files, that's cool I can hookup an after task to archive them away.

• 

  Dan Kubb (dkubb) April 15th, 2008 @ 03:35 AM

  • Milestone cleared.

• You flagged this item as spam.
  

  Sam Smoot April 27th, 2008 @ 10:12 PM

  • State changed from “open” to “invalid”

  Ok, I'm killing this until someone starts contributing some code to it. :-)