ETL framework for loading data into a Rails application

Question

I need to upload data for my Rails application from several providers (XML feed-based REST / SOAP) to the database on a regular basis. I have written a set of Rake tasks that start with cron jobs ever created. Each task reaches the endpoint of the partner-partner, analyzes the channel and loads it into the database.

Instead of writing Rake tasks, should an ETL scheme such as ActiveWarehouse ( http://activewarehouse.rubyforge.org/etl/ ) be used? Any suggestions on the best way to do this in Rails?

Answer 1

If you just load the data into a set of tables, and the example of use is simple, for example, simply adding new records or updating the basic ones, and your load meets your requirements, I would stick to this. You can, of course, use ActiveWarehouse, but that sounds like overkill. If, however, you need to support changing sizes (i.e., keep a history of data changes over time) or other data warehouse functions, then something like ActiveWarehouse will start to matter more.